Update in November 2024: This Cypress post was written in 2021. As such, the code samples are likely to be outdated and will probably will not work with the latest versions of Cypress, GitHub Actions and GitLab CI. Hopefully the advice itself is still useful!

Does this Cypress dilemma sound familiar? You’re an exploratory tester who happens to know a bit of JavaScript, so you decide to give Cypress a try. It’s got great documentation (seriously, it’s amazing), requires extremely minimal setup (check out my Cypress Starter Kit) and can provide a lot of value very quickly. After a while, you might find yourself running the tests multiple times a day. By now, you’ve probably demoed your tests to the rest of the team. “Wouldn’t it be great,” you think, “if these tests ran automatically so everyone could benefit from their insight?” There’s only one problem – your DevOps specialists are over-worked and under-resourced. What’s more, you don’t have the know-how to properly integrate Cypress into the frontend dev workflow.

That’s where this blog post comes in!

Cypress’ creators want you to run your tests against local or CI environments. Running tests in such environments enables you to use shortcuts to speed up your tests and simplify test setup. But not every tester has the laptop horsepower or the knowledge to properly run a local environment. Deeply integrating a test suite with existing CI pipelines is no mean feat; that’s why God made DevOps specialists. But we’ve already established that they tend to have a lot on their plate.

One solution* to this dilemma is to just run your tests against a production-like environment. We usually call these environments ‘Dev’, ‘Int’, ‘QA’ or ‘Staging’. Whatever you call them, they’re typically production-like environments where the whole team tests features and fixes before deploying them to production.

* A better solution would be to hire more DevOps specialists, get a better laptop and/or upskill yourself. But Rome wasn’t built in a day.

The case against

Running your tests against production-like environments has a few common drawbacks:

• We often use these environments for many different purposes. Your automated test runs might be competing with intensive performance testing or destructive security audits. Not to mention multiple human users with competing agendas.
• It’s not feasible to have complete control of the data and config for automated testing purposes. Doing so might hinder other valuable activities such as exploratory testing and stakeholder demos.
• We tend to hook up these environments to other ‘real’ components. As such, any tests you run on them may create unwanted load on third party services.
• Likewise, if your test environment is connected to multiple third party services then your tests will depend on these services. For your tests to pass, all services must be fully operational and able to handle the load created by intensive test runs.

So it’s clear that running your primary Cypress test suite on an isolated/sandboxed environment is preferable for a variety of reasons. But what if that’s not an option? Perhaps it’s not yet an option because the DevOps investment won’t be made until Cypress is demonstrating real value?

The case in favour

Despite these drawbacks, in my experience Cypress tests still run pretty well on production-like environments. Despite their threats, the Cypress police have yet to come knocking at my door. Running these tests regularly helps me and my colleagues to quickly spot multi-system failures. Inevitably, my dev colleagues have quickly joined me in advocating for Cypress to be integrated properly into CI/CD pipelines.

Running Cypress against production-like environments is not a perfect solution, perhaps not even a good one. But it will likely add some value, helping you and your team to deliver higher quality features more quickly. And some value is better than no value, right?

So how do you get there?

Config-as-Code FTW

In recent years there has been an explosion of CI/CD tools with a Configuration as Code (CaC) approach to code deployment, typically using YAML config files. Gone are the maze of Jenkins dashboards and standalone jobs. Instead, CaC tools typically use linear pipelines that run jobs in a predefined order. Sure, you can put in some conditional logic to run different jobs in different scenarios, but it’s often less chaotic than a sprawling Jenkins setup.

Examples of CaC tools include, but are not limited to:

• GitHub Actions
• GitLab CI
• BitBucket Pipelines
• CircleCI
• GoCD

In this blog post I’ll be using examples from GitLab CI and GitHub Actions, as those are the tools I’m familiar with. However you should be able to adapt them for your preferred tool with minimal difficulty.

Standalone Cypress job using GitHub Actions

Here’s the barebones GitHub Actions YAML file for my Cypress Starter Kit:

name: Run Cypress Tests

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  cypress-run:
    runs-on: ubuntu-16.04
    # Cypress Docker image with Chrome v83
    # and Firefox v77 pre-installed
    container: cypress/browsers:node12.18.0-chrome83-ff77
    steps:
      - uses: actions/checkout@v1
      - uses: cypress-io/github-action@v2
        with:
          browser: chrome
      - uses: actions/upload-artifact@v1
        if: failure()
        with:
          name: cypress-screenshots
          path: tests/cypress/screenshots
      - uses: actions/upload-artifact@v1
        if: always()
        with:
          name: cypress-videos
          path: tests/cypress/videos

This YAML file tells GitHub Actions to do several things:

• Run this pipeline whenever a branch is merged to the main branch.
• Also Run this pipeline whenever a pull request is raised against the main branch.
• Triggers a job called cypress-run on an Ubuntu 16.04 environment.
• Build and deploy a Docker image called cypress/browsers:node12.18.0-chrome83-ff77 to run the steps.
• Run the default GitHub checkout action to download the repository into the Docker image.
• Run the Cypress tests in Chrome using the official Cypress GitHub action.
• Stores screenshots (if tests fail) and videos (always) as artifacts for later viewing.

This pipeline is extremely simple, for two key reasons:

• I’m running my tests against someone else’s site, purely for demo purposes. As such, I don’t have to deploy the site before running the tests. Also, I’m assuming that the site I’m testing against will stay online indefinitely.
• I’m only testing against a single environment. There is no need to specify environment variables per test run, because they’ll always be the same.

If you’re already using GitHub Actions and are storing your Cypress tests in a standalone repo, you could copy the above file into your repository and tweak it as needed. You’re welcome.

Post-deployment Cypress job

Let’s use another example, this time using GitLab CI, to show how we might include a Cypress test run in a full deployment pipeline.

Imagine the following CI pipeline:

Setup > Check > Build > Validate > Deploy

This pipeline is designed to find problems as early as possible. Before running some basic code quality checks, the CI pipeline installs the application dependencies using a tool such as NPM or Yarn. If the Check stage passes, the pipeline then does a full build of the app. If the build succeeds, then the pipeline starts the Validate step to check that the app is working as the developers intended. Finally, the pipeline deploys the app to a designated environment – assuming that specific criteria (e.g. Git branch) are met.

We can represent this linear pipeline in a .gitlab-ci.yml file like so:

stages:
  - setup
  - check
  - build
  - validate
  - deploy

Each stage will have one or more jobs defined, each with criteria determining when and how they will run. The .gitlab-ci.yml file will also hold other important information like environment variables and the overall condition(s) for running the pipeline.

Extending the pipeline

Let’s add a post-deployment stage to our example pipeline:

stages:
  - setup
  - check
  - build
  - validate
  - deploy
  - test:awsdev

Next, we must define the corresponding Cypress job:

cypress-e2e-chrome:
  stage: test:awsdev
  extends:
    - .node
  image: cypress/browsers:node14.7.0-chrome84
  script:
    - yarn cypress run --browser chrome
  artifacts:
    expire_in: 1 week
    when: always
    paths:
      - tests/cypress/screenshots
      - tests/cypress/videos

This job (which was written by a DevOps-literate colleague) tells GitLab CI to perform the following actions:

• Run a job called cypress-e2e-chrome as part of the test:awsdev pipeline stage.
• Re-use the .node config (defined elsewhere in the file) as a baseline.
• Build and deploy a Docker image called cypress/browsers:node14.7.0-chrome84 to run the steps.
• Run the command yarn cypress run --browser chrome , defined in the package.json file.
• Store the contents of the screenshots and videos folders as artifacts for up to one week.

This is a great first step. But what happens if the Cypress tests are out of date, or if there’s a known issue with a feature? This basic config will fail the entire pipeline when Cypress tests are failing. The deployment takes place before we run the Cypress tests, so deployment isn’t actually being blocked here.

Improving Cypress job behaviour

Let’s change our pipeline configuration to reflect the current reality. At the bottom of the cypress-e2e-chrome YAML config, we can add the following line:

  when: manual

Specifying when: manual in this way causes the job to become optional and non-blocking. This means that a human must manually trigger the job, and that this job won’t fail the whole pipeline.

This is what a manual job looks like when it’s yet to be triggered:

And here’s what a failing-but-optional job looks like:

But what about situations where we do want to run Cypress automatically?

Controlling job behaviour with rules

To add more nuance and clarity to our GitLab CI job config, we can use rules to control when and how the job will run.

Here’s an example ruleset, which replaces the when: manual line:

  rules:
    - if: $CYPRESS
      when: always
      allow_failure: false
    - if: $CI_COMMIT_BRANCH == "main"
      when: always
      allow_failure: true
    - when: manual
      allow_failure: true

This ruleset works like this:

• If a user sets the $CYPRESS environment variable when they run the pipeline, the job will run automatically. If the Cypress tests fail, the whole pipeline will fail, preventing merging.
• When the Git branch is main, the job will run automatically. If the Cypress tests fail, the pipeline will show as ‘passing, with warnings’.
• In all other cases, the job must be triggered manually by a human. As with the main branch, the failure of the Cypress tests won’t fail the whole pipeline.

Other possible scenarios could include:

• If there are any changes to the Cypress test code, run the tests automatically. Fail the whole pipeline if the tests fail.
• Run the Cypress job automatically when the branch name includes ‘cypress’
• Run the job automatically on mainline branches like main or develop, but make it optional on all other branches.

As with everything in software development, the correct conditions to apply will depend on your requirements and context.

GitLab CI Cypress job – final result

If we put together everything in this section, here’s what we end up with:

stages:
  - setup
  - check
  - build
  - validate
  - deploy
  - test:awsdev

# existing pipeline config goes here

cypress-e2e-chrome:
  stage: test:awsdev
  extends:
    - .node
  image: cypress/browsers:node14.7.0-chrome84
  script:
    - yarn cypress run --browser chrome
  artifacts:
    expire_in: 1 week
    when: always
    paths:
      - tests/cypress/screenshots
      - tests/cypress/videos
  rules:
    - if: $CYPRESS
      when: always
      allow_failure: false
    - if: $CI_COMMIT_BRANCH == "main"
      when: always
      allow_failure: true
    - when: manual
      allow_failure: true

Do with this example what you will. You’re welcome.

Additional Resources

• GitLab CI documentation
• GitHub Actions documentation
• My Cypress Starter Kit, with an example package.json and standalone GitHub Actions flow.
• Cypress’ Continuous Integration Docs
• The #automation channel on the Ministry of Testing Slack community
• Friendly DevOps specialists, without whom I could not have written this blog post or understood its contents.