That’s where this blog post comes in!
Cypress’ creators want you to run your tests against local or CI environments. Running tests in such environments enables you to use shortcuts to speed up your tests and simplify test setup. But not every tester has the laptop horsepower or the knowledge to properly run a local environment. Deeply integrating a test suite with existing CI pipelines is no mean feat; that’s why God made DevOps specialists. But we’ve already established that they tend to have a lot on their plate.
One solution* to this dilemma is to just run your tests against a production-like environment. We usually call these environments ‘Dev’, ‘Int’, ‘QA’ or ‘Staging’. Whatever you call them, they’re typically production-like environments where the whole team tests features and fixes before deploying them to production.
* A better solution would be to hire more DevOps specialists, get a better laptop and/or upskill yourself. But Rome wasn’t built in a day.
The case against
Running your tests against production-like environments has a few common drawbacks:
- We often use these environments for many different purposes. Your automated test runs might be competing with intensive performance testing or destructive security audits. Not to mention multiple human users with competing agendas.
- It’s not feasible to have complete control of the data and config for automated testing purposes. Doing so might hinder other valuable activities such as exploratory testing and stakeholder demos.
- We tend to hook up these environments to other ‘real’ components. As such, any tests you run on them may create unwanted load on third party services.
- Likewise, if your test environment is connected to multiple third party services then your tests will depend on these services. For your tests to pass, all services must be fully operational and able to handle the load created by intensive test runs.
So it’s clear that running your primary Cypress test suite on an isolated/sandboxed environment is preferable for a variety of reasons. But what if that’s not an option? Perhaps it’s not yet an option because the DevOps investment won’t be made until Cypress is demonstrating real value?
The case in favour
Despite these drawbacks, in my experience Cypress tests still run pretty well on production-like environments. Despite their threats, the Cypress police have yet to come knocking at my door. Running these tests regularly helps me and my colleagues to quickly spot multi-system failures. Inevitably, my dev colleagues have quickly joined me in advocating for Cypress to be integrated properly into CI/CD pipelines.
Running Cypress against production-like environments is not a perfect solution, perhaps not even a good one. But it will likely add some value, helping you and your team to deliver higher quality features more quickly. And some value is better than no value, right?
So how do you get there?
In recent years there has been an explosion of CI/CD tools with a Configuration as Code (CaC) approach to code deployment, typically using YAML config files. Gone are the maze of Jenkins dashboards and standalone jobs. Instead, CaC tools typically use linear pipelines that run jobs in a predefined order. Sure, you can put in some conditional logic to run different jobs in different scenarios, but it’s often less chaotic than a sprawling Jenkins setup.
Examples of CaC tools include, but are not limited to:
In this blog post I’ll be using examples from GitLab CI and GitHub Actions, as those are the tools I’m familiar with. However you should be able to adapt them for your preferred tool with minimal difficulty.
Standalone Cypress job using GitHub Actions
Here’s the barebones GitHub Actions YAML file for my Cypress Starter Kit:
name: Run Cypress Tests on: push: branches: [ main ] pull_request: branches: [ main ] jobs: cypress-run: runs-on: ubuntu-16.04 # Cypress Docker image with Chrome v83 # and Firefox v77 pre-installed container: cypress/browsers:node12.18.0-chrome83-ff77 steps: - uses: actions/checkout@v1 - uses: cypress-io/github-action@v2 with: browser: chrome - uses: actions/upload-artifact@v1 if: failure() with: name: cypress-screenshots path: tests/cypress/screenshots - uses: actions/upload-artifact@v1 if: always() with: name: cypress-videos path: tests/cypress/videos
This YAML file tells GitHub Actions to do several things:
- Run this pipeline whenever a branch is merged to the
- Also Run this pipeline whenever a pull request is raised against the
- Triggers a job called
cypress-runon an Ubuntu 16.04 environment.
- Build and deploy a Docker image called
cypress/browsers:node12.18.0-chrome83-ff77to run the steps.
- Run the default GitHub checkout action to download the repository into the Docker image.
- Run the Cypress tests in Chrome using the official Cypress GitHub action.
- Stores screenshots (if tests fail) and videos (always) as artifacts for later viewing.
This pipeline is extremely simple, for two key reasons:
- I’m running my tests against someone else’s site, purely for demo purposes. As such, I don’t have to deploy the site before running the tests. Also, I’m assuming that the site I’m testing against will stay online indefinitely.
- I’m only testing against a single environment. There is no need to specify environment variables per test run, because they’ll always be the same.
If you’re already using GitHub Actions and are storing your Cypress tests in a standalone repo, you could copy the above file into your repository and tweak it as needed. You’re welcome.
Post-deployment Cypress job
Let’s use another example, this time using GitLab CI, to show how we might include a Cypress test run in a full deployment pipeline.
Imagine the following CI pipeline:
Setup > Check > Build > Validate > Deploy
This pipeline is designed to find problems as early as possible. Before running some basic code quality checks, the CI pipeline installs the application dependencies using a tool such as NPM or Yarn. If the
Check stage passes, the pipeline then does a full build of the app. If the build succeeds, then the pipeline starts the
Validate step to check that the app is working as the developers intended. Finally, the pipeline deploys the app to a designated environment – assuming that specific criteria (e.g. Git branch) are met.
We can represent this linear pipeline in a
.gitlab-ci.yml file like so:
stages: - setup - check - build - validate - deploy
Each stage will have one or more jobs defined, each with criteria determining when and how they will run. The
.gitlab-ci.yml file will also hold other important information like environment variables and the overall condition(s) for running the pipeline.
Extending the pipeline
Let’s add a post-deployment stage to our example pipeline:
stages: - setup - check - build - validate - deploy - test:awsdev
Next, we must define the corresponding Cypress job:
cypress-e2e-chrome: stage: test:awsdev extends: - .node image: cypress/browsers:node14.7.0-chrome84 script: - yarn cypress run --browser chrome artifacts: expire_in: 1 week when: always paths: - tests/cypress/screenshots - tests/cypress/videos
This job (which was written by a DevOps-literate colleague) tells GitLab CI to perform the following actions:
- Run a job called
cypress-e2e-chromeas part of the
- Re-use the
.nodeconfig (defined elsewhere in the file) as a baseline.
- Build and deploy a Docker image called
cypress/browsers:node14.7.0-chrome84to run the steps.
- Run the command
yarn cypress run --browser chrome, defined in the package.json file.
- Store the contents of the screenshots and videos folders as artifacts for up to one week.
And here’s how it looks in the pipeline:
This is a great first step. But what happens if the Cypress tests are out of date, or if there’s a known issue with a feature? This basic config will fail the entire pipeline when Cypress tests are failing. The deployment takes place before we run the Cypress tests, so deployment isn’t actually being blocked here.
Improving Cypress job behaviour
Let’s change our pipeline configuration to reflect the current reality. At the bottom of the
cypress-e2e-chrome YAML config, we can add the following line:
when: manual in this way causes the job to become optional and non-blocking. This means that a human must manually trigger the job, and that this job won’t fail the whole pipeline.
This is what a manual job looks like when it’s yet to be triggered:
And here’s what a failing-but-optional job looks like:
But what about situations where we do want to run Cypress automatically?
Controlling job behaviour with rules
To add more nuance and clarity to our GitLab CI job config, we can use rules to control when and how the job will run.
Here’s an example ruleset, which replaces the
when: manual line:
rules: - if: $CYPRESS when: always allow_failure: false - if: $CI_COMMIT_BRANCH == "main" when: always allow_failure: true - when: manual allow_failure: true
This ruleset works like this:
- If a user sets the $CYPRESS environment variable when they run the pipeline, the job will run automatically. If the Cypress tests fail, the whole pipeline will fail, preventing merging.
- When the Git branch is
main, the job will run automatically. If the Cypress tests fail, the pipeline will show as ‘passing, with warnings’.
- In all other cases, the job must be triggered manually by a human. As with the
mainbranch, the failure of the Cypress tests won’t fail the whole pipeline.
Other possible scenarios could include:
- If there are any changes to the Cypress test code, run the tests automatically. Fail the whole pipeline if the tests fail.
- Run the Cypress job automatically when the branch name includes ‘cypress’
- Run the job automatically on mainline branches like
develop, but make it optional on all other branches.
As with everything in software development, the correct conditions to apply will depend on your requirements and context.
GitLab CI Cypress job – final result
If we put together everything in this section, here’s what we end up with:
stages: - setup - check - build - validate - deploy - test:awsdev # existing pipeline config goes here cypress-e2e-chrome: stage: test:awsdev extends: - .node image: cypress/browsers:node14.7.0-chrome84 script: - yarn cypress run --browser chrome artifacts: expire_in: 1 week when: always paths: - tests/cypress/screenshots - tests/cypress/videos rules: - if: $CYPRESS when: always allow_failure: false - if: $CI_COMMIT_BRANCH == "main" when: always allow_failure: true - when: manual allow_failure: true
Do with this example what you will. You’re welcome.
- GitLab CI documentation
- GitHub Actions documentation
- My Cypress Starter Kit, with an example package.json and standalone GitHub Actions flow.
- Cypress’ Continuous Integration Docs
- The #automation channel on the Ministry of Testing Slack community
- Friendly DevOps specialists, without whom I could not have written this blog post or understood its contents.