Implementing Continuous Integration (CI) with End to End (E2E) Testing Using Cypress, Percy and CircleCI

Testing is one of the most important aspects of a good continuous integration/deployment pipeline. Without robust testing, we cannot guarantee that new versions of the system will work correctly or even worse, break existing working code.

Today I want to show our experience creating a continuous integration pipeline using Circle CI and Cypress. In Diffgram we needed to create an automated way of validating our new features and testing all the existing components of our frontend service.

Then end goal was to integrate our existing E2E testing tool Cypress and Percy to automate testing every time new PR’s where added to our repos. I will first go through our testing tools and the reason we chose them and after that, what we needed to do in order to create a fully automated testing pipeline.

Cypress and Percy: Fully Automated UI E2E Tests

Cypress is one of the fastest growing testing tools in the last couple years. It provides easy syntax and a complete framework for running, writing and analyzing tests. Since most of our frontend code is web based and we have tons of complex UI interactions that we want to test before publishing new changes, this was our first choice.

Cypress’s test runner provides handful tools to test all elements of or UI

However, Cypress was still missing a thing. Our product has a strong usage of canvas elements, and most of this are not DOM components that can be selected with common Cypress commands. We needed a visual test tool to compare how the UI looked and the end result after applying changes to the canvas with our UI controls. Here’s where Percy comes into play.

Percy allows us to take snapshots of the screen at any given time and compare visual changes in the UI when new versions of the code are created. This allowed us to see if our canvas components are working correctly and if all the related controllers are doing the right job.

Percy’s dashboard helps analyze visual changes in a centralized manner.

Setting up Percy and Cypress

Setting Percy and Cypress on our project was really easy. We already had a node project for our vue frontend, so it was just a matter of following the installation guide of Cypress to have our test runner.

After that, we saw that Percy has great integration with Cypress. So the way it worked is that we just added new commands to our core “cy” command that allowed us to take snapshots at specific points during a test.

Once this was setups we were able to locally run tests using cypress and percy tools, and we were ready to start building our CI Pipeline.

CI Pipeline: General Architecture

For our CI pipeline, we need to first ensure that we understand our services and components that need to be in place for a full E2E test to work. In our case we had 4 basic components

1. Database Server (Postgres)
2. Backend API (Flask)
3. Worker Long Running Service (Flask)
4. Web UI Frontend (VueJs)

In order for our pipeline to work, we need to ensure that all 4 services are available and that they are automatically destroyed every time a test finishes. This ensure that we don’t share state between test runs and that our previous DB data do not affect new test runs.

To accomplish this we’ve used CircleCI. CircleCI allows us to spin up servers dedicated specifically to run our tests. We’ve used a Dockerfile to setup and OS dependencies and circleCI’s YAML config file’s format to install libraries and start the dependent services.

So the Pipeline in general terms goes like this.

1. Pull a Dockerfile built in with all our required OS dependencies.
2. Configure Launch our DB service
3. Install our NPM dependencies.
4. Install our Python dependencies.
5. Start our Backend Services
6. Start Our Frontend Service
7. Run tests with Cypress and Percy.

CircleCI enables us to configure, run and monitor this pipelines with an easy to use web dashboard and a well documented configuration file for launching infrastructure to run the tests.

With CircleCI you can monitor all the test runs in a centralized dasboard.

Key Findings & Learnings

We’ve found CircleCI to be a great tool for building this types of pipelines. Few problems that we found was knowing how to wait for services to be available and running processes in background, which was easily handled by using the background: true command of CircleCI and some curl probes to wait for services startup.

Integrations with Github is pretty easy, with just a few click you can configure your repo to start tests and show results on every push in a Pull Request that merges to master.

Overall we are now having a fully working E2E testing pipeline that guarantees that all tests are passing before we are able to merge any branch to master and deploy our new code.