info
What you'll learn
- The difference between Code Coverage and Cypress's UI Coverage
- How to instrument your application code for code coverage
- How to collect code coverage from Cypress tests
- How to merge code coverage from parallel tests
- How to collect code coverage from different testing types
Introduction
As you write more and more end-to-end tests, you will find yourself wondering -do I need to write more tests? Are there parts of the application stilluntested? Are there parts of the application that perhaps are tested too much?
Cypress offers a couple of solutions to help answer these questions.
- Code coverage - find out which lines of the application's source code were
- UI coverage - find out which parts of the UI were testedexecuted
Code Coverage
Code coverage is a metric that helps you understand how much of your applicationcode is covered by your tests. If there are importantsections of the application's logic that were not executed from the tests,then a new test can be added to ensure that part of our application logic istested.
Computing the source code lines that were executed during the test is donethrough code coverage. Code coverage requires inserting additional countersinto your source code before running it using instrumentation..This document will be focused on code coverage and the instrumentation required to set it up.
UI Coverage
UI Coverage is a visual test coverage report based on the interactive elements that make up your application.It highlights areas of missed test coverage directly within every page of your application,that you can use to make data-driven testing decisions.
UI Coverage also seamlessly integrates with your Cypress tests, building on the tests you've alreadywritten — no extra instrumentation is required like with code coverage.
To learn more about UI coverage,please see our UI Coverage guide.
Instrumenting code
Instrumentation takes code that looks like this...
add.js
function add(a, b) {
return a + b
}
module.exports = { add }
...and parses it to find all functions, statements, and branches and theninserts counters into the code. For the above code it might look like this:
// this object counts the number of times each
// function and each statement is executed
const c = (window.__coverage__ = {
// "f" counts the number of times each function is called
// we only have a single function in the source code thus it starts with [0]
f: [0],
// "s" counts the number of times each statement is called
// we have 3 statements and they all start with 0
s: [0, 0, 0],
})
// the original code + increment statements
// uses "c" alias to "window.__coverage__" object
// the first statement defines the function, let's increment it
c.s[0]++
function add(a, b) {
// function is called and then the 2nd statement
c.f[0]++
c.s[1]++
return a + b
}
// 3rd statement is about to be called
c.s[2]++
module.exports = { add }
Imagine we load the above instrumented source file from our test spec file.Immediately some counters will be incremented!
add.cy.js
const { add } = require('./add')
// JavaScript engine has parsed and evaluated "add.js" source code
// which ran some of the increment statements
// __coverage__ has now
// f: [0] - function "add" was NOT executed
// s: [1, 0, 1] - first and third counters were incremented
// but the statement inside function "add" was NOT executed
We want to make sure every statement and function in the file add.js
has beenexecuted by our tests at least once. Thus we write a test:
add.cy.js
const { add } = require('./add')
it('adds numbers', () => {
expect(add(2, 3)).to.equal(5)
})
When the test calls add(2, 3)
, the counter increments inside the "add"function are executed, and the coverage object becomes:
{
// "f" keeps count of times each function was called
// we only have a single function in the source code
// thus it starts with [0]
f: [1],
// "s" keeps count of times each statement was called
// we have 3 statements, and they all start with 0
s: [1, 1, 1]
}
This single test has achieved 100% code coverage - every function and everystatement has been executed at least once. But, in real world applications,achieving 100% code coverage requires multiple tests.
Once the tests finish, the coverage object can be serialized and saved to diskso that a human-friendly report can be generated. The collected coverageinformation can also be sent to external services and help during pull requestreviews.
info
If you are unfamiliar with code coverage or want to learn more, take a look atthe "Understanding JavaScript Code Coverage" blog postPart 1andPart 2.
Cypress does not instrument your code - you need to do it yourself. The goldenstandard for JavaScript code instrumentation is the battle-hardenedIstanbul and, luckily, it plays very nicely with theCypress. You can instrument the code as a build step through one of two ways:
- Using the nyc module - a command-lineinterface for the Istanbul library
- As part of your code transpilation pipeline using the
babel-plugin-istanbul
tool.
Using NYC
To instrument the application code located in your src
folder and save it inan instrumented
folder use the following command:
npx nyc instrument --compact=false src instrumented
We are passing the --compact=false
flag to generate human-friendly output.
The instrumentation takes your original code like this fragment...
src/index.js
const store = createStore(reducer)
render(
<Provider store={store}>
<App />
</Provider>,
document.getElementById('root')
)
...and wraps each statement with additional counters that keep track of how manytimes each source line has been executed by the JavaScript runtime.
const store = (cov_18hmhptych.s[0]++, createStore(reducer))
cov_18hmhptych.s[1]++
render(
<Provider store={store}>
<App />
</Provider>,
document.getElementById('root')
)
Notice the calls to cov_18hmhptych.s[0]++
and cov_18hmhptych.s[1]++
thatincrement the statement counters. All counters and additional book-keepinginformation is stored in a single object attached to the browser's window
object. We can see the counters if we serve the instrumented
folder instead ofsrc
and open the application.
If we drill into the coverage object we can see the statements executed in eachfile. For example the file src/index.js
has the following information:
In green, we highlighted the 4 statements present in that file. The first threestatements were each executed once and the last statement was never executed (itprobably was inside an if
statement). By using the application, we can bothincrement the counters and flip some of the zero counters into positive numbers.
Using code transpilation pipeline
Instead of using the npx instrument
command, we can usebabel-plugin-istanbul
to instrument the code as part of its transpilation. Add this plugin to the.babelrc
file.
.babelrc
{
"presets": ["@babel/preset-react"],
"plugins": ["transform-class-properties", "istanbul"]
}
We can now serve the application and get instrumented code without anintermediate folder, but the result is the same instrumented code loaded by thebrowser, with the same window.__coverage__
object keeping track of theoriginal statements.
info
Check out@cypress/code-coverage#examples
for full example projects showing different code coverage setups.
A really nice feature of both nyc andbabel-plugin-istanbul
is that the source maps are generated automatically, allowing us to collect codecoverage information, but also interact with the original, non-instrumented codein the Developer Tools. In the screenshot above the bundle (green arrow) hascoverage counters, but the source mapped files in the green rectangle show theoriginal code.
info
The nyc
and babel-plugin-istanbul
only instrument the application code andnot 3rd party dependencies from node_modules
.
E2E code coverage
To handle code coverage collected during each test, there is a@cypress/code-coverage
Cypressplugin. It merges coverage from each test and saves the combined result. It alsocalls nyc
(its peer dependency) to generate static HTML reports for humanconsumption.
Install the plugin
info
Please consult the@cypress/code-coverage
documentation for up-to-date installation instructions.
npm install @cypress/code-coverage --save-dev
Then add the code below to thesupportFile andsetupNodeEvents function.
// cypress/support/e2e.js
import '@cypress/code-coverage/support'
- cypress.config.js
- cypress.config.ts
const { defineConfig } = require('cypress')
module.exports = defineConfig({
// setupNodeEvents can be defined in either
// the e2e or component configuration
e2e: {
setupNodeEvents(on, config) {
require('@cypress/code-coverage/task')(on, config)
// include any other plugin code...
// It's IMPORTANT to return the config object
// with any changed environment variables
return config
},
},
})
When you run the Cypress tests now, you should see a few commands after thetests finish. We have highlighted these commands using a green rectangle below.
After the tests complete, the final code coverage is saved to a .nyc_output
folder. It is a JSON file from which we can generate a report in a variety offormats. The@cypress/code-coverage
plugingenerates the HTML report automatically - you can open the coverage/index.html
page locally after the tests finish. You can also call nyc report
to generateother reports, for example, sending the coverage information to 3rd partyservices.
See code coverage summary
To see the summary of the code coverage after tests run, run the command below.
npx nyc report --reporter=text-summary
========= Coverage summary =======
Statements : 76.3% ( 103/135 )
Branches : 65.31% ( 32/49 )
Functions : 64% ( 32/50 )
Lines : 81.42% ( 92/113 )
==================================
info
Tip: store the coverage
folder as a build artifact on your continuousintegration server. Because the report is a static HTML page, some CIs can showit right from their web applications. The screenshot below shows the coveragereport stored on CircleCI. Clicking on index.html
shows the report right inthe browser.
Code coverage as a guide
Even a single test can cover a lot of the application code. For example, let'srun the following test that adds a few items, then marks one of them ascompleted.
- End-to-End Test
- Component Test
it('adds and completes todos', () => {
cy.visit('/')
cy.get('.new-todo')
.type('write code{enter}')
.type('write tests{enter}')
.type('deploy{enter}')
cy.get('.todo').should('have.length', 3)
cy.get('.todo').first().find('.toggle').check()
cy.get('.todo').first().should('have.class', 'completed')
})
After running the test and opening the HTML report, we see 76% code coverage inour application.
Even better, we can drill down into the individual source files to see what codewe missed. In our example application, the main state logic is in thesrc/reducers/todos.js
file. Let's see the code coverage in this file:
Notice how the ADD_TODO action was executed 3 times - because our test hasadded 3 todo items, and the COMPLETE_TODO action was executed just once -because our test has marked 1 todo item as completed.
The source lines not covered marked in yellow (the switch cases the test missed)and red (regular statements) are a great guide for writing more end-to-endtests. We need tests that delete todo items, edit them, mark all of them ascompleted at once and clear completed items. When we cover every switchstatement in src/reducers/todos.js
we probably will achieve close to 100% codecoverage. Even more importantly, we will cover the main features of theapplication the user is expected to use.
We can write more E2E tests.
The produced HTML report shows 99% code coverage
Every source file but 1 is covered at 100%. We can have great confidence in ourapplication, and safely refactor the code knowing that we have a robust set ofend-to-end tests.
If possible, we advise implementingvisual testing in addition to Cypressfunctional tests to avoid CSS and visual regressions.
Combining code coverage from parallel tests
If you execute Cypress tests inparallel, each machine endsup with a code coverage report that only shows a portion of the code exercised.Typically an external code coverage service would merge such partial reports foryou. If you do want to merge the reports yourself:
- on every machine running Cypress tests, copy the produced code coverage reportinto a common folder under a unique name to avoid overwriting it
- after all E2E tests finish, combine the reports yourself using
nyc merge
command
You can find an example of merging partial reports in ourcypress-io/cypress-example-conduit-app
E2E and unit code coverage
Let's look at the one file that has a "missed" line. It is thesrc/selectors/index.js
file shown below.
The source line not covered by the end-to-end tests shows an edge case NOTreachable from the UI. Yet this switch case is definitely worth testing - atleast to avoid accidentally changing its behavior during future refactoring.
We can directly test this piece of code by importing the getVisibleTodos
function from the Cypress spec file. In essence we are using Cypress as a unittesting tool (find more unit testing recipeshere).
Here is our test to confirm that the error is thrown.
// cypress/e2e/selectors.cy.js
import { getVisibleTodos } from '../../src/selectors'
describe('getVisibleTodos', () => {
it('throws an error for unknown visibility filter', () => {
expect(() => {
getVisibleTodos({
todos: [],
visibilityFilter: 'unknown-filter',
})
}).to.throw()
})
})
The test passes, even if there is no web application visited.
Previously we instrumented the application code (either using a build step orinserting a plugin into the Babel pipeline). In the example above, we are NOTloading an application, instead we are only running the test files bythemselves.
If we want to collect the code coverage from the unit tests, we need toinstrument the source code of our spec files. The simplest way to do this isto use the same .babelrc
withbabel-plugin-istanbul
and tell the Cypress built-in bundler to use .babelrc
when bundling specs. Onecan use the@cypress/code-coverage
pluginagain to do this by adding the code below to thesetupNodeEvents function.
- cypress.config.js
- cypress.config.ts
const { defineConfig } = require('cypress')
module.exports = defineConfig({
// setupNodeEvents can be defined in either
// the e2e or component configuration
e2e: {
setupNodeEvents(on, config) {
require('@cypress/code-coverage/task')(on, config)
// tell Cypress to use .babelrc file
// and instrument the specs files
// only the extra application files will be instrumented
// not the spec files themselves
on('file:preprocessor', require('@cypress/code-coverage/use-babelrc'))
return config
},
},
})
For reference, the .babelrc
file is shared between the example application andthe spec files, thus Cypress tests are transpiled the same way the applicationcode is transpiled.
{
"presets": ["@babel/preset-react"],
"plugins": ["transform-class-properties", "istanbul"]
}
When we run Cypress withbabel-plugin-istanbul
included and inspect the window.__coverage__
object in the spec iframe, weshould see the coverage information for the application source files.
The code coverage information in unit tests and end-to-end tests has the sameformat; the@cypress/code-coverage
pluginautomatically grabs both and saves the combined report. Thus we can see the codecoverage from the cypress/e2e/selectors.cy.js
file after running the test.
Our unit test is hitting the line we could not reach from the end-to-end tests,and if we execute all spec files - we will get 100% code coverage.
Full stack code coverage
A complex application might have a Node back end with its own complex logic.From the front end web application, the calls to the API go through layers ofcode. It would be nice to track what back end code has been exercised duringCypress end-to-end tests.
Are our end-to-end tests that are so effective at covering the web applicationcode also covering the back end server code?
Long story short: yes. You can collect the code coverage from the back end,and let the @cypress/code-coverage
plugin merge it with the front endcoverage, creating a single full stack report.
info
The full source code for this section can be found in thecypress-io/cypress-example-conduit-apprepository.
You can run your Node server and instrument it using nyc on the fly. Instead ofthe "normal" server start command, you can run the commandnpm run start:coverage
defined in the package.json
like this:
{
"scripts": {
"start": "node server",
"start:coverage": "nyc --silent node server"
}
}
In your server, insert another middleware from @cypress/code-coverage
. If youuse an Express server, include middleware/express
:
const express = require('express')
const app = express()
require('@cypress/code-coverage/middleware/express')(app)
If your server uses hapi, include middleware/hapi
if (global.__coverage__) {
require('@cypress/code-coverage/middleware/hapi')(server)
}
Tip: you can conditionally register the endpoint only if there is a globalcode coverage object, and you canexclude the middleware code from the coverage numbers:
/* istanbul ignore next */
if (global.__coverage__) {
require('@cypress/code-coverage/middleware/hapi')(server)
}
For any other server type, define a GET /__coverage__
endpoint and return theglobal.__coverage__
object.
if (global.__coverage__) {
// handle "GET __coverage__" requests
onRequest = (response) => {
response.sendJSON({ coverage: global.__coverage__ })
}
}
In order for the @cypress/code-coverage
plugin to know that it should requestthe back end coverage, add the new endpoint to the Cypress configurationenvironment settings under env.codeCoverage.url
key. For example, if theapplication back end is running at port 3000 and we are using the default "GET/coverage" endpoint, set the following:
- cypress.config.js
- cypress.config.ts
const { defineConfig } = require('cypress')
module.exports = defineConfig({
env: {
codeCoverage: {
url: 'http://localhost:3000/__coverage__',
},
},
})
From now on, the front end code coverage collected during end-to-end tests willbe merged with the code coverage from the instrumented back end code and savedin a single report. Here is an example report from thecypress-io/cypress-example-conduit-appexample:
You can explore the above combined full stack coverage report at thecoveralls.io/github/cypress-io/cypress-example-conduit-appdashboard. You can also find full stack code coverage in ourRealWorld App.
Even if you only want to measure the back end code coverage Cypress can help.
Videos
There is a series of videos we have recorded showing code coverage in Cypress
How to instrument react-scripts web application for code coverage
Get code coverage reports from Cypress tests
Excluding code from code coverage reports
Check code coverage robustly using 3rd party tool
Adding code coverage badge to your project
Show code coverage in commit status check
Checking code coverage on pull request
Examples
You can find full examples showing different code coverage setups in thefollowing repositories:
- cypress-io/cypress-realworld-appor RWA is a full stack example application that demonstrates best practicesand scalable strategies with Cypress in practical and realistic scenarios.The RWA achieves full code coverage with end-to-end testsacross multiple browsers anddevice sizes.
- lluia/cypress-typescript-coverage-exampleshows coverage for a React App that uses TypeScript.
- ericorruption/cypress-code-coverage-typescript-webpack-ts-loadershows how to collect coverage for a TypeScript + webpack project using
ts-loader
instead ofbabel-loader
.
Find the full list of examples linked incypress-io/code-coverage#external-examples.
See also
- UI Coverage
- The official@cypress/code-coverage plugin