Learn Istanbul

Learn how to use Istanbul to check/track Code Coverage in your JavaScript projects.

Why?

Like the road sign that is "Not In Use" most of the code being written never gets executed.

There are a few obvious issues with this:

if un-tested code remains in the codebase i t can contain unknown behaviour e.g. bugs.
untested features are more difficult to maintain without introducing breaking changes.
un-tested code can clutter a project and accumulates technical debt that wastes time.

What?

Code coverage tells you when code you have written is being executed so you can decide if un-covered lines are superfluous (and can be removed) or require additional testing.

The rest of this page will focus on practical usage example, so if you are completely new to Code Coverage we recommend you read the wikipedia article: https://en.wikipedia.org/wiki/Code_coverage first.

Istanbul is a code coverage analysis script you run when executing your unit tests: https://github.com/gotwarlost/istanbul/ we like it because it's simple and prints out nice html reports (see below)

How?

Installation

We prefer to install istanbul as a "devDependencies" in each of our projects:

npm install istanbul --save-dev

to check if the installation worked, (copy-paste and) run the following command in your terminal:

node_modules/.bin/istanbul help

Simple Example

For our first example create a file called test.js.

vi test.js

type (or copy-paste) the following code in the test.js file:

x = 42;
if(false)
     x =-1;

Now run the istanbul command to generate a coverage report:

node ./node_modules/.bin/istanbul cover test.js

Alternatively you can insert the line

"coverage": "istanbul cover ./test.js"

into the scripts section of your package.json and run the following command in your terminal:

npm run coverage

This will create a directory in your project called coverage where you will find the generated coverage reports. In our case: learn-istanbul/coverage/lcov-report/learning-istanbul/test1.js.html

If you open the test1.js**.html** file in your web browser you will see a visual coverage report:

Istanbul gives us four code coverage metrics:

Statements: How many of the statements in you code are executed.
Branches: Conditional statements create branches of code which may not be executed (e.g. if/else). This metric tells you how many of your branches have been executed.
Functions: The proportion of the functions you have defined which have been called.
Lines: The proportion of lines of code which have been executed.

when you click test.js to view the coverage for the file you see:

Two things to note in the example above:

we only get 66.67% coverage because the only 2/3 of the code is being run
the 3rd line never gets executed because false is always false!

This may be a trivial example but it shows exactly where the useless code is.

A more "Real World" Example

Try executing the mischief.js file by running npm test:

What is wrong with the following picture?

There are plenty of developers/organisations that can only dream about getting 96% code coverage! and yet when we inspect the detail, there's something big slipping through the net!

We have 100% functional code coverage, but only 50% "Branch" Coverage. This means one or more conditional execution branches is not being executed.

Most of the time it will be something innocuous but what if a disgruntled person slipped in something like:

if(employee.status === 'terminated' && employee.left - today() > 90) {
	selfDestruct();
}

The 97% Coverage is not looking so hot anymore ...

What if we add a Test that follows the branch containing the rogue code? We reach our mythical 100% Coverage:

And if we simply allow this code to be promoted without further checks, the rogue code will be in production and soon forgotten.

The solution here is to not rely (solely) on tools such as Istanbul to check code. Its essential would advocate a separation between the people writing the tests and the developers who write the code.

And there is still no substitute for Code Review!

Tracking Coverage as-a-Service

Knowing the coverage locally is nice, but it's way more useful to a team to track coverage over time using an independent service. Thankfully, there are a few you can chose from:

Codecov: https://codecov.io/#features
Coveralls: https://coveralls.io/features
Codeclimate: https://codeclimate.com/features
Codacy: https://codacy.com/product

We have used all the services and found that Codecov i s the clear winner for 3 reasons:

Easy to integrate with existing Continuous Integration (CI) setup e.g. Travis-CI
Great stats and visualisations
Pull request message informs if coverage has changed (see below)

Setup

Sign-up to use Codecov (Free) using your GitHub account: https://codecov.io/
Select the repository you want to track coverage for from your list.
Add a few lines to your CI configuration file e.g:

before_install:
  - pip install --user codecov
after_success:
  - codecov --file coverage/lcov.info --disable search

This installs the Codecov reporter tool on your CI and sends the lcov.info report (which gets generated by Istanbul) to Codecov where it's kept safely.

Real-world example in .travis.yml file: https://github.com/dwyl/hapi-auth-jwt2/blob/master/.travis.yml#L7-L10

Now when you create a pull request your CI will send a coverage report to Codecov and Codecov will leave a comment on the PR:

Yes, it's "possible" to have a "large" project with 100% Test Coverage. (ALL DWYL projects do!)

So you can see at a glance if new code is being added without corresponding tests ...

When the coverage is lower the Pull Request "fails":

Note: if you want to prevent people (your team) from creating Pull Requests with less than an agreed level of coverage (e.g 100%!!), add a coverage checking script and corresponding pre-commit hook. More on pre-commit hooks: https://github.com/dwyl/learn-pre-commit

Add a "Badge" to your Readme (Optional/Recommended)

Click on the Settings tab for your chosen repo, click on Badge and click Copy e.g:

Then paste the markdown into your Readme.

More about badges: https://github.com/dwyl/repo-badges

Further Reading:

JavaScript Code Coverage Dashboard with Codecov.io by @ariya https://ariya.io/2015/08/javascript-code-coverage-dashboard-with-codecov-io

Note: DWYL is not financially affiliated with Codecov, but we ❤️ @stevepeak and the product he has built. They are focussed on doing one thing really well and have innovated a lot in code coverage tracking making developers lives much better.

Background Reading

Ariay's basic tutorial: https://ariya.io/2012/12/javascript-code-coverage-with-istanbul
Jasmine Test Coverage: https://dzone.com/articles/code-coverage-jasmine-tests

Istanbul (the JavaScript Code Coverage tool) https://github.com/gotwarlost/istanbul should not to be confused with istanbul the desktop screen recorder, they are totally diferent animals! Shame about the name collision... :-(