Mutation Testing for Smart Contracts - A step by step guide

In my previous article, I wrote about the core concepts of mutation testing. With this post, I will show you the mutation testing theory can be applied in practice using the mutation testing tool Vertigo.

---

Photo by Bit Cloud on Unsplash

Setup

To start with mutation testing, we need to do three things:

• Install Truffle and Ganache
• Install Vertigo
• Get a project to test

Install Truffle and ganache by executing the following commands:

npm install -g truffle

Then install Vertigo by executing the following command:$ pip3 install eth_vertigo

With that out of the way, the only thing left to do is to get a Truffle project to run our mutation testing on.

In this article, I will use the MetaCoin truffle box as the target of the mutation testing run. You can easily download this project by executing the following commands in your terminal:$mkdirmetacoin$ cd metacoin $ truffle unbox metacoin

Configuration

We need to do a minor configuration tweak to ensure that Vertigo can run optimally. Vertigo can run the analysis for multiple mutants in parallel if there are enough test networks configured. Unfortunately, the default configuration for the MetaCoin box does not yet provide these networks, so you’ll have to add them yourself.

The following snippet shows my truffle-config.jsafter I’ve added two networks to use with mutation testing. Go ahead and copy-paste this snippet into the truffle-config.js file in the metacoin directory. You can also just add the two networks, if you already have some values configured in your project’s truffle-config.js.

module.exports = {
  networks: {
    vertigo_test_network_1: {
      host: "127.0.0.1",
      port: 8545,
      network_id: "*"
    },
    vertigo_test_network_2: {
      host: "127.0.0.1",
      port: 8546,
      network_id: "*"
    },
  }
}

Photo by Artem Maltsev on Unsplash

Let the magic happen

First, we need to start the two networks that will be used by Vertigo. You can do so by running the following commands in separate terminal windows (bonus: You can try to use tmuxinator with this config, to make life a bit easier):

$ ganache-cli -p 8545$ ganache-cli -p 8546

With everything set up, we can start the mutation testing by simply running the following command:

$ vertigo run --network vertigo_test_network_1 --network vertigo_test_network_2

The next step is very important!

Sit back. Enjoy. Drink some ☕

PS. You can optionally add the commandline option --output output.txt to the command, to write the analysis results to a file named output.txt.

Interpreting The Results

Once the analysis is finished you will be presented with the following information:… [] Running analysis on 5 mutants 100%|████████████████████████████████████████████████████| 5/5 [00:17<00:00,  4.38s/mutant] [] Done with campaign run [+] Report: Mutation testing report: Number of mutations:    5 Killed:                 4 / 5Mutations:[+] Survivors Mutation:    File: /Users/walker/Development/metacoin/contracts/MetaCoin.sol    Line nr: 19    Result: Lived    Original line:         if (balances[msg.sender] < amount) return false;Mutated line:         if (balances[msg.sender] ⇐ amount) return false; …

The important bit is the line that says Killed: 4/5. This line shows the number of killed mutants (4 in this case), compared to the total number of valid mutants (5 for MetaCoin). See my previous article for an explanation of these terms. Just one mutation slipped through the cracks! The mutation score for this project is 80%; pretty good.

Stay tuned for a deep dive into the detailed analysis results that Vertigo provides and how you can use them to improve your test suite’s quality.

---

Photo by Jason Leung on Unsplash

Looking to the future

Vertigo is a brand new mutation testing tool, and while features like parallelization and equivalent mutant detection help improve usability, there are still some things to improve:

Test Networks

The current setup process requires you to add network configurations to truffle-config.js and directly run ganache instances.

One of the next things that will be added to Vertigo is a feature that automates all this and gives a more streamlined experience.

Analysis Speed

For big projects like openzeppelin-contracts the mutation testing process can take some time. And sure, you can use features like mutant sampling to reduce the number of mutants that you need to test. However, it would be better if we spend less time on each mutant altogether. A soon-to-be-added feature achieves just that, by leveraging fine-grained coverage data and optimizing which unit tests to execute.

Of course, there is more in the pipeline! But you will have to stay tuned to hear about that!

---

Thanks to Bernhard Mueller for reviews on drafts of this article!