What if I told you The X-Men can make you a better software tester? Well they can’t but their supervillain, William Stryker, can.
This article is all about how to improve your unit tests using mutation testing. There are several mutation testing frameworks but the framework I’ll be talking about is called Stryker, aptly named after the supervillain from X-Men who wants to kill all of the mutants. I’ve been using Stryker for over two years now, in those two years I’ve implemented Stryker on three large enterprise projects and given several presentations on the framework.
Caution!
I want to start this article with a warning, mutation testing is not for the faint of heart. At first, it’s tedious, time-consuming and often drags out the testing phase of development, often requiring developers to write even more unit tests. Why even bother with it? Because it is a foolproof way to ensure your unit tests are actually providing value.
Every developer has seen a unit test that looks great but actually doesn’t test anything or simply exists to provide code coverage. These are the tests that mutation testing exposes. It’s the cherry on top for unit tests, the pick of destiny, any rockstar tester isn’t a true rockstar unless they have mutation testing checking their unit tests. Now that you’ve been warned let’s get into the details.
What is Mutation Testing?
The simple one sentence answer is mutation testing is testing for your unit tests. What does this mean? Mutation testing is done through a framework, like Stryker, that will mutate or change, specific parts of your code and check to see if that mutation causes a unit test to fail. If the mutant causes one of your existing unit tests to fail the mutant is considered “killed”. At the end of the mutation testing, a report is generated with a full list of killed and surviving mutants, similar to what you would see in a code coverage report using Jest or Jasmine.
A Simple Example
Let’s say you have some code that is generating a URL for an application. Hopefully, there is a unit test checking that URL path and ensuring it gets generated. The mutation testing framework would come in and mutate your endpoint to a null or empty string. What happens to your unit test? If it fails that’s good, it means your test is smart enough to know what the URL should be and the mutant will be considered killed. If your unit test doesn’t fail that means your unit test isn’t quite as good as you thought. The framework will list the mutant as survived and you’ll need to go back to the drawing board to write a better unit test. Below is an example of what a string mutation would do:
String url = "deadpool.com"goTo(url)
//mutated codeString url = ""goTo(url)
Mutation Testing Frameworks
Stryker is an open source mutation testing framework that has a very active development community. When I started using Stryker two years ago the framework was only available for JavaScript but it has continued to grow over the years and is now available for JavaScript, TypeScript, Scala, and C#. It has a CLI tool available for quick and easy setup and plays nicely with common testing frameworks like Jasmine and Jest. I have also used PIT Test for Java, the mutation testing concept is the same but the framework is more console based. While each mutation testing framework may vary slightly the concept remains the same and the important thing is to just start using one.
Photo by Artem Sapegin on Unsplash
Mutation Testing Pro’s & Con’s
When first starting to use mutation testing the process will be very slow. It can be difficult to kill certain mutants and having to rewrite unit tests is never fun. Over time the benefits are worth it though and just like anything in life the more you practice the faster the process will become.
Pro’s
- Higher quality unit tests
- No more writing pointless tests that only serve to boost a metric (like code coverage)
- More confidence that your application is fully tested
- Fewer production defects
- More bugs caught
- Opportunity for developers to receive instant feedback on the quality of their unit tests
- Stryker runs in parallel so it can run very fast if the resources are available
Con’s
- Adds an extra step in the development cycle
- Can take over an hour to run against a large codebase on a machine having four cores or less
- Configuration can be tricky to figure out if the CLI generated configuration doesn’t work for your project
Conclusion
Mutation testing still seems to be a rarity amongst developers but hopefully, this article has encouraged any developers reading to give it a shot. The setup may be slow and the first mutants may take a long time to kill but the more mutants killed the faster the process will become. Short of Wolverine offering unit testing tutoring, Stryker and mutation testing is as close as you can get to the X-Men improving your unit tests.