What is Mutation Testing and Why Should You Use It?

Do you know what is a better testing metric than code coverage? Does 100% code coverage mean your code is well-tested? How to generate unit test cases for free, without using AI?

The answer to all these questions is mutation testing. Interested? Read on 😉

Table of Contents

Code Coverage – Our Old Friend

You probably know what code coverage is. Traditionally, many teams tend to measure how good their tests’ suite is by percentage line / statements / functions coverage. Some even require this metric to be 100%. Let’s start with this well-known approach first.

We are working with TicketsPriceCalculator class today. It contains some simple business logic:

	export class TicketsPriceCalculator {
	calculateTotalTicketsPrice(tickets: Ticket[]): number {
	if (tickets.length === 0) {
	return 0;
	}

	let totalPrice = tickets.reduce((sum, ticket) => sum + ticket.price, 0);
	const businessClassTickets = tickets.filter(
	(ticket) => ticket.isBusinessClass
	).length;

	let totalDiscountToApply = 0;

	// Apply discounts based on total price
	if (totalPrice > 2000) {
	totalDiscountToApply += 0.25;
	} else if (totalPrice > 1000 && totalPrice <= 2000) {
	totalDiscountToApply += 0.2;
	} else if (totalPrice > 500 && totalPrice <= 1000) {
	totalDiscountToApply += 0.15;
	} else if (totalPrice >= 200 && totalPrice <= 500) {
	totalDiscountToApply += 0.1;
	}

	// Apply additional discount based on number of tickets
	if (tickets.length >= 10) {
	totalDiscountToApply += 0.05;
	}

	// Apply business class discounts
	if (businessClassTickets >= 10) {
	totalDiscountToApply += 0.2;
	} else if (businessClassTickets >= 5) {
	totalDiscountToApply += 0.1;
	}

	return totalPrice * (1 – totalDiscountToApply);
	}
	}

view raw ticketsPriceCalculator.ts hosted with ❤ by GitHub

As you can see, the class has one function whith returns the total price to be paid for a set of tickets. On the way, it applies various discounts based on some business rules. Fair enough.

It’s a critical piece of business logic for the app we’re working on, but it has never had any tests. We were asked to write unit tests for this class. Our manager strictly requires the code coverage to be 100%. Let’s do it then. We will go line by line, analyze the business logic and add the tests. Below, you can see the results of running those tests (you can find the tests code here):

Unit tests before using mutation testing. We have 8 unit tests, all passing. Our code coverage (line, functions, statements, branches) is 100%

Wow, that’s awesome! We did a great job. We have 8 unit tests which provide 100% statements, branch, functions and lines coverage!

So now we are really safe, aren’t we? 🤔

Yeah… until you know what mutation testing is 😎

What Is Mutation Testing?

The idea of mutation testing is to mutate (change) our code. Mutation testing tools introduce changes into our production code. These changes can be really anything – from reverting a condition of an if statement, to changing some operators or changing a string value. For instance, here are some logical mutations Stryker Mutator is able to introduce (you can find the list of all mutations supported by Stryker here):

Logical operator mutations supported by Stryker Mutator

In effect, they generate modified versions of our code, which are called mutants. Makes sense 😉

Each mutant contains a single modification in our source code. What happens next is crucial. Mutation testing tool takes this mutant (it is used as the code under tests now) and executes all of our unit tests against it. The result may be twofold:

all tests have passed. In other words, the mutant has survived. This is a bad situation. It implies that the modification of our source code was not detected by our tests
at least one test has failed. In other words, the mutant has been killed. This is a good scenario. It means that our tests detect changes in the production code and efficiently cover this particular scenario.

The flow is more or less like this schema from Peter Evans’ blog presents:

But why would anyone change the production code on purpose by introducing some stupid changes? It may seem counterintuitive for now, but let’s see how we can use it to our leverage.

Mutation Testing with Stryker Mutator

Let’s clearly state what we are starting with:

8 unit tests, 100% code coverage

Let’s now run mutation testing on our code. We are using TypeScript here, so let’s go with Stryker Mutator.

Here’s the result of running Stryker with npx stryker run on our code:

Mutation test result - first run. Mutation score 78%, survived mutants 15

As you can see, 53 mutants were killed, but 15 of them survived. This gives us a mutation score of 78%. Let’s call it mutation coverage from now on. So to make it clear, this is our starting situation:

8 unit tests, 100% code coverage, 78% mutation coverage

Let’s now see how to kill those remaining mutants.

Killing First Mutants

In order to kill the mutants and improve our mutation score, we need to write more unit tests. Sounds logical, doesn’t it?

If still not, let’s take a deeper look at mutation tests results in VS Code. Stryker lists details of every survived mutant, for example this one:

Stryker changed the greater or equal (>=) operator to greater operator (>). This is a very common and very useful mutation. It usually means that our tests don’t cover the edge case associated with this particular condition.

In this case, the edge case value seems to be 5. More precisely, what happens if the number of business class tickets is exactly 5? We don’t know, because there is no test for this particular edge case!

Let’s add it:

	test("should apply 10% discount if there are exactly 5 business class tickets", () => {
	const tickets = Array.from({ length: 5 }, () => ({
	price: 10,
	isBusinessClass: true,
	}));

	const totalPrice = calculator.calculateTotalTicketsPrice(tickets);
	expect(totalPrice).toBe(45); // 10% discount
	});

view raw ticketsPriceCalculator_edgeCase.test.ts hosted with ❤ by GitHub

The test passes. Let’s run mutation tests again:

Struker Mutator second run - already 14 mutants survived and almost 80% mutation coverage

That’s it! We have a new unit test and 1 more mutant killed 🎉 Current state:

9 (+1) unit tests, 100% code coverage, 79% mutation coverage

I think you’re now getting what this is all about. We should be adding unit tests to cover the edge cases detected by mutations until all (or most) mutants are killed.

Mutation Testing – Free Unit Tests Generator

After adding 1 more test, the results look as follows:

Struker mutator with 10 tests - 86% mutation coverage

10 (+2) unit tests, 100% code coverage, 86% mutation coverage

Finally, this is what we managed to get after a few minutes of work:

And we now have 17 unit tests in our suite! The final score looks like that:

🎊17 (+9) unit tests, 100% code coverage, 96% mutation coverage 🎊

Notice what happened here. We started with 8 unit tests and finished with 17! This is awesome! That’s how mutation testing is a free unit tests generator. And you don’t even need AI for that 😉

What’s more, the generated mutations even forced me to make a little tweak to the production code. It exposed some things that didn’t make sense. I could make this change to the production code, because my tests’ suite was already quite substantial.

Note that I didn’t push to gain 100% mutation coverage. The 3 mutants that survived don’t make much sense in our case. I encourage you to get the code and execute npx stryker run in the application folder to see for yourself 🙂

The percentage value doesn’t matter in that case. What matters is that we got 9 new unit tests and our tests’ suite is more robust. Now we can go and tell our manager that the business-critical code is well tested 😊

So What Now?

I hope you see how valuable mutation testing is. It’s not a perfect tool that suits all cases. For example, it’s not easy to use mutation testing for complex legacy code. However, if your logic is nicely isolated, it could be a great tool. Especially if your production code heavily depends on calculations and many conditions. You may be surprised how many edge cases mutation testing can find.

I also love using mutation testing with TDD. These two techniques nicely complement each other.

I encourage you to go and try mutation testing. There are different mutation tools for various programming languages:

C#, JavaScript, TypeScript, Scala: Stryker Mutator
Ruby: mutant
Java: pitest
PHP: Infection
C++: Mutate++

…and probably many more. Just do some research for the language you work with and give it a try 😉 Because testing is cool, isn’t it? 🙂

You can find the full source code used in this article here.

Code Coverage – Our Old Friend

What Is Mutation Testing?

Mutation Testing with Stryker Mutator

Killing First Mutants

Mutation Testing – Free Unit Tests Generator

So What Now?

Related Posts

Share This

Share if you liked it:

You may also like

5 Reasons To Not Use C# IDE For TypeScript Development

React State Management Basics

TypeScript Compiler Explained