A brief introduction to mutation tests

4 min readMay 10, 2022

Mutation testing is an approach designed to validate the quality of unit tests. The main idea of mutated testing is based on creating mutants programs. The mutant program is a slightly modified copy of the original program. The changes should be very small and should not affect the program overall. After creating a set of mutants, all the tests that are used for testing the original program are run both for the original program, and for mutants.

The process of mutation testing

The first step is preparing a set of mutants. Each mutant has to contain exactly one change in the comparison with the original program. Mutants created during this step are called first-order mutants. This step is called infection.
After creating a set of mutants the unit tests set is run both for the original program and for all the modified programs (mutants).
Test results from the original program are compared with test results run for each created mutant. For this reason, tests need to be stable. As a stable test, I consider a test that produces neither false-positive nor false-negative results. Unstable tests make mutation testing pointless.
If a mutant program will cause a different test result than the original program, the mutant is terminated. Otherwise, the mutant is kept alive. Alive mutants are called equivalent mutants.

The more mutants are killed the better. The number of alive mutants should strive for zero. The value of killed mutants divided by all the mutants is called a mutation score. A test that did not fail for any mutant is a good candidate to be verified in the context of its usability. Such a test is not sensitive to changes in the source code. There are a few reasons why such tests should be inspected:

A test may be badly written. For example, an assertion may check a condition that is always true.
A test may not check anything useful. For example, the code always produces a result that will cause a test’s success.

A low number of alive mutants is a factor that may indicate (but it does not have to!) that tests for the original program are well-designed.

Types of mutations

Statement mutations — a part of code from the original program is removed:

Remove if statements.
Remove else statement from if.

Value mutations — the default values in the original code are replaced:

Change default values for constants from the original code.
Change default parameters in methods or objects.
Change hardcoded values e.g. static string values.

Decision mutations — statements from the original code are changed:

Increment returned number values.
Replace in if conditions e.g x < y with x > y, or x < y with x =< y.
Replace a variable in condition with a static value e.g. change x === y to 100 === y.
Replace or remove && and || statements.
Change returned data type, e.g. instead of 5 return '5'.
Replace the arithmetic operators e.g. replace + with -.
Keyword replacement e.g. replacing true with false. or replace let with const.

Mutation testing technologies

The most trivial way to start the adventure with mutation testing is to create mutants manually. However, let’s be honest, it is frustrating and neither fast nor effective. Fortunately, some tools allow automating this process:

Stryker Mutator (JS, C#, Scala)
PIT (Java, Kotlin)
Infection (PHP)
mull (C, C++)
mutmut (Python)

Pros of mutation testing

Mutation tests check the real code coverage for testing statements. Nevertheless, there are dedicated tools for it such as Istanbul for JavaScript test frameworks. Note that, the code coverage metric in the context of tests’ quality is discussable. Code coverage means only that a test touched a covered part of the code. This metric says nothing about tests’ accuracy or completeness.
Allows for detecting human errors faster.
Provides a metric that could be used to measure the quality of existing tests.
Helps to find missing tests.
Sometimes it helps to detect a dead code.
Detects a high coupling in the code — a small change may cause a cascade effect. In such a case a mutant will cause a massive number of test fails.

Cons of mutation testing

The time required to generate mutants can be very long as well as running tests for all the generated mutants.
A lot of automatically generated mutants are redundant. It’s not a trivial thing to determine the usability of the mutant. In edge cases, nearly 99% of mutants can be redundant!
Without automation tools, mutation testing is extremely ineffective.
Mutation testing is applicable only to white-box testing.

Sources and extra materials