A brief introduction to mutation tests

The process of mutation testing

  1. The first step is preparing a set of mutants. Each mutant has to contain exactly one change in the comparison with the original program. Mutants created during this step are called first-order mutants. This step is called infection.
  2. After creating a set of mutants the unit tests set is run both for the original program and for all the modified programs (mutants).
  3. Test results from the original program are compared with test results run for each created mutant. For this reason, tests need to be stable. As a stable test, I consider a test that produces neither false-positive nor false-negative results. Unstable tests make mutation testing pointless.
  4. If a mutant program will cause a different test result than the original program, the mutant is terminated. Otherwise, the mutant is kept alive. Alive mutants are called equivalent mutants.
  • A test may be badly written. For example, an assertion may check a condition that is always true.
  • A test may not check anything useful. For example, the code always produces a result that will cause a test’s success.

Types of mutations

  • Remove if statements.
  • Remove else statement from if.
  • Change default values for constants from the original code.
  • Change default parameters in methods or objects.
  • Change hardcoded values e.g. static string values.
  • Increment returned number values.
  • Replace in if conditions e.g x < y with x > y, or x < y with x =< y.
  • Replace a variable in condition with a static value e.g. change x === y to 100 === y.
  • Replace or remove && and || statements.
  • Change returned data type, e.g. instead of 5 return '5'.
  • Replace the arithmetic operators e.g. replace + with -.
  • Keyword replacement e.g. replacing true with false. or replace let with const.

Mutation testing technologies

  • Stryker Mutator (JS, C#, Scala)
  • PIT (Java, Kotlin)
  • Infection (PHP)
  • mull (C, C++)
  • mutmut (Python)

Pros of mutation testing

  • Mutation tests check the real code coverage for testing statements. Nevertheless, there are dedicated tools for it such as Istanbul for JavaScript test frameworks. Note that, the code coverage metric in the context of tests’ quality is discussable. Code coverage means only that a test touched a covered part of the code. This metric says nothing about tests’ accuracy or completeness.
  • Allows for detecting human errors faster.
  • Provides a metric that could be used to measure the quality of existing tests.
  • Helps to find missing tests.
  • Sometimes it helps to detect a dead code.
  • Detects a high coupling in the code — a small change may cause a cascade effect. In such a case a mutant will cause a massive number of test fails.

Cons of mutation testing

  • The time required to generate mutants can be very long as well as running tests for all the generated mutants.
  • A lot of automatically generated mutants are redundant. It’s not a trivial thing to determine the usability of the mutant. In edge cases, nearly 99% of mutants can be redundant!
  • Without automation tools, mutation testing is extremely ineffective.
  • Mutation testing is applicable only to white-box testing.

Sources and extra materials

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store