Issues with mutation coverage score and how it evolves

[vmassol]

Hi,

I'm trying to understand how the mutation score evolves on the xwiki-commons-context module of XWiki. Right now this module has 91% of mutation score.

I did the following:

• Experiment A: I introduced a new Test class with an empty test method (annotated with @Test). Score remained the same when I was expecting it to drop since the quality of this new test should be very low compared to the other ones. I think it's not included in the score because it's not actually calling any code.

public class ExecutionContextExceptionTest
{
    @Test
    public void testXXX()
    {
    }
}

• Experiment B: I added some code in the previous test:

public class ExecutionContextExceptionTest
{
    @Test
    public void testXXX()
    {
        ExecutionContextException exception = new ExecutionContextException("hello");
        assertNotNull(exception);
    }
}

Result: the mutation score remained at 91%. Checking the report, I don't see ExecutionContextException which I was expecting to see since the tests is exercising it. Any idea?

• Experiment C: I removed one test method from ExecutionContextTest, namely inheritance() and the score didn't change (it remained at 91%). That doesn't look good. Maybe it reduced the score by less than 1% and the rounding kept it at 91%? Is that a way to have more precise percentages?

• Experiment D: I removed 2 test methods from ExecutionContextTest, namely inheritance() and illegalInheritance() and this time I got a score reduction: Mutation score of 84 is below threshold of 91. What's interesting is that the jump was pretty high: from 91% to 84%, when in experiment C there was less than 1% reduction (if any).

• Experiment E: I removed only illegalInheritance() and I got the same score reduction from 91% to 84%. So this means that inheritance() doesn't contribute to the mutation score.

My conclusions so far:

• You can introduce some new bad quality test (worse in average than existing tests) and still see no score reduction.
• You can even remove some test methods and still keep the same score, even though it's blatantly obvious that you've reduced the quality of the test.

More general conclusion: It doesn't seem that the mutation score really reflects the quality of the tests.

WDYT? Any ideas?

thx

[vmassol]

It probably means that no mutations were done on the code covered by inheritance() and thus the score didn't change. Right? How can I check what mutations were executed?

[oscarlvp]

Hi,
This is a really nice experiment.
Some comments:

• The mutation score evaluates the test suite as a whole and not individual test clases. Given this, experiment A, does not bring anything new. The test suite has the same effect over the mutants as before so the score should not change.
• Descartes does not mutate constructors. Therefore experiment B has no effect.
• Experiments C, D and E reveal that those two methods detect a non disjoint set of mutants. IT would be interesting to check what's happening with inheritance in more detail.

About the conclusions:

• PIT counts non-covered mutants in the score. A better strategy would be to evaluate the score only for covered mutants. The two new test methods added in the experiments do not strengthen the test suite, so they produce no change. A change would be noted if the score does not consider non-covered mutants and the new test case covers code that was not covered before or kills undetected mutants.
• The same goes when removing test methods. If other test methods kill the same mutants, then the test suite as a whole remains the same.

I agree with your general conclusion. Mutation score does not accurately reflects the quality of the test suite, even more using extreme mutation. It would be better to check the number of pseudo and partially tested methods.

The report produced by PIT in XML format or our own JSON reporter provide the test cases run for each mutant and also the test case that killed the mutant. It also says if the mutant was detected, not detected or not covered.

I think we can get some valuable information by inspecting the report files produced by each experiment.

[bbaudry]

Thanks @vmassol for these experiments!
As Oscar mentions, it might be more interesting to look at the number of methods killed and pseudo-tested, instead of the score.
Also, PIT/Descartes do not report if a mutant is killed by more than one test. So, indeed, the tools currently do not say which tests are more or less contributing to the detection of mutants.
As for your second conclusion: it would be interesting to discuss what makes you say that you reduce the quality of the tests by removing inheritance. Because, indeed, from the perspective of usual test quality metrics (coverage, mutation score, etc), removing this test has not reduced the quality.
If we can characterize better what is test quality, we can retrieve more relevant indicators.

Another experiment, which I believe can be very interesting to capture the impact of mutation testing would be as follows: keep the same tests, but remove the assertions.

Also, these experiments focus on what the mutation analysis tells about the test cases, but it's also interesting to look at what the mutation analysis says about the code: we can also look at the live mutants.
Have you tried evolving the code base instead of the test suite?

[vmassol]

As for your second conclusion: it would be interesting to discuss what makes you say that you reduce the quality of the tests by removing inheritance. Because, indeed, from the perspective of usual test quality metrics (coverage, mutation score, etc), removing this test has not reduced the quality

Removing the inheritance() test method reduces the coverage by 2% on this module. And that's normal since less code is covered. Globally the Test class containing inheritance() is less good in term of quality because it's exercising less code and even in term of mutation, the test has more risks of killing less mutants since it's testing less. Hence the lower quality for me. Don't you think?

[bbaudry]

Indeed, if coverage decreases, we can say that the test quality decreases.
And, clearly, Descartes is not meant to capture fine-grained changes in code coverage
But, it captures other aspects of test quality, such as the quality of assertions, or the ability of test cases to propagate erroneous states.
So, as you mentioned previously, coverage and pseudo-tested methods are complementary indicators about the quality of the test suite, and we need to refine the pros and cons of each one of them