Breaking mutation testing for failing tests

[psfinaki]

I wonder, what was the motivation for keeping running Stryker in cases when some (less than 50%) unit tests are failing? And why 50%?

The reason I ask is that I noticed this accidentally. We run mutation testing in parallel to unit testing - and the former currently happens to be faster usually. I made some changes to the code and saw Stryker's success but then I saw a unit test failing. My first thought was - it's some flaky crap so I was rerunning the pipeline without looking into details.

After this repeated a few times, I looked deeper and saw that Stryker was actually issuing a warning about failing tests but kept moving. What's the point? I don't imagine any good reasoning for pushing the code with failing tests anyway. Broken tests mean even bigger problems than surviving mutants.

[rouke-broersma]

We introduced it to make running stryker locally more flexible when you're still working on a feature. Perhaps you haven't fixed all unit tests yet but want to check that the unit tests you have fixed are covering the mutants you expected. The 50% was simply a guesttimate not really based on much more than 'yea on more than 50% failing tests is probably pretty useless to still run stryker'.

Perhaps we should not do this in CI runs? Or perhaps we should make this configurable? On the other hand I would expect that next to mutation testing you do also run your unit tests and that if you have a requirement for working unit tests I would think you would just fail your pipeline.

[psfinaki]

I see. Well yeah spiritually I see mutation testing as the next step after unit testing, in that unit tests are supposed to verify the behavior and mutants are then supposed to verify the quality of unit tests. But technically I can imagine people working in a different fashion so that can make sense.

Yes probably I'd see it configurable then, like either breaking mutation testing for failing unit tests or not. And the default will be breaking. Pipeline should be failing because of unit tests anyway, yes, and it was in our case, it's just this inconsistency was misleading before I got aware of this behavior.