Simple assignment mutator

[DavidHall]

I've noticed that simple assignment expressions with the = operator do not appear to generate mutations.

This appears to be covered by this feature #870 where the decision was made to not implement since the StatementMutator would cover much of the use cases.

In one of my projects I have identified untested code I was hoping to see caught by mutation testing:

    var result = new ResultWrapper();
    CashBalance cashBalance;

    try
    {
        cashBalance = await _balanceRepository.GetCashBalance();
        // I was hoping that this line would be caught by mutation testing
        result.CreditBalance = cashBalance.Amount;
    }

    return result;

If I remove that line above, no tests break and I'm also seeing no living mutants.

I have two questions:

• Is this actually a supported feature and I've missed something in how I'm using Stryker
• If not supported, is it possible and desirable to implement this, if even in a basic version like removing property assignments.

If the answer to the second question is true, I can create an issue ticket, and even try my hand at make a PR for it.

[rouke-broersma]

Does the statement mutator actually remove the statement? Could you show the report?

[DavidHall]

Here is the report (cleartexttree) from a minimal reproduction that I believe duplicates what I'm seeing in my project.

All mutants have been tested, and your mutation score has been calculated
All files [4/4 (100,00 %)]
├── Calculator.cs [1/1 (100,00 %)]
│ └── [Killed] Block removal mutation on line 6
│ ├── [-] {
│ │ var resultHolder = new ResultHolder();
│ │
│ │ var generator = new ResultValueGenerator();
│ │ resultHolder.Result = generator.GetResultValue();
│ │ resultHolder.Result = generator.GetFalseValue();
│ │
│ │ return resultHolder;
│ │ }
│ └── [+] {}
├── ResultHolder.cs [0/0 (N/A)]
└── ResultValueGenerator.cs [3/3 (100,00 %)]
├── [Killed] Block removal mutation on line 6
│ ├── [-] {
│ │ return 42;
│ │ }
│ └── [+] {}
├── [Killed] Block removal mutation on line 11
│ ├── [-] {
│ │ return -42;
│ │ }
│ └── [+] {}
└── [Killed] UnaryMinusExpression to UnaryPlusExpression mutation on line 12
├── [-] -42
└── [+] +42
[15:52:00 INF] Time Elapsed 00:00:14.2010076
[15:52:00 INF] The final mutation score is 100.00 %

So that doesn't appear to be generating the remove statement call.

[dupdob]

I can explain: the statement mutator does not remove assignment statement as of now, and this is done on purpose because it triggers compilation errors that are reported away from the mutation. And such errors lead to rolling back every mutation from the method.

In the previous exemple, removing result.CreditBalance = cashBalance.Amount; does not trigger an error, but removing ashBalance = await _balanceRepository.GetCashBalance();` would trigger an error on the next line.

In order to support this kind of (apparently) simple mutations, we need either to:

1. Implement assignment tracking to decide if we can safely remove an assignment or not (akin to implementing a significant portion of a compiler)
2. Or compile the project each time we inject a mutation to see if it triggers error (probably not difficult to implement, but would slow down mutation generation by a lot, but incremental build may help.
3. Or roll back mutation one by one in case of error (I made some attempts in that direction in the past months, but it is a lot harder than it looks)
4. Mutate assignment specifically such as replacing them with assignment to default, for example. But this may lead to semantically equivalent mutant, rendering them un killable (when the original assignment is to default).

I can't think of another alternative strategy at this stage.

[dupdob]

you can see #1953 and #2000 for related issues

[DavidHall]

@dupdob Thanks for the quick answer, that makes a lot of sense!

One thought is that removing property assignments should never trigger a compilation error, but I guess it would in some ways be worse to have mutations for some assignment statements and not others.

I wasn't aware that Stryker would need to compile for each mutation to check if assignments are valid or not. Or rather - I had assumed that Stryker was doing something like that anyway in the process of evaluating each mutation.

I'll dig into the source a little myself so I get a feel for how it all works.

One last question - would a feature flag that forces unsafe mutations and runs the compilation strategy you mention be desirable?

It was exactly this line of code that isn't covered by Stryker that led me to start using it, so I would happily accept occasonally enabling such a feature and living with a very slow run as a necessary evil.

[dupdob]

One thought is that removing property assignments should never trigger a compilation error

To put it short: Stryker does not know if this is a property assignment or a variable assignment or an indexer assignment for that matter.

One last question - would a feature flag that forces unsafe mutations and runs the compilation strategy you mention be desirable?

Could be, in theory. But just 'blindly' enabling it would not give you the expected result either, I am afraid: it would result in generating two 'assignment mutations' (and others, of course), once of each assignment. But, the resulting code would not compile and would result in Stryker removing all mutations. So no cigar either.

Stryker relies on a double edge strategy regarding having 'viable mutation' (i.e. that can compile):

1. it skip mutations that will lead to non compilable code, or mutations that step 2 may not roll back
2. it removes all mutations which relates to a build error using the location (file, line and column number). When a compilation error's location matches no mutations, Stryker removes all mutations from that method (or accessor, function...).