Running Stryker against diffs

[psfinaki]

Hey there, once again :)

So we are leveraging Stryker more and more, currently running it against a bunch of projects nightly.

And I am investigating the diff mode. I have made a change in a file in a project under test, committed it and run Stryker like dotnet stryker --project-file <...> --solution-path <...> --diff -gdt <...>.

However, it doesn't seem to make any difference (no pun intended) to just dotnet stryker --project-file <...> --solution-path <...>. The reports are the same. The logs reflect the new Stryker configuration but then don't output anything specific to that, with the same mutants created in the same files.

I have tried specifying the commit in the diff target, with the same result. Actually, if I specify non-existent commit or branch, Stryker doesn't seem to notice it.

Can you verify the feature is still working? Don't want to file a bug just yet, maybe I am missing something...

[rouke-broersma]

Hi @psfinaki

I'm taking a look right now

[rouke-broersma]

Looks like you're hitting #1290 which was fixed in 0.21.0.. We're aiming to fix this problem tomorrow and push out a new release as well.

[rouke-broersma]

0.22.0 is being released which should fix your issues

[psfinaki]

Thanks for your effort and the update, @Mobrockers :)

Is there any example on how to properly use it? I tried the latest Stryker version with the repo I attached in the previous post but it doesn't seem to identify any changes in the latest commit, showing 0 files changed.

I tried specifying both the branch and the commit. I see some new specific logs (e.g. Looking for branch matching master or Matched with canonical name refs/heads/master) but that's it, and the mutation score then is NaN.

[rouke-broersma]

When I run the diff feature on your example this is my result:

[15:29:52 INF] 1 files changed

PS C:\Users\RoukeB\Downloads\StrykerTest\StrykerTest.Test> dotnet stryker --diff -gdt 0730dff9bc254859476621926b8b2f13982c24f2

   _____ _              _               _   _ ______ _______  
  / ____| |            | |             | \ | |  ____|__   __| 
 | (___ | |_ _ __ _   _| | _____ _ __  |  \| | |__     | |    
  \___ \| __| '__| | | | |/ / _ \ '__| | . ` |  __|    | |    
  ____) | |_| |  | |_| |   <  __/ |    | |\  | |____   | |    
 |_____/ \__|_|   \__, |_|\_\___|_| (_)|_| \_|______|  |_|    
                   __/ |                                      
                  |___/                                       


 Version: 0.22.0 (beta)

[15:29:35 INF] Identifying project to mutate.
[15:29:40 INF] The project C:\Users\RoukeB\Downloads\StrykerTest\StrykerTest\StrykerTest.csproj will be mutated.
[15:29:43 INF] Analysis complete.
[15:29:43 INF] Building test project C:\Users\RoukeB\Downloads\StrykerTest\StrykerTest.Test\StrykerTest.Test.csproj (1/1)
[15:29:46 INF] Initializing test runners (VsTest)
[15:29:52 INF] 1 files changed
[15:29:52 INF] Changed file C:\Users\RoukeB\Downloads\StrykerTest\StrykerTest\Class1.cs
[15:29:52 INF] Total number of tests found: 2
[15:29:52 INF] Initial testrun started
[15:29:56 INF] Using 10019 ms as testrun timeout
[15:29:58 INF] 8 mutants created
[15:29:58 INF] Capture mutant coverage using 'perTest' mode.
[15:30:00 INF] 4     mutants got status NoCoverage.   Reason: Mutant has no test coverage
[15:30:00 INF] 4     total mutants are skipped for the above mentioned reasons
[15:30:00 INF] 4     total mutants will be tested
████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████100.00% │ Testing mutant 4 / 4 │ K 0 │ S 4 │ T 0 │ ~0m 00s │                                                    00:00:03

Killed:   0
Survived: 4
Timeout:  0

Your html report has been generated at:
 file:///C:/Users/RoukeB/Downloads/StrykerTest/StrykerTest.Test/StrykerOutput/2021-03-08.15-29-35/reports/mutation-report.html
You can open it in your browser of choice.
[15:30:03 INF] Time Elapsed 00:00:28.3406590
[15:30:03 INF] The final mutation score is 0.00 %

The git diff feature tests only those files that have been changed since the target you provide. So if you have a feature branch you might want to test those mutants that are new or that have been changed since master. You would then set -gdt to master. Or, you might want to test the mutations relevant in the latest commit. In that case you would set -gdt to the commit id of the previous commit.

[psfinaki]

Ohhh I see now. This is a great explanation - maybe worth copypasting as a note to the docs :) I thought for the branch target the diff was calculated between the two latest commits in that branch and for the commit target - for the changes in that commit. So I was putting all the wrong arguments :D

This makes sense and I confirm it works. Thanks for the explanations!