Issue when I trying to test Health Check on application

[lucas-llb]

Hi everyone,
I got an issue when I trying to test the Health Check on application.
Even when I test the route and the return message, Stryker still got survived mutants on the Health Check Options.
These images below is about my test and can help us:
repo:
https://github.com/lucas-llb/HealthCheckTest

Codes and images:
Startup(edited to code):

app.UseRouting(); app.UseEndpoints( endpoint => endpoint.MapHealthChecks( "/health", new HealthCheckOptions { ResponseWriter = async (context, report) => { var result = JsonConvert.SerializeObject( new HealthCheckResponse(report.Status.ToString()) ); await context.Response.WriteAsync(result); } } ) );

Test(Edited to code):

[Fact] public async Task Given_HealthCheckConfigured_When_ApplicationOn_Then_ShouldReturnApiHealthStatus200_And_ExpectedResponseBody() { var application = applicationFactory.CreateClient(); var response = await application.GetAsync("/health"); var responseCode = (int)response.StatusCode; var responseContent = await response.Content.ReadAsStringAsync(); responseCode.Should().Be(StatusCodes.Status200OK); responseContent.Should().Contain(APPLICATION_STATUS_RESPONSE); }

Mutation Report:

Anyone can help me with it?
Thanks in advance.

[rouke-broersma]

What happens to your unit test when you apply the mutation manually to your code? (make the endpoint string empty)

[lucas-llb]

I sent the reports of a project with only the test that I had the problem, but i think that its not getting survivors on it.
But this logs below is from my original project, where I get the issue, I think will be more helpfull.
logs.zip

[dupdob]

I am sorry, but this is getting very confusing. These logs shows that all mutants are killed except one, which is not related to your initial problem if I read the log properly.
Could you please start over by providing;

1. The report with the problem you face
2. The logs
3. ideally the original project or a stripped down project that does reproduce your original issue.

Short from that, I do not feel I am spending my time productively

[lucas-llb]

Sorry about the confusion. Let's try to start again.
Unfortunately i can't share the whole code because it's a closed source but i will share what I can.

Report:

and

Logs:
logs.zip

Stripped down project (stryker works properly here, the issue only happens in my original project, but it has the test I use):
https://github.com/lucas-llb/HealthCheckTest

[dupdob]

Thanks for the details. It would be better if I had access to the original project, but I have a theory and a workaround.

Evidences

I see there is only one survivor (app.CustomizeHealthCheckPoint()), it is tested against 4 tests, none of them matching the one in your example.
ProgramTests.Given_ThatTheEFCoreQueryAdaptersWhereRegistered_When_ApplicationStarts_Then_AllQueriesShouldBeResolvedWithoutErrors() ProgramTests.TestSwaggerDocGeneration(environment: \"Staging\", statusCode: NotFound) ProgramTests.TestSwaggerDocGeneration(environment: \"Production\", statusCode: NotFound) ProgramTests.TestSwaggerDocGeneration(environment: \"Development\", statusCode: OK) ProgramTests.Given_SerilogConfigurated_When_ApplicationStarts_Then_LogsShouldAppear()

Analysis

This is probably a limitation of coverage analysis: I do not have a clear idea of when and how a webapp is instantiated.
Your initialization logic happens in the app constructor, and and it seems the app is cached/reused across several tests. Caching object across tests results in incomplete coverage information, as in Stryker is not able to identify all tests that would be impacted by a mutation.

Fix

To this way, we have yet to find a reliable way around this problem. Stryker detects this kind of pattern when capturing coverage information, but only when caching is performed by the mutated project, not by some external framework.
The upcoming analysis function will offer Stryker comments allowing the user to tag code forcing it to be tested against all tests.

Workaround

As of now, the only workaround in Stryker is to disable coverage analysis. Can you please try and report if this fixes your problem (or not).
Alternatively, I recommend you not to use IClassFixture in xUnit and to use constructor and Dispose function instead see doc. Tests may be a bit slower in the hand, but it will ensure tests are properly isolated.

[lucas-llb]

Hello everyone.
I tried the suggetion about removing the IClassFixture and using the constructor and dispose. Now its working properly.
Thanks a lot @dupdob for the help!