System.ObjectDisposedException on Hangfire.InMemory.State.Dispatcher when running multiple integration test scenarios

[JenPullUp]

I am building integration tests with ReqNroll. In one of my tests, I call an endpoint using an HTTP client, and this endpoint queues a Hangfire job using an in-memory instance of Hangfire, which relies on an in-memory database. When I run this test on its own, it runs as expected. However, when I run all tests in a row, only the first test succeeds, and the others fail due to internal server errors. It seems like the Dispose is not working properly, and the Hangfire instance is trying to access data that has been disposed of (this is an assumption; I am not sure).

Here are some additional details:

Error Details: The internal server errors typically look like this:

2024-08-30 09:35:55] fail: TestProject.Api.Filters.GlobalApiExceptionFilter[0]
      An exception occurred while processing the request.
      Hangfire.BackgroundJobClientException: Background job creation failed. See inner exception for details.
       ---> System.ObjectDisposedException: Cannot access a disposed object.
      Object name: 'Hangfire.InMemory.State.Dispatcher`1[[System.UInt64, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken={}]]'.
         at Hangfire.InMemory.State.Dispatcher`1.ThrowObjectDisposedException() in /_/src/Hangfire.InMemory/State/Dispatcher.cs:line 170
         at Hangfire.InMemory.State.Dispatcher`1.QueryWriteAndWait(ICommand`2 query) in /_/src/Hangfire.InMemory/State/Dispatcher.cs:line 68
         at Hangfire.InMemory.State.DispatcherBase`1.QueryWriteAndWait[T](ICommand`2 query) in /_/src/Hangfire.InMemory/State/DispatcherBase.cs:line 168
         at Hangfire.InMemory.InMemoryConnection`1.CreateExpiredJob(Job job, IDictionary`2 parameters, DateTime createdAt, TimeSpan expireIn) in /_/src/Hangfire.InMemory/InMemoryConnection.cs:line 112
         at Hangfire.Client.CoreBackgroundJobFactory.<>c__DisplayClass15_0.<CreateBackgroundJobTwoSteps>b__0(Int32 _)
         at Hangfire.Client.CoreBackgroundJobFactory.RetryOnException[T](Int32& attemptsLeft, Func`2 action)
      --- End of stack trace from previous location ---

Test Setup: I am using ReqNRoll and NUnit. My test setup involves a webapplication factory which configures:

• external conntection stubs.
• local in memory db
• hangfire in memory

the setup and teardown look like this:

[BeforeScenario]
    public void Setup()
    {
        _httpClient = _factory.WithWebHostBuilder(builder =>
        {
            builder.ConfigureTestServices(services =>
            {
                services
                    .ConfigureExternalConnectionStubs()
                    .ConfigureLocalDb()
                    .ConfigureHangfire();
            });
        }).CreateClient();
        _potentialThreatRequests = [];
        _statusCodes = [];
        _httpContents = [];
        _cachedThreatResponses = [];
        _cachedProblemDetails = [];
    }

[AfterScenario]
    public void CleanUp()
    {
        _httpClient.Dispose();
    }

configure localDb looks like this:

public static IServiceCollection ConfigureLocalDb(this IServiceCollection services)
    {
        var descriptor = services.SingleOrDefault(d => d.ServiceType == typeof(DbContextOptions<TestContext>));

        if (descriptor != null)
        {
            _ = services.Remove(descriptor);
        }

        _ = services.AddDbContext<TestContext>(options =>
            options.UseInMemoryDatabase("TestDb"));

        return services;
    }

and configure hangfire looks like this:

public static IServiceCollection ConfigureHangfire(this IServiceCollection services)
    {
        RemoveExistingHangfireServices(services);
        AddInMemoryHangfireServices(services);

        return services;
    }

private static void RemoveExistingHangfireServices(IServiceCollection services)
    {
        var hangfireServices = services
            .Where(s => s
                .ServiceType.FullName
                ?.Contains("hangfire", StringComparison.InvariantCultureIgnoreCase) == true)
            .ToArray();

        foreach (var hangfirePart in hangfireServices)
        {
            _ = services.Remove(hangfirePart);
        }
    }

    private static void AddInMemoryHangfireServices(IServiceCollection services)
    {
        services.AddHangfire(config => config
            .SetDataCompatibilityLevel(CompatibilityLevel.Version_180)
            .UseSimpleAssemblyNameTypeSerializer()
            .UseIgnoredAssemblyVersionTypeResolver()
            .UseInMemoryStorage(new InMemoryStorageOptions
            {
                IdType = InMemoryStorageIdType.Long
            }));

        services.AddHangfireServer();
    }

Test Isolation: Each test should run in isolation, but it seems like state or resources are not being cleaned up properly between tests. I've tried tearing down the httpClient using dispose in the teardown and giving the hangfire job a jobstorage which gets reinitiated every testrun. In both cases the hangfire instance still tries to access memory after it's been disposed

Environment Differences: There don't appear to be any environment differences between running tests individually versus all together, but the issue only occurs when running all tests.

Version Information: I am using Hangfire version 1.8.5 Hangfire.InMemory version 0.10.1 and ReqNroll version 2.0.3.

Does anyone have suggestions on what might be causing this issue or how I can better isolate my tests to prevent this error? Thank you!

My upload teststep looks like this:

[When(@"the file is uploaded")]
    public async Task WhenFileUploaded()
    {
        var potentialThreat = new PotentialThreatRequest()
        {
            ReferenceId = _referenceId,
            FileName = _fileName,
            File = Convert.FromBase64String(_fileContents),
            SourceReferenceId = _sourceReferenceId
        };
        var result = await _httpClient.PostAsJsonAsync("/api/v1/potential-threats", potentialThreat);
        _httpContents.Add(result.Content);
        _statusCodes.Add(result.StatusCode);

        if(result.StatusCode == HttpStatusCode.OK)
        {
            _potentialThreatRequests.Add(potentialThreat);
        }
        await HangfireJobHelper.AwaitAnyJobCompletionAsync();
    }

private static class HangfireJobHelper
    {
        public static async Task AwaitAnyJobCompletionAsync()
        {
            var jobStorage = JobStorage.Current;
            var monitoringApi = jobStorage.GetMonitoringApi();

            while (true)
            {
                var processingJobs = monitoringApi.ProcessingJobs(0, int.MaxValue);
                if (!processingJobs.Any())
                {
                    break;
                }

                await Task.Delay(50); // Wait for 50 ms before checking again
            }
        }
    }

the only tests that fail are when i run this step twice to check the case where the same files are uploaded twice or a file is uploaded under a different file name.