A failure in snapshot save can cause unobserved task exception

[stedmanm]

Version Information
Version of Akka.NET? 1.5.28
Which Akka.NET Modules? Akka.Persistence

Describe the bug
When snapshot save fails, Akka will then attempt to delete the specified snapshot, if the delete snapshot operation throws an exception, this exception will become unobserved.

To Reproduce
This can most likely be reproduced in multiple ways, but this is what happened in our scenario.

1. Multiple load snapshot operations failed, causing an open circuit.
2. Save snapshot operation fails, because circuit is open.
3. Delete snapshot gets triggered, because snapshot save failed.
4. Delete snapshot operation returns a faulted task with an open circuit exception, this exception is never observed.

Expected behavior
The exception should get observed by Akka framework.

Actual behavior
Exception was not observed and caused our service to crash.

Screenshots

Environment
Windows Service and .NET 7

[Aaronontheweb]

Weird, though I had commented on this and classified it yesterday. We'll look into it!

[Aaronontheweb]

ah wait, yes I did - is this a duplicate of #7358 ? or a different flavor of the same problem

[stedmanm]

@Aaronontheweb Yes this is different problem than #7358, that also caused us an unobserved task exception, but it is also related to the SnapshotStore.