Reminder doesn't start after Kubernetes pod eviction

[aviviadisap]

Hi, I’m using .NET 6 and Orleans 3.7.2 with KubernetesHosting:
silo.ConfigureServices(services => services.UseKubernetesHosting());

We need to run a periodic single-silo task every 2 seconds. Since Orleans reminders have a minimum interval of 1 minute, we're simulating this with a long-running 1-year reminder and a while (shutdownCtk.IsCancelled == false) { doWork; sleep 2 seconds; } loop.

This approach works well after the initial deployment of the cluster. However, if Kubernetes evicts the pod running this reminder, the task doesn't resume on another pod until we manually restart one of the silos.

What might be causing this issue?

[ReubenBond]

It's hard to say what's causing this issue without code or logs - are you able to provide more details on your implementation? I assume "sleep 2 seconds" is a Task.Delay call?

Perhaps you want something similar to this: https://gist.github.com/ReubenBond/8dc47f68634faafe334fadc6f5444ad0

[aviviadisap]

We were able to workaround this using "ISyncWorker<GrainCancellationToken, SyncWorkStatus>, IGrainWithStringKey" grain for the long running loop, and reminder just activates the syncWorker (so we have a single silo with the reminder and single silo with hardcoded string key running single loop instance).

So we can close this issue, however the reminders in Kubernetes with K8s hosting feels glitchy.