No alerts after restart

[RuriRyan]

Hi, we have a couple of minor problems with the bot. We're running the bot inside of k8s and it works for the most part. We also have a group chat where the alerts are sent to.
One of the problems is, that if the pod is restarted/recreated after, e.g. a node failure, we don't get any alerts and we have to remove the bot from the group and add it again.
We use Bolt for storage and a PVC of course.
Is this intended or a bug somewhere?
The second problem/question is, is it possible to run the bot multiple times with the same credentials?

[metalmatze]

Hey,
yes this pretty much sounds like the way I run the bot on Kubernetes.
If there is a node failure the situation is sadly as you've described.
For those reason I highly recommend setting up a DeadManSnitch / Watchdog with a third party tool.

As this bot uses the libkv library, we also support consul or etcd as data stores. With that, it might be possible that your can run multiple instances of the bot. However, I still haven't tested that, but would be very interested in your findings, if you plan on looking into that. 👍
Let me know if you need help with that.

[tabnul]

For this reason i run 2 deployments of the bot. each with different telegram keys, and using its own alertmanager replica.
So if one alertmanager replica or alertmanager-bot fails , i will still get alerts from the second.
Downside is that in this way, i have 2 bots hammering me for every alert ;)