[SPIKE: 2 hrs] Automate setting up Grafana alerting via grafonnet

[sgibson91]

Context

In #5062, @sunu had difficulty creating alerts automatically via grafonnet. The current work-around is a one-time manual step to enable alerting in the grafana UI.

There is more experience within 2i2c staff members with using grafonnet, so perhaps we may find a way to unblock this with a spike?

Task list

Tasks

Beta Give feedback

1. Investigate whether or not it is possible to setup grafana alerting automatically via grafonnet

Definition of Done

• We have learned whether grafana alerting can be automatically setup via grafonnet
• The learnings of this spike have been added to this issue as a comment