Wrongly formatted alert rules are silently not loaded

[Abuelodelanada]

Enhancement Proposal

When relating Prometheus to a charm that provides alert rules (for instance cos-configuratior) and some of the alert rules are not well formatted, Prometheus silently does not load those alert rules.

So for instance if you relate prometheus to cos-configurator with this alert rule where the expr is wrongly indented, the alert rule is silently not loaded into Prometheus.

groups:
- name: Custom JVM Rules
  rules:
    - alert: Custom JVM Usage
      annotations:
        description: " The broker {{ $labels.instance }} has high memory usage ({{ $value }}>85%) for more than 10 minutes."
        summary: "Broker {{ $labels.instance }} :: Critical :: Memory usage is {{ $value }}%"
        expr:  ((sum without(area)(jvm_memory_bytes_used{juju_charm!=".*"}) / 1024 / 1024) / (sum without(area)(jvm_memory_bytes_max{juju_charm!=".*"}) / 1024 / 1024)) * 100 > 85
      for: 5m
      labels:
        severity: critical

Related issues:

• Add probe for alert rules prometheus-k8s-operator#650
• Add probe for detecting multiple cos-proxy apps related to the same app #162