Generate alert rules and integrate them into Prometheus

Prerequisites: Working Kubernetes with NooBaa and Prometheus installed and connected.

You run the following command: make all This will generate the .yaml files with the alerts/rules configurations. Currently, we are only interested in the file prometheus_alert_rules.yaml.

In order to add the generated rules to Prometheus, we use a custom CRD (kind: PrometheusRule). This is an example of such a CRD configuration (https://github.com/coreos/prometheus-operator/blob/master/example/user-guides/alerting/prometheus-example-rules.yaml):

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  creationTimestamp: null
  labels:
    prometheus: example
    role: alert-rules
  name: prometheus-example-rules
spec:
  groups:
  - name: ./example.rules
    rules:
    - alert: ExampleAlert
      expr: vector(1)

The next step is to convert our generated file to suit the structure of the CRD. Currently, we do it by hand. It should be handled by a script.

This results in CRD that looks like:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  creationTimestamp: null
  labels:
    prometheus: example
    role: alert-rules
  name: prometheus-example-rules
spec:
  groups:
  - name: system-capacity-alert.rules
    rules:
    - alert: NooBaaSystemCapacityWarning85
      annotations:
        description: "A NooBaa system is approaching its capacity, usage is more than 85%"
        message: "A NooBaa System Is Approaching Its Capacity"
        severity_level: warning
        storage_type: NooBaa
      expr: |
        NooBaa_system_capacity{} > 85
      labels:
        severity: warning
    - alert: NooBaaSystemCapacityWarning95
      annotations:
        description: "A NooBaa system is approaching its capacity, usage is more than 95%"
        message: "A NooBaa System Is Approaching Its Capacity"
        severity_level: warning
        storage_type: NooBaa
      expr: |
        NooBaa_system_capacity{} > 95
      labels:
        severity: warning
    - alert: NooBaaSystemCapacityWarning100
      annotations:
        description: "A NooBaa system approached its capacity, usage is at 100%"
        message: "A NooBaa System Approached Its Capacity"
        severity_level: warning
        storage_type: NooBaa
      expr: |
        NooBaa_system_capacity{} == 100
      labels:
        severity: warning

Notice that the label prometheus: example and role: alert-rules should be the same as the ruleSelector of your Prometheus CRD. Example of ruleSelector:

  ruleSelector:
    matchLabels:
      role: alert-rules
      prometheus: example

You can read more here (https://github.com/coreos/prometheus-operator/blob/master/Documentation/user-guides/alerting.md)

Now you just create this CRD and the Prometheus should load that by himself without any further input from our side.

You can check if the Prometheus applied that CRD, using the ConfigMap configuration. Just run the following command: oc get ConfigMap prometheus-example-rulefiles-0 -o yaml Change prometheus-example-rulefiles-0 to the relevant ConfigMap name.

In the .yaml you should see NooBaa alerts configured.

The last step is to browse into the Prometheus dashboard and click on Alerts tab. You will see NooBaa alerts there with all of the relevant info.