Add metrics endpoint in prometheus format

[agaudreault]

Description

The renovate on-prem application only exposes a /status endpoint that can be used to see what is the current state of the application, however, it is difficult to parse and send the data to a monitoring system. It has become a standard for Kubernetes open-source applications to expose a /metrics endpoint in the Prometheus format, so the community can monitor the application successfully and have confidence in their deployment.

Motivation

As a renovate on-prem user, I want a way to monitor the applications, investigate issues and receive alerts if my on-prem deployment is not behaving properly.

Implementation

• Application exposes a /metrics endpoint with Prometheus metrics
• ServiceMonitor is added to the chart
• Grafana dashboard template added to the repository
• Alerting guidelines

Example

Here are some of the metrics that could be helpful

# TYPE renovate_info gauge
# TYPE renovate_webhook_received counter
# TYPE renovate_webhook_received_timestamp_seconds gauge
# TYPE renovate_job_enqueue_total counter
# TYPE renovate_job_dispatch_total counter
# TYPE renovate_job_error_total counter
# TYPE renovate_job_queue_length gauge
# TYPE renovate_job_processed_total counter
# TYPE renovate_job_duration_seconds histogram
# TYPE renovate_job_last_completed_timestamp_seconds gauge

related to #220

[morremeyer]

For anyone coming across this: We've been using the prometheus communities' JSON exporter in the meantime.

Here's the configuration we're running:

We're using https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus-json-exporter in version 0.11.0 to deploy.

The values we have configured in the values file

serviceMonitor:
  enabled: true
  targets:
    - name: renovate-ce
      # Replace "the-endpoint-for-renovate" with the hostname for the renovate-ce service
      url: http://the-endpoint-for-renovate/api/status
      module: renovate

configuration:
  config: |
    ---
    modules:
      renovate:
        http_client_config:
          authorization:
            credentials_file: /credentials/renovate/renovate-ce.txt

        metrics:
          - name: renovate_organization_count
            valuetype: gauge
            path: "{ .app.organizationCount }"
            help: The number of organizations known to renovate

          - name: renovate_repository_count
            valuetype: gauge
            path: "{ .app.repositoryCount }"
            help: The number of repositories known to renovate

          - name: renovate_jobs_processed
            valuetype: counter
            path: "{ .jobs.history.processed }"
            help: The number of jobs processed since the instance has started

          - name: renovate_queue_size
            valuetype: gauge
            path: "{ .jobs.queue.size }"
            help: The number of jobs currently in the queue

additionalVolumes:
  - name: credentials-renovate-ce
    secret:
      secretName: credentials-renovate-ce

additionalVolumeMounts:
  - name: credentials-renovate-ce
    mountPath: /credentials/renovate/renovate-ce.txt
    subPath: mendRnvServerApiSecret

The secret

apiVersion: v1
kind: Secret
metadata:
  name: credentials-renovate-ce
  namespace: kube-prometheus-stack
type: Opaque
data:
  # This is the same as "mendRnvServerApiSecret" in the existingSecret configured in the renovate-ce values file at renovate.existingSecret` since that is the Bearer token.
  mendRnvServerApiSecret: Tm8gY2hhbmNlLCBJIHJlcGxhY2VkIHRoaXMgYmVmb3JlIHBhc3RpbmcgaXQ=

[morremeyer]

@nabeelsaabna Release 8.1.0 has a changelog entry

feat: prometheus metrics endpoint

Could you add documentation for that, please? I tried a few paths, but couldn't find it (/prometheus, /metrics, /api/metrics, /api/prometheus)

[Gabriel-Ladzaretti]

Its not enabled by default and you have to set MEND_RNV_PROMETHEUS_METRICS_ENABLED=true.
but the thing is that I forgot to add it for CE, although it was added for EE. please wait for the next release. apologies for the inconvenience.

[morremeyer]

No worries, thanks for the update!

[nabeelsaabna]

@morremeyer Thanks for the quick catch
https://github.com/mend/renovate-ce-ee/releases/tag/8.1.1 ✅

[morremeyer]

@nabeelsaabna Thanks, just tested and it works. I noticed that the endpoint is available without authentication, is that on purpose?

If yes, I think it would be great to have an option to not have it included in the Ingress resource, which currently exposes /.

Maybe the Ingress could just expose /api by default and /metrics only optionally, what do you think?

For everyone: Once you set MEND_RNV_PROMETHEUS_METRICS_ENABLED to "true", the metrics are available at /metrics.

[nabeelsaabna]

Agree, Ingress should not expose everything by default.

We now have:

• /health
• /metrics
• /webhook exposed by default
• /api exposed by default

[morremeyer]

I think it makes sense to have all of these as separate paths in the ingress that are configurable, where /webhook and /api are configured in the values.yaml to be enabled by default, and /health and /metrics are disabled by default.

This way, everyone can configure it to the needs of their setup.