Feature Request: k8s-infra support for collection application-level prometheus metrics

[jkgeyti-cogna]

The k8s-infra helm chart is convenient for cloud customers who do not wish to self-host the full stack, but unlike the signoz chart, the k8s-infra chart doesn't run an prometheus metrics collector for collecting pod metrics by default.

I had a conversation with @srikanthccv who came up with a solution using the override below, which seems like a sensible thing to be able to toggle on or off as part of the template, instead of having to override it like this.

Thanks for a great product.

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: signoz-collector
spec:
  values:
    otelDeployment:
      config:
        receivers:
          prometheus:
            config:
              scrape_configs:
              - job_name: pod_metrics
                kubernetes_sd_configs:
                - role: pod
                relabel_configs:
                - action: keep
                  regex: true
                  source_labels:
                  - __meta_kubernetes_pod_annotation_signoz_io_scrape
                - action: replace
                  regex: (.+)
                  source_labels:
                  - __meta_kubernetes_pod_annotation_signoz_io_path
                  target_label: __metrics_path__
                - action: replace
                  separator: ':'
                  source_labels:
                  - __meta_kubernetes_pod_ip
                  - __meta_kubernetes_pod_annotation_signoz_io_port
                  target_label: __address__
                - replacement: pod_metrics
                  target_label: job_name
                - action: replace
                  source_labels:
                  - __meta_kubernetes_namespace
                  target_label: k8s_namespace_name
                - action: replace
                  source_labels:
                  - __meta_kubernetes_pod_name
                  target_label: k8s_pod_name
                - action: replace
                  source_labels:
                  - __meta_kubernetes_pod_uid
                  target_label: k8s_pod_uid
                - action: replace
                  source_labels:
                  - __meta_kubernetes_pod_node_name
                  target_label: k8s_node_name
                - action: replace
                  source_labels:
                  - __meta_kubernetes_pod_ready
                  target_label: k8s_pod_ready
                - action: replace
                  source_labels:
                  - __meta_kubernetes_pod_phase
                  target_label: k8s_pod_phase
                scrape_interval: 60s
        service:
          pipelines:
            "metrics/internal":
              receivers: [prometheus]
              processors: [batch]
              exporters: []

[tuxillo]

Has there been any progress on this one?

[grandwizard28]

Apologies for the late response here, we have come up with a plan to incorporate this change. The wider discussion and tracking is being done in #593.

[prashant-shahi]

@jkgeyti-cogna @tuxillo We have added support for metrics scraping via annotations in the K8s-Infra chart from chart version k8s-infra-0.12.0.

If you have SigNoz running in the same cluster, remember to disable OtelCollectorMetrics to avoid data duplication.