prometheus.operator.podmonitors does not discover all targets consistently

[raphaelfan]

What's wrong?

We are encountering an issue with agent v0.37.2 flow mode with prometheus.operator.podmonitors. Clustering is enabled for both the agent and the component.

One of the podMonitor CR is discovering a different number of pods over time. We run this CR in multiple clusters. It would discover all targets, then after ~12hrs, it would only discover some of them for ~12hrs; after that, it would discover all targets again. The pattern then repeats itself. We are sure that the number of pods remained the same during that period.

Note that this is only happening in one CR, all the other scrape jobs are fine.

One interesting piece of information is that we have another CR that shares a similar name with the problematic one. For example, the working one is called “demo/v1-otelcollector” and the non-working one is called “demo/v1-otelcollector-tempo”. I wonder if this arrangement is hitting a bug.

We did not see the same issue with static mode operator.

Steps to reproduce

Create 2 deployments first. These will be monitored by podmonitors.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: v1-otelcollector
  namespace: unified-relay
  labels:
    purpose: debug
spec:
  replicas: 10
  selector:
    matchLabels:
      app: v1-otelcollector
  template:
    metadata:
      labels:
        app: v1-otelcollector
    spec:
      containers:
      - name: v1-otelcollector
        image: otel/opentelemetry-collector-contrib:0.53.0
        ports:
          - containerPort: 4317
            name: main
          - containerPort: 8888
            name: prom-metrics
        resources:
          requests:
            cpu: 10m
            memory: 100M
          limits:
            cpu: 100m
            memory: 200M
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: v1-otelcollector-tempo
  namespace: unified-relay
  labels:
    purpose: debug
spec:
  replicas: 15
  selector:
    matchLabels:
      app: v1-otelcollector-tempo
  template:
    metadata:
      labels:
        app: v1-otelcollector-tempo
    spec:
      containers:
      - name: v1-otelcollector-tempo
        image: otel/opentelemetry-collector-contrib:0.53.0
        ports:
          - containerPort: 4317
            name: main
          - containerPort: 8888
            name: prom-metrics
        resources:
          requests:
            cpu: 10m
            memory: 100M
          limits:
            cpu: 100m
            memory: 200M

Then create the podmonitors in the same namespace.

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  labels:
    purpose: debug
  name: v1-otelcollector
  namespace: unified-relay
spec:
  jobLabel: app.kubernetes.io/name
  namespaceSelector:
    matchNames:
      - unified-relay
  podMetricsEndpoints:
    - path: /metrics
      port: prom-metrics
  selector:
    matchLabels:
      app: v1-otelcollector
---
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  labels:
    purpose: debug
  name: v1-otelcollector-tempo
  namespace: unified-relay
spec:
  jobLabel: app.kubernetes.io/name
  namespaceSelector:
    matchNames:
      - unified-relay
  podMetricsEndpoints:
    - path: /metrics
      port: prom-metrics
  selector:
    matchLabels:
      app: v1-otelcollector-tempo

System information

No response

Software version

Grafana Agent v0.37.2

Configuration

No response

Logs

No response

[captncraig]

Thank you. This is a duplicate of #5679 which has a fix in #5862.