'otelcol.processor.k8sattributes' - Traces are showing up with Grafana Agent information, not their source.

[jseiser]

I am trying to make my traces more searchable, so I can look them up by namespace etc. What is happening is that all my traces just end up with attributes from the grafana agent pod they were processed by. We are running the grafana agent as a daemonset in flow mode.

    otelcol.processor.k8sattributes "default" {
    extract {
      label {
        from      = "pod"
        key_regex = "(.*)/(.*)"
        tag_name  = "$1.$2"
      }

      metadata = [
        "k8s.namespace.name",
        "k8s.deployment.name",
        "k8s.statefulset.name",
        "k8s.daemonset.name",
        "k8s.cronjob.name",
        "k8s.job.name",
        "k8s.node.name",
        "k8s.pod.name",
        "k8s.pod.uid",
        "k8s.pod.start_time",
      ]
    }
    output {
      metrics = [otelcol.processor.memory_limiter.default.input]
      logs    = [otelcol.processor.memory_limiter.default.input]
      traces  = [otelcol.processor.memory_limiter.default.input]
    }
  }

Which ends up with my traces looking like the below. Note this trace is from a pod in the namespace ebs-csi, so I would expect k8s.namespace.name = ebs-csi

app.kubernetes.io.instance  "grafana-agent"
app.kubernetes.io.name  "grafana-agent"
container.id    "230dcd1933a94746a2b75f73fe22e9f92772c15ec014eb04287ff0fab5ee4caf"
host.name   "ebs-csi-node-s48hp"
k8s.daemonset.name  "grafana-agent"
k8s.namespace.name  "grafana-agent"
k8s.node.name   "i-018bb590272435fe2.us-gov-west-1.compute.internal"
k8s.pod.ip  "10.2.30.17"
k8s.pod.name    "grafana-agent-2dwwl"
k8s.pod.start_time  "2024-05-01 21:44:10 +0000 UTC"
k8s.pod.uid "9854add6-4f12-4f42-80bd-0567d8934a01"
linkerd.io.control-plane-ns "linkerd"
linkerd.io.proxy-daemonset  "grafana-agent"
linkerd.io.workload-ns  "grafana-agent"
os.description  "Amazon Linux 2023 (Linux ebs-csi-node-s48hp 6.1.82 #1 SMP PREEMPT_DYNAMIC Fri Apr  5 22:26:15 UTC 2024 x86_64)"
os.type "linux"
process.command_args    
[
    "/bin/aws-ebs-csi-driver",
    "node",
    "--endpoint=unix:/csi/csi.sock",
    "--logging-format=text",
    "--v=2",
    "--enable-otel-tracing=true"
]
process.executable.name "aws-ebs-csi-driver"
process.executable.path "/usr/bin/aws-ebs-csi-driver"
process.owner   "root"
process.pid 1
process.runtime.description "go version go1.22.2 linux/amd64"
process.runtime.name    "go"
process.runtime.version "go1.22.2"
service.name    "ebs-csi-controller"

[jseiser]

Enabled debug, but its not much help.

{
  "ts": "2024-05-10T12:42:33.135310771Z",
  "level": "debug",
  "msg": "getting the pod",
  "component": "otelcol.processor.k8sattributes.default",
  "pod": {
    "Name": "grafana-agent-s9zhj",
    "Address": "10.2.31.81",
    "PodUID": "66f9f71a-d31c-4b6f-b14d-01ad1edf6e0c",
    "Attributes": {
      "k8s.daemonset.name": "grafana-agent",
      "k8s.namespace.name": "grafana-agent",
      "k8s.node.name": "i-0d81af472cc8a841b.us-gov-west-1.compute.internal",
      "k8s.pod.name": "grafana-agent-s9zhj",
      "k8s.pod.start_time": "2024-05-02 11:00:38 +0000 UTC",
      "k8s.pod.uid": "66f9f71a-d31c-4b6f-b14d-01ad1edf6e0c"
    },
    "StartTime": "2024-05-02T11:00:38Z",
    "Ignore": false,
    "Namespace": "grafana-agent",
    "HostNetwork": false,
    "Containers": {
      "ByID": null,
      "ByName": null
    },
    "DeletedAt": "0001-01-01T00:00:00Z"
  }
}