Beyla: context propagation fails in EKS

[saurabh3460]

Hi Team

We are currently in the process of evaluating Beyla with our sre-stack.

Our primary objective is to witness Beyla in action, particularly in scenarios where we can eliminate the need for explicit instrumentation of our applications for tracing.

We've encountered a blocker, where we can view individual traces but face difficulties when it comes to observing distributed traces. Below, you'll find the information needed to reproduce this issue.

Environment:

EKS:

k8s version: v1.27.9-eks-5e0fdde
OS image: Amazon Linux 2
Kernel version: 5.10.209-198.812.amzn2.x86_64

Test with beyla as a Daemonset

Having successfully viewed individual traces, our desire to also observe distributed traces led us to follow the instructions provided in this documentation

according to this section in doc we added this volume in daemonset.yaml

          volumeMounts:
          ...
            - name: security-volume
              mountPath: /sys/kernel/security
      ...
      volumes:
      ...
        - name: security-volume
          hostPath: 
            path: /sys/kernel/security

and we checked cat /sys/kernel/security/lockdown according to docs If that file exists and the mode is anything other than [none], Beyla will not be able to perform context propagation and distributed tracing will be disabled.

root@beyla-sn4td:/# cat /sys/kernel/security/lockdown
[none] integrity confidentiality

manifests we tried with

• configmap.yaml
• daemonset.yaml
• rbac.yaml

Pod Logs

time=2024-02-26T08:14:31.821Z level=INFO msg="instrumenting process" component=discover.TraceAttacher cmd=/usr/local/bin/node pid=11722
time=2024-02-26T08:14:31.821Z level=INFO msg="new process for already instrumented executable" component=discover.TraceAttacher pid=11957 child=[] exec=/usr/local/bin/envoy
time=2024-02-26T08:14:31.822Z level=INFO msg="new process for already instrumented executable" component=discover.TraceAttacher pid=11518 child=[] exec=/usr/local/bin/pilot-agent
time=2024-02-26T08:14:31.822Z level=INFO msg="new process for already instrumented executable" component=discover.TraceAttacher pid=11895 child=[] exec=/usr/local/bin/node
time=2024-02-26T08:14:31.822Z level=INFO msg="new process for already instrumented executable" component=discover.TraceAttacher pid=11794 child=[] exec=/usr/local/bin/pilot-agent
time=2024-02-26T08:14:31.822Z level=INFO msg="new process for already instrumented executable" component=discover.TraceAttacher pid=11476 child=[11496] exec=/usr/sbin/nginx-debug
time=2024-02-26T08:14:31.822Z level=INFO msg="new process for already instrumented executable" component=discover.TraceAttacher pid=11476 child=[11497] exec=/usr/sbin/nginx-debug
time=2024-02-26T08:14:31.822Z level=INFO msg="new process for already instrumented executable" component=discover.TraceAttacher pid=11758 child=[] exec=/usr/local/bin/node
time=2024-02-26T08:14:31.823Z level=INFO msg="new process for already instrumented executable" component=discover.TraceAttacher pid=11858 child=[] exec=/usr/local/bin/envoy
time=2024-02-26T08:14:31.823Z level=INFO msg="new process for already instrumented executable" component=discover.TraceAttacher pid=11824 child=[] exec=/usr/local/bin/pilot-agent
time=2024-02-26T08:14:32.486Z level=INFO msg="Kernel in lockdown mode or older than 5.17, trace info propagation in gRPC headers is disabled." component=grpc.Tracer
.
.
.
.
W0226 09:17:38.327512   34633 reflector.go:456] k8s.io/client-go/informers/factory.go:150: watch of *v1.ReplicaSet ended with: couldn't queue object: was expecting a ReplicaSet. Got: *kube.ReplicaSetInfo
W0226 09:17:39.270917   34633 reflector.go:456] k8s.io/client-go/informers/factory.go:150: watch of *v1.Pod ended with: couldn't queue object: was expecting a Pod. Got: *kube.PodInfo
2024/02/26 09:17:41 failed to upload metrics: rpc error: code = Unimplemented desc = unknown service opentelemetry.proto.collector.metrics.v1.MetricsService
2024/02/26 09:24:11 failed to upload metrics: rpc error: code = Unimplemented desc = unknown service opentelemetry.proto.collector.metrics.v1.MetricsService
2024/02/26 09:24:11 failed to upload metrics: rpc error: code = Unimplemented desc = unknown service opentelemetry.proto.collector.metrics.v1.MetricsService
2024/02/26 09:24:11 failed to upload metrics: rpc error: code = Unimplemented desc = unknown service opentelemetry.proto.collector.metrics.v1.MetricsService

Screenshot