Jaeger-Collector with Cassandra, occasionally seeing read tcp i/o timeout errors

[jay-annapureddy]

Hi Community,

I am using Jaeger Collector with Cassandra API of Azure Cosmos DB and occasionally I see read tcp i/o timeout errors, any suggestions on how to fix this error ?

Message: Failed to save span

Error:
Failed to insert span: failed to Exec query '[query statement="\n\t\tINSERT\n\t\tINTO traces(trace_id, span_id, span_hash, parent_id, operation_name, flags,\n\t\t\t\t start_time, duration, tags, logs, refs, process)\n\t\tVALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)" values=[3c65c6e584704d2ceb66c8a6f69ed733 6984534632810557311 -2304111467315990741 0 GET 0 1699513087558000 502 [{Key:otel.library.name ValueType:string ValueString:@opentelemetry/instrumentation-http ValueBool:false ValueInt64:0 ValueFloat64:0 ValueBinary:[]} {Key:otel.library.version ValueType:string ValueString:0.44.0 ValueBool:false ValueInt64:0 ValueFloat64:0 ValueBinary:[]} {Key:http.url ValueType:string ValueString:http://192.168.140.32:8882/probe/status/ready ValueBool:false ValueInt64:0 ValueFloat64:0 ValueBinary:[]} {Key:http.host ValueType:string ValueString:192.168.140.32:8882 ValueBool:false ValueInt64:0 ValueFloat64:0 ValueBinary:[]} {Key:net.host.name ValueType:string ValueString:192.168.140.32 ValueBool:false ValueInt64:0 ValueFloat64:0 ValueBinary:[]} {Key:http.method ValueType:string ValueString:GET ValueBool:false ValueInt64:0 ValueFloat64:0 ValueBinary:[]} {Key:http.scheme ValueType:string ValueString:http ValueBool:false ValueInt64:0 ValueFloat64:0 ValueBinary:[]} {Key:http.target ValueType:string ValueString:/probe/status/ready ValueBool:false ValueInt64:0 ValueFloat64:0 ValueBinary:[]} {Key:http.user_agent ValueType:string ValueString:kube-probe/1.27 ValueBool:false ValueInt64:0 ValueFloat64:0 ValueBinary:[]} {Key:http.flavor ValueType:string ValueString:1.1 ValueBool:false ValueInt64:0 ValueFloat64:0 ValueBinary:[]} {Key:net.transport ValueType:string ValueString:ip_tcp ValueBool:false ValueInt64:0 ValueFloat64:0 ValueBinary:[]} {Key:net.host.ip ValueType:string ValueString:::ffff:192.168.140.32 ValueBool:false ValueInt64:0 ValueFloat64:0 ValueBinary:[]} {Key:net.host.port ValueType:int64 ValueString: ValueBool:false ValueInt64:8882 ValueFloat64:0 ValueBinary:[]} {Key:net.peer.ip ValueType:string ValueString:::ffff:127.0.0.6 ValueBool:false ValueInt64:0 ValueFloat64:0 ValueBinary:[]} {Key:net.peer.port ValueType:int64 ValueString: ValueBool:false ValueInt64:36927 ValueFloat64:0 ValueBinary:[]} {Key:http.status_code ValueType:int64 ValueString: ValueBool:false ValueInt64:200 ValueFloat64:0 ValueBinary:[]} {Key:http.status_text ValueType:string ValueString:OK ValueBool:false ValueInt64:0 ValueFloat64:0 ValueBinary:[]} {Key:span.kind ValueType:string ValueString:server ValueBool:false ValueInt64:0 ValueFloat64:0 ValueBinary:[]} {Key:internal.span.format ValueType:string ValueString:proto ValueBool:false ValueInt64:0 ValueFloat64:0 ValueBinary:[]}] [] [] {ServiceName:demo-app.gef2 Tags:[{Key:telemetry.sdk.language ValueType:string ValueString:nodejs ValueBool:false ValueInt64:0 ValueFloat64:0 ValueBinary:[]} {Key:telemetry.sdk.name ValueType:string ValueString:opentelemetry ValueBool:false ValueInt64:0 ValueFloat64:0 ValueBinary:[]} {Key:telemetry.sdk.version ValueType:string ValueString:1.17.1 ValueBool:false ValueInt64:0 ValueFloat64:0 ValueBinary:[]} {Key:process.pid ValueType:int64 ValueString: ValueBool:false ValueInt64:7 ValueFloat64:0 ValueBinary:[]} {Key:process.executable.name ValueType:string ValueString:node ValueBool:false ValueInt64:0 ValueFloat64:0 ValueBinary:[]} {Key:process.executable.path ValueType:string ValueString:/usr/local/bin/node ValueBool:false ValueInt64:0 ValueFloat64:0 ValueBinary:[]} {Key:process.command_args ValueType:string ValueString:["/usr/local/bin/node","--max-http-header-size=32768","/app/dist/main.js","-p","8882"] ValueBool:false ValueInt64:0 ValueFloat64:0 ValueBinary:[]} {Key:process.runtime.version ValueType:string ValueString:18.18.2 ValueBool:false ValueInt64:0 ValueFloat64:0 ValueBinary:[]} {Key:process.runtime.name ValueType:string ValueString:nodejs ValueBool:false ValueInt64:0 ValueFloat64:0 ValueBinary:[]} {Key:process.runtime.description ValueType:string ValueString:Node.js ValueBool:false ValueInt64:0 ValueFloat64:0 ValueBinary:[]} {Key:process.command ValueType:string ValueString:/app/dist/main.js ValueBool:false ValueInt64:0 ValueFloat64:0 ValueBinary:[]} {Key:process.owner ValueType:string ValueString:root ValueBool:false ValueInt64:0 ValueFloat64:0 ValueBinary:[]}]}] consistency=LOCAL_ONE]': read tcp 192.168.141.12:60776->10.114.58.111:10350: i/o timeout

Stacktrace

github.com/jaegertracing/jaeger/cmd/collector/app.(*spanProcessor).saveSpan
github.com/jaegertracing/jaeger/cmd/collector/app/span_processor.go:164
github.com/jaegertracing/jaeger/cmd/collector/app.newSpanProcessor.ChainedProcessSpan.func3
github.com/jaegertracing/jaeger/cmd/collector/app/model_consumer.go:35
github.com/jaegertracing/jaeger/cmd/collector/app.(*spanProcessor).processItemFromQueue
github.com/jaegertracing/jaeger/cmd/collector/app/span_processor.go:204
github.com/jaegertracing/jaeger/cmd/collector/app.NewSpanProcessor.func1
github.com/jaegertracing/jaeger/cmd/collector/app/span_processor.go:78
github.com/jaegertracing/jaeger/pkg/queue.ConsumerFunc.Consume
github.com/jaegertracing/jaeger/pkg/queue/bounded_queue.go:105
github.com/jaegertracing/jaeger/pkg/queue.(*BoundedQueue).StartConsumersWithFactory.func1
github.com/jaegertracing/jaeger/pkg/queue/bounded_queue.go:84

Collector Configuration

apiVersion: apps/v1
kind: Deployment
metadata:
  name: jaeger-collector
  namespace: dejt
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jaeger-collector
  template:
    metadata:
      labels:
        app: jaeger-collector
      annotations:
        sidecar.istio.io/inject: "false"
        prometheus.io/port: "14269"
        prometheus.io/path: "/metrics"
        prometheus.io/scrape: "true"
    spec:
      containers:
        - name: jaeger-collector
          image: '#{DOCKER_REGISTRY_URL}#/#{JAEGER_REPOSITORY_NAME}#:#{JAEGER_VERSION}#'
          ports:
            - containerPort: 9411 # Default endpoint for Zipkin receiver.
            - containerPort: 4317 # Default endpoint for OpenTelemetry receiver gRPC.
            - containerPort: 4318 # Default endpoint for OpenTelemetry receiver HTTP.
            - containerPort: 14269 # Default endpoint for metrics.
          env:
            - name: LOG_LEVEL
              value: 'debug'
            - name: COLLECTOR_OTLP_ENABLED
              value: 'true'
            - name: COLLECTOR_ZIPKIN_HOST_PORT
              value: '9411'
            - name: COLLECTOR_QUEUE_SIZE
              value: '10000'
            - name: SPAN_STORAGE_TYPE
              value: cassandra
            - name: CASSANDRA_SERVERS
              value: '#{CASSANDRA_SERVER}#'
            - name: CASSANDRA_PORT
              value: '#{CASSANDRA_PORT}#'
            - name: CASSANDRA_USERNAME
              value: '#{CASSANDRA_USERNAME}#'
            - name: CASSANDRA_PASSWORD
              value: '#{CASSANDRA_PASSWORD}#'
            - name: CASSANDRA_KEYSPACE
              value: '#{CASSANDRA_KEYSPACE}#'
            - name: CASSANDRA_CONNECT_TIMEOUT
              value: '120s'
            - name: CASSANDRA_TLS_ENABLED
              value: 'true'
            - name: CASSANDRA_TLS_SKIP_HOST_VERIFY
              value: 'true'
            - name: CASSANDRA_INDEX_TAG_BLACKLIST
              value: 'client-uuid,span.kind,component,internal.span.format,ip,jaeger.version,sampler.type,sampler.param,http.method'
            - name: CASSANDRA_INDEX_LOGS
              value: 'false'

[yurishkuro]

There are too many possibilities why TCP timeout could happen - networking issues, Cassandra freezing for GC, etc.

[jay-annapureddy]

@yurishkuro thank for quick response, couple of questions

1. Is there a way to increase i/o read timeout ?
2. To not to loose any traces/spans of i/o read timeout or Azure Cosmos DB request rate too large (429) exceptions, is there a way to enforce reprocessing ?

Thank you,
Jay