Rust Tokio Trace Opentelemetry Pipeline appears to generate extra X-Ray traces

[sarmst22]

Hi folks, I've been working on getting the OpenTelemetry Collector AWS Lambda Extension layer working with the Rust open-telemetry implementation and Rust Lambda Runtime and I've had some successes.

I'm able to export via an otlp pipeline to X-Ray with trace information, however I've run into a roadblock as there appears to be a ton of dummy traces generated during the function invocation. Below is my current pipeline setup:

use lambda::{handler_fn, Context as LambdaContext, Error as LambdaError};
use opentelemetry::trace::Tracer;
use opentelemetry::{global, sdk::trace};
use opentelemetry_aws::trace::XrayPropagator;
use tracing::{error, info, instrument, span, warn};
use tracing::Instrument;
use tracing_futures::WithSubscriber;
use tracing_opentelemetry::OpenTelemetrySpanExt;
use tracing_subscriber::layer::SubscriberExt;
use tracing_subscriber::Registry;
use opentelemetry::KeyValue;
use opentelemetry::sdk::Resource;
use opentelemetry::{trace::FutureExt};

#[tokio::main]
async fn main() -> Result<(), LambdaError> {

    global::set_text_map_propagator(XrayPropagator::new());
    let otlp_exporter = opentelemetry_otlp::new_exporter().tonic();

    let tracer = opentelemetry_otlp::new_pipeline()
        .tracing()
        .with_trace_config(
            trace::config()
                .with_resource(Resource::new(vec![KeyValue::new("service.name", lambda_utilities::function_name(None))]))
                .with_sampler(trace::Sampler::AlwaysOn)
                .with_id_generator(trace::XrayIdGenerator::default())
        )
        .with_exporter(otlp_exporter)
        .install_batch(opentelemetry::runtime::Tokio)?;

    let otel_layer = tracing_opentelemetry::layer().with_tracer(tracer);

    let subscriber = Registry::default().with(otel_layer);

    tracing::subscriber::set_global_default(subscriber)?;

    let func = handler_fn(handler);

    lambda::run(func).await?;

    opentelemetry::global::force_flush_tracer_provider();
    opentelemetry::global::shutdown_tracer_provider();

    Ok(())
}

async fn handler(event: SqsEvent, context: LambdaContext) -> Result<(), LambdaError> {
    //internal logic
}

And my collector.yaml:

#collector.yaml in the root directory
#Set an environemnt variable 'OPENTELEMETRY_COLLECTOR_CONFIG_FILE' to '/var/task/<path/<to>/<file>'

receivers:
  otlp:
    protocols:
      grpc:
      http:

exporters:
  logging:
  awsxray:
  otlp:
    endpoint: 0.0.0.0:4317

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [logging, awsxray, otlp]
    metrics:
      receivers: [otlp]
      exporters: [logging]

Here's a json of the corresponding X-Ray Trace's I'm seeing come out the other side that are funky:

{
    "Id": "1-61732319-63616f4fcd639e2bcccff29b",
    "Duration": 0,
    "LimitExceeded": false,
    "Segments": [
        {
            "Id": "cef6d89760290342",
            "Document": {
                "id": "cef6d89760290342",
                "name": "parse_headers",
                "start_time": 1634935577.070729,
                "trace_id": "1-61732319-63616f4fcd639e2bcccff29b",
                "end_time": 1634935577.0707414,
                "fault": false,
                "error": false,
                "throttle": false,
                "aws": {
                    "xray": {
                        "auto_instrumentation": false
                    }
                },
                "metadata": {
                    "default": {
                        "code.filepath": "/Users/sarmst22/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.14.13/src/proto/h1/role.rs",
                        "code.namespace": "hyper::proto::h1::role",
                        "idle_ns": 4346,
                        "code.lineno": 70,
                        "busy_ns": 7941,
                        "otel.resource.service.name": "unknown_service"
                    }
                }
            }
        }
    ]
}

Anyone have any inclination of what I'm doing so very wrong here?

[mreimer]

Hello @sarmst22 , I don't have an answer to this very old question, but I'm wondering it worked out in the end. I'm currently trying to achieve the same thing and this post is the most practical example I could find. If you had success I would love to hear more about it.