Flow dropping only "HEAD" span

[EvertonSA]

Hi, I have the following tail sampling on my flow definition:

otelcol.processor.tail_sampling "this" {
    decision_wait               = "180s"
    num_traces                  = 1000
    expected_new_traces_per_sec = 30

    policy {
        name = "ingress-nginx-policy"
        type = "and"
        and {
            and_sub_policy {
                name = "is_ingress_nginx"
                type = "string_attribute"
                string_attribute {
                    key    = "service.name"
                    values = ["ingress-nginx-private"]
                }
            }
            and_sub_policy {
                name = "starts_with_slash_api"
                type = "string_attribute"
                string_attribute {
                    key    				   = "http.target"
                    values 				   = ["/api/.*"]
                    enabled_regex_matching = true
                }
            }
            and_sub_policy {
                name = "not_ugly_targets"
                type = "string_attribute"
                string_attribute {
                    key                    = "http.target"
                    enabled_regex_matching = true
                    invert_match           = true
                    values = [
                        "/[a-zA-Z0-9.-]+\\.(js|css|json|webmanifest|ico)$",
                        "/api/something-service/enddpoint/subpath.*",
                    ]
                }
            }
            and_sub_policy {
                name = "not_from_server_name"
                type = "string_attribute"
                string_attribute {
                    key    				   = "http.server_name"
                    enabled_regex_matching = true
                    invert_match           = true
                    values = [
                        ".*excluded.com",
                    ]
                }
            }
        }
    }

    policy {
        name = "this-backend-policy"
        type = "and"
        and {
            and_sub_policy {
                name = "is_from_allowed_apps"
                type = "string_attribute"
                string_attribute {
                    key    = "app.kubernetes.io.instance"
                    values = argument.allowed_traced_apps.value
                }
            }
            and_sub_policy {
                // https://github.com/open-telemetry/opentelemetry-java/blob/main/sdk-extensions/autoconfigure/README.md#sampler
                name = "is_service"
                type = "string_attribute"
                string_attribute {
                    key    				   = "app.kubernetes.io.name"
                    enabled_regex_matching = true
                    values = [
                        ".*-gateway",
                        ".*-service",
                    ]
                }
            }
            and_sub_policy {
                name = "not_ugly_targets"
                type = "string_attribute"
                string_attribute {
                    key                    = "http.target"
                    enabled_regex_matching = true
                    invert_match           = true
                    values = [
                        "/assets/.*",
                        "/actuator/.*",
                        "/oauth2/.*",
                        "/login.*",
                        "/[a-zA-Z0-9.-]+\\.(js|css|json|webmanifest|ico)$",
                         "/api/something-service/enddpoint/subpath.*",
                    ]
                }
            }
            and_sub_policy {
                name = "not_ugly_routes"
                type = "string_attribute"
                string_attribute {
                    key                    = "http.route"
                    enabled_regex_matching = true
                    invert_match           = true
                    values = [	
                        "frontend-ui",
                    ]
                }
            }
        }
    }

    //policy {
    //    name = "debug-policy"
    //    type = "always_sample"
    //}

    output {
        metrics = [otelcol.processor.memory_limiter.default.input]
        traces  = [otelcol.processor.memory_limiter.default.input]
        logs    = [otelcol.processor.memory_limiter.default.input]
    }
}

I though this above was invencible, until i started to notice following on my tempo:

the reason why those traces takes a shit long time is because this endpoint "/api/something-service/enddpoint/subpath.*" is a SSE stream. by looking at the traces, it looks to me the HTTP span was correctly dropped according to the above rule, but all subsequent SSE pushed events stayed on the collector:

I honestly don't know what else to do, because i cant drop all KEYS and HGETALL operations just becasue of the SSE stream.

Does someone have ideas on what can I do?

[jpkrohling]

Unfortunately, I think you are hitting a limitation of the tail-sampling processor: you configured the processor to wait 180s before making a decision, and once that happens, either all spans being held in memory are selected, or are dropped. From that point, if any new spans belonging to the same trace are seen, they are treated as a new trace by this processor, as we don't keep track of which traces were selected or dropped. In your case, you have spans arriving way past 180s, which explains why the root span was dropped but not the child spans.

[EvertonSA]

for the ones facing the same, my solution was to drop this at the application layer before sending to the grafana agent. see https://github.com/vmaleze/opentelemetry-java-ignore-spans