Monitor tab has no data even though the spans appear in search tab

[putthiwat0cha]

What happened?

I try to trace performance from my Py service
but even though the spans appear normally in search tab

the monitor tab show no data at all, in every kinds of spans at dropdown.

.
.
.

Steps to reproduce

This is "jaeger.yml"

services:
  jaeger:
    networks:
      - "backend"
    image: "jaegertracing/all-in-one:1.62.0"
    environment:
      - "METRICS_STORAGE_TYPE=prometheus"
      - "PROMETHEUS_SERVER_URL=http://prometheus:9090"
      - "PROMETHEUS_QUERY_SUPPORT_SPANMETRICS_CONNECTOR=true"
    ports:
      - "4317:4317"
      - "14269:14269"
      - "16686:16686"
    restart: "on-failure"
  otel:
    networks:
      - "backend"
    image: "otel/opentelemetry-collector-contrib:0.113.0"
    volumes:
      - "./otel.yml:/etc/otel-collector/otel.yml"
    command: [ "--config=/etc/otel-collector/otel.yml" ]
    depends_on:
      - "jaeger"
    ports:
      - "4318:4318"
      - "8889:8889"
    restart: "on-failure"
  microsim:
    networks:
      - "backend"
    image: "yurishkuro/microsim:v0.4.1"
    command: "-d 24h -s 500ms"
    environment:
      - "OTEL_EXPORTER_OTLP_ENDPOINT=http://otel:4318"
      - "OTEL_EXPORTER_OTLP_INSECURE=true"
    depends_on:
      - "otel"
    restart: "on-failure"
  prometheus:
    networks:
      - "backend"
    image: "prom/prometheus:v2.55.1"
    volumes:
      - "./prometheus.yml:/etc/prometheus/prometheus.yml"
    command: [ "--config.file=/etc/prometheus/prometheus.yml" ]
    depends_on:
      - "jaeger"
    ports:
      - "9090:9090"
    restart: "on-failure"
networks:
  backend:
    driver: "bridge"

.
.
.

This is "otel.yml"

receivers:
  otlp:
    protocols:
      grpc:
      http:
        endpoint: "0.0.0.0:4318"
exporters:
  prometheus:
    endpoint: "0.0.0.0:8889"
  otlp:
    endpoint: "http://jaeger:4317"
    tls:
      insecure: true
connectors:
  spanmetrics:
processors:
  batch:
service:
  pipelines:
    traces:
      receivers: [ "otlp" ]
      processors: [ "batch" ]
      exporters: [ "spanmetrics" ]
    metrics/spanmetrics:
      receivers: [ "spanmetrics" ]
      processors: [ "batch" ]
      exporters: [ "prometheus" ]

.
.
.

This is "prometheus.yml"

global:
  scrape_interval: "15s"
  evaluation_interval: "15s"
scrape_configs:
  - job_name: "tracing"
    metrics_path: "/metrics"
    static_configs:
    - targets: [ "otel:8889" ]
  - job_name: "all"
    metrics_path: "/metrics"
    static_configs:
    - targets: [ "jaeger:14269" ]

.
.
.

This is command to build Jaeger to Docker Container

docker-compose -f "D:/Jaeger/jaeger.yml" up -d

.
.
.

This is state of Prometheus target

.
.
.

This is "tracing.py"

import time
import http.client
import json
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import SERVICE_NAME, Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.metrics import get_meter
trace.set_tracer_provider(TracerProvider(resource=Resource.create({SERVICE_NAME: "test-service"})))
otlp_exporter = OTLPSpanExporter(endpoint="localhost:4317", insecure=True)
trace.get_tracer_provider().add_span_processor(BatchSpanProcessor(otlp_exporter))
meter = get_meter(__name__)
request_counter = meter.create_counter("http_requests", unit="1", description="Counts number of HTTP requests")
request_duration = meter.create_histogram("http_request_duration_seconds", unit="seconds", description="Measures the duration of HTTP requests")
tracer = trace.get_tracer(__name__)
url = "api.restful-api.dev"
endpoint = "/objects"
headers = {"Accept":"*/*", "Connection":"keep-alive", "Content-Type":"application/json"}
body = {"name":"Test Object", "data": {"year":2024, "price":1.00, "CPU model":"CoreX", "Hard disk size":"1 TB"}}
while True:
    with tracer.start_as_current_span("api_call_span", kind=trace.SpanKind.CLIENT) as root_span:
        try:
            start_time = time.time()
            with tracer.start_as_current_span("http_post_span", kind=trace.SpanKind.CLIENT):
                connection = http.client.HTTPSConnection(url, timeout=10)
                connection.request("POST", endpoint, body=json.dumps(body), headers=headers)
                response = connection.getresponse()
                response_body = response.read().decode("utf-8")
                elapsed_time = time.time() - start_time
                root_span.set_attribute("http.status_code", response.status)
                root_span.set_attribute("http.method", "POST")
                root_span.set_attribute("http.url", f"https://{url}{endpoint}")
                request_counter.add(1)
                request_duration.record(elapsed_time)
                print("Response Status:", response.status)
                print("Response Body:", response_body)    
        except Exception as e:
            root_span.record_exception(e)
            root_span.set_status(trace.Status(trace.StatusCode.ERROR, str(e)))
            print("An error occurred:", str(e))
        finally:
            connection.close()
    time.sleep(30)

.
.
.

This is command to run Py service script

python D:/Jaeger/test_script/tracing.py

.
.
.

Expected behavior

If spans from Py service already appear in search tab, the monitor tab should show data as well.

[yurishkuro]

we have extensive troubleshooting section https://www.jaegertracing.io/docs/1.63/spm/#troubleshooting

[putthiwat0cha]

we have extensive troubleshooting section https://www.jaegertracing.io/docs/1.63/spm/#troubleshooting

@yurishkuro

I already review document before building these scripts but it does not help me resolve this issue at all. Also tried to modified the scripts multiple times for whole weeks as well but still have no luck.

May you please test what I provided in "Steps to reproduce" then explain what I did wrong? I have no idea how to fix it anymore...

[yurishkuro]

Please post the output of the troubleshooting steps

[putthiwat0cha]

Please post the output of the troubleshooting steps

@yurishkuro

Do you mean this one?

I think I have provided enough information already. Now, may I ask, could you please help me check what is wrong with my scripts?

I am not that expert so I have no idea how to resolve this issue if your explanation has too little detail...

[yurishkuro]

Your screenshots are not showing any requests to the APIs for retrieving SPM data.

But I was asking about metrics in Prometheus: https://www.jaegertracing.io/docs/1.63/spm/#query-prometheus

[putthiwat0cha]

Your screenshots are not showing any requests to the APIs for retrieving SPM data.

But I was asking about metrics in Prometheus: https://www.jaegertracing.io/docs/1.63/spm/#query-prometheus

@yurishkuro

• duration_bucket
• duration_milliseconds_bucket
• duration_seconds_bucket
• calls
• calls_total

All of them return nothing, in both table and graph tab.

I think it will be more efficient to resolve this issue by letting you try building my script:

https://drive.google.com/file/d/1zpLS9ecTyAQwrknIRCSsDlL3ZBqowykB/view?usp=sharing

it use only 4 commands to build

docker login --username=DOCKER_ACCOUNT_USERNAME
docker-compose -f "D:/Jaeger/jaeger.yml" up -d
pip install opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp
python D:/Jaeger/test_script/tracing.py

[yurishkuro]

All of them return nothing, in both table and graph tab.

that means the metrics are not making it from otel collector to prometheus

[putthiwat0cha]

@yurishkuro

Ok, so, how can I resolve this issue? I check my scripts structure multiple time and unable to detect what is wrong with them.

Also, I tried asking this question with multiple AIs already (ChatGPT, Perplexity, Gemini, Claude, Copilot) and none of them provide useful answer. That's why I prefer you which is the developer of this project to help me instead.

[yurishkuro]

I don't have the time to debug an issue with your setup. We have a known working setup in docker-compose/monitor/docker-compose.yml, I suggest using that and then customizing. If you break it after it was working you'd know what you changed.

[putthiwat0cha]

@yurishkuro

Well, this might sound ridiculous, but the problem is that I already tried building Jaeger from your example "https://github.com/jaegertracing/jaeger/tree/main/docker-compose/monitor", and it does not work.

It gives the exact same result as my script.

Trust me, I had already relentlessly tried every possible solution by myself before I decided to come here and create this post to ask for help.

[yurishkuro]

• Did you wait 2-3min for the data to appear?
• Did you check that you're running correct Docker image version (latest may be old version)
• I just ran JAEGER_IMAGE_TAG=1.63.0 docker compose up and it's working fine

We test the docker-compose in our CI, here's the latest successful run that is able to retrieve data points.

[dosubot[bot]]

The issue you're experiencing with the monitor tab in Jaeger showing no data, despite spans appearing in the search tab, could be related to the configuration of the Service Performance Monitoring (SPM) feature. Here are some steps and checks you can perform to resolve this:

1. Span Kind Configuration: Ensure that the spans are correctly instrumented with the server span kind. The monitor tab relies on server spans to generate metrics, and if your spans are of kind client or unspecified, they might not be included in the metrics calculations ^[1].

2. OpenTelemetry Collector Configuration: Verify that your OpenTelemetry Collector configuration is correct. Ensure that the spanmetrics processor is properly set up to generate metrics from spans and that these metrics are being exported to Prometheus. Your otel.yml seems to be configured correctly, but double-check for any typos or misconfigurations ^[1].

3. Prometheus Configuration: Check that Prometheus is correctly scraping metrics from the OpenTelemetry Collector. Ensure that the spanmetrics.calls_total and spanmetrics.duration_bucket metrics are available in Prometheus. You can verify this by querying these metrics directly in the Prometheus UI ^[1].

4. Jaeger Configuration: Make sure that the Jaeger Query component is configured with METRICS_STORAGE_TYPE=prometheus and that the PROMETHEUS_SERVER_URL is correctly pointing to your Prometheus instance. This configuration is necessary for Jaeger to query metrics from Prometheus ^[1].

5. Check Logs and Metrics: Look at the logs of your Jaeger, OpenTelemetry Collector, and Prometheus services for any errors or warnings that might indicate configuration issues. Additionally, check the /metrics endpoint of your services to ensure that metrics are being generated and exported correctly ^[1].

By following these steps, you should be able to identify and resolve the issue with the monitor tab not displaying data. If the problem persists, consider running a minimal setup locally to isolate the issue further.

To continue talking to Dosu, mention @dosu.

[dosubot[bot]]

The issue you're experiencing, where the monitor tab in Jaeger shows no data despite spans appearing in the search tab, is likely related to the configuration of the Service Performance Monitoring (SPM) feature. Here are some steps and considerations to help resolve this:

1. Span Kind Configuration: Ensure that the spans from your Python service are correctly setting the span kind. The monitor tab relies on server span kinds to avoid double-counting. If your spans are client or unspecified, they might not appear in the monitor tab ^[1].

2. SPM Configuration: Verify that the OpenTelemetry Collector and Prometheus are correctly configured to handle span metrics. Your otel.yml configuration should have the spanmetrics connector properly set up, which it seems to be. Ensure that the Prometheus server is scraping the correct metrics from the OpenTelemetry Collector ^[1].

3. Prometheus Metrics: Check if the Prometheus server has the spanmetrics.calls_total and spanmetrics.duration_bucket metrics available. These are crucial for the monitor tab to display data. You can verify this by querying Prometheus directly ^[2].

4. Jaeger Configuration: Ensure that the Jaeger Query component is configured with METRICS_STORAGE_TYPE=prometheus and that the PROMETHEUS_SERVER_URL is correctly set. This configuration seems correct in your jaeger.yml file ^[1].

5. Check Logs and Metrics: Look at the logs for any errors related to metrics processing. Also, check the /metrics endpoint of your OpenTelemetry Collector to see if spans for specific services are being received. If there are issues, the result="err" metrics will be incremented, indicating a failure to reach the Prometheus server ^[1].

By ensuring these configurations are correct and checking the relevant metrics, you should be able to resolve the issue with the monitor tab not displaying data.

To continue talking to Dosu, mention @dosu.

[putthiwat0cha]

This issue is NOT resolved yet. Any answers from humans are welcome.

[putthiwat0cha]

@yurishkuro

After multiple attempts, these are my Yml scripts now;
,
,
,
jaeger.yml

services:
  jaeger:
    image: "jaegertracing/all-in-one:1.62.0"
    environment:
      - "METRICS_STORAGE_TYPE=prometheus"
      - "PROMETHEUS_SERVER_URL=http://prometheus:9090"
      - "PROMETHEUS_QUERY_NORMALIZE_CALLS=true"
      - "PROMETHEUS_QUERY_NORMALIZE_DURATION=true"
    ports:
      - "4317:4317"
      - "16686:16686"
    networks:
      - "backend"
    restart: "on-failure"
##############################################################
  otel:
    image: "otel/opentelemetry-collector-contrib:0.112.0"
    volumes:
      - "./otel.yml:/etc/otelcol/otel.yml"
    command: "--config /etc/otelcol/otel.yml"
    ports:
      - "4318:4318"
      - "8889:8889"
    networks:
      backend:
        aliases: ["spm"]
    restart: "on-failure"
##############################################################
  microsim:
    image: "yurishkuro/microsim:v0.4.1"
    command: "-d 24h -s 500ms"
    environment:
      - "OTEL_EXPORTER_OTLP_ENDPOINT=http://otel:4318"
      - "OTEL_EXPORTER_OTLP_INSECURE=true"
    networks:
      - "backend"
    restart: "on-failure"
##############################################################
  prometheus:
    image: "prom/prometheus:v2.55.1"
    volumes:
      - "./prometheus.yml:/etc/prometheus/prometheus.yml"
    ports:
      - "9090:9090"
    networks:
      - "backend"
    restart: "on-failure"
networks:
  backend:

,
,
,
prometheus.yml

global:
  scrape_interval: "15s"
  evaluation_interval: "15s"
#################################
scrape_configs:
  - job_name: "spm"
    metrics_path: "/metrics"
    static_configs:
    - targets: ["spm:8889"]

,
,
,
otel.yml

receivers:
  otlp:
    protocols:
      http:
        endpoint: "0.0.0.0:4318"
#############################################
exporters:
  prometheus:
    endpoint: "0.0.0.0:8889"
  otlp:
    endpoint: "jaeger:4317"
    tls:
      insecure: true
#############################################
connectors:
  spanmetrics:
processors:
  batch:
service:
  pipelines:
    traces:
      receivers: ["otlp"]
      processors: ["batch"]
      exporters: ["spanmetrics", "otlp"]
    metrics/spanmetrics:
      receivers: ["spanmetrics"]
      exporters: ["prometheus"]

,
,
,
docker build command
docker-compose -f "D:/JaegerTracing/jaeger.yml" up -d
,
,
,
screenshot

[putthiwat0cha]

This is my current Py service script (call API every 30 secs)

import time
import http.client
import json
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import SERVICE_NAME, Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.metrics import get_meter
trace.set_tracer_provider(TracerProvider(resource=Resource.create({SERVICE_NAME: "test-service"})))
otlp_exporter = OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces") #SPAN_URL
trace.get_tracer_provider().add_span_processor(BatchSpanProcessor(otlp_exporter))
meter = get_meter(__name__)
request_counter = meter.create_counter("http_requests", unit="1", description="Counts number of HTTP requests")
request_duration = meter.create_histogram("http_request_duration_seconds", unit="seconds", description="Measures the duration of HTTP requests")
tracer = trace.get_tracer(__name__)
url = "api.restful-api.dev" #FULL_URL=https://api.restful-api.dev/objects
endpoint = "/objects"
headers = {"Accept":"*/*", "Connection":"keep-alive", "Content-Type":"application/json"} #API_HEADER
body = {"name":"Test Object", "data": {"year":2024, "price":1.00, "CPU model":"CoreX", "Hard disk size":"1 TB"}} #API_BODY
while True:
    with tracer.start_as_current_span("api_call_span", kind=trace.SpanKind.CLIENT) as root_span:
        try:
            start_time = time.time()
            with tracer.start_as_current_span("http_post_span", kind=trace.SpanKind.CLIENT):
                connection = http.client.HTTPSConnection(url, timeout=10)
                connection.request("POST", endpoint, body=json.dumps(body), headers=headers)
                response = connection.getresponse()
                response_body = response.read().decode("utf-8")
                elapsed_time = time.time() - start_time
                root_span.set_attribute("http.status_code", response.status)
                root_span.set_attribute("http.method", "POST")
                root_span.set_attribute("http.url", f"https://{url}{endpoint}")
                request_counter.add(1)
                request_duration.record(elapsed_time)
                print("Response Status:", response.status)
                print("Response Body:", response_body)    
        except Exception as e:
            root_span.record_exception(e)
            root_span.set_status(trace.Status(trace.StatusCode.ERROR, str(e)))
            print("An error occurred:", str(e))
        finally:
            connection.close()
    time.sleep(30) #POST_EVERY_SEC

,
,
,
I run it using this command

,
,
,
this script is able to create span at Jaeger search tab

,
,
,
and at Jaeger monitor tab

[jmLee4]

Sorry I don't quite understand, could you please summarize how you solved the problem, and focus on which part of the modification solved the problem