Jaeger SPM failing with 501 Not Implemented #6320

a0ndr · 2024-12-07T17:17:38Z

a0ndr
Dec 7, 2024

Hello, I've been trying to figure this out for hours.

When trying to access the Monitor tab, a javascript error is thrown with a bunch of requests failing with 501.
If I try to access one of those endpoints, I get shown this message:

{"data":null,"total":0,"limit":0,"offset":0,"errors":[{"code":501,"msg":"metrics querying is currently disabled"}]}

But I've enabled it - at least I think so.
This is my config:

# jaeger.yml
service:
  extensions: [jaeger_storage, jaeger_query]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [jaeger_storage_exporter, spanmetrics]
    metrics/spanmetrics:
      receivers: [spanmetrics]
      exporters: [prometheus]
  telemetry:
    resource:
      service.name: jaeger
    metrics:
      level: detailed
      address: 0.0.0.0:8888
    logs:
      level: DEBUG

extensions:
  jaeger_query:
    storage:
      traces: some_storage
      metrics: some_metrics_storage
  jaeger_storage:
    backends:
      some_storage:
        memory:
          max_traces: 100000
    metric_backends:
      some_metrics_storage:
        prometheus:
          endpoint: http://prometheus:9090
          normalize_calls: true
          normalize_duration: true

connectors:
  spanmetrics:

receivers:
  otlp:
    protocols:
      grpc:
      http:
        endpoint: "0.0.0.0:4318"

processors:
  batch:

exporters:
  jaeger_storage_exporter:
    trace_storage: some_storage
  prometheus:
    endpoint: "0.0.0.0:8889"

It's just ^C ^V'd from the repo.

jaeger-ui.json:

{
  "monitor": {
    "menuEnabled": true
  },
  "dependencies": {
    "menuEnabled": false
  },
  "archiveEnabled": true
}

docker-compose.yml:

services:
  db:
    image: mariadb:10.11.7
    volumes:
      - db_data:/var/lib/mysql
    environment:
      - MARIADB_ROOT_PASSWORD=root
      - MARIADB_DATABASE=api
    ports:
      - "8081:3306"
  pma:
    image: phpmyadmin
    environment:
      - PMA_HOST=db
    ports:
      - "8082:80"
    depends_on:
      - db

  prometheus:
    image: prom/prometheus:latest
    ports:
      - "8084:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    networks:
      - ts

  jaeger:
    image: jaegertracing/all-in-one:latest
    command: ["--config-file", "/etc/jaeger/config.yml", "--query.ui-config", "/etc/jaeger/jaeger-ui.json"]
    ports:
      - "8085:16686"
      - "8086:8888"
      - "8087:8889"
      - "8088:4317"
      - "8089:4318"
    environment:
      - JAEGER_QUERY_METRICS_BACKEND=prometheus
      - JAEGER_QUERY_METRICS_PROMETHEUS_URL=http://prometheus:9090
    volumes:
      - ./jaeger.yml:/etc/jaeger/config.yml
      - ./jaeger-ui.json:/etc/jaeger/jaeger-ui.json
    networks:
      - ts

volumes:
    db_data:

networks:
  ts:
    name: ts

prometheus.yml:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'otel_metrics'
    static_configs:
      - targets: ['host.docker.internal:8080']
  - job_name: aggregated-trace-metrics
    static_configs:
      - targets: [ 'jaeger:8889' ]

Logs - Jaeger:

2024-12-07 17:51:18 2024/12/07 16:51:18 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2024-12-07 17:51:18 2024/12/07 16:51:18 application version: git-commit=65cff3c30823ea20d3dc48bae39d5685ae307da5, git-version=v1.64.0, build-date=2024-12-06T21:17:15Z
2024-12-07 17:51:18 {"level":"info","ts":1733590278.368889,"caller":"flags/service.go:99","msg":"Mounting metrics handler on admin server","route":"/metrics"}
2024-12-07 17:51:18 {"level":"info","ts":1733590278.3690712,"caller":"flags/service.go:105","msg":"Mounting expvar handler on admin server","route":"/debug/vars"}
2024-12-07 17:51:18 {"level":"info","ts":1733590278.3692749,"caller":"flags/admin.go:119","msg":"Mounting health check on admin server","route":"/"}
2024-12-07 17:51:18 {"level":"info","ts":1733590278.3694386,"caller":"flags/admin.go:133","msg":"Starting admin HTTP server","http-addr":":14269"}
2024-12-07 17:51:18 {"level":"info","ts":1733590278.3694565,"caller":"flags/admin.go:111","msg":"Admin server started","http.host-port":"[::]:14269","health-status":"unavailable"}
2024-12-07 17:51:18 {"level":"info","ts":1733590278.3695288,"caller":"[email protected]/clientconn.go:162","msg":"[core] original dial target is: \"localhost:4317\""}
2024-12-07 17:51:18 {"level":"info","ts":1733590278.3696969,"caller":"[email protected]/clientconn.go:440","msg":"[core] [Channel #1]Channel created"}
2024-12-07 17:51:18 {"level":"info","ts":1733590278.3697248,"caller":"[email protected]/clientconn.go:193","msg":"[core] [Channel #1]parsed dial target is: resolver.Target{URL:url.URL{Scheme:\"dns\", Opaque:\"\", User:(*url.Userinfo)(nil), Host:\"\", Path:\"/localhost:4317\", RawPath:\"\", OmitHost:false, ForceQuery:false, RawQuery:\"\", Fragment:\"\", RawFragment:\"\"}}"}
2024-12-07 17:51:18 {"level":"info","ts":1733590278.3697975,"caller":"[email protected]/clientconn.go:194","msg":"[core] [Channel #1]Channel authority set to \"localhost:4317\""}
2024-12-07 17:51:18 {"level":"info","ts":1733590278.3700867,"caller":"memory/factory.go:77","msg":"Memory storage initialized","configuration":{"MaxTraces":0}}
2024-12-07 17:51:18 {"level":"info","ts":1733590278.370965,"caller":"static/provider.go:149","msg":"Loading sampling strategies","filename":"/etc/jaeger/sampling_strategies.json"}
2024-12-07 17:51:18 {"level":"warn","ts":1733590278.3712661,"caller":"static/provider.go:71","msg":"Default operations level strategies will not be included for Ratelimiting service strategies.This behavior will be changed in future releases. Cf. https://github.com/jaegertracing/jaeger/issues/5270"}
2024-12-07 17:51:18 {"level":"info","ts":1733590278.3952115,"caller":"[email protected]/server.go:684","msg":"[core] [Server #2]Server created"}
2024-12-07 17:51:18 {"level":"info","ts":1733590278.3953838,"caller":"server/grpc.go:93","msg":"Starting jaeger-collector gRPC server","grpc.host-port":"[::]:14250"}
2024-12-07 17:51:18 {"level":"info","ts":1733590278.3953955,"caller":"server/http.go:45","msg":"Starting jaeger-collector HTTP server","http host-port":":14268"}
2024-12-07 17:51:18 {"level":"info","ts":1733590278.3957014,"caller":"app/collector.go:139","msg":"Not listening for Zipkin HTTP traffic, port not configured"}
2024-12-07 17:51:18 {"level":"info","ts":1733590278.396179,"caller":"handler/otlp_receiver.go:69","msg":"OTLP receiver status change","status":"StatusStarting"}
2024-12-07 17:51:18 {"level":"warn","ts":1733590278.396243,"caller":"[email protected]/warning.go:40","msg":"Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks.","documentation":"https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
2024-12-07 17:51:18 {"level":"info","ts":1733590278.3964453,"caller":"[email protected]/server.go:684","msg":"[core] [Server #4]Server created"}
2024-12-07 17:51:18 {"level":"info","ts":1733590278.3966029,"caller":"[email protected]/otlp.go:112","msg":"Starting GRPC server","endpoint":":4317"}
2024-12-07 17:51:18 {"level":"warn","ts":1733590278.3968182,"caller":"[email protected]/warning.go:40","msg":"Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks.","documentation":"https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
2024-12-07 17:51:18 {"level":"info","ts":1733590278.3957107,"caller":"[email protected]/server.go:880","msg":"[core] [Server #2 ListenSocket #3]ListenSocket created"}
2024-12-07 17:51:18 {"level":"info","ts":1733590278.3969643,"caller":"[email protected]/otlp.go:169","msg":"Starting HTTP server","endpoint":":4318"}
2024-12-07 17:51:18 {"level":"info","ts":1733590278.39699,"caller":"[email protected]/server.go:880","msg":"[core] [Server #4 ListenSocket #5]ListenSocket created"}
2024-12-07 17:51:18 {"level":"warn","ts":1733590278.3974583,"caller":"[email protected]/warning.go:40","msg":"Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks.","documentation":"https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
2024-12-07 17:51:18 {"level":"info","ts":1733590278.3975708,"caller":"[email protected]/server.go:684","msg":"[core] [Server #6]Server created"}
2024-12-07 17:51:18 {"level":"info","ts":1733590278.4258623,"caller":"app/static_handler.go:92","msg":"Using UI configuration","path":"/etc/jaeger/jaeger-ui.json"}
2024-12-07 17:51:18 {"level":"warn","ts":1733590278.4460382,"caller":"[email protected]/warning.go:40","msg":"Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks.","documentation":"https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
2024-12-07 17:51:18 {"level":"info","ts":1733590278.446269,"caller":"app/server.go:254","msg":"Query server started","http_addr":"[::]:16686","grpc_addr":"[::]:16685"}
2024-12-07 17:51:18 {"level":"info","ts":1733590278.4463565,"caller":"healthcheck/handler.go:118","msg":"Health Check state change","status":"ready"}
2024-12-07 17:51:18 {"level":"info","ts":1733590278.4463825,"caller":"app/server.go:334","msg":"Starting GRPC server","port":16685,"addr":":16685"}
2024-12-07 17:51:18 {"level":"info","ts":1733590278.4464035,"caller":"app/server.go:320","msg":"Starting HTTP server","port":16686,"addr":":16686"}
2024-12-07 17:51:18 {"level":"info","ts":1733590278.4464421,"caller":"[email protected]/server.go:880","msg":"[core] [Server #6 ListenSocket #7]ListenSocket created"}
2024-12-07 17:51:23 {"level":"info","ts":1733590283.37132,"caller":"[email protected]/clientconn.go:345","msg":"[core] [Channel #1]Channel exiting idle mode"}
2024-12-07 17:51:23 {"level":"info","ts":1733590283.398096,"caller":"[email protected]/resolver_wrapper.go:200","msg":"[core] [Channel #1]Resolver state updated: {\n  \"Addresses\": [\n    {\n      \"Addr\": \"127.0.0.1:4317\",\n      \"ServerName\": \"\",\n      \"Attributes\": null,\n      \"BalancerAttributes\": null,\n      \"Metadata\": null\n    },\n    {\n      \"Addr\": \"[::1]:4317\",\n      \"ServerName\": \"\",\n      \"Attributes\": null,\n      \"BalancerAttributes\": null,\n      \"Metadata\": null\n    }\n  ],\n  \"Endpoints\": [\n    {\n      \"Addresses\": [\n        {\n          \"Addr\": \"127.0.0.1:4317\",\n          \"ServerName\": \"\",\n          \"Attributes\": null,\n          \"BalancerAttributes\": null,\n          \"Metadata\": null\n        }\n      ],\n      \"Attributes\": null\n    },\n    {\n      \"Addresses\": [\n        {\n          \"Addr\": \"[::1]:4317\",\n          \"ServerName\": \"\",\n          \"Attributes\": null,\n          \"BalancerAttributes\": null,\n          \"Metadata\": null\n        }\n      ],\n      \"Attributes\": null\n    }\n  ],\n  \"ServiceConfig\": null,\n  \"Attributes\": null\n} (resolver returned new addresses)"}
2024-12-07 17:51:23 {"level":"info","ts":1733590283.3982933,"caller":"[email protected]/balancer_wrapper.go:109","msg":"[core] [Channel #1]Channel switches to new LB policy \"pick_first\""}
2024-12-07 17:51:23 {"level":"info","ts":1733590283.3983426,"caller":"gracefulswitch/gracefulswitch.go:193","msg":"[pick-first-lb] [pick-first-lb 0xc00066f080] Received new config {\n  \"shuffleAddressList\": false\n}, resolver state {\n  \"Addresses\": [\n    {\n      \"Addr\": \"127.0.0.1:4317\",\n      \"ServerName\": \"\",\n      \"Attributes\": null,\n      \"BalancerAttributes\": null,\n      \"Metadata\": null\n    },\n    {\n      \"Addr\": \"[::1]:4317\",\n      \"ServerName\": \"\",\n      \"Attributes\": null,\n      \"BalancerAttributes\": null,\n      \"Metadata\": null\n    }\n  ],\n  \"Endpoints\": [\n    {\n      \"Addresses\": [\n        {\n          \"Addr\": \"127.0.0.1:4317\",\n          \"ServerName\": \"\",\n          \"Attributes\": null,\n          \"BalancerAttributes\": null,\n          \"Metadata\": null\n        }\n      ],\n      \"Attributes\": null\n    },\n    {\n      \"Addresses\": [\n        {\n          \"Addr\": \"[::1]:4317\",\n          \"ServerName\": \"\",\n          \"Attributes\": null,\n          \"BalancerAttributes\": null,\n          \"Metadata\": null\n        }\n      ],\n      \"Attributes\": null\n    }\n  ],\n  \"ServiceConfig\": null,\n  \"Attributes\": null\n}"}
2024-12-07 17:51:23 {"level":"info","ts":1733590283.3983624,"caller":"[email protected]/balancer_wrapper.go:182","msg":"[core] [Channel #1 SubChannel #8]Subchannel created"}
2024-12-07 17:51:23 {"level":"info","ts":1733590283.3983724,"caller":"[email protected]/clientconn.go:544","msg":"[core] [Channel #1]Channel Connectivity change to CONNECTING"}
2024-12-07 17:51:23 {"level":"info","ts":1733590283.398412,"caller":"[email protected]/clientconn.go:1202","msg":"[core] [Channel #1 SubChannel #8]Subchannel Connectivity change to CONNECTING"}
2024-12-07 17:51:23 {"level":"info","ts":1733590283.3984275,"caller":"[email protected]/clientconn.go:1322","msg":"[core] [Channel #1 SubChannel #8]Subchannel picks a new address \"127.0.0.1:4317\" to connect"}
2024-12-07 17:51:23 {"level":"info","ts":1733590283.3984666,"caller":"pickfirst/pickfirst.go:184","msg":"[pick-first-lb] [pick-first-lb 0xc00066f080] Received SubConn state update: 0xc00066f110, {ConnectivityState:CONNECTING ConnectionError:<nil> connectedAddress:{Addr: ServerName: Attributes:<nil> BalancerAttributes:<nil> Metadata:<nil>}}"}
2024-12-07 17:51:23 {"level":"info","ts":1733590283.3989046,"caller":"[email protected]/clientconn.go:1202","msg":"[core] [Channel #1 SubChannel #8]Subchannel Connectivity change to READY"}
2024-12-07 17:51:23 {"level":"info","ts":1733590283.399079,"caller":"pickfirst/pickfirst.go:184","msg":"[pick-first-lb] [pick-first-lb 0xc00066f080] Received SubConn state update: 0xc00066f110, {ConnectivityState:READY ConnectionError:<nil> connectedAddress:{Addr:127.0.0.1:4317 ServerName:localhost:4317 Attributes:<nil> BalancerAttributes:<nil> Metadata:<nil>}}"}
2024-12-07 17:51:23 {"level":"info","ts":1733590283.3993762,"caller":"[email protected]/clientconn.go:544","msg":"[core] [Channel #1]Channel Connectivity change to READY"}

I tried everything. I just cannot figure it out. Any help is very appreciated

Answered by yurishkuro

Dec 7, 2024

image: jaegertracing/all-in-one:latest

This is Jaeger v1, but you are using v2 config. The image should be jaegertracing/jaeger

View full answer

@a0ndr · 2024-12-07T17:20:58Z

dosubot[bot]
bot Dec 7, 2024

Hey @a0ndr! I'm here to help you with your technical issues. If you need assistance with bugs, questions, or contributing, I'm here to assist you.

The 501 Not Implemented error you're encountering typically indicates that a requested feature or service is not supported by the server. In the context of Jaeger, this error often occurs when metrics querying is disabled or not properly configured ^[1].

Here are a few steps to troubleshoot and resolve the issue:

Verify Configuration: Ensure that the metrics section in your Jaeger configuration is correctly set up. Your configuration seems to have the necessary components, but double-check that the metrics store is correctly referenced in the jaeger_query extension:
```
extensions:
  jaeger_query:
    storage:
      traces: some_storage
      metrics: some_metrics_storage
```
Prometheus Configuration: Make sure that the Prometheus server is running and accessible at the specified endpoint (http://prometheus:9090). You can test this by accessing the Prometheus UI or using a tool like curl to check the endpoint.
Jaeger UI Configuration: Your jaeger-ui.json file correctly enables the Monitor tab with "menuEnabled": true. Ensure that this configuration file is correctly mounted and used by the Jaeger service.
Check Logs for Errors: Review the Jaeger logs for any errors or warnings related to metrics querying or Prometheus connectivity. The logs you provided don't show any obvious issues, but keep an eye out for any new log entries that might indicate a problem.
JavaScript Error: The TypeError indicating this.config is undefined suggests there might be an issue with how the configuration is being accessed in the Jaeger UI. The Jaeger UI handles such errors by issuing a warning and returning the default configuration ^[2]. Ensure that the UI configuration is correctly injected and accessible.

If these steps don't resolve the issue, consider checking the network connectivity between Jaeger and Prometheus, and ensure that all services are correctly started and configured in your Docker setup.

To continue talking to Dosu, mention @dosu.

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

0 replies

yurishkuro · 2024-12-07T17:28:06Z

yurishkuro
Dec 7, 2024
Maintainer

image: jaegertracing/all-in-one:latest

This is Jaeger v1, but you are using v2 config. The image should be jaegertracing/jaeger

2 replies

a0ndr Dec 7, 2024
Author

That seems to fix the SPM issue, however, now I cannot ingest any traces. The UI is just not showing my interface. I'm getting traces export: Post "http://localhost:8088/v1/traces": EOF and traces export: Post "http://localhost:8088/v1/traces": net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x00\x00\x06\x04\x00\x00\x00\x00\x00\x00\x05\x00\x00@\x00" errors. Neither gRPC or HTTP ports work
Jaeger isn't showing any errors in it's logs.

a0ndr Dec 7, 2024
Author

Okay, I don't know how, but I fixed it

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Jaeger - Distributed Tracing Platform

Jaeger SPM failing with 501 Not Implemented #6320

{{title}}

Replies: 2 comments 2 replies

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

Jaeger - Distributed Tracing Platform

Jaeger SPM failing with 501 Not Implemented #6320

a0ndr Dec 7, 2024

Replies: 2 comments · 2 replies

dosubot[bot] bot Dec 7, 2024

yurishkuro Dec 7, 2024 Maintainer

a0ndr Dec 7, 2024 Author

a0ndr Dec 7, 2024 Author

a0ndr
Dec 7, 2024

Replies: 2 comments 2 replies

dosubot[bot]
bot Dec 7, 2024

yurishkuro
Dec 7, 2024
Maintainer

a0ndr Dec 7, 2024
Author

a0ndr Dec 7, 2024
Author