[demo] update prometheus otlp config

[puckpuck]

Updates Prometheus config for OTLP based on this guide provided in Prometheus docs.

Also fixes this issue in the demo repo.

[julianocosta89]

LGTM!

Should we also think about adding translation_strategy?

otlp:
  # Ingest OTLP data keeping UTF-8 characters in metric/label names.
  translation_strategy: NoUTF8EscapingWithSuffixes

https://prometheus.io/docs/guides/opentelemetry/#utf-8

I'd go for a new PR though, as all dashboards will break 🫠

[puckpuck]

LGTM!

Should we also think about adding translation_strategy?

otlp:
  # Ingest OTLP data keeping UTF-8 characters in metric/label names.
  translation_strategy: NoUTF8EscapingWithSuffixes

https://prometheus.io/docs/guides/opentelemetry/#utf-8

I'd go for a new PR though, as all dashboards will break 🫠

I also considered adding this but realized it would break all dashboards, so I opted not to. This is absolutely something we should do in a future PR.

[Ron-Norman]

Just install chart version: 0.33.8 and it dashboards are working better but several dashboards have errors or no data. I am trying to determine what the default state should be. Build this out as training environment and want to get fully functional before we proceed to break things on out own :)

What should be working by default in a default config install? My assumption is the default install should have all metrics enabled and all dashboard panels working.

Demo Dashboard

• No data for Spanmetrics panels
• Application logs, both panels have errors

OpenTelemetry Collector

• Errors on Data Flows (Error: id field is required for nodes data frame)
• Receivers panels, not data
• No Data for CPU Usage, Service Instance Count, Uptime by Service Instance, Service Instance Details

Opentelemetry Collector Data Flow

• Prometheus scrape section panels are all working
• Most of the other panels are "no data" or error state

Spanmetrics Demo Dashboard

• all seem to be working other than "no data" on "op 7 Highest Endpoint Latencies Mean Over Range "