Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

OTEL plugin: No way to set variable values to resource attributes #12538

Open
1 task done
sriemer opened this issue Feb 5, 2024 · 10 comments · May be fixed by #13839
Open
1 task done

OTEL plugin: No way to set variable values to resource attributes #12538

sriemer opened this issue Feb 5, 2024 · 10 comments · May be fixed by #13839
Labels
plugins/opentelemetry

Comments

@sriemer
Copy link

sriemer commented Feb 5, 2024 •
edited
Loading

Is there an existing issue for this?

  • I have searched the existing issues

Kong version ($ kong version)

Kong 3.5.0

Current Behavior

All configured OTel resource attributes are taken from config as plain strings.
I do not see any possibility to set a variable value such as the current host name for example as resource attributes.

For span to OTel infrastructure linking on the Instana UI I need special OTel resource attributes such as host.name, process.pid, and container.id.

But no type of variable is resolved.

Expected Behavior

If I put

      resource_attributes:
        service.name: "kong-otel-demo"
        host.name: $host

into the OTel plugin config, then I would expect this to be resolved as the same value as from span attribute http.host. But it sends out "$host" instead. In my demo repo, the hostname of my Kong container would be "kong-gateway".

Steps To Reproduce

  1. clone our demo repo kong-tracing -> it is private for now
  2. get access to Instana UI with an own agent key and configure file .env
  3. try to configure Kong via kong.yml for resource attributes with variable values
  4. run ./run_kong_demo.sh # based on docker-compose
  5. check Instana agent trace logs for the span JSON data that gets sent out

Anything else?

Replacing the value of resource attributes with custom LUA code would be nice as well. I imagine something like:

local root_span = kong.tracing.active_span()
root_span:set_resource_attribute("host.name", ngx.var.host)

But I get an error "attempt to call method 'set_resource_attribute' (a nil value)".

Or just putting attributes where they belong to would be nice as well:

local root_span = kong.tracing.active_span()
root_span:set_attribute("resource.host.name", ngx.var.host)

Right now this adds a span attribute "resouce.host.name" instead of resolving this to a resource attribute
"tags":{"http.host":"kong-gateway" ... ,"resource":{"host.name":"kong-gateway",...}
@sriemer
Copy link
Author

sriemer commented Feb 12, 2024

@samugi Are you maybe available to look into this? TIA

@samugi
Copy link
Member

samugi commented Feb 12, 2024

@sriemer thank you for opening this issue.

The behavior you describe is expected: there is currently no support for parsing OpenTelemetry's resource_attributes as lua code. I've noted this and we may consider it based on interest. However, if you decide to submit a PR, I'll happily review it to expedite this!

Regarding your additional comments in the "Anything else?" section, the tracing PDK currently only allows setting span attributes, resource attributes are not part of a span's structure, see for example: https://github.com/open-telemetry/opentelemetry-proto/blob/main/examples/trace.json , so it would be required adding new support for this kind of customization in the tracing PDK.

@sriemer
Copy link
Author

sriemer commented Feb 13, 2024

Thanks for your reply and your offer. I might come back to it. But for now I have issues to get the container ID due to sandboxing. So I'm looking for solutions in Instana agent code right now.

@chronolaw
Copy link
Contributor

@sriemer, do you think that this issue is solved? Could you close it?

@garciasdos
Copy link

This is a common issue. I need support modifying OpenTelemetry's resource attributes too. For example, the New Relic OTEL agent asks for more information to be added on OpenTelemetry spans (https://docs.newrelic.com/docs/more-integrations/open-source-telemetry-integrations/opentelemetry/view-your-data/opentelemetry-transactions-page/)

@StarlightIbuki
Copy link
Contributor

@Water-Melon I remember that you've proposed a feature of variables in configuration. Do you have any idea?

@sriemer
Copy link
Author

sriemer commented Apr 29, 2024

What I did on our end is to go for the resource. span attribute prefix and to move those to resource attributes in the OpenTelemetryMapper of our Instana agent. I detect Kong from the scope name which is kong-internal in this case. Of cause this is only a workaround. I'm also able to find the Kong pid in the root pid namespace. But then finding out if that is the correct Kong with the correct service.instance.id is tricky.
Would be really better if the Kong OTel plugin would read from /proc/$pid/mountinfo and send the current container ID within spans.

@SuzyWangIBMer
Copy link

Is this feature request still on going? Any plan to support it in the near future? @samugi @Water-Melon

@StarlightIbuki
Copy link
Contributor

I think there are 2 features requested in this ticket: 1 for nested attributes and 1 for dynamic sources of custom attributes.
At least the latter should be simple: we can make resource_attributes's value referenceable. Any idea? @samugi

@samugi
Copy link
Member

samugi commented Oct 8, 2024 •
edited
Loading

@StarlightIbuki it would require supporting templates in order to render the resource attribute values as lua code, which is something we support in other plugins' fields, but is slightly more complex than just making the field "referenceable".
At the moment this is not being worked on, but some tracing modules are being reworked as part of different features. We might be able to squeeze this in too, I will follow up with an update if/when that happens.

@SuzyWangIBMer SuzyWangIBMer linked a pull request Nov 6, 2024 that will close this issue
Open
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
plugins/opentelemetry
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat(opentelemetry): support variable resource attributes SuzyWangIBMer/kong
6 participants
@sriemer @samugi @chronolaw @garciasdos @SuzyWangIBMer @StarlightIbuki