feat(opentelemetry): support variable resource attributes

[SuzyWangIBMer]

Summary

This change allows render resource attributes as lua code.
For example,

  config:
    traces_endpoint: "https://otlp-red-saas.instana.io:4318/v1/traces"
    resource_attributes:
      host.id: "$(headers.host)"

host.id is 0.0.0.0:9000 or depends on headers.host value.

Checklist

• The Pull Request has tests
• A changelog file has been created under changelog/unreleased/kong or skip-changelog label added on PR if changelog is unnecessary. README.md
• There is a user-facing docs PR against https://github.com/Kong/docs.konghq.com - PUT DOCS PR HERE

Issue reference

Fix #12538

[samugi]

failing test is expected, let's rebase after #13877 is merged

[samugi]

Thank you so much for your contribution, this is great! I left a few comments for things that I feel could be improved a bit. There's some code duplication with request-transformer that we may want to take care of, but that refactor can be done later.

[samugi]

keeping this variable at the module level's scope can be problematic because it will persist across different requests, which may introduce cross-request pollution. I'd recommend we just return the compiled resource attributes from the function that computes them.

[SuzyWangIBMer]

@samugi Thanks for quick response. I have updated the pr as suggested.

[samugi]

thanks for the quick action! I left a couple more suggestions, then I think it's good to go.

[samugi]

Suggested change

	options = {
	compiled_resource_attributes = otel_utils.read_resource_attributes(conf.resource_attributes)
	}
	local compiled, err = otel_utils.read_resource_attributes(conf.resource_attributes)
	if not compiled then
	ngx_log(ngx_WARN, _log_prefix, "resource attributes template failed to compile: ", err)
	end
	options.compiled_resource_attributes = compiled

[samugi]

Suggested change

	local resource_attributes = params.options.compiled_resource_attributes
	local resource_attributes = params.options.compiled_resource_attributes
	or conf.resource_attributes

[samugi]

Suggested change

	local resource_attributes = params.options.compiled_resource_attributes
	local resource_attributes = params.options.compiled_resource_attributes
	or conf.resource_attributes

[samugi]

Suggested change

	if err then
	return nil, err
	end
	if res then
	compiled_resource_attributes[current_name] = res
	end
	if not res then
	return nil, err
	end
	compiled_resource_attributes[current_name] = res

[samugi]

Suggested change

	local otel_utils = require "kong.plugins.opentelemetry.utils"
	local dynamic_hook = require "kong.dynamic_hook"
	local o11y_logs = require "kong.observability.logs"
	local kong_meta = require "kong.meta"
	local otel_utils = require "kong.plugins.opentelemetry.utils"
	local dynamic_hook = require "kong.dynamic_hook"
	local o11y_logs = require "kong.observability.logs"
	local kong_meta = require "kong.meta"
	local _log_prefix = otel_utils._log_prefix
	local ngx_log = ngx.log
	local ngx_WARN = ngx.WARN