Add an example of otel instrumentation usage

[grouzen]

It is a part of: #654

[dmytr]

Hi @grouzen,

Automatic instrumentation example should work with this patch: example.patch.txt (had to add .txt so GH would allow me to upload it).

Basically everything seemed to be correct, only that OTEL doesn't have instrumentation for async-http-client. On JVM 11+ it's possible to use Java's built-in async HTTP client as sttp backend. And Jaeger setup could be simplified.

I've got both client and server spans:

Update:

Actually there is instrumentation for async-http-client: https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/async-http-client. I'm not sure why it didn't work with this backend...

[grouzen]

Thanks a lot @dmytr ! I'll try to run it with your patch and let you know about the results.

[grouzen]

@dmytr Hey! I finally managed to try your solution, and it works great!

Unfortunately, I encountered another issue with manipulating traces manually via Tracing and ContextStorage.openTelemetryContext. It seems that StorageContext.openTelemetryContext fails to restore context properly when combining auto instrumentation and manual span management.
For example, we expect https://github.com/zio/zio-telemetry/pull/660/files#diff-05c0c9e710ad79f69a2b78edfb2428484268088d4ae606771a3c6167d1a32e54R33 piece of code to add the event and some attributes to newly created internal child span, but it doesn't.

Also, I've tried to use openTelemetryContext instead of fiberRef in TracingTest. Running it by sbt -J-javaagent:$OTEL_AGENT_PATH test makes all tests fail with errors like:

Exception in thread "zio-fiber-542" java.lang.RuntimeException: Current context is root!

[dmytr]

Hi @grouzen,

The problem with the current version of this example seems to be that it creates a separate tracer which is not linked to all the machinery provided by OTEL JVM agent.

Tracer has to be obtained using GlobalOpenTelemetry.getTracer(), which is instrumented by the agent.

Theoretically, manually created tracer should work as well, as it should be using the same Context storage. I'm not sure why it doesn't 🤷‍♂️

Also there is no need to extract trace in HTTP handler as this is handled by the agent.

And there is no need to add any extra dependencies to opentelemetryInstrumentationExample as everything is already provided by the agent. We only need to add a compatible Tapir backend.

With this patch everything works:

Regarding test, I'd expect that it's the same problem - it uses a separate tracer.

[grouzen]

@dmytr Thanks! You saved me a lot of hair on my head, GlobalOpenTelemetry.getTracer() works like a charm!

Theoretically, manually created tracer should work as well, as it should be using the same Context storage. I'm not sure why it doesn't man_shrugging

It is a good topic to investigate.

Also there is no need to extract trace in HTTP handler as this is handled by the agent.

Yeah, I know. My goal was to show how to manually create child spans from those provided by instrumentation.

And there is no need to add any extra dependencies to opentelemetryInstrumentationExample as everything is already provided by the agent. We only need to add a compatible Tapir backend.

Right!

Now I have the correct span hierarchy: