[FEATURE] OpenTelemetry / Monitoring support

[clovis1122]

Is your feature request related to a problem?

I'm trying to troubleshoot a performance issue.

What solution would you like?

It would be nice if OpenSearch had some basic open telemetry support. I'm using Sentry, and would like to have how long OpenSearch requests are taking on my performance tab.

What alternatives have you considered?

Manually tracking the OpenSearch requests for now...

Do you have any additional context?

[dblock]

@clovis1122 Are you saying you'd like to instrument the client? We'd like this. Are there clients that you know do it well?

[clovis1122]

So, I don't have an example of an OpenSearch client with OpenTelemetry instrumentation, or any similar client that can be used as an example. Although I recently saw some code on drizzle-orm that looks like a good way to implement it.

To add a bit more about my particular use case - I'm using Sentry for APM & I wrapped the call to client.search within Sentry's startSpan, like so:

	const matchingData = await startSpan(
		{ name: 'searchJobs', op: 'search' },
		(span) =>
			client
				.search(searchQuery)
				.then((resp) => {
					span.setAttributes({ took: resp.body.took });
					return resp;
				})
				.catch((err) => {
					span.setStatus({ code: 2 });
					throw err;
				}),

I also get automatic instrumentation from node's http clients which this project uses.

What I am noticing is that most of the times, my custom span instrumentation and node's http call happen within 10ms which is okay. However, sometimes I see 250ms difference between them, like here:

Having OpenTelemetry instrumentation built into the client would help me better understand the time spent there.

The other big spender I have is this one, but since most of the time is spent on the http request, I guess it has nothing to do with the client (so, adding APM may not help me at all). The first request seems to end on a socket hang up error, and so, it requires a second request to complete.

[dblock]

Makes sense @clovis1122. Would be awesome if someone (you?) could add this to the client.