[BUG] Error when initializing operational metrics

[ansjcy]

What is the bug?

We are seeing error when initializing the operational metrics

[2024-11-29T20:08:52,887][WARN ][stderr                   ] [214ba1566931c88a932ff8b566a00e96] java.lang.AssertionError
[2024-11-29T20:08:52,887][WARN ][stderr                   ] [214ba1566931c88a932ff8b566a00e96]  at io.opentelemetry.sdk.metrics.SdkMeter.checkValidInstrumentName(SdkMeter.java:163)
[2024-11-29T20:08:52,887][WARN ][stderr                   ] [214ba1566931c88a932ff8b566a00e96]  at io.opentelemetry.sdk.metrics.SdkMeter.counterBuilder(SdkMeter.java:88)
[2024-11-29T20:08:52,887][WARN ][stderr                   ] [214ba1566931c88a932ff8b566a00e96]  at org.opensearch.telemetry.metrics.OTelMetricsTelemetry.lambda$createCounter$0(OTelMetricsTelemetry.java:54)
[2024-11-29T20:08:52,887][WARN ][stderr                   ] [214ba1566931c88a932ff8b566a00e96]  at java.base/java.security.AccessController.doPrivileged(AccessController.java:319)
[2024-11-29T20:08:52,887][WARN ][stderr                   ] [214ba1566931c88a932ff8b566a00e96]  at org.opensearch.telemetry.metrics.OTelMetricsTelemetry.createCounter(OTelMetricsTelemetry.java:53)
[2024-11-29T20:08:52,887][WARN ][stderr                   ] [214ba1566931c88a932ff8b566a00e96]  at org.opensearch.telemetry.metrics.DefaultMetricsRegistry.createCounter(DefaultMetricsRegistry.java:33)
[2024-11-29T20:08:52,887][WARN ][stderr                   ] [214ba1566931c88a932ff8b566a00e96]  at org.opensearch.plugin.insights.core.metrics.OperationalMetricsCounter.createMetricCounter(OperationalMetricsCounter.java:89)
[2024-11-29T20:08:52,887][WARN ][stderr                   ] [214ba1566931c88a932ff8b566a00e96]  at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708)
[2024-11-29T20:08:52,887][WARN ][stderr                   ] [214ba1566931c88a932ff8b566a00e96]  at org.opensearch.plugin.insights.core.metrics.OperationalMetricsCounter.lambda$new$0(OperationalMetricsCounter.java:40)
[2024-11-29T20:08:52,887][WARN ][stderr                   ] [214ba1566931c88a932ff8b566a00e96]  at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:1024)
[2024-11-29T20:08:52,887][WARN ][stderr                   ] [214ba1566931c88a932ff8b566a00e96]  at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:762)
[2024-11-29T20:08:52,887][WARN ][stderr                   ] [214ba1566931c88a932ff8b566a00e96]  at org.opensearch.plugin.insights.core.metrics.OperationalMetricsCounter.<init>(OperationalMetricsCounter.java:40)
[2024-11-29T20:08:52,887][WARN ][stderr                   ] [214ba1566931c88a932ff8b566a00e96]  at org.opensearch.plugin.insights.core.metrics.OperationalMetricsCounter.initialize(OperationalMetricsCounter.java:51)
[2024-11-29T20:08:52,887][WARN ][stderr                   ] [214ba1566931c88a932ff8b566a00e96]  at org.opensearch.plugin.insights.QueryInsightsPlugin.createComponents(QueryInsightsPlugin.java:82)
[2024-11-29T20:08:52,887][WARN ][stderr                   ] [214ba1566931c88a932ff8b566a00e96]  at org.opensearch.node.Node.lambda$new$19(Node.java:1048)
[2024-11-29T20:08:52,887][WARN ][stderr                   ] [214ba1566931c88a932ff8b566a00e96]  at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273)
[2024-11-29T20:08:52,887][WARN ][stderr                   ] [214ba1566931c88a932ff8b566a00e96]  at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1708)
[2024-11-29T20:08:52,887][WARN ][stderr                   ] [214ba1566931c88a932ff8b566a00e96]  at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
[2024-11-29T20:08:52,887][WARN ][stderr                   ] [214ba1566931c88a932ff8b566a00e96]  at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
[2024-11-29T20:08:52,887][WARN ][stderr                   ] [214ba1566931c88a932ff8b566a00e96]  at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
[2024-11-29T20:08:52,887][WARN ][stderr                   ] [214ba1566931c88a932ff8b566a00e96]  at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
[2024-11-29T20:08:52,887][WARN ][stderr                   ] [214ba1566931c88a932ff8b566a00e96]  at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
[2024-11-29T20:08:52,887][WARN ][stderr                   ] [214ba1566931c88a932ff8b566a00e96]  at org.opensearch.node.Node.<init>(Node.java:1064)
[2024-11-29T20:08:52,887][WARN ][stderr                   ] [214ba1566931c88a932ff8b566a00e96]  at org.opensearch.node.Node.<init>(Node.java:457)
[2024-11-29T20:08:52,887][WARN ][stderr                   ] [214ba1566931c88a932ff8b566a00e96]  at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242)
[2024-11-29T20:08:52,887][WARN ][stderr                   ] [214ba1566931c88a932ff8b566a00e96]  at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242)
[2024-11-29T20:08:52,887][WARN ][stderr                   ] [214ba1566931c88a932ff8b566a00e96]  at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404)
[2024-11-29T20:08:52,887][WARN ][stderr                   ] [214ba1566931c88a932ff8b566a00e96]  at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181)
[2024-11-29T20:08:52,887][WARN ][stderr                   ] [214ba1566931c88a932ff8b566a00e96]  at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172)
[2024-11-29T20:08:52,887][WARN ][stderr                   ] [214ba1566931c88a932ff8b566a00e96]  at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
[2024-11-29T20:08:52,887][WARN ][stderr                   ] [214ba1566931c88a932ff8b566a00e96]  at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
[2024-11-29T20:08:52,887][WARN ][stderr                   ] [214ba1566931c88a932ff8b566a00e96]  at org.opensearch.cli.Command.main(Command.java:101)
[2024-11-29T20:08:52,887][WARN ][stderr                   ] [214ba1566931c88a932ff8b566a00e96]  at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138)
[2024-11-29T20:08:52,887][WARN ][stderr                   ] [214ba1566931c88a932ff8b566a00e96]  at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104)

How can one reproduce the bug?

See above, It is related to https://github.com/opensearch-project/query-insights/blame/main/src/main/java/org/opensearch/plugin/insights/core/metrics/OperationalMetricsCounter.java#L90.

The toString() method for OperationalMetric outputs the name and description of the metric. I'm pretty sure the quoted line above should really say PREFIX + metricName.name().toLowerCase(Locale.ROOT) + ".count".

The Otel SDKMeter JavaDoc related to the failing validation says:

Instrument names MUST conform to the following syntax.
They are not null or empty strings.
They are case-insensitive, ASCII strings.
The first character must be an alphabetic character.
Subsequent characters must belong to the alphanumeric characters, '_', '.', '/', and '-'.
They can have a maximum length of 255 characters.

The name+description strings produced by toString() contain space characters and parentheses, which are not valid according to those rules.

What is the expected behavior?

No error should be thrown

What is your host/environment?

Operating system, version.

Do you have any screenshots?

If applicable, add screenshots to help explain your problem.

Do you have any additional context?

Add any other context about the problem.

[ansjcy]

Triged - Let's target 2.19 to fix this and backport to older versions.