JVM crashed on MetricFamilySamples.<init)

[roy-20230331]

Version: simpleclient 0.15.0

Dear experts,

Our application crashed on following call stack, how to resolve it? Thanks!

A fatal error has been detected by the Java Runtime Environment:

SIGSEGV (0xb) at pc=0x00007f5882a53c4f, pid=1, tid=813

JRE version: OpenJDK Runtime Environment Corretto-17.0.10.8.1 (17.0.10+8) (build 17.0.10+8-LTS)

Java VM: OpenJDK 64-Bit Server VM Corretto-17.0.10.8.1 (17.0.10+8-LTS, mixed mode, tiered, compressed class ptrs, z gc, linux-amd64)

Problematic frame:

J 32247 c2 io.prometheus.client.Collector$MetricFamilySamples.(Ljava/lang/String;Ljava/lang/String;Lio/prometheus/client/Collector$Type;Ljava/lang/String;Ljava/util/List;)V (306 bytes) @ 0x00007f5882a53c4f [0x00007f5882a53c00+0x000000000000004f]

Core dump will be written. Default location: /tmp/coredumps/core-%e-%s-%u-%g-1-%t

If you would like to submit a bug report, please visit:

https://github.com/corretto/corretto-17/issues/

Stack: [0x00007f5208356000,0x00007f5208456000], sp=0x00007f5208453a70, free space=1014k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) J 32247 c2 io.prometheus.client.Collector$MetricFamilySamples.<init>(Ljava/lang/String;Ljava/lang/String;Lio/prometheus/client/Collector$Type;Ljava/lang/String;Ljava/util/List;)V (306 bytes) @ 0x00007f5882a53c4f [0x00007f5882a53c00+0x000000000000004f] J 34871 c2 io.micrometer.prometheus.MicrometerCollector$$Lambda$3800+0x0000000081849878.apply(Ljava/lang/Object;)Ljava/lang/Object; (12 bytes) @ 0x00007f5882e7874c [0x00007f5882e78640+0x000000000000010c] J 4774 c2 java.util.stream.ReferencePipeline$3$1.accept(Ljava/lang/Object;)V [email protected] (23 bytes) @ 0x00007f5880fbaf3c [0x00007f5880fbac80+0x00000000000002bc] J 25463 c2 java.util.HashMap$ValueSpliterator.forEachRemaining(Ljava/util/function/Consumer;)V [email protected] (186 bytes) @ 0x00007f5882291738 [0x00007f58822915c0+0x0000000000000178] J 10861 c2 java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Ljava/util/stream/PipelineHelper;Ljava/util/Spliterator;)Ljava/lang/Object; [email protected] (18 bytes) @ 0x00007f58815a1930 [0x00007f58815a1720+0x0000000000000210] J 16536 c2 java.util.stream.ReferencePipeline.collect(Ljava/util/stream/Collector;)Ljava/lang/Object; [email protected] (124 bytes) @ 0x00007f5881a3deb0 [0x00007f5881a3dce0+0x00000000000001d0] J 30548 c2 io.micrometer.prometheus.MicrometerCollector.collect()Ljava/util/List; (104 bytes) @ 0x00007f58827d67ac [0x00007f58827d5d00+0x0000000000000aac] J 34499 c2 io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement()V (139 bytes) @ 0x00007f5882b78090 [0x00007f5882b77da0+0x00000000000002f0] J 51204 c2 org.springframework.boot.actuate.metrics.export.prometheus.PrometheusScrapeEndpoint.scrape(Lorg/springframework/boot/actuate/metrics/export/prometheus/TextOutputFormat;Ljava/util/Set;)Lorg/springframework/boot/actuate/endpoint/web/WebEndpointResponse; (85 bytes) @ 0x00007f588483996c [0x00007f58848396c0+0x00000000000002ac] J 51241 c2 jdk.internal.reflect.GeneratedMethodAccessor1033.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (73 bytes) @ 0x00007f5884857d74 [0x00007f5884857cc0+0x00000000000000b4] J 48078 c2 org.springframework.boot.actuate.endpoint.invoke.reflect.ReflectiveOperationInvoker.invoke(Lorg/springframework/boot/actuate/endpoint/InvocationContext;)Ljava/lang/Object; (33 bytes) @ 0x00007f58840066c8 [0x00007f5884005b40+0x0000000000000b88] J 47964 c2 org.springframework.boot.actuate.endpoint.web.servlet.AbstractWebMvcEndpointHandlerMapping$ServletWebOperationAdapter.handle(Ljavax/servlet/http/HttpServletRequest;Ljava/util/Map;)Ljava/lang/Object; (123 bytes) @ 0x00007f588311e1fc [0x00007f588311d5e0+0x0000000000000c1c] J 48403 c2 jdk.internal.reflect.GeneratedMethodAccessor577.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (73 bytes) @ 0x00007f58840f5424 [0x00007f58840f5340+0x00000000000000e4] J 47991 c2 org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(Lorg/springframework/web/context/request/NativeWebRequest;Lorg/springframework/web/method/support/ModelAndViewContainer;[Ljava/lang/Object;)Ljava/lang/Object; (58 bytes) @ 0x00007f58830e668c [0x00007f58830e6140+0x000000000000054c] J 54805 c2 org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Lorg/springframework/web/method/HandlerMethod;)Lorg/springframework/web/servlet/ModelAndView; (350 bytes) @ 0x00007f58848f12dc [0x00007f58848ef840+0x0000000000001a9c] J 47934 c2 org.springframework.web.servlet.DispatcherServlet.doDispatch(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V (543 bytes) @ 0x00007f5883f9d1c4 [0x00007f5883f9b4e0+0x0000000000001ce4] J 47897 c2 org.springframework.web.servlet.FrameworkServlet.processRequest(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V (187 bytes) @ 0x00007f5883f7da3c [0x00007f5883f7b620+0x000000000000241c] J 47994 c2 javax.servlet.http.HttpServlet.service(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V (269 bytes) @ 0x00007f5883e2e4bc [0x00007f5883e2e060+0x000000000000045c] J 48005 c2 org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V (388 bytes) @ 0x00007f5883953be8 [0x00007f5883953680+0x0000000000000568] J 47951 c2 org.apache.catalina.core.StandardWrapperValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V (2665 bytes) @ 0x00007f5883fcb9d0 [0x00007f5883fcb6e0+0x00000000000002f0] J 47931 c2 org.apache.catalina.core.StandardContextValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V (175 bytes) @ 0x00007f5883f970a0 [0x00007f5883f96bc0+0x00000000000004e0] J 47950 c2 org.apache.catalina.authenticator.AuthenticatorBase.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V (810 bytes) @ 0x00007f5883fca0f0 [0x00007f5883fc9ec0+0x0000000000000230] J 47955 c2 org.apache.catalina.core.StandardHostValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V (554 bytes) @ 0x00007f5882864734 [0x00007f5882864100+0x0000000000000634] J 48065 c2 org.apache.catalina.valves.RemoteIpValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V (1450 bytes) @ 0x00007f5883ff2f44 [0x00007f5883ff1b00+0x0000000000001444] J 47869 c2 org.apache.catalina.connector.CoyoteAdapter.service(Lorg/apache/coyote/Request;Lorg/apache/coyote/Response;)V (908 bytes) @ 0x00007f5883f40c5c [0x00007f5883f40900+0x000000000000035c] J 52867 c2 org.apache.coyote.http11.Http11Processor.service(Lorg/apache/tomcat/util/net/SocketWrapperBase;)Lorg/apache/tomcat/util/net/AbstractEndpoint$Handler$SocketState; (1248 bytes) @ 0x00007f5884d0a178 [0x00007f5884d09e00+0x0000000000000378] J 47896 c2 org.apache.coyote.AbstractProcessorLight.process(Lorg/apache/tomcat/util/net/SocketWrapperBase;Lorg/apache/tomcat/util/net/SocketEvent;)Lorg/apache/tomcat/util/net/AbstractEndpoint$Handler$SocketState; (367 bytes) @ 0x00007f5883f7a0c8 [0x00007f5883f7a000+0x00000000000000c8] J 47878 c2 org.apache.coyote.AbstractProtocol$ConnectionHandler.process(Lorg/apache/tomcat/util/net/SocketWrapperBase;Lorg/apache/tomcat/util/net/SocketEvent;)Lorg/apache/tomcat/util/net/AbstractEndpoint$Handler$SocketState; (1272 bytes) @ 0x00007f5883f557b4 [0x00007f5883f55360+0x0000000000000454] J 47798 c2 org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun()V (620 bytes) @ 0x00007f5883efb4a0 [0x00007f5883efb3a0+0x0000000000000100] J 52320% c2 org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(Lorg/apache/tomcat/util/threads/ThreadPoolExecutor$Worker;)V (187 bytes) @ 0x00007f5884b71104 [0x00007f5884b70cc0+0x0000000000000444] j org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run()V+5 J 21563 c1 org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run()V (25 bytes) @ 0x00007f587b55b7ec [0x00007f587b55b760+0x000000000000008c] J 16222 c1 java.lang.Thread.run()V [email protected] (17 bytes) @ 0x00007f587ae53994 [0x00007f587ae53840+0x0000000000000154] v ~StubRoutines::call_stub V [libjvm.so+0x859824] JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, JavaThread*)+0x334 V [libjvm.so+0x85b30c] JavaCalls::call_virtual(JavaValue*, Handle, Klass*, Symbol*, Symbol*, JavaThread*)+0x20c V [libjvm.so+0x919760] thread_entry(JavaThread*, JavaThread*)+0x70 V [libjvm.so+0xed9a47] JavaThread::run()+0x127 V [libjvm.so+0xedcbcf] Thread::call_run()+0x9f V [libjvm.so+0xc4ebb1] thread_native_entry(Thread*)+0xe1

[fstab]

I guess this is caused by a mix of different versions of client_java. Spring boot ships with client_java, and you might have included a different version of client_java in your application dependencies. Run mvn dependency:tree (or the equivalent for Gradle) to see which versions of client_java dependencies you have, and make sure it's all the same version.

[zeitlinger]

How to avoid version conflicts: https://prometheus.github.io/client_java/getting-started/quickstart/#dependency-management