SIGSEGV during C2 CompileTask

[victornoel]

Please provide a brief summary of the bug

Hi, it seems that since we started using eclipse-temurin:21-jdk-noble (21.0.5+11) in our CI (via maven:3-eclipse-temurin-21), we get random JVM crashes during execution of the C2 compiler. It's always with the same trace and always during the compilation of com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector::_addFields.

I didn't find anything resembling this error in the OpenJDK JIRA nor anywhere on the internet :/

Did you test with the latest update version?

• Yes

Please provide steps to reproduce where possible

Not sure how, it's quite random

Expected Results

No crash :P

Actual Results

Random crashes :)

What Java Version are you using?

OpenJDK 64-Bit Server VM Temurin-21.0.5+11 (21.0.5+11-LTS, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)

What is your operating system and platform?

Ubuntu 24.04.1 LTS on x86

How did you install Java?

Docker image maven:3-eclipse-temurin-21

Did it work before?

Yes, I think in the previous version (21.0.4+7) but it's hard to exactly know.

Did you test with other Java versions?

Because it's using a docker image, I don't have a way to use an older version of Java unfortunately, but I will try to find a way to do that next.

Relevant log output

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007d5f17168bd9, pid=85, tid=103
#
# JRE version: OpenJDK Runtime Environment Temurin-21.0.5+11 (21.0.5+11) (build 21.0.5+11-LTS)
# Java VM: OpenJDK 64-Bit Server VM Temurin-21.0.5+11 (21.0.5+11-LTS, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# V  [libjvm.so+0xf0dbd9]  BoolNode::Ideal(PhaseGVN*, bool)+0x19
#
# Core dump will be written. Default location: /core.%e.85.%t
#
# If you would like to submit a bug report, please visit:
#   https://github.com/adoptium/adoptium-support/issues
#

---------------  S U M M A R Y ------------

Command Line: -javaagent:/cache/.m2/repository/org/jacoco/org.jacoco.agent/0.8.12/org.jacoco.agent-0.8.12-runtime.jar=destfile=/builds/server/target/jacoco-it.exec -Dmaven.repo.local=/cache/.m2/repository -Xmx8192m -XX:ActiveProcessorCount=2 -Xverify:none --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED /builds/server/target/surefire/surefirebooter9049070912728376033.jar /builds/server/target/surefire 2024-11-07T13-53-41_126-jvmRun1 surefire3272181257269677729tmp surefire_018442978085457632387tmp

Host: AMD EPYC 7B13, 16 cores, 9G, Ubuntu 24.04.1 LTS
Time: Thu Nov  7 14:07:52 2024 UTC elapsed time: 851.233062 seconds (0d 0h 14m 11s)

---------------  T H R E A D  ---------------

Current thread (0x00007d5f100cb350):  JavaThread "C2 CompilerThread0" daemon [_thread_in_native, id=103, stack(0x00007d5eed0b2000,0x00007d5eed1b2000) (1024K)]


Current CompileTask:
C2:851233 108248       4       com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector::_addFields (1097 bytes)

Stack: [0x00007d5eed0b2000,0x00007d5eed1b2000],  sp=0x00007d5eed1ad6d0,  free space=1005k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0xf0dbd9]  BoolNode::Ideal(PhaseGVN*, bool)+0x19
V  [libjvm.so+0xd5acfe]  PhaseIterGVN::transform_old(Node*)+0x9e
V  [libjvm.so+0x6b30e0]  Conv2BNode::Ideal(PhaseGVN*, bool)+0x180
V  [libjvm.so+0xd5acfe]  PhaseIterGVN::transform_old(Node*)+0x9e
V  [libjvm.so+0xd56c19]  PhaseIterGVN::optimize()+0xf9
V  [libjvm.so+0x67072f]  Compile::Optimize()+0xf9f
V  [libjvm.so+0x671e66]  Compile::Compile(ciEnv*, ciMethod*, int, Options, DirectiveSet*)+0xf26
V  [libjvm.so+0x597fbb]  C2Compiler::compile_method(ciEnv*, ciMethod*, int, bool, DirectiveSet*)+0x18b
V  [libjvm.so+0x67833e]  CompileBroker::invoke_compiler_on_method(CompileTask*)+0xc9e
V  [libjvm.so+0x67b538]  CompileBroker::compiler_thread_loop()+0x6a8
V  [libjvm.so+0x92a798]  JavaThread::thread_main_inner()+0x1d8
V  [libjvm.so+0xf7eeb8]  Thread::call_run()+0xa8
V  [libjvm.so+0xd08a8a]  thread_native_entry(Thread*)+0xda
C  [libc.so.6+0x9ca94]

[karianna]

@victornoel which version of Jackson are you usin and do you experience these crashes with Jacoco diasble?

[victornoel]

Hi @karianna, sorry for the delay.

So what I can already tell is that with 21.0.4+7 we don't get the problem (the CI has been using it for 4-5 days).

We use Jackson 2.17.2 (and seen it happen with 2.17.1 too.

We do use jacoco (0.8.11) indeed, and I haven't had the opportunity to test without it, but I will try to set that up as soon as I can.

[karianna]

Thanks @victornoel - It could well be a genuine 21.0.5 bug, but it could also be a genuine 21.0.5 C2 change that jacoco needs to adjust for (this is a not uncommon problem with trivial versions of Java and tools like it). If we can eliminate jacoco then it's easire for me to get this properly looked at upstream :-)

[victornoel]

@karianna yep, it makes sense, I will try it and come back to you. I'm not blocked by this btw

[victornoel]

I have setup a jacoco-less CI pipeline that runs regularly using maven:3-eclipse-temurin-21, we will see how it behaves after a few days :)

[vzedano]

Hello! I have the same issue in my company's pipeline after making a refactor on the code. Our GitHub action workers with temurin 21.0.5+11 have been upgraded on Friday and we started seeing issues right away on one of the two PRs.

It happens in 90% of the executions of the pipeline for the refactored code. Unfortunately, I can't provide it due to privacy policies, but if I am able to reproduce it, I'll let you know. Cheers!

Hereby, I attach the hs_err_pidXXXX.log stack.

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f66a3d0dbd9, pid=2560, tid=2578
#
# JRE version: OpenJDK Runtime Environment Temurin-21.0.5+11 (21.0.5+11) (build 21.0.5+11-LTS)
# Java VM: OpenJDK 64-Bit Server VM Temurin-21.0.5+11 (21.0.5+11-LTS, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# V  [libjvm.so+0xf0dbd9]  BoolNode::Ideal(PhaseGVN*, bool)+0x19
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E" (or dumping to /home/runner/work/pricing/pricing/core.2560)
#
# If you would like to submit a bug report, please visit:
#   https://github.com/adoptium/adoptium-support/issues
#

---------------  S U M M A R Y ------------

Command Line: -javaagent:/home/runner/.m2/repository/org/jacoco/org.jacoco.agent/0.8.12/org.jacoco.agent-0.8.12-runtime.jar=destfile=/home/runner/work/pricing/pricing/target/jacoco.exec,excludes=**/config/*:**/liquibase/executors/*:**/service/residualvalue/*:**/PricingFeature.*:**/MigrationController.* -XX:+EnableDynamicAgentLoading /home/runner/work/pricing/pricing/target/surefire/surefirebooter-20241118154552761_3.jar /home/runner/work/pricing/pricing/target/surefire 2024-11-18T15-45-52_707-jvmRun1 surefire-20241118154552761_1tmp surefire_0-20241118154552761_2tmp

Host: AMD EPYC 7763 64-Core Processor, 8 cores, 31G, Ubuntu 22.04.5 LTS
Time: Mon Nov 18 15:47:28 2024 UTC elapsed time: 96.009229 seconds (0d 0h 1m 36s)

---------------  T H R E A D  ---------------

Current thread (0x00007f669c135b30):  JavaThread "C2 CompilerThread0" daemon [_thread_in_native, id=2578, stack(0x00007f666d3fb000,0x00007f666d4fb000) (1024K)]


Current CompileTask:
C2:96009 56567 %     4       com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector::_addFields @ 117 (1097 bytes)

Stack: [0x00007f666d3fb000,0x00007f666d4fb000],  sp=0x00007f666d4f6660,  free space=1005k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0xf0dbd9]  BoolNode::Ideal(PhaseGVN*, bool)+0x19
V  [libjvm.so+0xd5acfe]  PhaseIterGVN::transform_old(Node*)+0x9e
V  [libjvm.so+0x6b30e0]  Conv2BNode::Ideal(PhaseGVN*, bool)+0x180
V  [libjvm.so+0xd5acfe]  PhaseIterGVN::transform_old(Node*)+0x9e
V  [libjvm.so+0xd56c19]  PhaseIterGVN::optimize()+0xf9
V  [libjvm.so+0x67072f]  Compile::Optimize()+0xf9f
V  [libjvm.so+0x671e66]  Compile::Compile(ciEnv*, ciMethod*, int, Options, DirectiveSet*)+0xf26
V  [libjvm.so+0x597fbb]  C2Compiler::compile_method(ciEnv*, ciMethod*, int, bool, DirectiveSet*)+0x18b
V  [libjvm.so+0x67833e]  CompileBroker::invoke_compiler_on_method(CompileTask*)+0xc9e
V  [libjvm.so+0x67b538]  CompileBroker::compiler_thread_loop()+0x6a8
V  [libjvm.so+0x92a798]  JavaThread::thread_main_inner()+0x1d8
V  [libjvm.so+0xf7eeb8]  Thread::call_run()+0xa8
V  [libjvm.so+0xd08a8a]  thread_native_entry(Thread*)+0xda

siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x000000000000002c

Registers:
RAX=0x00007f65742d1d90, RBX=0x00007f66a431eb50, RCX=0x00007f66008123d8, RDX=0x0000000000000001
RSP=0x00007f666d4f6660, RBP=0x00007f666d4f66e0, RSI=0x00007f666d4f6a30, RDI=0x00007f65742d1d58
R8 =0x0000000000000129, R9 =0x0000000000000000, R10=0x0000000000000000, R11=0x00007f6578b843c0
R12=0x00007f666d4f6a30, R13=0x0000000000000000, R14=0x00007f666d4f6a30, R15=0x00007f66a32b6db0
RIP=0x00007f66a3d0dbd9, EFLAGS=0x0000000000010206, CSGSFS=0x002b000000000033, ERR=0x0000000000000004
  TRAPNO=0x000000000000000e


Register to memory mapping:

RAX=0x00007f65742d1d90 points into unknown readable memory: 0x0000000000000000 | 00 00 00 00 00 00 00 00
RBX=0x00007f66a431eb50: <offset 0x000000000151eb50> in /usr/lib/jvm/temurin-21-jdk-amd64/lib/server/libjvm.so at 0x00007f66a2e00000
RCX=0x00007f66008123d8 points into unknown readable memory: 0x0000000000000000 | 00 00 00 00 00 00 00 00
RDX=0x0000000000000001 is an unknown value
RSP=0x00007f666d4f6660 is pointing into the stack for thread: 0x00007f669c135b30
RBP=0x00007f666d4f66e0 is pointing into the stack for thread: 0x00007f669c135b30
RSI=0x00007f666d4f6a30 is pointing into the stack for thread: 0x00007f669c135b30
RDI=0x00007f65742d1d58 points into unknown readable memory: 0x00007f66a42e4920 | 20 49 2e a4 66 7f 00 00
R8 =0x0000000000000129 is an unknown value
R9 =0x0 is null
R10=0x0 is null
R11=0x00007f6578b843c0 points into unknown readable memory: 0x00007f66a42c9900 | 00 99 2c a4 66 7f 00 00
R12=0x00007f666d4f6a30 is pointing into the stack for thread: 0x00007f669c135b30
R13=0x0 is null
R14=0x00007f666d4f6a30 is pointing into the stack for thread: 0x00007f669c135b30
R15=0x00007f66a32b6db0: <offset 0x00000000004b6db0> in /usr/lib/jvm/temurin-21-jdk-amd64/lib/server/libjvm.so at 0x00007f66a2e00000

Top of Stack: (sp=0x00007f666d4f6660)
0x00007f666d4f6660:   00007f65681a2e90 0000000000058298
0x00007f666d4f6670:   00007f66000e28a0 00007f66000e85c0
0x00007f666d4f6680:   00007f66000ea888 00007f669c133fa0
0x00007f666d4f6690:   00007f66000e85c0 00007f66000e28a0
0x00007f666d4f66a0:   00007f66000e85b0 00007f66a3ad1995
0x00007f666d4f66b0:   00007f65742d1d58 00007f66a431eb50
0x00007f666d4f66c0:   00007f666d4f6a30 00007f65742d1d58
0x00007f666d4f66d0:   00007f666d4f6a30 00007f66a32b6db0
0x00007f666d4f66e0:   00007f666d4f6720 00007f66a3b5acfe
0x00007f666d4f66f0:   00007f666d4f8a60 00007f66a3b5c1c0
0x00007f666d4f6700:   00007f666d4f6a30 00007f65742d1d58
0x00007f666d4f6710:   0000000000000000 00007f657402b810
0x00007f666d4f6720:   00007f666d4f6760 00007f66a34b30e0
0x00007f666d4f6730:   00007f6600334b18 00007f66a431eb50
0x00007f666d4f6740:   00007f666d4f6a30 00007f65f4807e90
0x00007f666d4f6750:   00007f666d4f6a30 00007f66a32b6db0
0x00007f666d4f6760:   00007f666d4f67a0 00007f66a3b5acfe
0x00007f666d4f6770:   00007f66a431e1c0 00007f666d4f8a60
0x00007f666d4f6780:   00007f666d4f6a30 00007f65f4807e90
0x00007f666d4f6790:   0000000000000135 00007f66a42ef708
0x00007f666d4f67a0:   00007f666d4f67f0 00007f66a3b56c19
0x00007f666d4f67b0:   00007f6600000000 00007f666d4f6a30
0x00007f666d4f67c0:   00007f666d4f67f0 00007f6600eabaf0
0x00007f666d4f67d0:   00007f666d4f8a60 00007f666d4f6a30
0x00007f666d4f67e0:   00007f666d4f73a0 00007f666d4f6880
0x00007f666d4f67f0:   00007f666d4f7d90 00007f66a347072f
0x00007f666d4f6800:   00007f666d4f8a60 00007f666d4f73a0
0x00007f666d4f6810:   00007f66000e28a0 00007f66000e85c0
0x00007f666d4f6820:   00007f666d4f6880 00007f66000e85c0
0x00007f666d4f6830:   00007f666d4f6a30 00007f666d4f73a0
0x00007f666d4f6840:   00007f666d4f6940 00007f669c001af0
0x00007f666d4f6850:   00007f669c133fa0 00007f6600636530 

Instructions: (pc=0x00007f66a3d0dbd9)
0x00007f66a3d0dad9:   00 4c 89 e7 e8 ae 0f dc ff 48 8d 05 5f 7c 5b 00
0x00007f66a3d0dae9:   4d 89 6c 24 38 41 c7 44 24 2c 04 00 00 00 48 83
0x00007f66a3d0daf9:   c0 10 49 89 04 24 66 48 8d 3d b9 a8 5d 00 66 66
0x00007f66a3d0db09:   48 e8 61 4f 39 ff 48 8b 00 48 8b 80 00 07 00 00
0x00007f66a3d0db19:   48 8b 80 80 00 00 00 48 8b 98 e0 02 00 00 49 8b
0x00007f66a3d0db29:   44 24 08 48 89 18 48 85 db 74 1b 48 8b 43 10 48
0x00007f66a3d0db39:   85 c0 74 12 8b 73 20 3b 73 24 74 68 8d 56 01 89
0x00007f66a3d0db49:   53 20 4c 89 24 f0 48 8d 05 02 80 5b 00 41 83 4c
0x00007f66a3d0db59:   24 30 10 41 c7 44 24 2c 04 18 00 00 48 83 c0 10
0x00007f66a3d0db69:   49 89 04 24 eb 04 90 45 31 e4 48 83 c4 08 4c 89
0x00007f66a3d0db79:   e0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 48 8d b8 80
0x00007f66a3d0db89:   02 00 00 31 d2 be 40 00 00 00 e8 c8 58 53 ff 49
0x00007f66a3d0db99:   89 c4 e9 11 ff ff ff 4c 89 ef ff d0 49 89 c5 e9
0x00007f66a3d0dba9:   28 ff ff ff 48 89 df e8 8b 12 dc ff 48 8b 43 10
0x00007f66a3d0dbb9:   8b 73 20 eb 87 66 90 55 48 89 e5 41 57 41 56 41
0x00007f66a3d0dbc9:   55 41 54 53 48 83 ec 58 48 8b 47 08 4c 8b 68 08
0x00007f66a3d0dbd9:   41 8b 45 2c 83 e0 7f 83 f8 40 0f 85 77 02 00 00
0x00007f66a3d0dbe9:   49 8b 45 00 49 89 fc 49 89 f6 4c 89 ef ff 10 41
0x00007f66a3d0dbf9:   89 c7 8d 80 56 ff ff ff 83 f8 01 0f 86 56 02 00
0x00007f66a3d0dc09:   00 41 8b 45 2c 25 ff 07 00 00 3d c0 04 00 00 0f
0x00007f66a3d0dc19:   94 c3 41 81 ff dd 01 00 00 0f 94 c0 08 c3 0f 85
0x00007f66a3d0dc29:   33 02 00 00 49 8b 45 08 4c 8b 48 10 4c 8b 50 08
0x00007f66a3d0dc39:   4c 89 4d c0 4d 85 d2 0f 84 1a 02 00 00 41 8b 44
0x00007f66a3d0dc49:   24 34 83 e0 fb 83 f8 02 0f 84 09 02 00 00 49 8b
0x00007f66a3d0dc59:   02 4c 89 d7 4c 89 55 b8 ff 10 4c 8b 4d c0 89 45
0x00007f66a3d0dc69:   c8 49 8b 01 4c 89 cf ff 10 4c 8b 55 b8 4c 8b 4d
0x00007f66a3d0dc79:   c0 41 89 c0 41 f6 42 30 10 74 24 41 f6 41 30 10
0x00007f66a3d0dc89:   75 1d 44 89 f8 83 e0 fd 83 f8 4d 74 12 41 81 f8
0x00007f66a3d0dc99:   02 01 00 00 0f 85 65 07 00 00 0f 1f 44 00 00 41
0x00007f66a3d0dca9:   83 ff 51 0f 84 46 05 00 00 41 83 ff 52 0f 85 74
0x00007f66a3d0dcb9:   01 00 00 41 f7 44 24 34 fb ff ff ff 0f 85 ad 01
0x00007f66a3d0dcc9:   00 00 83 7d c8 1d 0f 85 a3 01 00 00 41 83 f8 7c 


Stack slot to memory mapping:

stack at sp + 0 slots: 0x00007f65681a2e90 points into unknown readable memory: 0x00007f66a422cbc0 | c0 cb 22 a4 66 7f 00 00
stack at sp + 1 slots: 0x0000000000058298 is an unknown value
stack at sp + 2 slots: 0x00007f66000e28a0 points into unknown readable memory: 0x0000000000000000 | 00 00 00 00 00 00 00 00
stack at sp + 3 slots: 0x00007f66000e85c0 points into unknown readable memory: 0x747361662e6d6f63 | 63 6f 6d 2e 66 61 73 74
stack at sp + 4 slots: 0x00007f66000ea888 points into unknown readable memory: 0x0000000000000035 | 35 00 00 00 00 00 00 00
stack at sp + 5 slots: 0x00007f669c133fa0 points into unknown readable memory: 0x00000007f669c107 | 07 c1 69 f6 07 00 00 00
stack at sp + 6 slots: 0x00007f66000e85c0 points into unknown readable memory: 0x747361662e6d6f63 | 63 6f 6d 2e 66 61 73 74
stack at sp + 7 slots: 0x00007f66000e28a0 points into unknown readable memory: 0x0000000000000000 | 00 00 00 00 00 00 00 00

Update: when I ran the same pipeline with temurin 21.0.4+7.0.LTS the builds started passing.

[victornoel]

@vzedano hi, since you are so "lucky" that it fails often, would you mind checking if it fails with jacoco disabled (if you use it)? Is it also using Jackson and falling on the same place as in my report? Thanks

[victornoel]

Hi @karianna, after a week of running latest jdk 21 with jacoco disabled, I did not see the issue appear. What would be the next step then? Thanks!

[karianna]

@victornoel - I'd report this to the Jacoco team in that case. https://groups.google.com/g/jacoco

[victornoel]

@karianna will do, thx for all the help! I will report back here before closing this ticket.

[victornoel]

I created jacoco/jacoco#1801 and also posted a message in the ML.

[victornoel]

Hey @karianna, so I'm here acting just as a messenger (:P), it seems that jacoco people are happy to help us finding the problem, but they basically say (see jacoco/jacoco#1801 (comment)) that they don't see how they could fix this in jacoco to avoid such a bug because jacoco is about bytecode enhancement.

They did say that many time in the past helped track down JDK bugs and so they can help. Luckily it seems that @Godin may have also encountered that issue at its job but it's hard to reproduce. Ideally if we can get output from a fastdebug build of the JDK, that would greatly help.

Do you know if we have such an image at adoptium? Cheers

[ozlmulg]

++