Crash in JIT with vmState=0x0002000f

[pshipton]

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_sanity.openjdk_x86-64_linux_Release/9 - ub18-x86-1
jdk_lang_1
-Xdump:system:none -Xdump:heap:none -Xdump:system:events=gpf+abort+traceassert+corruptcache -XX:-JITServerTechPreviewMessage -XX:-UseCompressedOops
jdk/modules/scenarios/automaticmodules/RunWithAutomaticModules.java

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk11_j9_sanity.openjdk_x86-64_linux_Release/9/openjdk_test_output.tar.gz

15:27:08  JVMPORT030W /proc/sys/kernel/core_pattern setting "|/usr/share/apport/apport %p %s %c %d %P %E" specifies that the core dump is to be piped to an external program.  Attempting to rename either core or core.19458.
15:27:08  
15:27:08  JVMDUMP012E Error in System dump: The core file created by child process with pid = 19458 was not found. Expected to find core file with name "/home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_x86-64_linux_Release/aqa-tests/TKG/output_16331988297460/jdk_lang_1/work/scratch/0/core"
15:27:08  JVMDUMP032I JVM requested Java dump using '/home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_x86-64_linux_Release/aqa-tests/TKG/output_16331988297460/jdk_lang_1/work/scratch/0/javacore.20211002.160539.14594.0002.txt' in response to an event
15:27:08  JVMDUMP010I Java dump written to /home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_x86-64_linux_Release/aqa-tests/TKG/output_16331988297460/jdk_lang_1/work/scratch/0/javacore.20211002.160539.14594.0002.txt
15:27:08  Unhandled exception
15:27:08  Type=Segmentation error vmState=0x0002000f
15:27:08  J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
15:27:08  Handler1=00007FBCC97DD690 Handler2=00007FBCC953BAE0 InaccessibleAddress=0000000000000000
15:27:08  RDI=00007FBCA15AB860 RSI=0000000000000005 RAX=0000000000000008 RBX=00007FBCA15AB6D0
15:27:08  RCX=0000000000100000 RDX=0000000000000003 R8=0000000020000000 R9=0000000000000000
15:27:08  R10=00007FBCC2B47E80 R11=00007FBC716348BB R12=0000000000000000 R13=0000000000000000
15:27:08  R14=0000000000000000 R15=0000000000000000
15:27:08  RIP=00007FBCC39241B1 GS=0000 FS=0000 RSP=00007FBCA15AB5F0
15:27:08  EFlags=0000000000010206 CS=0033 RBP=00007FBCC3DED1E0 ERR=0000000000000004
15:27:08  TRAPNO=000000000000000E OLDMASK=0000000000000000 CR2=0000000000000000
15:27:08  xmm0 00007fbcc2fc7270 (f: 3271324160.000000, d: 6.939088e-310)
15:27:08  xmm1 0000000400000004 (f: 4.000000, d: 8.487983e-314)
15:27:08  xmm2 000000000000049b (f: 1179.000000, d: 5.825034e-321)
15:27:08  xmm3 feda0750feda0708 (f: 4275701504.000000, d: -1.115594e+303)
15:27:08  xmm4 0000000000000000 (f: 0.000000, d: 0.000000e+00)
15:27:08  xmm5 0000000000000005 (f: 5.000000, d: 2.470328e-323)
15:27:08  xmm6 0000000000000000 (f: 0.000000, d: 0.000000e+00)
15:27:08  xmm7 fedbeb08fedd1368 (f: 4275901184.000000, d: -1.196580e+303)
15:27:08  xmm8 ff597f00ff597ee0 (f: 4284055296.000000, d: -2.797496e+305)
15:27:08  xmm9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
15:27:08  xmm10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
15:27:08  xmm11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
15:27:08  xmm12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
15:27:08  xmm13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
15:27:08  xmm14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
15:27:08  xmm15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
15:27:08  Module=/home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_x86-64_linux_Release/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so
15:27:08  Module_base_address=00007FBCC2FCC000
15:27:08  Target=2_90_20211002_12 (Linux 4.15.0-143-generic)
15:27:08  CPU=amd64 (4 logical CPUs) (0x5e2804000 RAM)
15:27:08  ----------- Stack Backtrace -----------
15:27:08  (0x00007FBCC39241B1 [libj9jit29.so+0x9581b1])
15:27:08  (0x00007FBCC981C17E [libj9vm29.so+0x7e17e])
15:27:08  (0x00007FBCC2B47F46 [libj9gc29.so+0x41f46])
15:27:08  (0x00007FBCC2B3FD2D [libj9gc29.so+0x39d2d])
15:27:08  (0x00007FBCC2B3EA3F [libj9gc29.so+0x38a3f])
15:27:08  (0x00007FBCC2B415B2 [libj9gc29.so+0x3b5b2])
15:27:08  (0x00007FBCC2C7CE3B [libj9gc29.so+0x176e3b])
15:27:08  (0x00007FBCC2C317F7 [libj9gc29.so+0x12b7f7])
15:27:08  (0x00007FBCC2C31009 [libj9gc29.so+0x12b009])
15:27:08  (0x00007FBCC953C843 [libj9prt29.so+0x2a843])
15:27:08  (0x00007FBCC2C30B0F [libj9gc29.so+0x12ab0f])
15:27:08  (0x00007FBCC93054F6 [libj9thr29.so+0xe4f6])
15:27:08  (0x00007FBCCBD5D6DB [libpthread.so.0+0x76db])
15:27:08  clone+0x3f (0x00007FBCCB67171F [libc.so.6+0x12171f])

@AdamBrousseau there is a machine problem capturing the system core file, pls fix the machine.

[AdamBrousseau]

As per
#13462 (comment)
and
#13462 (comment)

root@ub18-x86-1:/home/jenkins# echo "core" > /proc/sys/kernel/core_pattern
root@ub18-x86-1:/home/jenkins# echo "line: kernel.core_pattern=core.%p" >> /etc/sysctl.conf

Try now.

Edit: Also did ub20x64-2

[pshipton]

Thanks, I was about to ask that all machines be checked. We can't necessarily repeat the crashes and generate another core, we need core collection to work when the failures occur.

[dmitripivkine]

First grinder for 10 test iterations passed, there is another one for 20: https://openj9-jenkins.osuosl.org/job/Grinder/257/

[dmitripivkine]

All jobs passed, running another grinder https://openj9-jenkins.osuosl.org/job/Grinder/258/

[JasonFengJ9]

An occurrence at an internal JDK17 0.29 release build job/Test_openjdk17_j9_special.system_x86-64_linux_Personal/61/tapResults/ (ub20x64rt1-8).

===============================================
Running test LambdaLoadTest_special_J9_5m_21 ...
===============================================
LambdaLoadTest_special_J9_5m_21 Start Time: Tue Oct 12 23:46:41 2021 Epoch Time (ms): 1634107601987
variation: Mode615
JVM_OPTIONS:  -Xcompressedrefs -Xgcpolicy:gencon -Xquickstart -Xjit 

LT  stderr Unhandled exception
LT  stderr Type=Segmentation error vmState=0x0002000f
LT  stderr J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
LT  stderr Handler1=00007F2F86D891C0 Handler2=00007F2F86AE6BA0 InaccessibleAddress=0000000000000050
LT  stderr RDI=00007F2F50008520 RSI=00007F2F5E48091B RAX=0000000000000000 RBX=00007F2F3F0F56F0
LT  stderr RCX=00007F2F3F8C46B9 RDX=00000000007340E8 R8=0000000020000000 R9=00007F2F3F0F56F0
LT  stderr R10=00007F2F84940EC0 R11=00007F2F254B8BB1 R12=0000000000000001 R13=0000000000153800
LT  stderr R14=0000000004600002 R15=0000000000000000
LT  stderr RIP=00007F2F85723054 GS=0000 FS=0000 RSP=00007F2F3F0F5610
LT  stderr EFlags=0000000000010246 CS=0033 RBP=00000000007340E0 ERR=0000000000000004
LT  stderr TRAPNO=000000000000000E OLDMASK=0000000000000000 CR2=0000000000000050
LT  stderr xmm0 00007f2f84dc0290 (f: 2229011200.000000, d: 6.909116e-310)
LT  stderr xmm1 0000000f0000000f (f: 15.000000, d: 3.182994e-313)
LT  stderr xmm2 000000000000000b (f: 11.000000, d: 5.434722e-323)
LT  stderr xmm3 0000000039626683 (f: 962750080.000000, d: 4.756617e-315)
LT  stderr xmm4 000000003df84178 (f: 1039679872.000000, d: 5.136701e-315)
LT  stderr xmm5 0036003139347648 (f: 959739456.000000, d: 1.223832e-307)
LT  stderr xmm6 000000003e883095 (f: 1049112704.000000, d: 5.183306e-315)
LT  stderr xmm7 000000003ba40000 (f: 1000603648.000000, d: 4.943639e-315)
LT  stderr xmm8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr xmm9 000000003e17cee7 (f: 1041747712.000000, d: 5.146917e-315)
LT  stderr xmm10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr xmm11 0000ff0000000000 (f: 0.000000, d: 1.385239e-309)
LT  stderr xmm12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr xmm13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr xmm14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr xmm15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr Module=/home/jenkins/workspace/Test_openjdk17_j9_special.system_x86-64_linux_Personal_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so
LT  stderr Module_base_address=00007F2F84DC5000
LT  stderr Target=2_90_20211012_200 (Linux 3.10.0-1160.42.2.el7.x86_64)
LT  stderr CPU=amd64 (4 logical CPUs) (0x1e8d0d000 RAM)
LT  stderr ----------- Stack Backtrace -----------
LT  stderr (0x00007F2F85723054 [libj9jit29.so+0x95e054])
LT  stderr (0x00007F2F86DC766E [libj9vm29.so+0x7e66e])
LT  stderr (0x00007F2F84940F86 [libj9gc29.so+0x41f86])
LT  stderr (0x00007F2F84938D6D [libj9gc29.so+0x39d6d])
LT  stderr (0x00007F2F84937A7F [libj9gc29.so+0x38a7f])
LT  stderr (0x00007F2F8493A5F2 [libj9gc29.so+0x3b5f2])
LT  stderr (0x00007F2F84A75E7B [libj9gc29.so+0x176e7b])
LT  stderr (0x00007F2F84A2A837 [libj9gc29.so+0x12b837])
LT  stderr (0x00007F2F84A2A049 [libj9gc29.so+0x12b049])
LT  stderr (0x00007F2F86AE7903 [libj9prt29.so+0x2a903])
LT  stderr (0x00007F2F84A29B4F [libj9gc29.so+0x12ab4f])
LT  stderr (0x00007F2F868B04F2 [libj9thr29.so+0xe4f2])
LT  stderr (0x00007F2F888C0EA5 [libpthread.so.0+0x7ea5])
LT  stderr clone+0x6d (0x00007F2F883E59FD [libc.so.6+0xfe9fd])
LT  stderr ---------------------------------------
LT  stderr JVMDUMP039I Processing dump event "gpf", detail "" at 2021/10/12 23:46:46 - please wait.

LambdaLoadTest_special_J9_5m_21_FAILED

[dmitripivkine]

I downloaded artifacts (results and jvm) to /team/Dmitri/13617/1/
Core location ./aqa-tests/TKG/output_16340984673980/LambdaLoadTest_special_J9_5m_21/20211012-234642-LambdaLoadTest/results/core.20211012.234646.24727.0001.dmp

The crash occur in jitWalkStackFrames()

#12 <signal handler called>
#13 0x00007f2f85723054 in jitWalkStackFrames () from /team/Dmitri/13617/1/jdk/lib/default/libj9jit29.so
#14 0x00007f2f86dc766e in walkStackFrames () from /team/Dmitri/13617/1/jdk/lib/default/libj9vm29.so
#15 0x00007f2f84940f86 in GC_VMThreadStackSlotIterator::scanSlots(J9VMThread*, J9VMThread*, void*, void (*)(J9JavaVM*, J9Object**, void*, J9StackWalkState*, void const*), bool, bool) () from /team/Dmitri/13617/1/jdk/lib/default/libj9gc29.so
#16 0x00007f2f84938d6d in MM_RootScanner::scanOneThread(MM_EnvironmentBase*, J9VMThread*, void*) () from /team/Dmitri/13617/1/jdk/lib/default/libj9gc29.so
#17 0x00007f2f84937a7f in MM_RootScanner::scanThreads(MM_EnvironmentBase*) () from /team/Dmitri/13617/1/jdk/lib/default/libj9gc29.so
#18 0x00007f2f8493a5f2 in MM_RootScanner::scanRoots(MM_EnvironmentBase*) () from /team/Dmitri/13617/1/jdk/lib/default/libj9gc29.so
#19 0x00007f2f84a75e7b in MM_Scavenger::workThreadGarbageCollect(MM_EnvironmentStandard*) () from /team/Dmitri/13617/1/jdk/lib/default/libj9gc29.so
#20 0x00007f2f84a2a837 in MM_ParallelDispatcher::workerEntryPoint(MM_EnvironmentBase*) () from /team/Dmitri/13617/1/jdk/lib/default/libj9gc29.so
#21 0x00007f2f84a2a049 in dispatcher_thread_proc2(OMRPortLibrary*, void*) () from /team/Dmitri/13617/1/jdk/lib/default/libj9gc29.so
#22 0x00007f2f86ae7903 in omrsig_protect () from /team/Dmitri/13617/1/jdk/lib/default/libj9prt29.so
#23 0x00007f2f84a29b4f in dispatcher_thread_proc () from /team/Dmitri/13617/1/jdk/lib/default/libj9gc29.so
#24 0x00007f2f868b04f2 in thread_wrapper () from /team/Dmitri/13617/1/jdk/lib/default/libj9thr29.so
#25 0x00007f2f888c0ea5 in start_thread () from /lib64/libpthread.so.0
#26 0x00007f2f883e59fd in clone () from /lib64/libc.so.6

Based on registers content I suspect java stack for thread !j9vmthread 0x005cf200
Unfortunately jdmpview can not show stack slots for this thread

@0xdaryl FYI

[babsingh]

This issue may be related to #13504 or #13499.

[pshipton]

Haven't seen this for a while, closing.

[cjjdespres]

This may have happened again in an internal test job/Test_openjdk11_j9_special.system_x86-64_linux_jit_Personal/405/. Also reported in a comment in the internal JITServer test failure thread.