Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmdLineTester_criu_jitserverPostRestore zLinux assertion JITServerPersistentCHTable.cpp:172 #17474

Open
JasonFengJ9 opened this issue May 30, 2023 · 36 comments · Fixed by #19088
Assignees
Labels
comp:jit comp:jitserver Artifacts related to JIT-as-a-Service project criu Used to track CRIU snapshot related work test failure

Comments

@JasonFengJ9
Copy link
Member

JasonFengJ9 commented May 30, 2023

Failure link

From an internal build(rhel7s390x-4-1):

java version "11.0.20" 2023-07-18
IBM Semeru Runtime Certified Edition 11.0.20+2 (build 11.0.20+2)
Eclipse OpenJ9 VM 11.0.20+2 (build master-3b029b0af, JRE 11 Linux s390x-64-Bit Compressed References 20230529_567 (JIT enabled, AOT enabled)
OpenJ9   - 3b029b0af
OMR      - f89af3519
JCL      - b111539de8 based on jdk-11.0.20+2)

Rerun in Grinder - Change TARGET to run only the failed test targets.

Optional info

Failure output (captured from console output)

[2023-05-30T07:45:59.292Z] variation: -Xjit:vlog=vlog
[2023-05-30T07:45:59.292Z] JVM_OPTIONS:  -Xjit:vlog=vlog 

[2023-05-30T07:46:19.061Z] Testing: Test -Xnoaot
[2023-05-30T07:46:19.061Z] Test start time: 2023/05/30 00:46:18 Pacific Standard Time
[2023-05-30T07:46:19.061Z] Running command: bash /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criuJitServerScript.sh /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_testList_1/openjdkbinary/j2sdk-image/bin " -Xjit:vlog=vlog " org.openj9.criu.OptionsFileTest "JitOptionsTest -XX:+UseJITServer -Xnoaot" 1 false
[2023-05-30T07:46:19.061Z] Time spent starting: 14 milliseconds
[2023-05-30T07:46:36.861Z] Time spent executing: 15909 milliseconds
[2023-05-30T07:46:36.862Z] Test result: FAILED
[2023-05-30T07:46:36.862Z] Output from test:
[2023-05-30T07:46:36.862Z]  [OUT] start running script
[2023-05-30T07:46:36.862Z]  [OUT] export GLIBC_TUNABLES=glibc.cpu.hwcaps=-XSAVEC,-XSAVE,-AVX2,-ERMS,-AVX,-AVX_Fast_Unaligned_Load
[2023-05-30T07:46:36.862Z]  [OUT] export LD_BIND_NOT=on
[2023-05-30T07:46:36.862Z]  [OUT] Starting /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_testList_1/openjdkbinary/j2sdk-image/bin/jitserver -XX:JITServerPort=51945
[2023-05-30T07:46:36.862Z]  [OUT]  9467 ?        00:00:00 jitserver
[2023-05-30T07:46:36.862Z]  [OUT] JITSERVER EXISTS
[2023-05-30T07:46:36.862Z]  [OUT] Pre-checkpoint
[2023-05-30T07:46:36.862Z]  [OUT] Performing CRIUSupport.checkpointJVM(), current thread name: main, Tue May 30 00:46:20 PDT 2023, System.currentTimeMillis(): 1685432780893, System.nanoTime(): 1685432660688976596
[2023-05-30T07:46:36.862Z]  [OUT] JVMJITM043W AOT load and compilation disabled post restore.
[2023-05-30T07:46:36.862Z]  [OUT] JVMJITM043W AOT load and compilation disabled post restore.
[2023-05-30T07:46:36.862Z]  [OUT] Post-checkpoint
[2023-05-30T07:46:36.862Z]  [OUT] Removed testOutput file
[2023-05-30T07:46:36.862Z]  [OUT] JITSERVER NO LONGER EXISTS
[2023-05-30T07:46:36.862Z]  [OUT] Terminating /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_testList_1/openjdkbinary/j2sdk-image/bin/jitserver -XX:JITServerPort=51945
[2023-05-30T07:46:36.862Z]  [OUT] finished script
[2023-05-30T07:46:36.862Z]  [ERR] JITServer is currently a technology preview. Its use is not yet supported.
[2023-05-30T07:46:36.862Z]  [ERR] 
[2023-05-30T07:46:36.862Z]  [ERR] JITServer is ready to accept incoming requests
[2023-05-30T07:46:36.862Z]  [ERR] /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criuJitServerScript.sh: line 88:  9547 Killed                  $2/java -XX:+EnableCRIUSupport -XX:JITServerPort=$JITSERVER_PORT $3 -cp "$1/criu.jar" $4 $5 -XX:JITServerPort=$JITSERVER_PORT $6 > testOutput 2>&1
[2023-05-30T07:46:36.862Z]  [ERR] Assertion failed at ../../../../../openj9/runtime/compiler/env/JITServerPersistentCHTable.cpp:172: classInfo
[2023-05-30T07:46:36.862Z]  [ERR] 	subclass info cannot be null: ensure subclasses are loaded before superclass
[2023-05-30T07:46:36.862Z]  [ERR] #0: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0xa494e6) [0x3ffa91494e6]
[2023-05-30T07:46:36.862Z]  [ERR] #1: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0xa56e80) [0x3ffa9156e80]
[2023-05-30T07:46:36.862Z]  [ERR] #2: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x6c0778) [0x3ffa8dc0778]
[2023-05-30T07:46:36.862Z]  [ERR] #3: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x6c23de) [0x3ffa8dc23de]
[2023-05-30T07:46:36.862Z]  [ERR] #4: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x2a95ea) [0x3ffa89a95ea]
[2023-05-30T07:46:36.862Z]  [ERR] #5: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x2a9a78) [0x3ffa89a9a78]
[2023-05-30T07:46:36.862Z]  [ERR] #6: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x200496) [0x3ffa8900496]
[2023-05-30T07:46:36.862Z]  [ERR] #7: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x183776) [0x3ffa8883776]
[2023-05-30T07:46:36.862Z]  [ERR] #8: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x183c66) [0x3ffa8883c66]
[2023-05-30T07:46:36.862Z]  [ERR] #9: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x183d4a) [0x3ffa8883d4a]
[2023-05-30T07:46:36.862Z]  [ERR] #10: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9prt29.so(+0x34546) [0x3ffb0134546]
[2023-05-30T07:46:36.862Z]  [ERR] #11: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x1841e6) [0x3ffa88841e6]
[2023-05-30T07:46:36.862Z]  [ERR] #12: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9thr29.so(+0x5a14) [0x3ffb0085a14]
[2023-05-30T07:46:36.862Z]  [ERR] #13: /lib64/libpthread.so.0(+0x8312) [0x3ffb0a88312]
[2023-05-30T07:46:36.862Z]  [ERR] #14: /lib64/libc.so.6(+0x10e232) [0x3ffb098e232]
[2023-05-30T07:46:36.862Z]  [ERR] 
[2023-05-30T07:46:36.862Z]  [ERR] Unhandled exception
[2023-05-30T07:46:36.862Z]  [ERR] Type=Unhandled trap vmState=0x00000000
[2023-05-30T07:46:36.862Z]  [ERR] J9Generic_Signal_Number=00000108 Signal_Number=00000005 Error_Value=00000000 Signal_Code=fffffffa
[2023-05-30T07:46:36.862Z]  [ERR] Handler1=000003FFB024CBA8 Handler2=000003FFB01333D8
[2023-05-30T07:46:36.862Z]  [ERR] gpr0=0000000000000005 gpr1=000003FF8E17B910 gpr2=0000000000000000 gpr3=0000000000002502
[2023-05-30T07:46:36.862Z]  [ERR] gpr4=0000000000000005 gpr5=000003FF000024FB gpr6=000003FF8E17A110 gpr7=000003FF7C0104A0
[2023-05-30T07:46:36.862Z]  [ERR] gpr8=000003FEF4A0FEC0 gpr9=000003FF7C012470 gpr10=000000000000053E gpr11=000003FFA959FAF0
[2023-05-30T07:46:36.862Z]  [ERR] gpr12=000003FFB0A99000 gpr13=000003FFA92F4418 gpr14=000003FFA8DC2386 gpr15=000003FF8E179FD0
[2023-05-30T07:46:36.862Z]  [ERR] psw=000003FFB0A90BC0 mask=0705200180000000 fpc=0008fe00 bea=000003FFB08D81E2
[2023-05-30T07:46:36.862Z]  [ERR] fpr0 4008000000000000 (f: 0.000000, d: 3.000000e+00)
[2023-05-30T07:46:36.862Z]  [ERR] fpr1 3fff000000000000 (f: 0.000000, d: 1.937500e+00)
[2023-05-30T07:46:36.862Z]  [ERR] fpr2 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-05-30T07:46:36.862Z]  [ERR] fpr3 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-05-30T07:46:36.862Z]  [ERR] fpr4 4008000000000000 (f: 0.000000, d: 3.000000e+00)
[2023-05-30T07:46:36.862Z]  [ERR] fpr5 00000000a000c000 (f: 2684403712.000000, d: 1.326272e-314)
[2023-05-30T07:46:36.862Z]  [ERR] fpr6 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-05-30T07:46:36.862Z]  [ERR] fpr7 3e3a35f300000000 (f: 0.000000, d: 6.102663e-09)
[2023-05-30T07:46:36.862Z]  [ERR] fpr8 000003fff3efae18 (f: 4092571136.000000, d: 2.172824e-311)
[2023-05-30T07:46:36.862Z]  [ERR] fpr9 000000000014feb0 (f: 1375920.000000, d: 6.797948e-318)
[2023-05-30T07:46:36.862Z]  [ERR] fpr10 000000000014fe88 (f: 1375880.000000, d: 6.797750e-318)
[2023-05-30T07:46:36.862Z]  [ERR] fpr11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-05-30T07:46:36.862Z]  [ERR] fpr12 000003ff8018f918 (f: 2149120256.000000, d: 2.171864e-311)
[2023-05-30T07:46:36.862Z]  [ERR] fpr13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-05-30T07:46:36.862Z]  [ERR] fpr14 000000000014feb8 (f: 1375928.000000, d: 6.797988e-318)
[2023-05-30T07:46:36.862Z]  [ERR] fpr15 000000000014fe90 (f: 1375888.000000, d: 6.797790e-318)
[2023-05-30T07:46:36.862Z]  [ERR] Module=/lib64/libpthread.so.0
[2023-05-30T07:46:36.862Z]  [ERR] Module_base_address=000003FFB0A80000 Symbol=raise
[2023-05-30T07:46:36.862Z]  [ERR] Symbol_address=000003FFB0A90B90
[2023-05-30T07:46:36.862Z]  [ERR] Target=2_90_20230529_567 (Linux 3.10.0-1160.90.1.el7.s390x)
[2023-05-30T07:46:36.862Z]  [ERR] CPU=s390x (4 logical CPUs) (0x1ec5df000 RAM)
[2023-05-30T07:46:36.862Z]  [ERR] ----------- Stack Backtrace -----------
[2023-05-30T07:46:36.862Z]  [ERR] raise+0x30 (0x000003FFB0A90BC0 [libpthread.so.0+0x10bc0])
[2023-05-30T07:46:36.862Z]  [ERR] _ZN2TR4trapEv+0x5e (0x000003FFA8DC2386 [libj9jit29.so+0x6c2386])
[2023-05-30T07:46:36.862Z]  [ERR]  (0x000003FFA8DC23E4 [libj9jit29.so+0x6c23e4])
[2023-05-30T07:46:36.862Z]  [ERR] _ZN26JITServerPersistentCHTable19commitModificationsERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x402 (0x000003FFA89A95EA [libj9jit29.so+0x2a95ea])
[2023-05-30T07:46:36.862Z]  [ERR] _ZN26JITServerPersistentCHTable8doUpdateEP11TR_J9VMBaseRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES9_+0x390 (0x000003FFA89A9A78 [libj9jit29.so+0x2a9a78])
[2023-05-30T07:46:36.862Z]  [ERR] _ZN2TR30CompilationInfoPerThreadRemote12processEntryER21TR_MethodToBeCompiledRN2J917J9SegmentProviderE+0x198e (0x000003FFA8900496 [libj9jit29.so+0x200496])
[2023-05-30T07:46:36.862Z]  [ERR] _ZN2TR24CompilationInfoPerThread14processEntriesEv+0x3c6 (0x000003FFA8883776 [libj9jit29.so+0x183776])
[2023-05-30T07:46:36.862Z]  [ERR] _ZN2TR24CompilationInfoPerThread3runEv+0x66 (0x000003FFA8883C66 [libj9jit29.so+0x183c66])
[2023-05-30T07:46:36.862Z]  [ERR] _Z30protectedCompilationThreadProcP13J9PortLibraryPN2TR24CompilationInfoPerThreadE+0x9a (0x000003FFA8883D4A [libj9jit29.so+0x183d4a])
[2023-05-30T07:46:36.862Z]  [ERR] omrsig_protect+0x366 (0x000003FFB0134546 [libj9prt29.so+0x34546])
[2023-05-30T07:46:36.862Z]  [ERR] _Z21compilationThreadProcPv+0x1fe (0x000003FFA88841E6 [libj9jit29.so+0x1841e6])
[2023-05-30T07:46:36.862Z]  [ERR] thread_wrapper+0x114 (0x000003FFB0085A14 [libj9thr29.so+0x5a14])
[2023-05-30T07:46:36.862Z]  [ERR] start_thread+0xea (0x000003FFB0A88312 [libpthread.so.0+0x8312])
[2023-05-30T07:46:36.862Z]  [ERR]  (0x000003FFB098E232 [libc.so.6+0x10e232])
[2023-05-30T07:46:36.862Z]  [ERR] ---------------------------------------
[2023-05-30T07:46:36.862Z]  [ERR] JVMDUMP039I Processing dump event "gpf", detail "" at 2023/05/30 00:46:25 - please wait.

[2023-05-30T07:46:43.805Z] ---TEST RESULTS---
[2023-05-30T07:46:43.805Z] Number of PASSED tests: 4 out of 5
[2023-05-30T07:46:43.805Z] Number of FAILED tests: 1 out of 5
[2023-05-30T07:46:43.805Z] 
[2023-05-30T07:46:43.805Z] ---SUMMARY OF FAILED TESTS---
[2023-05-30T07:46:43.805Z] Test -Xnoaot
[2023-05-30T07:46:43.805Z] -----------------------------
[2023-05-30T07:46:43.805Z] 
[2023-05-30T07:46:43.805Z] -----------------------------------
[2023-05-30T07:46:43.805Z] cmdLineTester_criu_jitserverPostRestore_2_FAILED

50x internal grinder - all passed

FYI @tajila

@JasonFengJ9 JasonFengJ9 added test failure criu Used to track CRIU snapshot related work labels May 30, 2023
@pshipton
Copy link
Member

https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.functional_s390x_linux_OMR_testList_1/345
cmdLineTester_criu_jitserverPostRestore_0

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk17_j9_sanity.functional_s390x_linux_OMR_testList_1/345/functional_test_output.tar.gz

Testing: Test -Xnoaot
Test start time: 2023/06/20 13:49:25 Coordinated Universal Time
Running command: bash /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_OMR_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criuJitServerScript.sh /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_OMR_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_OMR_testList_1/openjdkbinary/j2sdk-image/bin " -Xjit " org.openj9.criu.OptionsFileTest "JitOptionsTest -XX:+UseJITServer -Xnoaot" 1 false false
Time spent starting: 4 milliseconds
Time spent executing: 31422 milliseconds
Test result: FAILED
Output from test:
 [OUT] start running script
 [OUT] export GLIBC_TUNABLES=glibc.cpu.hwcaps=-XSAVEC,-XSAVE,-AVX2,-ERMS,-AVX,-AVX_Fast_Unaligned_Load
 [OUT] export LD_BIND_NOT=on
 [OUT] Starting /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_OMR_testList_1/openjdkbinary/j2sdk-image/bin/jitserver -XX:JITServerPort=41145
 [OUT] 1874145 ?        00:00:00 jitserver
 [OUT] JITSERVER EXISTS
 [OUT] Pre-checkpoint
 [OUT] Performing CRIUSupport.checkpointJVM(), current thread name: main, Tue Jun 20 13:49:28 UTC 2023, System.currentTimeMillis(): 1687268968582, System.nanoTime(): 1687268968575296781
 [OUT] JVMJITM043W AOT load and compilation disabled post restore.
 [OUT] JVMJITM043W AOT load and compilation disabled post restore.
 [OUT] Post-checkpoint
 [OUT] Removed test output files
 [OUT] JITSERVER NO LONGER EXISTS
 [OUT] Terminating /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_OMR_testList_1/openjdkbinary/j2sdk-image/bin/jitserver -XX:JITServerPort=41145
 [OUT] finished script
 [ERR] JITServer is currently a technology preview. Its use is not yet supported.
 [ERR] 
 [ERR] JITServer is ready to accept incoming requests
 [ERR] /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_OMR_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criuJitServerScript.sh: line 91: 1874224 Killed                  $2/java -XX:+EnableCRIUSupport -XX:JITServerPort=$JITSERVER_PORT $3 -cp "$1/criu.jar" $4 $5 -XX:JITServerPort=$JITSERVER_PORT $6 > testOutput 2>&1
 [ERR] Assertion failed at /home/jenkins/workspace/Build_JDK17_s390x_linux_OMR/openj9/runtime/compiler/env/JITServerPersistentCHTable.cpp:172: classInfo
 [ERR] 	subclass info cannot be null: ensure subclasses are loaded before superclass
 [ERR] #0: /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_OMR_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0xab141e) [0x3ff8ad3141e]
 [ERR] #1: /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_OMR_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0xabf158) [0x3ff8ad3f158]
 [ERR] #2: /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_OMR_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x6f71b0) [0x3ff8a9771b0]
 [ERR] #3: /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_OMR_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x6f8a8e) [0x3ff8a978a8e]
 [ERR] #4: /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_OMR_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x2b35d8) [0x3ff8a5335d8]
 [ERR] #5: /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_OMR_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x2b3ab6) [0x3ff8a533ab6]
 [ERR] #6: /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_OMR_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x2093b0) [0x3ff8a4893b0]
 [ERR] #7: /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_OMR_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x18dcae) [0x3ff8a40dcae]
 [ERR] #8: /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_OMR_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x18e1b4) [0x3ff8a40e1b4]
 [ERR] #9: /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_OMR_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x18e24c) [0x3ff8a40e24c]
 [ERR] #10: /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_OMR_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9prt29.so(+0x32ea0) [0x3ff8c032ea0]
 [ERR] #11: /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_OMR_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x18e68e) [0x3ff8a40e68e]
 [ERR] #12: /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_OMR_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9thr29.so(+0x8a5e) [0x3ff8bf88a5e]
 [ERR] #13: /lib/s390x-linux-gnu/libpthread.so.0(+0x7e66) [0x3ff8ca87e66]
 [ERR] #14: /lib/s390x-linux-gnu/libc.so.6(+0xfcbe6) [0x3ff8c97cbe6]
 [ERR] #15: [(nil)]
 [ERR] 
 [ERR] Unhandled exception
 [ERR] Type=Unhandled trap vmState=0x00000000
 [ERR] J9Generic_Signal_Number=00000108 Signal_Number=00000005 Error_Value=00000000 Signal_Code=fffffffa
 [ERR] Handler1=000003FF8C149400 Handler2=000003FF8C031D48
 [ERR] gpr0=0000000000000005 gpr1=FFFFFFFF00000000 gpr2=0000000000000000 gpr3=000003FF70CFBE90
 [ERR] gpr4=0000000000000000 gpr5=0000000000000008 gpr6=000003FF70CFC0D8 gpr7=00000000000019A3
 [ERR] gpr8=000003FF5C016E60 gpr9=000003FF6BE94AC0 gpr10=0000000000000002 gpr11=000003FF70CFBE90
 [ERR] gpr12=000003FF8B15E000 gpr13=000003FF8AEDE578 gpr14=000003FF8A978A3A gpr15=000003FF70CFBDF0
 [ERR] psw=000003FF8CA93B16 mask=0705000180000000 fpc=00080000 bea=000003FF8C9BEE0E
 [ERR] fpr0 000003ffc4cf9670 (f: 3301938688.000000, d: 2.172433e-311)
 [ERR] fpr1 000003ff8afa632c (f: 2331665152.000000, d: 2.171954e-311)
 [ERR] fpr2 000003ffc4cfb81f (f: 3301947392.000000, d: 2.172433e-311)
 [ERR] fpr3 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] fpr4 000003ff70cf8b80 (f: 1892649856.000000, d: 2.171737e-311)
 [ERR] fpr5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] fpr6 000003ff70cf9268 (f: 1892651648.000000, d: 2.171737e-311)
 [ERR] fpr7 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] fpr8 000003ff6be80010 (f: 1810366464.000000, d: 2.171696e-311)
 [ERR] fpr9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] fpr10 000003ff70bfd000 (f: 1891618816.000000, d: 2.171736e-311)
 [ERR] fpr11 000003fff2378500 (f: 4063724800.000000, d: 2.172809e-311)
 [ERR] fpr12 000002aa4019f670 (f: 1075443328.000000, d: 1.447732e-311)
 [ERR] fpr13 000003ffa01c2218 (f: 2686198272.000000, d: 2.172129e-311)
 [ERR] fpr14 0000000000000440 (f: 1088.000000, d: 5.375434e-321)
 [ERR] fpr15 000003ffa04d9908 (f: 2689440000.000000, d: 2.172130e-311)
 [ERR] Module=/lib/s390x-linux-gnu/libpthread.so.0
 [ERR] Module_base_address=000003FF8CA80000 Symbol=raise
 [ERR] Symbol_address=000003FF8CA93A90
 [ERR] Target=2_90_20230620_369 (Linux 5.4.0-137-generic)
 [ERR] CPU=s390x (4 logical CPUs) (0x1f58f7000 RAM)
 [ERR] ----------- Stack Backtrace -----------
 [ERR] raise+0x86 (0x000003FF8CA93B16 [libpthread.so.0+0x13b16])
 [ERR] _ZN2TR4trapEv+0x52 (0x000003FF8A978A3A [libj9jit29.so+0x6f8a3a])
 [ERR]  (0x000003FF8A978A94 [libj9jit29.so+0x6f8a94])
 [ERR] _ZN26JITServerPersistentCHTable19commitModificationsERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x330 (0x000003FF8A5335D8 [libj9jit29.so+0x2b35d8])
 [ERR] _ZN26JITServerPersistentCHTable8doUpdateEP11TR_J9VMBaseRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES9_+0x366 (0x000003FF8A533AB6 [libj9jit29.so+0x2b3ab6])
 [ERR] _ZN2TR30CompilationInfoPerThreadRemote12processEntryER21TR_MethodToBeCompiledRN2J917J9SegmentProviderE+0x1450 (0x000003FF8A4893B0 [libj9jit29.so+0x2093b0])
 [ERR] _ZN2TR24CompilationInfoPerThread14processEntriesEv+0x3ee (0x000003FF8A40DCAE [libj9jit29.so+0x18dcae])
 [ERR] _ZN2TR24CompilationInfoPerThread3runEv+0xac (0x000003FF8A40E1B4 [libj9jit29.so+0x18e1b4])
 [ERR] _Z30protectedCompilationThreadProcP13J9PortLibraryPN2TR24CompilationInfoPerThreadE+0x94 (0x000003FF8A40E24C [libj9jit29.so+0x18e24c])
 [ERR] omrsig_protect+0x3e8 (0x000003FF8C032EA0 [libj9prt29.so+0x32ea0])
 [ERR] _Z21compilationThreadProcPv+0x196 (0x000003FF8A40E68E [libj9jit29.so+0x18e68e])
 [ERR] thread_wrapper+0xf6 (0x000003FF8BF88A5E [libj9thr29.so+0x8a5e])
 [ERR] start_thread+0xd6 (0x000003FF8CA87E66 [libpthread.so.0+0x7e66])
 [ERR]  (0x000003FF8C97CBE6 [libc.so.6+0xfcbe6])
 [ERR]  (0x0000000000000000 [<unknown>+0x0])
 [ERR] ---------------------------------------

@pshipton pshipton added the comp:jitserver Artifacts related to JIT-as-a-Service project label Jun 20, 2023
@pshipton pshipton changed the title cmdLineTester_criu_jitserverPostRestore_2_FAILED zLinux Test -Xnoaot Unhandled trap vmState=0x00000000 cmdLineTester_criu_jitserverPostRestore zLinux JITServerPersistentCHTable.cpp:172 Jun 20, 2023
@pshipton pshipton changed the title cmdLineTester_criu_jitserverPostRestore zLinux JITServerPersistentCHTable.cpp:172 cmdLineTester_criu_jitserverPostRestore zLinux assertion JITServerPersistentCHTable.cpp:172 Jun 20, 2023
@tajila
Copy link
Contributor

tajila commented Jun 21, 2023

@dsouzai ^^

@dsouzai
Copy link
Contributor

dsouzai commented Jun 21, 2023

Because of this message:

[2023-05-30T07:46:36.862Z]  [OUT] JITSERVER NO LONGER EXISTS

It looks like the crash is on the JITserver instance, and not on the restored JVM (a crash in the restored JVM wouldn't bring down the jitserver instance).

@mpirvu do you have any ideas as to what could be going on here?

@mpirvu
Copy link
Contributor

mpirvu commented Jun 21, 2023

I have seen that message when the client was not generating a new clientUID after restore. From the diagnostic data I cannot tell the UID of the client before the checkpoint and after the restore to determine that they are different.

@dsouzai
Copy link
Contributor

dsouzai commented Jun 22, 2023

I have seen that message when the client was not generating a new clientUID after restore.

Why would that cause the JITServer instance to go down? Shouldn't it just reject the client?

@mpirvu
Copy link
Contributor

mpirvu commented Jun 22, 2023

The server thinks it's the same client and that it encountered a bug. Rather than reject the client and hide the bug, we prefer to fatal assert.

@dsouzai
Copy link
Contributor

dsouzai commented Jun 22, 2023

Because this message

2023-05-30T07:46:36.862Z]  [ERR] JITServer is currently a technology preview. Its use is not yet supported.
[2023-05-30T07:46:36.862Z]  [ERR] 
[2023-05-30T07:46:36.862Z]  [ERR] JITServer is ready to accept incoming requests

is coming from the jitserver, if it did fatal assert because the client failed to generate a new UID, we should see that assert in the output. However, maybe what I'll do is update the tests to have better separation of the the various things that get cat'd so that we can tell where messages come from.

@pshipton
Copy link
Member

https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.functional_s390x_linux_Nightly_testList_0/509
cmdLineTester_criu_jitserverPostRestore_2 -Xjit:vlog=vlog

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk17_j9_sanity.functional_s390x_linux_Nightly_testList_0/509/functional_test_output.tar.gz

TESTING:
*** Starting test suite: J9 Criu Command-Line Post Restore JITServer Option Tests ***
Testing: Generate Verbose Log
Test start time: 2023/06/28 01:14:41 Coordinated Universal Time
Running command: bash /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_Nightly_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criuJitServerScript.sh /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_Nightly_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/bin " -Xjit:vlog=vlog " org.openj9.criu.OptionsFileTest "JitOptionsTest -XX:+UseJITServer -Xjit:verbose={compilePerformance},verbose={CheckpointRestore},verbose={JITServer},verbose={JITServerConns},vlog=vlog" 1 false true
Time spent starting: 76 milliseconds
Time spent executing: 35743 milliseconds
Test result: FAILED
Output from test:
 [OUT] start running script
 [OUT] export GLIBC_TUNABLES=glibc.cpu.hwcaps=-XSAVEC,-XSAVE,-AVX2,-ERMS,-AVX,-AVX_Fast_Unaligned_Load
 [OUT] export LD_BIND_NOT=on
 [OUT] Starting /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/bin/jitserver -XX:JITServerPort=50096
 [OUT] 1247254 ?        00:00:00 jitserver
 [OUT] JITSERVER EXISTS
 [OUT] Pre-checkpoint
 [OUT] Performing CRIUSupport.checkpointJVM(), current thread name: main, Wed Jun 28 01:14:45 UTC 2023, System.currentTimeMillis(): 1687914885074, System.nanoTime(): 1687914885071522333
 [OUT] Post-checkpoint
 [OUT] JITSERVER NO LONGER EXISTS
 [OUT] Terminating /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/bin/jitserver -XX:JITServerPort=50096
 [OUT] finished script
 [ERR] JITServer is currently a technology preview. Its use is not yet supported.
 [ERR] 
 [ERR] JITServer is ready to accept incoming requests
 [ERR] /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_Nightly_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criuJitServerScript.sh: line 91: 1247333 Killed                  $2/java -XX:+EnableCRIUSupport -XX:JITServerPort=$JITSERVER_PORT $3 -cp "$1/criu.jar" $4 $5 -XX:JITServerPort=$JITSERVER_PORT $6 > testOutput 2>&1
 [ERR] Assertion failed at /home/jenkins/workspace/Build_JDK17_s390x_linux_Nightly/openj9/runtime/compiler/env/JITServerPersistentCHTable.cpp:172: classInfo
 [ERR] 	subclass info cannot be null: ensure subclasses are loaded before superclass
 [ERR] #0: /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0xab141e) [0x3ff92ab141e]
 [ERR] #1: /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0xabf158) [0x3ff92abf158]
 [ERR] #2: /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x6f7260) [0x3ff926f7260]
 [ERR] #3: /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x6f8b3e) [0x3ff926f8b3e]
 [ERR] #4: /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x2b3608) [0x3ff922b3608]
 [ERR] #5: /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x2b3ae6) [0x3ff922b3ae6]
 [ERR] #6: /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x2093e0) [0x3ff922093e0]
 [ERR] #7: /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x18dcae) [0x3ff9218dcae]
 [ERR] #8: /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x18e1b4) [0x3ff9218e1b4]
 [ERR] #9: /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x18e24c) [0x3ff9218e24c]
 [ERR] #10: /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9prt29.so(+0x32ea0) [0x3ff93db2ea0]
 [ERR] #11: /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x18e68e) [0x3ff9218e68e]
 [ERR] #12: /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9thr29.so(+0x8a5e) [0x3ff93d08a5e]
 [ERR] #13: /lib/s390x-linux-gnu/libpthread.so.0(+0x7e66) [0x3ff94807e66]
 [ERR] #14: /lib/s390x-linux-gnu/libc.so.6(+0xfcbe6) [0x3ff946fcbe6]
 [ERR] #15: [(nil)]
 [ERR] 
 [ERR] Unhandled exception
 [ERR] Type=Unhandled trap vmState=0x00000000
 [ERR] J9Generic_Signal_Number=00000108 Signal_Number=00000005 Error_Value=00000000 Signal_Code=fffffffa
 [ERR] Handler1=000003FF93EC9400 Handler2=000003FF93DB1D48
 [ERR] gpr0=0000000000000005 gpr1=FFFFFFFF00000000 gpr2=0000000000000000 gpr3=000003FF78A7BE90
 [ERR] gpr4=0000000000000000 gpr5=0000000000000008 gpr6=000003FF78A7C0D8 gpr7=000000000000196F
 [ERR] gpr8=000003FF6401FCE8 gpr9=000003FEFA297B80 gpr10=0000000000000002 gpr11=000003FF78A7BE90
 [ERR] gpr12=000003FF92EDE000 gpr13=000003FF92C5E778 gpr14=000003FF926F8AEA gpr15=000003FF78A7BDF0
 [ERR] psw=000003FF94813B16 mask=0705000180000000 fpc=00080000 bea=000003FF9473EE0E
 [ERR] fpr0 000003ffec3fa240 (f: 3963593216.000000, d: 2.172760e-311)
 [ERR] fpr1 000003ff92d26828 (f: 2463262720.000000, d: 2.172019e-311)
 [ERR] fpr2 000003ffec3fc569 (f: 3963602176.000000, d: 2.172760e-311)
 [ERR] fpr3 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] fpr4 000003ff78a78b80 (f: 2024246144.000000, d: 2.171802e-311)
 [ERR] fpr5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] fpr6 000003ff78a79268 (f: 2024247936.000000, d: 2.171802e-311)
 [ERR] fpr7 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] fpr8 000003fefa280010 (f: 4196925440.000000, d: 2.170753e-311)
 [ERR] fpr9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] fpr10 000003ff7897d000 (f: 2023215104.000000, d: 2.171801e-311)
 [ERR] fpr11 000003ffeb7f8ee0 (f: 3951005440.000000, d: 2.172754e-311)
 [ERR] fpr12 000002aa0ee44960 (f: 249842016.000000, d: 1.447325e-311)
 [ERR] fpr13 000003ff984d56d8 (f: 2555205376.000000, d: 2.172064e-311)
 [ERR] fpr14 0000000000000440 (f: 1088.000000, d: 5.375434e-321)
 [ERR] fpr15 000003ff98780928 (f: 2558003456.000000, d: 2.172066e-311)
 [ERR] Module=/lib/s390x-linux-gnu/libpthread.so.0
 [ERR] Module_base_address=000003FF94800000 Symbol=raise
 [ERR] Symbol_address=000003FF94813A90
 [ERR] Target=2_90_20230627_518 (Linux 5.4.0-137-generic)
 [ERR] CPU=s390x (4 logical CPUs) (0x1f58f7000 RAM)
 [ERR] ----------- Stack Backtrace -----------
 [ERR] raise+0x86 (0x000003FF94813B16 [libpthread.so.0+0x13b16])
 [ERR] _ZN2TR4trapEv+0x52 (0x000003FF926F8AEA [libj9jit29.so+0x6f8aea])
 [ERR]  (0x000003FF926F8B44 [libj9jit29.so+0x6f8b44])
 [ERR] _ZN26JITServerPersistentCHTable19commitModificationsERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x330 (0x000003FF922B3608 [libj9jit29.so+0x2b3608])
 [ERR] _ZN26JITServerPersistentCHTable8doUpdateEP11TR_J9VMBaseRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES9_+0x366 (0x000003FF922B3AE6 [libj9jit29.so+0x2b3ae6])
 [ERR] _ZN2TR30CompilationInfoPerThreadRemote12processEntryER21TR_MethodToBeCompiledRN2J917J9SegmentProviderE+0x1450 (0x000003FF922093E0 [libj9jit29.so+0x2093e0])
 [ERR] _ZN2TR24CompilationInfoPerThread14processEntriesEv+0x3ee (0x000003FF9218DCAE [libj9jit29.so+0x18dcae])
 [ERR] _ZN2TR24CompilationInfoPerThread3runEv+0xac (0x000003FF9218E1B4 [libj9jit29.so+0x18e1b4])
 [ERR] _Z30protectedCompilationThreadProcP13J9PortLibraryPN2TR24CompilationInfoPerThreadE+0x94 (0x000003FF9218E24C [libj9jit29.so+0x18e24c])
 [ERR] omrsig_protect+0x3e8 (0x000003FF93DB2EA0 [libj9prt29.so+0x32ea0])
 [ERR] _Z21compilationThreadProcPv+0x196 (0x000003FF9218E68E [libj9jit29.so+0x18e68e])
 [ERR] thread_wrapper+0xf6 (0x000003FF93D08A5E [libj9thr29.so+0x8a5e])
 [ERR] start_thread+0xd6 (0x000003FF94807E66 [libpthread.so.0+0x7e66])
 [ERR]  (0x000003FF946FCBE6 [libc.so.6+0xfcbe6])
 [ERR]  (0x0000000000000000 [<unknown>+0x0])

@JasonFengJ9
Copy link
Member Author

JDK21 s390x_linux(rhel7s390x-svl-rt4-1)

variation: -Xjit
JVM_OPTIONS:  -Xjit 

TESTING:
*** Starting test suite: J9 Criu Command-Line Post Restore JITServer Option Tests ***
Testing: Generate Verbose Log
Test start time: 2023/09/03 06:24:50 Pacific Standard Time
Running command: bash /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_s390x_linux_Personal_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criuJitServerScript.sh /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_s390x_linux_Personal_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_s390x_linux_Personal_testList_1/openjdkbinary/j2sdk-image/bin " -Xjit " org.openj9.criu.OptionsFileTest "JitOptionsTest -XX:+UseJITServer -Xjit:verbose={compilePerformance},verbose={CheckpointRestore},verbose={JITServer},verbose={JITServerConns},vlog=vlog" 1 false true
Time spent starting: 113 milliseconds
Time spent executing: 10294 milliseconds
Test result: FAILED
Output from test:
 [OUT] start running script
 [OUT] export GLIBC_TUNABLES=glibc.cpu.hwcaps=-XSAVEC,-XSAVE,-AVX2,-ERMS,-AVX,-AVX_Fast_Unaligned_Load
 [OUT] export LD_BIND_NOT=on
 [OUT] Starting /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_s390x_linux_Personal_testList_1/openjdkbinary/j2sdk-image/bin/jitserver -XX:JITServerPort=45559
 [OUT]  6393 ?        00:00:00 jitserver
 [OUT] JITSERVER EXISTS
 [OUT] Pre-checkpoint
 [OUT] Performing CRIUSupport.checkpointJVM(), current thread name: main, Sun Sep 03 06:24:53 PDT 2023, System.currentTimeMillis(): 1693747493699, System.nanoTime(): 1693747378908550128
 [OUT] Post-checkpoint
 [OUT] JITSERVER NO LONGER EXISTS
 [OUT] Terminating /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_s390x_linux_Personal_testList_1/openjdkbinary/j2sdk-image/bin/jitserver -XX:JITServerPort=45559
 [OUT] finished script
 [ERR] JITServer is currently a technology preview. Its use is not yet supported.
 [ERR] 
 [ERR] JITServer is ready to accept incoming requests
 [ERR] /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_s390x_linux_Personal_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criuJitServerScript.sh: line 91:  6472 Killed                  $2/java -XX:+EnableCRIUSupport -XX:JITServerPort=$JITSERVER_PORT $3 -cp "$1/criu.jar" $4 $5 -XX:JITServerPort=$JITSERVER_PORT $6 > testOutput 2>&1
 [ERR] Assertion failed at /home/jenkins/workspace/Build_JDK21_s390x_linux_Personal/openj9/runtime/compiler/env/JITServerPersistentCHTable.cpp:172: classInfo
 [ERR] 	subclass info cannot be null: ensure subclasses are loaded before superclass
 [ERR] #0: /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_s390x_linux_Personal_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0xaba7ae) [0x3ff8363a7ae]
 [ERR] #1: /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_s390x_linux_Personal_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0xac85a0) [0x3ff836485a0]
 [ERR] #2: /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_s390x_linux_Personal_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x6f78a4) [0x3ff832778a4]
 [ERR] #3: /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_s390x_linux_Personal_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x6f9196) [0x3ff83279196]
 [ERR] #4: /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_s390x_linux_Personal_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x2b06b8) [0x3ff82e306b8]
 [ERR] #5: /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_s390x_linux_Personal_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x2b0bb4) [0x3ff82e30bb4]
 [ERR] #6: /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_s390x_linux_Personal_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x20b7ec) [0x3ff82d8b7ec]
 [ERR] #7: /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_s390x_linux_Personal_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x18fbc6) [0x3ff82d0fbc6]
 [ERR] #8: /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_s390x_linux_Personal_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x1900cc) [0x3ff82d100cc]
 [ERR] #9: /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_s390x_linux_Personal_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x190164) [0x3ff82d10164]
 [ERR] #10: /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_s390x_linux_Personal_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9prt29.so(+0x32b28) [0x3ff84832b28]
 [ERR] #11: /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_s390x_linux_Personal_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x1905a6) [0x3ff82d105a6]
 [ERR] #12: /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_s390x_linux_Personal_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9thr29.so(+0x8efe) [0x3ff84788efe]
 [ERR] #13: /lib64/libpthread.so.0(+0x9ad8) [0x3ff85289ad8]
 [ERR] #14: /lib64/libc.so.6(+0x10baee) [0x3ff8518baee]
 [ERR] #15: [(nil)]
 [ERR] 
 [ERR] Unhandled exception
 [ERR] Type=Unhandled trap vmState=0x00000000
 [ERR] J9Generic_Signal_Number=00000108 Signal_Number=00000005 Error_Value=00000000 Signal_Code=fffffffa
 [ERR] Handler1=000003FF84949878 Handler2=000003FF848319D8
 [ERR] gpr0=000003FF689FBE50 gpr1=FFFFFFFF00000000 gpr2=0000000000000000 gpr3=000003FF689FBE50
 [ERR] gpr4=0000000000000000 gpr5=0000000000000008 gpr6=000003FF689FC098 gpr7=000000000000367B
 [ERR] gpr8=000003FF54010948 gpr9=000003FF63E8CD40 gpr10=0000000000000002 gpr11=000003FF83A86020
 [ERR] gpr12=000003FF83A4AF48 gpr13=000003FF852991A0 gpr14=000003FF83279142 gpr15=000003FF689FBDB0
 [ERR] psw=000003FF85295E5E mask=0705100180000000 fpc=00080000 bea=000003FF850FABC2
 [ERR] fpr0 0000000000000005 (f: 5.000000, d: 2.470328e-323)
 [ERR] fpr1 000003ff54000e08 (f: 1409289728.000000, d: 2.171498e-311)
 [ERR] fpr2 000003fff9f7dc5a (f: 4193770496.000000, d: 2.172874e-311)
 [ERR] fpr3 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] fpr4 000003ff689f8b10 (f: 1755286272.000000, d: 2.171669e-311)
 [ERR] fpr5 3ff0000000000000 (f: 0.000000, d: 1.000000e+00)
 [ERR] fpr6 0000000000000007 (f: 7.000000, d: 3.458460e-323)
 [ERR] fpr7 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] fpr8 000003ff63e80010 (f: 1676148736.000000, d: 2.171630e-311)
 [ERR] fpr9 000003ffc2b77978 (f: 3266803968.000000, d: 2.172416e-311)
 [ERR] fpr10 000003ff688fd000 (f: 1754255360.000000, d: 2.171668e-311)
 [ERR] fpr11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] fpr12 000002aa2ccb3470 (f: 751514752.000000, d: 1.447572e-311)
 [ERR] fpr13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] fpr14 000002aa2ccaeea0 (f: 751496832.000000, d: 1.447572e-311)
 [ERR] fpr15 000003ff8c7cba28 (f: 2356984320.000000, d: 2.171966e-311)
 [ERR] Module=/lib64/libpthread.so.0
 [ERR] Module_base_address=000003FF85280000 Symbol=raise
 [ERR] Symbol_address=000003FF85295DD0
 [ERR] Target=2_90_20230903_61 (Linux 5.14.21-150400.24.69-default)
 [ERR] CPU=s390x (4 logical CPUs) (0x1eaf17000 RAM)
 [ERR] ----------- Stack Backtrace -----------
 [ERR] raise+0x8e (0x000003FF85295E5E [libpthread.so.0+0x15e5e])
 [ERR] _ZN2TR4trapEv+0x52 (0x000003FF83279142 [libj9jit29.so+0x6f9142])
 [ERR]  (0x000003FF8327919C [libj9jit29.so+0x6f919c])
 [ERR] _ZN26JITServerPersistentCHTable19commitModificationsERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x330 (0x000003FF82E306B8 [libj9jit29.so+0x2b06b8])
 [ERR] _ZN26JITServerPersistentCHTable8doUpdateEP11TR_J9VMBaseRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES9_+0x384 (0x000003FF82E30BB4 [libj9jit29.so+0x2b0bb4])
 [ERR] _ZN2TR30CompilationInfoPerThreadRemote12processEntryER21TR_MethodToBeCompiledRN2J917J9SegmentProviderE+0x13cc (0x000003FF82D8B7EC [libj9jit29.so+0x20b7ec])
 [ERR] _ZN2TR24CompilationInfoPerThread14processEntriesEv+0x3ee (0x000003FF82D0FBC6 [libj9jit29.so+0x18fbc6])
 [ERR] _ZN2TR24CompilationInfoPerThread3runEv+0xac (0x000003FF82D100CC [libj9jit29.so+0x1900cc])
 [ERR] _Z30protectedCompilationThreadProcP13J9PortLibraryPN2TR24CompilationInfoPerThreadE+0x94 (0x000003FF82D10164 [libj9jit29.so+0x190164])
 [ERR] omrsig_protect+0x3d8 (0x000003FF84832B28 [libj9prt29.so+0x32b28])
 [ERR] _Z21compilationThreadProcPv+0x196 (0x000003FF82D105A6 [libj9jit29.so+0x1905a6])
 [ERR] thread_wrapper+0xf6 (0x000003FF84788EFE [libj9thr29.so+0x8efe])
 [ERR] start_thread+0xd8 (0x000003FF85289AD8 [libpthread.so.0+0x9ad8])
 [ERR]  (0x000003FF8518BAEE [libc.so.6+0x10baee])
 [ERR]  (0x0000000000000000 [<unknown>+0x0])
 [ERR] ---------------------------------------
 [ERR] JVMDUMP039I Processing dump event "gpf", detail "" at 2023/09/03 06:24:56 - please wait.

>> Success condition was found: [Output match: Killed]
>> Required condition was found: [Output match: Pre-checkpoint]
>> Success condition was found: [Output match: Post-checkpoint]
>> Failure condition was not found: [Output match: CRIU is not enabled]
>> Failure condition was not found: [Output match: Operation not permitted]
>> Success condition was not found: [Output match: Thread pid mismatch]
>> Success condition was not found: [Output match: do not match expected]
>> Success condition was not found: [Output match: Unable to create a thread:]
>> Success condition was found: [Output match: JITSERVER EXISTS]
>> Success condition was not found: [Output match: JITSERVER STILL EXISTS]
>> Failure condition was not found: [Output match: JITSERVER DOES NOT EXIST]
>> Failure condition was found: [Output match: JITSERVER NO LONGER EXISTS]
>> Failure condition was not found: [Output match: User requested Java dump using]

---TEST RESULTS---
Number of PASSED tests: 4 out of 5
Number of FAILED tests: 1 out of 5

---SUMMARY OF FAILED TESTS---
Generate Verbose Log
-----------------------------

-----------------------------------
cmdLineTester_criu_jitserverPostRestore_0_FAILED

@pshipton
Copy link
Member

pshipton commented Mar 7, 2024

@pshipton
Copy link
Member

pshipton commented Mar 8, 2024

@pshipton
Copy link
Member

https://openj9-jenkins.osuosl.org/job/Test_openjdk22_j9_sanity.functional_s390x_linux_Release_testList_1/4
cmdLineTester_criu_jitserverPostRestore_2

Testing: Test -Xnoaot
Test start time: 2024/03/17 15:11:30 Eastern Standard Time
Running command: bash /home/jenkins/workspace/Test_openjdk22_j9_sanity.functional_s390x_linux_Release_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criuJitServerScript.sh /home/jenkins/workspace/Test_openjdk22_j9_sanity.functional_s390x_linux_Release_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu /home/jenkins/workspace/Test_openjdk22_j9_sanity.functional_s390x_linux_Release_testList_1/jdkbinary/j2sdk-image/bin " -Xjit:vlog=vlog " org.openj9.criu.OptionsFileTest "JitOptionsTest -XX:+UseJITServer -Xnoaot" 1 false false
...
 [ERR] /home/jenkins/workspace/Test_openjdk22_j9_sanity.functional_s390x_linux_Release_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/jitserverconfig.sh: line 30: lsof: command not found
 [ERR] /home/jenkins/workspace/Test_openjdk22_j9_sanity.functional_s390x_linux_Release_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/jitserverconfig.sh: line 30: lsof: command not found
 [ERR] JITServer is currently a technology preview. Its use is not yet supported.
 [ERR] 
 [ERR] JITServer is ready to accept incoming requests
 [ERR] /home/jenkins/workspace/Test_openjdk22_j9_sanity.functional_s390x_linux_Release_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criuJitServerScript.sh: line 114: 10893 Killed                  $TEST_JDK_BIN/java -XX:+EnableCRIUSupport -XX:JITServerPort=$JITSERVER_PORT $JVM_OPTIONS -cp "$TEST_ROOT/criu.jar" $MAINCLASS $APP_ARGS -XX:JITServerPort=$JITSERVER_PORT $NUM_CHECKPOINT > testOutput 2>&1
 [ERR] Assertion failed at /home/jenkins/workspace/Build_JDK22_s390x_linux_Release/openj9/runtime/compiler/env/JITServerPersistentCHTable.cpp:172: classInfo
 [ERR] 	subclass info cannot be null: ensure subclasses are loaded before superclass

@JasonFengJ9
Copy link
Member Author

JDK17 s390x_linux(ubu22s390x-svl-rt8-1)

[2024-04-02T01:12:49.553Z] variation: -Xjit:vlog=vlog
[2024-04-02T01:12:49.553Z] JVM_OPTIONS:  -Xjit:vlog=vlog 

[2024-04-02T01:13:04.870Z] Testing: Test -Xnoaot
[2024-04-02T01:13:04.870Z] Test start time: 2024/04/02 01:13:03 Coordinated Universal Time
[2024-04-02T01:13:04.870Z] Running command: bash /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criuJitServerScript.sh /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_testList_0/jdkbinary/j2sdk-image/bin " -Xjit:vlog=vlog " org.openj9.criu.OptionsFileTest "JitOptionsTest -XX:+UseJITServer -Xnoaot" 1 false false
[2024-04-02T01:13:04.870Z] Time spent starting: 3 milliseconds
[2024-04-02T01:13:14.819Z] Time spent executing: 10596 milliseconds
[2024-04-02T01:13:14.819Z] Test result: FAILED
[2024-04-02T01:13:14.819Z] Output from test:
[2024-04-02T01:13:14.819Z]  [OUT] start running script
[2024-04-02T01:13:14.819Z]  [OUT] export GLIBC_TUNABLES=glibc.cpu.hwcaps=-XSAVEC,-XSAVE,-AVX2,-ERMS,-AVX,-AVX_Fast_Unaligned_Load
[2024-04-02T01:13:14.819Z]  [OUT] export LD_BIND_NOT=on
[2024-04-02T01:13:14.819Z]  [OUT] Starting /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_testList_0/jdkbinary/j2sdk-image/bin/jitserver -XX:JITServerPort=44959 -XX:JITServerHealthProbePort=47076 
[2024-04-02T01:13:14.819Z]  [OUT] 3735070 ?        00:00:00 jitserver
[2024-04-02T01:13:14.819Z]  [OUT] JITSERVER EXISTS
[2024-04-02T01:13:14.819Z]  [OUT] Pre-checkpoint
[2024-04-02T01:13:14.819Z]  [OUT] main: Tue Apr 02 01:13:06 UTC 2024, Performing CRIUSupport.checkpointJVM(), System.currentTimeMillis(): 1712020386429, System.nanoTime(): 1712020275214754523
[2024-04-02T01:13:14.819Z]  [OUT] JVMJITM043W AOT load and compilation disabled post restore.
[2024-04-02T01:13:14.819Z]  [OUT] JVMJITM043W AOT load and compilation disabled post restore.
[2024-04-02T01:13:14.819Z]  [OUT] Post-checkpoint
[2024-04-02T01:13:14.819Z]  [OUT] Removed test output files
[2024-04-02T01:13:14.819Z]  [OUT] JITSERVER NO LONGER EXISTS
[2024-04-02T01:13:14.819Z]  [OUT] Terminating /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_testList_0/jdkbinary/j2sdk-image/bin/jitserver -XX:JITServerPort=44959 -XX:JITServerHealthProbePort=47076 
[2024-04-02T01:13:14.819Z]  [OUT] finished script
[2024-04-02T01:13:14.819Z]  [ERR] JITServer is currently a technology preview. Its use is not yet supported.
[2024-04-02T01:13:14.819Z]  [ERR] 
[2024-04-02T01:13:14.819Z]  [ERR] JITServer is ready to accept incoming requests
[2024-04-02T01:13:14.819Z]  [ERR] /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_s390x_linux_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criuJitServerScript.sh: line 114: 3735148 Killed                  $TEST_JDK_BIN/java -XX:+EnableCRIUSupport -XX:JITServerPort=$JITSERVER_PORT $JVM_OPTIONS -cp "$TEST_ROOT/criu.jar" $MAINCLASS $APP_ARGS -XX:JITServerPort=$JITSERVER_PORT $NUM_CHECKPOINT > testOutput 2>&1
[2024-04-02T01:13:14.819Z]  [ERR] Assertion failed at /home/jenkins/workspace/build-scripts/jobs/jdk17u/jdk17u-linux-s390x-openj9-IBM/workspace/build/src/openj9/runtime/compiler/env/JITServerPersistentCHTable.cpp:172: classInfo
[2024-04-02T01:13:14.819Z]  [ERR] 	subclass info cannot be null: ensure subclasses are loaded before superclass

[2024-04-02T01:15:37.551Z] ---TEST RESULTS---
[2024-04-02T01:15:37.551Z] Number of PASSED tests: 10 out of 11
[2024-04-02T01:15:37.551Z] Number of FAILED tests: 1 out of 11
[2024-04-02T01:15:37.551Z] 
[2024-04-02T01:15:37.551Z] ---SUMMARY OF FAILED TESTS---
[2024-04-02T01:15:37.551Z] Test -Xnoaot
[2024-04-02T01:15:37.551Z] -----------------------------
[2024-04-02T01:15:37.551Z] 
[2024-04-02T01:15:37.551Z] -----------------------------------
[2024-04-02T01:15:37.551Z] cmdLineTester_criu_jitserverPostRestore_2_FAILED

@pshipton
Copy link
Member

@vij-singh
Copy link

@dsouzai Is this something we're looking at for the 0.48 release?

@dsouzai
Copy link
Contributor

dsouzai commented Sep 19, 2024

No; I spent some time a few months ago looking at this but couldn't make much progress. It's fairly intermittent and seems to only happen with JITServer + CRIU.

@pshipton
Copy link
Member

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_sanity.functional_s390x_linux_Release_testList_0/122
cmdLineTester_criu_jitserverPostRestore_3

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk11_j9_sanity.functional_s390x_linux_Release_testList_0/122/functional_test_output.tar.gz

01:08:55  Testing: Test -Xnoaot
01:08:55  Test start time: 2024/09/29 01:08:55 Eastern Standard Time
01:08:55  Running command: bash /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_Release_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criuJitServerScript.sh /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_Release_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_Release_testList_0/jdkbinary/j2sdk-image/bin " -XX:+JVMPortableRestoreMode " org.openj9.criu.OptionsFileTest "JitOptionsTest -XX:+UseJITServer -Xnoaot" 1 false false
01:08:55  Time spent starting: 2 milliseconds
01:09:11  Time spent executing: 14139 milliseconds
01:09:11  Test result: FAILED
01:09:11  Output from test:
01:09:11   [OUT] start running script
01:09:11   [OUT] export GLIBC_TUNABLES=glibc.cpu.hwcaps=-XSAVEC,-XSAVE,-AVX2,-ERMS,-AVX,-AVX_Fast_Unaligned_Load
01:09:11   [OUT] export LD_BIND_NOT=on
01:09:11   [OUT] Starting /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_Release_testList_0/jdkbinary/j2sdk-image/bin/jitserver -XX:JITServerPort=41861 -XX:JITServerHealthProbePort=40028 
01:09:11   [OUT] 1608135 ?        00:00:00 jitserver
01:09:11   [OUT] JITSERVER EXISTS
01:09:11   [OUT] Pre-checkpoint
01:09:11   [OUT] main: Sun Sep 29 01:08:57 EDT 2024, Performing CRIUSupport.checkpointJVM(), System.currentTimeMillis(): 1727586537385, System.nanoTime(): 1727586537382660007
01:09:11   [OUT] JVMJITM043W AOT load and compilation disabled post restore.
01:09:11   [OUT] Post-checkpoint
01:09:11   [OUT] Removed test output files
01:09:11   [OUT] JITSERVER NO LONGER EXISTS
01:09:11   [OUT] Terminating /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_Release_testList_0/jdkbinary/j2sdk-image/bin/jitserver -XX:JITServerPort=41861 -XX:JITServerHealthProbePort=40028 
01:09:11   [OUT] finished script
01:09:11   [ERR] /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_Release_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/jitserverconfig.sh: line 30: lsof: command not found
01:09:11   [ERR] /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_Release_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/jitserverconfig.sh: line 30: lsof: command not found
01:09:11   [ERR] JITServer is currently a technology preview. Its use is not yet supported.
01:09:11   [ERR] 
01:09:11   [ERR] JITServer is ready to accept incoming requests
01:09:11   [ERR] /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_Release_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criuJitServerScript.sh: line 114: 1608215 Killed                  $TEST_JDK_BIN/java -XX:+EnableCRIUSupport -XX:JITServerPort=$JITSERVER_PORT $JVM_OPTIONS -cp "$TEST_ROOT/criu.jar" $MAINCLASS $APP_ARGS -XX:JITServerPort=$JITSERVER_PORT $NUM_CHECKPOINT > testOutput 2>&1
01:09:11   [ERR] Assertion failed at /home/jenkins/workspace/Build_JDK11_s390x_linux_Release/openj9/runtime/compiler/env/JITServerPersistentCHTable.cpp:172: classInfo
01:09:11   [ERR] 	subclass info cannot be null: ensure subclasses are loaded before superclass
01:09:11   [ERR] #0: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_Release_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0xada82e) [0x3ffa345a82e]
01:09:11   [ERR] #1: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_Release_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0xae87d0) [0x3ffa34687d0]
01:09:11   [ERR] #2: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_Release_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x710dcc) [0x3ffa3090dcc]
01:09:11   [ERR] #3: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_Release_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x7126be) [0x3ffa30926be]
01:09:11   [ERR] #4: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_Release_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x2bd470) [0x3ffa2c3d470]
01:09:11   [ERR] #5: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_Release_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x2bd96c) [0x3ffa2c3d96c]
01:09:11   [ERR] #6: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_Release_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x210c92) [0x3ffa2b90c92]
01:09:11   [ERR] #7: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_Release_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x193432) [0x3ffa2b13432]
01:09:11   [ERR] #8: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_Release_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x193924) [0x3ffa2b13924]
01:09:11   [ERR] #9: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_Release_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x1939bc) [0x3ffa2b139bc]
01:09:11   [ERR] #10: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_Release_testList_0/jdkbinary/j2sdk-image/lib/default/libj9prt29.so(+0x32bb0) [0x3ffa46b2bb0]
01:09:11   [ERR] #11: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_Release_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x193dfe) [0x3ffa2b13dfe]
01:09:11   [ERR] #12: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_Release_testList_0/jdkbinary/j2sdk-image/lib/default/libj9thr29.so(+0x8a46) [0x3ffa4608a46]
01:09:11   [ERR] #13: /lib64/libpthread.so.0(+0x80de) [0x3ffa50880de]
01:09:11   [ERR] #14: /lib64/libc.so.6(+0x28b42) [0x3ffa4ea8b42]
01:09:11   [ERR] #15: [(nil)]
01:09:11   [ERR] 
01:09:11   [ERR] Unhandled exception
01:09:11   [ERR] Type=Unhandled trap vmState=0x00000000
01:09:11   [ERR] J9Generic_Signal_Number=00000108 Signal_Number=00000005 Error_Value=00000000 Signal_Code=fffffffa
01:09:11   [ERR] Handler1=000003FFA47C9618 Handler2=000003FFA46B1A60
01:09:11   [ERR] gpr0=000003FF8977DE48 gpr1=FFFFFFFF00000000 gpr2=0000000000000000 gpr3=000003FF8977DE48
01:09:11   [ERR] gpr4=0000000000000000 gpr5=0000000000000008 gpr6=000003FF8977E090 gpr7=000003FF84012060
01:09:11   [ERR] gpr8=000003FF84019480 gpr9=000003FF0A494900 gpr10=0000000000000002 gpr11=000003FFA38C1010
01:09:11   [ERR] gpr12=000003FFA3885000 gpr13=000003FFA50961F8 gpr14=000003FFA309266A gpr15=000003FF8977DDA8
01:09:11   [ERR] psw=000003FFA5093244 mask=0705000180000000 fpc=00080000 bea=000003FFA4FC6AF8
01:09:11   [ERR] fpr0=0000000000000005 (f: 5.000000, d: 2.470328e-323)
01:09:11   [ERR] fpr1=0000000000000001 (f: 1.000000, d: 4.940656e-324)
01:09:11   [ERR] fpr2=000003ffa501eca8 (f: 2768366848.000000, d: 2.172169e-311)
01:09:11   [ERR] fpr3=0000000000000007 (f: 7.000000, d: 3.458460e-323)
01:09:11   [ERR] fpr4=000003ff8977b220 (f: 2306322944.000000, d: 2.171941e-311)
01:09:11   [ERR] fpr5=000003ff84000de8 (f: 2214596096.000000, d: 2.171896e-311)
01:09:11   [ERR] fpr6=000003ff00000001 (f: 1.000000, d: 2.170802e-311)
01:09:11   [ERR] fpr7=d1d6b8188c58eefd (f: 2354638592.000000, d: -1.765429e+86)
01:09:11   [ERR] fpr8=000003ff0a480010 (f: 172490768.000000, d: 2.170887e-311)
01:09:11   [ERR] fpr9=0000000000000000 (f: 0.000000, d: 0.000000e+00)
01:09:11   [ERR] fpr10=00000000a0e15260 (f: 2699121152.000000, d: 1.333543e-314)
01:09:11   [ERR] fpr11=000003ff8b7f1f28 (f: 2340364032.000000, d: 2.171958e-311)
01:09:11   [ERR] fpr12=000003ff8977f910 (f: 2306341120.000000, d: 2.171941e-311)
01:09:11   [ERR] fpr13=0000000000000001 (f: 1.000000, d: 4.940656e-324)
01:09:11   [ERR] fpr14=0000000000101000 (f: 1052672.000000, d: 5.200891e-318)
01:09:11   [ERR] fpr15=000003ff84590968 (f: 2220427520.000000, d: 2.171899e-311)
01:09:11   [ERR] Module=/lib64/libpthread.so.0
01:09:11   [ERR] Module_base_address=000003FFA5080000 Symbol=raise
01:09:11   [ERR] Symbol_address=000003FFA50931B8
01:09:11   [ERR] Target=2_90_20240928_144 (Linux 4.18.0-513.24.1.el8_9.s390x)
01:09:11   [ERR] CPU=s390x (4 logical CPUs) (0x1d5ba6000 RAM)
01:09:11   [ERR] ----------- Stack Backtrace -----------
01:09:11   [ERR] raise+0x8c (0x000003FFA5093244 [libpthread.so.0+0x13244])
01:09:11   [ERR] _ZN2TR4trapEv+0x52 (0x000003FFA309266A [libj9jit29.so+0x71266a])
01:09:11   [ERR]  (0x000003FFA30926C4 [libj9jit29.so+0x7126c4])
01:09:11   [ERR] _ZN26JITServerPersistentCHTable19commitModificationsERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x330 (0x000003FFA2C3D470 [libj9jit29.so+0x2bd470])
01:09:11   [ERR] _ZN26JITServerPersistentCHTable8doUpdateEP11TR_J9VMBaseRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES9_+0x384 (0x000003FFA2C3D96C [libj9jit29.so+0x2bd96c])
01:09:11   [ERR] _ZN2TR30CompilationInfoPerThreadRemote12processEntryER21TR_MethodToBeCompiledRN2J917J9SegmentProviderE+0xd82 (0x000003FFA2B90C92 [libj9jit29.so+0x210c92])
01:09:11   [ERR] _ZN2TR24CompilationInfoPerThread14processEntriesEv+0x402 (0x000003FFA2B13432 [libj9jit29.so+0x193432])
01:09:11   [ERR] _ZN2TR24CompilationInfoPerThread3runEv+0xac (0x000003FFA2B13924 [libj9jit29.so+0x193924])
01:09:11   [ERR] _Z30protectedCompilationThreadProcP13J9PortLibraryPN2TR24CompilationInfoPerThreadE+0x94 (0x000003FFA2B139BC [libj9jit29.so+0x1939bc])
01:09:11   [ERR] omrsig_protect+0x3d8 (0x000003FFA46B2BB0 [libj9prt29.so+0x32bb0])
01:09:11   [ERR] _Z21compilationThreadProcPv+0x196 (0x000003FFA2B13DFE [libj9jit29.so+0x193dfe])
01:09:11   [ERR] thread_wrapper+0xf6 (0x000003FFA4608A46 [libj9thr29.so+0x8a46])
01:09:11   [ERR] start_thread+0xd6 (0x000003FFA50880DE [libpthread.so.0+0x80de])
01:09:11   [ERR]  (0x000003FFA4EA8B42 [libc.so.6+0x28b42])
01:09:11   [ERR]  (0x0000000000000000 [<unknown>+0x0])
01:09:11   [ERR] ---------------------------------------

@pshipton
Copy link
Member

https://openj9-jenkins.osuosl.org/job/Test_openjdk23_j9_sanity.functional_ppc64le_linux_Nightly_testList_0/41/
cmdLineTester_criu_jitserverAcrossCheckpoint_0 - ubu24-ppc64le-4

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk23_j9_sanity.functional_ppc64le_linux_Nightly_testList_0/41/functional_test_output.tar.gz

22:53:53  Testing: Enable JITServer specified Pre-Checkpoint but not explicitly enabled Post-Restore
22:53:53  Test start time: 2024/11/23 03:53:53 Coordinated Universal Time
22:53:53  Running command: bash /home/jenkins/workspace/Test_openjdk23_j9_sanity.functional_ppc64le_linux_Nightly_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criuJitServerScript.sh /home/jenkins/workspace/Test_openjdk23_j9_sanity.functional_ppc64le_linux_Nightly_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu /home/jenkins/workspace/Test_openjdk23_j9_sanity.functional_ppc64le_linux_Nightly_testList_0/jdkbinary/j2sdk-image/bin "  -XX:+UseJITServer -Xjit:verbose={compilePerformance},verbose={JITServer},verbose={JITServerConns},vlog=preCheckpointVlog" org.openj9.criu.OptionsFileTest "JitOptionsTest -Xjit:verbose={compilePerformance},verbose={CheckpointRestore},verbose={JITServer},verbose={JITServerConns},vlog=postRestoreVlog" 1 false true
22:53:53  Time spent starting: 2 milliseconds
22:54:03  Time spent executing: 9618 milliseconds
22:54:03  Test result: FAILED
22:54:03  Output from test:
22:54:03   [OUT] start running script
22:54:03   [OUT] export GLIBC_TUNABLES=glibc.cpu.hwcaps=-XSAVEC,-XSAVE,-AVX2,-ERMS,-AVX,-AVX_Fast_Unaligned_Load
22:54:03   [OUT] export LD_BIND_NOT=on
22:54:03   [OUT] Starting /home/jenkins/workspace/Test_openjdk23_j9_sanity.functional_ppc64le_linux_Nightly_testList_0/jdkbinary/j2sdk-image/bin/jitserver -XX:JITServerPort=57426 -XX:JITServerHealthProbePort=39121 
22:54:03   [OUT] 2262857 ?        00:00:00 jitserver
22:54:03   [OUT] JITSERVER EXISTS
22:54:03   [OUT] Pre-checkpoint
22:54:03   [OUT] main: Sat Nov 23 03:53:55 UTC 2024, Performing CRIUSupport.checkpointJVM(), System.currentTimeMillis(): 1732334035877, System.nanoTime(): 5698589620837107
22:54:03   [OUT] Post-checkpoint
22:54:03   [OUT] JITSERVER NO LONGER EXISTS
22:54:03   [OUT] Terminating /home/jenkins/workspace/Test_openjdk23_j9_sanity.functional_ppc64le_linux_Nightly_testList_0/jdkbinary/j2sdk-image/bin/jitserver -XX:JITServerPort=57426 -XX:JITServerHealthProbePort=39121 
22:54:03   [OUT] finished script
22:54:03   [ERR] 
22:54:03   [ERR] JITServer is ready to accept incoming requests
22:54:03   [ERR] /home/jenkins/workspace/Test_openjdk23_j9_sanity.functional_ppc64le_linux_Nightly_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criuJitServerScript.sh: line 114: 2262939 Killed                  $TEST_JDK_BIN/java -XX:+EnableCRIUSupport -XX:JITServerPort=$JITSERVER_PORT $JVM_OPTIONS -cp "$TEST_ROOT/criu.jar" $MAINCLASS $APP_ARGS -XX:JITServerPort=$JITSERVER_PORT $NUM_CHECKPOINT > testOutput 2>&1
22:54:03   [ERR] Assertion failed at /home/jenkins/workspace/Build_JDK23_ppc64le_linux_Nightly/openj9/runtime/compiler/env/JITServerPersistentCHTable.cpp:172: classInfo
22:54:03   [ERR] 	subclass info cannot be null: ensure subclasses are loaded before superclass
22:54:03   [ERR] #0: /home/jenkins/workspace/Test_openjdk23_j9_sanity.functional_ppc64le_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0xb7cdc0) [0x64b92817cdc0]
22:54:03   [ERR] #1: /home/jenkins/workspace/Test_openjdk23_j9_sanity.functional_ppc64le_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0xb8e80c) [0x64b92818e80c]
22:54:03   [ERR] #2: /home/jenkins/workspace/Test_openjdk23_j9_sanity.functional_ppc64le_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x765dfc) [0x64b927d65dfc]
22:54:03   [ERR] #3: /home/jenkins/workspace/Test_openjdk23_j9_sanity.functional_ppc64le_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x767aac) [0x64b927d67aac]
22:54:03   [ERR] #4: /home/jenkins/workspace/Test_openjdk23_j9_sanity.functional_ppc64le_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x2dd9d0) [0x64b9278dd9d0]
22:54:03   [ERR] #5: /home/jenkins/workspace/Test_openjdk23_j9_sanity.functional_ppc64le_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x2ddefc) [0x64b9278ddefc]
22:54:03   [ERR] #6: /home/jenkins/workspace/Test_openjdk23_j9_sanity.functional_ppc64le_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x202568) [0x64b927802568]
22:54:03   [ERR] #7: /home/jenkins/workspace/Test_openjdk23_j9_sanity.functional_ppc64le_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x175e80) [0x64b927775e80]
22:54:03   [ERR] #8: /home/jenkins/workspace/Test_openjdk23_j9_sanity.functional_ppc64le_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x176498) [0x64b927776498]
22:54:03   [ERR] #9: /home/jenkins/workspace/Test_openjdk23_j9_sanity.functional_ppc64le_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x176550) [0x64b927776550]
22:54:03   [ERR] #10: /home/jenkins/workspace/Test_openjdk23_j9_sanity.functional_ppc64le_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9prt29.so(+0x39cd4) [0x64b928d69cd4]
22:54:03   [ERR] #11: /home/jenkins/workspace/Test_openjdk23_j9_sanity.functional_ppc64le_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x176ab8) [0x64b927776ab8]
22:54:03   [ERR] #12: /home/jenkins/workspace/Test_openjdk23_j9_sanity.functional_ppc64le_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9thr29.so(+0xcc00) [0x64b92959cc00]
22:54:03   [ERR] #13: /lib/powerpc64le-linux-gnu/libc.so.6(+0xb2a5c) [0x64b9296b2a5c]
22:54:03   [ERR] 
22:54:03   [ERR] Unhandled exception
22:54:03   [ERR] Type=Unhandled trap vmState=0x00000000
22:54:03   [ERR] J9Generic_Signal_Number=00000108 Signal_Number=00000005 Error_Value=00000000 Signal_Code=fffffffa
22:54:03   [ERR] Handler1=000064B928E42C40 Handler2=000064B928D68820
22:54:03   [ERR] R0=00000000000000FA R1=000064B90C1FB510 R2=000064B929876D00 R3=0000000000000000
22:54:03   [ERR] R4=000000000022874D R5=0000000000000005 R6=2D3D544547524154 R7=0000000000000005
22:54:03   [ERR] R8=002D3D5445475241 R9=0000000000000000 R10=0000000000000000 R11=0000000000000000
22:54:03   [ERR] R12=0000000000000000 R13=000064B90C2068E0 R14=0000000000000001 R15=00000000002AE100
22:54:03   [ERR] R16=000064B9285C0778 R17=000064B9285EBAD8 R18=0000000000000001 R19=000064B90D109168
22:54:03   [ERR] R20=0000000000000000 R21=000064B90C1FB640 R22=000064B90C1FB688 R23=000064B90C1FB658
22:54:03   [ERR] R24=0000000000000030 R25=000064B900012E00 R26=000064B9278DB950 R27=000064B90001A7B0
22:54:03   [ERR] R28=0000000000000005 R29=000064B90001A8B8 R30=000064B90C1FF120 R31=000000000022874D
22:54:03   [ERR] NIP=000064B9296B55BC MSR=800000000280D033 ORIG_GPR3=0000000000228749 CTR=0000000000000000
22:54:03   [ERR] LINK=000064B9296B55BC XER=0000000000000000 CCR=0000000048884408 SOFTE=0000000000000001
22:54:03   [ERR] TRAP=0000000000003000 DAR=000064B928DD3C4C dsisr=0000000040000000 RESULT=0000000000000000
22:54:03   [ERR] FPR0=0000000000357e00 (f: 3505664.000000, d: 1.732028e-317)
22:54:03   [ERR] FPR1=3fe8000000000000 (f: 0.000000, d: 7.500000e-01)
22:54:03   [ERR] FPR2=4113400000000000 (f: 0.000000, d: 3.153920e+05)
22:54:03   [ERR] FPR3=0000002e00000021 (f: 33.000000, d: 9.761181e-313)
22:54:03   [ERR] FPR4=0000000000000000 (f: 0.000000, d: 0.000000e+00)
22:54:03   [ERR] FPR5=0000002600000027 (f: 39.000000, d: 8.063584e-313)
22:54:03   [ERR] FPR6=0000000000000000 (f: 0.000000, d: 0.000000e+00)
22:54:03   [ERR] FPR7=0000000000000000 (f: 0.000000, d: 0.000000e+00)
22:54:03   [ERR] FPR8=0000000000000000 (f: 0.000000, d: 0.000000e+00)
22:54:03   [ERR] FPR9=0000000000000000 (f: 0.000000, d: 0.000000e+00)
22:54:03   [ERR] FPR10=0000000000000000 (f: 0.000000, d: 0.000000e+00)
22:54:03   [ERR] FPR11=5b20296335613262 (f: 895562368.000000, d: 8.962195e+130)
22:54:03   [ERR] FPR12=6b646a2f305f7473 (f: 811562112.000000, d: 2.097354e+209)
22:54:03   [ERR] FPR13=616d692d6b647332 (f: 1801745152.000000, d: 2.067451e+161)
22:54:03   [ERR] FPR14=0000000000000000 (f: 0.000000, d: 0.000000e+00)
22:54:03   [ERR] FPR15=0000000000000000 (f: 0.000000, d: 0.000000e+00)
22:54:03   [ERR] FPR16=0000000000000000 (f: 0.000000, d: 0.000000e+00)
22:54:03   [ERR] FPR17=0000000000000000 (f: 0.000000, d: 0.000000e+00)
22:54:03   [ERR] FPR18=0000000000000000 (f: 0.000000, d: 0.000000e+00)
22:54:03   [ERR] FPR19=0000000000000000 (f: 0.000000, d: 0.000000e+00)
22:54:03   [ERR] FPR20=0000000000000000 (f: 0.000000, d: 0.000000e+00)
22:54:03   [ERR] FPR21=0000000000000000 (f: 0.000000, d: 0.000000e+00)
22:54:03   [ERR] FPR22=0000000000000000 (f: 0.000000, d: 0.000000e+00)
22:54:03   [ERR] FPR23=0000000000000000 (f: 0.000000, d: 0.000000e+00)
22:54:03   [ERR] FPR24=0000000000000000 (f: 0.000000, d: 0.000000e+00)
22:54:03   [ERR] FPR25=0000000000000000 (f: 0.000000, d: 0.000000e+00)
22:54:03   [ERR] FPR26=0000000000000000 (f: 0.000000, d: 0.000000e+00)
22:54:03   [ERR] FPR27=0000000000000000 (f: 0.000000, d: 0.000000e+00)
22:54:03   [ERR] FPR28=0000000000000000 (f: 0.000000, d: 0.000000e+00)
22:54:03   [ERR] FPR29=0000000000000000 (f: 0.000000, d: 0.000000e+00)
22:54:03   [ERR] FPR30=0000000000000000 (f: 0.000000, d: 0.000000e+00)
22:54:03   [ERR] FPR31=0000000000000000 (f: 0.000000, d: 0.000000e+00)
22:54:03   [ERR] Module=/lib/powerpc64le-linux-gnu/libc.so.6
22:54:03   [ERR] Module_base_address=000064B929600000 Symbol=pthread_kill
22:54:03   [ERR] Symbol_address=000064B9296B5430
22:54:03   [ERR] Target=2_90_20241123_43 (Linux 6.8.0-45-generic)
22:54:03   [ERR] CPU=ppc64le (4 logical CPUs) (0x1ea0e0000 RAM)
22:54:03   [ERR] ----------- Stack Backtrace -----------
22:54:03   [ERR] pthread_kill+0x18c (0x000064B9296B55BC [libc.so.6+0xb55bc])
22:54:03   [ERR] gsignal+0x2c (0x000064B92964BD5C [libc.so.6+0x4bd5c])
22:54:03   [ERR] _ZN2TR4trapEv+0x70 (0x000064B927D67A60 [libj9jit29.so+0x767a60])
22:54:03   [ERR] _ZN2TR15fatal_assertionEPKciS1_S1_z+0x30 (0x000064B927D67AB0 [libj9jit29.so+0x767ab0])
22:54:03   [ERR] _ZN26JITServerPersistentCHTable19commitModificationsERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x2e0 (0x000064B9278DD9D0 [libj9jit29.so+0x2dd9d0])
22:54:03   [ERR] _ZN26JITServerPersistentCHTable8doUpdateEP11TR_J9VMBaseRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES9_+0x37c (0x000064B9278DDEFC [libj9jit29.so+0x2ddefc])
22:54:03   [ERR] _ZN2TR30CompilationInfoPerThreadRemote12processEntryER21TR_MethodToBeCompiledRN2J917J9SegmentProviderE+0xa18 (0x000064B927802568 [libj9jit29.so+0x202568])
22:54:03   [ERR] _ZN2TR24CompilationInfoPerThread14processEntriesEv+0x410 (0x000064B927775E80 [libj9jit29.so+0x175e80])
22:54:03   [ERR] _ZN2TR24CompilationInfoPerThread3runEv+0xa8 (0x000064B927776498 [libj9jit29.so+0x176498])
22:54:03   [ERR] _Z30protectedCompilationThreadProcP13J9PortLibraryPN2TR24CompilationInfoPerThreadE+0xa0 (0x000064B927776550 [libj9jit29.so+0x176550])
22:54:03   [ERR] omrsig_protect+0x3e4 (0x000064B928D69CD4 [libj9prt29.so+0x39cd4])
22:54:03   [ERR] _Z21compilationThreadProcPv+0x1a8 (0x000064B927776AB8 [libj9jit29.so+0x176ab8])
22:54:03   [ERR] thread_wrapper+0x190 (0x000064B92959CC00 [libj9thr29.so+0xcc00])
22:54:03   [ERR]  (0x000064B9296B2A5C [libc.so.6+0xb2a5c])
22:54:03   [ERR] ---------------------------------------

@pshipton
Copy link
Member

pshipton commented Dec 9, 2024

https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_sanity.functional_ppc64le_linux_OMR_testList_1/85\
cmdLineTester_criu_jitserverPostRestore_2

21:47:57  Testing: Test -Xnoaot
21:47:57  Test start time: 2024/12/08 02:47:56 Coordinated Universal Time
21:47:57  Running command: bash /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_ppc64le_linux_OMR_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criuJitServerScript.sh /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_ppc64le_linux_OMR_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_ppc64le_linux_OMR_testList_1/jdkbinary/j2sdk-image/bin " -Xjit:vlog=vlog " org.openj9.criu.OptionsFileTest "JitOptionsTest -XX:+UseJITServer -Xnoaot" 1 false false
21:47:57  Time spent starting: 2 milliseconds
21:48:05  Time spent executing: 9778 milliseconds
21:48:05  Test result: FAILED
21:48:05  Output from test:
21:48:05   [OUT] start running script
21:48:05   [OUT] export GLIBC_TUNABLES=glibc.cpu.hwcaps=-XSAVEC,-XSAVE,-AVX2,-ERMS,-AVX,-AVX_Fast_Unaligned_Load
21:48:05   [OUT] export LD_BIND_NOT=on
21:48:05   [OUT] Starting /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_ppc64le_linux_OMR_testList_1/jdkbinary/j2sdk-image/bin/jitserver -XX:JITServerPort=48991 -XX:JITServerHealthProbePort=39408 
21:48:05   [OUT]  966619 ?        00:00:00 jitserver
21:48:05   [OUT] JITSERVER EXISTS
21:48:05   [OUT] Pre-checkpoint
21:48:05   [OUT] main: Sun Dec 08 02:47:58 UTC 2024, Performing CRIUSupport.checkpointJVM(), System.currentTimeMillis(): 1733626078731, System.nanoTime(): 6384166402994924
21:48:05   [OUT] JVMJITM043W AOT load and compilation disabled post restore.
21:48:05   [OUT] Post-checkpoint
21:48:05   [OUT] Removed test output files
21:48:05   [OUT] JITSERVER NO LONGER EXISTS
21:48:05   [OUT] Terminating /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_ppc64le_linux_OMR_testList_1/jdkbinary/j2sdk-image/bin/jitserver -XX:JITServerPort=48991 -XX:JITServerHealthProbePort=39408 
21:48:05   [OUT] finished script
21:48:05   [ERR] 
21:48:05   [ERR] JITServer is ready to accept incoming requests
21:48:05   [ERR] /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_ppc64le_linux_OMR_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criuJitServerScript.sh: line 114: 966697 Killed                  $TEST_JDK_BIN/java -XX:+EnableCRIUSupport -XX:JITServerPort=$JITSERVER_PORT $JVM_OPTIONS -cp "$TEST_ROOT/criu.jar" $MAINCLASS $APP_ARGS -XX:JITServerPort=$JITSERVER_PORT $NUM_CHECKPOINT > testOutput 2>&1
21:48:05   [ERR] Assertion failed at /home/jenkins/workspace/Build_JDK21_ppc64le_linux_OMR/openj9/runtime/compiler/env/JITServerPersistentCHTable.cpp:172: classInfo
21:48:05   [ERR] 	subclass info cannot be null: ensure subclasses are loaded before superclass
21:48:05   [ERR] #0: /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_ppc64le_linux_OMR_testList_1/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0xb844a0) [0x772a362c44a0]
21:48:05   [ERR] #1: /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_ppc64le_linux_OMR_testList_1/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0xb95ebc) [0x772a362d5ebc]
21:48:05   [ERR] #2: /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_ppc64le_linux_OMR_testList_1/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x76d8fc) [0x772a35ead8fc]
21:48:05   [ERR] #3: /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_ppc64le_linux_OMR_testList_1/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x76f5ac) [0x772a35eaf5ac]
21:48:05   [ERR] #4: /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_ppc64le_linux_OMR_testList_1/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x2de3f0) [0x772a35a1e3f0]
21:48:05   [ERR] #5: /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_ppc64le_linux_OMR_testList_1/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x2de91c) [0x772a35a1e91c]
21:48:05   [ERR] #6: /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_ppc64le_linux_OMR_testList_1/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x2027a8) [0x772a359427a8]
21:48:05   [ERR] #7: /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_ppc64le_linux_OMR_testList_1/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x176100) [0x772a358b6100]
21:48:05   [ERR] #8: /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_ppc64le_linux_OMR_testList_1/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x176718) [0x772a358b6718]
21:48:05   [ERR] #9: /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_ppc64le_linux_OMR_testList_1/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x1767d0) [0x772a358b67d0]
21:48:05   [ERR] #10: /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_ppc64le_linux_OMR_testList_1/jdkbinary/j2sdk-image/lib/default/libj9prt29.so(+0x39cd4) [0x772a36d89cd4]
21:48:05   [ERR] #11: /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_ppc64le_linux_OMR_testList_1/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x176d38) [0x772a358b6d38]
21:48:05   [ERR] #12: /home/jenkins/workspace/Test_openjdk21_j9_sanity.functional_ppc64le_linux_OMR_testList_1/jdkbinary/j2sdk-image/lib/default/libj9thr29.so(+0xcc00) [0x772a36d1cc00]
21:48:05   [ERR] #13: /lib/powerpc64le-linux-gnu/libc.so.6(+0xb5804) [0x772a376f5804]
21:48:05   [ERR] 
21:48:05   [ERR] Unhandled exception
21:48:05   [ERR] Type=Unhandled trap vmState=0x00000000
21:48:05   [ERR] J9Generic_Signal_Number=00000108 Signal_Number=00000005 Error_Value=00000000 Signal_Code=fffffffa
21:48:05   [ERR] Handler1=0000772A36E636B0 Handler2=0000772A36D88820
21:48:05   [ERR] R0=00000000000000FA R1=0000772A1B6AB420 R2=0000772A378C6E00 R3=0000000000000000
21:48:05   [ERR] R4=00000000000EBFDE R5=0000000000000005 R6=0000000000000073 R7=000000000000004E
21:48:05   [ERR] R8=FFFFFFFFFEFF0000 R9=0000000000000000 R10=0000000000000000 R11=0000000000000000
21:48:05   [ERR] R12=0000000000000000 R13=0000772A1B6B68E0 R14=0000000000000001 R15=00000000001C0000
21:48:05   [ERR] R16=0000772A36700778 R17=0000772A3672BAA0 R18=0000000000000001 R19=0000772985E69168
21:48:05   [ERR] R20=0000000000000000 R21=0000772A1B6AB5D0 R22=0000772A1B6AB618 R23=0000772A1B6AB5E8
21:48:05   [ERR] R24=0000000000000002 R25=0000772A14020EF0 R26=0000772A35A1C370 R27=0000772A1400F0F0
21:48:05   [ERR] R28=0000000000000005 R29=0000772A1400F118 R30=0000772A1B6AF0A0 R31=00000000000EBFDE
21:48:05   [ERR] NIP=0000772A376F866C MSR=800000000280D033 ORIG_GPR3=00000000000EBFDB CTR=0000000000000000
21:48:05   [ERR] LINK=0000772A376F866C XER=0000000000000000 CCR=0000000044884408 SOFTE=0000000000000001
21:48:05   [ERR] TRAP=0000000000003000 DAR=0000772A36DF3AAC dsisr=0000000040000000 RESULT=0000000000000000
21:48:05   [ERR] FPR0=000000000034cc00 (f: 3460096.000000, d: 1.709515e-317)
21:48:05   [ERR] FPR1=4039000000000000 (f: 0.000000, d: 2.500000e+01)
21:48:05   [ERR] FPR2=4055c00000000000 (f: 0.000000, d: 8.700000e+01)
21:48:05   [ERR] FPR3=0000002e00000021 (f: 33.000000, d: 9.761181e-313)
21:48:05   [ERR] FPR4=0000000000000000 (f: 0.000000, d: 0.000000e+00)
21:48:05   [ERR] FPR5=0000002600000027 (f: 39.000000, d: 8.063584e-313)
21:48:05   [ERR] FPR6=0000000000000000 (f: 0.000000, d: 0.000000e+00)
21:48:05   [ERR] FPR7=0000000000000000 (f: 0.000000, d: 0.000000e+00)
21:48:05   [ERR] FPR8=0000000000000000 (f: 0.000000, d: 0.000000e+00)
21:48:05   [ERR] FPR9=0000000000000000 (f: 0.000000, d: 0.000000e+00)
21:48:05   [ERR] FPR10=005d343038356636 (f: 943023680.000000, d: 6.498075e-307)
21:48:05   [ERR] FPR11=5b20293430383562 (f: 808990080.000000, d: 8.961797e+130)
21:48:05   [ERR] FPR12=616e69626b646a2f (f: 1801742848.000000, d: 2.137804e+161)
21:48:05   [ERR] FPR13=6c2f6567616d692d (f: 1634560256.000000, d: 1.321182e+213)
21:48:05   [ERR] FPR14=0000000000000000 (f: 0.000000, d: 0.000000e+00)
21:48:05   [ERR] FPR15=0000000000000000 (f: 0.000000, d: 0.000000e+00)
21:48:05   [ERR] FPR16=0000000000000000 (f: 0.000000, d: 0.000000e+00)
21:48:05   [ERR] FPR17=0000000000000000 (f: 0.000000, d: 0.000000e+00)
21:48:05   [ERR] FPR18=0000000000000000 (f: 0.000000, d: 0.000000e+00)
21:48:05   [ERR] FPR19=0000000000000000 (f: 0.000000, d: 0.000000e+00)
21:48:05   [ERR] FPR20=0000000000000000 (f: 0.000000, d: 0.000000e+00)
21:48:05   [ERR] FPR21=0000000000000000 (f: 0.000000, d: 0.000000e+00)
21:48:05   [ERR] FPR22=0000000000000000 (f: 0.000000, d: 0.000000e+00)
21:48:05   [ERR] FPR23=0000000000000000 (f: 0.000000, d: 0.000000e+00)
21:48:05   [ERR] FPR24=0000000000000000 (f: 0.000000, d: 0.000000e+00)
21:48:05   [ERR] FPR25=0000000000000000 (f: 0.000000, d: 0.000000e+00)
21:48:05   [ERR] FPR26=0000000000000000 (f: 0.000000, d: 0.000000e+00)
21:48:05   [ERR] FPR27=0000000000000000 (f: 0.000000, d: 0.000000e+00)
21:48:05   [ERR] FPR28=0000000000000000 (f: 0.000000, d: 0.000000e+00)
21:48:05   [ERR] FPR29=0000000000000000 (f: 0.000000, d: 0.000000e+00)
21:48:05   [ERR] FPR30=0000000000000000 (f: 0.000000, d: 0.000000e+00)
21:48:05   [ERR] FPR31=0000000000000000 (f: 0.000000, d: 0.000000e+00)
21:48:05   [ERR] Module=/lib/powerpc64le-linux-gnu/libc.so.6
21:48:05   [ERR] Module_base_address=0000772A37640000 Symbol=pthread_kill
21:48:05   [ERR] Symbol_address=0000772A376F84E0
21:48:05   [ERR] Target=2_90_20241208_94 (Linux 5.15.0-122-generic)
21:48:05   [ERR] CPU=ppc64le (4 logical CPUs) (0x1fb170000 RAM)
21:48:05   [ERR] ----------- Stack Backtrace -----------
21:48:05   [ERR] pthread_kill+0x18c (0x0000772A376F866C [libc.so.6+0xb866c])
21:48:05   [ERR] gsignal+0x2c (0x0000772A37691DFC [libc.so.6+0x51dfc])
21:48:05   [ERR] _ZN2TR4trapEv+0x70 (0x0000772A35EAF560 [libj9jit29.so+0x76f560])
21:48:05   [ERR] _ZN2TR15fatal_assertionEPKciS1_S1_z+0x30 (0x0000772A35EAF5B0 [libj9jit29.so+0x76f5b0])
21:48:05   [ERR] _ZN26JITServerPersistentCHTable19commitModificationsERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x2e0 (0x0000772A35A1E3F0 [libj9jit29.so+0x2de3f0])
21:48:05   [ERR] _ZN26JITServerPersistentCHTable8doUpdateEP11TR_J9VMBaseRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES9_+0x37c (0x0000772A35A1E91C [libj9jit29.so+0x2de91c])
21:48:05   [ERR] _ZN2TR30CompilationInfoPerThreadRemote12processEntryER21TR_MethodToBeCompiledRN2J917J9SegmentProviderE+0xa18 (0x0000772A359427A8 [libj9jit29.so+0x2027a8])
21:48:05   [ERR] _ZN2TR24CompilationInfoPerThread14processEntriesEv+0x410 (0x0000772A358B6100 [libj9jit29.so+0x176100])
21:48:05   [ERR] _ZN2TR24CompilationInfoPerThread3runEv+0xa8 (0x0000772A358B6718 [libj9jit29.so+0x176718])
21:48:05   [ERR] _Z30protectedCompilationThreadProcP13J9PortLibraryPN2TR24CompilationInfoPerThreadE+0xa0 (0x0000772A358B67D0 [libj9jit29.so+0x1767d0])
21:48:05   [ERR] omrsig_protect+0x3e4 (0x0000772A36D89CD4 [libj9prt29.so+0x39cd4])
21:48:05   [ERR] _Z21compilationThreadProcPv+0x1a8 (0x0000772A358B6D38 [libj9jit29.so+0x176d38])
21:48:05   [ERR] thread_wrapper+0x190 (0x0000772A36D1CC00 [libj9thr29.so+0xcc00])
21:48:05   [ERR]  (0x0000772A376F5804 [libc.so.6+0xb5804])
21:48:05   [ERR] ---------------------------------------

@hzongaro
Copy link
Member

@dsouzai, should this move out to the 0.51 release?

@mpirvu
Copy link
Contributor

mpirvu commented Dec 10, 2024

@dsouzai, should this move out to the 0.51 release?

Yes, in my opinion.

@dsouzai
Copy link
Contributor

dsouzai commented Dec 10, 2024

Yeah I agree, it's an issue that only seems to happen on zLinux with JITServer + CRIU. I've tried looking into it a while back but it's so intermittent that I haven't been able to make much progress.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
comp:jit comp:jitserver Artifacts related to JIT-as-a-Service project criu Used to track CRIU snapshot related work test failure
Projects
Status: Done
Status: In progress
Development

Successfully merging a pull request may close this issue.

8 participants