Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HCRLateAttachWorkload_previewEnabled crash vmState=0x00000000 #17907

Closed
pshipton opened this issue Aug 3, 2023 · 9 comments
Closed

HCRLateAttachWorkload_previewEnabled crash vmState=0x00000000 #17907

pshipton opened this issue Aug 3, 2023 · 9 comments
Labels
segfault Issues that describe segfaults / JVM crashes test failure

Comments

@pshipton
Copy link
Member

pshipton commented Aug 3, 2023 •
edited
Loading

https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_extended.system_s390x_linux_Nightly_testList_2/531 - ub20-390-2
HCRLateAttachWorkload_previewEnabled_0 -XX:+UseCompressedOops

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk17_j9_extended.system_s390x_linux_Nightly_testList_2/531/system_test_output.tar.gz

LT  07:49:33.912 - Completed 23.5%. Number of tests started=4480 (+596)
LT  stderr Unhandled exception
LT  stderr Type=Segmentation error vmState=0x00000000
LT  stderr J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
LT  stderr Handler1=000003FF933C94E0 Handler2=000003FF932B1D48 InaccessibleAddress=0000000000000000
LT  stderr gpr0=0000000000000000 gpr1=0000000001655540 gpr2=000000000DA4FDC0 gpr3=0000000000000000
LT  stderr gpr4=000003FF7191CF78 gpr5=0000000001870738 gpr6=0000000000000000 gpr7=0000000083B85318
LT  stderr gpr8=0000000000000000 gpr9=0000000084523390 gpr10=0000000000000001 gpr11=000000000DA10EE0
LT  stderr gpr12=0000000000000006 gpr13=0000000001653200 gpr14=000003FF7191CA68 gpr15=000003FF914FD158
LT  stderr psw=0000000000000000 mask=0705000180000000 fpc=00080000 bea=000003FF7191CA30
LT  stderr fpr0 41a615a000000000 (f: 0.000000, d: 1.852580e+08)
LT  stderr fpr1 45494000914fc798 (f: 2437924864.000000, d: 6.105077e+25)
LT  stderr fpr2 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr fpr3 3f336e21dbeabbfe (f: 3689593856.000000, d: 2.964814e-04)
LT  stderr fpr4 4549400000000000 (f: 0.000000, d: 6.105075e+25)
LT  stderr fpr5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr fpr6 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr fpr7 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr fpr8 00000000017cfb68 (f: 24968040.000000, d: 1.233585e-316)
LT  stderr fpr9 00000000017cfb30 (f: 24967984.000000, d: 1.233582e-316)
LT  stderr fpr10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr fpr11 000003ffc0bfb4c0 (f: 3233789184.000000, d: 2.172399e-311)
LT  stderr fpr12 0000000000000001 (f: 1.000000, d: 4.940656e-324)
LT  stderr fpr13 000002aa1dc22938 (f: 499263808.000000, d: 1.447448e-311)
LT  stderr fpr14 00000000017cfb28 (f: 24967976.000000, d: 1.233582e-316)
LT  stderr fpr15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr Target=2_90_20230801_542 (Linux 5.4.0-137-generic)
LT  stderr CPU=s390x (4 logical CPUs) (0x1f58f7000 RAM)
LT  stderr ----------- Stack Backtrace -----------
LT  stderr  (0x0000000000000000 [<unknown>+0x0])
LT  stderr ---------------------------------------
@pshipton pshipton added test failure segfault Issues that describe segfaults / JVM crashes labels Aug 3, 2023
@pshipton
Copy link
Member Author

pshipton commented Sep 7, 2023 •
edited
Loading

We got at least two of these in the 0.41 M1 builds.

https://hyc-runtimes-jenkins.swg-devops.com/job/Test_openjdk17_j9_extended.system_s390x_linux_testList_1/141/

https://hyc-runtimes-jenkins.swg-devops.com/job/Test_openjdk11_j9_extended.system_x86-64_windows_testList_2/123/

@dmitripivkine can you pls take a look.

@dmitripivkine
Copy link
Contributor

From core for build 141:
The crash occur an attempt to access stack allocated object located down (lower address) in the java stack and contains no object (garbage).
This object is referenced from Bytecode frame for net/adoptopenjdk/test/hcrAgent/agent/TransformerMakerThread.run()V and also mentioned in JIT-Frame-RegisterMap (jit_r12). There are two threads with identical pattern (both caused the crash). There is one of them:

<gc check (1): from debugger: THREAD STACKS: slot 21aee00(22873a0) -> 2287280: class pointer not in a class segment>

> !stackslots 0x21aee00
<21aee00> *** BEGIN STACK WALK, flags = 00400001 walkThread = 0x00000000021AEE00 ***
<21aee00> 	ITERATE_O_SLOTS
<21aee00> 	RECORD_BYTECODE_PC_OFFSET
<21aee00> Initial values: walkSP = 0x0000000002287298, PC = 0x0000000000000006, literals = 0x0000000000000000, A0 = 0x0000000002287378, j2iFrame = 0x0000000000000000, ELS = 0x000003FF9927EA10, decomp = 0x0000000000000000
<21aee00> JIT JNI call-out frame: bp = 0x00000000022872B8, sp = 0x0000000002287298, pc = 0x0000000000000006, cp = 0x0000000001E99090, arg0EA = 0x0000000002287378, flags = 0x0000000020000000
<21aee00> 	Method: java/lang/Thread.sleepImpl(JI)V !j9method 0x0000000001E9A240
<21aee00> JIT inline frame: bp = 0x0000000002287328, pc = 0x000003FF78BF0464, unwindSP = 0x00000000022872C0, cp = 0x0000000001E99090, arg0EA = 0x0000000000000000, jitInfo = 0x000003FF72307AF8
<21aee00> 	Method: java/lang/Thread.sleep(JI)V !j9method 0x0000000001E9A220
<21aee00> 	Bytecode index = 2, inlineDepth = 1, PC offset = 0x000003FF78BEFD9F
<21aee00> JIT frame: bp = 0x0000000002287328, pc = 0x000003FF78BF0464, unwindSP = 0x00000000022872C0, cp = 0x0000000001E99090, arg0EA = 0x0000000002287338, jitInfo = 0x000003FF72307AF8
<21aee00> 	Method: java/lang/Thread.sleep(J)V !j9method 0x0000000001E9A200
<21aee00> 	Bytecode index = 2, inlineDepth = 0, PC offset = 0x000000000000006A
<21aee00> 	stackMap=0x000003FF72307C02, slots=I16(0x0002) parmBaseOffset=I16(0x0018), parmSlots=U16(0x0000), localBaseOffset=I16(0x0000)
<21aee00> 	Described JIT temps starting at 0x0000000002287328 for IDATA(0x0000000000000000) slots
<21aee00> 	JIT-RegisterMap = UDATA(0x0000000000000000)
<21aee00> 	JIT-Frame-RegisterMap[0x00000000022872E0] = UDATA(0xE2D9254E03924DFE) (jit_r6)
<21aee00> 	JIT-Frame-RegisterMap[0x00000000022872E8] = UDATA(0x000003FF9A2A6558) (jit_r7)
<21aee00> 	JIT-Frame-RegisterMap[0x00000000022872F0] = UDATA(0x000003FF9A0785C0) (jit_r8)
<21aee00> 	JIT-Frame-RegisterMap[0x00000000022872F8] = UDATA(0x000003FF9927E630) (jit_r9)
<21aee00> 	JIT-Frame-RegisterMap[0x0000000002287300] = UDATA(0x000003FED8000DFD) (jit_r10)
<21aee00> 	JIT-Frame-RegisterMap[0x0000000002287308] = UDATA(0x000003FF787C7BF0) (jit_r11)
<21aee00> 	JIT-Frame-RegisterMap[0x0000000002287310] = UDATA(0x0000000002287280) (jit_r12) <---- pointer to bad stack allocated object
<21aee00> I2J values: PC = 0x000003FF9433D009, A0 = 0x0000000002287378, walkSP = 0x0000000002287348, literals = 0x0000000002195F78, JIT PC = 0x000003FF98D80FD0, pcAddress = 0x000003FF9927EA38, decomp = 0x0000000000000000
<21aee00> Bytecode frame: bp = 0x0000000002287358, sp = 0x0000000002287348, pc = 0x000003FF9433D009, cp = 0x00000000021963B0, arg0EA = 0x0000000002287378, flags = 0x0000000000000000
<21aee00> 	Method: net/adoptopenjdk/test/hcrAgent/agent/TransformerMakerThread.sleepNow(J)V !j9method 0x0000000002195F78
<21aee00> 	Bytecode index = 1
<21aee00> 	Using local mapper
<21aee00> 	Locals starting at 0x0000000002287378 for 0x0000000000000004 slots
<21aee00> 		I-Slot: a0[0x0000000002287378] = 0x000000008B9B3810
<21aee00> 		I-Slot: a1[0x0000000002287370] = 0x000001568B9B4858
<21aee00> 		I-Slot: a2[0x0000000002287368] = 0x0000000000000156
<21aee00> 		I-Slot: t3[0x0000000002287360] = 0x000007D098D80FD0
<21aee00> Bytecode frame: bp = 0x0000000002287390, sp = 0x0000000002287380, pc = 0x000003FF9433CEC0, cp = 0x00000000021963B0, arg0EA = 0x00000000022873A8, flags = 0x0000000000000000
<21aee00> 	Method: net/adoptopenjdk/test/hcrAgent/agent/TransformerMakerThread.run()V !j9method 0x0000000002195F58
<21aee00> 	Bytecode index = 48
<21aee00> 	Using local mapper
<21aee00> 	Locals starting at 0x00000000022873A8 for 0x0000000000000003 slots
<21aee00> 		O-Slot: a0[0x00000000022873A8] = 0x000000008BC80FB8
<21aee00> 		O-Slot: t1[0x00000000022873A0] = 0x0000000002287280 <---- pointer to bad stack allocated object
<21aee00> 		I-Slot: t2[0x0000000002287398] = 0x0000000000000000
<21aee00> JNI call-in frame: bp = 0x00000000022873D0, sp = 0x00000000022873B0, pc = 0x000003FF9A2A49A6, cp = 0x0000000000000000, arg0EA = 0x00000000022873D0, flags = 0x0000000000000000
<21aee00> 	New ELS = 0x0000000000000000
<21aee00> JNI native method frame: bp = 0x00000000022873F8, sp = 0x00000000022873D8, pc = 0x0000000000000007, cp = 0x0000000000000000, arg0EA = 0x00000000022873F8, flags = 0x0000000000000000
<21aee00> <end of stack>
<21aee00> *** END STACK WALK (rc = NONE) ***
>

0x02287270 :  000003fe d8000dfd 000003ff 9927e630 [ .............'.0 ]
0x02287280 :  000003ff 9a0785c0 000003ff 9a2a6558 [ .............*eX ] <----- there is no object
0x02287290 :  e2d9254e 03924dfe 00000000 01e9a240 [ ..%N..M........@ ]
0x022872A0 :  00000000 20000000 000003ff 78bf0464 [ .... .......x..d ]

@dmitripivkine
Copy link
Contributor

@tajila @pshipton

@pshipton
Copy link
Member Author

pshipton commented Sep 7, 2023

@TobiAjila can someone pls take a look, this may be a regression in 0.41, we need to decide if it's blocking.

@dmitripivkine
Copy link
Contributor

just FYI seems we do have issue regarding the same test #13504
Some of failures listed there looks very similar to this one. So, it might be not a regression after all.

@pshipton
Copy link
Member Author

pshipton commented Sep 7, 2023

Sorry, my bad. I'll move the comments to #13504 where they belong.

@pshipton pshipton mentioned this issue Sep 7, 2023
Closed
@pshipton pshipton removed the comp:vm label Sep 7, 2023
@pshipton
Copy link
Member Author

https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_extended.system_x86-64_mac_Nightly_testList_2/539/
HCRLateAttachWorkload_previewEnabled_1 -XX:-UseCompressedOops

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk17_j9_extended.system_x86-64_mac_Nightly_testList_2/539/system_test_output.tar.gz

LT  stderr Unhandled exception
LT  stderr Type=Segmentation error vmState=0x00000000
LT  stderr J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
LT  stderr Handler1=0000000000297C10 Handler2=00000000005B6970 InaccessibleAddress=0000000000000058
LT  stderr RDI=0000000000000020 RSI=0000000000000001 RAX=00007FCC1D819A20 RBX=00007FCC1F0B8D00
LT  stderr RCX=0000000000000000 RDX=000000000000002D R8=0000000000000001 R9=000000000063C838
LT  stderr R10=00007FCC1E008E00 R11=0000000000226238 R12=00007FCC1D90FDC8 R13=00007FCC1D90FB00
LT  stderr R14=00007000096528A8 R15=0000000000000000
LT  stderr RIP=00000000002A01BB GS=0000 FS=0000 RSP=0000700009651E10
LT  stderr RFlags=0000000000010206 CS=002B RBP=0000700009651E50 ERR=0000005800000004
LT  stderr TRAPNO=000000040000000E CPU=0058000000040000 FAULTVADDR=0000000000000058
LT  stderr XMM0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr XMM1 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr XMM2 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr XMM3 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr XMM4 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr XMM5 656b6f766e692f67 (f: 1852387200.000000, d: 3.557613e+180)
LT  stderr XMM6 000000003f800000 (f: 1065353216.000000, d: 5.263544e-315)
LT  stderr XMM7 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr XMM8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr XMM9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr XMM10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr XMM11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr XMM12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr XMM13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr XMM14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr XMM15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr Module=/Users/jenkins/workspace/Test_openjdk17_j9_extended.system_x86-64_mac_Nightly_testList_2/openjdkbinary/j2sdk-image/lib/default/libj9vm29.dylib
LT  stderr Module_base_address=0000000000260000 Symbol=getJNIMethodID
LT  stderr Symbol_address=00000000002A0150
LT  stderr Target=2_90_20230911_574 (Mac OS X 11.6.7)
LT  stderr CPU=amd64 (4 logical CPUs) (0x200000000 RAM)
LT  stderr ----------- Stack Backtrace -----------
LT  stderr getJNIMethodID+0x6c (0x00000000002A01BC [libj9vm29.dylib+0x401bc])
LT  stderr prepareToFixMemberNamesObjectIteratorCallback+0x158 (0x0000000000799778 [libj9jvmti29.dylib+0x33778])
LT  stderr j9mm_iterate_region_objects+0xdd (0x00000000028E56AD [libj9gc_full29.dylib+0xe56ad])
LT  stderr j9mm_iterate_all_objects+0xbb (0x00000000028E644B [libj9gc_full29.dylib+0xe644b])
LT  stderr prepareToFixMemberNames+0x46 (0x0000000000799606 [libj9jvmti29.dylib+0x33606])
LT  stderr redefineClassesCommon+0x31b (0x000000000076E08B [libj9jvmti29.dylib+0x808b])
LT  stderr jvmtiRetransformClasses+0x284 (0x000000000076E844 [libj9jvmti29.dylib+0x8844])
LT  stderr retransformClasses+0x1d1 (0x00000000030EA401 [libinstrument.dylib+0x4401])
LT  stderr ---------------------------------------

@tajila
Copy link
Contributor

tajila commented Sep 12, 2023

@babsingh

@babsingh
Copy link
Contributor

Based on the native stack from #17907 (comment), this seems to be a duplicate of

Issue: j9mm_iterate_all_objects() encounters dead objects.

Fix: is to use the J9VM_DEBUG_ATTRIBUTE_ALLOW_USER_HEAP_WALK flag. See #18076 (comment) for more details.

fyi @jdmpapin

@pshipton pshipton mentioned this issue Sep 12, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
segfault Issues that describe segfaults / JVM crashes test failure
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@babsingh @tajila @pshipton @dmitripivkine