Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HCRLateAttachWorkload_previewEnabled_1_FAILED - Segmentation error vmState=0x0002000f #13504

Closed
JasonFengJ9 opened this issue Sep 13, 2021 · 67 comments
Closed

HCRLateAttachWorkload_previewEnabled_1_FAILED - Segmentation error vmState=0x0002000f #13504

JasonFengJ9 opened this issue Sep 13, 2021 · 67 comments
Assignees
@a7ehuo
Labels
comp:jit segfault Issues that describe segfaults / JVM crashes test failure
Milestone
Release 0.44 (Jav...

Comments

@JasonFengJ9
Copy link
Member

Failure link

From an internal build job/Test_openjdk17_j9_extended.system_s390x_linux_Personal/12/tapResults/ (rhel7s390x-3-1):

openjdk version "17-internal" 2021-09-14
OpenJDK Runtime Environment (build 17-internal+0-adhoc.jenkins.BuildJDK17s390xlinuxPersonal)
Eclipse OpenJ9 VM (build v0.28.0-release-4b4e3ba6907, JRE 17 Linux s390x-64-Bit Compressed References 20210913_41 (JIT enabled, AOT enabled)
OpenJ9   - 4b4e3ba6907
OMR      - 407eef0153b
JCL      - b6c1a570c77 based on jdk-17+35)

Rerun in Grinder

Optional info

Failure output (captured from console output)

===============================================
Running test HCRLateAttachWorkload_previewEnabled_1 ...
===============================================
HCRLateAttachWorkload_previewEnabled_1 Start Time: Mon Sep 13 08:35:13 2021 Epoch Time (ms): 1631547313887
variation: Mode650
JVM_OPTIONS:  -XX:-UseCompressedOops 

LT  Starting thread 9
LT  Threads are all initialised.
LT  Entering the loop that will replace dead threads with live ones until we run out of time.
LT  stderr Unhandled exception
LT  stderr Type=Segmentation error vmState=0x0002000f
LT  stderr J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=fc01a270 Signal_Code=00000001
LT  stderr Handler1=000003FF94E48210 Handler2=000003FF94D32268 InaccessibleAddress=0000000000000000
LT  stderr gpr0=0000000000000068 gpr1=0000000000000000 gpr2=0000000099669966 gpr3=000003FF946F6132
LT  stderr gpr4=000003FF94C7E350 gpr5=0000000000424FF0 gpr6=0000000000424FF0 gpr7=0000000000424FF0
LT  stderr gpr8=000003FF94C7E350 gpr9=000003FF947667F0 gpr10=0000000000000002 gpr11=000003FF1C001AC8
LT  stderr gpr12=000003FF95F19000 gpr13=000003FF9544CF50 gpr14=000003FF944C8B88 gpr15=000003FF94C7DF30
LT  stderr psw=000003FF946219EA mask=0705000180000000 fpc=0008fe00 bea=000003FF94621948
LT  stderr fpr0 428f090600000000 (f: 0.000000, d: 4.265452e+12)
LT  stderr fpr1 3e638ee300000000 (f: 0.000000, d: 3.642990e-08)
LT  stderr fpr2 3e3ab28300000000 (f: 0.000000, d: 6.215952e-09)
LT  stderr fpr3 3b8b4e2300000000 (f: 0.000000, d: 7.227632e-22)
LT  stderr fpr4 be0044dd00000018 (f: 24.000000, d: -4.734901e-10)
LT  stderr fpr5 3e92499300000000 (f: 0.000000, d: 2.725035e-07)
LT  stderr fpr6 3ecccd5400000058 (f: 88.000000, d: 3.433473e-06)
LT  stderr fpr7 3e3a33d800000000 (f: 0.000000, d: 6.100748e-09)
LT  stderr fpr8 000000000f256800 (f: 254109696.000000, d: 1.255469e-315)
LT  stderr fpr9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr fpr10 000000000f256f78 (f: 254111616.000000, d: 1.255478e-315)
LT  stderr fpr11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr fpr12 0005cbe236e5370f (f: 920991488.000000, d: 8.060915e-309)
LT  stderr fpr13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr fpr14 000003ffc77f7bd0 (f: 3347020800.000000, d: 2.172455e-311)
LT  stderr fpr15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr Module=/home/jenkins/workspace/Test_openjdk17_j9_extended.system_s390x_linux_Personal_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9gc29.so
LT  stderr Module_base_address=000003FF94480000
LT  stderr Target=2_90_20210913_41 (Linux 3.10.0-1160.42.2.el7.s390x)
LT  stderr CPU=s390x (4 logical CPUs) (0x1ec5df000 RAM)
LT  stderr ----------- Stack Backtrace -----------
LT  stderr (0x000003FF946219EA [libj9gc29.so+0x1a19ea])
LT  stderr (0x000003FF944C8B88 [libj9gc29.so+0x48b88])
LT  stderr (0x000003FF94E8FD16 [libj9vm29.so+0x8fd16])
LT  stderr (0x000003FF94E9058E [libj9vm29.so+0x9058e])
LT  stderr (0x000003FF944C8C4C [libj9gc29.so+0x48c4c])
LT  stderr (0x000003FF944BF6E2 [libj9gc29.so+0x3f6e2])
LT  stderr (0x000003FF944BE002 [libj9gc29.so+0x3e002])
LT  stderr (0x000003FF944C1314 [libj9gc29.so+0x41314])
LT  stderr (0x000003FF9461A43E [libj9gc29.so+0x19a43e])
LT  stderr (0x000003FF945C6E38 [libj9gc29.so+0x146e38])
LT  stderr (0x000003FF945C64A8 [libj9gc29.so+0x1464a8])
LT  stderr (0x000003FF94D333D6 [libj9prt29.so+0x333d6])
LT  stderr (0x000003FF945C5E68 [libj9gc29.so+0x145e68])
LT  stderr (0x000003FF94C85A14 [libj9thr29.so+0x5a14])
LT  stderr (0x000003FF95F08312 [libpthread.so.0+0x8312])
LT  stderr (0x000003FF95D8E232 [libc.so.6+0x10e232])
LT  stderr ---------------------------------------
LT  stderr JVMDUMP039I Processing dump event "gpf", detail "" at 2021/09/13 08:36:03 - please wait.

LT  stderr JVMDUMP013I Processed dump event "gpf", detail "".
LT  stderr 0000000000433300: Unhandled exception while validating object in stack frame in thread Thread-58
LT  stderr 0000000000433300:	O-Slot=0000000000424FF0
LT  stderr 0000000000433300:	O-Slot value=0000000000424E98
LT  stderr 0000000000433300:	PC=000003FF903A1E25
LT  stderr 0000000000433300:	framesWalked=3
LT  stderr 0000000000433300:	arg0EA=0000000000424FF8
LT  stderr 0000000000433300:	walkSP=0000000000424FD0
LT  stderr 0000000000433300:	literals=0000000000416820
LT  stderr 0000000000433300:	jitInfo=0000000000000000
LT  stderr 0000000000433300:	method=0000000000416820 (net/adoptopenjdk/test/hcrAgent/agent/TransformerMakerThread.run()V) (Interpreted)
LT  stderr 0000000000433300:	stack=0000000000423868-0000000000425050
STF 08:36:17.637 - Monitoring Report Summary:
STF 08:36:17.637 -   o Process AG  has crashed unexpectedly
STF 08:36:17.637 -   o Process LT  ended with return code 255
STF 08:36:17.638 - Killing processes: AG  LT 
STF 08:36:17.638 -   o Process AG  pid 9540 is not running
STF 08:36:17.638 -   o Process LT  pid 9538 is not running
**FAILED** at step 3 (Wait for Agent to complete). Expected return value=0 Actual=1 at /home/jenkins/workspace/Test_openjdk17_j9_extended.system_s390x_linux_Personal_testList_1/aqa-tests/TKG/../TKG/output_1631545702618/HCRLateAttachWorkload_previewEnabled_1/20210913-083514-HCRLateAttachWorkload/execute.pl line 153.
STF 08:36:17.664 - **FAILED** execute script failed. Expected return value=0 Actual=1
STF 08:36:17.665 - 
STF 08:36:17.665 - ====================   T E A R D O W N   ====================
STF 08:36:17.665 - Running teardown: perl /home/jenkins/workspace/Test_openjdk17_j9_extended.system_s390x_linux_Personal_testList_1/aqa-tests/TKG/../TKG/output_1631545702618/HCRLateAttachWorkload_previewEnabled_1/20210913-083514-HCRLateAttachWorkload/tearDown.pl
STF 08:36:17.753 - TEARDOWN stage completed
STF 08:36:17.765 - 
STF 08:36:17.765 - =====================   R E S U L T S   =====================
STF 08:36:17.765 - Stage results:
STF 08:36:17.765 -   setUp:     pass
STF 08:36:17.765 -   execute:  *fail*
STF 08:36:17.765 -   teardown:  pass
STF 08:36:17.765 - 
STF 08:36:17.765 - Overall result: **FAILED**

HCRLateAttachWorkload_previewEnabled_1_FAILED

fyi @dmitripivkine
@JasonFengJ9 JasonFengJ9 added this to the Release 0.28 (Java 17) milestone Sep 13, 2021
@dmitripivkine
Copy link
Contributor

dmitripivkine commented Sep 14, 2021 •
edited
Loading

Bad O-slot is reported:

LT  stderr JVMDUMP013I Processed dump event "gpf", detail "".
LT  stderr 0000000000433300: Unhandled exception while validating object in stack frame in thread Thread-58
LT  stderr 0000000000433300: O-Slot=0000000000424FF0
LT  stderr 0000000000433300: O-Slot value=0000000000424E98
LT  stderr 0000000000433300: PC=000003FF903A1E25
LT  stderr 0000000000433300: framesWalked=3
LT  stderr 0000000000433300: arg0EA=0000000000424FF8
LT  stderr 0000000000433300: walkSP=0000000000424FD0
LT  stderr 0000000000433300: literals=0000000000416820
LT  stderr 0000000000433300: jitInfo=0000000000000000
LT  stderr 0000000000433300: method=0000000000416820 (net/adoptopenjdk/test/hcrAgent/agent/TransformerMakerThread.run()V) (Interpreted)
LT  stderr 0000000000433300: stack=0000000000423868-0000000000425050

this O-slot points to stack allocated object 0x424E98 but there is no object

0x00424E90 :  00000000 8a1c3038 00000001 00000008 [ ......08........ ] <---
0x00424EA0 :  00000000 00000000 00000001 00000008 [ ................ ]
0x00424EB0 :  57bf700d 26c4acd1 000003ff 89a7e648 [ W.p.&..........H ]
0x00424EC0 :  00000001 00000008 00000001 00000000 [ ................ ]

Unfortunately my jdmpview can not show stack slots for this thread. The Scavenger crashed an attempt to walk this frame as well.
Test results are downloaded.

@dmitripivkine
Copy link
Contributor

@pshipton @tajila Technically problematic frame is interpreted but most likely this bad object pointer has been passed from JIT frame. I am not sure should somebody from VM look to it

@tajila
Copy link
Contributor

tajila commented Sep 14, 2021

@fengxue-IS Can you please take a look at this

@pshipton
Copy link
Member

Tagging vm as well since vm is taking the first look.

@JasonFengJ9 JasonFengJ9 changed the title DK17 zLinux 0.28 release : HCRLateAttachWorkload_previewEnabled_1_FAILED - Segmentation error vmState=0x0002000f JDK17 zLinux 0.28 release : HCRLateAttachWorkload_previewEnabled_1_FAILED - Segmentation error vmState=0x0002000f Sep 14, 2021
@fengxue-IS
Copy link
Contributor

This failure is highly intermittent, reproduced with a recent nightly build in grinder (1/50).
The error log seem a bit different with a 4 unhandled exception (1 case of vmState=0x000000, 3 with vmState=0x0002000f)

[2021-09-14T16:47:08.423Z] LT  09:47:07.113 - Completed 33.3%. Number of tests started=10015 (+266)
[2021-09-14T16:47:20.238Z] LT  stderr Unhandled exception
[2021-09-14T16:47:20.238Z] LT  stderr Type=Segmentation error vmState=0x00000000
[2021-09-14T16:47:20.238Z] LT  stderr J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
[2021-09-14T16:47:20.238Z] LT  stderr Handler1=000003FFB70C7E80 Handler2=000003FFB6FB2268 InaccessibleAddress=0000000000000000
[2021-09-14T16:47:20.238Z] LT  stderr gpr0=000003FF98104520 gpr1=000003FF981041DC gpr2=000000000055D700 gpr3=0000000000000018
[2021-09-14T16:47:20.238Z] LT  stderr gpr4=000003FF98104538 gpr5=0000000000000008 gpr6=0000000000000000 gpr7=0000000000000300
[2021-09-14T16:47:20.238Z] LT  stderr gpr8=000003FFD105943B gpr9=0000000000000001 gpr10=0000000000000001 gpr11=0000000000000300
[2021-09-14T16:47:20.238Z] LT  stderr gpr12=0000000000000000 gpr13=000000000055D700 gpr14=000003FFB6665716 gpr15=000003FF951FC0B0
[2021-09-14T16:47:20.238Z] LT  stderr psw=000003FFB6652680 mask=0705100180000000 fpc=00080000 bea=000003FFB6665714
[2021-09-14T16:47:20.238Z] LT  stderr fpr0 476ccb0000000000 (f: 0.000000, d: 1.196013e+36)
[2021-09-14T16:47:20.238Z] LT  stderr fpr1 3f800000951fb7f8 (f: 2501883904.000000, d: 7.812504e-03)
[2021-09-14T16:47:20.238Z] LT  stderr fpr2 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2021-09-14T16:47:20.238Z] LT  stderr fpr3 000003ff951fb9e8 (f: 2501884416.000000, d: 2.172038e-311)
[2021-09-14T16:47:20.238Z] LT  stderr fpr4 3f80000000000000 (f: 0.000000, d: 7.812500e-03)
[2021-09-14T16:47:20.238Z] LT  stderr fpr5 000003ffb73dba40 (f: 3074275840.000000, d: 2.172321e-311)
[2021-09-14T16:47:20.238Z] LT  stderr fpr6 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2021-09-14T16:47:20.238Z] LT  stderr fpr7 000002aa07326bd0 (f: 120744912.000000, d: 1.447261e-311)
[2021-09-14T16:47:20.238Z] LT  stderr fpr8 0000000000553d98 (f: 5586328.000000, d: 2.760013e-317)
[2021-09-14T16:47:20.238Z] LT  stderr fpr9 0000000000553da0 (f: 5586336.000000, d: 2.760017e-317)
[2021-09-14T16:47:20.238Z] LT  stderr fpr10 0000000000553dd8 (f: 5586392.000000, d: 2.760044e-317)
[2021-09-14T16:47:20.238Z] LT  stderr fpr11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2021-09-14T16:47:20.238Z] LT  stderr fpr12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2021-09-14T16:47:20.238Z] LT  stderr fpr13 000002aa2cd14030 (f: 751910976.000000, d: 1.447573e-311)
[2021-09-14T16:47:20.238Z] LT  stderr fpr14 0000000000000001 (f: 1.000000, d: 4.940656e-324)
[2021-09-14T16:47:20.238Z] LT  stderr fpr15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2021-09-14T16:47:20.238Z] LT  stderr Module=/home/jenkins/workspace/Grinder_iteration_0/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so
[2021-09-14T16:47:20.238Z] LT  stderr Module_base_address=000003FFB5C80000
[2021-09-14T16:47:20.238Z] LT  stderr Target=2_90_20210913_23 (Linux 4.15.0-147-generic)
[2021-09-14T16:47:20.238Z] LT  stderr CPU=s390x (4 logical CPUs) (0xeed06000 RAM)
[2021-09-14T16:47:20.238Z] LT  stderr ----------- Stack Backtrace -----------
[2021-09-14T16:47:20.238Z] LT  stderr (0x000003FFB6652680 [libj9jit29.so+0x9d2680])
[2021-09-14T16:47:20.238Z] LT  stderr (0x000003FFB6665716 [libj9jit29.so+0x9e5716])
[2021-09-14T16:47:20.238Z] LT  stderr ---------------------------------------
[2021-09-14T16:47:20.238Z] LT  stderr JVMDUMP039I Processing dump event "gpf", detail "" at 2021/09/14 09:47:18 - please wait.
[2021-09-14T16:47:20.238Z] LT  stderr JVMDUMP032I JVM requested System dump using '/home/jenkins/workspace/Grinder_iteration_0/aqa-tests/TKG/output_16316378216234/HCRLateAttachWorkload_previewEnabled_1/20210914-094342-HCRLateAttachWorkload/results/core.20210914.094718.13789.0001.dmp' in response to an event
[2021-09-14T16:47:26.070Z] STF 09:47:24.817 - Found dump at: /home/jenkins/workspace/Grinder_iteration_0/aqa-tests/TKG/output_16316378216234/HCRLateAttachWorkload_previewEnabled_1/20210914-094342-HCRLateAttachWorkload/results/core.20210914.094718.13789.0001.dmp
[2021-09-14T16:47:26.070Z] LT  stderr JVMPORT030W /proc/sys/kernel/core_pattern setting "|/usr/share/apport/apport %p %s %c %d %P %E" specifies that the core dump is to be piped to an external program.  Attempting to rename either core or core.14436.
[2021-09-14T16:47:26.070Z] LT  stderr 
[2021-09-14T16:47:26.070Z] LT  stderr JVMDUMP010I System dump written to /home/jenkins/workspace/Grinder_iteration_0/aqa-tests/TKG/output_16316378216234/HCRLateAttachWorkload_previewEnabled_1/20210914-094342-HCRLateAttachWorkload/results/core.20210914.094718.13789.0001.dmp
[2021-09-14T16:47:26.070Z] LT  stderr Unhandled exception
[2021-09-14T16:47:26.070Z] LT  stderr Type=Segmentation error vmState=0x0002000f
[2021-09-14T16:47:26.070Z] LT  stderr J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
[2021-09-14T16:47:26.070Z] LT  stderr Handler1=000003FFB70C8210 Handler2=000003FFB6FB2268 InaccessibleAddress=FFFFFFFFFFFFF000
[2021-09-14T16:47:26.070Z] LT  stderr gpr0=0024E00000000000 gpr1=0000000000000000 gpr2=00000000008570B0 gpr3=0000000000000000
[2021-09-14T16:47:26.070Z] LT  stderr gpr4=000003FF2C2D5F0C gpr5=0000000000000000 gpr6=00000000008570B0 gpr7=00000000008577A0
[2021-09-14T16:47:26.070Z] LT  stderr gpr8=0000000004600002 gpr9=0000000000000000 gpr10=0000000000368100 gpr11=000003FFB6A7E330
[2021-09-14T16:47:26.070Z] LT  stderr gpr12=000003FFB84A6000 gpr13=0000000000000000 gpr14=0000000000000000 gpr15=000003FFB6A7E1A0
[2021-09-14T16:47:26.070Z] LT  stderr psw=000003FFB7110594 mask=0705100180000000 fpc=00080000 bea=000003FFB7110F42
[2021-09-14T16:47:26.070Z] LT  stderr fpr0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2021-09-14T16:47:26.070Z] LT  stderr fpr1 000003ffb6a7e088 (f: 3064455424.000000, d: 2.172316e-311)
[2021-09-14T16:47:26.070Z] LT  stderr fpr2 000003ffb819e1d8 (f: 3088704000.000000, d: 2.172328e-311)
[2021-09-14T16:47:26.070Z] LT  stderr fpr3 000003ffb6a7def8 (f: 3064454912.000000, d: 2.172316e-311)
[2021-09-14T16:47:26.070Z] LT  stderr fpr4 0000000000000007 (f: 7.000000, d: 3.458460e-323)
[2021-09-14T16:47:26.070Z] LT  stderr fpr5 000003ffb73dba40 (f: 3074275840.000000, d: 2.172321e-311)
[2021-09-14T16:47:26.070Z] LT  stderr fpr6 000003ff40000d40 (f: 1073745152.000000, d: 2.171332e-311)
[2021-09-14T16:47:26.070Z] LT  stderr fpr7 000002aa07326bd0 (f: 120744912.000000, d: 1.447261e-311)
[2021-09-14T16:47:26.070Z] LT  stderr fpr8 0000000000041000 (f: 266240.000000, d: 1.315400e-318)
[2021-09-14T16:47:26.070Z] LT  stderr fpr9 000003fff1ffc2a0 (f: 4060070656.000000, d: 2.172808e-311)
[2021-09-14T16:47:26.070Z] LT  stderr fpr10 000003ffb6a3f000 (f: 3064197120.000000, d: 2.172316e-311)
[2021-09-14T16:47:26.070Z] LT  stderr fpr11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2021-09-14T16:47:26.070Z] LT  stderr fpr12 0005cbf749005a59 (f: 1224759936.000000, d: 8.061362e-309)
[2021-09-14T16:47:26.070Z] LT  stderr fpr13 000002aa2cd14030 (f: 751910976.000000, d: 1.447573e-311)
[2021-09-14T16:47:26.070Z] LT  stderr fpr14 000003ffe35f93c0 (f: 3814691840.000000, d: 2.172686e-311)
[2021-09-14T16:47:26.070Z] LT  stderr fpr15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2021-09-14T16:47:26.070Z] LT  stderr Unhandled exception
[2021-09-14T16:47:26.070Z] LT  stderr Type=Segmentation error vmState=0x0002000f
[2021-09-14T16:47:26.070Z] LT  stderr Module=/home/jenkins/workspace/Grinder_iteration_0/openjdkbinary/j2sdk-image/lib/default/libj9vm29.so
[2021-09-14T16:47:26.070Z] LT  stderr Module_base_address=000003FFB7080000
[2021-09-14T16:47:26.070Z] LT  stderr J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
[2021-09-14T16:47:26.070Z] LT  stderr Handler1=000003FFB70C7E80 Handler2=000003FFB6FB2268 InaccessibleAddress=0000000000000000
[2021-09-14T16:47:26.070Z] LT  stderr Target=2_90_20210913_23 (Linux 4.15.0-147-generic)
[2021-09-14T16:47:26.070Z] LT  stderr CPU=s390x (4 logical CPUs) (0xeed06000 RAM)
[2021-09-14T16:47:26.070Z] LT  stderr ----------- Stack Backtrace -----------
[2021-09-14T16:47:26.070Z] LT  stderr gpr0=0000000000000068 gpr1=0000000000000000 gpr2=0000000099669966 gpr3=000003FFB5BF5F4A
[2021-09-14T16:47:26.070Z] LT  stderr gpr4=000003FF94679090 gpr5=000000000055AFC0 gpr6=000000000055AFC0 gpr7=000000000055AFC0
[2021-09-14T16:47:26.070Z] LT  stderr gpr8=000003FF94679090 gpr9=000003FFB5C667F0 gpr10=0000000000000002 gpr11=000003FFB05185F8
[2021-09-14T16:47:26.070Z] LT  stderr gpr12=00000000471B81E8 gpr13=000003FFB77CCF58 gpr14=000003FFB59C8B88 gpr15=000003FF94678C70
[2021-09-14T16:47:26.070Z] LT  stderr psw=000003FFB5B219EA mask=0705000180000000 fpc=00080000 bea=000003FFB5B21948
[2021-09-14T16:47:26.070Z] LT  stderr fpr0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2021-09-14T16:47:26.070Z] LT  stderr fpr1 000003ff946782d8 (f: 2489811712.000000, d: 2.172032e-311)
[2021-09-14T16:47:26.070Z] LT  stderr fpr2 000003ffb819e1d8 (f: 3088704000.000000, d: 2.172328e-311)
[2021-09-14T16:47:26.070Z] LT  stderr fpr3 000003ff94679d00 (f: 2489818368.000000, d: 2.172032e-311)
[2021-09-14T16:47:26.070Z] LT  stderr fpr4 0000000000000007 (f: 7.000000, d: 3.458460e-323)
[2021-09-14T16:47:26.070Z] LT  stderr fpr5 000003ffb73dba40 (f: 3074275840.000000, d: 2.172321e-311)
[2021-09-14T16:47:26.070Z] LT  stderr fpr6 000003ff0c003cd0 (f: 201342160.000000, d: 2.170901e-311)
[2021-09-14T16:47:26.070Z] LT  stderr fpr7 000002aa07326bd0 (f: 120744912.000000, d: 1.447261e-311)
[2021-09-14T16:47:26.070Z] LT  stderr fpr8 00720075006e0054 (f: 7209044.000000, d: 1.602212e-306)
[2021-09-14T16:47:26.070Z] LT  stderr fpr9 0074006500730074 (f: 7536756.000000, d: 1.780196e-306)
[2021-09-14T16:47:26.070Z] LT  stderr fpr10 0000000000000001 (f: 1.000000, d: 4.940656e-324)
[2021-09-14T16:47:26.070Z] LT  stderr fpr11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2021-09-14T16:47:26.070Z] LT  stderr fpr12 000003ffb0236500 (f: 2955109632.000000, d: 2.172262e-311)
[2021-09-14T16:47:26.070Z] LT  stderr fpr13 000002aa2cd14030 (f: 751910976.000000, d: 1.447573e-311)
[2021-09-14T16:47:26.070Z] LT  stderr fpr14 000003ffe35f93c0 (f: 3814691840.000000, d: 2.172686e-311)
[2021-09-14T16:47:26.070Z] LT  stderr fpr15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2021-09-14T16:47:26.070Z] LT  stderr Module=/home/jenkins/workspace/Grinder_iteration_0/openjdkbinary/j2sdk-image/lib/default/libj9gc29.so
[2021-09-14T16:47:26.070Z] LT  stderr Module_base_address=000003FFB5980000
[2021-09-14T16:47:26.070Z] LT  stderr Target=2_90_20210913_23 (Linux 4.15.0-147-generic)
[2021-09-14T16:47:26.070Z] LT  stderr CPU=s390x (4 logical CPUs) (0xeed06000 RAM)
[2021-09-14T16:47:26.070Z] LT  stderr ----------- Stack Backtrace -----------
[2021-09-14T16:47:26.070Z] LT  stderr (0x000003FFB7110594 [libj9vm29.so+0x90594])
[2021-09-14T16:47:26.070Z] LT  stderr (0x000003FFB59C8C4C [libj9gc29.so+0x48c4c])
[2021-09-14T16:47:26.070Z] LT  stderr (0x000003FFB59BF6E2 [libj9gc29.so+0x3f6e2])
[2021-09-14T16:47:26.070Z] LT  stderr (0x000003FFB59BE002 [libj9gc29.so+0x3e002])
[2021-09-14T16:47:26.070Z] LT  stderr (0x000003FFB59C1314 [libj9gc29.so+0x41314])
[2021-09-14T16:47:26.070Z] LT  stderr (0x000003FFB5B1A43E [libj9gc29.so+0x19a43e])
[2021-09-14T16:47:26.070Z] LT  stderr (0x000003FFB5AC6E38 [libj9gc29.so+0x146e38])
[2021-09-14T16:47:26.070Z] LT  stderr (0x000003FFB5AC64A8 [libj9gc29.so+0x1464a8])
[2021-09-14T16:47:26.070Z] LT  stderr (0x000003FFB6FB33D6 [libj9prt29.so+0x333d6])
[2021-09-14T16:47:26.070Z] LT  stderr (0x000003FFB5AC5E68 [libj9gc29.so+0x145e68])
[2021-09-14T16:47:26.070Z] LT  stderr (0x000003FFB6F05A14 [libj9thr29.so+0x5a14])
[2021-09-14T16:47:26.070Z] LT  stderr (0x000003FFB8287AA8 [libpthread.so.0+0x7aa8])
[2021-09-14T16:47:26.070Z] LT  stderr (0x000003FFB80FA11E [libc.so.6+0xfa11e])
[2021-09-14T16:47:26.070Z] LT  stderr (0x0000000000000000 [<unknown>+0x0])
[2021-09-14T16:47:26.070Z] LT  stderr ---------------------------------------
[2021-09-14T16:47:26.070Z] LT  stderr JVMDUMP039I Processing dump event "gpf", detail "" at 2021/09/14 09:47:24 - please wait.
[2021-09-14T16:47:26.070Z] LT  stderr (0x000003FFB5B219EA [libj9gc29.so+0x1a19ea])
[2021-09-14T16:47:26.070Z] LT  stderr (0x000003FFB59C8B88 [libj9gc29.so+0x48b88])
[2021-09-14T16:47:26.070Z] LT  stderr (0x000003FFB710FD16 [libj9vm29.so+0x8fd16])
[2021-09-14T16:47:26.070Z] LT  stderr (0x000003FFB711058E [libj9vm29.so+0x9058e])
[2021-09-14T16:47:26.070Z] LT  stderr (0x000003FFB59C8C4C [libj9gc29.so+0x48c4c])
[2021-09-14T16:47:26.070Z] LT  stderr (0x000003FFB59BF6E2 [libj9gc29.so+0x3f6e2])
[2021-09-14T16:47:26.070Z] LT  stderr (0x000003FFB59BE002 [libj9gc29.so+0x3e002])
[2021-09-14T16:47:26.070Z] LT  stderr (0x000003FFB59C1314 [libj9gc29.so+0x41314])
[2021-09-14T16:47:26.071Z] LT  stderr (0x000003FFB5B1A43E [libj9gc29.so+0x19a43e])
[2021-09-14T16:47:26.071Z] LT  stderr (0x000003FFB5AC6AA6 [libj9gc29.so+0x146aa6])
[2021-09-14T16:47:26.071Z] LT  stderr (0x000003FFB5B02414 [libj9gc29.so+0x182414])
[2021-09-14T16:47:26.071Z] LT  stderr (0x000003FFB5B11A94 [libj9gc29.so+0x191a94])
[2021-09-14T16:47:26.071Z] LT  stderr (0x000003FFB5B12ED8 [libj9gc29.so+0x192ed8])
[2021-09-14T16:47:26.071Z] LT  stderr (0x000003FFB5A9FCA6 [libj9gc29.so+0x11fca6])
[2021-09-14T16:47:26.071Z] LT  stderr (0x000003FFB5B807C8 [libj9gc29.so+0x2007c8])
[2021-09-14T16:47:26.071Z] LT  stderr (0x000003FFB5B5D3FE [libj9gc29.so+0x1dd3fe])
[2021-09-14T16:47:26.071Z] LT  stderr (0x000003FFB5ACE0BC [libj9gc29.so+0x14e0bc])
[2021-09-14T16:47:26.071Z] LT  stderr (0x000003FFB5ACD602 [libj9gc29.so+0x14d602])
[2021-09-14T16:47:26.071Z] LT  stderr (0x000003FFB5AD4500 [libj9gc29.so+0x154500])
[2021-09-14T16:47:26.071Z] LT  stderr (0x000003FFB59D03BC [libj9gc29.so+0x503bc])
[2021-09-14T16:47:26.071Z] LT  stderr (0x000003FFB664B008 [libj9jit29.so+0x9cb008])
[2021-09-14T16:47:26.071Z] LT  stderr (0x000003FFB6663968 [libj9jit29.so+0x9e3968])
[2021-09-14T16:47:26.071Z] LT  stderr ---------------------------------------
[2021-09-14T16:47:26.071Z] LT  stderr JVMDUMP039I Processing dump event "gpf", detail "" at 2021/09/14 09:47:24 - please wait.
[2021-09-14T16:47:26.071Z] LT  stderr Unhandled exception
[2021-09-14T16:47:26.071Z] LT  stderr Type=Segmentation error vmState=0x0002000f
[2021-09-14T16:47:26.071Z] LT  stderr J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
[2021-09-14T16:47:26.071Z] LT  stderr Handler1=000003FFB70C8210 Handler2=000003FFB6FB2268 InaccessibleAddress=0000000000000000
[2021-09-14T16:47:26.071Z] LT  stderr gpr0=0000000000000068 gpr1=0000000000000300 gpr2=0000000099669966 gpr3=000003FFB5BF5F4A
[2021-09-14T16:47:26.071Z] LT  stderr gpr4=000003FFB6AFE330 gpr5=000000000073CF80 gpr6=000000000073CF80 gpr7=000000000073CF80
[2021-09-14T16:47:26.071Z] LT  stderr gpr8=000003FFB6AFE330 gpr9=000003FFB5C667F0 gpr10=0000000000000002 gpr11=000003FF4C001D28
[2021-09-14T16:47:26.071Z] LT  stderr gpr12=000003FFB84A6000 gpr13=000003FFB77CCF58 gpr14=000003FFB59C8B88 gpr15=000003FFB6AFDF10
[2021-09-14T16:47:26.071Z] LT  stderr psw=000003FFB5B219EA mask=0705100180000000 fpc=00080000 bea=000003FFB5B21948
[2021-09-14T16:47:26.071Z] LT  stderr fpr0 42b78bfd6d7634df (f: 1836463360.000000, d: 2.589002e+13)
[2021-09-14T16:47:26.071Z] LT  stderr fpr1 000003ffb6afe088 (f: 3064979712.000000, d: 2.172316e-311)
[2021-09-14T16:47:26.071Z] LT  stderr fpr2 3e3ab283c90de255 (f: 3373130240.000000, d: 6.215955e-09)
[2021-09-14T16:47:26.071Z] LT  stderr fpr3 000003ffb6afd7f0 (f: 3064977408.000000, d: 2.172316e-311)
[2021-09-14T16:47:26.071Z] LT  stderr fpr4 bfe02691284dd9a3 (f: 676190656.000000, d: -5.047079e-01)
[2021-09-14T16:47:26.071Z] LT  stderr fpr5 000003ffb73dba40 (f: 3074275840.000000, d: 2.172321e-311)
[2021-09-14T16:47:26.071Z] LT  stderr fpr6 3f2996ac2d43aad2 (f: 759409344.000000, d: 1.952253e-04)
[2021-09-14T16:47:26.071Z] LT  stderr fpr7 000002aa07326bd0 (f: 120744912.000000, d: 1.447261e-311)
[2021-09-14T16:47:26.071Z] LT  stderr fpr8 0000000000041000 (f: 266240.000000, d: 1.315400e-318)
[2021-09-14T16:47:26.071Z] LT  stderr fpr9 000003fff1ffc2a0 (f: 4060070656.000000, d: 2.172808e-311)
[2021-09-14T16:47:26.071Z] LT  stderr fpr10 000003ffb6abf000 (f: 3064721408.000000, d: 2.172316e-311)
[2021-09-14T16:47:26.071Z] LT  stderr fpr11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2021-09-14T16:47:26.071Z] LT  stderr fpr12 0005cbf749005a59 (f: 1224759936.000000, d: 8.061362e-309)
[2021-09-14T16:47:26.071Z] LT  stderr fpr13 000002aa2cd14030 (f: 751910976.000000, d: 1.447573e-311)
[2021-09-14T16:47:26.071Z] LT  stderr fpr14 000003ffe35f93c0 (f: 3814691840.000000, d: 2.172686e-311)
[2021-09-14T16:47:26.071Z] LT  stderr fpr15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2021-09-14T16:47:26.071Z] LT  stderr Module=/home/jenkins/workspace/Grinder_iteration_0/openjdkbinary/j2sdk-image/lib/default/libj9gc29.so
[2021-09-14T16:47:26.071Z] LT  stderr Module_base_address=000003FFB5980000
[2021-09-14T16:47:26.071Z] LT  stderr Target=2_90_20210913_23 (Linux 4.15.0-147-generic)
[2021-09-14T16:47:26.071Z] LT  stderr CPU=s390x (4 logical CPUs) (0xeed06000 RAM)
[2021-09-14T16:47:26.071Z] LT  stderr ----------- Stack Backtrace -----------
[2021-09-14T16:47:26.071Z] LT  stderr JVMDUMP032I JVM requested System dump using '/home/jenkins/workspace/Grinder_iteration_0/aqa-tests/TKG/output_16316378216234/HCRLateAttachWorkload_previewEnabled_1/20210914-094342-HCRLateAttachWorkload/results/core.20210914.094724.13789.0003.dmp' in response to an event
[2021-09-14T16:47:26.071Z] LT  stderr (0x000003FFB5B219EA [libj9gc29.so+0x1a19ea])
[2021-09-14T16:47:26.071Z] LT  stderr (0x000003FFB59C8B88 [libj9gc29.so+0x48b88])
[2021-09-14T16:47:26.071Z] LT  stderr (0x000003FFB710FD16 [libj9vm29.so+0x8fd16])
[2021-09-14T16:47:26.071Z] LT  stderr (0x000003FFB711058E [libj9vm29.so+0x9058e])
[2021-09-14T16:47:26.071Z] LT  stderr (0x000003FFB59C8C4C [libj9gc29.so+0x48c4c])
[2021-09-14T16:47:26.071Z] LT  stderr (0x000003FFB59BF6E2 [libj9gc29.so+0x3f6e2])
[2021-09-14T16:47:26.071Z] LT  stderr (0x000003FFB59BE002 [libj9gc29.so+0x3e002])
[2021-09-14T16:47:26.071Z] LT  stderr (0x000003FFB59C1314 [libj9gc29.so+0x41314])
[2021-09-14T16:47:26.071Z] LT  stderr (0x000003FFB5B1A43E [libj9gc29.so+0x19a43e])
[2021-09-14T16:47:26.071Z] LT  stderr (0x000003FFB5AC6E38 [libj9gc29.so+0x146e38])
[2021-09-14T16:47:26.071Z] LT  stderr (0x000003FFB5AC64A8 [libj9gc29.so+0x1464a8])
[2021-09-14T16:47:26.071Z] LT  stderr (0x000003FFB6FB33D6 [libj9prt29.so+0x333d6])
[2021-09-14T16:47:26.071Z] LT  stderr (0x000003FFB5AC5E68 [libj9gc29.so+0x145e68])
[2021-09-14T16:47:26.071Z] LT  stderr (0x000003FFB6F05A14 [libj9thr29.so+0x5a14])
[2021-09-14T16:47:26.071Z] LT  stderr (0x000003FFB8287AA8 [libpthread.so.0+0x7aa8])
[2021-09-14T16:47:26.071Z] LT  stderr (0x000003FFB80FA11E [libc.so.6+0xfa11e])
[2021-09-14T16:47:26.071Z] LT  stderr (0x0000000000000000 [<unknown>+0x0])
[2021-09-14T16:47:26.071Z] LT  stderr ---------------------------------------
[2021-09-14T16:47:26.071Z] LT  stderr JVMDUMP039I Processing dump event "gpf", detail "" at 2021/09/14 09:47:24 - please wait.
[2021-09-14T16:47:26.071Z] AG  stderr core file generated - /home/jenkins/workspace/Grinder_iteration_0/aqa-tests/TKG/output_16316378216234/HCRLateAttachWorkload_previewEnabled_1/20210914-094342-HCRLateAttachWorkload/results/core.20210914.094718.13789.0001.dmp
[2021-09-14T16:47:29.462Z] AG  stderr Exception in thread "main" java.net.SocketException: Broken pipe
[2021-09-14T16:47:29.462Z] AG  stderr 	at java.base/sun.nio.ch.NioSocketImpl.implWrite(NioSocketImpl.java:420)
[2021-09-14T16:47:29.462Z] AG  stderr 	at java.base/sun.nio.ch.NioSocketImpl.write(NioSocketImpl.java:440)
[2021-09-14T16:47:29.462Z] AG  stderr 	at java.base/sun.nio.ch.NioSocketImpl$2.write(NioSocketImpl.java:826)
[2021-09-14T16:47:29.462Z] AG  stderr 	at java.base/java.net.Socket$SocketOutputStream.write(Socket.java:1035)
[2021-09-14T16:47:29.463Z] AG  stderr 	at java.base/java.net.Socket$SocketOutputStream.write(Socket.java:1031)
[2021-09-14T16:47:29.463Z] AG  stderr 	at java.base/openj9.internal.tools.attach.target.AttachmentConnection.streamSend(AttachmentConnection.java:105)
[2021-09-14T16:47:29.463Z] AG  stderr 	at jdk.attach/com.ibm.tools.attach.attacher.OpenJ9VirtualMachine.detach(OpenJ9VirtualMachine.java:201)
[2021-09-14T16:47:29.463Z] AG  stderr 	at net.adoptopenjdk.test.hcrAgent.agent.Attacher.main(Attacher.java:85)
[2021-09-14T16:47:29.463Z] STF 09:47:28.834 - Found dump at: /home/jenkins/workspace/Grinder_iteration_0/aqa-tests/TKG/output_16316378216234/HCRLateAttachWorkload_previewEnabled_1/20210914-094342-HCRLateAttachWorkload/results/javacore.20210914.094724.13789.0006.txt
[2021-09-14T16:47:29.463Z] STF 09:47:28.834 - Found dump at: /home/jenkins/workspace/Grinder_iteration_0/aqa-tests/TKG/output_16316378216234/HCRLateAttachWorkload_previewEnabled_1/20210914-094342-HCRLateAttachWorkload/results/core.20210914.094724.13789.0003.dmp
[2021-09-14T16:47:29.463Z] STF 09:47:28.834 - Found dump at: /home/jenkins/workspace/Grinder_iteration_0/aqa-tests/TKG/output_16316378216234/HCRLateAttachWorkload_previewEnabled_1/20210914-094342-HCRLateAttachWorkload/results/Snap.20210914.094724.13789.0007.trc
[2021-09-14T16:47:29.463Z] STF 09:47:28.834 - **FAILED** Process AG  ended with exit code (1) and not the expected exit code/s (0)
[2021-09-14T16:47:29.463Z] AG  stderr javacore file generated - /home/jenkins/workspace/Grinder_iteration_0/aqa-tests/TKG/output_16316378216234/HCRLateAttachWorkload_previewEnabled_1/20210914-094342-HCRLateAttachWorkload/results/javacore.20210914.094724.13789.0006.txt
[2021-09-14T16:47:29.463Z] AG  stderr core file generated - /home/jenkins/workspace/Grinder_iteration_0/aqa-tests/TKG/output_16316378216234/HCRLateAttachWorkload_previewEnabled_1/20210914-094342-HCRLateAttachWorkload/results/core.20210914.094724.13789.0003.dmp
[2021-09-14T16:47:29.463Z] AG  stderr Snap file generated - /home/jenkins/workspace/Grinder_iteration_0/aqa-tests/TKG/output_16316378216234/HCRLateAttachWorkload_previewEnabled_1/20210914-094342-HCRLateAttachWorkload/results/Snap.20210914.094724.13789.0007.trc
[2021-09-14T16:47:29.463Z] LT  stderr JVMPORT030W /proc/sys/kernel/core_pattern setting "|/usr/share/apport/apport %p %s %c %d %P %E" specifies that the core dump is to be piped to an external program.  Attempting to rename either core or core.14444.
[2021-09-14T16:47:29.463Z] LT  stderr 
[2021-09-14T16:47:29.463Z] LT  stderr JVMDUMP010I System dump written to /home/jenkins/workspace/Grinder_iteration_0/aqa-tests/TKG/output_16316378216234/HCRLateAttachWorkload_previewEnabled_1/20210914-094342-HCRLateAttachWorkload/results/core.20210914.094724.13789.0003.dmp
[2021-09-14T16:47:29.463Z] LT  stderr JVMDUMP032I JVM requested Java dump using '/home/jenkins/workspace/Grinder_iteration_0/aqa-tests/TKG/output_16316378216234/HCRLateAttachWorkload_previewEnabled_1/20210914-094342-HCRLateAttachWorkload/results/javacore.20210914.094724.13789.0006.txt' in response to an event
[2021-09-14T16:47:29.463Z] LT  stderr JVMDUMP010I Java dump written to /home/jenkins/workspace/Grinder_iteration_0/aqa-tests/TKG/output_16316378216234/HCRLateAttachWorkload_previewEnabled_1/20210914-094342-HCRLateAttachWorkload/results/javacore.20210914.094724.13789.0006.txt
[2021-09-14T16:47:29.463Z] LT  stderr JVMDUMP032I JVM requested Snap dump using '/home/jenkins/workspace/Grinder_iteration_0/aqa-tests/TKG/output_16316378216234/HCRLateAttachWorkload_previewEnabled_1/20210914-094342-HCRLateAttachWorkload/results/Snap.20210914.094724.13789.0007.trc' in response to an event
[2021-09-14T16:47:29.463Z] LT  stderr JVMDUMP010I Snap dump written to /home/jenkins/workspace/Grinder_iteration_0/aqa-tests/TKG/output_16316378216234/HCRLateAttachWorkload_previewEnabled_1/20210914-094342-HCRLateAttachWorkload/results/Snap.20210914.094724.13789.0007.trc
[2021-09-14T16:47:29.463Z] LT  stderr JVMDUMP032I JVM requested JIT dump using '/home/jenkins/workspace/Grinder_iteration_0/aqa-tests/TKG/output_16316378216234/HCRLateAttachWorkload_previewEnabled_1/20210914-094342-HCRLateAttachWorkload/results/jitdump.20210914.094724.13789.0008.dmp' in response to an event
[2021-09-14T16:47:29.463Z] LT  stderr JVMDUMP051I JIT dump occurred in 'GC Worker' thread 0x0000000000368100
[2021-09-14T16:47:29.463Z] LT  stderr JVMDUMP010I JIT dump written to /home/jenkins/workspace/Grinder_iteration_0/aqa-tests/TKG/output_16316378216234/HCRLateAttachWorkload_previewEnabled_1/20210914-094342-HCRLateAttachWorkload/results/jitdump.20210914.094724.13789.0008.dmp

Will look into the generated cores to see what caused the crash

@pshipton
Copy link
Member

Just fyi 1/50 is not a blocker, and this isn't marked as a blocker, we could move it out.

@pshipton pshipton added the segfault Issues that describe segfaults / JVM crashes label Sep 21, 2021
@hzongaro
Copy link
Member

I haven't had any luck reproducing this failure with either the original build or a more recent build. Grinder runs on each showed 0/50 failures.

@JasonFengJ9
Copy link
Member Author

An occurrence at JDK17 0.28 release build Test_openjdk17_j9_extended.system_s390x_linux_Personal/27/tapResults/ (rhel7s390x-3-4)

===============================================
Running test HCRLateAttachWorkload_previewEnabled_0 ...
===============================================
HCRLateAttachWorkload_previewEnabled_0 Start Time: Sat Sep 25 06:42:38 2021 Epoch Time (ms): 1632577358874
variation: Mode150
JVM_OPTIONS:  -XX:+UseCompressedOops 

openjdk version "17-internal" 2021-09-14
OpenJDK Runtime Environment (build 17-internal+0-adhoc.jenkins.BuildJDK17s390xlinuxPersonal)
Eclipse OpenJ9 VM (build v0.28.0-release-35fad07466a, JRE 17 Linux s390x-64-Bit Compressed References 20210925_59 (JIT enabled, AOT enabled)
OpenJ9   - 35fad07466a
OMR      - e5975af44fc
JCL      - 41171df43e2 based on jdk-17+35)

LT  06:46:12.157 - Completed 33.4%. Number of tests started=5223 (+313)
LT  stderr Unhandled exception
LT  stderr Type=Segmentation error vmState=0x0002000f
LT  stderr Unhandled exception
LT  stderr Type=Segmentation error vmState=0x0002000f
LT  stderr J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=ba66fe70 Signal_Code=00000001
LT  stderr Handler1=000003FF879C8210 Handler2=000003FF878B2268 InaccessibleAddress=0000000000000000
LT  stderr J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=7455c270 Signal_Code=00000001
LT  stderr Handler1=000003FF879C8210 Handler2=000003FF878B2268 InaccessibleAddress=0000000000000000
LT  stderr gpr0=0000000000000068 gpr1=0000000000000000 gpr2=0000000099669966 gpr3=000003FF7FFF6152
LT  stderr gpr4=000003FF87C7E350 gpr5=0000000000573980 gpr6=0000000000573980 gpr7=0000000000573980
LT  stderr gpr8=000003FF87C7E350 gpr9=000003FF800667F0 gpr10=0000000000000002 gpr11=000003FF1C001AC8
LT  stderr gpr12=000003FF8CB19000 gpr13=000003FF87FCCF58 gpr14=000003FF7FDC8B88 gpr15=000003FF87C7DF30
LT  stderr psw=000003FF7FF21A0A mask=0705000180000000 fpc=0008fe00 bea=000003FF7FF21968
LT  stderr gpr0=0000000000000068 gpr1=0000000000000000 gpr2=0000000099669966 gpr3=000003FF7FFF6152
LT  stderr gpr4=000003FF8787E350 gpr5=000000000069F620 gpr6=000000000069F620 gpr7=000000000069F620
LT  stderr gpr8=000003FF8787E350 gpr9=000003FF800667F0 gpr10=0000000000000002 gpr11=000003FF14001AC8
LT  stderr gpr12=000003FF8CB19000 gpr13=000003FF87FCCF58 gpr14=000003FF7FDC8B88 gpr15=000003FF8787DF30
LT  stderr psw=000003FF7FF21A0A mask=0705000180000000 fpc=0008fe00 bea=000003FF7FF21968
LT  stderr fpr0 42b779ec00000000 (f: 0.000000, d: 2.581242e+13)
LT  stderr fpr1 3e638e5400000000 (f: 0.000000, d: 3.642583e-08)
LT  stderr fpr2 3e3ab28300000000 (f: 0.000000, d: 6.215952e-09)
LT  stderr fpr3 3b05717400000000 (f: 0.000000, d: 2.217172e-24)
LT  stderr fpr4 3dbec76f00000000 (f: 0.000000, d: 2.799337e-11)
LT  stderr fpr5 3e92493e00000000 (f: 0.000000, d: 2.724842e-07)
LT  stderr fpr6 3eccccec00000000 (f: 0.000000, d: 3.433284e-06)
LT  stderr fpr7 3e3a334e00000000 (f: 0.000000, d: 6.100258e-09)
LT  stderr fpr8 0000000029778800 (f: 695699456.000000, d: 3.437212e-315)
LT  stderr fpr9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr fpr10 0000000029778f78 (f: 695701376.000000, d: 3.437221e-315)
LT  stderr fpr11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr fpr12 0005ccd209ea69de (f: 166357472.000000, d: 8.066004e-309)
LT  stderr fpr13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr fpr14 000003ffe59f7e80 (f: 3852434944.000000, d: 2.172705e-311)
LT  stderr fpr15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr fpr0 42b765c400000000 (f: 0.000000, d: 2.572585e+13)
LT  stderr fpr1 3e638e4800000000 (f: 0.000000, d: 3.642549e-08)
LT  stderr fpr2 3e3ab28300000000 (f: 0.000000, d: 6.215952e-09)
LT  stderr fpr3 3ae2324a00000000 (f: 0.000000, d: 4.703670e-25)
LT  stderr fpr4 3daf2c2f00000000 (f: 0.000000, d: 1.417565e-11)
LT  stderr fpr5 3e92493700000000 (f: 0.000000, d: 2.724826e-07)
LT  stderr fpr6 3ecccce300000000 (f: 0.000000, d: 3.433268e-06)
LT  stderr fpr7 3e3a334300000000 (f: 0.000000, d: 6.100219e-09)
LT  stderr fpr8 0000000029778800 (f: 695699456.000000, d: 3.437212e-315)
LT  stderr fpr9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr fpr10 0000000029778f78 (f: 695701376.000000, d: 3.437221e-315)
LT  stderr fpr11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr fpr12 0005ccd209ea69de (f: 166357472.000000, d: 8.066004e-309)
LT  stderr fpr13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr fpr14 000003ffe59f7e80 (f: 3852434944.000000, d: 2.172705e-311)
LT  stderr fpr15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr Module=/home/jenkins/workspace/Test_openjdk17_j9_extended.system_s390x_linux_Personal_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9gc29.so
LT  stderr Module_base_address=000003FF7FD80000
LT  stderr Module=/home/jenkins/workspace/Test_openjdk17_j9_extended.system_s390x_linux_Personal_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9gc29.so
LT  stderr Module_base_address=000003FF7FD80000
LT  stderr Target=2_90_20210925_59 (Linux 3.10.0-1160.42.2.el7.s390x)
LT  stderr CPU=s390x (4 logical CPUs) (0x1ec5df000 RAM)
LT  stderr Target=2_90_20210925_59 (Linux 3.10.0-1160.42.2.el7.s390x)
LT  stderr ----------- Stack Backtrace -----------
LT  stderr CPU=s390x (4 logical CPUs) (0x1ec5df000 RAM)
LT  stderr ----------- Stack Backtrace -----------
LT  stderr (0x000003FF7FF21A0A [libj9gc29.so+0x1a1a0a])
LT  stderr (0x000003FF7FDC8B88 [libj9gc29.so+0x48b88])
LT  stderr (0x000003FF87A0FD16 [libj9vm29.so+0x8fd16])
LT  stderr (0x000003FF87A1058E [libj9vm29.so+0x9058e])
LT  stderr (0x000003FF7FDC8C4C [libj9gc29.so+0x48c4c])
LT  stderr (0x000003FF7FDBF6E2 [libj9gc29.so+0x3f6e2])
LT  stderr (0x000003FF7FDBE002 [libj9gc29.so+0x3e002])
LT  stderr (0x000003FF7FDC1314 [libj9gc29.so+0x41314])
LT  stderr (0x000003FF7FF1A45E [libj9gc29.so+0x19a45e])
LT  stderr (0x000003FF7FEC6E58 [libj9gc29.so+0x146e58])
LT  stderr (0x000003FF7FEC64C8 [libj9gc29.so+0x1464c8])
LT  stderr (0x000003FF878B33D6 [libj9prt29.so+0x333d6])
LT  stderr (0x000003FF7FEC5E88 [libj9gc29.so+0x145e88])
LT  stderr (0x000003FF87805A14 [libj9thr29.so+0x5a14])
LT  stderr (0x000003FF8CB08312 [libpthread.so.0+0x8312])
LT  stderr (0x000003FF8C98E232 [libc.so.6+0x10e232])
LT  stderr ---------------------------------------
LT  stderr (0x000003FF7FF21A0A [libj9gc29.so+0x1a1a0a])
LT  stderr (0x000003FF7FDC8B88 [libj9gc29.so+0x48b88])
LT  stderr (0x000003FF87A0FD16 [libj9vm29.so+0x8fd16])
LT  stderr (0x000003FF87A1058E [libj9vm29.so+0x9058e])
LT  stderr (0x000003FF7FDC8C4C [libj9gc29.so+0x48c4c])
LT  stderr (0x000003FF7FDBF6E2 [libj9gc29.so+0x3f6e2])
LT  stderr (0x000003FF7FDBE002 [libj9gc29.so+0x3e002])
LT  stderr (0x000003FF7FDC1314 [libj9gc29.so+0x41314])
LT  stderr (0x000003FF7FF1A45E [libj9gc29.so+0x19a45e])
LT  stderr (0x000003FF7FEC6E58 [libj9gc29.so+0x146e58])
LT  stderr (0x000003FF7FEC64C8 [libj9gc29.so+0x1464c8])
LT  stderr (0x000003FF878B33D6 [libj9prt29.so+0x333d6])
LT  stderr (0x000003FF7FEC5E88 [libj9gc29.so+0x145e88])
LT  stderr (0x000003FF87805A14 [libj9thr29.so+0x5a14])
LT  stderr (0x000003FF8CB08312 [libpthread.so.0+0x8312])
LT  stderr (0x000003FF8C98E232 [libc.so.6+0x10e232])
LT  stderr ---------------------------------------
LT  stderr JVMDUMP039I Processing dump event "gpf", detail "" at 2021/09/25 06:46:19 - please wait.
LT  stderr JVMDUMP039I Processing dump event "gpf", detail "" at 2021/09/25 06:46:19 - please wait.
LT  stderr Unhandled exception
LT  stderr Type=Segmentation error vmState=0x0002000f
LT  stderr J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=ef204270 Signal_Code=00000001
LT  stderr Handler1=000003FF879C7E80 Handler2=000003FF878B2268 InaccessibleAddress=0000000000000000
LT  stderr gpr0=0000000000000068 gpr1=0000000000000000 gpr2=0000000099669966 gpr3=000003FF7FFF6152
LT  stderr gpr4=000003FF80EFD020 gpr5=00000000006CF1C0 gpr6=00000000006CF1C0 gpr7=00000000006CF1C0
LT  stderr gpr8=000003FF80EFD020 gpr9=000003FF800667F0 gpr10=0000000000000002 gpr11=000003FF886EAFA8
LT  stderr gpr12=0000000000000186 gpr13=000003FF87FCCF58 gpr14=000003FF7FDC8B88 gpr15=000003FF80EFCC00
LT  stderr psw=000003FF7FF21A0A mask=0705000180000000 fpc=0008fe00 bea=000003FF7FF21968
LT  stderr fpr0 42b78def00000000 (f: 0.000000, d: 2.589837e+13)
LT  stderr fpr1 3e638e6300000000 (f: 0.000000, d: 3.642626e-08)
LT  stderr fpr2 3e3ab28300000000 (f: 0.000000, d: 6.215952e-09)
LT  stderr fpr3 3b1b4cb600000000 (f: 0.000000, d: 5.645436e-24)
LT  stderr fpr4 3dce5a8e00000000 (f: 0.000000, d: 5.521311e-11)
LT  stderr fpr5 3e92494747ae147b (f: 1202590848.000000, d: 2.724863e-07)
LT  stderr fpr6 3eccccf700000000 (f: 0.000000, d: 3.433304e-06)
LT  stderr fpr7 3e3a335d00000000 (f: 0.000000, d: 6.100311e-09)
LT  stderr fpr8 00720075006e0054 (f: 7209044.000000, d: 1.602212e-306)
LT  stderr fpr9 0074006500730074 (f: 7536756.000000, d: 1.780196e-306)
LT  stderr fpr10 0000000000000001 (f: 1.000000, d: 4.940656e-324)
LT  stderr fpr11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr fpr12 000003ff8822a050 (f: 2283970560.000000, d: 2.171930e-311)
LT  stderr fpr13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr fpr14 000003ffe59f7e80 (f: 3852434944.000000, d: 2.172705e-311)
LT  stderr fpr15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr Module=/home/jenkins/workspace/Test_openjdk17_j9_extended.system_s390x_linux_Personal_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9gc29.so
LT  stderr Module_base_address=000003FF7FD80000
LT  stderr Target=2_90_20210925_59 (Linux 3.10.0-1160.42.2.el7.s390x)
LT  stderr CPU=s390x (4 logical CPUs) (0x1ec5df000 RAM)
LT  stderr ----------- Stack Backtrace -----------
LT  stderr (0x000003FF7FF21A0A [libj9gc29.so+0x1a1a0a])
LT  stderr (0x000003FF7FDC8B88 [libj9gc29.so+0x48b88])
LT  stderr (0x000003FF87A0FD16 [libj9vm29.so+0x8fd16])
LT  stderr (0x000003FF87A1058E [libj9vm29.so+0x9058e])
LT  stderr (0x000003FF7FDC8C4C [libj9gc29.so+0x48c4c])
LT  stderr (0x000003FF7FDBF6E2 [libj9gc29.so+0x3f6e2])
LT  stderr (0x000003FF7FDBE002 [libj9gc29.so+0x3e002])
LT  stderr (0x000003FF7FDC1314 [libj9gc29.so+0x41314])
LT  stderr (0x000003FF7FF1A45E [libj9gc29.so+0x19a45e])
LT  stderr (0x000003FF7FEC6AC6 [libj9gc29.so+0x146ac6])
LT  stderr (0x000003FF7FF02434 [libj9gc29.so+0x182434])
LT  stderr (0x000003FF7FF11AB4 [libj9gc29.so+0x191ab4])
LT  stderr (0x000003FF7FF12EF8 [libj9gc29.so+0x192ef8])
LT  stderr (0x000003FF7FE9FCA6 [libj9gc29.so+0x11fca6])
LT  stderr (0x000003FF7FF807E8 [libj9gc29.so+0x2007e8])
LT  stderr (0x000003FF7FF5CFA8 [libj9gc29.so+0x1dcfa8])
LT  stderr (0x000003FF7FECF050 [libj9gc29.so+0x14f050])
LT  stderr (0x000003FF7FECF276 [libj9gc29.so+0x14f276])
LT  stderr (0x000003FF7FECE032 [libj9gc29.so+0x14e032])
LT  stderr (0x000003FF7FED468E [libj9gc29.so+0x15468e])
LT  stderr (0x000003FF7FDCF182 [libj9gc29.so+0x4f182])
LT  stderr (0x000003FF80A4A2C2 [libj9jit29.so+0x9ca2c2])
LT  stderr (0x000003FF80A638A8 [libj9jit29.so+0x9e38a8])
LT  stderr ---------------------------------------
LT  stderr JVMDUMP039I Processing dump event "gpf", detail "" at 2021/09/25 06:46:19 - please wait.

LT  stderr JVMDUMP013I Processed dump event "gpf", detail "".
LT  stderr 0000000000441300: Unhandled exception while validating object in stack frame in thread Thread-400
LT  stderr 0000000000441300:	O-Slot=000000000069F620
LT  stderr 0000000000441300:	O-Slot value=000000000069F500
LT  stderr 0000000000441300:	PC=000003FF8839E04B
LT  stderr 0000000000441300:	framesWalked=3
LT  stderr 0000000000441300:	arg0EA=000000000069F628
LT  stderr 0000000000441300:	walkSP=000000000069F600
LT  stderr 0000000000441300:	literals=000000000040D668
LT  stderr 0000000000441300:	jitInfo=0000000000000000
LT  stderr 0000000000441300:	method=000000000040D668 (net/adoptopenjdk/test/hcrAgent/agent/TransformerMakerThread.run()V) (Interpreted)
LT  stderr 0000000000441300:	stack=000000000069DEA0-000000000069F680
STF 06:46:32.182 - Monitoring Report Summary:
STF 06:46:32.182 -   o Process AG  has crashed unexpectedly
STF 06:46:32.182 -   o Process LT  ended with return code 255
STF 06:46:32.183 - Killing processes: AG  LT 
STF 06:46:32.183 -   o Process AG  pid 20362 is not running
STF 06:46:32.183 -   o Process LT  pid 20361 is not running
**FAILED** at step 3 (Wait for Agent to complete). Expected return value=0 Actual=1 at /home/jenkins/workspace/Test_openjdk17_j9_extended.system_s390x_linux_Personal_testList_0/aqa-tests/TKG/../TKG/output_16325757266048/HCRLateAttachWorkload_previewEnabled_0/20210925-064239-HCRLateAttachWorkload/execute.pl line 153.
STF 06:46:32.196 - **FAILED** execute script failed. Expected return value=0 Actual=1
STF 06:46:32.196 - 
STF 06:46:32.196 - ====================   T E A R D O W N   ====================
STF 06:46:32.196 - Running teardown: perl /home/jenkins/workspace/Test_openjdk17_j9_extended.system_s390x_linux_Personal_testList_0/aqa-tests/TKG/../TKG/output_16325757266048/HCRLateAttachWorkload_previewEnabled_0/20210925-064239-HCRLateAttachWorkload/tearDown.pl
STF 06:46:32.269 - TEARDOWN stage completed
STF 06:46:32.278 - 
STF 06:46:32.278 - =====================   R E S U L T S   =====================
STF 06:46:32.278 - Stage results:
STF 06:46:32.278 -   setUp:     pass
STF 06:46:32.278 -   execute:  *fail*
STF 06:46:32.278 -   teardown:  pass
STF 06:46:32.278 - 
STF 06:46:32.278 - Overall result: **FAILED**

HCRLateAttachWorkload_previewEnabled_0_FAILED

@JasonFengJ9
Copy link
Member Author

JasonFengJ9 commented Oct 25, 2021 •
edited
Loading

Moved to #13499 (comment)

@tajila
Copy link
Contributor

tajila commented Oct 26, 2021

@babsingh Can you investigate this issue, Jack is busy with another one

@babsingh
Copy link
Contributor

babsingh commented Oct 27, 2021 •
edited
Loading

@JasonFengJ9 Is the LambdaLoadTest_special_J9_5m_20 failure reported in #13504 (comment) similar to the issue reported in #13499 or #13617?

This issue primarily covers the HCRLateAttachWorkload_previewEnabled_0 failure which originates from a segfault in libj9gc29.so.

@babsingh babsingh mentioned this issue Oct 27, 2021
Closed
@JasonFengJ9
Copy link
Member Author

Is the LambdaLoadTest_special_J9_5m_20 failure reported in #13504 (comment) similar to the issue reported in #13499 or #13617?

Yeah, #13504 (comment) (and an earlier #13504 (comment)) are similar w/ #13499 (comment) instead, in which LambdaLoadTest_special hit a segmentation error vmState=0x00000000 at libj9jit29.so followed by Segmentation error vmState=0x0002000f. Will re-post there.

Please focus on HCRLateAttachWorkload_previewEnabled_1_FAILED in this issue.

@JasonFengJ9 JasonFengJ9 mentioned this issue Oct 28, 2021
Closed
@babsingh
Copy link
Contributor

babsingh commented Nov 1, 2021

Grinders

Personal grinder 1: JIT enabled. 37 failures out of 1950 tests. Link to core files.

Personal grinder 2: -Xint JIT disabled. Still running.

Related Test Code

  • StringModifierAgent.agentmain: This class is a java agent, designed to create and execute multiple TransformerMakerThreads, and to replace dead threads with new ones for the duration of the test. Each thread inserts a transformer into the JVM. These transformers modify the class "String" in harmless ways, to ensure we can do so without breaking Java. The threads add their transformer, trigger a retransformation of String, remove their transformer, and die.

  • TransformerMakerThread.run: This method creates, adds, and removes a new transformer from the JVM via the Instrumentation object. The random pauses allow other, concurrent threads to add other transformers, or remove them, before we trigger retransformation. This increases the number of code paths that this test might cover based upon its duration, as we can be testing unique combinations of transformers from the first moment of the test to the end, conceivably days from now.

Stackslots

<5877300> *** BEGIN STACK WALK, flags = 00400001 walkThread = 0x0000000005877300 ***
<5877300>       ITERATE_O_SLOTS
<5877300>       RECORD_BYTECODE_PC_OFFSET
<5877300> Initial values: walkSP = 0x0000000005B461D8, PC = 0x0000000000000006, literals = 0x0000000000000000, A0 = 0x0000000005B462B8, j2iFrame = 0x0000000000000000, ELS = 0x000003FFB127EA30, decomp = 0x0000000000000000
<5877300> JIT JNI call-out frame: bp = 0x0000000005B461F8, sp = 0x0000000005B461D8, pc = 0x0000000000000006, cp = 0x00000000054FF890, arg0EA = 0x0000000005B462B8, flags = 0x0000000020000000
<5877300>       Method: java/lang/Thread.sleep(JI)V !j9method 0x0000000005500B00
<5877300> JIT frame: bp = 0x0000000005B46268, pc = 0x000003FF8AC94C5E, unwindSP = 0x0000000005B46200, cp = 0x00000000054FF890, arg0EA = 0x0000000005B46278, jitInfo = 0x000003FF50D6F188
<5877300>       Method: java/lang/Thread.sleep(J)V !j9method 0x0000000005500AE0
<5877300>       Bytecode index = 2, inlineDepth = 0, PC offset = 0x0000000000000064
<5877300>       stackMap=0x000003FF50D6F281, slots=I16(0x0002) parmBaseOffset=I16(0x0018), parmSlots=U16(0x0000), localBaseOffset=I16(0x0000)
<5877300>       Described JIT temps starting at 0x0000000005B46268 for IDATA(0x0000000000000000) slots
<5877300>       JIT-RegisterMap = UDATA(0x0000000000000000)
<5877300>       JIT-Frame-RegisterMap[0x0000000005B46220] = UDATA(0x000003FF00000582) (jit_r6)
<5877300>       JIT-Frame-RegisterMap[0x0000000005B46228] = UDATA(0x0000000000000582) (jit_r7)
<5877300>       JIT-Frame-RegisterMap[0x0000000005B46230] = UDATA(0x0000000000000000) (jit_r8)
<5877300>       JIT-Frame-RegisterMap[0x0000000005B46238] = UDATA(0x0000000088899D00) (jit_r9)
<5877300>       JIT-Frame-RegisterMap[0x0000000005B46240] = UDATA(0x000000008889A4C8) (jit_r10)
<5877300>       JIT-Frame-RegisterMap[0x0000000005B46248] = UDATA(0x00000005DEECE66D) (jit_r11)
<5877300>       JIT-Frame-RegisterMap[0x0000000005B46250] = UDATA(0x000003FF8B17FE08) (jit_r12)
<5877300> I2J values: PC = 0x000003FFAC3AA00D, A0 = 0x0000000005B462B8, walkSP = 0x0000000005B46288, literals = 0x000000000582A1E8, JIT PC = 0x000003FFB0EF8698, pcAddress = 0x000003FFB127EA58, decomp = 0x0000000000000000
<5877300> Bytecode frame: bp = 0x0000000005B46298, sp = 0x0000000005B46288, pc = 0x000003FFAC3AA00D, cp = 0x00000000058647B0, arg0EA = 0x0000000005B462B8, flags = 0x0000000000000000
<5877300>       Method: net/adoptopenjdk/test/hcrAgent/agent/TransformerMakerThread.sleepNow(J)V !j9method 0x000000000582A1E8
<5877300>       Bytecode index = 1
<5877300>       Using local mapper
<5877300>       Locals starting at 0x0000000005B462B8 for 0x0000000000000004 slots
<5877300>               I-Slot: a0[0x0000000005B462B8] = 0x0000000088899D00
<5877300>               I-Slot: a1[0x0000000005B462B0] = 0x0000045D8C847720
<5877300>               I-Slot: a2[0x0000000005B462A8] = 0x000000000000045D
<5877300>               I-Slot: t3[0x0000000005B462A0] = 0x000007D0B0EF8698
<5877300> Bytecode frame: bp = 0x0000000005B462D0, sp = 0x0000000005B462C0, pc = 0x000003FFAC3A9EEB, cp = 0x00000000058647B0, arg0EA = 0x0000000005B462E8, flags = 0x0000000000000000
<5877300>       Method: net/adoptopenjdk/test/hcrAgent/agent/TransformerMakerThread.run()V !j9method 0x000000000582A1C8
<5877300>       Bytecode index = 87
<5877300>       Using local mapper
<5877300>       Locals starting at 0x0000000005B462E8 for 0x0000000000000003 slots
<5877300>               O-Slot: a0[0x0000000005B462E8] = 0x0000000088899D00 <--- TransformerMakerThread
<5877300>               O-Slot: t1[0x0000000005B462E0] = 0x00000000893AF510 <--- StringTransformer
<5877300>               I-Slot: t2[0x0000000005B462D8] = 0x0000000000000000
<5877300> JNI call-in frame: bp = 0x0000000005B46310, sp = 0x0000000005B462F0, pc = 0x000003FFB1AA3DEE, cp = 0x0000000000000000, arg0EA = 0x0000000005B46310, flags = 0x0000000000000000
<5877300>       New ELS = 0x0000000000000000
<5877300> JNI native method frame: bp = 0x0000000005B46338, sp = 0x0000000005B46318, pc = 0x0000000000000007, cp = 0x0000000000000000, arg0EA = 0x0000000005B46338, flags = 0x0000000000000000
<5877300> <end of stack>
<5877300> *** END STACK WALK (rc = NONE) ***

> j9object 0x0000000088899D00 // Corrupt: not an instance of TransformerMakerThread
!J9Object 0x0000000088899D00 {
	struct J9Class* clazz = !j9class 0x8587FF00 //
	Object flags = 0x00000004;
Unable to read object clazz at 0x0000000088899D00 (clazz = 0x000000008587FF00)

> j9object 0x00000000893AF510 // Good: an instance of StringTransformer
!J9Object 0x00000000893AF510 {
	struct J9Class* clazz = !j9class 0x5864D00 // net/adoptopenjdk/test/hcrAgent/agent/StringTransformer
	Object flags = 0x00000000;
	I lockword = 0x00000000 (offset = 0) (java/lang/Object) <hidden>
	J seed = 0xED6F159C721639FB (offset = 4) (net/adoptopenjdk/test/hcrAgent/agent/StringTransformer)
	I percentage = 0x00000032 (offset = 12) (net/adoptopenjdk/test/hcrAgent/agent/StringTransformer)
	I threadNumber = 0x00000004 (offset = 16) (net/adoptopenjdk/test/hcrAgent/agent/StringTransformer)

Problem

The two O-slots for TransformerMakerThread.run should be an instance of TransformerMakerThread and StringTransformer respectively. While executing the code within TransformerMakerThread.run, the O-slots get corrupt. I have observed multiple scenarios: only the first O-slot is corrupt; only the second O-slot is corrupt; and both O-slots are corrupt. The corrupt O-slots are identified when the GC validates the objects in the stack frames. Sometimes, the O-slots are replaced with a stack address; I have observed the stack address either point to random data or to an indexable object. Just by looking at the core file, I am unable to find where and when the stack gets corrupt.

Looking at TransformerMakerThread.run's bytecodes: An instance of StringTransformer is created at the start. At bytecode 87, the thread sleeps. The stack corruption happens between the creation of StringTransformer and TransformerMakerThread.sleepNow. No class transformation has occurred yet. The culprit bytecode may be invokedynamic ... bsm, which was rewritten to support OJDK MHs.

    0 newdup 11 net/adoptopenjdk/test/hcrAgent/agent/StringTransformer
    3 dup
    4 aload0getfield
    5 getfield 9 net/adoptopenjdk/test/hcrAgent/agent/TransformerMakerThread.randomNumberGenerator Ljava/util/Random;
    8 invokevirtual 12 java/util/Random.nextLong()J
   11 aload0getfield
   12 getfield 6 net/adoptopenjdk/test/hcrAgent/agent/TransformerMakerThread.percentage I
   15 aload0getfield
   16 getfield 10 net/adoptopenjdk/test/hcrAgent/agent/TransformerMakerThread.threadNumber I
   19 invokespecial 13 net/adoptopenjdk/test/hcrAgent/agent/StringTransformer.<init>(JII)V
   22 astore1
   23 aload0getfield
   24 getfield 10 net/adoptopenjdk/test/hcrAgent/agent/TransformerMakerThread.threadNumber I
   27 invokedynamic 0 bsm #0:makeConcatWithConstants(I)Ljava/lang/String;
   30 nop
   31 nop
   32 iconst2
   33 invokestatic 14 net/adoptopenjdk/test/hcrAgent/agent/AgentLogger.printThis(Ljava/lang/String;I)V
   36 aload0
   37 aload0getfield
   38 getfield 9 net/adoptopenjdk/test/hcrAgent/agent/TransformerMakerThread.randomNumberGenerator Ljava/util/Random;
   41 sipush 2000
   44 invokevirtual 16 java/util/Random.nextInt(I)I
   47 i2l
   48 invokespecial 17 net/adoptopenjdk/test/hcrAgent/agent/TransformerMakerThread.sleepNow(J)V
   51 aload0getfield
   52 getfield 10 net/adoptopenjdk/test/hcrAgent/agent/TransformerMakerThread.threadNumber I
   55 invokedynamic 1 bsm #1:makeConcatWithConstants(I)Ljava/lang/String;
   58 nop
   59 nop
   60 iconst2
   61 invokestatic 14 net/adoptopenjdk/test/hcrAgent/agent/AgentLogger.printThis(Ljava/lang/String;I)V
   64 aload0getfield
   65 getfield 4 net/adoptopenjdk/test/hcrAgent/agent/TransformerMakerThread.inst Ljava/lang/instrument/Instrumentation;
   68 aload1
   69 iconst1
   70 invokeinterface2
   72 invokeinterface 18 java/lang/instrument/Instrumentation.addTransformer(Ljava/lang/instrument/ClassFileTransformer;Z)V
   75 aload0
   76 aload0getfield
   77 getfield 9 net/adoptopenjdk/test/hcrAgent/agent/TransformerMakerThread.randomNumberGenerator Ljava/util/Random;
   80 sipush 2000
   83 invokevirtual 16 java/util/Random.nextInt(I)I
   86 i2l
   87 invokespecial 17 net/adoptopenjdk/test/hcrAgent/agent/TransformerMakerThread.sleepNow(J)V

I am re-running the grinder with -Xint in order to isolate the problem to either the VM or JIT domain.

@babsingh
Copy link
Contributor

babsingh commented Nov 2, 2021

@llxia I ran a grinder (Link to Grinder #19186) for HCRLateAttachWorkload_previewEnabled_1 with EXTRA_OPTIONS = -Xint (JIT disabled). But, the test didn't run with -Xint. The core files (from the failures) show that the JIT is still enabled. How can I pass -Xint to HCRLateAttachWorkload_previewEnabled_1 in the grinder?

@fengxue-IS The stack corruption seen in this issue may be related to the stack corruption seen in #13162.

@pshipton
Copy link
Member

pshipton commented Sep 8, 2023

As the failure rate seems to have increased, I've moved this out of the backlog and into the 0.41 milestone. Previously it was claimed to be highly intermittent.

@hzongaro fyi

@hzongaro
Copy link
Member

hzongaro commented Sep 8, 2023

Annabelle @a7ehuo, may I ask you to take a crack at this one? It looks like it has started failing more frequently and we might have a duplicate issue reported in #18098.

@JasonFengJ9 JasonFengJ9 mentioned this issue Sep 9, 2023
Open
@dmitripivkine dmitripivkine mentioned this issue Sep 11, 2023
Open
@a7ehuo
Copy link
Contributor

a7ehuo commented Sep 11, 2023 •
edited
Loading

I took a look at the core from build 141 (core.20230907.141440.1476192.0001.dmp) [1]. What I've found so far is not much more than what @dmitripivkine and @babsingh had previously found. The investigation is ongoing.

net/adoptopenjdk/test/hcrAgent/agent/TransformerMakerThread.run()V ran in 10 threads [2]. Eight threads ran the JIT'd TransformerMakerThread.run. Two threads run interpreted TransformerMakerThread.run. The btyecode frame of TransformerMakerThread.run from both threads (0x021aee00 and 0x21b3a00) contains corrupted O-Slot value [3].

The btyecode frame of TransformerMakerThread.run from both threads is at bytecode index 48: 48 invokespecial 17 net/adoptopenjdk/test/hcrAgent/agent/TransformerMakerThread.sleepNow(J)V [4]. I plan to take a look at java/lang/Thread.sleep(J)V next since that's the JIT frame after TransformerMakerThread.sleepNow and also shows the corrupted O-Slot value in jit_r12 as @dmitripivkine has pointed out.

[1]

00000000021AEE00: Unhandled exception while validating object in stack frame in thread Thread-60
00000000021AEE00:	O-Slot=00000000022873A0
00000000021AEE00:	O-Slot value=0000000002287280
00000000021AEE00:	PC=000003FF9433CEC0
00000000021AEE00:	framesWalked=3
00000000021AEE00:	arg0EA=00000000022873A8
00000000021AEE00:	walkSP=0000000002287380
00000000021AEE00:	literals=0000000002195F58
00000000021AEE00:	jitInfo=0000000000000000
00000000021AEE00:	method=0000000002195F58 (net/adoptopenjdk/test/hcrAgent/agent/TransformerMakerThread.run()V) (Interpreted)
00000000021AEE00:	stack=0000000002285C20-0000000002287400

[2]

	JIT frame -> !stack 0x021b6000	!j9vmthread 0x021b6000	!j9thread 0x3ff18066690	tid 0x1686e3 (1476323) // (Thread-52)
	JIT frame -> !stack 0x021bab00	!j9vmthread 0x021bab00	!j9thread 0x3ff18073880	tid 0x1686e6 (1476326) // (Thread-53)
	JIT frame -> !stack 0x021b8500	!j9vmthread 0x021b8500	!j9thread 0x3ff1c00a998	tid 0x1686e7 (1476327) // (Thread-54)
	JIT frame -> !stack 0x021a7d00	!j9vmthread 0x021a7d00	!j9thread 0x3ff18063ac0	tid 0x1686e8 (1476328) // (Thread-55)
	JIT frame -> !stack 0x021b1400	!j9vmthread 0x021b1400	!j9thread 0x3ff18073d78	tid 0x1686e9 (1476329) // (Thread-56)
	JIT frame -> !stack 0x021bd100	!j9vmthread 0x021bd100	!j9thread 0x3ff18066198	tid 0x1686ea (1476330) // (Thread-57)
	JIT frame -> !stack 0x021aa200	!j9vmthread 0x021aa200	!j9thread 0x3ff18065ca0	tid 0x1686eb (1476331) // (Thread-58)
	JIT frame -> !stack 0x021ac800	!j9vmthread 0x021ac800	!j9thread 0x3ff18063fb8	tid 0x1686ec (1476332) // (Thread-59)
	Bytecode frame -> !stack 0x021aee00	!j9vmthread 0x021aee00	!j9thread 0x3ff18074270	tid 0x1686f1 (1476337) // (Thread-60)
	Bytecode frame -> !stack 0x021b3a00	!j9vmthread 0x021b3a00	!j9thread 0x3ff1c00ae90	tid 0x1686f2 (1476338) // (Thread-61)

[3]

...
<21aee00> Bytecode frame: bp = 0x0000000002287390, sp = 0x0000000002287380, pc = 0x000003FF9433CEC0, cp = 0x00000000021963B0, arg0EA = 0x00000000022873A8, flags = 0x0000000000000000
<21aee00> 	Method: net/adoptopenjdk/test/hcrAgent/agent/TransformerMakerThread.run()V !j9method 0x0000000002195F58
<21aee00> 	Bytecode index = 48
<21aee00> 	Using local mapper
<21aee00> 	Locals starting at 0x00000000022873A8 for 0x0000000000000003 slots
<21aee00> 		O-Slot: a0[0x00000000022873A8] = 0x000000008BC80FB8 //<-- Obj - {net/adoptopenjdk/test/hcrAgent/agent/TransformerMakerThread}
<21aee00> 		O-Slot: t1[0x00000000022873A0] = 0x0000000002287280 //<-- Corrupted
<21aee00> 		I-Slot: t2[0x0000000002287398] = 0x0000000000000000
<21aee00> JNI call-in frame: bp = 0x00000000022873D0, sp = 0x00000000022873B0, pc = 0x000003FF9A2A49A6, cp = 0x0000000000000000, arg0EA = 0x00000000022873D0, flags = 0x0000000000000000
<21aee00> 	New ELS = 0x0000000000000000
<21aee00> JNI native method frame: bp = 0x00000000022873F8, sp = 0x00000000022873D8, pc = 0x0000000000000007, cp = 0x0000000000000000, arg0EA = 0x00000000022873F8, flags = 0x0000000000000000
<21aee00> <end of stack>
<21aee00> *** END STACK WALK (rc = NONE) ***

...
<21b3a00> Bytecode frame: bp = 0x0000000002288BF0, sp = 0x0000000002288BE0, pc = 0x000003FF9433CEC0, cp = 0x00000000021963B0, arg0EA = 0x0000000002288C08, flags = 0x0000000000000000
<21b3a00> 	Method: net/adoptopenjdk/test/hcrAgent/agent/TransformerMakerThread.run()V !j9method 0x0000000002195F58
<21b3a00> 	Bytecode index = 48
<21b3a00> 	Using local mapper
<21b3a00> 	Locals starting at 0x0000000002288C08 for 0x0000000000000003 slots
<21b3a00> 		O-Slot: a0[0x0000000002288C08] = 0x000000008BC810D8 //<-- Obj - {net/adoptopenjdk/test/hcrAgent/agent/TransformerMakerThread}
<21b3a00> 		O-Slot: t1[0x0000000002288C00] = 0x0000000002288AE0 //<-- Corrupted
<21b3a00> 		I-Slot: t2[0x0000000002288BF8] = 0x0000000000000000
<21b3a00> JNI call-in frame: bp = 0x0000000002288C30, sp = 0x0000000002288C10, pc = 0x000003FF9A2A49A6, cp = 0x0000000000000000, arg0EA = 0x0000000002288C30, flags = 0x0000000000000000
<21b3a00> 	New ELS = 0x0000000000000000
<21b3a00> JNI native method frame: bp = 0x0000000002288C58, sp = 0x0000000002288C38, pc = 0x0000000000000007, cp = 0x0000000000000000, arg0EA = 0x0000000002288C58, flags = 0x0000000000000000
<21b3a00> <end of stack>
<21b3a00> *** END STACK WALK (rc = NONE) ***

[4]

    0 newdup 11 net/adoptopenjdk/test/hcrAgent/agent/StringTransformer
    3 dup 
    4 aload0getfield 
    5 getfield 9 net/adoptopenjdk/test/hcrAgent/agent/TransformerMakerThread.randomNumberGenerator Ljava/util/Random;
    8 invokevirtual 12 java/util/Random.nextLong()J
   11 aload0getfield 
   12 getfield 6 net/adoptopenjdk/test/hcrAgent/agent/TransformerMakerThread.percentage I
   15 aload0getfield 
   16 getfield 10 net/adoptopenjdk/test/hcrAgent/agent/TransformerMakerThread.threadNumber I
   19 invokespecial 13 net/adoptopenjdk/test/hcrAgent/agent/StringTransformer.<init>(JII)V
   22 astore1 
   23 aload0getfield 
   24 getfield 10 net/adoptopenjdk/test/hcrAgent/agent/TransformerMakerThread.threadNumber I
   27 invokedynamic 0 bsm #0:makeConcatWithConstants(I)Ljava/lang/String;
   30 nop 
   31 nop 
   32 iconst2 
   33 invokestatic 14 net/adoptopenjdk/test/hcrAgent/agent/AgentLogger.printThis(Ljava/lang/String;I)V
   36 aload0 
   37 aload0getfield 
   38 getfield 9 net/adoptopenjdk/test/hcrAgent/agent/TransformerMakerThread.randomNumberGenerator Ljava/util/Random;
   41 sipush 2000
   44 invokevirtual 16 java/util/Random.nextInt(I)I
   47 i2l 
   48 invokespecial 17 net/adoptopenjdk/test/hcrAgent/agent/TransformerMakerThread.sleepNow(J)V //<--- 
   51 aload0getfield

@a7ehuo
Copy link
Contributor

a7ehuo commented Sep 15, 2023 •
edited
Loading

The failure is highly intermittent. The test running in multiple threads adds random method calls to the public methods in java/lang/String. Because of the frequent class redefinition events, the compilation of TransformerMakerThread.run is most of the time interrupted. To recreate the scenario where TransformerMakerThread.run is compiled like when the GC assert happens, I added count=1 in order for TransformerMakerThread.run to be compiled. So far I ran the test 50 times, all passed.

java/lang/Thread.sleep(J)V calls the native implementation of java/lang/Thread.sleepImpl(JI)V [1]. The %r12 before calling java/lang/Thread.sleepImpl(JI)V (instruction 0x3ff78bf043a) stores the address 0x3ff78bf0500 that contains the J9Method pointer 0x1e9a240 for java/lang/Thread.sleepImpl(JI)V [2].

The invalid O-Slot value 0x2287280 at 0x22873a0 from the previous bytecode frame for TransformerMakerThread.run is supposed to be an object of StringTransformer [3]. I don't see how the saved jit_r12 at 0x2287310 ends up containing the same O-Slot value while the real register %r12 is supposed to contain a pointer of the J9Method pointer 0x1e9a240 for java/lang/Thread.sleepImpl(JI)V. @0xdaryl I'm wondering if you might know which part of the code stores these jit_r6, jit_r7, ... jit_r12 onto the stack. Meanwhile I'm also not too sure I'm looking at the right direction. All the stored values for jit_r6, jit_r7, ... jit_r12 don't seem to make much sense. For example, jit_r11 points to the code of java/lang/Thread.startImpl but none of these Thread.sleep methods invoke Thread.startImpl. I wonder if all these values for jit_r6, jit_r7, ... jit_r12 on the stack are meant to be garbage in this case and it is a coincidence that jit_r12 on the stack happens to be the same value as the invalid O-Slot value from the previous bytecode frame for TransformerMakerThread.run.

[1]

interpreted : net/adoptopenjdk/test/hcrAgent/agent/TransformerMakerThread.run()V
interpreted :    net/adoptopenjdk/test/hcrAgent/agent/TransformerMakerThread.sleepNow(J)V
JIT         :       java/lang/Thread.sleep(J)V
JIT(inlined):         java/lang/Thread.sleep(JI)V
Native Impl :            java/lang/Thread.sleepImpl(JI)V

[2]

0x3ff78bf03f4 {java/lang/Thread.sleep} -6                        c54000060010 bprp      4, 0x3ff78bf03f4, 0x3ff78cb0414 {net/.../TransformerMakerThread.run} +78362
0x3ff78bf03fa {java/lang/Thread.sleep} +0                        e31050000004 lg        %r1, 0(%r5)
0x3ff78bf0400 {java/lang/Thread.sleep} +6                        e3e05ff8ff24 stg       %r14, -8(%r5)
0x3ff78bf0406 {java/lang/Thread.sleep} +12                       e3505f90ff71 lay       %r5, -0x70(%r5)
0x3ff78bf040c {java/lang/Thread.sleep} +18                       e350d0500021 clg       %r5, 0x50(%r13)  J9VMThread.stackOverflowMark
0x3ff78bf0412 {java/lang/Thread.sleep} +24                       a74400b5     jl        0x3ff78bf057c C>> +386 
0x3ff78bf0416 {java/lang/Thread.sleep} +28                       eb6c50200024 stmg      %r6, %r12, 0x20(%r5)  <<< ^+404
0x3ff78bf041c {java/lang/Thread.sleep} +34                       e31050700024 stg       %r1, 0x70(%r5) 
0x3ff78bf0422 {java/lang/Thread.sleep} +40                       b9040031     lgr       %r3, %r1
0x3ff78bf0426 {java/lang/Thread.sleep} +44                       b904002d     lgr       %r2, %r13            //<---- first parameter for JNI call. r13 is methodMetaDataRealRegister or J9VMThread?
0x3ff78bf042a {java/lang/Thread.sleep} +48                       c06043a0db4a larl      %r6, 0x4000000babe   //<---- generateLoadLiteralPoolAddress 
0x3ff78bf0430 {java/lang/Thread.sleep} +54                       a5610000     iihl      %r6, 0
0x3ff78bf0434 {java/lang/Thread.sleep} +58                       1700         xr        %r0, %r0
0x3ff78bf0436 {java/lang/Thread.sleep} +60                       b9140080     lgfr      %r8, %r0
0x3ff78bf043a {java/lang/Thread.sleep} +64                       c0c000000063 larl      %r12, 0x3ff78bf0500 {java/lang/Thread.sleep} +262 //<---- 0x3ff78bf0500: 0x0000000001e9a240 J9Method - {java/lang/Thread.sleepImpl}
0x3ff78bf0440 {java/lang/Thread.sleep} +70                       e3505fd8ff71 lay       %r5, -0x28(%r5)      //<----  adjust sp 
0x3ff78bf0446 {java/lang/Thread.sleep} +76                       d217d028c028 mvc       0x28(0x18, %r13), 0x28(%r12) 
0x3ff78bf044c {java/lang/Thread.sleep} +82                       e350d0200024 stg       %r5, 0x20(%r13)  J9VMThread.sp //<---- store out jsp
0x3ff78bf0452 {java/lang/Thread.sleep} +88                       d2275000c000 mvc       0(0x28, %r5), 0(%r12)
0x3ff78bf0458 {java/lang/Thread.sleep} +94                       e3e0c0500004 lg        %r14, 0x50(%r12) 
0x3ff78bf045e {java/lang/Thread.sleep} +100                      b9040048     lgr       %r4, %r8
0x3ff78bf0462 {java/lang/Thread.sleep} +104          0:0         0dee         basr      %r14, %r14 Call>>  // lload0  //<---- call Thread.sleepImpl 
0x3ff78bf0464 {java/lang/Thread.sleep} +106                      e350d0200004 lg        %r5, 0x20(%r13)  J9VMThread.sp  /<---- pc 
0x3ff78bf046a {java/lang/Thread.sleep} +112                      e350d0300008 ag        %r5, 0x30(%r13)  J9VMThread.literals
0x3ff78bf0470 {java/lang/Thread.sleep} +118                      c01f00030000 llilf     %r1, 0x30000  CONST 0x30000 
...

(kca) what (0x3ff78bf0500)
0x3ff78bf0500: 0x0000000001e9a240 J9Method - {java/lang/Thread.sleepImpl}

[3]

--------------------------------------
0x22872c0: 0x0000000000000000 Unknown!                                //<---- unwindSP for java/lang/Thread.sleep(J)V and java/lang/Thread.sleep(JI)V 
0x22872c8: 0x0000000000000000 Unknown!
0x22872d0: 0x000000000e828db0 Obj - {java/lang/invoke/BoundMethodHandle$Species_LL}T
0x22872d8: 0x000000008b9b4820 Obj - {[B}
0x22872e0: 0xe2d9254e03924dfe Unknown!                                //<---- jit_r6
0x22872e8: 0x000003ff9a2a6558 Ptr Unknown!                            //<---- jit_r7
0x22872f0: 0x000003ff9a0785c0 Ptr Unknown!                            //<---- jit_r8
0x22872f8: 0x000003ff9927e630 Ptr Unknown!                            //<---- jit_r9
0x2287300: 0x000003fed8000dfd Ptr Unknown!                            //<---- jit_r10
0x2287308: 0x000003ff787c7bf0 {java/lang/Thread.startImpl} +236       //<---- jit_r11
0x2287310: 0x0000000002287280 on-java-stack(0x21aee00) Ptr Unknown!   //<---- jit_r12 same value as in 0x22873a0
0x2287318: 0x000003ff9927e630 Ptr Unknown!
0x2287320: 0x000003fed8000dfd Ptr Unknown!
0x2287328: 0x000003ff98d80fd0 Ptr Unknown!                            //<---- bp for java/lang/Thread.sleep(J)V and java/lang/Thread.sleep(JI)V 
--------------------------------------
0x2287330: 0x0000000000000156 Unknown!
0x2287338: 0x000003ff9433cea3 Ptr Unknown!                            //<---- arg0EA for java/lang/Thread.sleep(J)V, points to local slot 0 on stack
0x2287340: 0x000003ff9433cea3 Ptr Unknown!
--------------------------------------
0x2287348: 0x0000000002195f58 J9Method - {net/adoptopenjdk/test/hcrAgent/agent/TransformerMakerThread.run}                     // <---- walkSP for TransformerMakerThread.sleepNow
0x2287350: 0x000003ff9433cec0 Ptr Unknown!
0x2287358: 0x00000000022873a8 on-java-stack(0x21aee00) Ptr Obj - {net/adoptopenjdk/test/hcrAgent/agent/TransformerMakerThread} // <---- bp for TransformerMakerThread.sleepNow
--------------------------------------
0x2287360: 0x000007d098d80fd0 Unknown!
0x2287368: 0x0000000000000156 Unknown!
0x2287370: 0x000001568b9b4858 Unknown!
0x2287378: 0x000000008b9b3810 Moved Obj - {net/adoptopenjdk/test/hcrAgent/agent/TransformerMakerThread} //<---- arg0EA for TransformerMakerThread.sleepNow, arg0EA for JIT JNI call-out frame java/lang/Thread.sleepImpl, A0 for I2J   
--------------------------------------
0x2287380: 0x0000000000000000 Unknown!                                                                  // <---- sp for TransformerMakerThread.run
0x2287388: 0x000003ff9a2a49a6 Ptr Unknown!
0x2287390: 0x00000000022873d0 on-java-stack(0x21aee00) Ptr on-java-stack(0x21aee00) Unknown!            // <---- bp for TransformerMakerThread.run
--------------------------------------
0x2287398: 0x0000000000000000 Unknown!
0x22873a0: 0x0000000002287280 on-java-stack(0x21aee00) Ptr Unknown!                                     //<--- corrupted, supposed to be object StringTransformer. same value as in jit_r12 at 0x2287310
0x22873a8: 0x000000008bc80fb8 Obj - {net/adoptopenjdk/test/hcrAgent/agent/TransformerMakerThread}       //<--- arg0EA for TransformerMakerThread.run
--------------------------------------
0x22873b0: 0x0000000000000000 Unknown!                                                                  //<---- sp for  JNI call-in frame
0x22873b8: 0x0000000000000000 Unknown!
0x22873c0: 0x0000000000000000 Unknown!
0x22873c8: 0x0000000000000007 Unknown!
0x22873d0: 0x00000000022873fa on-java-stack(0x21aee00) Ptr Unknown!                                     //<----- bp and arg0EA for JNI call-in frame

@a7ehuo
Copy link
Contributor

a7ehuo commented Sep 21, 2023

Ran in Grinder: FR 6/500. @pshipton This issue has been around for awhile and the failure rate is 1.2%. I'm still investigating. Should it be moved out of 0.41 into the next release?

@pshipton
Copy link
Member

Yup, thanks.

@a7ehuo
Copy link
Contributor

a7ehuo commented Oct 19, 2023 •
edited
Loading

After adding JIT options to try to capture some compilation logs [1], I reproduced the issue once (FR: about 1/400) with all interpreted methods on the stack frames [2]. @babsingh So far I'm not sure how the object could be corrupted by JIT'd methods since there is no JIT'd method on the stack. Could this problem be investigated again from the VM's perspective? Thanks!

[1]

JIT: env var TR_Options is set to verbose,vlog=/root/home/ahuo/src/issues_triage/13504_HCRLateAttachWorkload_previewEnabled_18098_HCRLateAttachWorkload_1/debug/logs/vlogs/vlog,{net/adoptopenjdk/test/hcrAgent/agent/TransformerMakerThread.run*,java/util/TreeMap.containsValue*,java/lang/Thread.sleep*}(traceFull,traceILGen,traceCG,log=/root/home/ahuo/src/issues_triage/13504_HCRLateAttachWorkload_previewEnabled_18098_HCRLateAttachWorkload_1/debug/logs/jitlogs/jitlog)

[2]

LT  stderr 000000000190B200: Unhandled exception while validating object in stack frame in thread Thread-54
LT  stderr 000000000190B200:    O-Slot=000000000190FC80
LT  stderr 000000000190B200:    O-Slot value=000000000190FB38 // <----
LT  stderr 000000000190B200:    PC=000003FFA83DF10F
LT  stderr 000000000190B200:    framesWalked=4
LT  stderr 000000000190B200:    arg0EA=000000000190FC88
LT  stderr 000000000190B200:    walkSP=000000000190FC60
LT  stderr 000000000190B200:    literals=00000000018DEE20
LT  stderr 000000000190B200:    jitInfo=0000000000000000
LT  stderr 000000000190B200:    method=00000000018DEE20 (net/adoptopenjdk/test/hcrAgent/agent/TransformerMakerThread.run()V) (Interpreted)
LT  stderr 000000000190B200:    stack=000000000190E4F8-000000000190FCE0

3XMTHREADINFO      "Thread-54" J9VMThread:0x000000000190B200, omrthread_t:0x000003FF3000C708, java/lang/Thread:0x00000000CA1E0518, state:R, prio=10
3XMJAVALTHREAD            (java/lang/Thread getId:0x4F, isDaemon:true)
3XMJAVALTHRCCL            jdk/internal/loader/ClassLoaders$AppClassLoader(0x000000000DD72680)
3XMTHREADINFO1            (native thread ID:0xA25F7, native priority:0xA, native policy:UNKNOWN, vmstate:CW, vm thread flags:0x00000001)
3XMTHREADINFO2            (native stack address range from:0x000003FFAD8C0000, to:0x000003FFAD900000, size:0x40000)
3XMCPUTIME               CPU usage total: 0.000624939 secs, current category="Application"
3XMHEAPALLOC             Heap bytes allocated since last GC cycle=0 (0x0)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at java/lang/Thread.sleepImpl(Native Method)
4XESTACKTRACE                at java/lang/Thread.sleep(Bytecode PC:2)
4XESTACKTRACE                at java/lang/Thread.sleep(Bytecode PC:2)
4XESTACKTRACE                at net/adoptopenjdk/test/hcrAgent/agent/TransformerMakerThread.sleepNow(Bytecode PC:1)
4XESTACKTRACE                at net/adoptopenjdk/test/hcrAgent/agent/TransformerMakerThread.run(Bytecode PC:87)

> !stackslots 0x0190b200
<190b200> *** BEGIN STACK WALK, flags = 00400001 walkThread = 26,259,968 ***
<190b200> 	ITERATE_O_SLOTS
<190b200> 	RECORD_BYTECODE_PC_OFFSET
<190b200> Initial values: walkSP = 0x000000000190FB90, PC = 0x0000000000000003, literals = 0x0000000000000000, A0 = 0x000000000190FBC8, j2iFrame = 0x0000000000000000, decomp = 0x0000000000000000
<190b200> INL native method frame: bp = 0x000000000190FBB0, sp = 0x000000000190FB90, pc = 0x0000000000000003, cp = 0x0000000000000000, arg0EA = 0x000000000190FBC8, flags = 0x0000000000000000
<190b200> 	Method: java/lang/Thread.sleepImpl(JI)V !j9method 0x0000000001578240
<190b200> 	Using signature mapper
<190b200> 	Arguments starting at 0x000000000190FBC8 for UDATA(0x0000000000000003) slots
<190b200> 		I-Slot: a0[0x000000000190FBC8] = 0x00000000C9E0DE08
<190b200> 		I-Slot: a1[0x000000000190FBC0] = 0x00000000000006D6
<190b200> 		I-Slot: a2[0x000000000190FBB8] = 0x0000000000000000
<190b200> Bytecode frame: bp = 0x000000000190FBE0, sp = 0x000000000190FBD0, pc = 0x000003FF554B980A, cp = 0x0000000001577090, arg0EA = 0x000000000190FBF8, flags = 0x0000000000000000
<190b200> 	Method: java/lang/Thread.sleep(JI)V !j9method 0x0000000001578220
<190b200> 	Bytecode index = 2
<190b200> 	Using local mapper
<190b200> 	Locals starting at 0x000000000190FBF8 for 0x0000000000000003 slots
<190b200> 		I-Slot: a0[0x000000000190FBF8] = 0x000003FFAF47A5E8
<190b200> 		I-Slot: a1[0x000000000190FBF0] = 0x00000000000006D6
<190b200> 		I-Slot: a2[0x000000000190FBE8] = 0x0000000000000000
<190b200> Bytecode frame: bp = 0x000000000190FC10, sp = 0x000000000190FC00, pc = 0x000003FF554B97E2, cp = 0x0000000001577090, arg0EA = 0x000000000190FC20, flags = 0x0000000000000000
<190b200> 	Method: java/lang/Thread.sleep(J)V !j9method 0x0000000001578200
<190b200> 	Bytecode index = 2
<190b200> 	Using local mapper
<190b200> 	Locals starting at 0x000000000190FC20 for 0x0000000000000002 slots
<190b200> 		I-Slot: a0[0x000000000190FC20] = 0x00000007C9E0DDE0
<190b200> 		I-Slot: a1[0x000000000190FC18] = 0x00000000000006D6
<190b200> Bytecode frame: bp = 0x000000000190FC38, sp = 0x000000000190FC28, pc = 0x000003FFA83DF231, cp = 0x00000000018DF2B0, arg0EA = 0x000000000190FC58, flags = 0x0000000000000000
<190b200> 	Method: net/adoptopenjdk/test/hcrAgent/agent/TransformerMakerThread.sleepNow(J)V !j9method 0x00000000018DEE40
<190b200> 	Bytecode index = 1
<190b200> 	Using local mapper
<190b200> 	Locals starting at 0x000000000190FC58 for 0x0000000000000004 slots
<190b200> 		I-Slot: a0[0x000000000190FC58] = 0x00000000C9DDABE8
<190b200> 		I-Slot: a1[0x000000000190FC50] = 0x000006D6C9E0DDC8
<190b200> 		I-Slot: a2[0x000000000190FC48] = 0x00000000000006D6
<190b200> 		I-Slot: t3[0x000000000190FC40] = 0x000007D00190FC88
<190b200> Bytecode frame: bp = 0x000000000190FC70, sp = 0x000000000190FC60, pc = 0x000003FFA83DF10F, cp = 0x00000000018DF2B0, arg0EA = 0x000000000190FC88, flags = 0x0000000000000000
<190b200> 	Method: net/adoptopenjdk/test/hcrAgent/agent/TransformerMakerThread.run()V !j9method 0x00000000018DEE20
<190b200> 	Bytecode index = 87
<190b200> 	Using local mapper
<190b200> 	Locals starting at 0x000000000190FC88 for 0x0000000000000003 slots
<190b200> 		O-Slot: a0[0x000000000190FC88] = 0x00000000CA1E0518
<190b200> 		O-Slot: t1[0x000000000190FC80] = 0x000000000190FB38 // <----
<190b200> 		I-Slot: t2[0x000000000190FC78] = 0x0000000000000000
<190b200> JNI call-in frame: bp = 0x000000000190FCB0, sp = 0x000000000190FC90, pc = 0x000003FFAF43437E, cp = 0x0000000000000000, arg0EA = 0x000000000190FCB0, flags = 0x0000000000000000
<190b200> 	New ELS = 0x0000000000000000
<190b200> JNI native method frame: bp = 0x000000000190FCD8, sp = 0x000000000190FCB8, pc = 0x0000000000000007, cp = 0x0000000000000000, arg0EA = 0x000000000190FCD8, flags = 0x0000000000000000
<190b200> <end of stack>
<190b200> *** END STACK WALK (rc = NONE) ***

@babsingh
Copy link
Contributor

@a7ehuo Can you share your core files (e.g. via Box)?

@a7ehuo
Copy link
Contributor

a7ehuo commented Oct 19, 2023

Can you share your core files (e.g. via Box)?

The logs are updated to a Box folder and have been shared offline. Please check. Thank you!

@babsingh
Copy link
Contributor

Although TransformerMakerThread.run()V is interpreted now, it was compiled previously. For a compiled a method to run interpreted, it goes through decompilation, and on-stack-replacement (OSR) sets up the stack for the interpreted method from the compiled version. The bad object on the stack has most likely come from the JIT during OSR. @a7ehuo Can you check if this analysis holds true?

Source code

  • performDecompile temporarily copies the outgoing arguments to the C stack, then rebuilds the interpreter stack frames, and then pushes the outgoing arguments on the Java stack.

    openj9/runtime/codert_vm/decomp.cpp

    Lines 780 to 818 in b1abbfd

    performDecompile(J9VMThread *currentThread, J9JITDecompileState *decompileState, J9JITDecompilationInfo *decompRecord, J9OSRFrame *osrFrame, UDATA numberOfFrames)
    {
    J9JavaVM *vm = currentThread->javaVM;
    UDATA outgoingArgs[255];
    UDATA outgoingArgCount = decompileState->argCount;
    UDATA inlineDepth = numberOfFrames - 1;
    Trc_Decomp_performDecompile_Entry(currentThread);
    dumpStack(currentThread, "before decompilation");
    if (FALSE == decompRecord->usesOSR) {
    /* Not compiled with OSR - copy stack slots from JIT frame into OSR frame */
    UDATA *jitTempBase = ((UDATA *) (((U_8 *) decompileState->bp) + ((J9JITStackAtlas *) decompileState->metaData->gcStackAtlas)->localBaseOffset)) + decompileState->metaData->tempOffset;
    UDATA *osrTempBase = ((UDATA*)(osrFrame + 1)) + osrFrame->maxStack;
    J9ROMMethod * romMethod = J9_ROM_METHOD_FROM_RAM_METHOD(osrFrame->method);
    UDATA argCount = J9_ARG_COUNT_FROM_ROM_METHOD(romMethod);
    UDATA tempCount = osrFrame->numberOfLocals - argCount;
    UDATA pendingStackHeight = osrFrame->pendingStackHeight;
    /* Decompiling without OSR means old-style single-frame FSD */
    Assert_CodertVM_true(vm->jitConfig->fsdEnabled);
    Assert_CodertVM_true(1 == numberOfFrames);
    /* The pending slots immediately precede the temps in both the OSR frame and the JIT frame, so copy them all at once */
    memcpy(osrTempBase - pendingStackHeight, jitTempBase - pendingStackHeight, (tempCount + pendingStackHeight) * sizeof(UDATA));
    }
    /* Temporarily copy the outgoing arguments to the C stack */
    memcpy(outgoingArgs, decompileState->sp, outgoingArgCount * sizeof(UDATA));
    /* Rebuild the interpreter stack frames */
    buildInlineStackFrames(currentThread, decompileState, decompRecord, inlineDepth, osrFrame);
    /* Push the outgoing arguments onto the java stack */
    currentThread->sp -= outgoingArgCount;
    memcpy(currentThread->sp, outgoingArgs, outgoingArgCount * sizeof(UDATA));
    Trc_Decomp_performDecompile_Exit(currentThread, currentThread->sp, currentThread->literals, currentThread->pc);

  • dumpStack can be enabled for debugging. -verbose:stackwalk or -verbose:stackwalk=<LEVEL> are the cmdline options to enable it. It will allow you to trace the stack arguments as they are moved from the JIT to interpreter during decompilation.

    openj9/runtime/codert_vm/decomp.cpp

    Lines 224 to 230 in b1abbfd

    dumpStack(J9VMThread *currentThread, char const *msg)
    {
    J9JavaVM *vm = currentThread->javaVM;
    if (NULL != vm->verboseStackDump) {
    vm->verboseStackDump(currentThread, msg);
    }
    }

Details from DDR

<190b200> Bytecode frame: bp = 0x000000000190FC70, sp = 0x000000000190FC60, pc = 0x000003FFA83DF10F, cp = 0x00000000018DF2B0, arg0EA = 0x000000000190FC88, flags = 0x0000000000000000
<190b200> 	Method: net/adoptopenjdk/test/hcrAgent/agent/TransformerMakerThread.run()V !j9method 0x00000000018DEE20
<190b200> 	Bytecode index = 87
<190b200> 	Using local mapper
<190b200> 	Locals starting at 0x000000000190FC88 for 0x0000000000000003 slots
<190b200> 		O-Slot: a0[0x000000000190FC88] = 0x00000000CA1E0518
<190b200> 		O-Slot: t1[0x000000000190FC80] = 0x000000000190FB38 // <---- Bad Object
<190b200> 		I-Slot: t2[0x000000000190FC78] = 0x0000000000000000


// The method is .../TransformerMakerThread.run()V

> !j9method 0x00000000018DEE20
J9Method at 0x18dee20 {
  Fields for J9Method:
	0x0: U8* bytecodes = !j9x 0x000003FFA83DF0B8 // "�"
	0x8: struct J9ConstantPool* constantPool = !j9constantpool 0x00000000018DF2B0 (flags = 0x0)
	0x10: void* methodRunAddress = !j9x 0x0000000000000018
	0x18: volatile void* extra = !j9x 0x000003FF8CAEE5FA <-- This method was compiled previously
}
Signature: net/adoptopenjdk/test/hcrAgent/agent/TransformerMakerThread.run()V !bytecodes 0x00000000018DEE20
ROM Method: !j9rommethod 0x000003FFA83DF0A4
Next Method: !j9method 0x00000000018DEE40

// extra != NULL: This method was compiled previously

> !snapformat -t 0x0190b200
Formatting trace using format dat files J9TraceFormat.dat, OMRTraceFormat.dat from /root/openj9-openjdk-jdk21/build/linux-x86_64-server-release/images/jdk/lib
22:21:47.811155302 *0x190B200 j9trc.5              Event       Thread started VMthread = 0x190b200, name = Thread-54, nativeID = 0x3ff3000c708
22:21:47.811158760  0x190B200 j9vm.16              Entry      >runJavaThread
22:21:47.811172464  0x190B200 j9vm.97              Event       Allocated new profiling buffer at 0x3ff18026c90
22:21:47.811176672  0x190B200 j9vm.682             Entry      >WalkStackFrames - walkThread=0x190b200 flags=0x4100000 sp=0x190fa60 a0=0x190fa80 pc=0x5 literals=0x0 els=0x3ffad8fea10 j2i=0x0
22:21:47.811184470  0x190B200 j9vm.683             Exit       <WalkStackFrames - walkThread=0x190b200, rc=0
22:21:47.811192860  0x190B200 j9vm.682             Entry      >WalkStackFrames - walkThread=0x190b200 flags=0x24100000 sp=0x190fa60 a0=0x190fa80 pc=0x5 literals=0x0 els=0x3ffad8fea10 j2i=0x0
22:21:47.811193159  0x190B200 j9vm.683             Exit       <WalkStackFrames - walkThread=0x190b200, rc=0
22:21:47.811197799  0x190B200 j9codertvm(j9jit).59 Event       pending count from stack mapper is 4
22:21:47.811198693  0x190B200 j9codertvm(j9jit).33 Entry      >jitInterpreterPCFromWalkState - Lookup interp PC for 0x3ff8caef946
22:21:47.811199044  0x190B200 j9codertvm(j9jit).37 Exit       <jitInterpreterPCFromWalkState - interp pc = 0x3ffa83df0c3 (stack map)
22:21:47.811201175  0x190B200 j9codertvm(j9jit).51 Event       at current PC
22:21:47.811201573  0x190B200 j9codertvm(j9jit).121 Entry      >DecompileAtCurrentPC entry
22:21:47.811201847  0x190B200 j9vm.682             Entry      >WalkStackFrames - walkThread=0x190b200 flags=0x66240000 sp=0x190fa60 a0=0x190fa80 pc=0x5 literals=0x0 els=0x3ffad8fea10 j2i=0x0
22:21:47.811202401  0x190B200 j9codertvm(j9jit).57 Entry      >decompileMethodFrameIterator
22:21:47.811202785  0x190B200 j9codertvm(j9jit).58 Event       outgoing arg count is 0
22:21:47.811203171  0x190B200 j9codertvm(j9jit).63 Exit       <decompileMethodFrameIterator
22:21:47.811203939  0x190B200 j9codertvm(j9jit).57 Entry      >decompileMethodFrameIterator
22:21:47.811204030  0x190B200 j9codertvm(j9jit).62 Exit       <decompileMethodFrameIterator - finish: previous is BC - set flag
22:21:47.811204124  0x190B200 j9vm.683             Exit       <WalkStackFrames - walkThread=0x190b200, rc=0
22:21:47.811204226  0x190B200 j9codertvm(j9jit).71 Entry      >performDecompile
22:21:47.811205020  0x190B200 j9codertvm(j9jit).73 Event       building bytecode frame at 0x190fc60
22:21:47.811205179  0x190B200 j9codertvm(j9jit).114 Exit       <performDecompile - new SP = 0x190fc40, literals = 0x18dee20, pc = 0x3ffa83df0c3
22:21:47.811205651  0x190B200 j9codertvm(j9jit).122 Exit       <DecompileAtCurrentPC exit

// <performDecompile - ... literals = 0x18dee20: The method was decompiled

@babsingh
Copy link
Contributor

@gacholio Is -verbose:stackwalk=0 the correct option to debug issues during OSR/decompilation?

@gacholio
Copy link
Contributor

Yes, that will dump the stack at various points throughout the decompilation process.

@JasonFengJ9
Copy link
Member Author

JasonFengJ9 commented Oct 20, 2023 •
edited
Loading

JDK11 s390x_linux(ubu22s390x-svl-rt4-1)

java version "11.0.21" 2023-10-17
IBM Semeru Runtime Certified Edition 11.0.21.0 (build 11.0.21+9)
Eclipse OpenJ9 VM 11.0.21.0 (build v0.41.0-release-9c5799b80, JRE 11 Linux s390x-64-Bit Compressed References 20231018_672 (JIT enabled, AOT enabled)
OpenJ9   - 9c5799b80
OMR      - fa7b6ddc7
JCL      - 6f4cc08025 based on jdk-11.0.21+9)

[2023-10-18T22:11:04.784Z] variation: Mode150
[2023-10-18T22:11:04.784Z] JVM_OPTIONS:  -XX:+UseCompressedOops 

[2023-10-18T22:11:42.476Z] LT  22:11:41.901 -   5 JUnit[net.adoptopenjdk.test.util.treemap.TreeMapTest]  Weighting=1 
[2023-10-18T22:11:51.188Z] LT  stderr Unhandled exception
[2023-10-18T22:11:51.188Z] LT  stderr Type=Segmentation error vmState=0x0002000f
[2023-10-18T22:11:51.188Z] LT  stderr J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
[2023-10-18T22:11:51.188Z] LT  stderr Handler1=000003FFA6C4D9A0 Handler2=000003FFA6B333D8 InaccessibleAddress=0000000000000000
[2023-10-18T22:11:51.188Z] LT  stderr gpr0=0000000000000068 gpr1=0000000000000000 gpr2=0000000099669966 gpr3=000003FFA4E0E2A6
[2023-10-18T22:11:51.188Z] LT  stderr gpr4=000003FFA6AFE238 gpr5=0000000001AED410 gpr6=0000000001AED410 gpr7=0000000001AED410
[2023-10-18T22:11:51.188Z] LT  stderr gpr8=000003FFA6AFE238 gpr9=000003FFA4E82890 gpr10=0000000000000002 gpr11=000003FF34002D88
[2023-10-18T22:11:51.188Z] LT  stderr gpr12=000003FFA78B0F90 gpr13=000003FFA73CA010 gpr14=000003FFA4BCC8E0 gpr15=000003FFA6AFDDF0
[2023-10-18T22:11:51.188Z] LT  stderr psw=000003FFA4D3A782 mask=0705000180000000 fpc=00080000 bea=000003FFA4D3A6E0
[2023-10-18T22:11:51.188Z] LT  stderr fpr0 4290639f33208be0 (f: 857771008.000000, d: 4.505015e+12)
[2023-10-18T22:11:51.188Z] LT  stderr fpr1 4033000000000000 (f: 0.000000, d: 1.900000e+01)
[2023-10-18T22:11:51.188Z] LT  stderr fpr2 3e3ab2838ec924cc (f: 2395546880.000000, d: 6.215954e-09)
[2023-10-18T22:11:51.188Z] LT  stderr fpr3 000003ff34000f48 (f: 872419136.000000, d: 2.171233e-311)
[2023-10-18T22:11:51.188Z] LT  stderr fpr4 bfe0139c2733c539 (f: 657704256.000000, d: -5.023938e-01)
[2023-10-18T22:11:51.188Z] LT  stderr fpr5 f6c55708b9b132ce (f: 3115397888.000000, d: -1.343941e+264)
[2023-10-18T22:11:51.188Z] LT  stderr fpr6 402a56ef8ec924cc (f: 2395546880.000000, d: 1.316980e+01)
[2023-10-18T22:11:51.188Z] LT  stderr fpr7 000003ffd5efc36c (f: 3589260032.000000, d: 2.172575e-311)
[2023-10-18T22:11:51.188Z] LT  stderr fpr8 000003ffa6abf000 (f: 2796285952.000000, d: 2.172183e-311)
[2023-10-18T22:11:51.188Z] LT  stderr fpr9 0000000000041000 (f: 266240.000000, d: 1.315400e-318)
[2023-10-18T22:11:51.188Z] LT  stderr fpr10 000003ffa023c9f0 (f: 2686700032.000000, d: 2.172129e-311)
[2023-10-18T22:11:51.188Z] LT  stderr fpr11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-10-18T22:11:51.188Z] LT  stderr fpr12 000003ffa6aff900 (f: 2796550400.000000, d: 2.172183e-311)
[2023-10-18T22:11:51.188Z] LT  stderr fpr13 0000000000000001 (f: 1.000000, d: 4.940656e-324)
[2023-10-18T22:11:51.188Z] LT  stderr fpr14 000003ffa6abf000 (f: 2796285952.000000, d: 2.172183e-311)
[2023-10-18T22:11:51.188Z] LT  stderr fpr15 0000000000876590 (f: 8873360.000000, d: 4.384022e-317)
[2023-10-18T22:11:51.188Z] LT  stderr Module=/home/jenkins/workspace/Test_openjdk11_j9_extended.system_s390x_linux_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9gc29.so
[2023-10-18T22:11:51.188Z] LT  stderr Module_base_address=000003FFA4B80000
[2023-10-18T22:11:51.188Z] LT  stderr Target=2_90_20231018_672 (Linux 5.15.0-86-generic)
[2023-10-18T22:11:51.188Z] LT  stderr CPU=s390x (4 logical CPUs) (0x1f5c26000 RAM)
[2023-10-18T22:11:51.188Z] LT  stderr ----------- Stack Backtrace -----------
[2023-10-18T22:11:51.188Z] LT  stderr Unhandled exception
[2023-10-18T22:11:51.188Z] LT  stderr Type=Segmentation error vmState=0x0002000f
[2023-10-18T22:11:51.188Z] LT  stderr J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
[2023-10-18T22:11:51.188Z] LT  stderr Handler1=000003FFA6C4D610 Handler2=000003FFA6B333D8 InaccessibleAddress=0000000000000000
[2023-10-18T22:11:51.188Z] LT  stderr gpr0=0000000000000068 gpr1=0000000000000000 gpr2=0000000099669966 gpr3=000003FFA4E0E2A6
[2023-10-18T22:11:51.188Z] LT  stderr gpr4=000003FF771FBBC8 gpr5=0000000001B430B0 gpr6=0000000001B430B0 gpr7=0000000001B430B0
[2023-10-18T22:11:51.188Z] LT  stderr gpr8=000003FF771FBBC8 gpr9=000003FFA4E82890 gpr10=0000000000000002 gpr11=000003FF200482F8
[2023-10-18T22:11:51.188Z] LT  stderr gpr12=000000008B35D6F0 gpr13=000003FFA73CA010 gpr14=000003FFA4BCC8E0 gpr15=000003FF771FB780
[2023-10-18T22:11:51.188Z] LT  stderr psw=000003FFA4D3A782 mask=0705000180000000 fpc=00080000 bea=000003FFA4D3A6E0
[2023-10-18T22:11:51.188Z] LT  stderr fpr0 428e087f974e1bfb (f: 2538478592.000000, d: 4.127731e+12)
[2023-10-18T22:11:51.188Z] LT  stderr fpr1 4033000000000000 (f: 0.000000, d: 1.900000e+01)
[2023-10-18T22:11:51.188Z] LT  stderr fpr2 3e3ab283766010a6 (f: 1986007168.000000, d: 6.215954e-09)
[2023-10-18T22:11:51.188Z] LT  stderr fpr3 000003ff10000a08 (f: 268438016.000000, d: 2.170934e-311)
[2023-10-18T22:11:51.188Z] LT  stderr fpr4 bfe019f48b77b72f (f: 2339878656.000000, d: -5.031684e-01)
[2023-10-18T22:11:51.188Z] LT  stderr fpr5 f6c55708b9b132ce (f: 3115397888.000000, d: -1.343941e+264)
[2023-10-18T22:11:51.188Z] LT  stderr fpr6 4029ea282c2f1c38 (f: 741284928.000000, d: 1.295734e+01)
[2023-10-18T22:11:51.188Z] LT  stderr fpr7 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-10-18T22:11:51.188Z] LT  stderr fpr8 0000000001a944e8 (f: 27870440.000000, d: 1.376983e-316)
[2023-10-18T22:11:51.188Z] LT  stderr fpr9 0000000001a944f0 (f: 27870448.000000, d: 1.376983e-316)
[2023-10-18T22:11:51.188Z] LT  stderr fpr10 0000000001a94528 (f: 27870504.000000, d: 1.376986e-316)
[2023-10-18T22:11:51.188Z] LT  stderr fpr11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-10-18T22:11:51.188Z] LT  stderr fpr12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2023-10-18T22:11:51.188Z] LT  stderr fpr13 0000000000000001 (f: 1.000000, d: 4.940656e-324)
[2023-10-18T22:11:51.188Z] LT  stderr fpr14 0000000000000001 (f: 1.000000, d: 4.940656e-324)
[2023-10-18T22:11:51.188Z] LT  stderr fpr15 0000000000876590 (f: 8873360.000000, d: 4.384022e-317)
[2023-10-18T22:11:51.188Z] LT  stderr Module=/home/jenkins/workspace/Test_openjdk11_j9_extended.system_s390x_linux_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9gc29.so
[2023-10-18T22:11:51.188Z] LT  stderr Module_base_address=000003FFA4B80000
[2023-10-18T22:11:51.188Z] LT  stderr Target=2_90_20231018_672 (Linux 5.15.0-86-generic)
[2023-10-18T22:11:51.189Z] LT  stderr CPU=s390x (4 logical CPUs) (0x1f5c26000 RAM)
[2023-10-18T22:11:51.189Z] LT  stderr ----------- Stack Backtrace -----------
[2023-10-18T22:11:51.189Z] LT  stderr _ZN23MM_ScavengerRootScanner11doStackSlotEPP8J9ObjectPvPKv+0xf2 (0x000003FFA4D3A782 [libj9gc29.so+0x1ba782])
[2023-10-18T22:11:51.189Z] LT  stderr vmThreadStackDoOSlotIterator+0x30 (0x000003FFA4BCC8E0 [libj9gc29.so+0x4c8e0])
[2023-10-18T22:11:51.189Z] LT  stderr walkBytecodeFrameSlots+0x246 (0x000003FFA6C95BFE [libj9vm29.so+0x95bfe])
[2023-10-18T22:11:51.189Z] LT  stderr walkStackFrames+0x6cc (0x000003FFA6C9679C [libj9vm29.so+0x9679c])
[2023-10-18T22:11:51.189Z] LT  stderr _ZN28GC_VMThreadStackSlotIterator9scanSlotsEP10J9VMThreadS1_PvPFvP8J9JavaVMPP8J9ObjectS2_P16J9StackWalkStatePKvEbb+0x4e (0x000003FFA4BCCA16 [libj9gc29.so+0x4ca16])
[2023-10-18T22:11:51.189Z] LT  stderr _ZN14MM_RootScanner13scanOneThreadEP18MM_EnvironmentBaseP10J9VMThreadPv+0x14a (0x000003FFA4BC265A [libj9gc29.so+0x4265a])
[2023-10-18T22:11:51.189Z] LT  stderr _ZN14MM_RootScanner11scanThreadsEP18MM_EnvironmentBase+0xe2 (0x000003FFA4BC0F7A [libj9gc29.so+0x40f7a])
[2023-10-18T22:11:51.189Z] LT  stderr _ZN14MM_RootScanner9scanRootsEP18MM_EnvironmentBase+0x64 (0x000003FFA4BC465C [libj9gc29.so+0x4465c])
[2023-10-18T22:11:51.189Z] LT  stderr _ZN12MM_Scavenger24workThreadGarbageCollectEP22MM_EnvironmentStandard+0x34e (0x000003FFA4D33F26 [libj9gc29.so+0x1b3f26])
[2023-10-18T22:11:51.189Z] LT  stderr _ZN21MM_ParallelDispatcher16workerEntryPointEP18MM_EnvironmentBase+0x218 (0x000003FFA4CD4DE0 [libj9gc29.so+0x154de0])
[2023-10-18T22:11:51.189Z] LT  stderr _Z23dispatcher_thread_proc2P14OMRPortLibraryPv+0x138 (0x000003FFA4CD3750 [libj9gc29.so+0x153750])
[2023-10-18T22:11:51.189Z] LT  stderr omrsig_protect+0x366 (0x000003FFA6B34546 [libj9prt29.so+0x34546])
[2023-10-18T22:11:51.189Z] LT  stderr dispatcher_thread_proc+0x60 (0x000003FFA4CD33C8 [libj9gc29.so+0x1533c8])
[2023-10-18T22:11:51.189Z] LT  stderr thread_wrapper+0x114 (0x000003FFA6A85A14 [libj9thr29.so+0x5a14])
[2023-10-18T22:11:51.189Z] LT  stderr  (0x000003FFA7496296 [libc.so.6+0x96296])
[2023-10-18T22:11:51.189Z] LT  stderr  (0x000003FFA750FD5E [libc.so.6+0x10fd5e])
[2023-10-18T22:11:51.189Z] LT  stderr ---------------------------------------
[2023-10-18T22:11:51.189Z] LT  stderr JVMDUMP039I Processing dump event "gpf", detail "" at 2023/10/18 22:11:50 - please wait.
[2023-10-18T22:11:51.189Z] LT  stderr JVMDUMP032I JVM requested System dump using '/home/jenkins/workspace/Test_openjdk11_j9_extended.system_s390x_linux_testList_0/aqa-tests/TKG/output_1697664994449/HCRLateAttachWorkload_previewEnabled_0/20231018-221104-HCRLateAttachWorkload/results/core.20231018.221150.844541.0001.dmp' in response to an event
[2023-10-18T22:11:51.189Z] LT  stderr _ZN23MM_ScavengerRootScanner11doStackSlotEPP8J9ObjectPvPKv+0xf2 (0x000003FFA4D3A782 [libj9gc29.so+0x1ba782])
[2023-10-18T22:11:51.189Z] LT  stderr vmThreadStackDoOSlotIterator+0x30 (0x000003FFA4BCC8E0 [libj9gc29.so+0x4c8e0])
[2023-10-18T22:11:51.189Z] LT  stderr walkBytecodeFrameSlots+0x246 (0x000003FFA6C95BFE [libj9vm29.so+0x95bfe])
[2023-10-18T22:11:51.189Z] LT  stderr walkStackFrames+0x6cc (0x000003FFA6C9679C [libj9vm29.so+0x9679c])
[2023-10-18T22:11:51.189Z] LT  stderr _ZN28GC_VMThreadStackSlotIterator9scanSlotsEP10J9VMThreadS1_PvPFvP8J9JavaVMPP8J9ObjectS2_P16J9StackWalkStatePKvEbb+0x4e (0x000003FFA4BCCA16 [libj9gc29.so+0x4ca16])
[2023-10-18T22:11:51.189Z] LT  stderr _ZN14MM_RootScanner13scanOneThreadEP18MM_EnvironmentBaseP10J9VMThreadPv+0x14a (0x000003FFA4BC265A [libj9gc29.so+0x4265a])
[2023-10-18T22:11:51.189Z] LT  stderr _ZN14MM_RootScanner11scanThreadsEP18MM_EnvironmentBase+0xe2 (0x000003FFA4BC0F7A [libj9gc29.so+0x40f7a])
[2023-10-18T22:11:51.189Z] LT  stderr _ZN14MM_RootScanner9scanRootsEP18MM_EnvironmentBase+0x64 (0x000003FFA4BC465C [libj9gc29.so+0x4465c])
[2023-10-18T22:11:51.189Z] LT  stderr _ZN12MM_Scavenger24workThreadGarbageCollectEP22MM_EnvironmentStandard+0x34e (0x000003FFA4D33F26 [libj9gc29.so+0x1b3f26])
[2023-10-18T22:11:51.189Z] LT  stderr _ZN21MM_ParallelDispatcher3runEP18MM_EnvironmentBaseP7MM_Taskm+0xae (0x000003FFA4CD4ABE [libj9gc29.so+0x154abe])
[2023-10-18T22:11:51.189Z] LT  stderr _ZN12MM_Scavenger8scavengeEP18MM_EnvironmentBase+0xdc (0x000003FFA4D1BBDC [libj9gc29.so+0x19bbdc])
[2023-10-18T22:11:51.189Z] LT  stderr _ZN12MM_Scavenger24mainThreadGarbageCollectEP18MM_EnvironmentBaseP22MM_AllocateDescriptionbb+0x266 (0x000003FFA4D2B30E [libj9gc29.so+0x1ab30e])
[2023-10-18T22:11:51.189Z] LT  stderr _ZN12MM_Scavenger22internalGarbageCollectEP18MM_EnvironmentBaseP17MM_MemorySubSpaceP22MM_AllocateDescription+0xd34 (0x000003FFA4D2C754 [libj9gc29.so+0x1ac754])
[2023-10-18T22:11:51.189Z] LT  stderr _ZN12MM_Collector14garbageCollectEP18MM_EnvironmentBaseP17MM_MemorySubSpaceP22MM_AllocateDescriptionjP28MM_ObjectAllocationInterfaceS3_P20MM_AllocationContext+0xd6 (0x000003FFA4CAAF16 [libj9gc29.so+0x12af16])
[2023-10-18T22:11:51.189Z] LT  stderr _ZN26MM_MemorySubSpaceSemiSpace23allocationRequestFailedEP18MM_EnvironmentBaseP22MM_AllocateDescriptionN17MM_MemorySubSpace14AllocationTypeEP28MM_ObjectAllocationInterfacePS4_S8_+0x320 (0x000003FFA4D96E40 [libj9gc29.so+0x216e40])
[2023-10-18T22:11:51.189Z] LT  stderr _ZN24MM_MemorySubSpaceGeneric11allocateTLHEP18MM_EnvironmentBaseP22MM_AllocateDescriptionP28MM_ObjectAllocationInterfaceP17MM_MemorySubSpaceS7_b+0x418 (0x000003FFA4D79870 [libj9gc29.so+0x1f9870])
[2023-10-18T22:11:51.189Z] LT  stderr _ZN23MM_TLHAllocationSupport7refreshEP18MM_EnvironmentBaseP22MM_AllocateDescriptionb+0x552 (0x000003FFA4CDE072 [libj9gc29.so+0x15e072])
[2023-10-18T22:11:51.189Z] LT  stderr _ZN23MM_TLHAllocationSupport15allocateFromTLHEP18MM_EnvironmentBaseP22MM_AllocateDescriptionb+0x122 (0x000003FFA4CDE2D2 [libj9gc29.so+0x15e2d2])
[2023-10-18T22:11:51.189Z] LT  stderr _ZN25MM_TLHAllocationInterface14allocateObjectEP18MM_EnvironmentBaseP22MM_AllocateDescriptionP14MM_MemorySpaceb+0x144 (0x000003FFA4CDC6DC [libj9gc29.so+0x15c6dc])
[2023-10-18T22:11:51.189Z] LT  stderr _Z21OMR_GC_AllocateObjectP12OMR_VMThreadP25MM_AllocateInitialization+0x25e (0x000003FFA4CE39F6 [libj9gc29.so+0x1639f6])
[2023-10-18T22:11:51.189Z] LT  stderr J9AllocateObject+0x20a (0x000003FFA4BD342A [libj9gc29.so+0x5342a])
[2023-10-18T22:11:51.189Z] LT  stderr old_slow_jitNewObject+0x360 (0x000003FFA5A7B8A8 [libj9jit29.so+0xb7b8a8])
[2023-10-18T22:11:51.189Z] LT  stderr jitNewObject+0x48 (0x000003FFA5A95328 [libj9jit29.so+0xb95328])
[2023-10-18T22:11:51.189Z] LT  stderr ---------------------------------------
[2023-10-18T22:11:51.189Z] LT  stderr JVMDUMP039I Processing dump event "gpf", detail "" at 2023/10/18 22:11:50 - please wait.
[2023-10-18T22:11:53.339Z] STF 22:11:53.062 - Found dump at: /home/jenkins/workspace/Test_openjdk11_j9_extended.system_s390x_linux_testList_0/aqa-tests/TKG/output_1697664994449/HCRLateAttachWorkload_previewEnabled_0/20231018-221104-HCRLateAttachWorkload/results/javacore.20231018.221150.844541.0003.txt
[2023-10-18T22:11:53.339Z] STF 22:11:53.062 - Found dump at: /home/jenkins/workspace/Test_openjdk11_j9_extended.system_s390x_linux_testList_0/aqa-tests/TKG/output_1697664994449/HCRLateAttachWorkload_previewEnabled_0/20231018-221104-HCRLateAttachWorkload/results/core.20231018.221150.844541.0001.dmp
[2023-10-18T22:11:53.339Z] LT  stderr JVMDUMP010I System dump written to /home/jenkins/workspace/Test_openjdk11_j9_extended.system_s390x_linux_testList_0/aqa-tests/TKG/output_1697664994449/HCRLateAttachWorkload_previewEnabled_0/20231018-221104-HCRLateAttachWorkload/results/core.20231018.221150.844541.0001.dmp
[2023-10-18T22:11:53.339Z] LT  stderr JVMDUMP032I JVM requested Java dump using '/home/jenkins/workspace/Test_openjdk11_j9_extended.system_s390x_linux_testList_0/aqa-tests/TKG/output_1697664994449/HCRLateAttachWorkload_previewEnabled_0/20231018-221104-HCRLateAttachWorkload/results/javacore.20231018.221150.844541.0003.txt' in response to an event
[2023-10-18T22:11:55.441Z] AG  stderr javacore file generated - /home/jenkins/workspace/Test_openjdk11_j9_extended.system_s390x_linux_testList_0/aqa-tests/TKG/output_1697664994449/HCRLateAttachWorkload_previewEnabled_0/20231018-221104-HCRLateAttachWorkload/results/javacore.20231018.221150.844541.0003.txt
[2023-10-18T22:11:55.441Z] AG  stderr core file generated - /home/jenkins/workspace/Test_openjdk11_j9_extended.system_s390x_linux_testList_0/aqa-tests/TKG/output_1697664994449/HCRLateAttachWorkload_previewEnabled_0/20231018-221104-HCRLateAttachWorkload/results/core.20231018.221150.844541.0001.dmp
[2023-10-18T22:11:55.441Z] LT  stderr JVMDUMP032I JVM requested System dump using '/home/jenkins/workspace/Test_openjdk11_j9_extended.system_s390x_linux_testList_0/aqa-tests/TKG/output_1697664994449/HCRLateAttachWorkload_previewEnabled_0/20231018-221104-HCRLateAttachWorkload/results/core.20231018.221150.844541.0002.dmp' in response to an event
[2023-10-18T22:11:55.441Z] LT  stderr JVMDUMP012E Error in Java dump: /home/jenkins/workspace/Test_openjdk11_j9_extended.system_s390x_linux_testList_0/aqa-tests/TKG/output_1697664994449/HCRLateAttachWorkload_previewEnabled_0/20231018-221104-HCRLateAttachWorkload/results/javacore.20231018.221150.844541.0003.txt
[2023-10-18T22:11:57.957Z] STF 22:11:55.383 - Found dump at: /home/jenkins/workspace/Test_openjdk11_j9_extended.system_s390x_linux_testList_0/aqa-tests/TKG/output_1697664994449/HCRLateAttachWorkload_previewEnabled_0/20231018-221104-HCRLateAttachWorkload/results/javacore.20231018.221150.844541.0005.txt
[2023-10-18T22:11:57.957Z] STF 22:11:55.383 - Found dump at: /home/jenkins/workspace/Test_openjdk11_j9_extended.system_s390x_linux_testList_0/aqa-tests/TKG/output_1697664994449/HCRLateAttachWorkload_previewEnabled_0/20231018-221104-HCRLateAttachWorkload/results/core.20231018.221150.844541.0002.dmp
[2023-10-18T22:11:57.957Z] STF 22:11:55.383 - Found dump at: /home/jenkins/workspace/Test_openjdk11_j9_extended.system_s390x_linux_testList_0/aqa-tests/TKG/output_1697664994449/HCRLateAttachWorkload_previewEnabled_0/20231018-221104-HCRLateAttachWorkload/results/Snap.20231018.221150.844541.0004.trc
[2023-10-18T22:11:57.957Z] LT  stderr JVMDUMP010I System dump written to /home/jenkins/workspace/Test_openjdk11_j9_extended.system_s390x_linux_testList_0/aqa-tests/TKG/output_1697664994449/HCRLateAttachWorkload_previewEnabled_0/20231018-221104-HCRLateAttachWorkload/results/core.20231018.221150.844541.0002.dmp
[2023-10-18T22:11:57.957Z] LT  stderr JVMDUMP032I JVM requested Java dump using '/home/jenkins/workspace/Test_openjdk11_j9_extended.system_s390x_linux_testList_0/aqa-tests/TKG/output_1697664994449/HCRLateAttachWorkload_previewEnabled_0/20231018-221104-HCRLateAttachWorkload/results/javacore.20231018.221150.844541.0005.txt' in response to an event
[2023-10-18T22:11:57.957Z] LT  stderr JVMDUMP032I JVM requested Snap dump using '/home/jenkins/workspace/Test_openjdk11_j9_extended.system_s390x_linux_testList_0/aqa-tests/TKG/output_1697664994449/HCRLateAttachWorkload_previewEnabled_0/20231018-221104-HCRLateAttachWorkload/results/Snap.20231018.221150.844541.0004.trc' in response to an event
[2023-10-18T22:11:57.957Z] LT  stderr JVMDUMP010I Snap dump written to /home/jenkins/workspace/Test_openjdk11_j9_extended.system_s390x_linux_testList_0/aqa-tests/TKG/output_1697664994449/HCRLateAttachWorkload_previewEnabled_0/20231018-221104-HCRLateAttachWorkload/results/Snap.20231018.221150.844541.0004.trc
[2023-10-18T22:11:57.957Z] LT  stderr JVMDUMP032I JVM requested JIT dump using '/home/jenkins/workspace/Test_openjdk11_j9_extended.system_s390x_linux_testList_0/aqa-tests/TKG/output_1697664994449/HCRLateAttachWorkload_previewEnabled_0/20231018-221104-HCRLateAttachWorkload/results/jitdump.20231018.221150.844541.0006.dmp' in response to an event
[2023-10-18T22:11:57.957Z] LT  stderr JVMDUMP051I JIT dump occurred in 'GC Worker' thread 0x00000000016B5C00
[2023-10-18T22:11:57.957Z] LT  stderr JVMDUMP010I JIT dump written to /home/jenkins/workspace/Test_openjdk11_j9_extended.system_s390x_linux_testList_0/aqa-tests/TKG/output_1697664994449/HCRLateAttachWorkload_previewEnabled_0/20231018-221104-HCRLateAttachWorkload/results/jitdump.20231018.221150.844541.0006.dmp
[2023-10-18T22:11:57.957Z] LT  stderr JVMDUMP013I Processed dump event "gpf", detail "".
[2023-10-18T22:11:57.957Z] LT  stderr 0000000001874100: Unhandled exception while validating object in stack frame in thread Thread-55
[2023-10-18T22:11:57.957Z] LT  stderr 0000000001874100:	O-Slot=0000000001AED410
[2023-10-18T22:11:57.957Z] LT  stderr 0000000001874100:	O-Slot value=0000000001AED2C0
[2023-10-18T22:11:57.957Z] LT  stderr 0000000001874100:	PC=000003FFA032D140
[2023-10-18T22:11:57.957Z] LT  stderr 0000000001874100:	framesWalked=4
[2023-10-18T22:11:57.957Z] LT  stderr 0000000001874100:	arg0EA=0000000001AED418
[2023-10-18T22:11:57.957Z] LT  stderr 0000000001874100:	walkSP=0000000001AED3F0
[2023-10-18T22:11:57.957Z] LT  stderr 0000000001874100:	literals=0000000001858D78
[2023-10-18T22:11:57.957Z] LT  stderr 0000000001874100:	jitInfo=0000000000000000
[2023-10-18T22:11:57.957Z] LT  stderr 0000000001874100:	method=0000000001858D78 (net/adoptopenjdk/test/hcrAgent/agent/TransformerMakerThread.run()V) (Interpreted)
[2023-10-18T22:11:57.957Z] LT  stderr 0000000001874100:	stack=0000000001AE7C58-0000000001AED470
[2023-10-18T22:11:57.957Z] AG  stderr javacore file generated - /home/jenkins/workspace/Test_openjdk11_j9_extended.system_s390x_linux_testList_0/aqa-tests/TKG/output_1697664994449/HCRLateAttachWorkload_previewEnabled_0/20231018-221104-HCRLateAttachWorkload/results/javacore.20231018.221150.844541.0005.txt
[2023-10-18T22:11:57.957Z] AG  stderr core file generated - /home/jenkins/workspace/Test_openjdk11_j9_extended.system_s390x_linux_testList_0/aqa-tests/TKG/output_1697664994449/HCRLateAttachWorkload_previewEnabled_0/20231018-221104-HCRLateAttachWorkload/results/core.20231018.221150.844541.0002.dmp
[2023-10-18T22:11:57.957Z] AG  stderr Snap file generated - /home/jenkins/workspace/Test_openjdk11_j9_extended.system_s390x_linux_testList_0/aqa-tests/TKG/output_1697664994449/HCRLateAttachWorkload_previewEnabled_0/20231018-221104-HCRLateAttachWorkload/results/Snap.20231018.221150.844541.0004.trc
[2023-10-18T22:11:57.957Z] AG  stderr Exception in thread "main" java.net.SocketException: Broken pipe (Write failed)
[2023-10-18T22:11:57.957Z] AG  stderr 	at java.base/java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:110)
[2023-10-18T22:11:57.957Z] AG  stderr 	at java.base/java.net.SocketOutputStream.write(SocketOutputStream.java:129)
[2023-10-18T22:11:57.957Z] AG  stderr 	at java.base/openj9.internal.tools.attach.target.AttachmentConnection.streamSend(AttachmentConnection.java:105)
[2023-10-18T22:11:57.957Z] AG  stderr 	at jdk.attach/com.ibm.tools.attach.attacher.OpenJ9VirtualMachine.detach(OpenJ9VirtualMachine.java:206)
[2023-10-18T22:11:57.957Z] AG  stderr 	at net.adoptopenjdk.test.hcrAgent.agent.Attacher.main(Attacher.java:85)
[2023-10-18T22:11:57.957Z] STF 22:11:55.797 - **FAILED** Process AG  ended with exit code (1) and not the expected exit code/s (0)
[2023-10-18T22:11:57.957Z] STF 22:11:55.797 - Monitoring Report Summary:
[2023-10-18T22:11:57.957Z] STF 22:11:55.797 -   o Process AG  has crashed unexpectedly
[2023-10-18T22:11:57.957Z] STF 22:11:55.797 -   o Process LT  ended with return code 255
[2023-10-18T22:11:57.957Z] STF 22:11:55.800 - Killing processes: AG  LT 
[2023-10-18T22:11:57.957Z] STF 22:11:55.800 -   o Process AG  pid 844543 is not running
[2023-10-18T22:11:57.957Z] STF 22:11:55.800 -   o Process LT  pid 844541 is not running
[2023-10-18T22:11:57.958Z] **FAILED** at step 3 (Wait for Agent to complete). Expected return value=0 Actual=1 at /home/jenkins/workspace/Test_openjdk11_j9_extended.system_s390x_linux_testList_0/aqa-tests/TKG/../TKG/output_1697664994449/HCRLateAttachWorkload_previewEnabled_0/20231018-221104-HCRLateAttachWorkload/execute.pl line 157.
[2023-10-18T22:11:57.958Z] STF 22:11:55.988 - **FAILED** execute script failed. Expected return value=0 Actual=1
[2023-10-18T22:11:57.958Z] STF 22:11:55.988 - 
[2023-10-18T22:11:57.958Z] STF 22:11:55.988 - ====================   T E A R D O W N   ====================
[2023-10-18T22:11:57.958Z] STF 22:11:55.988 - Running teardown: perl /home/jenkins/workspace/Test_openjdk11_j9_extended.system_s390x_linux_testList_0/aqa-tests/TKG/../TKG/output_1697664994449/HCRLateAttachWorkload_previewEnabled_0/20231018-221104-HCRLateAttachWorkload/tearDown.pl
[2023-10-18T22:11:57.958Z] STF 22:11:56.090 - TEARDOWN stage completed
[2023-10-18T22:11:57.958Z] STF 22:11:56.103 - 
[2023-10-18T22:11:57.958Z] STF 22:11:56.107 - =====================   R E S U L T S   =====================
[2023-10-18T22:11:57.958Z] STF 22:11:56.107 - Stage results:
[2023-10-18T22:11:57.958Z] STF 22:11:56.107 -   setUp:     pass
[2023-10-18T22:11:57.958Z] STF 22:11:56.107 -   execute:  *fail*
[2023-10-18T22:11:57.958Z] STF 22:11:56.107 -   teardown:  pass
[2023-10-18T22:11:57.958Z] STF 22:11:56.107 - 
[2023-10-18T22:11:57.958Z] STF 22:11:56.107 - Overall result: **FAILED**
[2023-10-18T22:11:57.958Z] -----------------------------------
[2023-10-18T22:11:57.958Z] HCRLateAttachWorkload_previewEnabled_0_FAILED

50x grinder - 2/50 failed

@JasonFengJ9
Copy link
Member Author

JasonFengJ9 commented Oct 20, 2023 •
edited
Loading

JDK17 s390x_linux(ubu20s390x-svl-rt3-1)

[2023-10-18T23:23:54.659Z] variation: Mode150
[2023-10-18T23:23:54.659Z] JVM_OPTIONS:  -XX:+UseCompressedOops 

[2023-10-18T23:24:15.685Z] LT  Starting thread 9
[2023-10-18T23:24:15.685Z] LT  Threads are all initialised.
[2023-10-18T23:24:15.685Z] LT  Entering the loop that will replace dead threads with live ones until we run out of time.
[2023-10-18T23:24:40.087Z] LT  stderr Unhandled exception
[2023-10-18T23:24:40.087Z] LT  stderr Type=Segmentation error vmState=0x0002000f

[2023-10-18T23:24:43.134Z] HCRLateAttachWorkload_previewEnabled_0_FAILED

50x grinder - 1/50 failed

@hzongaro
Copy link
Member

Annabelle @a7ehuo, I'm wondering whether it's possible this could be related to the problems that are fixed by pull requests #14074 and #18315.

@a7ehuo
Copy link
Contributor

a7ehuo commented Oct 24, 2023

@a7ehuo Can you check if this analysis holds true?

Thanks @babsingh for narrowing down the issue! This one is on my TODO list. I'll get to it as soon as possible.

I'm wondering whether it's possible this could be related to the problems that are fixed by pull requests #14074 and #18315.

Interesting! Thanks for pointing it out @hzongaro ! I'll take a look along with the information from @babsingh

@hzongaro
Copy link
Member

Moving this to 0.44 while @a7ehuo continues to investigate.

@pshipton pshipton mentioned this issue Nov 22, 2023
Closed
@a7ehuo
Copy link
Contributor

a7ehuo commented Nov 29, 2023 •
edited
Loading

After many runs, I reproduced the issue once with an old build from September with OSR and EA traces. With help from @hzongaro, we concluded this is the same issue that has been solved by PR #14074.

openjdk version "17.0.9-internal" 2023-10-17
OpenJDK Runtime Environment (build 17.0.9-internal+0-adhoc.root.openj9-openjdk-jdk17)
Eclipse OpenJ9 VM (build master-1ae04ef63, JRE 17 Linux s390x-64-Bit Compressed References 20231104_000000 (JIT enabled, AOT enabled)
OpenJ9   - 1ae04ef63
OMR      - f8fecbefe
JCL      - fdaefd4721b based on jdk-17.0.9+6)

Below is the detailed analysis from the reproduced case:

The VM trace shows the PC is 0x3ff822dba06 when jitInduceOSRAtCurrentPCAndRecompile is called [1].
0x3ff822dba06 is mapped to block_1871 [2] that contains the helper call jitInduceOSRAtCurrentPCAndRecompile.
The exception out block for block_1871 is block_9 which falls through to block_10 [3]

The invalid object 0x21E7520 [4] is stored on scratchBufferOffset at offset 4 and 16 [5].

block_10 (prepareForOSR) shows [3]:
OSRScratchBuffer offset 4 : aRegLoad GPR9 #402
OSRScratchBuffer offset 16: aRegLoad aRegLoad GPR11 #403

block_1871 (jitInduceOSRAtCurrentPCAndRecompile) shows [2]:
Both #402 and #403 are objects from #4569. #4569 is a stack allocated object of StringTransformer by EA [6]. Neither #403 nor #402 is heapified on the path from block_2 to block_1871.

block_2: n49599n loadaddr  <temp slot 162>[#4569  Auto] 
     |   
block_2748
     |    
block_1871: jitInduceOSRAtCurrentPCAndRecompile 
     | exception out
block_9
     |
block_10: prepareForOSR

Henry found #402 and #403 missing from the call to eaEscapeHelper that’s added to block_1871 by PreEscapeAnalysis [7], and that’s what’s supposed to trigger the heapification code to be generated. Hence this is the same issue solved by PR #14074.

[1]

00:16:25.588709833  0x00000000021e9d00 j9codertvm(j9jit).131 Entry      >jitInduceOSRAtCurrentPCAndRecompile entry: jitReturnAddress = 0x3ff822dba06 parmCount 0

[2]

n27693n   BBStart <block_1871> (freq 5) (cold)                                                [     0x3ff612bbfe0] bci=[-1,11,58] rc=0 vc=16082 vn=- li=- udi=- nc=0
n27716n   lstore  <pending push temp 2>[#404  Auto] [flags 0x4 0x800 ]                        [     0x3ff612bc710] bci=[-1,8,58] rc=0 vc=16082 vn=- li=1002 udi=712 nc=1
n27715n     lload  <temp slot 17>[#1255  Auto] [flags 0x4 0x0 ] (cannotOverflow )             [     0x3ff612bc6c0] bci=[-1,8,58] rc=1 vc=16082 vn=- li=- udi=2327 nc=0 flg=0x1000
n27718n   astore  <pending push temp 0>[#402  Auto] [flags 0x7 0x800 ] (X!=0 )                [     0x3ff612bc7b0] bci=[-1,8,58] rc=0 vc=16082 vn=- li=1003 udi=713 nc=1 flg=0x4
n27717n     loadaddr  <temp slot 162>[#4569  Auto] [flags 0x60010008 0x0 ] (X!=0 cannotOverflow nodePointsToNonNull cannotTrackLocalUses escapesInColdBlock )  [     0x3ff612bc760] bci=[-1,8,58] rc=1 vc=16082 vn=- li=- udi=2328 nc=0 flg=0x9004
n27720n   astore  <pending push temp 1>[#403  Auto] [flags 0x7 0x800 ] (X!=0 )                [     0x3ff612bc850] bci=[-1,8,58] rc=0 vc=16082 vn=- li=1004 udi=714 nc=1 flg=0x4
n27719n     loadaddr  <temp slot 162>[#4569  Auto] [flags 0x60010008 0x0 ] (X!=0 cannotOverflow nodePointsToNonNull cannotTrackLocalUses escapesInColdBlock )  [     0x3ff612bc800] bci=[-1,8,58] rc=1 vc=16082 vn=- li=- udi=2329 nc=0 flg=0x9004
n27700n   astore  st<auto slot 1>[#412  Auto] [flags 0x7 0x0 ] (X==0 StoredValueIsIrrelevant )  [     0x3ff612bc210] bci=[-1,11,58] rc=0 vc=16082 vn=- li=- udi=715 nc=1 flg=0x20002
n27699n     aconst NULL (X==0 X>=0 X<=0 )                                                     [     0x3ff612bc1c0] bci=[-1,11,58] rc=1 vc=16082 vn=- li=- udi=- nc=0 flg=0x302
n27702n   istore  <pending push temp 1>[#424  Auto] [flags 0x3 0x800 ] (StoredValueIsIrrelevant )  [     0x3ff612bc2b0] bci=[-1,11,58] rc=0 vc=16082 vn=- li=- udi=716 nc=1 flg=0x20000
n27701n     iconst 0xdead (X!=0 X>=0 )                                                        [     0x3ff612bc260] bci=[-1,11,58] rc=1 vc=16082 vn=- li=- udi=- nc=0 flg=0x104
n27710n   istore  <pending push temp 0>[#459  Auto] [flags 0x3 0x800 ] (StoredValueIsIrrelevant )  [     0x3ff612bc530] bci=[-1,11,58] rc=0 vc=16082 vn=- li=- udi=717 nc=1 flg=0x20000
n27709n     iconst 0xdead (X!=0 X>=0 )                                                        [     0x3ff612bc4e0] bci=[-1,11,58] rc=1 vc=16082 vn=- li=- udi=- nc=0 flg=0x104
n27696n   treetop ()                                                                          [     0x3ff612bc0d0] bci=[-1,11,58] rc=0 vc=16082 vn=- li=- udi=- nc=1 flg=0x8
n27695n     call  jitInduceOSRAtCurrentPCAndRecompile[#90  helper Method] [flags 0x400 0x0 ] ()  [     0x3ff612bc080] bci=[-1,11,58] rc=1 vc=16082 vn=- li=- udi=- nc=0 flg=0x8
n27698n   athrow  jitThrowException[#24  helper Method] [flags 0x400 0x0 ] (Unsigned ThrowInsertedByOSR )  [     0x3ff612bc170] bci=[-1,11,58] rc=0 vc=16082 vn=- li=- udi=- nc=1 flg=0x4008
n27697n     aload  ExceptionMeta[#326  MethodMeta +64] [flags 0x207 0x0 ] ()                  [     0x3ff612bc120] bci=[-1,11,58] rc=1 vc=16082 vn=- li=- udi=- nc=0 flg=0x8
n27694n   BBEnd </block_1871> (cold) =====                                                    [     0x3ff612bc030] bci=[-1,11,58] rc=0 vc=16082 vn=- li=- udi=- nc=0

[3]

n400n     BBStart <block_10> (freq 0) (cold)                                                  [     0x3ff61396cc0] bci=[-1,0,58] rc=0 vc=14 vn=- li=10 udi=- nc=1
n60455n     GlRegDeps ()                                                                      [     0x3fedf93c080] bci=[-1,0,58] rc=1 vc=14 vn=- li=10 udi=- nc=5 flg=0x20
n60456n       iRegLoad GPR6   <pending push temp 0>[#459  Auto] [flags 0x3 0x800 ] (cannotOverflow SeenRealReference )  [     0x3fedf93c0d0] bci=[-1,0,58] rc=2 vc=14 vn=- li=10 udi=- nc=0 flg=0x9000
n60457n       iRegLoad GPR7   <pending push temp 1>[#424  Auto] [flags 0x3 0x800 ] (cannotOverflow SeenRealReference )  [     0x3fedf93c120] bci=[-1,0,58] rc=2 vc=14 vn=- li=10 udi=- nc=0 flg=0x9000
n60458n       lRegLoad GPR8   <pending push temp 2>[#404  Auto] [flags 0x4 0x800 ] (cannotOverflow SeenRealReference )  [     0x3fedf93c170] bci=[-1,0,58] rc=2 vc=14 vn=- li=10 udi=- nc=0 flg=0x9000
n60459n       aRegLoad GPR9   <pending push temp 0>[#402  Auto] [flags 0x4007 0x800 ] (SeenRealReference )  [     0x3fedf93c1c0] bci=[-1,0,58] rc=2 vc=14 vn=- li=10 udi=- nc=0 flg=0x8000
n60460n       aRegLoad GPR11   <pending push temp 1>[#403  Auto] [flags 0x4007 0x800 ] (SeenRealReference )  [     0x3fedf93c210] bci=[-1,0,58] rc=2 vc=14 vn=- li=10 udi=- nc=0 flg=0x8000
n441n     tabort  transactionAbort[#119  helper Method] [flags 0x400 0x0 ] (X==0 X>=0 X<=0 )  [     0x3ff61397990] bci=[-1,0,58] rc=0 vc=14 vn=- li=10 udi=- nc=0 flg=0x302
n62688n   istorei  <generic int shadow>[#4657  Shadow] [flags 0x603 0x0 ]                     [     0x3fedf5f7a80] bci=[-1,0,58] rc=0 vc=14 vn=- li=10 udi=- nc=2
n62687n     aladd                                                                             [     0x3fedf5f7a30] bci=[-1,0,58] rc=1 vc=14 vn=- li=10 udi=- nc=2
n62684n       aload  OSRScratchBuffer[#352  MethodMeta +2464] [flags 0x10207 0x0 ]            [     0x3fedf5f7940] bci=[-1,0,58] rc=7 vc=14 vn=- li=10 udi=- nc=0
n62686n       i2l                                                                             [     0x3fedf5f79e0] bci=[-1,0,58] rc=1 vc=14 vn=- li=10 udi=- nc=1
n62685n         iconst 0 (X==0 X>=0 X<=0 )                                                    [     0x3fedf5f7990] bci=[-1,0,58] rc=1 vc=14 vn=- li=10 udi=- nc=0 flg=0x302
n60456n     ==>iRegLoad
n62692n   astorei  <generic int shadow>[#4658  Shadow] [flags 0x603 0x0 ]                     [     0x3fedf5f7bc0] bci=[-1,0,58] rc=0 vc=14 vn=- li=10 udi=- nc=2
n62691n     aladd                                                                             [     0x3fedf5f7b70] bci=[-1,0,58] rc=1 vc=14 vn=- li=10 udi=- nc=2
n62684n       ==>aload
n62690n       i2l                                                                             [     0x3fedf5f7b20] bci=[-1,0,58] rc=1 vc=14 vn=- li=10 udi=- nc=1
n62689n         iconst 4 (X!=0 X>=0 )                                                         [     0x3fedf5f7ad0] bci=[-1,0,58] rc=1 vc=14 vn=- li=10 udi=- nc=0 flg=0x104
n60459n     ==>aRegLoad
n62696n   istorei  <generic int shadow>[#4659  Shadow] [flags 0x603 0x0 ]                     [     0x3fedf5f7d00] bci=[-1,0,58] rc=0 vc=14 vn=- li=10 udi=- nc=2
n62695n     aladd                                                                             [     0x3fedf5f7cb0] bci=[-1,0,58] rc=1 vc=14 vn=- li=10 udi=- nc=2
n62684n       ==>aload
n62694n       i2l                                                                             [     0x3fedf5f7c60] bci=[-1,0,58] rc=1 vc=14 vn=- li=10 udi=- nc=1
n62693n         iconst 12 (X!=0 X>=0 )                                                        [     0x3fedf5f7c10] bci=[-1,0,58] rc=1 vc=14 vn=- li=10 udi=- nc=0 flg=0x104
n60457n     ==>iRegLoad
n62700n   astorei  <generic int shadow>[#4660  Shadow] [flags 0x603 0x0 ]                     [     0x3fedf5f7e40] bci=[-1,0,58] rc=0 vc=14 vn=- li=10 udi=- nc=2
n62699n     aladd                                                                             [     0x3fedf5f7df0] bci=[-1,0,58] rc=1 vc=14 vn=- li=10 udi=- nc=2
n62684n       ==>aload
n62698n       i2l                                                                             [     0x3fedf5f7da0] bci=[-1,0,58] rc=1 vc=14 vn=- li=10 udi=- nc=1
n62697n         iconst 16 (X!=0 X>=0 )                                                        [     0x3fedf5f7d50] bci=[-1,0,58] rc=1 vc=14 vn=- li=10 udi=- nc=0 flg=0x104
n60460n     ==>aRegLoad
n62704n   astorei  <generic int shadow>[#4661  Shadow] [flags 0x603 0x0 ]                     [     0x3fedf5f7f80] bci=[-1,0,58] rc=0 vc=14 vn=- li=10 udi=- nc=2
n62703n     aladd                                                                             [     0x3fedf5f7f30] bci=[-1,0,58] rc=1 vc=14 vn=- li=10 udi=- nc=2
n62684n       ==>aload
n62702n       i2l                                                                             [     0x3fedf5f7ee0] bci=[-1,0,58] rc=1 vc=14 vn=- li=10 udi=- nc=1
n62701n         iconst 24 (X!=0 X>=0 )                                                        [     0x3fedf5f7e90] bci=[-1,0,58] rc=1 vc=14 vn=- li=10 udi=- nc=0 flg=0x104
n418n       aconst NULL (X==0 )                                                               [     0x3ff61397260] bci=[-1,0,58] rc=1 vc=14 vn=- li=10 udi=- nc=0 flg=0x2
n62708n   lstorei  <generic int shadow>[#4662  Shadow] [flags 0x603 0x0 ]                     [     0x3fedf5f80c0] bci=[-1,0,58] rc=0 vc=14 vn=- li=10 udi=- nc=2
n62707n     aladd                                                                             [     0x3fedf5f8070] bci=[-1,0,58] rc=1 vc=14 vn=- li=10 udi=- nc=2
n62684n       ==>aload
n62706n       i2l                                                                             [     0x3fedf5f8020] bci=[-1,0,58] rc=1 vc=14 vn=- li=10 udi=- nc=1
n62705n         iconst 32 (X!=0 X>=0 )                                                        [     0x3fedf5f7fd0] bci=[-1,0,58] rc=1 vc=14 vn=- li=10 udi=- nc=0 flg=0x104
n60458n     ==>lRegLoad
n62712n   istorei  <generic int shadow>[#4663  Shadow] [flags 0x603 0x0 ]                     [     0x3fedf5f8200] bci=[-1,0,58] rc=0 vc=14 vn=- li=10 udi=- nc=2
n62711n     aladd                                                                             [     0x3fedf5f81b0] bci=[-1,0,58] rc=1 vc=14 vn=- li=10 udi=- nc=2
n62684n       ==>aload
n62710n       i2l                                                                             [     0x3fedf5f8160] bci=[-1,0,58] rc=1 vc=14 vn=- li=10 udi=- nc=1
n62709n         iconst 40 (X!=0 X>=0 )                                                        [     0x3fedf5f8110] bci=[-1,0,58] rc=1 vc=14 vn=- li=10 udi=- nc=0 flg=0x104
n424n       iconst 0 (X==0 X>=0 X<=0 )                                                        [     0x3ff61397440] bci=[-1,0,58] rc=1 vc=14 vn=- li=10 udi=- nc=0 flg=0x302
n62717n   astorei  <generic int shadow>[#4664  Shadow] [flags 0x603 0x0 ]                     [     0x3fedf5f8390] bci=[-1,0,58] rc=0 vc=14 vn=- li=10 udi=- nc=2
n62716n     aladd                                                                             [     0x3fedf5f8340] bci=[-1,0,58] rc=1 vc=14 vn=- li=10 udi=- nc=2
n62682n       aload  OSRBuffer[#351  MethodMeta +2448] [flags 0x10207 0x0 ]                   [     0x3fedf5f78a0] bci=[-1,0,58] rc=4 vc=14 vn=- li=10 udi=- nc=0
n62715n       i2l                                                                             [     0x3fedf5f82f0] bci=[-1,0,58] rc=1 vc=14 vn=- li=10 udi=- nc=1
n62714n         iadd                                                                          [     0x3fedf5f82a0] bci=[-1,0,58] rc=1 vc=14 vn=- li=10 udi=- nc=2
n62683n           iload  osrFrameIndex[#353  MethodMeta +348] [flags 0x203 0x0 ]              [     0x3fedf5f78f0] bci=[-1,0,58] rc=5 vc=14 vn=- li=10 udi=- nc=0
n62713n           iconst 64 (X!=0 X>=0 )                                                      [     0x3fedf5f8250] bci=[-1,0,58] rc=1 vc=14 vn=- li=10 udi=- nc=0 flg=0x104
n427n       aconst NULL (X==0 )                                                               [     0x3ff61397530] bci=[-1,0,58] rc=1 vc=14 vn=- li=10 udi=- nc=0 flg=0x2
n62722n   astorei  <generic int shadow>[#4665  Shadow] [flags 0x603 0x0 ]                     [     0x3fedf5f8520] bci=[-1,0,58] rc=0 vc=14 vn=- li=10 udi=- nc=2
n62721n     aladd                                                                             [     0x3fedf5f84d0] bci=[-1,0,58] rc=1 vc=14 vn=- li=10 udi=- nc=2
n62682n       ==>aload
n62720n       i2l                                                                             [     0x3fedf5f8480] bci=[-1,0,58] rc=1 vc=14 vn=- li=10 udi=- nc=1
n62719n         iadd                                                                          [     0x3fedf5f8430] bci=[-1,0,58] rc=1 vc=14 vn=- li=10 udi=- nc=2
n62683n           ==>iload
n62718n           iconst 120 (X!=0 X>=0 )                                                     [     0x3fedf5f83e0] bci=[-1,0,58] rc=1 vc=14 vn=- li=10 udi=- nc=0 flg=0x104
n430n       aload  this<'this' parm Lnet/adoptopenjdk/test/hcrAgent/agent/TransformerMakerThread;>[#397  Parm] [flags 0xc0000107 0x0 ] ()  [     0x3ff61397620] bci=[-1,0,58] rc=1 vc=14 vn=- li=10 udi=- nc=0 flg=0x8
n62727n   astorei  <generic int shadow>[#4666  Shadow] [flags 0x603 0x0 ]                     [     0x3fedf5f86b0] bci=[-1,0,58] rc=0 vc=14 vn=- li=10 udi=- nc=2
n62726n     aladd                                                                             [     0x3fedf5f8660] bci=[-1,0,58] rc=1 vc=14 vn=- li=10 udi=- nc=2
n62682n       ==>aload
n62725n       i2l                                                                             [     0x3fedf5f8610] bci=[-1,0,58] rc=1 vc=14 vn=- li=10 udi=- nc=1
n62724n         iadd                                                                          [     0x3fedf5f85c0] bci=[-1,0,58] rc=1 vc=14 vn=- li=10 udi=- nc=2
n62683n           ==>iload
n62723n           iconst 112 (X!=0 X>=0 )                                                     [     0x3fedf5f8570] bci=[-1,0,58] rc=1 vc=14 vn=- li=10 udi=- nc=0 flg=0x104
n433n       aload  st<auto slot 1>[#412  Auto] [flags 0x4007 0x0 ]                            [     0x3ff61397710] bci=[-1,0,58] rc=1 vc=14 vn=- li=10 udi=- nc=0
n62732n   astorei  <generic int shadow>[#4667  Shadow] [flags 0x603 0x0 ]                     [     0x3fedf5f8840] bci=[-1,0,58] rc=0 vc=14 vn=- li=10 udi=- nc=2
n62731n     aladd                                                                             [     0x3fedf5f87f0] bci=[-1,0,58] rc=1 vc=14 vn=- li=10 udi=- nc=2
n62682n       ==>aload
n62730n       i2l                                                                             [     0x3fedf5f87a0] bci=[-1,0,58] rc=1 vc=14 vn=- li=10 udi=- nc=1
n62729n         iadd                                                                          [     0x3fedf5f8750] bci=[-1,0,58] rc=1 vc=14 vn=- li=10 udi=- nc=2
n62683n           ==>iload
n62728n           iconst 104 (X!=0 X>=0 )                                                     [     0x3fedf5f8700] bci=[-1,0,58] rc=1 vc=14 vn=- li=10 udi=- nc=0 flg=0x104
n436n       aconst NULL (X==0 )                                                               [     0x3ff61397800] bci=[-1,0,58] rc=1 vc=14 vn=- li=10 udi=- nc=0 flg=0x2
n440n     treetop ()                                                                          [     0x3ff61397940] bci=[-1,0,58] rc=0 vc=14 vn=- li=10 udi=- nc=1 flg=0x8
n439n       call  prepareForOSR[#55  helper Method] [flags 0x400 0x0 ] ()                     [     0x3ff613978f0] bci=[-1,0,58] rc=1 vc=14 vn=- li=10 udi=- nc=3 flg=0x28
n404n         loadaddr  vmThread[#487  MethodMeta] [flags 0x200 0x0 ] ()                      [     0x3ff61396e00] bci=[-1,0,58] rc=1 vc=14 vn=- li=10 udi=- nc=0 flg=0x8
n405n         iconst -1 (X!=0 X<=0 )                                                          [     0x3ff61396e50] bci=[-1,0,58] rc=1 vc=14 vn=- li=10 udi=- nc=0 flg=0x204
n62733n       iconst 0x70009 (X!=0 X>=0 )                                                     [     0x3fedf5f8890] bci=[-1,0,58] rc=1 vc=14 vn=- li=10 udi=- nc=0 flg=0x104
n62736n   istore  osrFrameIndex[#353  MethodMeta +348] [flags 0x203 0x0 ]                     [     0x3fedf5f8980] bci=[-1,0,58] rc=0 vc=14 vn=- li=10 udi=- nc=1
n62735n     iadd                                                                              [     0x3fedf5f8930] bci=[-1,0,58] rc=1 vc=14 vn=- li=10 udi=- nc=2
n62683n       ==>iload
n62734n       iconst 128 (X!=0 X>=0 )                                                         [     0x3fedf5f88e0] bci=[-1,0,58] rc=1 vc=14 vn=- li=10 udi=- nc=0 flg=0x104
n443n     igoto ()                                                                            [     0x3ff61397a30] bci=[-1,0,58] rc=0 vc=14 vn=- li=10 udi=- nc=1 flg=0x8
n442n       aload  osrReturnAddress[#354  MethodMeta +2456] [flags 0x10207 0x0 ] ()           [     0x3ff613979e0] bci=[-1,0,58] rc=1 vc=14 vn=- li=10 udi=- nc=0 flg=0x8
n401n     BBEnd </block_10> (cold) =====                                                      [     0x3ff61396d10] bci=[-1,0,58] rc=0 vc=14 vn=- li=10 udi=- nc=0

[4]

LT  stderr 00000000021E9D00: Unhandled exception while validating object in stack frame in thread Thread-59
LT  stderr 00000000021E9D00:    O-Slot=00000000021E7680
LT  stderr 00000000021E9D00:    O-Slot value=00000000021E7520
LT  stderr 00000000021E9D00:    PC=000003FFA0300BB0
LT  stderr 00000000021E9D00:    framesWalked=4
LT  stderr 00000000021E9D00:    arg0EA=00000000021E7688
LT  stderr 00000000021E9D00:    walkSP=00000000021E7660
LT  stderr 00000000021E9D00:    literals=00000000021B9BA8
LT  stderr 00000000021E9D00:    jitInfo=0000000000000000
LT  stderr 00000000021E9D00:    method=00000000021B9BA8 (net/adoptopenjdk/test/hcrAgent/agent/TransformerMakerThread.run()V) (Interpreted)
LT  stderr 00000000021E9D00:    stack=00000000021E5EF8-00000000021E76E0

[5]

#OSR:  21e9d00 prepareForOSR at 000003FF822DBA06 (startPC 000003FF822D73FA +17932) at -1:b numSharingSyms:7 totalSlots:9 vmThread=00000000021E9D00
#OSRd: 21E9D00   Jitted body:    net/adoptopenjdk/test/hcrAgent/agent/TransformerMakerThread.run()V
#OSRd: 21E9D00   osrBuffer=000003FEFC02EE18 osrFrame=000003FEFC02EE28, osrReturnAddress=000003FF9F9175D0 osrScratchBuffer=000003FEFC02EEA8 osrJittedFrameCopy=000003FEFC02EEE8
#OSRd: 21E9D00     OSRBuffer: numberOfFrames=1 jitPC=000003FF822DBA06
#OSRd: 21E9D00     OSRFrame: j9method=00000000021B9BA8 bytecodePC=a0300b8b numberOfLocals=3 maxStack=6 pendingStackHeight=4 monitorEnterRecords=0000000000000000
#OSRd: 21E9D00       stack  3: 0000000000000000
#OSRd: 21E9D00       stack  2: 0000000000000000
#OSRd: 21E9D00       stack  1: 0000000000000000
#OSRd: 21E9D00       stack  0: 0000000000000000
#OSRd: 21E9D00       local  2: 0000000000000000
#OSRd: 21E9D00       local  1: 00000000CAB775C0
#OSRd: 21E9D00       local  0: 00000000CA4D5330
#OSRd: 21E9D00   47 mappings
#OSRd: 21E9D00     Skip mapping @17926 <= 17932 with 3 symbols
#OSRd: 21E9D00   Found mapping @17972 > 17932
#OSRd: 21E9D00   Copying 3 symbols
#OSRd: 21E9D00     Symbol #0 osrFrameDataOffset=96 scratchBufferOffset=4 size=8 data:0x21e7520
#OSRd: 21E9D00     Symbol #1 osrFrameDataOffset=88 scratchBufferOffset=16 size=8 data:0x21e7520
#OSRd: 21E9D00     Symbol #2 osrFrameDataOffset=72 scratchBufferOffset=32 size=8 data:0x9c83547c
#OSRd: 21E9D00   prepareForOSR returning

[6]

"Trees after Escape Analysis"

n49603n   astorei  <vft-symbol>[#323  Shadow] [flags 0x18607 0x0 ]                            [     0x3fee8768070] bci=[-1,0,58] rc=0 vc=0 vn=- li=- udi=- nc=2
n49599n     loadaddr  <temp slot 162>[#4569  Auto] [flags 0x60010008 0x0 ] (highWordZero Unsigned X!=0 cannotOverflow nodePointsToNonNull cannotTrackLocalUses escapesInColdBlock )  [     0x3fee8767f30] bci=[-1,0,58] rc=2 vc=14018 vn=- li=- udi=- nc=0 flg=0xd00c
n49602n     aiadd                                                                             [     0x3fee8768020] bci=[-1,0,58] rc=1 vc=0 vn=- li=- udi=- nc=2
n49600n       loadaddr  net/adoptopenjdk/test/hcrAgent/agent/StringTransformer[#398  Static] [flags 0x18307 0x0 ]  [     0x3fee8767f80] bci=[-1,0,58] rc=1 vc=14018 vn=- li=- udi=- nc=0
n49601n       iconst 0 (X==0 X>=0 X<=0 )                                                      [     0x3fee8767fd0] bci=[-1,0,58] rc=1 vc=0 vn=- li=- udi=- nc=0 flg=0x302

[7]

n27693n   BBStart <block_1871> (freq 5) (cold)                                                [     0x3ff612bbfe0] bci=[-1,11,58] rc=0 vc=14018 vn=- li=- udi=- nc=0
n27716n   lstore  <pending push temp 2>[#404  Auto] [flags 0x4 0x800 ]                        [     0x3ff612bc710] bci=[-1,8,58] rc=0 vc=14018 vn=- li=1046 udi=725 nc=1
n27715n     lload  <temp slot 17>[#1255  Auto] [flags 0x4 0x0 ] (cannotOverflow )             [     0x3ff612bc6c0] bci=[-1,8,58] rc=1 vc=14018 vn=- li=2271 udi=2231 nc=0 flg=0x1000
n27718n   astore  <pending push temp 0>[#402  Auto] [flags 0x7 0x800 ] (X!=0 )                [     0x3ff612bc7b0] bci=[-1,8,58] rc=0 vc=14018 vn=- li=1047 udi=726 nc=1 flg=0x4
n27717n     aload  <pending push temp 0>[#402  Auto] [flags 0x7 0x800 ] (X!=0 )               [     0x3ff612bc760] bci=[-1,8,58] rc=1 vc=14018 vn=- li=2272 udi=2232 nc=0 flg=0x4
n27720n   astore  <pending push temp 1>[#403  Auto] [flags 0x7 0x800 ] (X!=0 )                [     0x3ff612bc850] bci=[-1,8,58] rc=0 vc=14018 vn=- li=1048 udi=727 nc=1 flg=0x4
n27719n     aload  <temp slot 18>[#1256  Auto] [flags 0x7 0x0 ] (X!=0 )                       [     0x3ff612bc800] bci=[-1,8,58] rc=1 vc=14018 vn=- li=2273 udi=2233 nc=0 flg=0x4
n27700n   astore  st<auto slot 1>[#412  Auto] [flags 0x7 0x0 ] (X==0 StoredValueIsIrrelevant )  [     0x3ff612bc210] bci=[-1,11,58] rc=0 vc=14018 vn=- li=- udi=728 nc=1 flg=0x20002
n27699n     aconst NULL (X==0 X>=0 X<=0 )                                                     [     0x3ff612bc1c0] bci=[-1,11,58] rc=1 vc=14018 vn=- li=- udi=- nc=0 flg=0x302
n27702n   istore  <pending push temp 1>[#424  Auto] [flags 0x3 0x800 ] (StoredValueIsIrrelevant )  [     0x3ff612bc2b0] bci=[-1,11,58] rc=0 vc=14018 vn=- li=- udi=729 nc=1 flg=0x20000
n27701n     iconst 0xdead (X!=0 X>=0 )                                                        [     0x3ff612bc260] bci=[-1,11,58] rc=1 vc=14018 vn=- li=- udi=- nc=0 flg=0x104
n27710n   istore  <pending push temp 0>[#459  Auto] [flags 0x3 0x800 ] (StoredValueIsIrrelevant )  [     0x3ff612bc530] bci=[-1,11,58] rc=0 vc=14018 vn=- li=- udi=730 nc=1 flg=0x20000
n27709n     iconst 0xdead (X!=0 X>=0 )                                                        [     0x3ff612bc4e0] bci=[-1,11,58] rc=1 vc=14018 vn=- li=- udi=- nc=0 flg=0x104
n27696n   treetop ()                                                                          [     0x3ff612bc0d0] bci=[-1,11,58] rc=0 vc=14018 vn=- li=- udi=- nc=1 flg=0x8
n27695n     call  jitInduceOSRAtCurrentPCAndRecompile[#90  helper Method] [flags 0x400 0x0 ] ()  [     0x3ff612bc080] bci=[-1,11,58] rc=1 vc=14018 vn=- li=- udi=- nc=0 flg=0x8
n46264n   treetop                                                                             [     0x3fee8216cc0] bci=[-1,11,58] rc=0 vc=14018 vn=- li=- udi=- nc=1
n46231n     call  <eaEscapeHelper>[#357  helper Method] [flags 0x400 0x0 ] ()                 [     0x3fee8216270] bci=[-1,11,58] rc=1 vc=14018 vn=- li=- udi=- nc=32 flg=0x20
n46232n       aload  this<'this' parm Lnet/adoptopenjdk/test/hcrAgent/agent/TransformerMakerThread;>[#397  Parm] [flags 0x40000107 0x0 ]  [     0x3fee82162c0] bci=[-1,11,58] rc=1 vc=14018 vn=- li=- udi=2234 nc=0
n46233n       aload  <pending push temp 2>[#444  Auto] [flags 0x7 0x800 ]                     [     0x3fee8216310] bci=[-1,11,58] rc=1 vc=14018 vn=- li=- udi=2235 nc=0
n46234n       aload  <pending push temp 4>[#446  Auto] [flags 0x7 0x800 ]                     [     0x3fee8216360] bci=[-1,11,58] rc=1 vc=14018 vn=- li=- udi=2236 nc=0
n46235n       aload  <temp slot 5>[#1219  Auto] [flags 0x7 0x0 ]                              [     0x3fee82163b0] bci=[-1,11,58] rc=1 vc=14018 vn=- li=2274 udi=2237 nc=0
n46236n       aload  <temp slot 6>[#1220  Auto] [flags 0x7 0x0 ]                              [     0x3fee8216400] bci=[-1,11,58] rc=1 vc=14018 vn=- li=2275 udi=2238 nc=0
n46237n       aload  <temp slot 11>[#1244  Auto] [flags 0x7 0x0 ]                             [     0x3fee8216450] bci=[-1,11,58] rc=1 vc=14018 vn=- li=2276 udi=2239 nc=0
n46238n       aload  <temp slot 12>[#1245  Auto] [flags 0x7 0x0 ]                             [     0x3fee82164a0] bci=[-1,11,58] rc=1 vc=14018 vn=- li=2277 udi=2240 nc=0
n46239n       aload  <temp slot 19>[#1708  Auto] (obj2) [flags 0x20000007 0x0 ]               [     0x3fee82164f0] bci=[-1,11,58] rc=1 vc=14018 vn=- li=2278 udi=2241 nc=0
n46240n       aload  <temp slot 21>[#1757  Auto] [flags 0x7 0x0 ]                             [     0x3fee8216540] bci=[-1,11,58] rc=1 vc=14018 vn=- li=2279 udi=2242 nc=0
n46241n       aload  <temp slot 22>[#1758  Auto] [flags 0x7 0x0 ]                             [     0x3fee8216590] bci=[-1,11,58] rc=1 vc=14018 vn=- li=2280 udi=2243 nc=0
n46242n       aload  <temp slot 29>[#2194  Auto] (obj4) [flags 0x20000007 0x0 ]               [     0x3fee82165e0] bci=[-1,11,58] rc=1 vc=14018 vn=- li=2281 udi=2244 nc=0
n46243n       aload  <temp slot 32>[#2348  Auto] [flags 0x7 0x0 ]                             [     0x3fee8216630] bci=[-1,11,58] rc=1 vc=14018 vn=- li=2282 udi=2245 nc=0
n46244n       aload  <temp slot 33>[#2349  Auto] [flags 0x7 0x0 ]                             [     0x3fee8216680] bci=[-1,11,58] rc=1 vc=14018 vn=- li=2283 udi=2246 nc=0
n46245n       aload  <temp slot 40>[#2774  Auto] (obj6) [flags 0x20000007 0x0 ]               [     0x3fee82166d0] bci=[-1,11,58] rc=1 vc=14018 vn=- li=2284 udi=2247 nc=0
n46246n       aload  <temp slot 42>[#2811  Auto] (obj7) [flags 0x20000007 0x0 ]               [     0x3fee8216720] bci=[-1,11,58] rc=1 vc=14018 vn=- li=2285 udi=2248 nc=0
n46247n       aload  <temp slot 46>[#3239  Auto] (obj9) [flags 0x20000007 0x0 ]               [     0x3fee8216770] bci=[-1,11,58] rc=1 vc=14018 vn=- li=2286 udi=2249 nc=0
n46248n       aload  <temp slot 48>[#3266  Auto] [flags 0x7 0x0 ]                             [     0x3fee82167c0] bci=[-1,11,58] rc=1 vc=14018 vn=- li=2287 udi=2250 nc=0
n46249n       aload  <temp slot 49>[#3267  Auto] [flags 0x7 0x0 ]                             [     0x3fee8216810] bci=[-1,11,58] rc=1 vc=14018 vn=- li=2288 udi=2251 nc=0
n46250n       aload  <temp slot 56>[#3692  Auto] (obj11) [flags 0x20000007 0x0 ]              [     0x3fee8216860] bci=[-1,11,58] rc=1 vc=14018 vn=- li=2289 udi=2252 nc=0
n46251n       aload  <temp slot 60>[#4169  Auto] (obj13) [flags 0x20000007 0x0 ]              [     0x3fee82168b0] bci=[-1,11,58] rc=1 vc=14018 vn=- li=2290 udi=2253 nc=0
n46252n       aload  <temp slot 132>[#4426  Auto] [flags 0x7 0x0 ]                            [     0x3fee8216900] bci=[-1,11,58] rc=1 vc=14018 vn=- li=2291 udi=2254 nc=0
n46253n       aload  <temp slot 134>[#4438  Auto] [flags 0x7 0x0 ]                            [     0x3fee8216950] bci=[-1,11,58] rc=1 vc=14018 vn=- li=2292 udi=2255 nc=0
n46254n       aload  <temp slot 136>[#4495  Auto] [flags 0x7 0x0 ]                            [     0x3fee82169a0] bci=[-1,11,58] rc=1 vc=14018 vn=- li=2293 udi=2256 nc=0
n46255n       aload  <temp slot 138>[#4507  Auto] [flags 0x7 0x0 ]                            [     0x3fee82169f0] bci=[-1,11,58] rc=1 vc=14018 vn=- li=2294 udi=2257 nc=0
n46256n       aload  <temp slot 140>[#4522  Auto] [flags 0x7 0x0 ]                            [     0x3fee8216a40] bci=[-1,11,58] rc=1 vc=14018 vn=- li=2295 udi=2258 nc=0
n46257n       aload  <temp slot 142>[#4543  Auto] [flags 0x7 0x0 ]                            [     0x3fee8216a90] bci=[-1,11,58] rc=1 vc=14018 vn=- li=2296 udi=2259 nc=0
n46258n       aload  <temp slot 147>[#4554  Auto] [flags 0x7 0x0 ]                            [     0x3fee8216ae0] bci=[-1,11,58] rc=1 vc=14018 vn=- li=2297 udi=2260 nc=0
n46259n       aload  <temp slot 148>[#4555  Auto] [flags 0x7 0x0 ]                            [     0x3fee8216b30] bci=[-1,11,58] rc=1 vc=14018 vn=- li=2298 udi=2261 nc=0
n46260n       aload  <temp slot 149>[#4556  Auto] [flags 0x10007 0x0 ]                        [     0x3fee8216b80] bci=[-1,11,58] rc=1 vc=14018 vn=- li=2299 udi=2262 nc=0
n46261n       aload  <temp slot 151>[#4558  Auto] [flags 0x7 0x0 ]                            [     0x3fee8216bd0] bci=[-1,11,58] rc=1 vc=14018 vn=- li=2300 udi=2263 nc=0
n46262n       aload  <temp slot 152>[#4559  Auto] [flags 0x10007 0x0 ]                        [     0x3fee8216c20] bci=[-1,11,58] rc=1 vc=14018 vn=- li=2301 udi=2264 nc=0
n46263n       aload  <temp slot 155>[#4562  Auto] [flags 0x10007 0x0 ]                        [     0x3fee8216c70] bci=[-1,11,58] rc=1 vc=14018 vn=- li=2302 udi=2265 nc=0
n27698n   athrow  jitThrowException[#24  helper Method] [flags 0x400 0x0 ] (Unsigned ThrowInsertedByOSR )  [     0x3ff612bc170] bci=[-1,11,58] rc=0 vc=14018 vn=- li=- udi=- nc=1 flg=0x4008
n27697n     aload  ExceptionMeta[#326  MethodMeta +64] [flags 0x207 0x0 ] ()                  [     0x3ff612bc120] bci=[-1,11,58] rc=1 vc=14018 vn=- li=- udi=- nc=0 flg=0x8
n27694n   BBEnd </block_1871> (cold) =====                                                    [     0x3ff612bc030] bci=[-1,11,58] rc=0 vc=14018 vn=- li=- udi=- nc=0

@gacholio
Copy link
Contributor

Closing as duplicate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@a7ehuo a7ehuo

Labels
comp:jit segfault Issues that describe segfaults / JVM crashes test failure
Projects
None yet
Milestone
Release 0.44 (Java 8, 11, 17, 21) Apr...
Development

No branches or pull requests