DDR vthreads command no longer works

[keithc-ca]

Java -version output

$ java -Xdump:system:events=vmstop -version
openjdk version "21-internal" 2023-09-19
OpenJDK Runtime Environment (build 21-internal-adhoc.keithc.jdknext)
Eclipse OpenJ9 VM (build main-55b92677405, JRE 21 Linux amd64-64-Bit Compressed References 20230323_000000 (JIT enabled, AOT enabled)
OpenJ9   - 55b92677405
OMR      - e6796be5b15
JCL      - 80e1dc85e7e based on jdk-21+13)

Summary of problem

Core dumps created by VM which include the changes in #16855 will not have the field J9JavaVM.virtualThreadLinkNextOffset. Attempts to use the !vthreads command will always just fail:

> !vthreads
Problem running command:
com.ibm.j9ddr.tools.ddrinteractive.DDRInteractiveCommandException: java.lang.NoSuchFieldException
	at com.ibm.j9ddr.vm29.tools.ddrinteractive.VirtualThreadsCommand.run(VirtualThreadsCommand.java:64)
	at openj9.dtfj/com.ibm.j9ddr.tools.ddrinteractive.Context.tryCommand(Context.java:229)
	at openj9.dtfj/com.ibm.j9ddr.tools.ddrinteractive.Context.execute(Context.java:202)
	at openj9.dtfj/com.ibm.j9ddr.tools.ddrinteractive.DDRInteractive.execute(DDRInteractive.java:356)
	at openj9.dtfj/com.ibm.j9ddr.command.CommandReader.processLine(CommandReader.java:78)
	at openj9.dtfj/com.ibm.j9ddr.tools.ddrinteractive.DDRInteractive.processLine(DDRInteractive.java:331)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:578)
	at openj9.dtfjview/com.ibm.jvm.dtfjview.CombinedContext.executeDDRInteractiveCommand(CombinedContext.java:258)
	at openj9.dtfjview/com.ibm.jvm.dtfjview.CombinedContext.execute(CombinedContext.java:169)
	at openj9.dtfjview/com.ibm.jvm.dtfjview.Session.execute(Session.java:813)
	at openj9.dtfjview/com.ibm.jvm.dtfjview.Session.execute(Session.java:767)
	at openj9.dtfjview/com.ibm.jvm.dtfjview.tools.ToolsRegistry.executeJdmpviewCommand(ToolsRegistry.java:183)
	at openj9.dtfjview/com.ibm.jvm.dtfjview.tools.ToolsRegistry.execute(ToolsRegistry.java:107)
	at openj9.dtfjview/com.ibm.jvm.dtfjview.tools.ToolsRegistry.execute(ToolsRegistry.java:88)
	at openj9.dtfjview/com.ibm.jvm.dtfjview.tools.ToolsRegistry.recordAndExecute(ToolsRegistry.java:72)
	at openj9.dtfjview/com.ibm.jvm.dtfjview.Session.runInteractive(Session.java:741)
	at openj9.dtfjview/com.ibm.jvm.dtfjview.Session.run(Session.java:665)
	at openj9.dtfjview/com.ibm.jvm.dtfjview.DTFJView.launch(DTFJView.java:52)
	at openj9.dtfjview/com.ibm.jvm.dtfjview.DTFJView.main(DTFJView.java:46)
Caused by: java.lang.NoSuchFieldException
	at com.ibm.j9ddr.vm29.pointer.generated.J9JavaVMPointer.liveVirtualThreadList(Unknown Source)
	at com.ibm.j9ddr.vm29.tools.ddrinteractive.VirtualThreadsCommand.displayVirtualThreads(VirtualThreadsCommand.java:75)
	at com.ibm.j9ddr.vm29.tools.ddrinteractive.VirtualThreadsCommand.run(VirtualThreadsCommand.java:61)
	... 19 more

[keithc-ca]

FYI @fengxue-IS @gacholio @tajila

[babsingh]

@thallium Can you provide links to the PRs where !vthreads and !continuationstack are being reworked?

I don't believe !vthreads has been formally documented. So, it does not need to be fixed for the 0.37 release.

[babsingh]

Alone !vthreads is not sufficient, both !vthreads and !continuationstack are needed for complete diagnostic support.

For the JDK19 release, the easiest approach would be to disable the !vthreads cmd for resolving the reported DDR error.

[fengxue-IS]

I agree with @babsingh , Without !continuationstack, the use of !vthreads is only to list out all the allocated vthread objects, it wouldn't be very useful if DDR user cannot see the associated stack with the vthread as nothing in the vthread object it self can go wrong.

[keithc-ca]

Unfortunately, if the standard were that it was "formally documented", we would have very few commands. I don't think that's a helpful measure.

Yes, the output of !vthreads includes !continuationstack which is not yet implemented, but other helpful information provided, so I think we should try to keep it. Unless there's some other way to enumerate VirtualThread objects, I suggest we use JavaHeap.getObjects(), checking that the class is java.lang.VirtualThread.

[babsingh]

JDK19 is the first Openj9 release where !vthreads will be publicly published. We do have an option to push !vthreads to the next release since it does not exist in any prior OpenJ9 release.

[gacholio]

This sort of thing is for VM developers, not users. Why would we document it at all (other than in the !help list)?

[babsingh]

Why would we document it at all (other than in the !help list)?

Yes, it does not need to be documented. I used the wrong phrase formally documented -> published in an earlier OpenJ9 release.

Not sure how quickly #17093 will be merged. @pshipton When are the JDK19 release builds scheduled?

[pshipton]

There is no fixed schedule other than asap. We are waiting for the things in the milestone plan to be resolved in order to proceed with RC1. Typically we allow DDR updates to occur after a release since you can use a later build to investigate core files created by earlier builds. Are there quick updates we can make as Keith suggests in #17084 (comment), or is it better to defer this?

[babsingh]

Are there quick updates we can make as Keith suggests in #17084 (comment), or is it better to defer this?

#17093 should be the first preference. @keithc-ca Let us know if #17093 is close to being merged. If not, we can explore the JavaHeap.getObjects suggestion.

[gacholio]

It should be noted that both of these solutions are unrealible if the heap is in a bad state (i.e. in the middle of a GC). The JVMTI code also flushes some GC stuff before walking the continuation list, which DDR will of course not be able to do. I have no solution for this.

[babsingh]

The JVMTI code also flushes some GC stuff before walking the continuation list, which DDR will of course not be able to do.

Option 1: We can invoke j9gc_flush_nonAllocationCaches_for_walk prior to system dump (core file) generation. There will be a timing gap if the system dump is not created under exclusive VM access i.e the GC will be able to update the continuation list.

Option 2: Iterate through GC's nonAllocationCaches in DDR.

@amicic @LinHu2016 @dmitripivkine for thoughts?

[keithc-ca]

Let us know if #17093 is close to being merged.

Yes, I expect it's close.

[dmitripivkine]

The JVMTI code also flushes some GC stuff before walking the continuation list, which DDR will of course not be able to do.

Option 1: We can invoke j9gc_flush_nonAllocationCaches_for_walk prior to system dump (core file) generation. There will be a timing gap if the system dump is not created under exclusive VM access i.e the GC will be able to update the continuation list.

Option 2: Iterate through GC's nonAllocationCaches in DDR.

@amicic @LinHu2016 @dmitripivkine for thoughts?

I believe nonAllocationCaches should be flushed already before core generation. There is very similar logic for Ownable Synchronizers lists in gccheck can be used as example.

[pshipton]

Reopen until we have the backport in 0.37