[JDK 24] jlink fails on Windows/Mac with '<binary-file> has been modified' error

[Thihup]

Using JLink with JDK 24 fails with:

Error: C:\hostedtoolcache\windows\Java_Temurin-Hotspot_jdk\24.0.0-ea.24.0.ea\x64\bin\j2pkcs11.dll has been modified

Build link: https://github.com/adoptium/temurin24-binaries/releases/download/jdk-24%2B24-ea-beta/OpenJDK-jdk_x64_windows_hotspot_24_24-ea.zip

[karianna]

Please report this upstream to OpenJDK - https://mail.openjdk.org/mailman/listinfo/core-libs-dev

[judovana]

It do not necessarily be an upstream bug. It can be really downstream. The jmod files and the dll files are bound together. If the signing (or any other operation after build) changes the dll, the jmods have to be adjusted.

Forgive me if the details are not exactly precise, as I was dealing with this on rpm level, and had to rebuild jmods after the build was done.

[sxa]

Interesting - are you getting the same failure with the nightly Temurin builds of 21 or 23 or is it specific to 24. Our signing process should be the same across all versions.

[judovana]

Interesting - are you getting the same failure with the nightly Temurin builds of 21 or 23 or is it specific to 24. Our signing process should be the same across all versions.

Hi! I can not provide much more info. I hit it during packing the October CPU, it may be even unrelated. But I hit the same "... .so has been modified". Please ping gnu_andrew/frederic They may know more. Especially if it can imapct windows packaging.

[judovana]

I forget to mention, that I hit it on all used .so files, not jsut j2pkcs11. So maybe the j2pkcs11 is new in jdk24 (which I think it is) and thus maybe some logic missed it?

[sxa]

This looks like it changed between 24+23 and 24+24 which was when we implemented the options for JEP493 in #4035

@jerboaa Is this expected? The error message doesn't feel right to me but you're in a better place to comment. I'll re-open until we're sure this is working as expected.

[jerboaa]

This looks like it changed between 24+23 and 24+24 which was when we implemented the options for JEP493 in #4035

@jerboaa Is this expected? The error message doesn't feel right to me but you're in a better place to comment. I'll re-open until we're sure this is working as expected.

@sxa It's not expected. The way JEP 493 detects modified files is by generating a 512 sha of the file at build time when jlink runs during the build. So if Temurin adds something to those DLLs (like a signature) after, it would consider them modified. I'm not clear how the signing of DLLs work and if that is needed.

FWIW, adding --verbose to the jlink invocation would say it's linking from the run-time image. If that's what it is, then it's caused by the change in #4035 and would need some fix to get the signing step properly working. Ideally, the signing would happen before the final run-time image is linked.

[jerboaa]

Aren't Mac OSX builds also doing some signing? Would those have a similar issue?

[jerboaa]

@Thihup As a work-around you might be able to use --ignore-modified-runtime which demotes the issue to a warning. I would only recommend using it if you are sure it's OK, though.

[sxa]

@jerboaa Is this expected?

@sxa It's not expected. The way JEP 493 detects modified files is by generating a 512 sha of the file at build time when jlink runs during the build. So if Temurin adds something to those DLLs (like a signature) after, it would consider them modified. I'm not clear how the signing of DLLs work and if that is needed.

That makes sense and is what I suspected might be the case when I reopened this - thanks for confirming.

Aren't Mac OSX builds also doing some signing? Would those have a similar issue?

Yes I would expect so. I just tried windows quickly at midnight last night when I had an idea about the potential cause

that I hit it on all used .so files

@judovana To be clear on this comment, did you mean .so in the "general shared library" sense or have you seen this on Linux too which uses .so (I wouldn't expect to hit it there since we don't sign/modify the files on that platform).

FYI @andrew-m-leonard

[judovana]

@sxa we are doing nasty stuff in rpms. We build unstripped portables, and then strip and repack them. That was the reason for ...has been modified error. Regenerating jmods also on stripped form was the cure, thats why the issue it triggered the bell.

[jerboaa]

adoptium/ci-jenkins-pipelines#1145 seems related (and might need some work).

[Thihup]

Same issue with JDK 24+25 https://github.com/adoptium/temurin24-binaries/releases/download/jdk-24%2B25-ea-beta/OpenJDK-jdk_x64_windows_hotspot_24_25-ea.zip

[sxa]

Same issue with JDK 24+25

Yep that's expected - the change was made between +23 and +24 so will affect later versions too until we have a resolution

[andrew-m-leonard]

@sxa It's not expected. The way JEP 493 detects modified files is by generating a 512 sha of the file at build time when jlink runs during the build. So if Temurin adds something to those DLLs (like a signature) after, it would consider them modified. I'm not clear how the signing of DLLs work and if that is needed.

@jerboaa @sxa
The Temurin rough build & signing process for say Windows DLLs is:

1. Make complete exploded image (which presumably would include jlink build...):

• Compile...
• Link into DLL

2. Sign all DLLs...
3. Assemble exploded image to create a build/image/...

• Assemble JMODs, zips, images and CDS

So I suspect the jlink run in in stage (1), we'd need to re-run it in (3) maybe...?

[andrew-m-leonard]

Actually, i've got a new thought on this, I suspect this is to do with the secondary signing job "sign_build", which is run after the JDK build stage. @sxa I need to have a think, do/should we be running that? eg: https://ci.adoptium.net/job/build-scripts/job/release/job/sign_build/29528/

[jerboaa]

@sxa It's not expected. The way JEP 493 detects modified files is by generating a 512 sha of the file at build time when jlink runs during the build. So if Temurin adds something to those DLLs (like a signature) after, it would consider them modified. I'm not clear how the signing of DLLs work and if that is needed.

@jerboaa @sxa The Temurin rough build & signing process for say Windows DLLs is:

1. Make complete exploded image (which presumably would include jlink build...):


* Compile...

* Link into DLL


2. Sign  all DLLs...

3. Assemble exploded image to create a  build/image/...


* Assemble JMODs, zips, images and CDS

So I suspect the jlink run in in stage (1), we'd need to re-run it in (3) maybe...?

The exploded image shouldn't include lib/modules, so I doubt jlink has been run at that point. So it might be that just step 3 might need changing. Here is the OpenJDK build invocation which generates a linkable runtime (if so configured):
https://github.com/openjdk/jdk/blob/ac3bbf7ddc1f6a55682264d66282e6ee8825f47a/make/Images.gmk#L99-L113

Whenever the above runs is relevant.

It essentially adds the --generate-linkable-runtime option to jlink.

[judovana]

Probably stupid question, but also the content of zips is signed, or just msi?

[andrew-m-leonard]

https://ci.adoptium.net/job/build-scripts/job/release/job/sign_build/29528/

Signing ./bin/j2pkcs11.dll
Signing ./bin/j2pkcs11.dll using Eclipse Foundation codesign service

[sxa]

Probably stupid question, but also the content of zips is signed, or just msi?

The files within the zips are also signed. The files that the MSI and zip lays down on the users' machine should be identical.

[jerboaa]

@smlambert Do we have some basic jlink tests that run on Temurin builds? A simple smoke test like running jlink --add-modules java.base --output testimage should reproduce this issue and serve as a regression test. If so I can work on that. I suppose the regular smoke tests run after signing on Windows/Mac?

[smlambert]

There are no jlink tests currently in the smoke tests bucket, as you say, adding a quick test at that point is a good idea.

There are other test targets that do exercise jlink (including the jdk_tools target from openjdk), and they do not look healthy for JDK24, but no one has triaged or investigated those at the moment.

Example: https://ci.adoptium.net/job/Test_openjdk24_hs_extended.openjdk_x86-64_linux_testList_0/18/testReport/junit/tools_jpackage_share_JLinkOptionsTest/java/JLinkOptionsTest/

[jerboaa]

There are no jlink tests currently in the smoke tests bucket, as you say, adding a quick test at that point is a good idea.

Thanks. I'll add that to my TODO.

There are other test targets that do exercise jlink (including the jdk_tools target from openjdk), and they do not look healthy for JDK24, but no one has triaged or investigated those at the moment.

Linux is OK. The basic jdk/tools/jlink tests pass.

Example: https://ci.adoptium.net/job/Test_openjdk24_hs_extended.openjdk_x86-64_linux_testList_0/18/testReport/junit/tools_jpackage_share_JLinkOptionsTest/java/JLinkOptionsTest/

Those are all jpackage tests. I'll look into those. That's more of an upstream task.

[judovana]

@smlambert Do we have some basic jlink tests that run on Temurin builds? A simple smoke test like running jlink --add-modules java.base --output testimage should reproduce this issue and serve as a regression test. If so I can work on that. I suppose the regular smoke tests run after signing on Windows/Mac?

We have one. @sefroberg is supposed to contribute it to aqa-tests. He wanted the do that during summer, but the interruption on container work-flow delayed it.

[jerboaa]

There are no jlink tests currently in the smoke tests bucket, as you say, adding a quick test at that point is a good idea.

Thanks. I'll add that to my TODO.

Smoke test PR with basic jlink is here:
#4067

[jerboaa]

@sxa I thought @andrew-m-leonard is working on this? FWIW, I don't have enough experience with windows/mac signing for temurin builds to be a good assignee for this. Happy to help guide when it comes to how a linkable runtime is assembled.

[sxa]

@sxa I thought @andrew-m-leonard is working on this?

Yep Andrew plans to be on it this week ;-)

[merks]

@jerboaa

The use of --ignore-modified-runtime was a good hint for making progress with 24+26! Indeed it downgrades to warnings:

Warning: /Users/genie.justj/jenkins/workspace/build-jres/jdk-24+26/Contents/Home/lib/libjdwp.dylib has been modified

Thanks!

[jerboaa]

@jerboaa

The use of --ignore-modified-runtime was a good hint for making progress with 24+26! Indeed it downgrades to warnings:

Warning: /Users/genie.justj/jenkins/workspace/build-jres/jdk-24+26/Contents/Home/lib/libjdwp.dylib has been modified

@merks Glad to hear it. Please remove the option once this issue is fixed, though. It's not a good idea to keep it.

[merks]

Please remove the option once this issue is fixed, though. It's not a good idea to keep it.

Absolutely! The script only does this specifically for Java 24 and I will retest/revert when Java 24+27 comes out. But now I can complete the build for Java 24+26:

https://ci.eclipse.org/justj/job/build-jres/

[jerboaa]

Please remove the option once this issue is fixed, though. It's not a good idea to keep it.

Absolutely! The script only does this specifically for Java 24 and I will retest/revert when Java 24+27 comes out. But now I can complete the build for Java 24+26:

https://ci.eclipse.org/justj/job/build-jres/

Excellent. Thank you!

[jerboaa]

All:

New JDK 24 EA builds should be available with via the API that have this issue fixed (jdk 24+28 and better). Please report here if this isn't working for you.

[merks]

FYI, the JustJ Temurin Indexer is only finding 24+27 for Linux aarch64:

def java24Adoptium = [
  label: 'Java 24 (Adoptium) Early Access Beta',
  BUILD_DESCRIPTION: 'Adoptium Java 24+28',
  PUBLISH_LOCATION_PREFIX: "${defaultPrefix}",
  JUSTJ_MANIFEST_URL: "${justjURL}/${defaultPrefix}/24/downloads/latest/justj.manifest",
  JDK_URLS_WINDOWS: '''
    https://github.com/adoptium/temurin24-binaries/releases/download/jdk-24+28-ea-beta/OpenJDK24U-jdk_x64_windows_hotspot_24_28-ea.zip
    ''',
  JDK_URLS_WINDOWS_AARCH64: '''
    https://github.com/adoptium/temurin24-binaries/releases/download/jdk-24+28-ea-beta/OpenJDK24U-jdk_aarch64_windows_hotspot_24_28-ea.zip
    ''',
  JDK_URLS_MACOS: '''
    https://github.com/adoptium/temurin24-binaries/releases/download/jdk-24+28-ea-beta/OpenJDK24U-jdk_x64_mac_hotspot_24_28-ea.tar.gz
    ''',
  JDK_URLS_MACOS_AARCH64: '''
    https://github.com/adoptium/temurin24-binaries/releases/download/jdk-24+28-ea-beta/OpenJDK24U-jdk_aarch64_mac_hotspot_24_28-ea.tar.gz
    ''',
  JDK_URLS_LINUX: '''
    https://github.com/adoptium/temurin24-binaries/releases/download/jdk-24+28-ea-beta/OpenJDK24U-jdk_x64_linux_hotspot_24_28-ea.tar.gz
    ''',
  JDK_URLS_LINUX_AARCH64: '''
    https://github.com/adoptium/temurin24-binaries/releases/download/jdk-24+27-ea-beta/OpenJDK24U-jdk_aarch64_linux_hotspot_24_27-ea.tar.gz
    ''',
  JDK_URLS_LINUX_PPC64LE: '''
    https://github.com/adoptium/temurin24-binaries/releases/download/jdk-24+28-ea-beta/OpenJDK24U-jdk_ppc64le_linux_hotspot_24_28-ea.tar.gz
    ''',
  JDK_URLS_LINUX_RISCV64: '''
    https://github.com/adoptium/temurin24-binaries/releases/download/jdk-24+28-ea-beta/OpenJDK24U-jdk_riscv64_linux_hotspot_24_28-ea.tar.gz
    ''',
  BUILD_TYPE: 'nightly',
  PROMOTE: 'true'
]

This definitely doesn't exist yet:

https://github.com/adoptium/temurin24-binaries/releases/download/jdk-24+28-ea-beta/OpenJDK24U-jdk_aarch64_linux_hotspot_24_28-ea.tar.gz

It's been this way for quite a few days...

In any case, I did locally test build JREs for Windows x64 without the workaround and definitely the problem is fixed for that. I can't confirm macos without an actual build and for that I generally need all the JDKs at the same version. But I do expect the problem is fixed there too. 😄

[sxa]

Looks like we had a build failure on that one: https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk24/job/jdk24-linux-aarch64-temurin/3/console

FYI @andrew-m-leonard @adamfarley - do we have any plans to re-run this one?

[adamfarley]

Looks like we had a build failure on that one: https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk24/job/jdk24-linux-aarch64-temurin/3/console

FYI @andrew-m-leonard @adamfarley - do we have any plans to re-run this one?

Not me. I've rerun it anyway.

https://ci.adoptium.net/job/build-scripts/job/openjdk24-pipeline/62/

Not sure why it timed out without an error. Maybe we need better error reporting, assuming docker reported anything and we just ignored it for some reason.