Status of Reproducible comparing tests on linux #3967

sophia-guo · 2024-10-02T17:39:03Z

PPC64le and aarch64 was enabled on Sep 25th. adoptium/ci-jenkins-pipelines#1115. There is no data yet as no betaTrigger builds triggered with test enabled.

Recent linux update was merged in Sep 6th. .

Most recent ea pipeline jobs are triggered by test disabled (not sure why, for example https://ci.adoptium.net/job/build-scripts/job/openjdk21-pipeline/333/), which means most reproducible comparing tests are quite old and may not be worth looking at. FYI @smlambert @andrew-m-leonard

Only one test build with jdk24 is with recent update. https://trss.adoptium.net/output/test?id=66f72b85d24e1b006ee0dd01 @andrew-m-leonard triggered by https://ci.adoptium.net/job/build-scripts/job/openjdk24-pipeline/25/, test jenkins job is https://ci.adoptium.net/job/Test_openjdk24_hs_special.system_x86-64_linux/11/. Both job and rerun job have same results. 6 files differs.

17:51:56  Number of files: 105604
17:51:56  Comparing jdk-24+17 with tarJDK/jdk-24+17 ... output to file: reprotest.diff
17:52:00  Files jdk-24+17/jmods/expanded_java.base.jmod/lib/classlist and tarJDK/jdk-24+17/jmods/expanded_java.base.jmod/lib/classlist differ
17:52:00  Files jdk-24+17/jmods/expanded_java.base.jmod/lib/server/libjvm.so and tarJDK/jdk-24+17/jmods/expanded_java.base.jmod/lib/server/libjvm.so differ
17:52:00  Files jdk-24+17/lib/classlist and tarJDK/jdk-24+17/lib/classlist differ
17:52:00  Files jdk-24+17/lib/server/classes.jsa and tarJDK/jdk-24+17/lib/server/classes.jsa differ
17:52:00  Files jdk-24+17/lib/server/classes_nocoops.jsa and tarJDK/jdk-24+17/lib/server/classes_nocoops.jsa differ
17:52:00  Files jdk-24+17/lib/server/libjvm.so and tarJDK/jdk-24+17/lib/server/libjvm.so differ
17:52:00  Number of differences: 6
17:52:00  ReproduciblePercent = 99.99 %
17:52:00  Differences found..., logged in: reprotest.diff

@andrew-m-leonard I remember you mentioned those CDS files differences ( 4 files) are a known issue due to the classlist ipv6 difference. Compared with the job I tested before https://ci.adoptium.net/job/Test_openjdk23_hs_special.system_x86-64_linux_testList_0/29/console, number of files increased and two more file differs. jmods/expanded_java.base.jmod/lib/server/libjvm.so and /lib/server/libjvm.so and tarJDK/jdk-24+17/lib/server/libjvm.so. Are those two differences expected?

09:59:33  Number of files: 104722
09:59:33  Comparing jdk-23+36 with tarJDK/jdk-23+36 ... output to file: reprotest.diff
09:59:37  Files jdk-23+36/jmods/expanded_java.base.jmod/lib/classlist and tarJDK/jdk-23+36/jmods/expanded_java.base.jmod/lib/classlist differ
09:59:37  Files jdk-23+36/lib/classlist and tarJDK/jdk-23+36/lib/classlist differ
09:59:37  Files jdk-23+36/lib/server/classes.jsa and tarJDK/jdk-23+36/lib/server/classes.jsa differ
09:59:37  Files jdk-23+36/lib/server/classes_nocoops.jsa and tarJDK/jdk-23+36/lib/server/classes_nocoops.jsa differ
09:59:37  Number of differences: 4
09:59:37  ReproduciblePercent = 99.99 %
09:59:37  Differences found..., logged in: reprotest.diff``

The text was updated successfully, but these errors were encountered:

sophia-guo · 2024-10-02T19:46:44Z

I see , most of September openjdk pipeline jobs disabled tests job as it's in Release period, which causes only one job with jdk24 available. adoptium/temurin#54 (comment)

andrew-m-leonard · 2024-10-03T10:48:35Z

@sophia-guo jdk-21.0.5+9 EA is building now.... so you should have a new test to look at soon...

The above differences need looking at, for Linux they should be 100% identical, I suspect an "environmental" difference that we need to investigate in this issue. I suspect one of:

Differing TimeZone?
Differing ipv6 enablement?

@sophia-guo What would be really useful output in the test, is if the "classlist" file appears in the diff, then do a verbose content diff just for that file, as that will reveal any of the differences suggested above ^

sophia-guo · 2024-10-03T19:12:09Z

I think you mentioned @andrew-m-leonard

I suspect you've added the CDS files for Linux due to the classlist ipv6 difference, that is a known issue, and we need to ensure we have the same ipv6 setting on the reproduce VM.

sophia-guo · 2024-10-07T15:30:02Z

The latest results -
linux-x64 https://ci.adoptium.net/job/Test_openjdk21_hs_special.system_x86-64_linux/27/artifact/aqa-tests/TKG/output_17279540359679/Test_openjdk21_hs_special.system_x86-64_linux.tap/*view*/

        Cleaning any Temurin build-scripts release file BUILD_INFO from tarJDK/jdk-21.0.5+9
        Number of files: 108609
        Comparing jdk-21.0.5+9 with tarJDK/jdk-21.0.5+9 ... output to file: reprotest.diff
        Files jdk-21.0.5+9/jmods/expanded_java.base.jmod/lib/classlist and tarJDK/jdk-21.0.5+9/jmods/expanded_java.base.jmod/lib/classlist differ
        Files jdk-21.0.5+9/lib/classlist and tarJDK/jdk-21.0.5+9/lib/classlist differ
        Files jdk-21.0.5+9/lib/server/classes.jsa and tarJDK/jdk-21.0.5+9/lib/server/classes.jsa differ
        Files jdk-21.0.5+9/lib/server/classes_nocoops.jsa and tarJDK/jdk-21.0.5+9/lib/server/classes_nocoops.jsa differ
        Number of differences: 4
        ReproduciblePercent = 99.99 %
        Differences found..., logged in: reprotest.diff

linux-aarch64 and ppc64le have the same result
https://ci.adoptium.net/job/Test_openjdk21_hs_special.system_ppc64le_linux/1/artifact/aqa-tests/TKG/output_17279518397907/Test_openjdk21_hs_special.system_ppc64le_linux.tap/*view*/

https://ci.adoptium.net/job/Test_openjdk21_hs_special.system_ppc64le_linux/1/artifact/aqa-tests/TKG/output_17279518397907/Test_openjdk21_hs_special.system_ppc64le_linux.tap/*view*/)

        Comparing jdk-21.0.5+9 with tarJDK/jdk-21.0.5+9 ... output to file: reprotest.diff
        Only in jdk-21.0.5+9: jdk-21.0.5+9
        Only in jdk-21.0.5+9: jdk-21.0.5+9-static-libs
        Files jdk-21.0.5+9/jmods/expanded_java.base.jmod/lib/classlist and tarJDK/jdk-21.0.5+9/jmods/expanded_java.base.jmod/lib/classlist differ
        Files jdk-21.0.5+9/lib/classlist and tarJDK/jdk-21.0.5+9/lib/classlist differ
        Files jdk-21.0.5+9/lib/server/classes.jsa and tarJDK/jdk-21.0.5+9/lib/server/classes.jsa differ
        Files jdk-21.0.5+9/lib/server/classes_nocoops.jsa and tarJDK/jdk-21.0.5+9/lib/server/classes_nocoops.jsa differ
        Number of differences: 6
        ReproduciblePercent = 99.99 %
        Differences found..., logged in: reprotest.diff

sophia-guo · 2024-10-11T13:58:14Z

#3988

sophia-guo · 2024-10-30T14:01:58Z

If the "classlist" file appears in the diff, do a verbose content diff just for that file by 'diff -r,'
https://github.com/adoptium/temurin-build/compare/master...sophia-guo:openjdk-build:LinuxClass?expand=1

results:

 23:33:55  Cleaning any Temurin build-scripts release file BUILD_INFO from tarJDK/jdk-21.0.4+7
23:33:56  Number of files: 108594
23:33:56  Comparing jdk-21.0.4+7 with tarJDK/jdk-21.0.4+7 ... output to file: reprotest.diff
23:34:00  Files jdk-21.0.4+7/jmods/expanded_java.base.jmod/lib/classlist and tarJDK/jdk-21.0.4+7/jmods/expanded_java.base.jmod/lib/classlist differ
23:34:00  Files jdk-21.0.4+7/lib/classlist and tarJDK/jdk-21.0.4+7/lib/classlist differ
23:34:00  Files jdk-21.0.4+7/lib/server/classes.jsa and tarJDK/jdk-21.0.4+7/lib/server/classes.jsa differ
23:34:00  Files jdk-21.0.4+7/lib/server/classes_nocoops.jsa and tarJDK/jdk-21.0.4+7/lib/server/classes_nocoops.jsa differ
23:34:00  diff -c on jdk-21.0.4+7/jmods/expanded_java.base.jmod/lib/classlist and tarJDK/jdk-21.0.4+7/jmods/expanded_java.base.jmod/lib/classlist
23:34:00  *** jdk-21.0.4+7/jmods/expanded_java.base.jmod/lib/classlist	Wed Oct 30 03:33:26 2024
23:34:00  --- tarJDK/jdk-21.0.4+7/jmods/expanded_java.base.jmod/lib/classlist	Wed Oct 30 03:33:51 2024
23:34:00  ***************
23:34:00  *** 351,369 ****
23:34:00    java/lang/reflect/Type
23:34:00    java/math/BigInteger
23:34:00    java/math/RoundingMode
23:34:00  - java/net/DefaultInterface
23:34:00    java/net/Inet4Address
23:34:00    java/net/Inet4AddressImpl
23:34:00  - java/net/Inet6Address
23:34:00  - java/net/Inet6Address$Inet6AddressHolder
23:34:00  - java/net/Inet6AddressImpl
23:34:00    java/net/InetAddress
23:34:00    java/net/InetAddress$1
23:34:00    java/net/InetAddress$InetAddressHolder
23:34:00    java/net/InetAddress$PlatformResolver
23:34:00    java/net/InetAddressImpl
23:34:00  - java/net/InterfaceAddress
23:34:00  - java/net/NetworkInterface
23:34:00    java/net/URI
23:34:00    java/net/URI$1
23:34:00    java/net/URI$Parser
23:34:00  --- 351,363 ----
23:34:00  diff -c on jdk-21.0.4+7/lib/classlist and tarJDK/jdk-21.0.4+7/lib/classlist
23:34:00  *** jdk-21.0.4+7/lib/classlist	Wed Oct 30 03:10:54 2024
23:34:00  --- tarJDK/jdk-21.0.4+7/lib/classlist	Wed Oct 30 03:30:04 2024
23:34:00  ***************
23:34:00  *** 351,369 ****
23:34:00    java/lang/reflect/Type
23:34:00    java/math/BigInteger
23:34:00    java/math/RoundingMode
23:34:00  - java/net/DefaultInterface
23:34:00    java/net/Inet4Address
23:34:00    java/net/Inet4AddressImpl
23:34:00  - java/net/Inet6Address
23:34:00  - java/net/Inet6Address$Inet6AddressHolder
23:34:00  - java/net/Inet6AddressImpl
23:34:00    java/net/InetAddress
23:34:00    java/net/InetAddress$1
23:34:00    java/net/InetAddress$InetAddressHolder
23:34:00    java/net/InetAddress$PlatformResolver
23:34:00    java/net/InetAddressImpl
23:34:00  - java/net/InterfaceAddress
23:34:00  - java/net/NetworkInterface
23:34:00    java/net/URI
23:34:00    java/net/URI$1
23:34:00    java/net/URI$Parser
23:34:00  --- 351,363 ----
23:34:00  diff -c on jdk-21.0.4+7/lib/server/classes.jsa and tarJDK/jdk-21.0.4+7/lib/server/classes.jsa
23:34:00  Binary files jdk-21.0.4+7/lib/server/classes.jsa and tarJDK/jdk-21.0.4+7/lib/server/classes.jsa differ
23:34:00  diff -c on jdk-21.0.4+7/lib/server/classes_nocoops.jsa and tarJDK/jdk-21.0.4+7/lib/server/classes_nocoops.jsa
23:34:00  Binary files jdk-21.0.4+7/lib/server/classes_nocoops.jsa and tarJDK/jdk-21.0.4+7/lib/server/classes_nocoops.jsa differ
23:34:00  Number of differences: 4
23:34:00  ReproduciblePercent = 99.99 %
23:34:00  Differences found..., logged in: reprotest.diff
23:34:00  -----------------------------------

https://ci.adoptium.net/job/Grinder/11263/

Using https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk21u/job/jdk21u-linux-x64-temurin/286/artifact/workspace/target/OpenJDK21U-jdk_x64_linux_hotspot_2024-10-15-21-58.tar.gz tag: 21.0.4+7

sophia-guo · 2024-10-30T14:03:14Z

@andrew-m-leonard #3967 (comment) updated the verbose output.

andrew-m-leonard · 2024-10-30T14:46:52Z

thanks @sophia-guo so yes the problem is one env has ipv6 enabled the other doesn't... we need to fix one of the environments

andrew-m-leonard · 2024-10-30T14:47:30Z

We maybe able to dynamically disable it using a sysctl maybe ?

sophia-guo · 2024-10-30T15:27:06Z

temurin build environment is adoptopenjdk/centos7_build_image and reproducible is uisng centos:7. It seems adoptopenjdk/centos7_build_image IPV6 is enabled while centos:7 isn't? @andrew-m-leonard are you talking about disable it in temurin build?

sophia-guo · 2024-10-30T15:37:08Z

Google notes

Red-hat based distributions, Change to the root user.
Type these commands: disable

sysctl -w net.ipv6.conf.all.disable_ipv6=1
sysctl -w net.ipv6.conf.default.disable_ipv6=1
sysctl -w net.ipv6.conf.tun0.disable_ipv6=1

To re-enable IPv6, type these commands:
sysctl -w net.ipv6.conf.all.disable_ipv6=0
sysctl -w net.ipv6.conf.default.disable_ipv6=0
sysctl -w net.ipv6.conf.tun0.disable_ipv6=0
sysctl -p

andrew-m-leonard · 2024-10-31T09:24:51Z

temurin build environment is adoptopenjdk/centos7_build_image and reproducible is uisng centos:7. It seems adoptopenjdk/centos7_build_image IPV6 is enabled while centos:7 isn't? @andrew-m-leonard are you talking about disable it in temurin build?

@sophia-guo i'm thinking the test script could dynamically enable in the container possibly using sysctl, would need testing to see if it works?

sophia-guo · 2024-10-31T20:43:31Z

Did a try. Seems this can be done by configure namespaced kernel parameters (sysctls) at runtime (--sysctl). Note Docker does not support changing sysctls inside of a container that also modify the host system. https://docs.docker.com/reference/cli/docker/container/run/#sysctl

Grinder https://ci.adoptium.net/job/Grinder/11270/

andrew-m-leonard · 2024-11-01T09:48:33Z

Did a try. Seems this can be done by configure namespaced kernel parameters (sysctls) at runtime (--sysctl). Note Docker does not support changing sysctls inside of a container that also modify the host system. https://docs.docker.com/reference/cli/docker/container/run/#sysctl

Grinder https://ci.adoptium.net/job/Grinder/11270/

@sophia-guo Looks good?

19:01:02  ReproduciblePercent = 100 %
19:01:02  Compare identical !

sophia-guo · 2024-11-01T14:26:10Z

@andrew-m-leonard yes, the PR is here #4020

github-project-automation bot added this to Adoptium Backlog Oct 2, 2024

github-project-automation bot moved this to Todo in Adoptium Backlog Oct 2, 2024

github-actions bot added aarch Issues that affect or relate to the aarch ARCHITECTURE jenkins Issues that enhance or fix our jenkins server testing Issues that enhance or fix our test suites x-linux Issues that affect or relate to the x64/x32 LINUX OS labels Oct 2, 2024

sophia-guo changed the title ~~Status of Reproduciable comparing tests on linux~~ Status of Reproducible comparing tests on linux Oct 2, 2024

sophia-guo mentioned this issue Oct 31, 2024

Enable IPV6 at runtime #4020

Merged

sophia-guo self-assigned this Nov 1, 2024

sophia-guo added this to 2024 4Q Adoptium Plan Nov 1, 2024

sophia-guo removed this from Adoptium Backlog Nov 1, 2024

sophia-guo moved this to In Progress in 2024 4Q Adoptium Plan Nov 1, 2024

sophia-guo closed this as completed in #4020 Nov 4, 2024

github-project-automation bot moved this from In Progress to Done in 2024 4Q Adoptium Plan Nov 4, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Status of Reproducible comparing tests on linux #3967

Status of Reproducible comparing tests on linux #3967

sophia-guo commented Oct 2, 2024 •

edited

Loading

sophia-guo commented Oct 2, 2024

andrew-m-leonard commented Oct 3, 2024

sophia-guo commented Oct 3, 2024

sophia-guo commented Oct 7, 2024

sophia-guo commented Oct 11, 2024

sophia-guo commented Oct 30, 2024 •

edited

Loading

sophia-guo commented Oct 30, 2024

andrew-m-leonard commented Oct 30, 2024

andrew-m-leonard commented Oct 30, 2024

sophia-guo commented Oct 30, 2024

sophia-guo commented Oct 30, 2024

andrew-m-leonard commented Oct 31, 2024 •

edited

Loading

sophia-guo commented Oct 31, 2024 •

edited

Loading

andrew-m-leonard commented Nov 1, 2024

sophia-guo commented Nov 1, 2024

Status of Reproducible comparing tests on linux #3967

Status of Reproducible comparing tests on linux #3967

Comments

sophia-guo commented Oct 2, 2024 • edited Loading

sophia-guo commented Oct 2, 2024

andrew-m-leonard commented Oct 3, 2024

sophia-guo commented Oct 3, 2024

sophia-guo commented Oct 7, 2024

sophia-guo commented Oct 11, 2024

sophia-guo commented Oct 30, 2024 • edited Loading

sophia-guo commented Oct 30, 2024

andrew-m-leonard commented Oct 30, 2024

andrew-m-leonard commented Oct 30, 2024

sophia-guo commented Oct 30, 2024

sophia-guo commented Oct 30, 2024

andrew-m-leonard commented Oct 31, 2024 • edited Loading

sophia-guo commented Oct 31, 2024 • edited Loading

andrew-m-leonard commented Nov 1, 2024

sophia-guo commented Nov 1, 2024

sophia-guo commented Oct 2, 2024 •

edited

Loading

sophia-guo commented Oct 30, 2024 •

edited

Loading

andrew-m-leonard commented Oct 31, 2024 •

edited

Loading

sophia-guo commented Oct 31, 2024 •

edited

Loading