Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update getjdkbinary to correctly handle Temurin debugimage and static-libs folders #5750

Merged
merged 16 commits into from
Nov 20, 2024
Merged

Update getjdkbinary to correctly handle Temurin debugimage and static-libs folders #5750

merged 16 commits into from
Nov 20, 2024

Conversation

andrew-m-leonard
Copy link
Contributor

@andrew-m-leonard andrew-m-leonard commented Nov 13, 2024 •
edited
Loading

Fixes #5749

To resolve the above issue, this PR:

  • Updates the debugimage "overlay" logic to handle the differing Temurin and IBM/semeru debugimage archive structure, with differing number of top-level folders that need strippping
  • Add support for Temurin static-libs uncompressing
  • Add safety check to ensure the archive contents are moved correctly and not overlaying within an existing expansion of another archive. If a new type of archive is added this check will fail to ensure we don't corrupt the j2sdk-image content with unknown material

Grinders:

@andrew-m-leonard andrew-m-leonard self-assigned this Nov 13, 2024
@andrew-m-leonard andrew-m-leonard marked this pull request as draft November 13, 2024 11:33
@andrew-m-leonard andrew-m-leonard marked this pull request as ready for review November 14, 2024 08:43
@llxia
Copy link
Contributor

llxia commented Nov 14, 2024 •
edited
Loading

We need to test other cases (e.g., openj9, vmfarm, zOS, etc.). This PR should not be merged until we test these cases.

smlambert
smlambert requested changes Nov 14, 2024
View reviewed changes
Copy link
Contributor

@smlambert smlambert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Put a temp hold on it to allow @llxia to verify other use cases still function

@llxia llxia self-requested a review November 14, 2024 23:48
@llxia
Copy link
Contributor

llxia commented Nov 14, 2024 •
edited
Loading

Failed on Windows:

00:01:18.078  Uncompressing file: ibm-semeru-certified-jdk_x64_windows_JDK11U_2024-11-13-23-01.zip ...
00:01:44.169  List files in jdkbinary folder...
00:01:44.170  total 300804
00:01:44.170  -rw-r--r--+ 1 jenkins None  68818399 Nov 14 15:41 ibm-semeru-certified-debugimage_x64_windows_JDK11U_2024-11-13-23-01.zip
00:01:44.170  -rw-r--r--+ 1 jenkins None 231312038 Nov 14 15:41 ibm-semeru-certified-jdk_x64_windows_JDK11U_2024-11-13-23-01.zip
00:01:44.170  -rw-r--r--+ 1 jenkins None   7886951 Nov 14 15:41 ibm-semeru-certified-testimage_x64_windows_JDK11U_2024-11-13-23-01.zip
00:01:44.170  drwxr-xr-x+ 1 jenkins None         0 Nov 14 15:41 tmp
00:01:44.170  List files in jdkbinary/tmp folder...
00:01:44.170  total 4
00:01:44.170  drwxrwxr-x+ 1 jenkins None 0 Nov 13 17:06 jdk-11.0.25+9
00:01:44.170  Moving directory jdk-11.0.25+9/ to ../j2sdk-image
00:01:44.170  Uncompressing file: ibm-semeru-certified-testimage_x64_windows_JDK11U_2024-11-13-23-01.zip ...
00:01:45.147  List files in jdkbinary folder...
00:01:45.147  total 300808
00:01:45.147  -rw-r--r--+ 1 jenkins None  68818399 Nov 14 15:41 ibm-semeru-certified-debugimage_x64_windows_JDK11U_2024-11-13-23-01.zip
00:01:45.147  -rw-r--r--+ 1 jenkins None 231312038 Nov 14 15:41 ibm-semeru-certified-jdk_x64_windows_JDK11U_2024-11-13-23-01.zip
00:01:45.147  -rw-r--r--+ 1 jenkins None   7886951 Nov 14 15:41 ibm-semeru-certified-testimage_x64_windows_JDK11U_2024-11-13-23-01.zip
00:01:45.148  drwxrwxr-x+ 1 jenkins None         0 Nov 13 17:06 j2sdk-image
00:01:45.148  drwxr-xr-x+ 1 jenkins None         0 Nov 14 15:41 tmp
00:01:45.148  List files in jdkbinary/tmp folder...
00:01:45.148  total 4
00:01:45.148  drwxrwxr-x+ 1 jenkins None 0 Nov 13 17:03 jdk-11.0.25+9-test-image
00:01:45.148  Moving directory jdk-11.0.25+9-test-image/ to ../openjdk-test-image
00:01:46.209  Uncompressing ibm-semeru-certified-debugimage_x64_windows_JDK11U_2024-11-13-23-01.zip over ./j2sdk-image...
00:01:46.210  unzip:  cannot find or open ibm-semeru-certified-debugimage_x64_windows_JDK11U_2024-11-13-23-01.zip, ibm-semeru-certified-debugimage_x64_windows_JDK11U_2024-11-13-23-01.zip.zip or ibm-semeru-certified-debugimage_x64_windows_JDK11U_2024-11-13-23-01.zip.ZIP.

Unzip is fine without using this PR

@smlambert
Copy link
Contributor

Hi Lan, can you share a public Grinder link (or a rerun in Grinder link from the hyc Jenkins run) so Andrew can test modifications he makes?

@llxia
Copy link
Contributor

llxia commented Nov 15, 2024

Openj9 Windows is fine https://openj9-jenkins.osuosl.org/view/Test/job/Grinder/3982/. This problem happened when using the windows zip from the internal artifactory. I do not think rerun in Grinder link from the hyc Jenkins would help in this case.

@llxia
Copy link
Contributor

llxia commented Nov 15, 2024 •
edited
Loading 

00:03:00.861  Moving directory test-images/ to ../openjdk-test-image
00:03:00.861  Uncompressing debug-image.tar.gz over ./j2sdk-image...
00:03:06.899  
00:03:06.900  gzip: write_error():stdout: EDC5140I Broken pipe.

@andrew-m-leonard
Copy link
Contributor Author

Failed on Windows:

00:01:18.078  Uncompressing file: ibm-semeru-certified-jdk_x64_windows_JDK11U_2024-11-13-23-01.zip ...
00:01:44.169  List files in jdkbinary folder...
00:01:44.170  total 300804
00:01:44.170  -rw-r--r--+ 1 jenkins None  68818399 Nov 14 15:41 ibm-semeru-certified-debugimage_x64_windows_JDK11U_2024-11-13-23-01.zip
00:01:44.170  -rw-r--r--+ 1 jenkins None 231312038 Nov 14 15:41 ibm-semeru-certified-jdk_x64_windows_JDK11U_2024-11-13-23-01.zip
00:01:44.170  -rw-r--r--+ 1 jenkins None   7886951 Nov 14 15:41 ibm-semeru-certified-testimage_x64_windows_JDK11U_2024-11-13-23-01.zip
00:01:44.170  drwxr-xr-x+ 1 jenkins None         0 Nov 14 15:41 tmp
00:01:44.170  List files in jdkbinary/tmp folder...
00:01:44.170  total 4
00:01:44.170  drwxrwxr-x+ 1 jenkins None 0 Nov 13 17:06 jdk-11.0.25+9
00:01:44.170  Moving directory jdk-11.0.25+9/ to ../j2sdk-image
00:01:44.170  Uncompressing file: ibm-semeru-certified-testimage_x64_windows_JDK11U_2024-11-13-23-01.zip ...
00:01:45.147  List files in jdkbinary folder...
00:01:45.147  total 300808
00:01:45.147  -rw-r--r--+ 1 jenkins None  68818399 Nov 14 15:41 ibm-semeru-certified-debugimage_x64_windows_JDK11U_2024-11-13-23-01.zip
00:01:45.147  -rw-r--r--+ 1 jenkins None 231312038 Nov 14 15:41 ibm-semeru-certified-jdk_x64_windows_JDK11U_2024-11-13-23-01.zip
00:01:45.147  -rw-r--r--+ 1 jenkins None   7886951 Nov 14 15:41 ibm-semeru-certified-testimage_x64_windows_JDK11U_2024-11-13-23-01.zip
00:01:45.148  drwxrwxr-x+ 1 jenkins None         0 Nov 13 17:06 j2sdk-image
00:01:45.148  drwxr-xr-x+ 1 jenkins None         0 Nov 14 15:41 tmp
00:01:45.148  List files in jdkbinary/tmp folder...
00:01:45.148  total 4
00:01:45.148  drwxrwxr-x+ 1 jenkins None 0 Nov 13 17:03 jdk-11.0.25+9-test-image
00:01:45.148  Moving directory jdk-11.0.25+9-test-image/ to ../openjdk-test-image
00:01:46.209  Uncompressing ibm-semeru-certified-debugimage_x64_windows_JDK11U_2024-11-13-23-01.zip over ./j2sdk-image...
00:01:46.210  unzip:  cannot find or open ibm-semeru-certified-debugimage_x64_windows_JDK11U_2024-11-13-23-01.zip, ibm-semeru-certified-debugimage_x64_windows_JDK11U_2024-11-13-23-01.zip.zip or ibm-semeru-certified-debugimage_x64_windows_JDK11U_2024-11-13-23-01.zip.ZIP.

Unzip is fine without using this PR

@llxia thank you Lan, I see the issue

@andrew-m-leonard
Copy link
Contributor Author

andrew-m-leonard commented Nov 15, 2024 •
edited
Loading

Re-test Windows using upstream artifacts: https://ci.adoptium.net/job/Grinder/11404

@andrew-m-leonard
Copy link
Contributor Author

andrew-m-leonard commented Nov 15, 2024 •
edited
Loading

Re-test s390x upstream artifacts: https://ci.adoptium.net/job/Grinder/11390/ - SUCCESS

@andrew-m-leonard
Copy link
Contributor Author 

00:03:00.861  Moving directory test-images/ to ../openjdk-test-image
00:03:00.861  Uncompressing debug-image.tar.gz over ./j2sdk-image...
00:03:06.899  
00:03:06.900  gzip: write_error():stdout: EDC5140I Broken pipe.

* [rerun](https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder_testList_0/2344/) on the same machine fyrec21g - failed with the same error

* [rerun without using this PR](https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder_testList_0/2345/) on the same machine fyrec21g - no `EDC5140I Broken pipe` error

Hi Lan,
I've fixed the zip problem, but the zOS gzip "Broken pipe" I can't see any obvious issue,
please can you retry on zOS with and without the PR ?
thanks

@llxia
Copy link
Contributor

llxia commented Nov 15, 2024 •
edited
Loading

sxa
sxa reviewed Nov 15, 2024
View reviewed changes
get.sh Show resolved Hide resolved
sxa
sxa reviewed Nov 15, 2024
View reviewed changes
get.sh Show resolved Hide resolved
sxa
sxa reviewed Nov 18, 2024
View reviewed changes
get.sh Outdated Show resolved Hide resolved
@andrew-m-leonard andrew-m-leonard marked this pull request as draft November 18, 2024 14:16
@andrew-m-leonard
Copy link
Contributor Author

andrew-m-leonard commented Nov 18, 2024 •
edited
Loading

Grinder on all platforms using "upstream" job output:

@andrew-m-leonard andrew-m-leonard marked this pull request as ready for review November 19, 2024 09:49
@andrew-m-leonard andrew-m-leonard requested a review from sxa November 19, 2024 09:49
@andrew-m-leonard
Copy link
Contributor Author

Thank you @llxia

sxa
sxa approved these changes Nov 19, 2024
View reviewed changes
Copy link
Member

@sxa sxa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving, but with a number of suggested updates for clarity

get.sh Outdated Show resolved Hide resolved
get.sh Outdated Show resolved Hide resolved
get.sh Outdated Show resolved Hide resolved
get.sh Outdated Show resolved Hide resolved
@andrew-m-leonard
Copy link
Contributor Author

andrew-m-leonard commented Nov 19, 2024 •
edited
Loading

Sanity check with latest review updates: https://ci.adoptium.net/job/Grinder/11496/ - SUCCESS

@andrew-m-leonard
Copy link
Contributor Author

I think we're good now @ShelleyLambert ?

@sxa
Copy link
Member

sxa commented Nov 19, 2024

I think we're good now @ShelleyLambert ?

I think you want @smlambert :-)

@smlambert
Copy link
Contributor

@llxia - some additional changes have been made to this PR, if you wish to rerun a couple of your Grinders to verify the last of these changes are good, please do so. I will lift the block on this PR once I get the thumbs up from you, as it LGTM.

@llxia
Copy link
Contributor

llxia commented Nov 19, 2024 •
edited
Loading

windows - passed

vmfarm xlinux - unzip passed

vmfarm windows - unzip passed

openj9 windows - passed

zos fyrec21g - passed

zos - passed

smlambert
smlambert approved these changes Nov 20, 2024
View reviewed changes
Copy link
Contributor

@smlambert smlambert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks @andrew-m-leonard and all 👍

@smlambert smlambert merged commit 2c34c05 into adoptium:master Nov 20, 2024
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@llxia llxia llxia approved these changes

@sxa sxa sxa approved these changes

@smlambert smlambert smlambert approved these changes

@karianna karianna karianna approved these changes

@sophia-guo sophia-guo Awaiting requested review from sophia-guo

Assignees

@andrew-m-leonard andrew-m-leonard

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

get.sh getjdkbinary() not correct uncompressing temurin debug-image and does not handle static-libs archive
5 participants
@andrew-m-leonard @llxia @smlambert @sxa @karianna