Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(Tests pending) TOOL-11960 Install ca-certifcates to delphix-platform such that openjdk11 can install #317

Closed
wants to merge 1 commit into from

Conversation

blim747
Copy link

@blim747 blim747 commented Jul 30, 2021

Background:

As part of TOOL-11707, we are trying to install Java 11 on the DCoL hosts because the Jenkins project is planning on deprecating Java 8 support in the near future. DCoL is used as a Jenkins agent.

Problem:

When testing TOOL-11707, I tried to install openjdk11 but I kept receiving this error:

18:37:05  task path: /var/tmp/jenkins/workspace/devops-gate/master/appliance-build-stage1/post-push/appliance-build/live-build/misc/ansible-roles/appliance-build.dcenter/tasks/main.yml:51
18:38:15  FAILED - RETRYING: appliance-build.dcenter : apt (3 retries left).
18:39:53  FAILED - RETRYING: appliance-build.dcenter : apt (2 retries left).
18:41:32  FAILED - RETRYING: appliance-build.dcenter : apt (1 retries left).
18:43:26  fatal: <binary>: FAILED! => {"attempts": 3, "cache*update_time": 1627609377, "cache_updated": true, "changed": false, "msg": "'/usr/bin/apt-get -y -o \"Dpkg::Options::=--force-confdef\" -o \"Dpkg::Options::=--force-confold\"      install 'openjdk-11-jdk-headless'' failed: E: Can not write log (Is /dev/pts mounted?) - posix_openpt (2: No such file or directory)\nhead: cannot open '/etc/ssl/certs/java/cacerts' for reading: No such file or directory\nthe keytool command requires a mounted proc fs (/proc).\ndpkg: error processing package ca-certificates-java (--configure):\n installed ca-certificates-java package post-installation script subprocess returned error exit status 1\ndpkg: dependency problems prevent configuration of openjdk-11-jre-headless:amd64:\n openjdk-11-jre-headless:amd64 depends on ca-certificates-java; however:\n  Package ca-certificates-java is not configured yet.\n\ndpkg: error processing package openjdk-11-jre-headless:amd64 (--configure):\n dependency problems - leaving unconfigured\ndpkg: dependency problems prevent configuration of openjdk-11-jdk-headless:amd64:\n openjdk-11-jdk-headless:amd64 depends on openjdk-11-jre-headless (= 11.0.11<ins>9-0ubuntu2<sub>18.04); however:\n  Package openjdk-11-jre-headless:amd64 is not configured yet.\n\ndpkg: error processing package openjdk-11-jdk-headless:amd64 (--configure):\n dependency problems - leaving unconfigured\nthe keytool command requires a mounted proc fs (/proc).\nErrors were encountered while processing:\n ca-certificates-java\n openjdk-11-jre-headless:amd64\n openjdk-11-jdk-headless:amd64\nE: Sub-process /usr/bin/dpkg returned an error code (1)\n", "rc": 100, "stderr": "E: Can not write log (Is /dev/pts mounted?) - posix_openpt (2: No such file or directory)\nhead: cannot open '/etc/ssl/certs/java/cacerts' for reading: No such file or directory\nthe keytool command requires a mounted proc fs (/proc).\ndpkg: error processing package ca-certificates-java (--configure):\n installed ca-certificates-java package post-installation script subprocess returned error exit status 1\ndpkg: dependency problems prevent configuration of openjdk-11-jre-headless:amd64:\n openjdk-11-jre-headless:amd64 depends on ca-certificates-java; however:\n  Package ca-certificates-java is not configured yet.\n\ndpkg: error processing package openjdk-11-jre-headless:amd64 (--configure):\n dependency problems - leaving unconfigured\ndpkg: dependency problems prevent configuration of openjdk-11-jdk-headless:amd64:\n openjdk-11-jdk-headless:amd64 depends on openjdk-11-jre-headless (= 11.0.11</ins>9-0ubuntu2</sub>18.04); however:\n  Package openjdk-11-jre-headless:amd64 is not configured yet.\n\ndpkg: error processing package openjdk-11-jdk-headless:amd64 (--configure):\n dependency problems - leaving unconfigured\nthe keytool command requires a mounted proc fs (/proc).\nErrors were encountered while processing:\n ca-certificates-java\n openjdk-11-jre-headless:amd64\n openjdk-11-jdk-headless:amd64\nE: Sub-process /usr/bin/dpkg returned an error code (1)\n", "stderr_lines": <"E: Can not write log (Is /dev/pts mounted?) - posix_openpt (2: No such file or directory)", "head: cannot open '/etc/ssl/certs/java/cacerts' for reading: No such file or directory", "the keytool command requires a mounted proc fs (/proc).", "dpkg: error processing package ca-certificates-java (--configure):", " installed ca-certificates-java package post-installation script subprocess returned error exit status 1", "dpkg: dependency problems prevent configuration of openjdk-11-jre-headless:amd64:", " openjdk-11-jre-headless:amd64 depends on ca-certificates-java; however:", "  Package ca-certificates-java is not configured yet.", "", "dpkg: error processing package openjdk-11-jre-headless:amd64 (--configure):", " dependency problems - leaving unconfigured", "dpkg: dependency problems prevent configuration of openjdk-11-jdk-headless:amd64:", " openjdk-11-jdk-headless:amd64 depends on openjdk-11-jre-headless (= 11.0.11<ins>9-0ubuntu2<sub>18.04); however:", "  Package openjdk-11-jre-headless:amd64 is not configured yet.", "", "dpkg: error processing package openjdk-11-jdk-headless:amd64 (--configure):", " dependency problems - leaving unconfigured", "the keytool command requires a mounted proc fs (/proc).", "Errors were encountered while processing:", " ca-certificates-java", " openjdk-11-jre-headless:amd64", " openjdk-11-jdk-headless:amd64", "E: Sub-process /usr/bin/dpkg returned an error code (1)">, "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nopenjdk-11-jdk-headless is already the newest version (11.0.11</ins>9-0ubuntu2</sub>18.04).\n0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.\n3 not fully installed or removed.\nAfter this operation, 0 B of additional disk space will be used.\nSetting up ca-certificates-java (20180516ubuntu1<sub>18.04.1) ...\nProcessing triggers for ca-certificates (20210119</sub>18.04.1) ...\nUpdating certificates in /etc/ssl/certs...\n0 added, 0 removed; done.\nRunning hooks in /etc/ca-certificates/update.d...\n\nE: /etc/ca-certificates/update.d/jks-keystore exited with code 1.\ndone.\n", "stdout*lines": <"Reading package lists...", "Building dependency tree...", "Reading state information...", "openjdk-11-jdk-headless is already the newest version (11.0.11+9-0ubuntu2<sub>18.04).", "0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.", "3 not fully installed or removed.", "After this operation, 0 B of additional disk space will be used.", "Setting up ca-certificates-java (20180516ubuntu1</sub>18.04.1) ...", "Processing triggers for ca-certificates (20210119<sub>18.04.1) ...", "Updating certificates in /etc/ssl/certs...", "0 added, 0 removed; done.", "Running hooks in /etc/ca-certificates/update.d...", "", "E: /etc/ca-certificates/update.d/jks-keystore exited with code 1.", "done.">}

As @basil mentions in this GitHub PR comment, adding ca-certificates-java to the delphix-platform package list should resolve this.

The current workaround is to install adoptopenjdk-java11-jdk but this install is Linux distribution dependent (we plan on upgrading as part of TOOL-7726) and also requires setting apt keys and repository, a bit of a maintenance burden.

Solution:

Add the package. Test the change for this in conjunction with the change for TOOL-11707, which is being reviewed in this appliance-build PR (ID 576).

Testing:

appliance-build-orchestrator-pre-push job: http://ops.jenkins.delphix.com/job/devops-gate/job/master/job/appliance-build-orchestrator-pre-push/1078/console

See next comment: #317 (comment)

Related tickets:

  • TOOL-11707 PR, adding Java 11 to internal-dcenter appliance-build
  • TOOL-11962: since ca-certificates-java is only being added to ESX and internal-dcenter will require this package, it makes sense to only build the internal-dcenter variant for ESX. This will also moderately decrease the number of appliance-build-stage1 post-push jobs from 42 (6 variants * 7 platforms) to 36 (5 variants * 7 platforms + 1 internal-dcenter for ESX)

@blim747
Copy link
Author

blim747 commented Jul 30, 2021

Happy to say testing is no longer pending. @basil's suggestion to install ca-certificates-java in delphix-platform worked:

delphix@blim-t11707-try3:/usr/lib/jvm$ ls
adoptopenjdk-java8-jdk-amd64  java-1.11.0-openjdk-amd64  java-11-openjdk-amd64
delphix@blim-t11707-try3:/usr/lib/jvm$ java -version
openjdk version "1.8.0_282"
OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_282-b08)
OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.282-b08, mixed mode)
delphix@blim-t11707-try3:/usr/lib/jvm$ sudo update-alternatives --display java
java - manual mode
  link best version is /usr/lib/jvm/java-11-openjdk-amd64/bin/java
  link currently points to /usr/lib/jvm/adoptopenjdk-java8-jdk-amd64/bin/java
  link java is /usr/bin/java
  slave java.1.gz is /usr/share/man/man1/java.1.gz
/usr/lib/jvm/adoptopenjdk-java8-jdk-amd64/bin/java - priority 318
/usr/lib/jvm/java-11-openjdk-amd64/bin/java - priority 1111
  slave java.1.gz: /usr/lib/jvm/java-11-openjdk-amd64/man/man1/java.1.gz
delphix@blim-t11707-try3:/usr/lib/jvm$

appliance-build-orchestrator-pre-push job: http://ops.jenkins.delphix.com/job/devops-gate/job/master/job/appliance-build-orchestrator-pre-push/1078/console
The post-push job that built internal-dcenter-esx.ova: http://ops.jenkins.delphix.com/job/devops-gate/job/master/job/appliance-build-stage1/job/post-push/116208/, the build lives at s3://dev-de-images/builds/jenkins-ops/devops-gate/master/appliance-build/master/pre-push/845/internal-dcenter-esx/internal-dcenter-esx.ova on S3
Due to the internal-dev variant taking a while to build, I manually imported the image into a DCoL VM here: http://collins.d.delphix.com:28565/job/devops-gate/job/tool-11959-default-esx-credential/job/sync-ova-into-dcenter/1/console

@blim747 blim747 marked this pull request as ready for review July 30, 2021 22:36
@blim747 blim747 changed the title [Testing pending] TOOL-11960 Install ca-certifcates to delphix-platform such that openjdk11 can install TOOL-11960 Install ca-certifcates to delphix-platform such that openjdk11 can install Jul 30, 2021
debian/rules Outdated
@@ -93,7 +93,7 @@ DEPENDS += pam-challenge-response, \
# Platform-specific dependencies
DEPENDS.aws = nvme-cli,
DEPENDS.azure = walinuxagent,
DEPENDS.esx = open-vm-tools,
DEPENDS.esx = open-vm-tools, ca-certificates-java,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this package specific to ESX? IMO, we should try to minimize the differences between the various cloud platforms, so I'd like to install this package in all cases, rather than specific to ESX, if possible.

Copy link
Author

@blim747 blim747 Aug 2, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This package is only consumed by a variant that is only used by ESX. As discussed in this Slack thread in #dcol-dev, this package is only consumed by the internal-dcenter variant, whose only consumer is ESX. To keep package installations as minimal as possible I decided to only install it on ESX. In addition, as part of TOOL-11962 I plan to deprecate the ability to build the internal-dcenter variant for any platform aside from ESX. Doing so will decrease the number of appliance-build-stage1 jobs from 42 (6 variants * 7 platforms) to 36 (5 variants * 7 platforms), a 14% decrease (which saves time and resources on wasted builds for variant-platform combos that aren't useful)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To keep package installations as minimal as possible I decided to only install it on ESX.

Please correct me if I'm mistaken, but I interpret this to mean that we could install this package on all platforms without adverse effects. Thus, my preference would be to do that. IMO, I value consistency of the product across the cloud platforms, even when that isn't strictly necessary, than intentional inconsistency to avoid installing packages on platforms that may not need the package.

With that said, I'm not sure what our "official" stance is w.r.t. cloud platform consistency, so others may disagree.

In addition, as part of TOOL-11962 I plan to deprecate the ability to build the internal-dcenter variant for any platform aside from ESX.

Sure, that's fine, but tangential to this specific issue/PR. For example, this new package will not be installed on non-dcenter variants, so I just want to make sure this change is "correct" for the non-dcenter cases as well.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd tend to agree with Prakash on that. Unless ca-certificates-java causes some unwanted side-effects on other platforms or takes an excessive amount of space I'd say it's preferable to install it on all platforms. Moreover, given that most of our automated testing is run on AWS, if that package does cause some side effects we'd be more likely to catch it this way.

Copy link
Author

@blim747 blim747 Aug 2, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you both for your feedback. I'll install the package on all platforms and re-build with all testing (I am unable to run tests on internal-dcenter variant as show in this attempted job)

@blim747 blim747 force-pushed the t11960-add-ca-certificate branch from bb80018 to 783084c Compare August 2, 2021 15:40
Copy link
Contributor

@prakashsurya prakashsurya left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, pending testing. Please get a successful git-ab-pre-push run with this change, in addition to any dcenter specific testing.

@blim747
Copy link
Author

blim747 commented Aug 2, 2021

Thanks for the review. Here's the appliance-build-orchestrator job: http://ops.jenkins.delphix.com/job/devops-gate/job/master/job/appliance-build-orchestrator-pre-push/1089/console, it's still running. As mentioned in my edited reply here above I cannot run the dx-integration-tests with the internal-dcenter variant.

@blim747 blim747 changed the title TOOL-11960 Install ca-certifcates to delphix-platform such that openjdk11 can install (Tests pending) TOOL-11960 Install ca-certifcates to delphix-platform such that openjdk11 can install Aug 2, 2021
@prakashsurya
Copy link
Contributor

I'm requesting a clean run of git-ab-pre-push without any additional parameters, and with only this change; that will ensure we test the non-dcenter case(s).

@blim747
Copy link
Author

blim747 commented Aug 2, 2021

@prakashsurya thanks for double checking my test and it makes sense, here's the job kicked off from git ab-pre-push: http://selfservice.jenkins.delphix.com/job/devops-gate/job/master/job/appliance-build-orchestrator-pre-push/5820/parameters/

There were no arguments except -a to allow for duplicate jobs testing the same branch as discovered in this 2019 #platform Slack post. I'll still be running the previously kicked off test as well for the internal-dcenter variant specifically.

@blim747
Copy link
Author

blim747 commented Aug 4, 2021

(copied from this JIRA comment for TOOL-11960)

This causes a regression. Java 11 is a dependency of ca-certificates-java which then causes the delphix-mgmt stack to fail to startup because it is using Java 11 rather than Java 8. Where the bug was found is in this #platform Slack thread and this blackbox-self-service job. The root cause was found in this reply by John Gallagher

The new approach is to install ca-certifcates-java to the appliance-build's live-build/config/package-lists as described in this #dcol-dev Slack thread. This then requires variant-specific package lists for the appliance-build live-build, which is raised in this appliance-build Github issue

@blim747 blim747 closed this Aug 4, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

3 participants