Add JDK21 agents (build)

[dduportal]

Goal: provide an agent template on each controller (where needed) allowing users to have a default JDK21 when running their pipelines, regardless of the JDK used for running the agent (see #4121 about agent JDK runtime)

• Developers can use build tools as an alternative but usage of pipeline library is preferred

Task list:

• Provide JDK21 in agent templates
  • All in one template
    • JDK21 EA available since https://github.com/jenkins-infra/packer-images/releases/tag/1.20.0 (ref. Add JDK21 support for developers to pre-built and try #3709)
    • JDK21 LTS available since https://github.com/jenkins-infra/packer-images/releases/tag/1.32.0
  • Windows inbound containers - since feat(maven-jdk21-nanoserver): adding jdk21 to windows container docker-inbound-agents#98 (ref. Add JDK21 support for developers to pre-built and try #3709)
• ci.jenkins.io
  • Add a Linux Container template maven-21 since Add JDK21 support for developers to pre-built and try #3709
  • Windows Container template (ACI) maven-21 since Add JDK21 support for developers to pre-built and try #3709
• trusted.ci.jenkins.io
• cert.ci.jenkins.io
• infra.ci.jenkins.io

[dduportal]

Update: #4122 indicates that we need to find a solution for trusted.ci.jenkins.io:

• We use Azure VM template which allows running an initialization script when starting the agent process but not setting custom environment variables for the agent process
  • Let's try setting up the JAVA_HOME to JDK21 on a test template => if it work, we can define maven-xx labels on trusted.ci to ensure it stays coherent with ci.jio.
  • Otherwise, we'll have to update the pipelines so they define their own env. var JAVA_HOME.
• Note: we'll have to do the same with EC2 in the future though (once we exhaust Azure credits and start using AWS credits)

[dduportal]

Update: #4122 indicates that we need to find a solution for trusted.ci.jenkins.io:

* We use Azure VM template which allows running an initialization script when starting the agent process but not setting custom environment variables for the agent process
  
  * Let's try setting up the JAVA_HOME to JDK21 on a test template => if it work, we can define `maven-xx` labels on trusted.ci to ensure it stays coherent with ci.jio.
  * Otherwise, we'll have to update the pipelines so they define their own env. var JAVA_HOME.

* Note: we'll have to do the same with EC2 in the future though (once we exhaust Azure credits and start using AWS credits)

Tested with success using Linux Azure VM agents:

• When using inbound mode (such as in ci.jenkins.io) we already have logic in the agent systemD unit file defined in https://github.com/jenkins-infra/jenkins-infra/blob/dc673c579eddc81c604638b53b2e6d1f506a0dd7/dist/profile/templates/jenkinscontroller/casc/clouds.yaml.erb#L111C26-L113 => we only have to update the template. Tested with success with the code (JCasc excerpt) below:

agentWorkspace: "/home/jenkins/agent"
builtInImage: "Ubuntu 22.04 LTS"
credentialsId: "jenkinsvmagents-userpass"
diskType: "managed"
doNotUseMachineIfInitFails: true
encryptionAtHost: true
ephemeralOSDisk: true
executeInitScriptAsRoot: true
existingStorageAccountName: "cijenkinsioagentssub"
imageReference:
  galleryImageDefinition: "jenkins-agent-ubuntu-22.04-amd64"
  galleryImageVersion: "1.70.1"
  galleryName: "prod_packer_images"
  galleryResourceGroup: "prod-packer-images"
  gallerySubscriptionId: "<redacted>"
imageTopLevelType: "advanced"
initScript: |
  #!/bin/sh
  set -eux

  # Setup Datadog service
  (
    systemctl stop datadog-agent.service
    mkdir -p /var/log/datadog /etc/datadog-agent
    sed 's/api_key:.*/api_key: <redacted>/' /etc/datadog-agent/datadog.yaml.example > /etc/datadog-agent/datadog.yaml
    sed -i 's/# site:.*/site: datadoghq.com/' /etc/datadog-agent/datadog.yaml
    chown dd-agent:dd-agent /etc/datadog-agent/datadog.yaml
    chmod 640 /etc/datadog-agent/datadog.yaml
    chown dd-agent:dd-agent /var/log/datadog
    chmod 770 /var/log/datadog
    systemctl daemon-reload
    systemctl enable datadog-agent.service
    systemctl start datadog-agent.service
  ) 2>&1 | tee /var/log/agent-init-datadog.log
  # Setup Jenkins Agent Service
  (
    # Argument provided by the Azure-VM plugin
    export JENKINS_URL="^${1}" # Always ends with a '/'
    export AGENT_NAME="^${2}"
    export AGENT_SECRET="^${3}"

    export USER=jenkins
    export AGENT_WORKDIR='/home/jenkins/agent'
    export AGENT_JAR="^${AGENT_WORKDIR}/agent.jar"
    export AGENT_SECRETFILE="^${AGENT_WORKDIR}/agent-secret"
    export AGENT_URL="^${JENKINS_URL}computer/^${AGENT_NAME}/jenkins-agent.jnlp"
    export JENKINS_JAVA_OPTS='-XX:+PrintCommandLineFlags'
    export ARTIFACT_CACHING_PROXY_PROVIDER='azure'
    export JAVA_HOME='/opt/jdk-21'
    export JENKINS_JAVA_BIN='/opt/jdk-17/bin/java'

    mkdir -p "^${AGENT_WORKDIR}"
    chown "^${USER}:^${USER}" "^${AGENT_WORKDIR}"
    curl --silent --show-error --location --output "^${AGENT_JAR}" "^${JENKINS_URL}jnlpJars/agent.jar"
    touch "^${AGENT_SECRETFILE}"
    echo "^${AGENT_SECRET}" > "^${AGENT_SECRETFILE}"
    cat <<- EOF >/etc/systemd/system/jenkins-agent.service
    [Unit]
    Description=Jenkins Inbound Agent
    Wants=network.target
    After=network.target

    [Service]
    ExecStart=^${JENKINS_JAVA_BIN} ^${JENKINS_JAVA_OPTS} -jar ^${AGENT_JAR} -jnlpUrl ^${AGENT_URL} -secret @^${AGENT_SECRETFILE} -workDir ^${AGENT_WORKDIR}
    User=^${USER}
    WorkingDirectory=^${AGENT_WORKDIR}
    Restart=on-failure
    RestartSec=10
    Environment="JAVA_HOME=^${JAVA_HOME}"
    Environment="ARTIFACT_CACHING_PROXY_PROVIDER=^${ARTIFACT_CACHING_PROXY_PROVIDER}"
    Environment="PATH=^${JAVA_HOME}/bin:/home/jenkins/.asdf/shims:/home/jenkins/.asdf/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"

    [Install]
    WantedBy=multi-user.target
  EOF

    systemctl daemon-reload
    systemctl enable jenkins-agent
    systemctl start jenkins-agent || systemctl status jenkins-agent
  ) 2>&1 | tee /var/log/agent-init-jenkins.log

  # Remove jenkins user from sudoers
  rm -f /etc/sudoers.d/90-cloud-init-users
labels: "vm-maven-21"
launcher: "inbound"
licenseType: "Classic"
location: "East US 2"
maxVirtualMachinesLimit: 5
maximumDeploymentSize: 10
noOfParallelJobs: 1
osDiskSize: 150
osType: "Linux"
retentionStrategy: "azureVMCloudOnce"
storageAccountNameReferenceType: "existing"
storageAccountType: "Standard_LRS"
subnetName: "public-jenkins-sponsorship-vnet-ci_jenkins_io_agents"
templateDesc: "Test by DDU to have dynamically provisioned Ubuntu 22.04 LTS\
  \ machine using JDK21 for build tools"
templateName: "ubuntu-22-jdk21-ddu"
usageMode: NORMAL
usePrivateIP: true
virtualMachineSize: "Standard_D4ads_v5"
virtualNetworkName: "public-jenkins-sponsorship-vnet"
virtualNetworkResourceGroupName: "public-jenkins-sponsorship"

• Also a success with the SSH launcher (used in trusted.ci for instance instead of inbound). The trick was to add (in the init script code) the same instructions we use in packer-image in https://github.com/jenkins-infra/packer-images/blob/96b1ac213f5c653f695b62389a4c759e60fa8985/provisioning/ubuntu-provision.sh#L364-L365 but setting the default JDK with an even higher priority to override it:

export DEFAULT_JDK=/opt/jdk-21
# 2000 is higher than the 1000 provided in template
update-alternatives --install /usr/bin/java java "/opt/jdk-${DEFAULT_JDK}/bin/java" 2000

# Last line: takes precedence over the already defined `JAVA_HOME`
echo "JAVA_HOME=/opt/jdk-${DEFAULT_JDK}" >> /etc/environment

[dduportal]

Update: #4122 indicates that we need to find a solution for trusted.ci.jenkins.io:

* We use Azure VM template which allows running an initialization script when starting the agent process but not setting custom environment variables for the agent process
  
  * Let's try setting up the JAVA_HOME to JDK21 on a test template => if it work, we can define `maven-xx` labels on trusted.ci to ensure it stays coherent with ci.jio.
  * Otherwise, we'll have to update the pipelines so they define their own env. var JAVA_HOME.

* Note: we'll have to do the same with EC2 in the future though (once we exhaust Azure credits and start using AWS credits)

Tested with success using Linux Azure VM agents:

=> we can start the work to define maven-17 and maven-21 labels for Linux Ubuntu VM templates on trusted.ci and cert.ci immediately (cc @smerle33 )
=> With Windows VM, tried the inbound launcher without success: I'm missing how to update env. vars in the current working session (e.g. in the init script): the service launcher does not seem to pick my value (ping @MarkEWaite @timja have you already done this kind of thing?). Ref. init powershell code in https://github.com/jenkins-infra/jenkins-infra/blob/dc673c579eddc81c604638b53b2e6d1f506a0dd7/dist/profile/templates/jenkinscontroller/casc/clouds.yaml.erb#L24-L50 which starts the inbound service stuff

[timja]

Also a success with the SSH launcher

You can also simply set the javaPath variable in JCasC for SSH launcher

inbound launcher without success for Windows

Your code looks fine, I'd login to the agent and cat the file to see its been replaced appropriately and check the logs for the service (along with trying to start it manually if needed)

[dduportal]

Also a success with the SSH launcher

You can also simply set the javaPath variable in JCasC for SSH launcher

We already do this (ref. https://github.com/jenkins-infra/jenkins-infra/blob/dc673c579eddc81c604638b53b2e6d1f506a0dd7/dist/profile/templates/jenkinscontroller/casc/clouds.yaml.erb#L153).
But the JCasC javaPath attribute is only used to specify a JDK binary for the agent runtime (will be tracked in #4121).

Here, we want the set the default java available for builds, which is set by 2 variables:

• Set up or override JAVA_HOME (picked by Maven for instance)
• Update PATH to have JAVA_HOME/bin so a call to java (or java.exe) would resolve to the expected Java

On Kubernetes plugin or EC2 plugin, we define the environment variables in the Jenkins Cloud config UI and the variables are applied to the agent.jar process when it starts.
But with AzureVM, this feature does not exist as far as I can tell: we rely on the custom init script instead.

On Linux, we have solutions due to the init script using SystemD (inbound launcher) or update-alternative (Ubuntu) to solve the problem => no complicated env. loading path to manage.

But on Windows I'm not sure which solution to take: we usually set up the registry to have the env. vars setup which requires a reboot/restart/reload/reloggin for this change to be taken in account. As we tend to use packer and Docker, this problem do not exist (we build the image, and then when it is instanciated, the new env. is loaded).
=> in the case here, the init script of the AzureVM plugin does not re-log/reload so the new env. var values are not picked up by the agent.jar process.

[timja]

Have a look at this for Windows: https://github.com/winsw/winsw/blob/v3/docs/xml-config-file.md

[dduportal]

Have a look at this for Windows: https://github.com/winsw/winsw/blob/v3/docs/xml-config-file.md

Coool, that will do the trick!

[dduportal]

• [infra.ci.jenkins.io] Azure VM agent init script not in sync with Puppet code #4244 opened to fix an infra.ci error found while deploying the Linux agents
• Azure Windows SSH agents:
  • Adding variables during the initScript of an Azure SSH agent behaves as expected: JAVA_HOME works well and Maven uses the correct Java
  • However OpenSSH server seems to filter the Path variable: we can add to it but we cannot remove the existing elements from it. As such, I would want to remove defining a default JDK in the Windows packer image (https://github.com/jenkins-infra/packer-images/blob/bc253d92b0d51d20f7b77ff8ed1cf7f25ab171db/provisioning/windows-provision.ps1#L300) and only define it during the init script

[dduportal]

Update: no solution works to change the Path variables with SSH Windows agent:

• Tried changing Path on the 3 levels (Machine, User, Process)
• Tried restarting SSH server as part of the init script
• Tried the setx commands
• Tried changing the OpenSSH and restart

=> The initScript (not cloud init!) being executed in the context of a SSH session, I fear the only way is to change the template setup.

As such, let's remove any definition of a default JDK on the jenkins-infra/packer-images.
We'll let @jayfranco999 work on this

[jayfranco999]

Update: BREAKING CHANGE: removing the default_jdk env variable will introduce failure in goss tess harness for windows SSH agents and will result in build failures.

As per #4124 (comment) :

Any definition of a 'default JDK' and 'update-alternatives' on the jenkins-infra/packer-images has been removed from the ubuntu-provisioning script. The corresponding goss test patch has also been added to the goss test harness to check installation of maven.

Next steps are to introduce the same feature parity for the windows SSH agents with updated goss test harness for it.

[jayfranco999]

Update:

BREAKING CHANGE: All definitions of Default JDK has been removed from windows templates on jenkins-infra/packer-images.

As per #4124 (comment) :

If the solution holds good, we can proceed to introduce feature parity in the goss-common tests for both Linux and Windows SSH agents.

[jayfranco999]

Update: Created an updatecli PR that tracks and updates the end-dates every 90 days for the Azure credentials used by packer-resources.

This is required for the release of packer-images 2.0.0 (without default_jdk definitions) since the build fails due to expired ASP credentials.

[jayfranco999]

Update: jenkins-infra/jenkins-infra#3656

packer-images version 2.0.0 is now being used.

Added init script to handle javaHome path for Windows SSH agents since default_jdk is no longer used.

Added the following windows SSH AZ VM templates:

cert.ci:
win-2019-maven-17, win-2019-maven-21

trusted.ci
win-2019-maven-17, win-2019-maven-21
win-2022-maven-17, win-2022-maven-21

Fixed labels for windows maven-11 templates.

[dduportal]

Update:

• ci.jio, cert.ci.jio and trusted.cio now all have JDK11, JDK17 and JDK21 agents for both Linux and Windows, with label parity 🥳
  • feat(cert.ci&trusted.ci):Added AZ SSH VM templates for win-2019 & win-2022 jenkins-infra#3656
  • feat(controllers) ensure the jdk and maven labels are present on all controllers jenkins-infra#3659

@jayfranco999 there is nothing to do on infra.ci for the Windows agents. We use them only for Docker, so no need to clutter the configuration. We'll switch the existing Windows templates (on infra.ci) from JDK11 to JDK21 as part of #4127