Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jenkins maintenance #1299

Merged
merged 29 commits into from
Sep 2, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@

### Changed

- Jenkins maintenance ([#1299](https://github.com/opendevstack/ods-core/pull/1299)) and update java version in Jenkins ([#1295](https://github.com/opendevstack/ods-core/issues/1295))

### Fixed

## [4.5.1] - 2024-07-17
Expand Down
14 changes: 7 additions & 7 deletions configuration-sample/ods-core.env.sample
Original file line number Diff line number Diff line change
Expand Up @@ -217,10 +217,10 @@ CONFLUENCE_URL=http://192.168.56.31:8090
# For UBI8-based images (OpenShift 4):
# - RHEL variant: https://catalog.redhat.com/software/containers/ocp-tools-4/jenkins-rhel8/5fe1f38288e9c2f788526306
# - Example: registry.redhat.io/ocp-tools-4/jenkins-rhel8:v4.14.0
# - Last tested: registry.redhat.io/ocp-tools-4/jenkins-rhel8:v4.14.0-1706517686
# - Last tested: registry.redhat.io/ocp-tools-4/jenkins-rhel8:v4.14.0-1723454631
# - Community variant: https://quay.io/repository/openshift/origin-jenkins?tab=tags
# - Example: quay.io/openshift/origin-jenkins:4.6
JENKINS_MASTER_BASE_FROM_IMAGE=registry.redhat.io/ocp-tools-4/jenkins-rhel8:v4.14.0-1706517686
JENKINS_MASTER_BASE_FROM_IMAGE=registry.redhat.io/ocp-tools-4/jenkins-rhel8:v4.14.0-1723454631

# Dockerfile to use for Jenkins master.
# Use "Dockerfile.ubi8" for both OpenShift 3.11 and 4 (UBI8 base image)
Expand All @@ -230,10 +230,10 @@ JENKINS_MASTER_DOCKERFILE_PATH=Dockerfile.ubi8
# For UBI8-based images (OpenShift 4):
# - RHEL variant: https://catalog.redhat.com/software/containers/ocp-tools-4/jenkins-agent-base-rhel8/6241e3457847116cf8577aea
# - Example: registry.redhat.io/ocp-tools-4/jenkins-agent-base-rhel8:v4.14.0
# - Last tested: registry.redhat.io/ocp-tools-4/jenkins-agent-base-rhel8:v4.14.0-1706516367
# - Last tested: registry.redhat.io/ocp-tools-4/jenkins-agent-base-rhel8:v4.14.0-1723453106
# - Community variant: https://quay.io/repository/openshift/origin-jenkins-agent-base?tab=tags
# - Example: quay.io/openshift/origin-jenkins-agent-base:4.6
JENKINS_AGENT_BASE_FROM_IMAGE=registry.redhat.io/ocp-tools-4/jenkins-agent-base-rhel8:v4.14.0-1706516367
JENKINS_AGENT_BASE_FROM_IMAGE=registry.redhat.io/ocp-tools-4/jenkins-agent-base-rhel8:v4.14.0-1723453106

# Dockerfile to use for Jenkins agents.
# Use "Dockerfile.ubi8" for both OpenShift 3.11 and 4 (UBI8 base image)
Expand All @@ -242,16 +242,16 @@ JENKINS_AGENT_DOCKERFILE_PATH=Dockerfile.ubi8
# Snyk CLI binary distribution url
# Leave empty to avoid installing Snyk.
# Releases are published at https://github.com/snyk/snyk/releases.
# Latest tested version is v1.1284.0.
JENKINS_AGENT_BASE_SNYK_DISTRIBUTION_URL=https://github.com/snyk/snyk/releases/download/v1.1284.0/snyk-linux
# Latest tested version is v1.1292.4.
JENKINS_AGENT_BASE_SNYK_DISTRIBUTION_URL=https://github.com/snyk/snyk/releases/download/v1.1292.4/snyk-linux

# AquaSec CLI binary distribution url
# Leave empty to avoid installing AquaSec.
# Releases are published at https://download.aquasec.com/scanner
# Check Aqua versions backward compatibility at https://docs.aquasec.com/docs/version-compatibility-of-components#section-backward-compatibility-across-two-major-versions
# To Download the aquaSec scanner cli and check their documentaion requires a valid account on aquasec.com
# Latest tested version is 2022.4.517
# Example: https://<USER>:<PASSWORD>@download.aquasec.com/scanner/2022.4.517/scannercli
# Example: https://<USER>:<PASSWORD>@download.aquasec.com/scanner/2022.4.587/scannercli
JENKINS_AGENT_BASE_AQUASEC_SCANNERCLI_URL=

# Repository of shared library
Expand Down
39 changes: 15 additions & 24 deletions jenkins/agent-base/Dockerfile.ubi8
Original file line number Diff line number Diff line change
Expand Up @@ -3,15 +3,15 @@ FROM quay.io/openshift/origin-jenkins-agent-base
SHELL ["/bin/bash", "-o", "pipefail", "-c"]

# SONAR_SCANNER_VERSION above 4.8.x require java 17 to run.
ENV SONAR_SCANNER_VERSION=4.8.1.3023 \
CNES_REPORT_VERSION=4.2.0 \
ENV SONAR_SCANNER_VERSION=5.0.1.3006 \
CNES_REPORT_VERSION=4.3.0 \
TAILOR_VERSION=1.3.4 \
SOPS_VERSION=3.8.1 \
HELM_VERSION=3.14.3 \
HELM_PLUGIN_DIFF_VERSION=3.9.5 \
HELM_PLUGIN_SECRETS_VERSION=4.6.0 \
SOPS_VERSION=3.9.0 \
HELM_VERSION=3.15.4 \
HELM_PLUGIN_DIFF_VERSION=3.9.9 \
HELM_PLUGIN_SECRETS_VERSION=4.6.1 \
GIT_LFS_VERSION=3.5.1 \
TRIVY_VERSION=0.50.1 \
TRIVY_VERSION=0.54.1 \
JAVA_GC_OPTS="-XX:+UseParallelGC -XX:MinHeapFreeRatio=5 -XX:MaxHeapFreeRatio=10 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90"

ARG APP_DNS
Expand All @@ -22,27 +22,25 @@ ARG AQUASEC_SCANNERCLI_URL
COPY yum.repos.d/ubi.repo /etc/yum.repos.d/ubi.repo

COPY ensure_java_jre_is_adequate.sh /usr/local/bin/
COPY ./set-default-java.sh /etc/profile.d/set-default-java.sh

RUN cd /etc/yum.repos.d && rm -f localdev-* ci-rpm-mirrors.repo \
&& ensure_java_jre_is_adequate.sh \
&& yum -y install make glibc-langpack-en openssl \
&& yum -y install make glibc-langpack-en openssl skopeo \
&& yum -y update \
&& yum clean all \
&& rm -rf /var/cache/yum/*

#
# WARNING: We do not install java 8 nor java 11 in this image because they are already intalled in it.
#
&& rm -rf /var/cache/yum/* \
&& skopeo --version

# Copy use java scripts.
COPY use-j*.sh /usr/local/bin/
RUN chmod +x /usr/local/bin/use-j*.sh && \
chmod ugo+s /usr/local/bin/use-j*.sh && \
sh -c 'chmod ugo+s $(which alternatives)' && \
ls -la /usr/local/bin/use-j*.sh && \
echo "--- STARTS JDK 11 TESTS ---" && \
use-j11.sh && \
echo "--- ENDS JDK 11 TESTS ---"
echo "--- STARTS JDK 17 TESTS ---" && \
use-j17.sh && \
echo "--- ENDS JDK 17 TESTS ---"

COPY ./import_certs.sh /usr/local/bin/import_certs.sh
COPY ./fix_java_certs_permissions.sh /usr/local/bin/fix_java_certs_permissions.sh
Expand Down Expand Up @@ -73,7 +71,7 @@ RUN cd /tmp \

# Install Helm.
RUN cd /tmp \
&& dnf install -y https://github.com/mozilla/sops/releases/download/v${SOPS_VERSION}/sops-${SOPS_VERSION}.x86_64.rpm \
&& dnf install -y https://github.com/mozilla/sops/releases/download/v${SOPS_VERSION}/sops-${SOPS_VERSION}-1.x86_64.rpm \
&& mkdir -p /tmp/helm \
&& curl -sSLO https://get.helm.sh/helm-v${HELM_VERSION}-linux-amd64.tar.gz \
&& tar -zxvf helm-v${HELM_VERSION}-linux-amd64.tar.gz -C /tmp/helm \
Expand Down Expand Up @@ -133,15 +131,8 @@ RUN mv /usr/local/bin/run-jnlp-client /usr/local/bin/openshift-run-jnlp-client \

COPY ods-run-jnlp-client.sh /usr/local/bin/run-jnlp-client

# Add skopeo.
RUN yum install -y skopeo \
&& yum clean all \
&& rm -rf /var/cache/yum/* \
&& skopeo --version

# Fix permissions.
RUN mkdir -p /home/jenkins/.config && chmod -R g+w /home/jenkins/.config \
&& mkdir -p /home/jenkins/.cache && chmod -R g+w /home/jenkins/.cache \
&& mkdir -p /home/jenkins/.sonar && chmod -R g+w /home/jenkins/.sonar \
&& mkdir -p /tmp/aqua && chmod -R g+w /tmp/aqua

46 changes: 24 additions & 22 deletions jenkins/agent-base/ensure_java_jre_is_adequate.sh
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@ set -eu -o pipefail

ME="$(basename $0)"
JAVA_INSTALLED_PKGS_LOGS="/tmp/java_installed_pkgs.log"
JAVA_11_INSTALLED_PKGS_LOGS="/tmp/java_11_installed_pkgs.log"
rm -fv ${JAVA_INSTALLED_PKGS_LOGS} ${JAVA_11_INSTALLED_PKGS_LOGS}
JAVA_17_INSTALLED_PKGS_LOGS="/tmp/java_17_installed_pkgs.log"
rm -fv ${JAVA_INSTALLED_PKGS_LOGS} ${JAVA_17_INSTALLED_PKGS_LOGS}

NEEDS_DEVEL=${1-""}
PKG_NAME_TAIL="headless"
Expand All @@ -20,26 +20,26 @@ echo "${ME}: Needs development packages? ${NEEDS_DEVEL}"
echo " "
echo "${ME}: Listing versions of java installed: "
yum list installed | grep -i "\(java\|jre\)" | tee -a ${JAVA_INSTALLED_PKGS_LOGS}
touch ${JAVA_11_INSTALLED_PKGS_LOGS}
grep -i "java-11" ${JAVA_INSTALLED_PKGS_LOGS} > ${JAVA_11_INSTALLED_PKGS_LOGS} || echo "No java 11 packages found."
touch ${JAVA_17_INSTALLED_PKGS_LOGS}
grep -i "java-17" ${JAVA_INSTALLED_PKGS_LOGS} > ${JAVA_17_INSTALLED_PKGS_LOGS} || echo "No java 17 packages found."

NEEDS_INSTALLATION="true"
if [ -f ${JAVA_11_INSTALLED_PKGS_LOGS} ]; then
if grep -qi "${PKG_NAME_TAIL}" ${JAVA_11_INSTALLED_PKGS_LOGS} ; then
if [ -f ${JAVA_17_INSTALLED_PKGS_LOGS} ]; then
if grep -qi "${PKG_NAME_TAIL}" ${JAVA_17_INSTALLED_PKGS_LOGS} ; then
NEEDS_INSTALLATION="false"
fi
fi

# We need devel package in masters to have jar binary.
if [ "true" == "${NEEDS_INSTALLATION}" ]; then
echo "${ME}:Java-11 is *not* installed. Installing..."
echo "${ME}:Java-17 is *not* installed. Installing..."
if [ "true" == "${NEEDS_DEVEL}" ]; then
yum -y install java-11-openjdk-devel
yum -y install java-17-openjdk-devel
else
yum -y install java-11-openjdk-headless
yum -y install java-17-openjdk-headless
fi
else
echo "${ME}: Java-11 is already installed."
echo "${ME}: Java-17 is already installed."
fi

if grep -qi "java-1.8" ${JAVA_INSTALLED_PKGS_LOGS} ; then
Expand All @@ -49,19 +49,21 @@ else
echo "${ME}: Java-8 is not installed. Correct."
fi

rm -fv ${JAVA_INSTALLED_PKGS_LOGS} ${JAVA_11_INSTALLED_PKGS_LOGS}

echo " "
echo "${ME}: Checking java tool versions: "
if [ "true" == "${NEEDS_DEVEL}" ]; then
jar --version
if grep -qi "java-11" ${JAVA_INSTALLED_PKGS_LOGS} ; then
echo "${ME}: Java-11 is installed. Removing..."
yum -y remove java-11*
else
echo "${ME}: Java-11 is not installed. Correct."
fi

NO_JAVA_LINK="false"
java -version || NO_JAVA_LINK="true"
if [ "true" == "${NO_JAVA_LINK}" ]; then
JAVA_HOME_FOLDER=$(ls -lah /usr/lib/jvm | grep "java-11-openjdk-11.*\.x86_64" | awk '{print $NF}' | head -1)
JAVA_HOME="/usr/lib/jvm/${JAVA_HOME_FOLDER}"
alternatives --set java ${JAVA_HOME}/bin/java
if grep -qi "java-21" ${JAVA_INSTALLED_PKGS_LOGS} ; then
echo "${ME}: Java-21 is installed. Removing..."
yum -y remove java-21*
else
echo "${ME}: Java-21 is not installed. Correct."
fi

rm -fv ${JAVA_INSTALLED_PKGS_LOGS} ${JAVA_17_INSTALLED_PKGS_LOGS}

source /etc/profile.d/set-default-java.sh
java -version
7 changes: 7 additions & 0 deletions jenkins/agent-base/set-default-java.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
#!/bin/bash
set -eu -o pipefail

JAVA_HOME_FOLDER=$(ls -lah /usr/lib/jvm | grep "java-17-openjdk-.*\.x86_64" | awk '{print $NF}' | head -1)
export JAVA_HOME="/usr/lib/jvm/${JAVA_HOME_FOLDER}"
export USE_JAVA_VERSION=java-17
alternatives --set java ${JAVA_HOME}/bin/java
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
#!/bin/bash

JAVA_HOME_FOLDER=$(ls -lah /usr/lib/jvm | grep "java-11-openjdk-11.*\.x86_64" | awk '{print $NF}' | head -1)
JAVA_VERSION="11"
JAVA_HOME_FOLDER=$(ls -lah /usr/lib/jvm | grep "java-17-openjdk-.*\.x86_64" | awk '{print $NF}' | head -1)
JAVA_VERSION="17"

function msg_and_exit() {
echo "ERROR: ${1}"
Expand Down Expand Up @@ -36,8 +36,3 @@ else
msg_and_exit "Cannot configure JAVA_HOME environment variable to ${JAVA_HOME}"
fi
echo "JAVA_HOME: $JAVA_HOME"

rm -fv /etc/profile.d/set-default-java.sh
echo "export JAVA_HOME=${JAVA_HOME}" >> /etc/profile.d/set-default-java.sh
echo "export USE_JAVA_VERSION=java-11" >> /etc/profile.d/set-default-java.sh
chmod +x /etc/profile.d/set-default-java.sh
13 changes: 7 additions & 6 deletions jenkins/master/Dockerfile.ubi8
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
FROM quay.io/openshift/origin-jenkins

ENV JAVA_HOME /usr/lib/jvm/jre-11
ENV JAVA_HOME /usr/lib/jvm/jre-17

# ODS defaults, available to use within pipelines.
ARG ODS_NAMESPACE
Expand All @@ -14,12 +14,16 @@ ENV JENKINS_JAVA_OVERRIDES="-Dhudson.tasks.MailSender.SEND_TO_UNKNOWN_USERS=true

USER root

# Add UBI repositories.
COPY yum.repos.d/ubi.repo /etc/yum.repos.d/ubi.repo

COPY ./scripts_for_usr-local-bin/* /usr/local/bin/
RUN import_certs.sh \
&& rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io.key \
RUN rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io.key \
&& disable_yum_repository.sh /etc/yum.repos.d/ci-rpm-mirrors.repo \
/etc/yum.repos.d/localdev-* /etc/yum.repos.d/epel.repo \
&& ensure_java_jre_is_adequate.sh master \
&& yum -y update \
&& import_certs.sh \
&& fix_openshift_scripts.sh \
&& clean_yum_cache.sh

Expand All @@ -43,6 +47,3 @@ RUN cd /tmp \
&& tailor version

USER jenkins



17 changes: 8 additions & 9 deletions jenkins/master/plugins.ubi8.txt
Original file line number Diff line number Diff line change
@@ -1,14 +1,13 @@
# Aditional plugins
greenballs:1.15.1
sonar:2.17.2
blueocean:1.27.9
email-ext:2.104
ansicolor:1.0.4
kubernetes-credentials:0.11
kubernetes-client-api:6.10.0-240.v57880ce8b_0b_2
kubernetes:4186.v1d804571d5d4
junit:1259.v65ffcef24a_88
audit-trail:361.v82cde86c784e
credentials:1337.v60b_d7b_c7b_c9f
workflow-multibranch:773.vc4fe1378f1d5
git:5.2.1

# Bundled plugins
token-macro:400.v35420b_922dcb_
email-ext:2.104
junit:Version1256.v002534a_5f33e
blueocean:1.27.9
kubernetes:4174.v4230d0ccd951
openshift-sync:1.1.0.802.v45585f8cdc07
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@ set -eu -o pipefail

ME="$(basename $0)"
JAVA_INSTALLED_PKGS_LOGS="/tmp/java_installed_pkgs.log"
JAVA_11_INSTALLED_PKGS_LOGS="/tmp/java_11_installed_pkgs.log"
rm -fv ${JAVA_INSTALLED_PKGS_LOGS} ${JAVA_11_INSTALLED_PKGS_LOGS}
JAVA_17_INSTALLED_PKGS_LOGS="/tmp/java_17_installed_pkgs.log"
rm -fv ${JAVA_INSTALLED_PKGS_LOGS} ${JAVA_17_INSTALLED_PKGS_LOGS}

NEEDS_DEVEL=${1-""}
PKG_NAME_TAIL="headless"
Expand All @@ -20,26 +20,26 @@ echo "${ME}: Needs development packages? ${NEEDS_DEVEL}"
echo " "
echo "${ME}: Listing versions of java installed: "
yum list installed | grep -i "\(java\|jre\)" | tee -a ${JAVA_INSTALLED_PKGS_LOGS}
touch ${JAVA_11_INSTALLED_PKGS_LOGS}
grep -i "java-11" ${JAVA_INSTALLED_PKGS_LOGS} > ${JAVA_11_INSTALLED_PKGS_LOGS} || echo "No java 11 packages found."
touch ${JAVA_17_INSTALLED_PKGS_LOGS}
grep -i "java-17" ${JAVA_INSTALLED_PKGS_LOGS} > ${JAVA_17_INSTALLED_PKGS_LOGS} || echo "No java 17 packages found."

NEEDS_INSTALLATION="true"
if [ -f ${JAVA_11_INSTALLED_PKGS_LOGS} ]; then
if grep -qi "${PKG_NAME_TAIL}" ${JAVA_11_INSTALLED_PKGS_LOGS} ; then
if [ -f ${JAVA_17_INSTALLED_PKGS_LOGS} ]; then
if grep -qi "${PKG_NAME_TAIL}" ${JAVA_17_INSTALLED_PKGS_LOGS} ; then
NEEDS_INSTALLATION="false"
fi
fi

# We need devel package in masters to have jar binary.
if [ "true" == "${NEEDS_INSTALLATION}" ]; then
echo "${ME}:Java-11 is *not* installed. Installing..."
echo "${ME}:Java-17 is *not* installed. Installing..."
if [ "true" == "${NEEDS_DEVEL}" ]; then
yum -y install java-11-openjdk-devel
yum -y install java-17-openjdk-devel
else
yum -y install java-11-openjdk-headless
yum -y install java-17-openjdk-headless
fi
else
echo "${ME}: Java-11 is already installed."
echo "${ME}: Java-17 is already installed."
fi

if grep -qi "java-1.8" ${JAVA_INSTALLED_PKGS_LOGS} ; then
Expand All @@ -49,12 +49,26 @@ else
echo "${ME}: Java-8 is not installed. Correct."
fi

rm -fv ${JAVA_INSTALLED_PKGS_LOGS} ${JAVA_11_INSTALLED_PKGS_LOGS}
if grep -qi "java-11" ${JAVA_INSTALLED_PKGS_LOGS} ; then
echo "${ME}: Java-11 is installed. Removing..."
yum -y remove java-11*
else
echo "${ME}: Java-11 is not installed. Correct."
fi

if grep -qi "java-21" ${JAVA_INSTALLED_PKGS_LOGS} ; then
echo "${ME}: Java-21 is installed. Removing..."
yum -y remove java-21*
else
echo "${ME}: Java-21 is not installed. Correct."
fi

rm -fv ${JAVA_INSTALLED_PKGS_LOGS} ${JAVA_17_INSTALLED_PKGS_LOGS}

NO_JAVA_LINK="false"
java -version || NO_JAVA_LINK="true"
if [ "true" == "${NO_JAVA_LINK}" ]; then
JAVA_HOME_FOLDER=$(ls -lah /usr/lib/jvm | grep "java-11-openjdk-11.*\.x86_64" | awk '{print $NF}' | head -1)
JAVA_HOME_FOLDER=$(ls -lah /usr/lib/jvm | grep "java-17-openjdk-17.*\.x86_64" | awk '{print $NF}' | head -1)
JAVA_HOME="/usr/lib/jvm/${JAVA_HOME_FOLDER}"
alternatives --set java ${JAVA_HOME}/bin/java
fi
Expand Down
Loading
Loading