From 8b9a1906251a11fbbcd53dff4dda17e6a283ce12 Mon Sep 17 00:00:00 2001 From: judovana Date: Fri, 22 Nov 2024 09:20:53 +0100 Subject: [PATCH] Global image for external (#5553) * Added initial concept of configurable image for external * Added proper defautls to survive with -u * Added forgotten use of providers in cleaning of images for build_all * Removed usage of variable in method name to make it readable, find-able and more generic * excluded EXTERNAL_AQA_IMAGE from build_all.sh * Now honor EXTERNAL_AQA_CONTAINER_CLEAN also after external run * Now using the userset image (if set) * reworked roeprties so they can work in multi-image environment instead of hardcoded ubuntu/ubi images a generic approach was set generic_packages are those which ahve same name on all supported iamges REGEX_packages cen set special packages per distribution eg ubuntu_packages will be installed on ubuntu only eg (fedora|ubi|rhel|centos)_packages will be iisntalled on RH systems This already supports versioned ones, so eg centos:10_packages will indeed restrict the packages to os centos 10 This introduced hackish way how to handle hardcoded criu-openj9-images in this multi-image environment * Added explicit search for java in /opt/java/openjdk It seems that default contianer was not searching directly in, as the linking on rest of the system led to it. Now all images I tried properly tests mounted java if it is there. if not, default search is still on and works fine for both default with mount, without mount or for other images with java on path without mount * Added set of warning for case the mounted jdk is missing or TEST_JAVA is not part of the mount commnad * Added example of to build_image.sh * Retuerned parsing ov version. It is used over all scripts, but not set * Deduct os on correct place, after the hack is applied * Exiting build_all if version is missing. he variable is used (wihtout quotes) * Added missign make dependecies to all systems for criu * Removed common depenencies * Properly expand build-essentials which are not in fedora, and ascii doc It is quite hard to tell what is build-essentials i feodra world, but generally it seems to be: make automake gcc gcc-c++ kernel-devel Asciidocs seems to be on all systems * Changed decvlaration of pacakges in criu-ubi-portable-checkpoint to generic all `git wget perl make` install fie on all tried OSes * Improved exit message for unknown os * better readme for EXTERNAL_AQA_IMAGE Co-authored-by: Shelley Lambert * decalred -> declared * Removed the nasty warning warning... by nicer header * Update external/build_all.sh Co-authored-by: Martijn Verburg * Update external/common_functions.sh Co-authored-by: Martijn Verburg * Added whitesapces between # and first char * More === text ===.. isntead of text text text * added examples of usages for EXTERNAL_AQA_IMAGE * Fixed typos in variables gobalMatchingKeys -> globalMatchingKeys getFullTOpenJ9Image -> getFullOpenJ9Image --------- Co-authored-by: Shelley Lambert Co-authored-by: Martijn Verburg --- external/README.md | 3 + external/build.xml | 2 +- external/build_all.sh | 10 ++ external/build_image.sh | 2 + external/camel/test.properties | 2 +- external/common.properties | 6 + external/common_functions.sh | 68 ++++++++-- external/criu-functional/test.properties | 5 +- .../criu-portable-checkpoint/test.properties | 5 +- .../test.properties | 2 +- external/derby/test.properties | 3 +- external/dockerfile_functions.sh | 60 ++++++--- external/elasticsearch/test.properties | 2 +- external/external.sh | 27 ++++ external/functional-test/test.properties | 6 +- external/jacoco/test.properties | 2 +- external/jenkins/test.properties | 2 +- external/kafka/test.properties | 2 +- external/lucene-solr/test.properties | 3 +- external/netty/test.properties | 4 +- external/openliberty-mp-tck/test.properties | 2 +- external/payara-mp-tck/test.properties | 2 +- external/provider.sh | 122 +++++++++++++++++- external/quarkus/test.properties | 2 +- external/quarkus_openshift/test.properties | 2 +- external/quarkus_quickstarts/test.properties | 2 +- external/scala/test.properties | 3 +- external/spring/test.properties | 2 +- external/system-test/test.properties | 2 +- external/tck-ubi-test/test.properties | 3 +- external/test_base_functions.sh | 14 +- external/tomcat/test.properties | 5 +- external/tomee/test.properties | 2 +- external/wildfly/test.properties | 2 +- external/wycheproof/test.properties | 4 +- external/zookeeper/test.properties | 4 +- 36 files changed, 330 insertions(+), 59 deletions(-) create mode 100644 external/common.properties diff --git a/external/README.md b/external/README.md index 05d1978e44..e08dce56bd 100644 --- a/external/README.md +++ b/external/README.md @@ -20,6 +20,9 @@ EXTERNAL_AQA_CONTAINER_CLEAN=true|false ``` If EXTERNAL_AQA_CONTAINER_CLEAN is false, then the image is not cleaned after the `make _tests...` targets are finished. +## Configuring base image +By default, Eclipse Temurin JDK of identical version as your JDK is used. You can see, that `print_image_args` is taking all arguments to properly set registry url, image name and version. `EXTERNAL_AQA_IMAGE` variable describes the usual image ID in form of `optional_registry/path/name:tag` to allow for alternate images besides the default to be used. E.g.: `export EXTERNAL_AQA_IMAGE=fedora:41` or `export EXTERNAL_AQA_IMAGE=centos:stream9` + ## Running External tests locally To run any AQA tests locally, you follow the same pattern: diff --git a/external/build.xml b/external/build.xml index e7066291b1..f01b347196 100644 --- a/external/build.xml +++ b/external/build.xml @@ -89,7 +89,7 @@ - + diff --git a/external/build_all.sh b/external/build_all.sh index e22d0c81bb..bd13894f4b 100755 --- a/external/build_all.sh +++ b/external/build_all.sh @@ -16,6 +16,16 @@ set -o pipefail source $(dirname "$0")/common_functions.sh +if [ -n "${EXTERNAL_AQA_IMAGE}" ] ; then\ + echo "EXTERNAL_AQA_IMAGE is declared as $EXTERNAL_AQA_IMAGE; it would break this script. Unset it." + exit 1 +fi + +if [ -z "${version}" ] ; then\ + echo "'version' (of JDK, e.g., 17) is mandatory variable for this script to run" + exit 1 +fi + # Cleanup any old containers and images echo "===============================================================================" echo " Cleaning up images " diff --git a/external/build_image.sh b/external/build_image.sh index 8e277b392d..671a0241d9 100755 --- a/external/build_image.sh +++ b/external/build_image.sh @@ -25,6 +25,7 @@ if [ $# -ne 9 ] && [ $# -ne 10 ]; then echo "The supported tests are ${supported_tests}" echo echo "usage: $0 test version vm os package build platform check_external_custom" + echo "eg: $0 jacoco 17 openj9 ubi jdk full linux_x86-64 default 0" echo "test = ${supported_tests}" echo "vm = ${supported_jvms}" echo "os = ${supported_os}" @@ -43,6 +44,7 @@ check_external_custom=$9 if [[ "${check_external_custom}" == "0" ]]; then set_test $1 fi +set_version $2 set_vm $3 set_os $4 set_package $5 diff --git a/external/camel/test.properties b/external/camel/test.properties index 4872d6277a..4d352da816 100644 --- a/external/camel/test.properties +++ b/external/camel/test.properties @@ -3,4 +3,4 @@ test_results="testResults" tag_version="2.7.0" environment_variable="MODE=java" - ubuntu_packages="git" +generic_packages="git" diff --git a/external/common.properties b/external/common.properties new file mode 100644 index 0000000000..f17c6b2cb2 --- /dev/null +++ b/external/common.properties @@ -0,0 +1,6 @@ +# those packages are included in all tests and all distributions +generic_packages="" +# those packages are included in all tests and all ubuntu +ubuntu_packages="" +# those packages are included in all tests and all Red Hat distributions +(fedora|ubi|rhel|centos)_packages="" diff --git a/external/common_functions.sh b/external/common_functions.sh index 1b8686d78c..19bd25abb6 100755 --- a/external/common_functions.sh +++ b/external/common_functions.sh @@ -12,6 +12,9 @@ # See the License for the specific language governing permissions and # limitations under the License. # + +source $(dirname "$0")/provider.sh + # Supported JVMs supported_jvms="hotspot openj9" @@ -28,6 +31,11 @@ supported_builds="full" # Supported tests supported_tests="external_custom camel criu-functional criu-portable-checkpoint criu-portable-restore criu-ubi-portable-checkpoint criu-ubi-portable-restore derby elasticsearch jacoco jenkins functional-test kafka lucene-solr openliberty-mp-tck payara-mp-tck quarkus quarkus_quickstarts scala system-test tck-ubi-test tomcat tomee wildfly wycheproof netty spring zookeeper" +# Set a valid version +function set_version() { + version=$1 +} + function check_os() { os=$1 @@ -161,11 +169,34 @@ function set_base_docker_registry_dir() { # Reading properties of test.properties file function getProperty() { - PROP_KEY=$1 - PROP_VALUE=`cat $PROPERTY_FILE | grep "$PROP_KEY" | cut -d'=' -f 2-` + local PROP_KEY="${1}" + local FILE=${2:-""} + if [ -z "${FILE}" ] ; then + FILE="${PROPERTY_FILE}" + fi + local PROP_VALUE=`cat "${FILE}" | grep -v "^#" | grep "^${PROP_KEY}=" | cut -d'=' -f 2-` echo `sed -e 's/^"//' -e 's/"$//' <<<"$PROP_VALUE"` } +# Getting matching keys for given OS +function getMatchingPackagesKeys() { + local FILE=${1} + local OS=$(getImageOs) + local TAG=$(getImageTag) + local KEYS="" + while read line; do + if echo "$line" | grep -q "^#" ; then + continue + fi + local keyCandidate=$(echo $line | cut -d'=' -f 1) + local strippedKeyCandidate=$(echo $keyCandidate | sed "s/_packages//") + if echo "${OS}:${TAG}" | grep -Eqe ".*${strippedKeyCandidate}.*" ; then + KEYS="$KEYS $keyCandidate" + fi + done < "${FILE}" + echo ${KEYS} +} + # Used for external_custom tests function set_external_custom_test_info(){ test=$1 @@ -175,17 +206,20 @@ function set_external_custom_test_info(){ test_results="testResults" tag_version="${EXTERNAL_REPO_BRANCH}" environment_variable="MODE=java" - ubuntu_packages="git" maven_version="3.8.5" + packages="git" } # Set the valid OSes for the current architectures. function set_test_info() { - test=$1 + local test=$1 check_external_custom_test=$2 - cd ../ - path_to_file=$(pwd) + local path_to_file=$(cd $(dirname "$0") && pwd) echo ${path_to_file} + # global settings will be amend to local ones + local GLOBAL_PROPERTY_FILE=${path_to_file}/common.properties + local global_generic_packages=$(getProperty "generic_packages" "${GLOBAL_PROPERTY_FILE}") + PROPERTY_FILE=${path_to_file}/${test}/test.properties github_url=$(getProperty "github_url") test_options=$(getProperty "test_options") @@ -204,14 +238,28 @@ function set_test_info() { maven_version=$(getProperty "maven_version") environment_variable=$(getProperty "environment_variable") localPropertyFile=$(getProperty "localPropertyFile") - ubuntu_packages=$(getProperty "ubuntu_packages") - ubi_packages=$(getProperty "ubi_packages") + local local_generic_packages=$(getProperty "generic_packages") + + packages="$global_generic_packages $local_generic_packages" + local globalMatchingKeys=$(getMatchingPackagesKeys "${GLOBAL_PROPERTY_FILE}") + for key in ${globalMatchingKeys} ; do + packages="$packages $(getProperty $key" "${GLOBAL_PROPERTY_FILE}\")" + done + local localMatchingKeys=$(getMatchingPackagesKeys "${PROPERTY_FILE}") + for key in ${localMatchingKeys} ; do + packages="$packages $(getProperty "$key" "${PROPERTY_FILE}")" + done } function cleanup_images() { + local container_rm=$(getExternalImageCommand) + if [ "${EXTERNAL_AQA_CONTAINER_CLEAN}" == "false" ] ; then + echo "to debug, put '-i --entrypoint /bin/bash' before container name" + container_rm="echo to clean, run manually: $container_rm" + fi # Delete any old containers that have exited. - docker rm $(docker ps -a | grep "Exited" | awk '{ print $1 }') 2>/dev/null + ${container_rm} rm $(docker ps -a | grep "Exited" | awk '{ print $1 }') 2>/dev/null # Delete any old images for our target_repo on localhost. - docker rmi -f $(docker images | grep -e "adoptopenjdk" | awk '{ print $3 }' | sort | uniq) 2>/dev/null + ${container_rm} rmi -f $(docker images | grep -e "adoptopenjdk" | awk '{ print $3 }' | sort | uniq) 2>/dev/null } diff --git a/external/criu-functional/test.properties b/external/criu-functional/test.properties index 821735dc98..5042fa81e2 100644 --- a/external/criu-functional/test.properties +++ b/external/criu-functional/test.properties @@ -2,5 +2,8 @@ github_url="https://github.com/adoptium/aqa-tests.git" test_results="testResults" gradle_version="5.1" environment_variable="MODE=java CC=gcc-7 CXX=g++-7" -ubuntu_packages="ant-contrib build-essential git asciidoc" +generic_packages="git asciidoc" +ubuntu_packages="ant-contrib build-essential" +fedora_packages="ant-contrib" +(fedora|ubi|rhel|centos)_packages=make automake gcc gcc-c++ kernel-devel criu_version="latest" diff --git a/external/criu-portable-checkpoint/test.properties b/external/criu-portable-checkpoint/test.properties index 821735dc98..5042fa81e2 100644 --- a/external/criu-portable-checkpoint/test.properties +++ b/external/criu-portable-checkpoint/test.properties @@ -2,5 +2,8 @@ github_url="https://github.com/adoptium/aqa-tests.git" test_results="testResults" gradle_version="5.1" environment_variable="MODE=java CC=gcc-7 CXX=g++-7" -ubuntu_packages="ant-contrib build-essential git asciidoc" +generic_packages="git asciidoc" +ubuntu_packages="ant-contrib build-essential" +fedora_packages="ant-contrib" +(fedora|ubi|rhel|centos)_packages=make automake gcc gcc-c++ kernel-devel criu_version="latest" diff --git a/external/criu-ubi-portable-checkpoint/test.properties b/external/criu-ubi-portable-checkpoint/test.properties index d82b772f93..305fcebd75 100644 --- a/external/criu-ubi-portable-checkpoint/test.properties +++ b/external/criu-ubi-portable-checkpoint/test.properties @@ -2,4 +2,4 @@ github_url="https://github.com/adoptium/aqa-tests.git" test_results="testResults" gradle_version="5.1" environment_variable="MODE=java" -ubi_packages="git wget perl" +generic_packages="git wget perl make" diff --git a/external/derby/test.properties b/external/derby/test.properties index e9e043a1f0..f8b944186b 100644 --- a/external/derby/test.properties +++ b/external/derby/test.properties @@ -1,4 +1,5 @@ github_url="https://github.com/apache/derby.git" localPropertyFile="local.properties" tag_version="trunk" -ubuntu_packages="ant-optional git wget tar" +ubuntu_packages="ant-optional" +generic_packages="git wget tar" diff --git a/external/dockerfile_functions.sh b/external/dockerfile_functions.sh index 8c4e2290db..10898a8008 100755 --- a/external/dockerfile_functions.sh +++ b/external/dockerfile_functions.sh @@ -84,34 +84,50 @@ print_image_args() { local build=$7 local base_docker_registry_dir="$8" - image_name="docker.io/library/eclipse-temurin" - tag="" - if [[ "${package}" == "jre" ]]; then - tag="${version}-jre" - else - tag="${version}-jdk" - fi + image_name="$(getTemurinImageName)" + tag="$(getTemurinImageTag "${version}" "${package}")" if [[ "${vm}" == "openj9" ]]; then if [[ "${os}" == "ubuntu" ]]; then - image_name="docker.io/ibm-semeru-runtimes" - tag=open-${tag} + image_name="$(getOpenJ9ImageName)" + tag="$(getOpenJ9ImageTag "${version}" "${package}")" elif [[ "${os}" == *"ubi"* && "${test}" != *"criu"* ]]; then + if isExternalImageEnabled ; then + echo "openj9 ubi and custom EXTERNAL_AQA_IMAGE are not compatible" + exit 1 + fi image_name="registry.access.redhat.com/$base_docker_registry_dir" tag="latest" + EXTERNAL_AQA_IMAGE="${image_name}:${tag}" else + if isExternalImageEnabled ; then + echo "openj9 ubi and custom EXTERNAL_AQA_IMAGE are not compatible" + exit 1 + fi # os is ubi, and test is criu # temporarily all ubi based testing use internal base image image_name="$DOCKER_REGISTRY_URL/$base_docker_registry_dir" tag="latest" + EXTERNAL_AQA_IMAGE="${image_name}:${tag}" fi fi image="${image_name}:${tag}" - echo -e "ARG IMAGE=${image}" \ + if isExternalImageEnabled ; then + os=$(getImageOs) + echo -e \ + "ARG IMAGE=${image}" \ + "\nARG OS=${os}" \ + "\nARG IMAGE_VERSION=nightly" \ + "\nARG TAG=${tag}" \ + "\nFROM \$IMAGE\n" >> ${file} + else + echo -e \ + "ARG IMAGE=${image}" \ "\nARG OS=${os}" \ "\nARG IMAGE_VERSION=nightly" \ "\nARG TAG=${tag}" \ "\nFROM \$IMAGE\n" >> ${file} + fi } print_test_tag_arg() { @@ -133,8 +149,6 @@ print_result_comment_arg() { # Select the ubuntu OS packages print_ubuntu_pkg() { local file=$1 - local packages=$2 - echo -e "RUN apt-get update \\" \ "\n\t&& apt-get install -qq -y --no-install-recommends software-properties-common \\" \ "\n\t&& apt-get install -qq -y --no-install-recommends gnupg \\" \ @@ -148,8 +162,6 @@ print_ubuntu_pkg() { # Select the ubuntu OS packages print_ubi_pkg() { local file=$1 - local packages=$2 - echo -e "RUN dnf install -y ${packages} \\" \ "\n\t&& dnf clean all " >> ${file} echo -e "\nENV LANG='en_US.UTF-8' LANGUAGE='en_US:en' LC_ALL='en_US.UTF-8'" >> ${file} @@ -212,6 +224,9 @@ print_ant_install() { local file=$1 local ant_version=$2 local os=$3 + if isExternalImageEnabled ; then + os=$(getImageOs) + fi echo -e "ARG ANT_VERSION=${ant_version}" \ "\nENV ANT_VERSION=\$ANT_VERSION" \ @@ -501,6 +516,9 @@ print_testInfo_env() { local OS=$3 local version=$4 local vm=$5 + if isExternalImageEnabled ; then + OS=$(getImageOs) + fi echo -e "ENV APPLICATION_NAME=${test}" \ "\nENV APPLICATION_TAG=${test_tag}" \ "\nENV OS_TAG=${OS}" \ @@ -593,7 +611,6 @@ generate_dockerfile() { base_docker_registry_dir="$9" check_external_custom_test=$10 - if [[ "$check_external_custom_test" == "1" ]]; then tag_version=${EXTERNAL_REPO_BRANCH} fi @@ -603,7 +620,7 @@ generate_dockerfile() { else set_test_info ${test} ${check_external_custom_test} fi - packages=$(echo ${os}_packages | sed 's/-/_/') + jhome="/opt/java/openjdk" mkdir -p `dirname ${file}` 2>/dev/null @@ -612,9 +629,18 @@ generate_dockerfile() { print_legal ${file}; print_adopt_test ${file} ${test}; print_image_args ${file} ${test} ${os} ${version} ${vm} ${package} ${build} "${base_docker_registry_dir}"; + # print_image_args is setting up hackisch EXTERNAL_AQA_IMAGE for some corner cases + osDeducted=$(getImageOs) print_result_comment_arg ${file}; print_test_tag_arg ${file} ${test} ${tag_version}; - print_${os}_pkg ${file} "${!packages}"; + if echo ${osDeducted} | grep -i -e ubuntu -e debian ; then + print_ubuntu_pkg ${file} + elif echo ${osDeducted} | grep -i -e ubi -e fedora -e rhel -e centos ; then + print_ubi_pkg ${file} + else + echo "unknown os: ${osDeducted} (${os})" + exit 1 + fi if [[ ! -z ${ant_version} ]]; then print_ant_install ${file} ${ant_version} ${os}; diff --git a/external/elasticsearch/test.properties b/external/elasticsearch/test.properties index ad98cb8dec..0e19f6a8d0 100644 --- a/external/elasticsearch/test.properties +++ b/external/elasticsearch/test.properties @@ -1,5 +1,5 @@ github_url="https://github.com/elastic/elasticsearch.git" tag_version="v8.1.2" test_results="testResults" -ubuntu_packages="git wget unzip" +generic_packages="git wget unzip" diff --git a/external/external.sh b/external/external.sh index 80fbe2c220..584f66bf87 100755 --- a/external/external.sh +++ b/external/external.sh @@ -19,6 +19,33 @@ set -e source $(dirname "$0")/provider.sh +if [ -z "${EXTRA_DOCKER_ARGS}" ] ; then + echo \ +" # ================= WARNING ================= WARNING ================= WARNING ================= # + # EXTRA_DOCKER_ARGS are not set. You will be testing java which is already in container and not TEST_JDK_HOME + # TEST_JDK_HOME is set to $TEST_JDK_HOME but will not be used. See test_base_functions.sh for order of search + # You should set your's TEST_JDK_HOME to mount to /opt/java/openjdk, eg: # + # export EXTRA_DOCKER_ARGS=\"-v \$TEST_JDK_HOME:/opt/java/openjdk\" # + # ================= WARNING ================= WARNING ================= WARNING ================= #" +else + echo \ +" # =================== Info =================== Info =================== Info =================== # + # EXTRA_DOCKER_ARGS set as \"$EXTRA_DOCKER_ARGS\" # + # =================== Info =================== Info =================== Info =================== #" + if echo "${EXTRA_DOCKER_ARGS}" | grep -q "$TEST_JDK_HOME" ; then + echo \ +" # =================== Info =================== Info =================== Info =================== # + # TEST_JDK_HOME of $TEST_JDK_HOME is used in EXTRA_DOCKER_ARGS # + # =================== Info =================== Info =================== Info =================== #" + else + echo \ +" # ================= WARNING ================= WARNING ================= WARNING ================= # + # TEST_JDK_HOME of $TEST_JDK_HOME is NOT used in EXTRA_DOCKER_ARGS # + # ================= WARNING ================= WARNING ================= WARNING ================= #" + fi +fi + + tag=nightly docker_os=ubuntu build_type=full diff --git a/external/functional-test/test.properties b/external/functional-test/test.properties index 6faca16b72..d58312a2b1 100644 --- a/external/functional-test/test.properties +++ b/external/functional-test/test.properties @@ -2,4 +2,8 @@ github_url="https://github.com/adoptium/aqa-tests.git" test_results="testResults" gradle_version="5.1" environment_variable="MODE=java CC=gcc-7 CXX=g++-7" -ubuntu_packages="ant-contrib build-essential git asciidoc" \ No newline at end of file +generic_packages="git asciidoc" +ubuntu_packages="ant-contrib build-essential" +fedora_packages="ant-contrib" +(fedora|ubi|rhel|centos)_packages=make automake gcc gcc-c++ kernel-devel + diff --git a/external/jacoco/test.properties b/external/jacoco/test.properties index e231b3f517..dba0e7d858 100644 --- a/external/jacoco/test.properties +++ b/external/jacoco/test.properties @@ -3,5 +3,5 @@ test_command="mvn clean verify" test_results="testResults" tag_version="v0.8.7" environment_variable="MODE=java" -ubuntu_packages="git wget tar" maven_version="3.8.5" +generic_packages="git wget tar" diff --git a/external/jenkins/test.properties b/external/jenkins/test.properties index ae2460c8a1..685cf7fe66 100644 --- a/external/jenkins/test.properties +++ b/external/jenkins/test.properties @@ -1,5 +1,5 @@ github_url="https://github.com/jenkinsci/jenkins.git" test_results="testResults" tag_version="jenkins-2.342" -ubuntu_packages="git wget tar" maven_version="3.8.5" +generic_packages="git wget tar" diff --git a/external/kafka/test.properties b/external/kafka/test.properties index faf2d45a2d..66b9c227bd 100644 --- a/external/kafka/test.properties +++ b/external/kafka/test.properties @@ -1,3 +1,3 @@ github_url="https://github.com/apache/kafka.git" tag_version="3.1.0" -ubuntu_packages="git wget unzip" +generic_packages="git wget unzip" diff --git a/external/lucene-solr/test.properties b/external/lucene-solr/test.properties index 6c53376c6b..4f54bf8f7a 100644 --- a/external/lucene-solr/test.properties +++ b/external/lucene-solr/test.properties @@ -1,3 +1,4 @@ github_url="https://github.com/apache/lucene-solr.git" tag_version="releases/lucene-solr/8.11.1" -ubuntu_packages="ant-optional ivy git wget tar" +generic_packages="ivy git wget tar" +ubuntu_packages="ant-optional" diff --git a/external/netty/test.properties b/external/netty/test.properties index 9ba0f519f8..ed068ae7bf 100644 --- a/external/netty/test.properties +++ b/external/netty/test.properties @@ -2,5 +2,7 @@ github_url="https://github.com/netty/netty.git" test_commands="mvn clean package" test_results="testResults" tag_version="netty-4.1.75.Final" -ubuntu_packages="git autoconf automake libtool make wget tar gcc-multilib libaio-dev" +generic_packages="autoconf automake libtool make git wget tar" +ubuntu_packages="gcc-multilib libaio-dev" +fedora_packages="gcc libaio-devel" diff --git a/external/openliberty-mp-tck/test.properties b/external/openliberty-mp-tck/test.properties index 2ccedf9d5d..a3588b3d55 100644 --- a/external/openliberty-mp-tck/test.properties +++ b/external/openliberty-mp-tck/test.properties @@ -1,4 +1,4 @@ github_url="https://github.com/OpenLiberty/open-liberty.git" test_results="testResults" tag_version="gm-23.0.0.5" -ubuntu_packages="ant-optional maven git wget tar" +generic_packages="maven git wget tar" diff --git a/external/payara-mp-tck/test.properties b/external/payara-mp-tck/test.properties index 89cbcd799b..45c86ebd75 100644 --- a/external/payara-mp-tck/test.properties +++ b/external/payara-mp-tck/test.properties @@ -3,5 +3,5 @@ test_command="mvn --batch-mode clean verify" test_results="testResults" test_options="-pl !MicroProfile-OpenTracing/tck-runner,!MicroProfile-JWT-Auth,!MicroProfile-JWT-Auth/tck-arquillian-extension,!MicroProfile-JWT-Auth/tck-runner" tag_version="2.0" -ubuntu_packages="git wget tar" maven_version="3.8.5" +generic_packages="git wget tar" diff --git a/external/provider.sh b/external/provider.sh index c07d1cb8de..4e566556ec 100755 --- a/external/provider.sh +++ b/external/provider.sh @@ -13,10 +13,16 @@ # limitations under the License. # - ################################################################### ## This script walks through all external tests ## -## and determines which virtualisation to use ## +## and determines ## +## 1) which virtualisation to use ## +## 2) which image to use ## +################################################################### + + +################################################################### +## part *1* which virtualisation to use ## ## It recognizes two environment variables: ## ## EXTERNAL_AQA_RUNNER=docker|podman|... ## ## EXTERNAL_AQA_SUDO=sudo||runas ## @@ -60,3 +66,115 @@ function getProviderTile() { echo "$(getSudoNice)/$(getProviderNice)" } +##################################################################### +## part *2* which image to use ## +## It recognizes one complex environment variable: ## +## EXTERNAL_AQA_IMAGE=optional_url/image:optional_version ## +## defaults to: ## +## docker.io/library/eclipse-temurin:${JDK_VERSION}-jdk" ## +## for most of the calls. Defaults to ## +## docker.io/library/ibm-semeru-runtimes:open-${JDK_VERSION}-jdk ## +## for openj9 ## +## for prepare under "default" circumstances the domain is omitted ## +## for sometimes, the "latest" can slip as tag and for several ## +## cases the registry.access.redhat.com are used as domain. ## +## the: ## +## ARG OS ## +## ARG IMAGE_VERSION ## +## ARG TAG ## +## are deducted from this ## +#################################################################### + +function isExternalImageEnabled() { + if [ -n "${EXTERNAL_AQA_IMAGE}" ] ; then + return 0 + else + return 1 + fi +} + +function getFullTemurinImage() { + local JDK_VERSION="${1:-0}" + local jreSdk="${2:-jdk}" + if [ -z "${EXTERNAL_AQA_IMAGE}" ]; then + echo "docker.io/library/eclipse-temurin:${JDK_VERSION}-$jreSdk" + else + echo "${EXTERNAL_AQA_IMAGE}" + fi +} + +function getFullOpenJ9Image() { + local JDK_VERSION="${1:-0}" + local jreSdk="${2:-jdk}" + if [ -z "${EXTERNAL_AQA_IMAGE}" ]; then + echo "docker.io/library/ibm-semeru-runtimes:open-${JDK_VERSION}-$jreSdk" + else + echo "${EXTERNAL_AQA_IMAGE}" + fi +} + +function getImageOs() { + if [ -z "${EXTERNAL_AQA_IMAGE}" ]; then + echo "ubuntu" + else + local osAndTag="${EXTERNAL_AQA_IMAGE##*/}" + local os="${osAndTag%%:*}" + echo "${os}" + fi +} + +function getImageOsVersion() { + echo "nightly" +} + +function getTemurinImageTag() { + getImageTag `getFullTemurinImage ${1:-0} ${2:-jdk}` +} + +function getOpenJ9ImageTag() { + getImageTag `getFullOpenJ9Image ${1:-0} ${2:-jdk}` +} + +function getImageTag() { + local image="$1" + local osAndTag="${image##*/}" + local tag=${osAndTag##*:} + if [ "${tag}" = "${osAndTag}" ] ; then + echo "latest" + else + echo "${tag}" + fi +} + +function getTemurinImageName() { + getImageName `getFullTemurinImage ${1:-0} ${2:-jdk}` +} + +function getOpenJ9ImageName() { + getImageName `getFullOpenJ9Image ${1:-0} ${2:-jdk}` +} + +function getImageName() { + local image="$1" + local name="${image%%:*}" + echo "${name}" +} + +function getTemurinImageTag() { + getImageTag `getFullTemurinImage ${1:-0} ${2:-jdk}` +} + +function getOpenJ9ImageTag() { + getImageTag `getFullOpenJ9Image ${1:-0} ${2:-jdk}` +} + +function getImageTag() { + local image="$1" + local osAndTag="${image##*/}" + local tag=${osAndTag##*:} + if [ "${tag}" = "${osAndTag}" ] ; then + echo "latest" + else + echo "${tag}" + fi +} diff --git a/external/quarkus/test.properties b/external/quarkus/test.properties index bb743e4454..d9d629c5ce 100644 --- a/external/quarkus/test.properties +++ b/external/quarkus/test.properties @@ -3,4 +3,4 @@ test_command="./mvnw -pl '!:quarkus-documentation' clean install" test_results="testResults" tag_version="2.7.5.Final" environment_variable="MODE=java" -ubuntu_packages="git wget" +generic_packages="git wget" diff --git a/external/quarkus_openshift/test.properties b/external/quarkus_openshift/test.properties index a6a6f73e59..2e26a2ffea 100644 --- a/external/quarkus_openshift/test.properties +++ b/external/quarkus_openshift/test.properties @@ -2,4 +2,4 @@ github_url="https://github.com/quarkus-qe/quarkus-openshift-test-suite.git" test_results="testResults" tag_version="2.7.2.Final" environment_variable="MODE=java" -ubuntu_packages="git wget" +generic_packages="git wget" diff --git a/external/quarkus_quickstarts/test.properties b/external/quarkus_quickstarts/test.properties index a5519b70b1..ce9267e039 100644 --- a/external/quarkus_quickstarts/test.properties +++ b/external/quarkus_quickstarts/test.properties @@ -9,5 +9,5 @@ test_command="mvn -pl !:hibernate-orm-multi-tenancy-quickstart,\ test_results="testResults" tag_version="2.7.2.Final" environment_variable="MODE=java" -ubuntu_packages="git wget tar" maven_version="3.8.5" +generic_packages="git wget tar" diff --git a/external/scala/test.properties b/external/scala/test.properties index b80ddbb4c5..9e2b90ee09 100644 --- a/external/scala/test.properties +++ b/external/scala/test.properties @@ -1,5 +1,6 @@ github_url="https://github.com/scala/scala.git" tag_version="v2.13.8" sbt_version="1.6.2" -ubuntu_packages="git wget tar curl gpg gpg-agent" +generic_packages="gpg git wget tar curl" +ubuntu_packages="gpg-agent" test_options="jvm pos neg" diff --git a/external/spring/test.properties b/external/spring/test.properties index af54400e80..d1ddefab4a 100644 --- a/external/spring/test.properties +++ b/external/spring/test.properties @@ -1,4 +1,4 @@ github_url="https://github.com/spring-projects/spring-boot.git" test_results="testResults" -ubuntu_packages="git wget unzip" tag_version="v2.6.6" +generic_packages="git wget unzip" diff --git a/external/system-test/test.properties b/external/system-test/test.properties index e294118187..004370760c 100644 --- a/external/system-test/test.properties +++ b/external/system-test/test.properties @@ -1,4 +1,4 @@ github_url="https://github.com/adoptium/aqa-tests.git" test_results="testResults" gradle_version="5.1" -ubuntu_packages="git wget unzip" +generic_packages="git wget unzip" diff --git a/external/tck-ubi-test/test.properties b/external/tck-ubi-test/test.properties index 8631b96613..da58c67ca6 100644 --- a/external/tck-ubi-test/test.properties +++ b/external/tck-ubi-test/test.properties @@ -2,7 +2,8 @@ github_url="https://github.com/adoptium/aqa-tests.git" test_results="testResults" gradle_version="5.1" environment_variable="MODE=java CC=gcc-7 CXX=g++-7" -ubi_packages="git wget perl tzdata openssl ca-certificates fontconfig glibc-langpack-en gzip tar" +generic_packages="git wget tar" +(fedora|ubi|rhel|centos)_packages="perl tzdata openssl ca-certificates fontconfig glibc-langpack-en gzip" ant_version="1.10.5" ant_contrib_version="1.0b3" jdk_install="true" diff --git a/external/test_base_functions.sh b/external/test_base_functions.sh index ad653de6f7..1451634819 100755 --- a/external/test_base_functions.sh +++ b/external/test_base_functions.sh @@ -25,7 +25,17 @@ echo_env_var() { } echo_java_version() { - if [ -d /java/jre/bin ];then + if [ -d /opt/java/openjdk/jre/bin ];then + echo "Using /opt mounted Java8" + export JAVA_BIN=/opt/java/openjdk/jre/bin + export JAVA_HOME=/opt/java/openjdk + export PATH=$JAVA_BIN:$PATH + elif [ -d /opt/java/openjdk/bin ]; then + echo "Using /opt mounted Java" + export JAVA_BIN=/opt/java/openjdk/bin + export JAVA_HOME=/opt/java/openjdk + export PATH=$JAVA_BIN:$PATH + elif [ -d /java/jre/bin ];then echo "Using mounted Java8" export JAVA_BIN=/java/jre/bin export JAVA_HOME=/java @@ -47,4 +57,4 @@ echo_java_version() { echo "=JAVA VERSION OUTPUT BEGIN=" java -version echo "=JAVA VERSION OUTPUT END=" -} \ No newline at end of file +} diff --git a/external/tomcat/test.properties b/external/tomcat/test.properties index 9d156c83cb..948548cf44 100644 --- a/external/tomcat/test.properties +++ b/external/tomcat/test.properties @@ -1,3 +1,6 @@ github_url="https://github.com/apache/tomcat.git" tag_version="10.0.7" -ubuntu_packages="ant-optional openssl git wget tar make gcc linux-libc-dev libc6-dev perl build-essential automake autoconf libssl-dev subversion libtool-bin" +generic_packages="openssl make gcc perl automake autoconf subversion git wget tar" +ubuntu_packages="ant-optional linux-libc-dev libc6-dev build-essential libssl-dev libtool-bin" +fedora_packages="glibc glibc-devel openssl-devel libtool" +(fedora|ubi|rhel|centos)_packages=make automake gcc gcc-c++ kernel-devel diff --git a/external/tomee/test.properties b/external/tomee/test.properties index 794d067e0c..b99dafd8f4 100644 --- a/external/tomee/test.properties +++ b/external/tomee/test.properties @@ -1,5 +1,5 @@ github_url="https://github.com/apache/tomee.git" test_results="testResults" tag_version="tomee-8.0.2" -ubuntu_packages="git wget tar" maven_version="3.8.5" +generic_packages="git wget tar" diff --git a/external/wildfly/test.properties b/external/wildfly/test.properties index 7a38ac5c11..887ed82933 100644 --- a/external/wildfly/test.properties +++ b/external/wildfly/test.properties @@ -1,5 +1,5 @@ github_url="https://github.com/wildfly/wildfly.git" test_results="" tag_version="26.0.1.Final" -ubuntu_packages="git wget tar" maven_version="3.8.5" +generic_packages="git wget tar" diff --git a/external/wycheproof/test.properties b/external/wycheproof/test.properties index 84a31ca22b..5c792212d8 100644 --- a/external/wycheproof/test.properties +++ b/external/wycheproof/test.properties @@ -2,4 +2,6 @@ github_url="https://github.com/google/wycheproof.git" test_results="testResults" tag_version="master" bazel_version="1.2.1" -ubuntu_packages="git wget unzip zip g++" +generic_packages="git wget unzip zip" +ubuntu_packages="g++" +(fedora|ubi|rhel|centos)_packages="gcc-c++" diff --git a/external/zookeeper/test.properties b/external/zookeeper/test.properties index 1825eefc11..a2ef6f632b 100644 --- a/external/zookeeper/test.properties +++ b/external/zookeeper/test.properties @@ -2,5 +2,5 @@ github_url="https://github.com/apache/zookeeper.git" test_commands="mvn clean package" test_results="testResults" tag_version="release-3.8.0-1" -ubuntu_packages="git make wget tar" -maven_version="3.8.5" \ No newline at end of file +generic_packages="make git wget tar" +maven_version="3.8.5"