-
Notifications
You must be signed in to change notification settings - Fork 1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix github actions build failure #1054
Changes from all commits
9b3bd26
59c5995
73281a2
266890d
67b6a34
7a15fd9
e5d45f8
fe8b32d
e19a0c4
9d37ad5
e98200f
230f675
6f6e90d
8f69377
00f0842
1697e0b
b4de760
426ed3f
ae804f6
be919bb
bc58f7e
b697a05
3a91408
11ab4d5
c6fb425
30b7fc0
0b6b4c3
2564b30
b089632
343d2d3
537db4c
855dfaa
717e671
c2a0aa0
34e2018
6664c0f
b3f2aa4
dbcf157
a31d7cb
68692c9
f04ee49
65a734f
4b93e0e
66f1021
584c10e
d55f66d
6b284c4
34995d0
f9b079b
4eabc42
8cce300
9f8fa55
9494794
d6a32e9
d5ead6a
8b14f4e
3f038bf
8d77355
531e180
eb50a09
23c6a26
00d6c3b
0c72878
49bd794
c3777b6
1a9e163
d25d209
3c41edb
0e63735
502d59a
10fbd06
696e9a5
0e6bf04
3134798
98dbbd1
06ba485
33ff2ed
fdcda5e
8e0e323
19e8602
fe92872
4e293a7
823dd3e
2c8ee76
41f356d
8bed5c3
ed2bf4b
878d76a
9a470c7
1fe5b15
81b49a8
46117cf
0e538d7
c579d48
4c5d89b
a139085
7c223ae
9f7be4b
45ebe0f
2943c9d
79bcb89
f2795b5
c733c24
640f1bf
0419451
45624ab
a7cc9d7
619e74c
020f117
f715e97
a9fe5ad
0e27c68
22bbd24
795f0a5
24419d0
1db6394
76873ba
f921c73
94468af
8d49413
6e37088
a536330
aa7c81d
8ee5b3a
ecbe35f
b068344
40d72df
111ac26
305aecd
5c57db4
e5923a8
b16152e
cfffc6d
0ac03f0
9923b01
fb81fee
765b77c
609ffa3
2be3a26
c6c9489
ca3a1c4
8d6dba5
d158e88
35f8fbc
04c04f9
aad6c37
a7a37ad
1407fc5
2ed5072
0f4c477
6b6fe6f
dcde155
2a1ef4a
b507c5e
fdf148a
de31f65
3a3522d
3d52745
249ed84
df27866
a201e93
9340505
cc6bde5
0032a14
6dc5c42
bebdb7c
cad4be0
39c3c68
28f0618
1479efd
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -43,6 +43,11 @@ jobs: | |
# needed for .withReuse(true) to work | ||
echo "testcontainers.reuse.enable=true" > ~/.testcontainers.properties | ||
- checkout | ||
- restore_cache: | ||
keys: | ||
- spring-cloud-kubernetes-{{ .Branch }}-{{ checksum "pom.xml" }} | ||
- spring-cloud-kubernetes-{{ .Branch }} | ||
- spring-cloud-kubernetes | ||
- attach_workspace: | ||
at: /tmp/docker | ||
- run: | ||
|
@@ -60,19 +65,74 @@ jobs: | |
########################################################################################################################### | ||
################################################# Build test support dependency ########################################### | ||
cd spring-cloud-kubernetes-test-support | ||
.././mvnw clean install | ||
.././mvnw clean install | ||
cd .. | ||
|
||
########################################################################################################################### | ||
##################################################### Split and run tests ################################################# | ||
CLASSNAMES=$(circleci tests glob "**/src/test/**/**.java" | grep -v 'Fabric8IstioIT' \ | ||
| xargs grep -l '@Test' \ | ||
| sed 's/.*src.test.java.//g' | sed 's@/@.@g' \ | ||
| sed 's/.\{5\}$//' \ | ||
| circleci tests split --split-by=timings) | ||
echo $CLASSNAMES | ||
TEST_ARG=$(echo $CLASSNAMES | sed 's/ /,/g') | ||
echo $TEST_ARG | ||
|
||
# - find all tests | ||
# - exclude Fabric8IstionIT | ||
# - only take classes that have @Test inside them | ||
# - ignore the ones that have 'abstract class'. we do this because otherwise we would pass | ||
# to -DtestsToRun an abstract class, and it will not run anything. | ||
# - drop the "begining" xxx/src/test/java | ||
# - replace / with . | ||
# - drop last ".java" | ||
# - replace newline with space | ||
|
||
PLAIN_TEST_CLASSNAMES=($(find . -name '*.java' \ | ||
| grep 'src/test/java' \ | ||
| grep -v 'Fabric8IstioIT' \ | ||
| xargs grep -l '@Test' \ | ||
| xargs grep -L 'abstract class' \ | ||
| sed 's/.*src.test.java.//g' \ | ||
| sed 's@/@.@g' \ | ||
| sed 's/.\{5\}$//')) | ||
|
||
# classes that have @Test and are abstract, for example: "LabeledSecretWithPrefixTests" | ||
# - exclude Fabric8IstionIT | ||
# - only take classes that have @Test inside them | ||
# - only take classes that are abstract | ||
# - drop everything up until the last "/" | ||
# - drop ".java" | ||
|
||
ABSTRACT_TEST_CLASSNAMES_COMMAND="find . -name '*.java' \ | ||
| grep 'src/test/java' \ | ||
| grep -v 'Fabric8IstioIT' \ | ||
| xargs grep -l '@Test' \ | ||
| xargs grep -l 'abstract class' \ | ||
| sed 's/.*\///g' \ | ||
| sed 's/.java//g'" | ||
|
||
# find classes that extend abstract test classes | ||
DERIVED_FROM_ABSTRACT_CLASSES_COMMAND="find . -name '*.java' \ | ||
| grep 'src/test/java' \ | ||
| grep -v 'Fabric8IstioIT' \ | ||
| xargs grep -l 'extends replace_me ' \ | ||
| sed 's/.*src.test.java.//g' \ | ||
| sed 's@/@.@g' \ | ||
| sed 's/.\{5\}$//'" | ||
|
||
while read class_name; do | ||
replaced=$(echo ${DERIVED_FROM_ABSTRACT_CLASSES_COMMAND/replace_me/"$class_name"}) | ||
result=($(eval $replaced)) | ||
PLAIN_TEST_CLASSNAMES+=(${result[@]}) | ||
done < <(eval $ABSTRACT_TEST_CLASSNAMES_COMMAND) | ||
|
||
IFS=$'\n' | ||
SORTED_TEST_CLASSNAMES=( $(sort \<<< "${PLAIN_TEST_CLASSNAMES[*]}" | uniq -u) ) | ||
unset IFS | ||
|
||
printf "%s\n" "${SORTED_TEST_CLASSNAMES[@]}" > file.txt | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I have not found a way to make circleci happy, without writing tests to a file first. At least it works like this |
||
CLASSNAMES=( $(cat file.txt | circleci tests split --split-by=timings) ) | ||
rm file.txt | ||
|
||
TEST_ARG=$(echo ${CLASSNAMES[@]} | sed 's/ /,/g') | ||
echo '----------------------- tests ---------------------' | ||
echo ${TEST_ARG[@]} | ||
echo '---------------------------------------------------' | ||
|
||
./mvnw -s .settings.xml -DfailIfNoTests=false -DtestsToRun=$TEST_ARG -e clean org.jacoco:jacoco-maven-plugin:prepare-agent install \ | ||
-U -P sonar -nsu --batch-mode -Dmaven.test.redirectTestOutputToFile=true \ | ||
-Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn | ||
|
@@ -110,7 +170,7 @@ jobs: | |
- run: | ||
name: dependencies | ||
command: | | ||
./mvnw -s .settings.xml -U dependency:resolve-plugins dependency:go-offline -B -Dservice.occurence=${_SERVICE_OCCURENCE} || true | ||
./mvnw -s .settings.xml -U dependency:resolve-plugins dependency:go-offline -B || true | ||
- run: | ||
name: build | ||
command: | | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,11 @@ | ||
name: build controllers project | ||
description: build controllers project | ||
runs: | ||
using: "composite" | ||
steps: | ||
- name: build controllers project | ||
shell: bash | ||
run: | | ||
cd spring-cloud-kubernetes-controllers | ||
.././mvnw -T 1C clean install | ||
cd .. |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,12 @@ | ||
name: build integration tests project without tests | ||
description: build integration tests project without tests | ||
runs: | ||
using: "composite" | ||
steps: | ||
- name: build integration tests project without tests | ||
shell: bash | ||
run: | | ||
cd spring-cloud-kubernetes-integration-tests | ||
# build the images, but dont run the tests | ||
.././mvnw -T 1C clean install -DskipTests | ||
cd .. |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,11 @@ | ||
name: cache | ||
description: cache | ||
runs: | ||
using: "composite" | ||
steps: | ||
- uses: actions/cache@v2 | ||
with: | ||
path: ~/.m2/repository | ||
key: ${{ runner.os }}-cache-${{ env.BRANCH_NAME }}-${{ hashFiles('**/pom.xml') }} | ||
restore-keys: | | ||
${{ runner.os }}-cache-${{ env.BRANCH_NAME }}-${{ hashFiles('**/pom.xml') }} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,15 @@ | ||
# we are not going to use upload/download actions, since it is slower than cache usage | ||
# as key to the cache we are going to use DOCKER_IMAGES_KEY, which is the same as GITHUB_RUN_ID | ||
|
||
name: download docker images | ||
description: download docker images | ||
runs: | ||
using: "composite" | ||
steps: | ||
- uses: actions/cache@v2 | ||
with: | ||
path: /tmp/docker/images | ||
key: docker-images-cache-${{ env.DOCKER_IMAGES_KEY }} | ||
restore-keys: docker-images-cache-${{ env.DOCKER_IMAGES_KEY }} | ||
|
||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,10 @@ | ||
name: echo saved images | ||
description: echo saved images | ||
runs: | ||
using: "composite" | ||
steps: | ||
- name: echo saved images | ||
shell: bash | ||
run: | | ||
VIEW=$(ls -l /tmp/docker/images) | ||
echo "${VIEW}" |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,10 @@ | ||
name: sets environment variables | ||
description: sets environment variables | ||
runs: | ||
using: "composite" | ||
steps: | ||
- name: set env variables | ||
shell: bash | ||
run: | | ||
echo "BRANCH_NAME=$(echo $GITHUB_HEAD_REF)" >> $GITHUB_ENV | ||
echo "DOCKER_IMAGES_KEY=$(echo $GITHUB_RUN_ID)" >> $GITHUB_ENV |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,12 @@ | ||
name: run fabric8 istio integration test | ||
description: run fabric8 istio integration test | ||
runs: | ||
using: "composite" | ||
steps: | ||
- name: run fabric8 istio integration test | ||
shell: bash | ||
run: | | ||
docker load -i /tmp/docker/images/spring-cloud-kubernetes-fabric8-istio-it.tar | ||
cd spring-cloud-kubernetes-integration-tests/spring-cloud-kubernetes-fabric8-istio-it/ | ||
../.././mvnw clean install -Dskip.build.image=true | ||
cd ../.. |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,14 @@ | ||
# build a lightweight project so that dependencies for the | ||
# maven-surefire-plugin are downloaded and thus cached also | ||
|
||
name: build fabric8 istio project | ||
description: build fabric8 istio projects | ||
runs: | ||
using: "composite" | ||
steps: | ||
- name: build fabric8 istio project | ||
shell: bash | ||
run: | | ||
cd spring-cloud-kubernetes-fabric8-istio | ||
.././mvnw -s ../.settings.xml clean install -U | ||
cd .. |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,22 @@ | ||
name: install docker images | ||
description: load docker images from tar | ||
runs: | ||
using: "composite" | ||
steps: | ||
- name: load docker images | ||
shell: bash | ||
run: | | ||
|
||
# get the name of the images to load from children of spring-cloud-kubernetes-controllers | ||
cd spring-cloud-kubernetes-controllers | ||
while read controller_image; do | ||
docker load -i /tmp/docker/images/${controller_image}.tar | ||
done < <(mvn -Dexec.executable='echo' -Dexec.args='${project.artifactId}' exec:exec -q | grep -v 'spring-cloud-kubernetes-controllers') | ||
cd .. | ||
|
||
# get the name of the images to load from children of spring-cloud-kubernetes-integration-tests | ||
cd spring-cloud-kubernetes-integration-tests | ||
while read integration_test_image; do | ||
docker load -i /tmp/docker/images/${integration_test_image}.tar | ||
done < <(mvn -Dexec.executable='echo' -Dexec.args='${project.artifactId}' exec:exec -q | grep -v 'spring-cloud-kubernetes-integration-tests') | ||
cd .. |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,15 @@ | ||
name: save controller docker images | ||
description: save controller docker images | ||
runs: | ||
using: "composite" | ||
steps: | ||
- name: save controller docker images | ||
shell: bash | ||
run: | | ||
mkdir -p /tmp/docker/images | ||
TAG=$(./mvnw help:evaluate -Dexpression=project.version -q -DforceStdout) | ||
cd spring-cloud-kubernetes-controllers | ||
while read controller_image; do | ||
docker save -o /tmp/docker/images/${controller_image}.tar docker.io/springcloud/${controller_image}:$TAG | ||
done < <(mvn -Dexec.executable='echo' -Dexec.args='${project.artifactId}' exec:exec -q | grep -v 'spring-cloud-kubernetes-controllers') | ||
cd .. |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,15 @@ | ||
name: save integration tests docker images | ||
description: save integration tests docker images | ||
runs: | ||
using: "composite" | ||
steps: | ||
- name: save integration tests docker images | ||
shell: bash | ||
run: | | ||
mkdir -p /tmp/docker/images | ||
TAG=$(./mvnw help:evaluate -Dexpression=project.version -q -DforceStdout) | ||
cd spring-cloud-kubernetes-integration-tests | ||
while read integ_test; do | ||
docker save -o /tmp/docker/images/${integ_test}.tar docker.io/springcloud/${integ_test}:$TAG | ||
done < <(mvn -Dexec.executable='echo' -Dexec.args='${project.artifactId}' exec:exec -q | grep -v 'spring-cloud-kubernetes-integration-tests') | ||
cd .. |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,9 @@ | ||
name: setup project | ||
description: setup project | ||
runs: | ||
using: "composite" | ||
steps: | ||
- uses: actions/setup-java@v2 | ||
with: | ||
distribution: 'temurin' | ||
java-version: '17' |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,15 @@ | ||
# we are not going to use upload/download actions, since it is slower than cache usage | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I am using github composite actions, as it keeps the code much more understandable and manageable. |
||
# as key to the cache we are going to use DOCKER_IMAGES_KEY, which is the same as GITHUB_RUN_ID | ||
|
||
name: upload docker images | ||
description: upload docker images | ||
runs: | ||
using: "composite" | ||
steps: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. circleci, unlike github actions, has a common path for the entire workflow, so for example:
github actions does not have such support, they recommend to use upload/download actions. It turns out that uploading our images, currently takes 15 minutes, far too much. So instead, I am using cache with a slight hack - using |
||
- uses: actions/cache@v2 | ||
with: | ||
path: /tmp/docker/images | ||
key: docker-images-cache-${{ env.DOCKER_IMAGES_KEY }} | ||
restore-keys: docker-images-cache-${{ env.DOCKER_IMAGES_KEY }} | ||
|
||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
restore cache for tests