Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix github actions build failure #1054

Merged
merged 170 commits into from
Aug 17, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
170 commits
Select commit Hold shift + click to select a range
9b3bd26
test
wind57 Jul 26, 2022
59c5995
add timeout
wind57 Jul 26, 2022
73281a2
trigger
wind57 Jul 26, 2022
266890d
add timeout to 60 min
wind57 Jul 26, 2022
67b6a34
test
wind57 Jul 26, 2022
7a15fd9
test
wind57 Jul 26, 2022
e5d45f8
test
wind57 Jul 26, 2022
fe8b32d
fix what tests each instance is running
wind57 Jul 27, 2022
e19a0c4
github fix for abstract tests
wind57 Jul 27, 2022
9d37ad5
remove file
wind57 Jul 27, 2022
e98200f
removed echo
wind57 Jul 27, 2022
230f675
test circleci
wind57 Jul 27, 2022
6f6e90d
escape
wind57 Jul 27, 2022
8f69377
single?
wind57 Jul 27, 2022
00f0842
test
wind57 Jul 27, 2022
1697e0b
bigger timeout
wind57 Jul 27, 2022
b4de760
add a bit more debugging
wind57 Jul 27, 2022
426ed3f
fix circleci
wind57 Jul 27, 2022
ae804f6
split in half
wind57 Jul 28, 2022
be919bb
test
wind57 Jul 28, 2022
bc58f7e
test
wind57 Jul 28, 2022
b697a05
test
wind57 Jul 28, 2022
3a91408
test
wind57 Jul 28, 2022
11ab4d5
test
wind57 Jul 28, 2022
c6fb425
test
wind57 Jul 28, 2022
30b7fc0
test
wind57 Jul 28, 2022
0b6b4c3
test
wind57 Jul 28, 2022
2564b30
test
wind57 Jul 28, 2022
b089632
test
wind57 Jul 28, 2022
343d2d3
test
wind57 Jul 28, 2022
537db4c
test
wind57 Jul 28, 2022
855dfaa
test
wind57 Jul 28, 2022
717e671
test
wind57 Jul 28, 2022
c2a0aa0
test
wind57 Jul 28, 2022
34e2018
test
wind57 Jul 28, 2022
6664c0f
test
wind57 Jul 28, 2022
b3f2aa4
test
wind57 Jul 28, 2022
dbcf157
test
wind57 Jul 28, 2022
a31d7cb
test
wind57 Jul 28, 2022
68692c9
test
wind57 Jul 28, 2022
f04ee49
test
wind57 Jul 28, 2022
65a734f
test
wind57 Jul 28, 2022
4b93e0e
test
wind57 Jul 28, 2022
66f1021
test
wind57 Jul 28, 2022
584c10e
test
wind57 Jul 28, 2022
d55f66d
test
wind57 Jul 28, 2022
6b284c4
test
wind57 Jul 28, 2022
34995d0
test
wind57 Jul 28, 2022
f9b079b
test
wind57 Jul 28, 2022
4eabc42
test
wind57 Jul 28, 2022
8cce300
test
wind57 Jul 28, 2022
9f8fa55
test
wind57 Jul 28, 2022
9494794
test
wind57 Jul 28, 2022
d6a32e9
test
wind57 Jul 28, 2022
d5ead6a
test
wind57 Jul 28, 2022
8b14f4e
test
wind57 Jul 28, 2022
3f038bf
test
wind57 Jul 28, 2022
8d77355
test
wind57 Jul 28, 2022
531e180
test
wind57 Jul 28, 2022
eb50a09
test
wind57 Jul 28, 2022
23c6a26
circleci should work now
wind57 Jul 28, 2022
00d6c3b
circleci should work now
wind57 Jul 28, 2022
0c72878
drop splitting as it seems un-needed
wind57 Jul 28, 2022
49bd794
remove file when done
wind57 Jul 28, 2022
c3777b6
test
wind57 Jul 29, 2022
1a9e163
test
wind57 Jul 29, 2022
d25d209
test
wind57 Jul 29, 2022
3c41edb
test
wind57 Jul 29, 2022
0e63735
test
wind57 Jul 29, 2022
502d59a
test
wind57 Jul 29, 2022
10fbd06
test
wind57 Jul 29, 2022
696e9a5
test
wind57 Jul 29, 2022
0e6bf04
test
wind57 Jul 29, 2022
3134798
test
wind57 Jul 29, 2022
98dbbd1
test
wind57 Jul 29, 2022
06ba485
test
wind57 Jul 29, 2022
33ff2ed
test
wind57 Jul 29, 2022
fdcda5e
test
wind57 Jul 29, 2022
8e0e323
test
wind57 Jul 29, 2022
19e8602
test
wind57 Jul 29, 2022
fe92872
test
wind57 Jul 29, 2022
4e293a7
test
wind57 Jul 29, 2022
823dd3e
test
wind57 Jul 29, 2022
2c8ee76
test
wind57 Jul 29, 2022
41f356d
test
wind57 Jul 29, 2022
8bed5c3
test
wind57 Jul 29, 2022
ed2bf4b
test
wind57 Jul 29, 2022
878d76a
trigger
wind57 Jul 29, 2022
9a470c7
trigger
wind57 Jul 30, 2022
1fe5b15
test
wind57 Jul 30, 2022
81b49a8
test
wind57 Jul 30, 2022
46117cf
test
wind57 Jul 30, 2022
0e538d7
test
wind57 Jul 30, 2022
c579d48
test
wind57 Jul 30, 2022
4c5d89b
test
wind57 Jul 30, 2022
a139085
test
wind57 Jul 30, 2022
7c223ae
test
wind57 Jul 30, 2022
9f7be4b
test
wind57 Jul 30, 2022
45ebe0f
test
wind57 Jul 30, 2022
2943c9d
test
wind57 Jul 30, 2022
79bcb89
test
wind57 Jul 30, 2022
f2795b5
test
wind57 Jul 30, 2022
c733c24
test
wind57 Jul 30, 2022
640f1bf
test
wind57 Jul 30, 2022
0419451
test
wind57 Jul 30, 2022
45624ab
test
wind57 Jul 30, 2022
a7cc9d7
test
wind57 Jul 30, 2022
619e74c
test
wind57 Jul 30, 2022
020f117
test
wind57 Jul 30, 2022
f715e97
test
wind57 Jul 30, 2022
a9fe5ad
test
wind57 Jul 30, 2022
0e27c68
test
wind57 Jul 30, 2022
22bbd24
test
wind57 Jul 30, 2022
795f0a5
test
wind57 Jul 31, 2022
24419d0
trigger
wind57 Jul 31, 2022
1db6394
test
wind57 Jul 31, 2022
76873ba
test
wind57 Jul 31, 2022
f921c73
test
wind57 Jul 31, 2022
94468af
test
wind57 Aug 1, 2022
8d49413
test
wind57 Aug 1, 2022
6e37088
test
wind57 Aug 1, 2022
a536330
test
wind57 Aug 1, 2022
aa7c81d
test
wind57 Aug 1, 2022
8ee5b3a
test
wind57 Aug 1, 2022
ecbe35f
test
wind57 Aug 1, 2022
b068344
test
wind57 Aug 1, 2022
40d72df
test
wind57 Aug 1, 2022
111ac26
test
wind57 Aug 1, 2022
305aecd
test
wind57 Aug 1, 2022
5c57db4
test
wind57 Aug 1, 2022
e5923a8
test
wind57 Aug 1, 2022
b16152e
test
wind57 Aug 1, 2022
cfffc6d
test
wind57 Aug 1, 2022
0ac03f0
test
wind57 Aug 1, 2022
9923b01
test
wind57 Aug 1, 2022
fb81fee
test
wind57 Aug 1, 2022
765b77c
test
wind57 Aug 1, 2022
609ffa3
trigger
wind57 Aug 1, 2022
2be3a26
test
wind57 Aug 1, 2022
c6c9489
test
wind57 Aug 1, 2022
ca3a1c4
test
wind57 Aug 1, 2022
8d6dba5
test
wind57 Aug 1, 2022
d158e88
Merge branch 'main' into fix-build-on-github
wind57 Aug 1, 2022
35f8fbc
added README
wind57 Aug 3, 2022
04c04f9
refactor load images
wind57 Aug 4, 2022
aad6c37
enable reuse
wind57 Aug 4, 2022
a7a37ad
increaase timeout
wind57 Aug 4, 2022
1407fc5
increaase timeout
wind57 Aug 4, 2022
2ed5072
test
wind57 Aug 4, 2022
0f4c477
test
wind57 Aug 4, 2022
6b6fe6f
test
wind57 Aug 4, 2022
dcde155
test
wind57 Aug 4, 2022
2a1ef4a
no return
wind57 Aug 4, 2022
b507c5e
test
wind57 Aug 4, 2022
fdf148a
test
wind57 Aug 4, 2022
de31f65
test
wind57 Aug 4, 2022
3a3522d
test
wind57 Aug 4, 2022
3d52745
test
wind57 Aug 4, 2022
249ed84
test
wind57 Aug 4, 2022
df27866
test
wind57 Aug 4, 2022
a201e93
test
wind57 Aug 4, 2022
9340505
test
wind57 Aug 4, 2022
cc6bde5
test
wind57 Aug 4, 2022
0032a14
test
wind57 Aug 4, 2022
6dc5c42
test
wind57 Aug 4, 2022
bebdb7c
test
wind57 Aug 4, 2022
cad4be0
disable test
wind57 Aug 6, 2022
39c3c68
pom
wind57 Aug 16, 2022
28f0618
Merge branch 'main' into fix-build-on-github
wind57 Aug 17, 2022
1479efd
enable test back
wind57 Aug 17, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
80 changes: 70 additions & 10 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,11 @@ jobs:
# needed for .withReuse(true) to work
echo "testcontainers.reuse.enable=true" > ~/.testcontainers.properties
- checkout
- restore_cache:
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

restore cache for tests

keys:
- spring-cloud-kubernetes-{{ .Branch }}-{{ checksum "pom.xml" }}
- spring-cloud-kubernetes-{{ .Branch }}
- spring-cloud-kubernetes
- attach_workspace:
at: /tmp/docker
- run:
Expand All @@ -60,19 +65,74 @@ jobs:
###########################################################################################################################
################################################# Build test support dependency ###########################################
cd spring-cloud-kubernetes-test-support
.././mvnw clean install
.././mvnw clean install
cd ..

###########################################################################################################################
##################################################### Split and run tests #################################################
CLASSNAMES=$(circleci tests glob "**/src/test/**/**.java" | grep -v 'Fabric8IstioIT' \
| xargs grep -l '@Test' \
| sed 's/.*src.test.java.//g' | sed 's@/@.@g' \
| sed 's/.\{5\}$//' \
| circleci tests split --split-by=timings)
echo $CLASSNAMES
TEST_ARG=$(echo $CLASSNAMES | sed 's/ /,/g')
echo $TEST_ARG

# - find all tests
# - exclude Fabric8IstionIT
# - only take classes that have @Test inside them
# - ignore the ones that have 'abstract class'. we do this because otherwise we would pass
# to -DtestsToRun an abstract class, and it will not run anything.
# - drop the "begining" xxx/src/test/java
# - replace / with .
# - drop last ".java"
# - replace newline with space

PLAIN_TEST_CLASSNAMES=($(find . -name '*.java' \
| grep 'src/test/java' \
| grep -v 'Fabric8IstioIT' \
| xargs grep -l '@Test' \
| xargs grep -L 'abstract class' \
| sed 's/.*src.test.java.//g' \
| sed 's@/@.@g' \
| sed 's/.\{5\}$//'))

# classes that have @Test and are abstract, for example: "LabeledSecretWithPrefixTests"
# - exclude Fabric8IstionIT
# - only take classes that have @Test inside them
# - only take classes that are abstract
# - drop everything up until the last "/"
# - drop ".java"

ABSTRACT_TEST_CLASSNAMES_COMMAND="find . -name '*.java' \
| grep 'src/test/java' \
| grep -v 'Fabric8IstioIT' \
| xargs grep -l '@Test' \
| xargs grep -l 'abstract class' \
| sed 's/.*\///g' \
| sed 's/.java//g'"

# find classes that extend abstract test classes
DERIVED_FROM_ABSTRACT_CLASSES_COMMAND="find . -name '*.java' \
| grep 'src/test/java' \
| grep -v 'Fabric8IstioIT' \
| xargs grep -l 'extends replace_me ' \
| sed 's/.*src.test.java.//g' \
| sed 's@/@.@g' \
| sed 's/.\{5\}$//'"

while read class_name; do
replaced=$(echo ${DERIVED_FROM_ABSTRACT_CLASSES_COMMAND/replace_me/"$class_name"})
result=($(eval $replaced))
PLAIN_TEST_CLASSNAMES+=(${result[@]})
done < <(eval $ABSTRACT_TEST_CLASSNAMES_COMMAND)

IFS=$'\n'
SORTED_TEST_CLASSNAMES=( $(sort \<<< "${PLAIN_TEST_CLASSNAMES[*]}" | uniq -u) )
unset IFS

printf "%s\n" "${SORTED_TEST_CLASSNAMES[@]}" > file.txt
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have not found a way to make circleci happy, without writing tests to a file first. At least it works like this

CLASSNAMES=( $(cat file.txt | circleci tests split --split-by=timings) )
rm file.txt

TEST_ARG=$(echo ${CLASSNAMES[@]} | sed 's/ /,/g')
echo '----------------------- tests ---------------------'
echo ${TEST_ARG[@]}
echo '---------------------------------------------------'

./mvnw -s .settings.xml -DfailIfNoTests=false -DtestsToRun=$TEST_ARG -e clean org.jacoco:jacoco-maven-plugin:prepare-agent install \
-U -P sonar -nsu --batch-mode -Dmaven.test.redirectTestOutputToFile=true \
-Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn
Expand Down Expand Up @@ -110,7 +170,7 @@ jobs:
- run:
name: dependencies
command: |
./mvnw -s .settings.xml -U dependency:resolve-plugins dependency:go-offline -B -Dservice.occurence=${_SERVICE_OCCURENCE} || true
./mvnw -s .settings.xml -U dependency:resolve-plugins dependency:go-offline -B || true
- run:
name: build
command: |
Expand Down
11 changes: 11 additions & 0 deletions .github/workflows/composites/build-controllers-project/action.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
name: build controllers project
description: build controllers project
runs:
using: "composite"
steps:
- name: build controllers project
shell: bash
run: |
cd spring-cloud-kubernetes-controllers
.././mvnw -T 1C clean install
cd ..
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
name: build integration tests project without tests
description: build integration tests project without tests
runs:
using: "composite"
steps:
- name: build integration tests project without tests
shell: bash
run: |
cd spring-cloud-kubernetes-integration-tests
# build the images, but dont run the tests
.././mvnw -T 1C clean install -DskipTests
cd ..
11 changes: 11 additions & 0 deletions .github/workflows/composites/cache/action.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
name: cache
description: cache
runs:
using: "composite"
steps:
- uses: actions/cache@v2
with:
path: ~/.m2/repository
key: ${{ runner.os }}-cache-${{ env.BRANCH_NAME }}-${{ hashFiles('**/pom.xml') }}
restore-keys: |
${{ runner.os }}-cache-${{ env.BRANCH_NAME }}-${{ hashFiles('**/pom.xml') }}
15 changes: 15 additions & 0 deletions .github/workflows/composites/download-docker-images/action.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# we are not going to use upload/download actions, since it is slower than cache usage
# as key to the cache we are going to use DOCKER_IMAGES_KEY, which is the same as GITHUB_RUN_ID

name: download docker images
description: download docker images
runs:
using: "composite"
steps:
- uses: actions/cache@v2
with:
path: /tmp/docker/images
key: docker-images-cache-${{ env.DOCKER_IMAGES_KEY }}
restore-keys: docker-images-cache-${{ env.DOCKER_IMAGES_KEY }}


10 changes: 10 additions & 0 deletions .github/workflows/composites/echo-saved-images/action.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
name: echo saved images
description: echo saved images
runs:
using: "composite"
steps:
- name: echo saved images
shell: bash
run: |
VIEW=$(ls -l /tmp/docker/images)
echo "${VIEW}"
10 changes: 10 additions & 0 deletions .github/workflows/composites/env-variables/action.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
name: sets environment variables
description: sets environment variables
runs:
using: "composite"
steps:
- name: set env variables
shell: bash
run: |
echo "BRANCH_NAME=$(echo $GITHUB_HEAD_REF)" >> $GITHUB_ENV
echo "DOCKER_IMAGES_KEY=$(echo $GITHUB_RUN_ID)" >> $GITHUB_ENV
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
name: run fabric8 istio integration test
description: run fabric8 istio integration test
runs:
using: "composite"
steps:
- name: run fabric8 istio integration test
shell: bash
run: |
docker load -i /tmp/docker/images/spring-cloud-kubernetes-fabric8-istio-it.tar
cd spring-cloud-kubernetes-integration-tests/spring-cloud-kubernetes-fabric8-istio-it/
../.././mvnw clean install -Dskip.build.image=true
cd ../..
14 changes: 14 additions & 0 deletions .github/workflows/composites/fabric8-istio/action.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# build a lightweight project so that dependencies for the
# maven-surefire-plugin are downloaded and thus cached also

name: build fabric8 istio project
description: build fabric8 istio projects
runs:
using: "composite"
steps:
- name: build fabric8 istio project
shell: bash
run: |
cd spring-cloud-kubernetes-fabric8-istio
.././mvnw -s ../.settings.xml clean install -U
cd ..
22 changes: 22 additions & 0 deletions .github/workflows/composites/load-docker-images/action.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
name: install docker images
description: load docker images from tar
runs:
using: "composite"
steps:
- name: load docker images
shell: bash
run: |

# get the name of the images to load from children of spring-cloud-kubernetes-controllers
cd spring-cloud-kubernetes-controllers
while read controller_image; do
docker load -i /tmp/docker/images/${controller_image}.tar
done < <(mvn -Dexec.executable='echo' -Dexec.args='${project.artifactId}' exec:exec -q | grep -v 'spring-cloud-kubernetes-controllers')
cd ..

# get the name of the images to load from children of spring-cloud-kubernetes-integration-tests
cd spring-cloud-kubernetes-integration-tests
while read integration_test_image; do
docker load -i /tmp/docker/images/${integration_test_image}.tar
done < <(mvn -Dexec.executable='echo' -Dexec.args='${project.artifactId}' exec:exec -q | grep -v 'spring-cloud-kubernetes-integration-tests')
cd ..
15 changes: 15 additions & 0 deletions .github/workflows/composites/save-controller-images/action.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
name: save controller docker images
description: save controller docker images
runs:
using: "composite"
steps:
- name: save controller docker images
shell: bash
run: |
mkdir -p /tmp/docker/images
TAG=$(./mvnw help:evaluate -Dexpression=project.version -q -DforceStdout)
cd spring-cloud-kubernetes-controllers
while read controller_image; do
docker save -o /tmp/docker/images/${controller_image}.tar docker.io/springcloud/${controller_image}:$TAG
done < <(mvn -Dexec.executable='echo' -Dexec.args='${project.artifactId}' exec:exec -q | grep -v 'spring-cloud-kubernetes-controllers')
cd ..
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
name: save integration tests docker images
description: save integration tests docker images
runs:
using: "composite"
steps:
- name: save integration tests docker images
shell: bash
run: |
mkdir -p /tmp/docker/images
TAG=$(./mvnw help:evaluate -Dexpression=project.version -q -DforceStdout)
cd spring-cloud-kubernetes-integration-tests
while read integ_test; do
docker save -o /tmp/docker/images/${integ_test}.tar docker.io/springcloud/${integ_test}:$TAG
done < <(mvn -Dexec.executable='echo' -Dexec.args='${project.artifactId}' exec:exec -q | grep -v 'spring-cloud-kubernetes-integration-tests')
cd ..
9 changes: 9 additions & 0 deletions .github/workflows/composites/setup/action.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
name: setup project
description: setup project
runs:
using: "composite"
steps:
- uses: actions/setup-java@v2
with:
distribution: 'temurin'
java-version: '17'
15 changes: 15 additions & 0 deletions .github/workflows/composites/upload-docker-images/action.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# we are not going to use upload/download actions, since it is slower than cache usage
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am using github composite actions, as it keeps the code much more understandable and manageable.

# as key to the cache we are going to use DOCKER_IMAGES_KEY, which is the same as GITHUB_RUN_ID

name: upload docker images
description: upload docker images
runs:
using: "composite"
steps:
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

circleci, unlike github actions, has a common path for the entire workflow, so for example:

  • I can put images into /tmp/docker/images in one step
  • reference these images via docker load -i in any step I would like

github actions does not have such support, they recommend to use upload/download actions. It turns out that uploading our images, currently takes 15 minutes, far too much.

So instead, I am using cache with a slight hack - using DOCKER_IMAGES_KEY (echo "DOCKER_IMAGES_KEY=$(echo $GITHUB_RUN_ID)" >> $GITHUB_ENV) that I set in env-variables composite action and use that. Since GITHUB_RUN_ID is unique per workflow, this achieves the "upload/download" without respective actions, but using cache action instead. It shrinks this step to 1 minute.

- uses: actions/cache@v2
with:
path: /tmp/docker/images
key: docker-images-cache-${{ env.DOCKER_IMAGES_KEY }}
restore-keys: docker-images-cache-${{ env.DOCKER_IMAGES_KEY }}


Loading