[feat][build] Add support for ARM image (#12)

* [Dockerfile] Enable retries for apt-get when building Pulsar docker image (apache#14513) - also reduce default timeout to 30 seconds - prevents issues where apt repository doesn't respond (cherry picked from commit d3f6fe3) * PIP-155: Removed Python 2 support (apache#15376) * Remove Pulsar Client Build for Python 2.7 * Remove outdated homebrew files (source of truth is upstream homebrew) * Remove Python 2.7 build references; print error in some cases * Update python client tests to run with python client for python 3.5m * PIP-155: Removed Python 2 support * Fixed invocation in pulsar-build image * Fixed clang-format-10 indent differences * Fixed script invocation with wrong python * We don't need to rebuild the manylinux image each time * Fixed image name * Reverted back to use newer protobuf * Fixed image name * Fixed missing python3 in centos:7 image * Use python3 for gtest-parallel * Show bash commands in docker-tests.sh * Fixed gh action issue with git directory permissions * Fixed python to 3 * Fixed custom_logger_test.py * Fixed path in run_python_instance_tests.sh * Function runtime should use python3 * Fixed function runtime test python expectation * Fixed presto worker launcher * Fixed notes on how to format C++ code Co-authored-by: Michael Marshall <[email protected]> (cherry picked from commit 2b2e0c5) Signed-off-by: Zixuan Liu <[email protected]> * [improve][docker] Switch to Temurin JDK (apache#17129) Signed-off-by: Zixuan Liu <[email protected]> (cherry picked from commit 4378856) Signed-off-by: Zixuan Liu <[email protected]> * [refactor][ci] Build the docker image with docker-maven-plugin (apache#17148) (cherry picked from commit a68b58d) Signed-off-by: Zixuan Liu <[email protected]> * [feat][build] Support ARM64-based docker images (apache#17733) (cherry picked from commit 9a2aeb2) Signed-off-by: Zixuan Liu <[email protected]> * PIP-209: Removed C++/Python clients from main repo (apache#17881) * PIP-209: Removed C++/Python clients from main repo * Removed python directory from Docekrfile * Fixed python client version argument scoping * Fixed handling of pulsar.functions.serde (cherry picked from commit f3c547b) Signed-off-by: Zixuan Liu <[email protected]> * [improve][build] Avoid building image multiple times (apache#17208) Signed-off-by: Zixuan Liu <[email protected]> (cherry picked from commit 79a97a9) * [improve] Allow to build and push multi-arch Docker images (apache#19432) Co-authored-by: Lari Hotari <[email protected]> Co-authored-by: Yong Zhang <[email protected]> Co-authored-by: Zixuan Liu <[email protected]> Co-authored-by: tison <[email protected]> (cherry picked from commit 4190e40) Signed-off-by: Zixuan Liu <[email protected]> * [fix][build] Fix publish image script (apache#20305) Signed-off-by: Zixuan Liu <[email protected]> (cherry picked from commit 94c7bf3) Signed-off-by: Zixuan Liu <[email protected]> * [fix][build] Fix the pulsar-all image may use the wrong upstream image (apache#20435) Signed-off-by: Zike Yang <[email protected]> Co-authored-by: Lari Hotari <[email protected]> (cherry picked from commit d7f3558) Signed-off-by: Zixuan Liu <[email protected]> * [feat][build] Adapt to Python client to be compatible with ARM arch Signed-off-by: Zixuan Liu <[email protected]> * [fix][build] Configure git-commit-id-plugin to skip git describe (apache#20550) (cherry picked from commit 05f7e62) * [improve][misc] Include native epoll library for Netty for arm64 From apache#22319 Signed-off-by: Zixuan Liu <[email protected]> * [fix][misc] Rename all shaded Netty native libraries From apache#22415 Signed-off-by: Zixuan Liu <[email protected]> * [cleanup][build] Cleanup -Ddocker.nocache=true Signed-off-by: Zixuan Liu <[email protected]> * [fix][build] Fix ubuntu mirror Signed-off-by: Zixuan Liu <[email protected]> * [fix][build] Fix license Signed-off-by: Zixuan Liu <[email protected]> * [fix][build] Downgrade docker-maven to 0.43.3 Signed-off-by: Zixuan Liu <[email protected]> --------- Signed-off-by: Zixuan Liu <[email protected]> Signed-off-by: Zike Yang <[email protected]> Co-authored-by: Lari Hotari <[email protected]> Co-authored-by: Matteo Merli <[email protected]> Co-authored-by: Michael Marshall <[email protected]> Co-authored-by: tison <[email protected]> Co-authored-by: Yong Zhang <[email protected]> Co-authored-by: Zike Yang <[email protected]> Co-authored-by: Lari Hotari <[email protected]>
ascentstream · May 10, 2024 · 67a2d52 · 67a2d52
1 parent 0e1969d
commit 67a2d52
Show file tree

Hide file tree

Showing 513 changed files with 423 additions and 68,372 deletions.
diff --git a/.github/actions/tune-runner-vm/action.yml b/.github/actions/tune-runner-vm/action.yml
@@ -70,6 +70,14 @@ runs:
 
             # stop Azure Linux agent to save RAM
             sudo systemctl stop walinuxagent.service || true
+          
+            # enable docker experimental mode which is
+            # required for using "docker build --squash" / "-Ddocker.squash=true"
+            daemon_json="$(sudo cat /etc/docker/daemon.json  | jq '.experimental = true')"
+            echo "$daemon_json" | sudo tee /etc/docker/daemon.json
+            # restart docker daemon
+            sudo systemctl restart docker
+            echo '::endgroup::'
 
             # show memory
             free -m

diff --git a/.github/workflows/ci-cpp-build-centos7.yaml b/.github/workflows/ci-cpp-build-centos7.yaml
diff --git a/.github/workflows/ci-cpp-build-windows.yaml b/.github/workflows/ci-cpp-build-windows.yaml
diff --git a/.github/workflows/ci-integration-backwards-compatibility.yaml b/.github/workflows/ci-integration-backwards-compatibility.yaml
@@ -98,11 +98,11 @@ jobs:
 
       - name: build pulsar image
         if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
-        run: mvn -B -f docker/pulsar/pom.xml install -am -Pdocker,-main -DskipTests -Ddocker.nocache=true
+        run: mvn -B -f docker/pulsar/pom.xml install -am -Pdocker,-main -DskipTests -DUBUNTU_MIRROR=http://azure.archive.ubuntu.com/ubuntu/
 
       - name: build pulsar-all image
         if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
-        run: mvn -B -f docker/pulsar-all/pom.xml install -am -Pdocker,-main -DskipTests -Ddocker.nocache=true
+        run: mvn -B -f docker/pulsar-all/pom.xml install -am -Pdocker,-main -DskipTests -DUBUNTU_MIRROR=http://azure.archive.ubuntu.com/ubuntu/
 
       - name: build artifacts and docker image
         if: ${{ steps.check_changes.outputs.docs_only != 'true' }}

diff --git a/.github/workflows/ci-integration-cli.yaml b/.github/workflows/ci-integration-cli.yaml
@@ -88,11 +88,11 @@ jobs:
 
       - name: build pulsar image
         if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
-        run: mvn -B -f docker/pulsar/pom.xml install -am -Pdocker,-main -DskipTests -Ddocker.nocache=true
+        run: mvn -B -f docker/pulsar/pom.xml install -am -Pdocker,-main -DskipTests -DUBUNTU_MIRROR=http://azure.archive.ubuntu.com/ubuntu/
 
       - name: build pulsar-all image
         if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
-        run: mvn -B -f docker/pulsar-all/pom.xml install -am -Pdocker,-main -DskipTests -Ddocker.nocache=true
+        run: mvn -B -f docker/pulsar-all/pom.xml install -am -Pdocker,-main -DskipTests -DUBUNTU_MIRROR=http://azure.archive.ubuntu.com/ubuntu/
 
       - name: build artifacts and docker image
         if: ${{ steps.check_changes.outputs.docs_only != 'true' }}

diff --git a/.github/workflows/ci-integration-function.yaml b/.github/workflows/ci-integration-function.yaml
@@ -88,11 +88,11 @@ jobs:
 
       - name: build pulsar image
         if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
-        run: mvn -B -f docker/pulsar/pom.xml install -am -Pdocker,-main -DskipTests -Ddocker.nocache=true
+        run: mvn -B -f docker/pulsar/pom.xml install -am -Pdocker,-main -DskipTests -DUBUNTU_MIRROR=http://azure.archive.ubuntu.com/ubuntu/
 
       - name: build pulsar-all image
         if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
-        run: mvn -B -f docker/pulsar-all/pom.xml install -am -Pdocker,-main -DskipTests -Ddocker.nocache=true
+        run: mvn -B -f docker/pulsar-all/pom.xml install -am -Pdocker,-main -DskipTests -DUBUNTU_MIRROR=http://azure.archive.ubuntu.com/ubuntu/
 
       - name: build artifacts and docker image
         if: ${{ steps.check_changes.outputs.docs_only != 'true' }}

diff --git a/.github/workflows/ci-integration-messaging.yaml b/.github/workflows/ci-integration-messaging.yaml
@@ -88,15 +88,15 @@ jobs:
 
       - name: build pulsar image
         if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
-        run: mvn -B -f docker/pulsar/pom.xml install -am -Pdocker,-main -DskipTests -Ddocker.nocache=true
+        run: mvn -B -f docker/pulsar/pom.xml install -am -Pdocker,-main -DskipTests -DUBUNTU_MIRROR=http://azure.archive.ubuntu.com/ubuntu/
 
       - name: build pulsar-all image
         if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
-        run: mvn -B -f docker/pulsar-all/pom.xml install -am -Pdocker,-main -DskipTests -Ddocker.nocache=true
+        run: mvn -B -f docker/pulsar-all/pom.xml install -am -Pdocker,-main -DskipTests -DUBUNTU_MIRROR=http://azure.archive.ubuntu.com/ubuntu/
 
       - name: build artifacts and docker image
         if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
-        run: mvn -B -f tests/docker-images/pom.xml install -am -Pdocker,-main -DskipTests -Ddocker.nocache=true
+        run: mvn -B -f tests/docker-images/pom.xml install -am -Pdocker,-main -DskipTests
 
       - name: run integration tests
         if: ${{ steps.check_changes.outputs.docs_only != 'true' }}

diff --git a/.github/workflows/ci-integration-process.yaml b/.github/workflows/ci-integration-process.yaml
@@ -87,11 +87,11 @@ jobs:
 
       - name: build pulsar image
         if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
-        run: mvn -B -f docker/pulsar/pom.xml install -am -Pdocker,-main -DskipTests -Ddocker.nocache=true
+        run: mvn -B -f docker/pulsar/pom.xml install -am -Pdocker,-main -DskipTests -DUBUNTU_MIRROR=http://azure.archive.ubuntu.com/ubuntu/
 
       - name: build pulsar-all image
         if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
-        run: mvn -B -f docker/pulsar-all/pom.xml install -am -Pdocker,-main -DskipTests -Ddocker.nocache=true
+        run: mvn -B -f docker/pulsar-all/pom.xml install -am -Pdocker,-main -DskipTests -DUBUNTU_MIRROR=http://azure.archive.ubuntu.com/ubuntu/
 
       - name: build artifacts and docker image
         if: ${{ steps.check_changes.outputs.docs_only != 'true' }}

diff --git a/.github/workflows/ci-integration-pulsar-io-ora.yaml b/.github/workflows/ci-integration-pulsar-io-ora.yaml
@@ -98,11 +98,11 @@ jobs:
 
       - name: build pulsar image
         if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
-        run: mvn -B -f docker/pulsar/pom.xml install -am -Pdocker,-main -DskipTests -Ddocker.nocache=true
+        run: mvn -B -f docker/pulsar/pom.xml install -am -Pdocker,-main -DskipTests -DUBUNTU_MIRROR=http://azure.archive.ubuntu.com/ubuntu/
 
       - name: build pulsar-all image
         if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
-        run: mvn -B -f docker/pulsar-all/pom.xml install -am -Pdocker,-main -DskipTests -Ddocker.nocache=true
+        run: mvn -B -f docker/pulsar-all/pom.xml install -am -Pdocker,-main -DskipTests -DUBUNTU_MIRROR=http://azure.archive.ubuntu.com/ubuntu/
 
       - name: build artifacts and docker image
         if: ${{ steps.check_changes.outputs.docs_only != 'true' }}

diff --git a/.github/workflows/ci-integration-pulsar-io.yaml b/.github/workflows/ci-integration-pulsar-io.yaml
@@ -98,11 +98,11 @@ jobs:
 
       - name: build pulsar image
         if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
-        run: mvn -B -f docker/pulsar/pom.xml install -am -Pdocker,-main -DskipTests -Ddocker.nocache=true
+        run: mvn -B -f docker/pulsar/pom.xml install -am -Pdocker,-main -DskipTests -DUBUNTU_MIRROR=http://azure.archive.ubuntu.com/ubuntu/
 
       - name: build pulsar-all image
         if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
-        run: mvn -B -f docker/pulsar-all/pom.xml install -am -Pdocker,-main -DskipTests -Ddocker.nocache=true
+        run: mvn -B -f docker/pulsar-all/pom.xml install -am -Pdocker,-main -DskipTests -DUBUNTU_MIRROR=http://azure.archive.ubuntu.com/ubuntu/
 
       - name: build artifacts and docker image
         if: ${{ steps.check_changes.outputs.docs_only != 'true' }}

diff --git a/.github/workflows/ci-integration-schema.yaml b/.github/workflows/ci-integration-schema.yaml
@@ -86,10 +86,10 @@ jobs:
         run: mvn -q -B -ntp clean install -DskipTests
       - name: build pulsar image
         if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
-        run: mvn -B -f docker/pulsar/pom.xml install -am -Pdocker,-main -DskipTests -Ddocker.nocache=true
+        run: mvn -B -f docker/pulsar/pom.xml install -am -Pdocker,-main -DskipTests -DUBUNTU_MIRROR=http://azure.archive.ubuntu.com/ubuntu/
       - name: build pulsar-all image
         if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
-        run: mvn -B -f docker/pulsar-all/pom.xml install -am -Pdocker,-main -DskipTests -Ddocker.nocache=true
+        run: mvn -B -f docker/pulsar-all/pom.xml install -am -Pdocker,-main -DskipTests -DUBUNTU_MIRROR=http://azure.archive.ubuntu.com/ubuntu/
       - name: build artifacts and docker image
         if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
         run: mvn -B -f tests/docker-images/pom.xml install -am -Pdocker,-main -DskipTests

diff --git a/.github/workflows/ci-integration-sql.yaml b/.github/workflows/ci-integration-sql.yaml
@@ -88,11 +88,11 @@ jobs:
 
       - name: build pulsar image
         if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
-        run: mvn -B -f docker/pulsar/pom.xml install -am -Pdocker,-main -DskipTests -Ddocker.nocache=true
+        run: mvn -B -f docker/pulsar/pom.xml install -am -Pdocker,-main -DskipTests -DUBUNTU_MIRROR=http://azure.archive.ubuntu.com/ubuntu/
 
       - name: build pulsar-all image
         if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
-        run: mvn -B -f docker/pulsar-all/pom.xml install -am -Pdocker,-main -DskipTests -Ddocker.nocache=true
+        run: mvn -B -f docker/pulsar-all/pom.xml install -am -Pdocker,-main -DskipTests -DUBUNTU_MIRROR=http://azure.archive.ubuntu.com/ubuntu/
 
       - name: build artifacts and docker pulsar latest test image
         if: ${{ steps.check_changes.outputs.docs_only != 'true' }}

diff --git a/.github/workflows/ci-integration-standalone.yaml b/.github/workflows/ci-integration-standalone.yaml
@@ -87,11 +87,11 @@ jobs:
 
       - name: build pulsar image
         if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
-        run: mvn -B -f docker/pulsar/pom.xml install -am -Pdocker,-main -DskipTests -Ddocker.nocache=true
+        run: mvn -B -f docker/pulsar/pom.xml install -am -Pdocker,-main -DskipTests -DUBUNTU_MIRROR=http://azure.archive.ubuntu.com/ubuntu/
 
       - name: build pulsar-all image
         if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
-        run: mvn -B -f docker/pulsar-all/pom.xml install -am -Pdocker,-main -DskipTests -Ddocker.nocache=true
+        run: mvn -B -f docker/pulsar-all/pom.xml install -am -Pdocker,-main -DskipTests -DUBUNTU_MIRROR=http://azure.archive.ubuntu.com/ubuntu/
 
       - name: build artifacts and docker image
         if: ${{ steps.check_changes.outputs.docs_only != 'true' }}

diff --git a/.github/workflows/ci-integration-thread.yaml b/.github/workflows/ci-integration-thread.yaml
@@ -87,11 +87,11 @@ jobs:
 
       - name: build pulsar image
         if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
-        run: mvn -B -f docker/pulsar/pom.xml install -am -Pdocker,-main -DskipTests -Ddocker.nocache=true
+        run: mvn -B -f docker/pulsar/pom.xml install -am -Pdocker,-main -DskipTests -DUBUNTU_MIRROR=http://azure.archive.ubuntu.com/ubuntu/
 
       - name: build pulsar-all image
         if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
-        run: mvn -B -f docker/pulsar-all/pom.xml install -am -Pdocker,-main -DskipTests -Ddocker.nocache=true
+        run: mvn -B -f docker/pulsar-all/pom.xml install -am -Pdocker,-main -DskipTests -DUBUNTU_MIRROR=http://azure.archive.ubuntu.com/ubuntu/
 
       - name: build artifacts and docker image
         if: ${{ steps.check_changes.outputs.docs_only != 'true' }}

diff --git a/.github/workflows/ci-integration-tiered-filesystem.yaml b/.github/workflows/ci-integration-tiered-filesystem.yaml
@@ -87,11 +87,11 @@ jobs:
 
       - name: build pulsar image
         if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
-        run: mvn -B -f docker/pulsar/pom.xml install -am -Pdocker,-main -DskipTests -Ddocker.nocache=true
+        run: mvn -B -f docker/pulsar/pom.xml install -am -Pdocker,-main -DskipTests -DUBUNTU_MIRROR=http://azure.archive.ubuntu.com/ubuntu/
 
       - name: build pulsar-all image
         if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
-        run: mvn -B -f docker/pulsar-all/pom.xml install -am -Pdocker,-main -DskipTests -Ddocker.nocache=true
+        run: mvn -B -f docker/pulsar-all/pom.xml install -am -Pdocker,-main -DskipTests -DUBUNTU_MIRROR=http://azure.archive.ubuntu.com/ubuntu/
 
       - name: build artifacts and docker image
         if: ${{ steps.check_changes.outputs.docs_only != 'true' }}

diff --git a/.github/workflows/ci-integration-tiered-jcloud.yaml b/.github/workflows/ci-integration-tiered-jcloud.yaml
@@ -87,11 +87,11 @@ jobs:
 
       - name: build pulsar image
         if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
-        run: mvn -B -f docker/pulsar/pom.xml install -am -Pdocker,-main -DskipTests -Ddocker.nocache=true
+        run: mvn -B -f docker/pulsar/pom.xml install -am -Pdocker,-main -DskipTests -DUBUNTU_MIRROR=http://azure.archive.ubuntu.com/ubuntu/
 
       - name: build pulsar-all image
         if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
-        run: mvn -B -f docker/pulsar-all/pom.xml install -am -Pdocker,-main -DskipTests -Ddocker.nocache=true
+        run: mvn -B -f docker/pulsar-all/pom.xml install -am -Pdocker,-main -DskipTests -DUBUNTU_MIRROR=http://azure.archive.ubuntu.com/ubuntu/
 
       - name: build artifacts and docker image
         if: ${{ steps.check_changes.outputs.docs_only != 'true' }}