Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix release pipeline #851

Merged
merged 5 commits into from
Dec 2, 2024
Merged

Fix release pipeline #851

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
f2e02d4
fix release build scripts (p1)
graebm Nov 29, 2024
8af5313
Install golang according to official instructions
graebm Nov 30, 2024
2a44390
not sure what this GO_PATH stuff was about
graebm Nov 30, 2024
0d9e994
put back GO_PATH, with comments explaining it
graebm Dec 2, 2024
3bf4580
install golang with via package manager because it's simple
graebm Dec 2, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,9 @@ if (BUILD_DEPS)
list(APPEND AWSLC_CMAKE_ARGUMENTS -DFIPS=ON)
list(APPEND AWSLC_CMAKE_ARGUMENTS -DDISABLE_PERL=OFF)

# Pick up GO_PATH env-var, set by aws-crt-builder when cross-compiling, see:
# https://github.com/awslabs/aws-crt-builder/blob/31307c808ed9f2ea1eb16503b25a9b582f886481/builder/imports/golang.py#L84
# https://github.com/awslabs/aws-crt-builder/blob/31307c808ed9f2ea1eb16503b25a9b582f886481/builder/actions/cmake.py#L110
if (DEFINED ENV{GO_PATH})
list(APPEND AWSLC_CMAKE_ARGUMENTS -DGO_EXECUTABLE=$ENV{GO_PATH}/go)
message(STATUS "Overriding GO_EXECUTABLE to ${GO_EXECUTABLE}")
Expand Down
16 changes: 13 additions & 3 deletions codebuild/cd/generic-unix-build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,10 +29,20 @@ chmod a+x builder
GIT_TAG=$(git describe --tags)

./builder build -p aws-crt-java --target=$AWS_CRT_TARGET run_tests=false
# Builder corss-compiles the shared lib to `target/cmake-build/aws-crt-java/`, move it to the expected path for mvn to generate the jar.
mv target/cmake-build/aws-crt-java/* target/cmake-build/

# When cross-compiling with builder, the shared lib gets an extra "/aws-crt-java/" in its path.
# Move it to expected location.
if [ -d target/cmake-build/aws-crt-java/lib ]; then
mv target/cmake-build/aws-crt-java/lib target/cmake-build/lib
fi

# Double check that shared lib is where we expect
if ! find target/cmake-build/lib -type f -name "*.so" | grep -q .; then
echo "No .so files found"
exit 1
fi

JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64 mvn -B package -DskipTests -Dshared-lib.skip=true -Dcrt.classifier=$CLASSIFIER

aws s3 cp --recursive --include "*.so" target/cmake-build/lib s3://aws-crt-java-pipeline/${GIT_TAG}/lib
aws s3 cp --recursive --exclude "*" --include "*.so" target/cmake-build/lib s3://aws-crt-java-pipeline/${GIT_TAG}/lib
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding --exclude "*" because:

https://docs.aws.amazon.com/cli/latest/reference/s3/#use-of-exclude-and-include-filters

Note that, by default, all files are included. This means that providing only an --include filter will not change what files are transferred. --include will only re-include files that have been excluded from an --exclude filter. If you only want to upload files with a particular extension, you need to first exclude all files, then re-include the files with the particular extension.

This wasn't causing any bugs, but it's code that wasn't doing what it was supposed to be doing, so fixed this here (and in a few other locations)

aws s3 cp target/ s3://aws-crt-java-pipeline/${GIT_TAG}/jar/ --recursive --exclude "*" --include "aws-crt*.jar"
15 changes: 13 additions & 2 deletions codebuild/cd/linux-aarch64-fips-build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,19 @@ chmod a+x builder
GIT_TAG=$(git describe --tags)

./builder build -p aws-crt-java run_tests=false --target=linux-arm64 --cmake-extra=-DCRT_FIPS=ON
mv target/cmake-build/aws-crt-java/* target/cmake-build/

# When cross-compiling with builder, the shared lib gets an extra "/aws-crt-java/" in its path.
# Move it to expected location.
if [ -d target/cmake-build/aws-crt-java/lib ]; then
mv target/cmake-build/aws-crt-java/lib target/cmake-build/lib
fi

# Double check that shared lib is where we expect
if ! find target/cmake-build/lib -type f -name "*.so" | grep -q .; then
echo "No .so files found"
exit 1
fi

JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64 mvn -B package -DskipTests -Dshared-lib.skip=true -Dcrt.classifier=linux-aarch_64-fips

aws s3 cp --recursive --include "*.so" target/cmake-build/lib s3://aws-crt-java-pipeline/${GIT_TAG}/fips_lib
aws s3 cp --recursive --exclude "*" --include "*.so" target/cmake-build/lib s3://aws-crt-java-pipeline/${GIT_TAG}/fips_lib
5 changes: 2 additions & 3 deletions codebuild/cd/manylinux-x64-fips-build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,8 @@ phases:
- git submodule update --init
# double check aws-lc is the FIPS approved branch.
- bash ./codebuild/cd/test-fips-branch.sh
- curl -OL https://go.dev/dl/go1.21.6.linux-amd64.tar.gz && mkdir ./go
- tar -C ./go -xvf go1.21.6.linux-amd64.tar.gz
- export PATH=$PATH:./go/go/bin
# aws-lc FIPS build requires golang for codegen
- yum install -y golang
- mvn -B package -DskipTests -Dcrt.classifier=linux-x86_64-fips -Dcmake.crt_fips=ON

post_build:
Expand Down
8 changes: 7 additions & 1 deletion codebuild/cd/musl-linux-build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,5 +23,11 @@ docker container prune -f
# Upload the artifacts to S3
export GIT_TAG=$(git describe --tags)

aws s3 cp --recursive --include "*.so" target/cmake-build/lib s3://aws-crt-java-pipeline/${GIT_TAG}/lib
# Double check that shared lib is where we expect
if ! find target/cmake-build/lib -type f -name "*.so" | grep -q .; then
echo "No .so files found"
exit 1
fi

aws s3 cp --recursive --exclude "*" --include "*.so" target/cmake-build/lib s3://aws-crt-java-pipeline/${GIT_TAG}/lib
aws s3 cp target/ s3://aws-crt-java-pipeline/${GIT_TAG}/jar/ --recursive --exclude "*" --include "aws-crt*.jar"
Loading