Automate AWS CDK and Terraform tests w/ GitHub Actions (#21)

* Add more descriptive instructions * Fix make log output contaminating JSON output * Fix CLI and SDK version mismatch * Point to localhost if running outside container * Typo * Add nit comment * Fix nit * Add weekly github actions run * Nits and fixes * Fix setup stage * Fixes * Temporarily show all env vars * Remove unnecessary cd * Inline injection of auth token * Attempt no compose flags * Add support for amd64 arch * Fail the last step if one of the tests failed * Run localstack in background * Force recreate docker compose * Use `DOCKER_DEFAULT_PLATFORM` instead * Remove disallowed flag * Revert "Use `DOCKER_DEFAULT_PLATFORM` instead" This reverts commit d750e9c. # Conflicts: # .github/workflows/weekly_test.yml * Adopt multi-arch for Dockerfile.js.layer * Add missing npm packages * Fixes * Destroy the stacks and fix aws cdk pytest test * Add localhost.localstack.cloud host to /etc/hosts * Rename docker compose files * Parallelize pipelines and run tests in isolated env * Fix docker compose file paths * Remove unnecessary sudo * Add missing curl dependency * Add more missing packages * Fix installation of terraform * More fixes * Fix sourcing of profile in non-interactive mode, missing deps again, exit on first error fix, non-expandable bash instructions * Run make setup as well * Optimized Dockerfile + fix bug on amd64 arch * DNS fix, AWS CLI bug workaround * Install docker CLI required for GH actions * Many fixes * Temporary not set arch * Attempt arch fix for awscdk * Add temporary log * Remove arch completely from awscdk * cdktf fixes * Temporarily disable arch on cdktf * Arch fixes and docs * Improve docs a bit * Run tests on main and on PRs * Address PR comments * Update weekly_test.yml * Attempt lower timeout * Attempt Python version 3.10 instead of 3.11 * Add verbosity to pyenv install * Increase timeout to 60 mins * Attempt to catch *"x86_64"* pattern * Temporarily run only one test * Add docker layer caching * Revert "Temporarily run only one test" This reverts commit 7c928d0. * Remove docker layer caching because we run out of space * Set max concurrency to 1 because it's way faster * Fix undefined archList --------- Co-authored-by: Robert - Localstack <[email protected]>
localstack-samples · Jan 8, 2024 · e6aaaac · e6aaaac
1 parent 11cd419
commit e6aaaac
Show file tree

Hide file tree

Showing 28 changed files with 447 additions and 52 deletions.
diff --git a/.dockerignore b/.dockerignore
@@ -0,0 +1,2 @@
+.venv/
+.git/
diff --git a/.github/workflows/weekly_test.yml b/.github/workflows/weekly_test.yml
@@ -0,0 +1,43 @@
+name: Weekly CDK and Terraform Test
+
+on:
+  schedule:
+    - cron: '0 9 * * 1'  # Run every Monday at 9:00 UTC
+  push:
+    branches:
+      - main
+  pull_request:
+    branches:
+      - '*'
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    timeout-minutes: 60
+    # Only run one test at a time because it seems to be much faster.
+    # Possible reason: lots of memory swapping.
+    concurrency: 1
+    strategy:
+      matrix:
+        test_name: [awscdk, awscdktf]
+    steps:
+    # Checkout code
+    - name: Checkout code
+      uses: actions/checkout@v2
+
+    # Install prerequisites
+    - name: Set up Docker Buildx
+      uses: docker/setup-buildx-action@v1
+
+    # Start Localstack
+    - name: Test
+      env:
+        LOCALSTACK_AUTH_TOKEN: ${{ secrets.LOCALSTACK_AUTH_TOKEN }}
+        CI_TEST_NAME: ${{ matrix.test_name }}
+      run: |
+        # Unset stale org secret
+        printenv
+        unset LOCALSTACK_API_KEY
+
+        echo "Starting Localstack and running tests"
+        make run-ci-test
diff --git a/README.md b/README.md
@@ -42,24 +42,38 @@ To deploy your infrastructure, follow the steps below.
 
 ### Prerequisites
 
-1. [Install LATEST AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html)
-2. [Install Terraform](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli)
-3. [Install JQ](https://jqlang.github.io/jq/download/)
-4. Install Node Version Manager (NVM)
-   https://github.com/nvm-sh/nvm#installing-and-updating
-5. Select Node version 18
+
+1. [Install unzip](https://www.tecmint.com/install-zip-and-unzip-in-linux/)
+
+2. [Install LATEST AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html)
+
+3. [Install Terraform](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli)
+
+4. [Install JQ](https://jqlang.github.io/jq/download/)
+
+5. [Install Docker](https://docs.docker.com/engine/install/)
+
+6. [Install Node Version Manager (NVM)](https://github.com/nvm-sh/nvm#installing-and-updating)
+
+7. Select Node version 18
 
 ```shell
 nvm install 18
 ```
 
-6. Install Terraform CDK
+8. Install Terraform CDK
    Install cdktf in the node 18 version you just installed in step (4).
 
 ```shell
-npm install --global cdktf-cli@latest
+npm install --global cdktf-cli@^0.18.0
 ```
 
+9. Install `make`, `gcc`, `g++`, etc. For MacOS, run `brew install make gcc openssl readline sqlite3 xz` and for Ubuntu machines run `apt install build-essential libbz2-dev libssl-dev libreadline-dev libffi-dev zlib1g-dev libsqlite3-dev liblzma-dev`.
+
+10. Install `zlib1g-dev`. For MacOS, run `xcode-select --install` and for Ubuntu machines run `apt install zlib1g-dev`.
+
+11. [Install Pyenv](https://github.com/pyenv/pyenv#installation). Make sure the [prerequisites](https://github.com/pyenv/pyenv/wiki/Common-build-problems#prerequisites) are also there.
+
 ### Steps
 
 From the working directory:
@@ -68,18 +82,18 @@ From the working directory:
    your actual LocalStack key.
 
       ```bash
-      export LOCALSTACK_API_KEY=<your key>
+      export LOCALSTACK_AUTH_TOKEN=<your key>
       ```
 
-2. Start LocalStack
+2. Start LocalStack.
 
       ```bash
       make start-localstack
       ```
 
 ![Start LocalStack](./docs/img/start-localstack.png "Start LocalStack")
 
-3. Setup an AWS_PROFILE for LocalStack
+3. Setup an AWS_PROFILE for LocalStack.
 
 #### Add this to your `~/.aws/config` file
 
@@ -98,11 +112,21 @@ aws_access_key_id=test
 aws_secret_access_key=test
 ```
 
+4. Setup the virtual Python environment.
+
+```sh
+pyenv install 3.11 && \
+   pyenv local 3.11 && \
+   python -m venv .venv && \
+   . .venv/bin/activate && \
+   pip install -r devops-tooling/requirements.txt
+```
+
 # IaC Pipelines
 
 ## Terraform CDK Instructions
 
-[Solution Guide for Terrform CDK](./docs/README-cdktf.md "Solution Guide for TerraformCDK")
+[Solution Guide for Terraform CDK](./docs/README-cdktf.md "Solution Guide for TerraformCDK")
 
 ## AWS CDK
 
@@ -112,6 +136,25 @@ aws_secret_access_key=test
 
 [Solution Guide for Pulumi](./docs/README-pulumi.md "Solution Guide for Pulumi")
 
+# Tests
+
+First export the following env vars:
+
+```bash
+export LOCALSTACK_AUTH_TOKEN=<auth-token>
+export DOCKER_COMPOSE_FLAGS="--build"
+export BUILDKIT_PROGRESS=plain
+```
+
+And then run the AWS CDK and the Terraform-based AWS CDK deployments:
+
+```
+export CI_TEST_NAME=awscdk make run-ci-test
+export CI_TEST_NAME=awscdktf make run-ci-test
+```
+
+***Note: If you run the above tests with Rosetta turned on, and still want to go with `arm64`, you need to export `export OVERRIDE_LOCAL_ARCH=arm64`.***
+
 # Hot Reloading!
 
 The Lambda is setup for hot reloading in this project on LocalStack by default. After everything is deployed by
@@ -124,4 +167,4 @@ Run watch to do a build whenever code changes in the Lambda.
 
 ```shell
 make watch-lambda
-```
+```
diff --git a/devops-tooling/awscdk.makefile b/devops-tooling/awscdk.makefile
@@ -4,9 +4,9 @@ awscdkinstall:
 awscdkbootstrap: iac-shared awscdkinstall build
 	cd $(STACK_DIR) && $(CDK_CMD) bootstrap
 awscdkdeploy: iac-shared
-	cd $(STACK_DIR) && $(CDK_CMD) deploy $(TFSTACK_NAME)
+	cd $(STACK_DIR) && $(CDK_CMD) deploy $(TFSTACK_NAME) --require-approval=never
 awscdkdestroy: iac-shared
-	cd $(STACK_DIR) && $(CDK_CMD) destroy $(TFSTACK_NAME)
+	cd $(STACK_DIR) && $(CDK_CMD) destroy $(TFSTACK_NAME) --require-approval=never
 awscdkoutput:
 	@aws cloudformation describe-stacks \
   --stack-name $(TFSTACK_NAME) \
@@ -27,15 +27,15 @@ local-awscdk-destroy: awscdkdestroy
 local-awscdk-output: awscdkoutput
 
 local-awscdk-test:
-	make local-awscdk-output > auto_tests/iac-output.json;
+	@$(MAKE)  --silent local-awscdk-output > auto_tests/iac-output.json;
 	make test
 
 local-awscdk-invoke:
-	@APIGW=$$(make local-awscdk-output | jq -r '.apigwUrl') && \
+	@APIGW=$$($(MAKE) --silent local-awscdk-output | jq -r '.apigwUrl') && \
 	curl "http://$${APIGW}";
 
 local-awscdk-invoke-loop:
-	@APIGW=$$(make local-awscdk-output | jq -r '.apigwUrl') && \
+	@APIGW=$$($(MAKE) --silent local-awscdk-output | jq -r '.apigwUrl') && \
 	sh run-lambdas.sh "http://$${APIGW}"
 
 local-awscdk-clean:

diff --git a/devops-tooling/cdktf.makefile b/devops-tooling/cdktf.makefile
@@ -1,9 +1,9 @@
 # Add --ignore-missing-stack-dependencies. Waiting for issue below to be fixed.
 # https://github.com/hashicorp/terraform-cdk/issues/2976
 cdktfdeploy: iac-shared
-	cd $(STACK_DIR) && cdktf deploy $(TFSTACK_NAME) --ignore-missing-stack-dependencies
+	cd $(STACK_DIR) && cdktf deploy $(TFSTACK_NAME) --ignore-missing-stack-dependencies --auto-approve
 cdktfdestroy: iac-shared
-	cd $(STACK_DIR) && cdktf destroy $(TFSTACK_NAME) --ignore-missing-stack-dependencies
+	cd $(STACK_DIR) && cdktf destroy $(TFSTACK_NAME) --ignore-missing-stack-dependencies --auto-approve
 cdktfinstall:
 	cd $(STACK_DIR) && npm install
 	cd $(STACK_DIR) && cdktf get

diff --git a/devops-tooling/ci/Dockerfile b/devops-tooling/ci/Dockerfile
@@ -0,0 +1,20 @@
+FROM ubuntu:latest
+
+ENV BASH_ENV=~/.profile
+
+COPY devops-tooling/ci/setup.sh /tmp/setup.sh
+RUN chmod +x /tmp/setup.sh && /tmp/setup.sh
+
+COPY devops-tooling/ci/awscdk.sh /usr/local/bin/awscdk
+RUN chmod +x /usr/local/bin/awscdk
+
+COPY devops-tooling/ci/awscdktf.sh /usr/local/bin/awscdktf
+RUN chmod +x /usr/local/bin/awscdktf
+
+COPY devops-tooling/ci/bootstrap.sh /usr/local/bin/bootstrap
+RUN chmod +x /usr/local/bin/bootstrap
+
+COPY devops-tooling/requirements.txt /tmp/requirements.txt
+
+ENTRYPOINT ["/bin/bash"]
+CMD ["bootstrap"]
diff --git a/devops-tooling/ci/awscdk.sh b/devops-tooling/ci/awscdk.sh
@@ -0,0 +1,31 @@
+#!/bin/bash
+
+# Exit immediately if a command exits with a non-zero status
+set -e
+
+# Create AWS config/credentials
+make setup-aws
+
+export AWS_PROFILE=localstack
+export AWS_CONFIG_FILE=/root/.aws/config
+export AWS_SHARED_CREDENTIALS_FILE=/root/.aws/credentials
+
+# The endpoint is not getting picked up from the profile in the config file.
+export AWS_ENDPOINT_URL="http://${ENDPOINT_HOST}:4566"
+
+# Setup AWS CDK stacks
+make local-awscdk-bootstrap
+make local-awscdk-deploy
+
+# Test AWS CDK stacks
+make local-awscdk-test
+make local-awscdk-invoke
+
+# Cleanup
+make local-awscdk-destroy
+make local-awscdk-clean
+
+curl -X POST \
+     -H "Content-Type: application/json" \
+     -d '{"action": "kill"}' \
+     http://${ENDPOINT_HOST}:4566/_localstack/health
diff --git a/devops-tooling/ci/awscdktf.sh b/devops-tooling/ci/awscdktf.sh
@@ -0,0 +1,33 @@
+#!/bin/bash
+
+# Exit immediately if a command exits with a non-zero status
+set -e
+
+# Create AWS config/credentials
+make setup-aws
+
+export AWS_PROFILE=localstack
+export AWS_CONFIG_FILE=/root/.aws/config
+export AWS_SHARED_CREDENTIALS_FILE=/root/.aws/credentials
+
+# The endpoint is not getting picked up from the profile in the config file.
+export AWS_ENDPOINT_URL="http://${ENDPOINT_HOST}:4566"
+
+# Setup Terraform stacks
+make local-cdktf-install
+make local-cdktf-clean # Prevents flakiness.
+make local-cdktf-vpc-deploy
+make local-cdktf-deploy
+
+# Test Terraform stacks
+make local-cdktf-test
+make local-cdktf-invoke
+
+# Cleanup
+make local-cdktf-destroy
+make local-cdktf-clean
+
+curl -X POST \
+     -H "Content-Type: application/json" \
+     -d '{"action": "kill"}' \
+     http://${ENDPOINT_HOST}:4566/_localstack/health
diff --git a/devops-tooling/ci/bootstrap.sh b/devops-tooling/ci/bootstrap.sh
@@ -0,0 +1,25 @@
+#!/bin/bash
+
+# Exit immediately if a command exits with a non-zero status
+set -e
+
+# Check if MAPPING_DIR_NAME and CI_TEST_NAME are set
+if [ -z "${MAPPING_DIR_NAME}" ] || [ -z "${CI_TEST_NAME}" ]; then
+  echo "MAPPING_DIR_NAME and CI_TEST_NAME must be set"
+  exit 1
+fi
+
+# cd to mount directory
+cd ${MAPPING_DIR_NAME}
+
+# Initialize Python env
+eval "$(pyenv init --path)"
+pyenv global 3.11
+
+# Create Python virtual environment and install dependencies
+python3 -m venv .venv
+source .venv/bin/activate
+pip3 install --no-cache-dir -r /tmp/requirements.txt
+
+# Replace current shell with the CI test script 
+exec ${CI_TEST_NAME}
diff --git a/devops-tooling/ci/setup.sh b/devops-tooling/ci/setup.sh
@@ -0,0 +1,67 @@
+#!/bin/bash
+
+# Exit immediately if a command exits with a non-zero status
+set -e
+
+# Update system packages
+apt-get update && apt-get install -y \
+  curl \
+  unzip \
+  lsb-release \
+  software-properties-common \
+  git \
+  build-essential \
+  libbz2-dev \
+  libssl-dev \
+  libreadline-dev \
+  libffi-dev \
+  zlib1g-dev \
+  libsqlite3-dev \
+  liblzma-dev \
+  gnupg \
+  gnupg1 \
+  gnupg2 \
+  jq
+
+# Setup AWS CLI
+arch=$(uname -m)
+curl "https://d1vvhvl2y92vvt.cloudfront.net/awscli-exe-linux-$arch.zip" -o "awscliv2.zip"
+unzip awscliv2.zip
+./aws/install
+
+
+# Setup Terraform
+if [ "$arch" = "aarch64" ]; then
+    arch="arm64"
+elif [ "$arch" = "x86_64" ]; then
+    arch="amd64"
+fi
+curl -fsSL https://apt.releases.hashicorp.com/gpg | apt-key add -
+apt-add-repository "deb [arch=$arch] https://apt.releases.hashicorp.com $(lsb_release -cs) main" -y
+apt-get update && apt-get install terraform -y
+
+# Install Docker client
+curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
+add-apt-repository "deb [arch=$arch] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" -y
+apt-get update
+apt-get install -y docker-ce
+
+# Setup NVM and Node.js
+curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.38.0/install.sh | bash
+echo 'export NVM_DIR=$HOME/.nvm' >> ~/.profile
+echo '[ -s "$NVM_DIR/nvm.sh" ] && \. $NVM_DIR/nvm.sh' >> ~/.profile
+source ~/.profile
+nvm install 18
+nvm use 18
+
+# Setup Pyenv and Python
+curl https://pyenv.run | bash
+echo 'export PYENV_ROOT=$HOME/.pyenv' >> ~/.profile
+echo 'export PATH=$PYENV_ROOT/bin:$PATH' >> ~/.profile
+source ~/.profile
+eval "$(pyenv init --path)"
+pyenv install 3.11 -vvv
+pyenv global 3.11
+
+# Install Terraform CDK
+npm install --global cdktf-cli@^0.18.0 aws-cdk-local aws-cdk