Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kubelet config e2e tests for vsphere Bottlerocket #8386

Closed
wants to merge 32 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
1c4e43a
Override the nginx image properly in tinkerbell stack chart (#8319)
eks-distro-pr-bot Jun 13, 2024
d0b1628
Update branch name to release-0.20 in Makefile (#8320)
rahulbabu95 Jun 13, 2024
434226d
Fix e2e tests creating old bundle workload clusters (#8330)
eks-distro-pr-bot Jun 14, 2024
d1b4222
Updated prometheus pod labels for e2e test (#8332)
eks-distro-pr-bot Jun 17, 2024
38737d8
Ensure Chart.yaml is modified for Tinkerbell stack and CRDs Helm char…
eks-distro-pr-bot Jun 17, 2024
44868d0
Fix issue where incorrect flag is getting passed to helm commands (#8…
eks-distro-pr-bot Jun 18, 2024
0814535
ensure release branch is included in teest cluster name (#8342)
eks-distro-pr-bot Jun 18, 2024
42482d6
Log ssm command details when an e2e test finishes (#8352)
eks-distro-pr-bot Jun 19, 2024
11fcfb5
Set registry username and password as expected with new tink stack (#…
eks-distro-pr-bot Jun 20, 2024
14d3812
Support installing previous version of EKS-A CLI using Homebrew (#8360)
eks-distro-pr-bot Jun 21, 2024
0215fb3
Kubelet config e2e tests for vsphere and Docker (#8363)
eks-distro-pr-bot Jun 22, 2024
c014c30
trigger staging bundles for v0.20.0 (#8358)
rahulbabu95 Jun 23, 2024
a15067d
Handle nil commandOut (#8367)
eks-distro-pr-bot Jun 24, 2024
4ad9666
Change delete timeout to 30 mins (#8368)
eks-distro-pr-bot Jun 24, 2024
d2be185
Add descriptive error logs for download, rename and upload operations…
eks-distro-pr-bot Jun 24, 2024
3d6c158
Create S3 downloader using source session instead of release session …
eks-distro-pr-bot Jun 24, 2024
ed17522
Trigger staging CLI for release 0.20 (#8376)
rahulbabu95 Jun 24, 2024
49f9985
[release-0.20] Kubelet config e2e tests for Nutanix, Cloudstack and T…
eks-distro-pr-bot Jun 24, 2024
a9428e9
Revert "ensure release branch is included in teest cluster name (#833…
eks-distro-pr-bot Jun 24, 2024
305485f
lower concurrent vsphere e2e run count (#8382)
eks-distro-pr-bot Jun 25, 2024
43a4462
Upload RTOS artifacts to separate bucket (#8384)
eks-distro-pr-bot Jun 25, 2024
7573aac
Bump braces from 3.0.2 to 3.0.3 in /docs (#8297)
dependabot[bot] Jun 13, 2024
833270c
Add Changelog for v0.19.7 (#8322)
sp1999 Jun 14, 2024
26d8b0e
[PR BOT] Generate release testdata files (#8328)
eks-distro-pr-bot Jun 14, 2024
d73edc7
Update brew to v0.19.7 (#8327)
sp1999 Jun 14, 2024
3907dab
Bump codecov/codecov-action from 4.4.1 to 4.5.0 (#8334)
dependabot[bot] Jun 17, 2024
cb5b0aa
[PR BOT] Generate release testdata files (#8340)
eks-distro-pr-bot Jun 18, 2024
62bfa72
Skip multi-protocol Service test in conformance flow (#8349)
abhay-krishna Jun 18, 2024
2bcca0f
Fetch latest E2E test AMI ID dynamically (#8238)
abhay-krishna Jun 20, 2024
654ea0b
Update the HookOS customization doc: (#8181)
jacobweinstock Jun 20, 2024
64180bc
Add comments about why cloudstack tests are skipped (#8364)
ahreehong Jun 24, 2024
6e93d69
Kubelet config e2e tests for vsphere Bottlerocket
mitalipaygude Jun 25, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/go-coverage.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ jobs:
- name: Run go test with coverage
run: COVER_PROFILE=coverage.txt make coverage-unit-test
- name: Codecov upload
uses: codecov/codecov-action@v4.4.1
uses: codecov/codecov-action@v4.5.0
with:
files: ./coverage.txt
token: ${{ secrets.CODECOV_TOKEN }}
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ UNIT_TEST_PACKAGE_EXCLUSION_REGEX ?=mocks$
UNIT_TEST_PACKAGES ?= $$($(GO) list ./... | grep -vE "$(UNIT_TEST_PACKAGE_EXCLUSION_REGEX)")

## ensure local execution uses the 'main' branch bundle
BRANCH_NAME?=main
BRANCH_NAME?=release-0.20
# DEV_GIT_VERSION should be something like v0.19.0-dev+latest, depending on the base branch
# https://github.com/aws/eks-anywhere/blob/main/docs/developer/manifests.md#locally-building-the-cli
ifneq ($(PULL_BASE_REF),) # PULL_BASE_REF originates from prow
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ version: 0.2
env:
variables:
INTEGRATION_TEST_MAX_EC2_COUNT: 180
INTEGRATION_TEST_MAX_CONCURRENT_TEST_COUNT: 100
INTEGRATION_TEST_MAX_CONCURRENT_TEST_COUNT: 80
EKSA_GIT_KNOWN_HOSTS: "/tmp/known_hosts"
EKSA_GIT_PRIVATE_KEY: "/tmp/private-key"
INTEGRATION_TEST_INFRA_CONFIG: "/tmp/test-infra.yml"
Expand Down
18 changes: 17 additions & 1 deletion cmd/integration_test/build/script/create_infra_config.sh
Original file line number Diff line number Diff line change
Expand Up @@ -17,11 +17,27 @@ set -e
set -x
set -o pipefail

REPO_ROOT=$(git rev-parse --show-toplevel)
source $REPO_ROOT/test/e2e/E2E_AMI_FILTER_VARS

INTEGRATION_TEST_AMI_ID=$(aws ec2 describe-images \
--profile ${AWS_PROFILE} \
--owners ${AMI_OWNER_ID_FILTER} \
--filters "Name=name,Values=${AMI_NAME_FILTER}" "Name=description,Values=${AMI_DESCRIPTION_FILTER}" \
--query 'sort_by(Images, &CreationDate)[-1].[ImageId]' \
--output text
)

if [ -z "$INTEGRATION_TEST_AMI_ID" ]; then
echo "INTEGRATION_TEST_AMI_ID cannot be empty. Exiting"
exit 1
fi

cat << EOF > ${INTEGRATION_TEST_INFRA_CONFIG}
---

ec2:
amiId: ${INTEGRATION_TEST_AL2_AMI_ID}
amiId: ${INTEGRATION_TEST_AMI_ID}
subnetId: ${INTEGRATION_TEST_SUBNET_ID}

vSphere:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,191 +8,62 @@ description: >
Customizing HookOS for EKS Anywhere on Bare Metal
---

To initially network boot bare metal machines used in EKS Anywhere clusters, Tinkerbell acquires a kernel and initial ramdisk that is referred to as the HookOS.
To network boot bare metal machines in EKS Anywhere clusters, machines acquire a kernel and initial ramdisk that is referred to as HookOS.
A default HookOS is provided when you create an EKS Anywhere cluster.
However, there may be cases where you want to override the default HookOS, such as to add drivers required to boot your particular type of hardware.
However, there may be cases where you want and/or need to customize the default HookOS, such as to add drivers required to boot your particular type of hardware.

The following procedure describes how to get the Tinkerbell stack’s Hook/Linuxkit OS built locally.
For more information on Tinkerbell’s Hook Installation Environment, see the [Tinkerbell Hook repo](https://github.com/tinkerbell/hook).
The following procedure describes how to customize and build HookOS.
For more information on Tinkerbell’s HookOS Installation Environment, see the [Tinkerbell Hook repo](https://github.com/tinkerbell/hook).

1. Clone the hook repo or your fork of that repo:

```bash
git clone https://github.com/tinkerbell/hook.git
cd hook/
```

1. Pull down the commit that EKS Anywhere is tracking for Hook:

```bash
git checkout -b <new-branch> 03a67729d895635fe3c612e4feca3400b9336cc9
```

>**_NOTE_**: This commit number can be obtained from the [EKS-A build tooling repo](https://github.com/aws/eks-anywhere-build-tooling/blob/main/projects/tinkerbell/hook/GIT_TAG).
>

1. Make changes shown in the following `diff` in the `Makefile` located in the root of the repo using your favorite editor.

```bash
diff --git a/Makefile b/Makefile
index e7fd844..8e87c78 100644
--- a/Makefile
+++ b/Makefile
@@ -2,7 +2,7 @@
### !!NOTE!!
# If this is changed then a fresh output dir is required (`git clean -fxd` or just `rm -rf out`)
# Handling this better shows some of make's suckiness compared to newer build tools (redo, tup ...) where the command lines to tools invoked isn't tracked by make
-ORG := quay.io/tinkerbell
+ORG := localhost:5000/tinkerbell
# makes sure there's no trailing / so we can just add them in the recipes which looks nicer
ORG := $(shell echo "${ORG}" | sed 's|/*$$||')

```

Changes above change the ORG variable to use a local registry (`localhost:5000`)

1. Make changes shown in the following `diff` in the `rules.mk` located in the root of the repo using your favorite editor.

```bash
diff --git a/rules.mk b/rules.mk
index b2c5133..64e32b1 100644
--- a/rules.mk
+++ b/rules.mk
@@ -22,7 +22,7 @@ ifeq ($(ARCH),aarch64)
ARCH = arm64
endif

-arches := amd64 arm64
+arches := amd64
modes := rel dbg

hook-bootkit-deps := $(wildcard hook-bootkit/*)
@@ -87,13 +87,12 @@ push-hook-bootkit push-hook-docker:
docker buildx build --platform $$platforms --push -t $(ORG)/$(container):$T $(container)

.PHONY: dist
-dist: out/$T/rel/amd64/hook.tar out/$T/rel/arm64/hook.tar ## Build tarballs for distribution
+dist: out/$T/rel/amd64/hook.tar ## Build tarballs for distribution
dbg-dist: out/$T/dbg/$(ARCH)/hook.tar ## Build debug enabled tarball
dist dbg-dist:
for f in $^; do
case $$f in
*amd64*) arch=x86_64 ;;
- *arm64*) arch=aarch64 ;;
*) echo unknown arch && exit 1;;
esac
d=$$(dirname $$(dirname $$f))

```

Above changes are for the `docker build` command to only build for the immediately required platform (amd64 in this case) to save time.
## System requirements

- `>= 2G memory`
- `>= 4 CPU cores` # the more cores the better for kernel building.
- `>= 20G disk space`

1. Modify the `hook.yaml` file located in the root of the repo with the following changes:
## Dependencies

```bash
diff --git a/hook.yaml b/hook.yaml

index 0c5d789..b51b35e 100644

net: host
--- a/hook.yaml
+++ b/hook.yaml
@@ -1,5 +1,5 @@
kernel:
- image: quay.io/tinkerbell/hook-kernel:5.10.85 (http://quay.io/tinkerbell/hook-kernel:5.10.85)
+ image: localhost:5000/tinkerbell/hook-kernel:5.10.85
cmdline: "console=tty0 console=ttyS0 console=ttyAMA0 console=ttysclp0"
init:
- linuxkit/init:v0.8
@@ -42,7 +42,7 @@ services:
binds:
- /var/run:/var/run
- name: docker
- image: quay.io/tinkerbell/hook-docker:0.0 (http://quay.io/tinkerbell/hook-docker:0.0)
+ image: localhost:5000/tinkerbell/hook-docker:0.0
capabilities:
- all
net: host
@@ -64,7 +64,7 @@ services:
- /var/run/docker
- /var/run/worker
- name: bootkit
- image: quay.io/tinkerbell/hook-bootkit:0.0 (http://quay.io/tinkerbell/hook-bootkit:0.0)
+ image: localhost:5000/tinkerbell/hook-bootkit:0.0
capabilities:
- all
```

The changes above are for using local registry (localhost:5000) for hook-docker, hook-bootkit, and hook-kernel.
Be sure to install all the following dependencies.

>**_NOTE_**: You may also need to modify the `hook.yaml` file if you want to add or change components that are used to build up the image. So far, for example, we have needed to change versions of `init` and `getty` and inject SSH keys. Take a look at the [LinuxKit Examples](https://github.com/linuxkit/linuxkit/tree/master/examples) site for examples.
>
- `jq`
- `envsubst`
- `pigz`
- `docker`
- `curl`
- `bash` >= 4.4
- `git`
- `findutils`

1. Make any planned custom modifications to the files under `hook`, if you are only making changes to `bootkit` or `tink-docker`.


1. If you are modifying the kernel, such as to change kernel config parameters to add or modify drivers, follow these steps:

* Change into kernel directory and make a local image for amd64 architecture:
1. Clone the Hook repo or your fork of that repo:

```bash
cd kernel; make kconfig_amd64
```

* Run the image

```bash
docker run --rm -ti -v $(pwd):/src:z quay.io/tinkerbell/kconfig
```

* You can now navigate to the source code and run the UI for configuring the kernel:

```bash
cd linux-5-10
make menuconfig
```

* Once you have changed the necessary kernel configuration parameters, copy the new configuration:

```bash
cp .config /src/config-5.10.x-x86_64
git clone https://github.com/tinkerbell/hook.git
cd hook/
```

Exit out of container into the repo’s kernel directory and run make:
1. Run the Linux kernel [menuconfig](https://en.wikipedia.org/wiki/Menuconfig) TUI and configuring the kernel as needed. Be sure to save the config before exiting. The result of this step will be a modified kernel configuration file (`./kernel/configs/generic-6.6.y-x86_64`).

```bash
/linux-5.10.85 # exit
user1 % make
./build.sh kernel-config hook-latest-lts-amd64
```

1. Install Linuxkit based on instructions from the [LinuxKit](https://github.com/linuxkit/linuxkit) page.


1. Ensure that the `linuxkit` tool is in your PATH:
1. Build the kernel container image. The result of this step will be a container image. Use `docker images quay.io/tinkerbell/hook-kernel` to see it.

```bash
export PATH=$PATH:/home/tink/linuxkit/bin
./build.sh kernel hook-latest-lts-amd64
```

1. Start a local registry:
1. Build the HookOS kernel and initramfs artifacts. The result of this step will be the kernel and initramfs. These files are located at `./out/hook/vmlinuz-latest-lts-x86_64` and `./out/hook/initramfs-latest-lts-x86_64` respectively.

```bash
docker run -d -p 5000:5000 --name registry registry:2
./build.sh linuxkit hook-latest-lts-amd64
```

1. Compile by running the following in the root of the repo:

```bash
make dist
```
1. Artifacts will be put under the `dist` directory in the repo’s root:
1. Rename the kernel and initramfs files to `vmlinuz-x86_64` and `initramfs-x86_64` respectively.

```bash
./initramfs-aarch64
./initramfs-x86_64
./vmlinuz-aarch64
./vmlinuz-x86_64
mv ./out/hook/vmlinuz-latest-lts-x86_64 ./out/hook/vmlinuz-x86_64
mv ./out/hook/initramfs-latest-lts-x86_64 ./out/hook/initramfs-x86_64
```

1. To use the kernel (`vmlinuz`) and initial ram disk (`initramfs`) when you build your cluster, see the description of the `hookImagesURLPath` field in your Bare Metal configuration file.
1. To use the kernel (`vmlinuz-x86_64`) and initial ram disk (`initramfs-x86_64`) when you build your EKS Anywhere cluster, see the description of the [`hookImagesURLPath`]({{< relref "../bare-spec#hookimagesurlpath-optional" >}}) field in your cluster configuration file.
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,7 @@ To ensure the Prometheus package is installed correctly in the cluster, a user c

Port forward Prometheus to local host `9090`:
```bash
export PROM_SERVER_POD_NAME=$(kubectl get pods --namespace <namespace> -l "app=prometheus,component=server" -o jsonpath="{.items[0].metadata.name")
export PROM_SERVER_POD_NAME=$(kubectl get pods --namespace <namespace> -l "app.kubernetes.io/name=prometheus,app.kubernetes.io/component=server" -o jsonpath="{.items[0].metadata.name")
kubectl port-forward $PROM_SERVER_POD_NAME -n <namespace> 9090
```

Expand Down
30 changes: 30 additions & 0 deletions docs/content/en/docs/whatsnew/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,36 @@ description: >
* When upgrading to a new minor version, a new OS image must be created using the new image-builder CLI pertaining to that release.
{{% /alert %}}

## [v0.19.7](https://github.com/aws/eks-anywhere/releases/tag/v0.19.7)
### Supported OS version details
| | vSphere | Bare Metal | Nutanix | CloudStack | Snow |
|:-------------------:|:-------:|:----------:|:-------:|:----------:|:----:|
| Ubuntu 20.04 | ✔ | ✔ | ✔ | — | ✔ |
| Ubuntu 22.04 | ✔ | ✔ | ✔ | — | — |
| Bottlerocket 1.19.2 | ✔ | \* | — | — | — |
| RHEL 8.x | ✔ | ✔ | ✔ | ✔ | — |
| RHEL 9.x | — | — | ✔ | ✔ | — |

\* [EKS Anywhere issue regarding deprecation of Bottlerocket bare metal variants](https://github.com/aws/eks-anywhere/issues/7754)

### Upgraded
- Cluster API Provider Nutanix: `v1.3.3` to `v1.3.5`
- Image Builder: `v0.1.24` to `v0.1.26`
- EKS Distro:
- `v1-25-eks-39` to [`v1-25-eks-40`](https://distro.eks.amazonaws.com/releases/1-25/40/)
- `v1-26-eks-35` to [`v1-26-eks-38`](https://distro.eks.amazonaws.com/releases/1-26/38/)
- `v1-27-eks-29` to [`v1-27-eks-32`](https://distro.eks.amazonaws.com/releases/1-27/32/)
- `v1-28-eks-22` to [`v1-28-eks-25`](https://distro.eks.amazonaws.com/releases/1-28/25/)
- `v1-29-eks-11` to [`v1-29-eks-14`](https://distro.eks.amazonaws.com/releases/1-29/14/)

### Changed
- Updated cluster status reconciliation logic for worker node groups with [autoscaling](https://anywhere.eks.amazonaws.com/docs/getting-started/optional/autoscaling/) configuration [#8254](https://github.com/aws/eks-anywhere/pull/8254)
- Added logic to apply new hardware on baremetal cluster upgrades [#8288](https://github.com/aws/eks-anywhere/pull/8288)

### Fixed
- Fixed bug when installer does not create CCM secret for Nutanix workload cluster [#8191](https://github.com/aws/eks-anywhere/pull/8191)
- Fixed upgrade workflow for registry mirror certificates in EKS Anywhere packages [#7114](https://github.com/aws/eks-anywhere/issues/7114)

## [v0.19.6](https://github.com/aws/eks-anywhere/releases/tag/v0.19.6)
### Supported OS version details
| | vSphere | Bare Metal | Nutanix | CloudStack | Snow |
Expand Down
28 changes: 14 additions & 14 deletions docs/package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading