-
Notifications
You must be signed in to change notification settings - Fork 25
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add self-hosted runner, break E2E tests into labeled Specs
* Break provider tests into seperate files with labels representing either onprem or cloud providers. * Add new jobs to CI workflow which dictate where tests will run, onprem provider tests like vSphere will run on self-hosted runners since they will use internal resources to test. Cloud provider tests will use the existing workflow since they can access providers without network access and can take advantage of the much larger GitHub hosted pool. Hosted/Self-hosted tests can run concurrently. * Make Cleanup depend on the cloud-e2etest only. * Use new GINKGO_LABEL_FILTER to manage what tests run where. * Move controller validatation into BeforeSuite since the controller needs to be up and ready for each provider test, this will also enable us to add controller specific test cases later and make those run without the "test e2e" flag. * Seperate self-hosted and hosted test concurrency groups * Update docs with test filtering instructions * Ensure a Release exists for the custom build.version we deploy * Move all e2e related helpers into e2e dir * Add new clusteridentity package for creating ClusterIdentity kind's and associated Secret's. * Merge PR workflows together * Make sure VERSION gets passed across jobs * Ensure uniqueness among deployed ManagedClusters, simplify MANAGED_CLUSTER_NAME in CI to prevent Azure availabilitySetName validation error. * Default Azure test templates to uswest2 to prevent issues with AvailabilityZone. * Use the same concurrency-group across all jobs, except Cleanup which intentionally does not belong to a concurrency-group. * Use Setup Go across jobs for caching. * Support patching other hosted clusters to status.ready with a common patching helper. * Move VSphere delete into AfterEach to serve as cleanup. * Add support for cleaning Azure resources. * Prevent ginkgo from timing out tests. * Use azure-disk CSI driver we deploy via templates. Signed-off-by: Kyle Squizzato <[email protected]>
- Loading branch information
Showing
37 changed files
with
1,632 additions
and
1,227 deletions.
There are no files selected for viewing
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,241 @@ | ||
name: CI | ||
on: | ||
pull_request: | ||
types: | ||
- labeled | ||
- opened | ||
- synchronize | ||
- reopened | ||
branches: | ||
- main | ||
- release-* | ||
paths-ignore: | ||
- 'config/**' | ||
- '**.md' | ||
env: | ||
GO_VERSION: '1.22' | ||
REGISTRY_REPO: 'oci://ghcr.io/mirantis/hmc/charts-ci' | ||
|
||
jobs: | ||
build: | ||
concurrency: | ||
group: build-${{ github.head_ref || github.run_id }} | ||
cancel-in-progress: true | ||
name: Build and Unit Test | ||
runs-on: ubuntu-latest | ||
outputs: | ||
version: ${{ steps.vars.outputs.version }} | ||
clustername: ${{ steps.vars.outputs.clustername }} | ||
steps: | ||
- name: Checkout repository | ||
uses: actions/checkout@v4 | ||
with: | ||
fetch-depth: 0 | ||
- name: Setup Go | ||
uses: actions/setup-go@v5 | ||
with: | ||
go-version: ${{ env.GO_VERSION }} | ||
cache: false | ||
- name: Lint | ||
uses: golangci/golangci-lint-action@v6 | ||
with: | ||
args: --timeout 10m0s | ||
- name: Verify all generated pieces are up-to-date | ||
run: make generate-all && git add -N . && git diff --exit-code | ||
- name: Unit tests | ||
run: | | ||
make test | ||
- name: Set up Buildx | ||
uses: docker/setup-buildx-action@v3 | ||
- name: Login to GHCR | ||
uses: docker/[email protected] | ||
with: | ||
registry: ghcr.io | ||
username: ${{ github.actor }} | ||
password: ${{ secrets.GITHUB_TOKEN }} | ||
- name: Get outputs | ||
id: vars | ||
run: | | ||
echo "version=$(git describe --tags --always)" >> $GITHUB_OUTPUT | ||
echo "clustername=ci-$(date +%s | cut -b6-10)" >> $GITHUB_OUTPUT | ||
- name: Build and push HMC controller image | ||
uses: docker/build-push-action@v6 | ||
with: | ||
build-args: | | ||
LD_FLAGS=-s -w -X github.com/Mirantis/hmc/internal/build.Version=${{ steps.vars.outputs.version }} | ||
context: . | ||
platforms: linux/amd64 | ||
tags: | | ||
ghcr.io/mirantis/hmc/controller-ci:${{ steps.vars.outputs.version }} | ||
push: true | ||
cache-from: type=gha | ||
cache-to: type=gha,mode=max | ||
- name: Prepare and push HMC template charts | ||
run: | | ||
make hmc-chart-release | ||
make helm-push | ||
controller-e2etest: | ||
name: E2E Controller | ||
if: ${{ contains( github.event.pull_request.labels.*.name, 'skip for now') }} | ||
runs-on: ubuntu-latest | ||
needs: build | ||
concurrency: | ||
group: controller-${{ github.head_ref || github.run_id }} | ||
cancel-in-progress: true | ||
outputs: | ||
clustername: ${{ needs.build.outputs.clustername }} | ||
version: ${{ needs.build.outputs.version }} | ||
steps: | ||
- name: Checkout repository | ||
uses: actions/checkout@v4 | ||
with: | ||
fetch-depth: 0 | ||
- name: Setup kubectl | ||
uses: azure/setup-kubectl@v4 | ||
- name: Run E2E tests | ||
env: | ||
GINKGO_LABEL_FILTER: 'controller' | ||
MANAGED_CLUSTER_NAME: ${{ needs.build.outputs.clustername }} | ||
IMG: 'ghcr.io/mirantis/hmc/controller-ci:${{ needs.build.outputs.version }}' | ||
VERSION: ${{ needs.build.outputs.version }} | ||
run: | | ||
make test-e2e | ||
- name: Archive test results | ||
if: ${{ failure() }} | ||
uses: actions/upload-artifact@v4 | ||
with: | ||
name: cloud-e2etest-logs | ||
path: | | ||
test/e2e/*.log | ||
provider-cloud-e2etest: | ||
name: E2E Cloud Providers | ||
runs-on: ubuntu-latest | ||
if: ${{ contains( github.event.pull_request.labels.*.name, 'test e2e') }} | ||
needs: build | ||
concurrency: | ||
group: cloud-${{ github.head_ref || github.run_id }} | ||
cancel-in-progress: true | ||
outputs: | ||
clustername: ${{ needs.build.outputs.clustername }} | ||
version: ${{ needs.build.outputs.version }} | ||
env: | ||
AWS_REGION: us-west-2 | ||
AWS_ACCESS_KEY_ID: ${{ secrets.CI_AWS_ACCESS_KEY_ID }} | ||
AWS_SECRET_ACCESS_KEY: ${{ secrets.CI_AWS_SECRET_ACCESS_KEY }} | ||
AZURE_REGION: westus2 | ||
AZURE_SUBSCRIPTION_ID: ${{ secrets.CI_AZURE_SUBSCRIPTION_ID }} | ||
AZURE_TENANT_ID: ${{ secrets.CI_AZURE_TENANT_ID }} | ||
AZURE_CLIENT_ID: ${{ secrets.CI_AZURE_CLIENT_ID }} | ||
AZURE_CLIENT_SECRET: ${{ secrets.CI_AZURE_CLIENT_SECRET }} | ||
steps: | ||
- name: Checkout repository | ||
uses: actions/checkout@v4 | ||
with: | ||
fetch-depth: 0 | ||
- name: Setup Go | ||
uses: actions/setup-go@v5 | ||
with: | ||
go-version: ${{ env.GO_VERSION }} | ||
- name: Setup kubectl | ||
uses: azure/setup-kubectl@v4 | ||
- uses: actions/checkout@v4 | ||
- name: Run E2E tests | ||
env: | ||
GINKGO_LABEL_FILTER: 'provider:cloud' | ||
MANAGED_CLUSTER_NAME: ${{ needs.build.outputs.clustername }} | ||
IMG: 'ghcr.io/mirantis/hmc/controller-ci:${{ needs.build.outputs.version }}' | ||
VERSION: ${{ needs.build.outputs.version }} | ||
run: | | ||
make test-e2e | ||
- name: Archive test results | ||
if: ${{ failure() }} | ||
uses: actions/upload-artifact@v4 | ||
with: | ||
name: cloud-e2etest-logs | ||
path: | | ||
test/e2e/*.log | ||
provider-onprem-e2etest: | ||
name: E2E On-Prem Providers | ||
runs-on: self-hosted | ||
if: ${{ contains( github.event.pull_request.labels.*.name, 'test e2e') }} | ||
needs: build | ||
concurrency: | ||
group: onprem-${{ github.head_ref || github.run_id }} | ||
cancel-in-progress: true | ||
outputs: | ||
clustername: ${{ needs.build.outputs.clustername }} | ||
version: ${{ needs.build.outputs.version }} | ||
env: | ||
VSPHERE_USER: ${{ secrets.CI_VSPHERE_USER }} | ||
VSPHERE_PASSWORD: ${{ secrets.CI_VSPHERE_PASSWORD }} | ||
VSPHERE_SERVER: ${{ secrets.CI_VSPHERE_SERVER }} | ||
VSPHERE_THUMBPRINT: ${{ secrets.CI_VSPHERE_THUMBPRINT }} | ||
VSPHERE_DATACENTER: ${{ secrets.CI_VSPHERE_DATACENTER }} | ||
VSPHERE_DATASTORE: ${{ secrets.CI_VSPHERE_DATASTORE }} | ||
VSPHERE_RESOURCEPOOL: ${{ secrets.CI_VSPHERE_RESOURCEPOOL }} | ||
VSPHERE_FOLDER: ${{ secrets.CI_VSPHERE_FOLDER }} | ||
VSPHERE_CONTROL_PLANE_ENDPOINT: ${{ secrets.CI_VSPHERE_CONTROL_PLANE_ENDPOINT }} | ||
VSPHERE_VM_TEMPLATE: ${{ secrets.CI_VSPHERE_VM_TEMPLATE }} | ||
VSPHERE_NETWORK: ${{ secrets.CI_VSPHERE_NETWORK }} | ||
VSPHERE_SSH_KEY: ${{ secrets.CI_VSPHERE_SSH_KEY }} | ||
steps: | ||
- name: Checkout repository | ||
uses: actions/checkout@v4 | ||
with: | ||
fetch-depth: 0 | ||
- name: Setup Go | ||
uses: actions/setup-go@v5 | ||
with: | ||
go-version: ${{ env.GO_VERSION }} | ||
- name: Setup kubectl | ||
uses: azure/setup-kubectl@v4 | ||
- name: Run E2E tests | ||
env: | ||
GINKGO_LABEL_FILTER: 'provider:onprem' | ||
MANAGED_CLUSTER_NAME: ${{ needs.build.outputs.clustername }} | ||
IMG: 'ghcr.io/mirantis/hmc/controller-ci:${{ needs.build.outputs.version }}' | ||
VERSION: ${{ needs.build.outputs.version }} | ||
run: | | ||
make test-e2e | ||
- name: Archive test results | ||
if: ${{ failure() }} | ||
uses: actions/upload-artifact@v4 | ||
with: | ||
name: onprem-e2etest-logs | ||
path: | | ||
test/e2e/*.log | ||
cleanup: | ||
name: Cleanup | ||
needs: | ||
- build | ||
- provider-cloud-e2etest | ||
runs-on: ubuntu-latest | ||
if: ${{ always() && !contains(needs.provider-cloud-e2etest.result, 'skipped') && contains(needs.build.result, 'success') }} | ||
timeout-minutes: 15 | ||
steps: | ||
- name: Checkout repository | ||
uses: actions/checkout@v4 | ||
with: | ||
fetch-depth: 0 | ||
- name: Setup Go | ||
uses: actions/setup-go@v5 | ||
with: | ||
go-version: ${{ env.GO_VERSION }} | ||
- name: AWS Test Resources | ||
env: | ||
AWS_REGION: us-west-2 | ||
AWS_ACCESS_KEY_ID: ${{ secrets.CI_AWS_ACCESS_KEY_ID }} | ||
AWS_SECRET_ACCESS_KEY: ${{ secrets.CI_AWS_SECRET_ACCESS_KEY }} | ||
AZURE_REGION: westus2 | ||
AZURE_SUBSCRIPTION_ID: ${{ secrets.CI_AZURE_SUBSCRIPTION_ID }} | ||
AZURE_TENANT_ID: ${{ secrets.CI_AZURE_TENANT_ID }} | ||
AZURE_CLIENT_ID: ${{ secrets.CI_AZURE_CLIENT_ID }} | ||
AZURE_CLIENT_SECRET: ${{ secrets.CI_AZURE_CLIENT_SECRET }} | ||
CLUSTER_NAME: '${{ needs.build.outputs.clustername }}' | ||
run: | | ||
make dev-aws-nuke | ||
make dev-azure-nuke |
Oops, something went wrong.