Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[e2e] Support to provide e2e testing configuration #752

Draft
wants to merge 3 commits into
base: main
Choose a base branch
from
Draft

[e2e] Support to provide e2e testing configuration #752

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
262b16f
[e2e] Support to provide e2e testing configuration
eromanova Nov 11, 2024
4b4c5b3
[e2e] Fix adopted cluster testing template
eromanova Jan 22, 2025
e889ca1
Address review comments
eromanova Jan 22, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 10 additions & 10 deletions .github/workflows/build_test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ jobs:
runs-on: ubuntu-latest
outputs:
version: ${{ steps.vars.outputs.version }}
clustername: ${{ steps.vars.outputs.clustername }}
clusterprefix: ${{ steps.vars.outputs.clusterprefix }}
pr: ${{ steps.pr.outputs.result }}
permissions:
packages: write
Expand Down Expand Up @@ -83,7 +83,7 @@ jobs:
run: |
GIT_VERSION=$(git describe --tags --always)
echo "version=${GIT_VERSION:1}" >> $GITHUB_OUTPUT
echo "clustername=ci-$(date +%s | cut -b6-10)" >> $GITHUB_OUTPUT
echo "clusterprefix=ci-$(date +%s | cut -b6-10)" >> $GITHUB_OUTPUT
- name: Build and push KCM controller image
uses: docker/build-push-action@v6
with:
Expand All @@ -109,7 +109,7 @@ jobs:
group: controller-${{ github.head_ref || github.run_id }}
cancel-in-progress: true
outputs:
clustername: ${{ needs.build.outputs.clustername }}
clusterprefix: ${{ needs.build.outputs.clusterprefix }}
version: ${{ needs.build.outputs.version }}
pr: ${{ needs.build.outputs.pr }}
steps:
Expand All @@ -123,7 +123,7 @@ jobs:
- name: Run E2E tests
env:
GINKGO_LABEL_FILTER: 'controller'
CLUSTER_DEPLOYMENT_NAME: ${{ needs.build.outputs.clustername }}
CLUSTER_DEPLOYMENT_PREFIX: ${{ needs.build.outputs.clusterprefix }}
IMG: 'ghcr.io/k0rdent/kcm/controller-ci:${{ needs.build.outputs.version }}'
VERSION: ${{ needs.build.outputs.version }}
run: |
Expand All @@ -147,7 +147,7 @@ jobs:
group: cloud-${{ github.head_ref || github.run_id }}
cancel-in-progress: true
outputs:
clustername: ${{ needs.build.outputs.clustername }}
clusterprefix: ${{ needs.build.outputs.clusterprefix }}
version: ${{ needs.build.outputs.version }}
pr: ${{ needs.build.outputs.pr }}
env:
Expand All @@ -174,7 +174,7 @@ jobs:
- name: Run E2E tests
env:
GINKGO_LABEL_FILTER: 'provider:cloud'
CLUSTER_DEPLOYMENT_NAME: ${{ needs.build.outputs.clustername }}
CLUSTER_DEPLOYMENT_PREFIX: ${{ needs.build.outputs.clusterprefix }}
IMG: 'ghcr.io/k0rdent/kcm/controller-ci:${{ needs.build.outputs.version }}'
VERSION: ${{ needs.build.outputs.version }}
run: |
Expand All @@ -198,7 +198,7 @@ jobs:
group: onprem-${{ github.head_ref || github.run_id }}
cancel-in-progress: true
outputs:
clustername: ${{ needs.build.outputs.clustername }}
clusterprefix: ${{ needs.build.outputs.clusterprefix }}
version: ${{ needs.build.outputs.version }}
pr: ${{ needs.build.outputs.pr }}
env:
Expand Down Expand Up @@ -229,7 +229,7 @@ jobs:
- name: Run E2E tests
env:
GINKGO_LABEL_FILTER: 'provider:onprem'
CLUSTER_DEPLOYMENT_NAME: ${{ needs.build.outputs.clustername }}
CLUSTER_DEPLOYMENT_PREFIX: ${{ needs.build.outputs.clusterprefix }}
IMG: 'ghcr.io/k0rdent/kcm/controller-ci:${{ needs.build.outputs.version }}'
VERSION: ${{ needs.build.outputs.version }}
run: |
Expand All @@ -252,7 +252,7 @@ jobs:
if: ${{ always() && !contains(needs.provider-cloud-e2etest.result, 'skipped') && contains(needs.build.result, 'success') }}
timeout-minutes: 15
outputs:
clustername: ${{ needs.build.outputs.clustername }}
clusterprefix: ${{ needs.build.outputs.clusterprefix }}
version: ${{ needs.build.outputs.version }}
pr: ${{ needs.build.outputs.pr }}
steps:
Expand All @@ -275,7 +275,7 @@ jobs:
AZURE_TENANT_ID: ${{ secrets.CI_AZURE_TENANT_ID }}
AZURE_CLIENT_ID: ${{ secrets.CI_AZURE_CLIENT_ID }}
AZURE_CLIENT_SECRET: ${{ secrets.CI_AZURE_CLIENT_SECRET }}
CLUSTER_NAME: '${{ needs.build.outputs.clustername }}'
CLUSTER_NAME: '${{ needs.build.outputs.clusterprefix }}'
run: |
make dev-aws-nuke
make dev-azure-nuke
6 changes: 5 additions & 1 deletion Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,14 +108,18 @@ tidy: ## Run 'go mod tidy' against code.
test: generate-all envtest tidy external-crd ## Run tests.
KUBEBUILDER_ASSETS="$(shell $(ENVTEST) use $(ENVTEST_K8S_VERSION) --bin-dir $(LOCALBIN) -p path)" go test $$(go list ./... | grep -v /e2e) -coverprofile cover.out

# E2E_CONFIG_B64 contains the configuration for e2e testing.
E2E_CONFIG_B64 ?= ""
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i don't quite understand the purpose of the env, I cannot find the place where the variable is being set with a value, could you elaborate on that?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

after the CR i got the idea, but still can't see who would benefit from the env var? i can't say that's it is very handy to manually generate a configuration using go structs, then encode it and put in the var, hence the questions

Copy link
Contributor Author

@eromanova eromanova Jan 22, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should have a way to pass the testing configuration to the Makefile (either locally or on CI). As the testing configuration is kinda complex I decided to avoid multiple variables and have a single env var with the encoded yaml configuration. I assumed the user would follow the guide of all available e2e testing configuration options and define that before running tests. For example:

config="aws: []
azure: []"
make E2E_CONFIG_B64="$(echo "$config" | base64)" test-e2e

It's b64 encoded to avoid multiline strings and all the problems that may occur due to that. But probably it's not needed.
Anyway, I’ll give it some more thought as there’s probably a better way to achieve what I’m aiming for.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's enough to guide how to properly set the configuration with a (almost) full config for a single provider to illustrate available fields, etc. I am also ok with encoding just wanted to understand the use case, for who, and when the var should be utilized


# Utilize Kind or modify the e2e tests to load the image locally, enabling
# compatibility with other vendors.
.PHONY: test-e2e
test-e2e: cli-install ## Run the e2e tests using a Kind k8s instance as the management cluster.
@if [ "$$GINKGO_LABEL_FILTER" ]; then \
ginkgo_label_flag="-ginkgo.label-filter=$$GINKGO_LABEL_FILTER"; \
fi; \
KIND_CLUSTER_NAME="kcm-test" KIND_VERSION=$(KIND_VERSION) go test ./test/e2e/ -v -ginkgo.v -ginkgo.timeout=3h -timeout=3h $$ginkgo_label_flag
KIND_CLUSTER_NAME="kcm-test" KIND_VERSION=$(KIND_VERSION) E2E_CONFIG_B64=$(E2E_CONFIG_B64) \
go test ./test/e2e/ -v -ginkgo.v -ginkgo.timeout=3h -timeout=3h $$ginkgo_label_flag

.PHONY: lint
lint: golangci-lint fmt vet ## Run golangci-lint linter & yamllint
Expand Down
9 changes: 3 additions & 6 deletions docs/dev.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -175,13 +175,10 @@ IMG="ghcr.io/k0rdent/kcm/controller-ci:v0.0.1-179-ga5bdf29" \
Optionally, the `NO_CLEANUP=1` env var can be used to disable `After` nodes from
running within some specs, this will allow users to debug tests by re-running
them without the need to wait a while for an infrastructure deployment to occur.
For subsequent runs the `CLUSTER_DEPLOYMENT_NAME=<cluster name>` env var should be
passed to tell the test what cluster name to use so that it does not try to
generate a new name and deploy a new cluster.

Tests that run locally use autogenerated names like `12345678-e2e-test` while
tests that run in CI use names such as `ci-1234567890-e2e-test`. You can always
pass `CLUSTER_DEPLOYMENT_NAME=` from the get-go to customize the name used by the
Tests that run locally use autogenerated names prefixes like `e2e-test-12345` while
tests that run in CI use names such as `ci-12345`. You can always
pass `CLUSTER_DEPLOYMENT_PREFIX=` from the get-go to customize the prefix used by the
test.

### Filtering test runs
Expand Down
19 changes: 19 additions & 0 deletions internal/utils/utils.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
// Copyright 2024
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package utils

func PtrTo[T any](v T) *T {
return &v
}
2 changes: 1 addition & 1 deletion test/e2e/clusterdeployment/aws/aws.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.

// Package aws contains specific helpers for testing a managed cluster
// Package aws contains specific helpers for testing a cluster deployment
// that uses the AWS infrastructure provider.
package aws

Expand Down
80 changes: 32 additions & 48 deletions test/e2e/clusterdeployment/clusterdeployment.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,16 +19,17 @@ import (
_ "embed"
"fmt"
"os"
"strings"

"github.com/a8m/envsubst"
"github.com/google/uuid"
. "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"
"gopkg.in/yaml.v3"
"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
"sigs.k8s.io/cluster-api/api/v1beta1"

"github.com/K0rdent/kcm/test/e2e/kubeclient"
"github.com/K0rdent/kcm/test/e2e/templates"
"github.com/K0rdent/kcm/test/utils"
)

Expand All @@ -40,20 +41,6 @@ const (
ProviderAzure ProviderType = "infrastructure-azure"
ProviderVSphere ProviderType = "infrastructure-vsphere"
ProviderAdopted ProviderType = "infrastructure-internal"
providerLabel = "cluster.x-k8s.io/provider"
)

type Template string

const (
TemplateAWSStandaloneCP Template = "aws-standalone-cp"
TemplateAWSHostedCP Template = "aws-hosted-cp"
TemplateAWSEKS Template = "aws-eks"
TemplateAzureHostedCP Template = "azure-hosted-cp"
TemplateAzureStandaloneCP Template = "azure-standalone-cp"
TemplateVSphereStandaloneCP Template = "vsphere-standalone-cp"
TemplateVSphereHostedCP Template = "vsphere-hosted-cp"
TemplateAdoptedCluster Template = "adopted-cluster"
)
eromanova marked this conversation as resolved.
Show resolved Hide resolved

//go:embed resources/aws-standalone-cp.yaml.tpl
Expand Down Expand Up @@ -91,45 +78,42 @@ func FilterAllProviders() []string {
}

func GetProviderLabel(provider ProviderType) string {
return fmt.Sprintf("%s=%s", providerLabel, provider)
return fmt.Sprintf("%s=%s", v1beta1.ProviderNameLabel, provider)
}

func setClusterName(templateName Template) {
var generatedName string

mcName := os.Getenv(EnvVarClusterDeploymentName)
if mcName == "" {
mcName = "e2e-test-" + uuid.New().String()[:8]
func GenerateClusterName(postfix string) string {
mcPrefix := os.Getenv(EnvVarClusterDeploymentPrefix)
if mcPrefix == "" {
mcPrefix = "e2e-test-" + uuid.New().String()[:8]
}

providerName := strings.Split(string(templateName), "-")[0]

// Append the provider name to the cluster name to ensure uniqueness between
// different deployed ClusterDeployments.
generatedName = fmt.Sprintf("%s-%s", mcName, providerName)
if strings.Contains(string(templateName), "hosted") {
generatedName = fmt.Sprintf("%s-%s", mcName, "hosted")
}
// TODO: quick w/a. The cluster names' generator will be refactored in https://github.com/k0rdent/kcm/pull/752
if strings.Contains(string(templateName), "eks") {
generatedName = fmt.Sprintf("%s-%s", mcName, "eks")
if postfix != "" {
return fmt.Sprintf("%s-%s", mcPrefix, postfix)
}
return mcPrefix
}

func setClusterName(name string) {
GinkgoT().Setenv(EnvVarClusterDeploymentName, name)
}

GinkgoT().Setenv(EnvVarClusterDeploymentName, generatedName)
func setTemplate(templateName string) {
GinkgoT().Setenv(EnvVarClusterDeploymentTemplate, templateName)
}

// GetUnstructured returns an unstructured ClusterDeployment object based on the
// provider and template.
func GetUnstructured(templateName Template) *unstructured.Unstructured {
func GetUnstructured(templateType templates.Type, clusterName, template string) *unstructured.Unstructured {
GinkgoHelper()

setClusterName(templateName)
setClusterName(clusterName)
setTemplate(template)

var clusterDeploymentTemplateBytes []byte
switch templateName {
case TemplateAWSStandaloneCP:
switch templateType {
case templates.TemplateAWSStandaloneCP:
clusterDeploymentTemplateBytes = awsStandaloneCPClusterDeploymentTemplateBytes
case TemplateAWSHostedCP:
case templates.TemplateAWSHostedCP:
// Validate environment vars that do not have defaults are populated.
// We perform this validation here instead of within a Before block
// since we populate the vars from standalone prior to this step.
Expand All @@ -139,20 +123,20 @@ func GetUnstructured(templateName Template) *unstructured.Unstructured {
EnvVarAWSSecurityGroupID,
})
clusterDeploymentTemplateBytes = awsHostedCPClusterDeploymentTemplateBytes
case TemplateAWSEKS:
case templates.TemplateAWSEKS:
clusterDeploymentTemplateBytes = awsEksClusterDeploymentTemplateBytes
case TemplateVSphereStandaloneCP:
case templates.TemplateVSphereStandaloneCP:
clusterDeploymentTemplateBytes = vsphereStandaloneCPClusterDeploymentTemplateBytes
case TemplateVSphereHostedCP:
case templates.TemplateVSphereHostedCP:
clusterDeploymentTemplateBytes = vsphereHostedCPClusterDeploymentTemplateBytes
case TemplateAzureHostedCP:
case templates.TemplateAzureHostedCP:
clusterDeploymentTemplateBytes = azureHostedCPClusterDeploymentTemplateBytes
case TemplateAzureStandaloneCP:
case templates.TemplateAzureStandaloneCP:
clusterDeploymentTemplateBytes = azureStandaloneCPClusterDeploymentTemplateBytes
case TemplateAdoptedCluster:
case templates.TemplateAdoptedCluster:
clusterDeploymentTemplateBytes = adoptedClusterDeploymentTemplateBytes
default:
Fail(fmt.Sprintf("Unsupported template: %s", templateName))
Fail(fmt.Sprintf("Unsupported template type: %s", templateType))
}

clusterDeploymentConfigBytes, err := envsubst.Bytes(clusterDeploymentTemplateBytes)
Expand All @@ -175,12 +159,12 @@ func ValidateDeploymentVars(v []string) {
}

func ValidateClusterTemplates(ctx context.Context, client *kubeclient.KubeClient) error {
templates, err := client.ListClusterTemplates(ctx)
clusterTemplates, err := client.ListClusterTemplates(ctx)
if err != nil {
return fmt.Errorf("failed to list cluster templates: %w", err)
}

for _, template := range templates {
for _, template := range clusterTemplates {
valid, found, err := unstructured.NestedBool(template.Object, "status", "valid")
if err != nil {
return fmt.Errorf("failed to get valid flag for template %s: %w", template.GetName(), err)
Expand Down
10 changes: 6 additions & 4 deletions test/e2e/clusterdeployment/constants.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,10 +16,12 @@ package clusterdeployment

const (
// Common
EnvVarClusterDeploymentName = "CLUSTER_DEPLOYMENT_NAME"
EnvVarControlPlaneNumber = "CONTROL_PLANE_NUMBER"
EnvVarWorkerNumber = "WORKER_NUMBER"
EnvVarNamespace = "NAMESPACE"
EnvVarClusterDeploymentName = "CLUSTER_DEPLOYMENT_NAME"
EnvVarClusterDeploymentPrefix = "CLUSTER_DEPLOYMENT_PREFIX"
EnvVarClusterDeploymentTemplate = "CLUSTER_DEPLOYMENT_TEMPLATE"
EnvVarControlPlaneNumber = "CONTROL_PLANE_NUMBER"
EnvVarWorkerNumber = "WORKER_NUMBER"
EnvVarNamespace = "NAMESPACE"
// EnvVarNoCleanup disables After* cleanup in provider specs to allow for
// debugging of test failures.
EnvVarNoCleanup = "NO_CLEANUP"
Expand Down
Loading
Loading