Skip to content

Latest commit

 

History

History
361 lines (253 loc) · 15.6 KB

TESTING.md

File metadata and controls

361 lines (253 loc) · 15.6 KB
Raw

Testing guidelines

Following guide will help you decide what to test and how to go about it when changing code in this repository.

Testing levels

In KIC, we use several levels of testing:

Unit tests

Unit tests verify the functionality of one or more func or structs in an isolated environment.

These tests are independent of the environment they are run in and require no setup or teardown code.

NOTE: Some of these tests do require envtest setup to be present e.g. this one or this one. This setup is handled automatically via setup-envtest Makefile target which is a dependency of mentioned above unit test targets.

As part of #4099 this organization might change and tests that do require this setup might end up in a separate place.

These tests can either be written in a way that do not require any mocks or fakes, or using e.g. Kubernetes related fakes like for instance TestAddressesFromEndpointSlice does.

How to run

Unit tests can be either run via appropriate go test ... invocation or using any of the provided Makefile targets:

  • test.unit run unit tests with standard verbose output on stderr
  • test.unit.pretty run unit tests with opinionated output format, one line per package

Unit tests are typically located in the same package as the code they test and they do not use any build tags.

envtest based tests

envtest based tests use a middle ground approach, somewhere between unit and integration tests. They run tests within an isolated, limited environment which allows some basic Kubernetes cluster actions like CRUD operations on Kubernetes entities and running controllers against this environment.

Ensuring that the environment is ready for tests is done by:

  • setup-envtest which ensures that api-server and etcd binaries are present on the local system
  • Makefile invocation of setup-envtest use which does what's described above

There already exist several helpers for working with envtest environment which can be found at test/envtest/.

NOTE: Currently, these tests that do require envtest setup and use a special build tags: envtest. This is in place so that simply running go test ... without any special build tags would pass without requiring any manual setup from the developer (no failure due to missing envtest related binaries or environment variables).

Test definitions can be found both in dedicated directory - test/envtest - and throughout the codebase next to code that they are testing. This approach however may change over time. Suggestion for new envtest based tests is to be placed in test/envtest and to test code as a black box, i.e. using only exported methods (which will be enforced via running them from this dedicated package).

How to run

envtest based tests can be run via test.envtest Makefile target.

One can also use ./bin/setup-envtest use -p env to obtain the asset dir environment variable:

./bin/setup-envtest use -p env
export KUBEBUILDER_ASSETS='/Users/username/Library/Application Support/io.kubebuilder.envtest/k8s/1.27.1-darwin-arm64'

and use that to manually run the tests:

$ eval $(./bin/setup-envtest use -p env)
$ go test -v -count 1 -tags envtest ./pkg/to/test
...

Integration tests

KIC's integration tests rely on its controller manager being run in the same process as the tests via manager.Run().

These tests rely on a test suite setup using TestMain().

Said setup will either create a new kind cluster or use an existing one to run the tests against.

Most of the setup - and cleanup after the tests have run - is being done using ktf.

Currently, the cluster is being shared across all the tests that are a part of the test suite so special care needs to be taken in order to clean up after the tests that run. The typical approach is to run them in a dedicated, disposable Kubernetes namespace created just for the purposes of that test.

How to run

Integration tests are located under tests/integration/ and use integration_tests build tag.

You can run them using one of the dedicated Makefile targets:

  • test.integration run all integration tests with standard verbose output on stderr. This will run tests for dbless, postgres, enterprise and non enterprise.

  • test.integration.dbless run all dbless integration tests with standard verbose output on stderr. The output will also include controllers' logs.

  • test.integration.postgres run all postgres integration tests with standard verbose output on stderr. The output will also include controllers' logs.

Through GOTESTFLAGS you can specify custom flags that will be passed to go test. This can allow you to run a subset of all the tests for faster feedback times, e.g.:

make test.integration.dbless GOTESTFLAGS="-count 1 -run TestUDPRouteEssentials"

Isolated integration tests

Similarly to KIC's integration tests, isolated integration tests rely on its controller manager being run in the same process as the tests via manager.Run().

These tests rely on kubernetes-sigs/e2e-framework for setup, teardown, tests filtering etc.

Said setup will either create a new kind cluster or use an existing one to run the tests against.

Most of the setup - and cleanup after the tests have run - is being done using ktf.

Currently, the cluster is being shared across all the tests that are a part of the test suite so special care needs to be taken in order to clean up after the tests that run. The typical approach is to run them in a dedicated, disposable Kubernetes namespace created just for the purposes of that test.

Difference between isolated and regular integration tests

There are a couple of key differences between isolated and regular integration tests:

  • In isolated tests each test's Feature gets its own Kong deployment through the means of ktf's Kong addon.

  • In isolated tests each test's Feature gets its own controller manager instance started which is configured against the above mentioned kong instance. Said controller manager instance will be configured to only watch Feature's dedicated namespace (more info below) via --watch-namespace. You can add more namespaces for the controller manager to watch via ControllerManagerOptAdditionalWatchNamespace featureSetup option:

    ...
WithSetup("deploy kong addon into cluster", featureSetup(
  helpers.ControllerManagerOptAdditionalWatchNamespace("my-additional-namespace"),
)).
...

  • In regular integration tests each test gets its own Kubernetes namespace through manual creation via helper functions like:

    ns, cleaner := helpers.Setup(ctx, t, env)

    Which will automatically remove the namspace when the test is finished.

    In isolated tests each test's Feature will get its own Kubernetes namespace which you can get via:

    namespace := GetNamespaceForT(ctx, t)

Room for improvement

  • Eventually the whole integration suite could be migrated to use this setup. When that happens, we can add logic into setup which would make tests that don't need this level of separation to reuse a common installation of Kong (e.g. to its default - kong - namespace).

    This way we'll have the best of both worlds:

    • seprate tests where it's needed
    • shared, when it's not and where speed is the priority

How to run

Tests are located under tests/integration/isolated and use integration_tests build tag.

You can run them using one of the dedicated Makefile targets:

  • test.integration.isolated.dbless run all dbless isolated integration tests with standard verbose output on stderr. The output will also include controllers' logs.

Through GOTESTFLAGS you can specify custom flags that will be passed to go test. This can allow you to run a subset of all the tests for faster feedback times, e.g.:

make test.integration.isolated.dbless GOTESTFLAGS="-count 1 -run TestUDPRouteEssentials"

You can also specify e2e-framework's flags e.g. to filter tests via labels.

make test.integration.isolated.dbless E2E_FRAMEWORK_FLAGS="-labels=kind=UDPRoute,example=true"

Kong integration tests

Tests located under test/kongintegration/ allow verifying individual components of KIC against a running Kong Gateway instance.

Every test in this package requires a Kong instance to be running, but do not require a Kubernetes cluster nor a full KIC deployment. Each test is responsible for setting up its own Kong instance and cleaning it up after the test is finished. testcontainers-go is used for that purpose.

Examples of tests that could fit into this category are:

  • Ensuring that a component responsible for configuring Kong Gateway works as expected
  • Ensuring that configuration generated by a translator is accepted by Kong Gateway

How to run

The only requirement for running Kong integration tests is to have a docker daemon running.

All tests can be run using the following Makefile targets:

# Verbose output
make test.kongintegration

# Pretty output
make test.kongintegration.pretty

By default, kong:latest image is used for running Kong Gateway. This can be changed by setting the TEST_KONG_IMAGE and TEST_KONG_TAG environment variables.

Potential issues

testcontainer's Garbage Collector

In case your environment doesn't allow running containers in privileged mode, you may encounter issues with testcontainer's garbage collector (Ryuk) which requires it. In Makefile targets we disable the garbage collector by setting TESTCONTAINERS_RYUK_DISABLED environment variable to true.

If you encounter issues with this while running tests directly (e.g. when debugging in an IDE), you can fix it by:

  • setting TESTCONTAINERS_RYUK_DISABLED environment variable to true in your IDE configuration,
  • adding ryuk.disabled=true to .testcontainers.properties file in your home directory (see Configuration locations for exact locations depending on your OS).

End-to-End (E2E) tests

End to end tests in KIC are used to test features or use cases which rely on KIC being deployed in-cluster.

For instance:

These tests deploy KIC and Kong Gateway in a cluster using the requested image(s) which could be customized via dedicated environment variables like:

On CI, those are being run both in kind and GKE environments.

How to run

E2E tests are located under tests/e2e/ and use e2e_tests build tag.

You can run them using one of the dedicated Makefile targets:

  • test.e2e run all e2e tests.

Through GOTESTFLAGS you can specify custom flags that will be passed to go test.

E2E_TEST_RUN is also available to specify the name of the test to run. This is being used in CI where Github Actions' matrix specifies tests to run in each workflow.

Exemplar local invocation can look like this:

make test.e2e GOTESTFLAGS="-count 1" E2E_TEST_RUN=TestDeployAllInOneDBLESS
Konnect tests

Some of the E2E tests are Konnect-specific and are being run only when the TEST_KONG_KONNECT_ACCESS_TOKEN environment variable is set. You can create the access token using the Konnect UI. The tests run against a .tech Konnect environment therefore the token you obtain must be created for an account in that environment.

Exemplary local invocation for running a Konnect test can look like this:

make test.e2e GOTESTFLAGS="-count 1" E2E_TEST_RUN=TestKonnectConfigPush TEST_KONG_KONNECT_ACCESS_TOKEN=<token>