KEP-0008: Multi-cluster - Proposals for TestSuite Changes

[Miles-Garnsey]

What this PR does / why we need it:

KEP-0008 is currently light on detail. To get it ready for implementation this PR fleshes out proposed changes to the TestSuite API.

The changes proposed are designed to allow for the creation of multiple KinD clusters. They aim to leave room for the future addition of external clusters, and local clusters (etcd and apiserver running uncontainerised).

[kensipe]

a number of questions / comments which will likely drive some organization

1. there has been a history of folks wanting kuttl to be the cluster creation tool as well as the testing tool. We prefer cluster creation to be a pre step to kuttl. We did add kind as a convenience, but I often wonder if we should have. it is great for some CI capabilities. A concern is supporting more cluster types. while k3s and others could be interesting.. it would add a lot of dependencies for things that isn't the pure intention of kuttl. If someone wants gcp, or minikube or k3s... we don't have any plans to support the cluster spin up... but we happily support the testing.
2. It would be good to start with the testing components first... what does it mean to support that? how is it expected to report back. Examples of questions we need to solve for.
   a. can a teststep span clusters or is it single cluster only
   b. can we look at the existing tests as a set of testsuites defined by a cluster and adjust our current tests as a first pass to have an overarching cluster (which is 1 by default... providing backward compat)
   c. assuming a suite is for a cluster.. is there any relationship between the testsuites?

generally we think of TestSuites-> TestSteps -> Steps. Are we designing for a cluster that contains TestSuites? or ? If this is the course, it feels like we should define a name for that "ClusterSuites"? or "ClusterTests".

Love the contribution and drive. Thank you!

[kensipe]

and thanks for the DCO!

[kensipe]

and to add to the thinking... can the cluster tests run in parallel? or is it gated in some way... I can image the need for a clusterwide setup / assert to confirm we can execute each cluster. I can imagine 1 cluster testsuite being required after another. etc. It is the lifecycle of tests which should come to a good mental model on.

[Miles-Garnsey]

I can see why you'd say that the TestStep/reporting could be a good place to start. The reason I started with the TestSuite was that there needs to be some way to identify the context in which the tests will run, which is always going to entail both a kubeconfig file and a context within it.

It seemed ideal to have some sort of keyed data structure (I chose a map in this case) that could identify all of the involved clusters across the entire TestSuite, the keys can then be referred to in the TestSteps and assertions/errors.

It would certainly be simpler to have the TestSuite declare only the kubeconfig/context pair and avoid thinking about setting up clusters. Should I remove discussion of setting up clusters from this PR?

The real question here is - what key do we want to use to refer to a cluster? The original proposal called for a path to a kubeconfig yaml but that seemed insufficient given a single kubeconfig can contain multiple contexts.

PS: I've raised separate PRs for discussion on the TestSteps here, and asserts/errors here. I wanted to take @kensipe's advice and keep the PRs small.

[Miles-Garnsey]

I was going to answer the remainder of the questions in the TestStep PR, but I think they're too closely related to TestSuite. Maybe this should have been a single PR.

can a teststep span clusters or is it single cluster only

I think that a TestStep should be able to run on multiple clusters, yes. With the constraint that all of the scripts/commandss in the TestStep must run on the same set of clusters.

can we look at the existing tests as a set of testsuites defined by a cluster and adjust our current tests as a first pass to have an overarching cluster (which is 1 by default... providing backward compat)

I don't quite understand the question here. I think that the regular single-cluster behaviour needs to remain 100% unchanged. Backwards compatibility is obviously a priority.

assuming a suite is for a cluster.. is there any relationship between the testsuites?

Ah, I was envisaging a TestSuite would span multiple clusters (as with TestStep). Are there reasons that we'd want multiple TestSuites?