Run VSphere tests in self-hosted, refactor test suite into spec per provider, support label filtering, get all tests green #360
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
squizzi
added
the
test e2e
squizzi
force-pushed
the
vsphere-tests-on-self-hosted
branch
2 times, most recently
from
dc5f8d2 to
605513d
Compare
This comment was marked as resolved.
This comment was marked as resolved.
squizzi
force-pushed
the
vsphere-tests-on-self-hosted
branch
5 times, most recently
from
92cc4f5 to
d4f0344
Compare
This was referenced Sep 24, 2024
squizzi
force-pushed
the
vsphere-tests-on-self-hosted
branch
from
d4f0344 to
fea83d7
Compare
squizzi
changed the title
squizzi
force-pushed
the
vsphere-tests-on-self-hosted
branch
4 times, most recently
from
f76cfe0 to
e19f75f
Compare
squizzi
changed the title
squizzi
force-pushed
the
vsphere-tests-on-self-hosted
branch
8 times, most recently
from
7560f52 to
00088a6
Compare
|
kylewuolle
previously approved these changes
Oct 10, 2024
a13x5
previously requested changes
Oct 10, 2024
There was a problem hiding this comment.
Some minor notes here and there.
I don't have major concerns regarding actual tests, apart from the fact that standalone and hosted CP tests are squashed together.
I understand the reason for that, since we're reusing standalone cluster and it must be created anyway for hosted CP tests, but I see the following cons of this approach:
- standalone and hosted tests can't be run separately
- If hosted CP fails (and it will a lot I'm sure) it will fail the whole AWS test, which makes it slightly harder to interpret the test.
I know that it's e2e tests and we should run the whole suite, but I believe it makes sense to run only on the release or ad-hoc. Most changes could be tested by simply running tests on standalone cluster.
I will not block this PR because of that, but I a task for the future us would be nice.
templates/cluster/vsphere-standalone-cp/templates/k0scontrolplane.yaml
Outdated
Show resolved
Hide resolved
templates/cluster/vsphere-standalone-cp/templates/k0scontrolplane.yaml
Outdated
Show resolved
Hide resolved
squizzi
force-pushed
the
vsphere-tests-on-self-hosted
branch
2 times, most recently
from
80fc91c to
9c6a1b4
Compare
squizzi
force-pushed
the
vsphere-tests-on-self-hosted
branch
from
9c6a1b4 to
4ee982a
Compare
squizzi
force-pushed
the
vsphere-tests-on-self-hosted
branch
2 times, most recently
from
9630f10 to
803e93c
Compare
slysunkin
previously approved these changes
Oct 15, 2024
squizzi
force-pushed
the
vsphere-tests-on-self-hosted
branch
from
803e93c to
eb66984
Compare
* Break provider tests into seperate files with labels representing either onprem or cloud providers. * Add new jobs to CI workflow which dictate where tests will run, onprem provider tests like vSphere will run on self-hosted runners since they will use internal resources to test. Cloud provider tests will use the existing workflow since they can access providers without network access and can take advantage of the much larger GitHub hosted pool. Hosted/Self-hosted tests can run concurrently. * Make Cleanup depend on the cloud-e2etest only. * Use new GINKGO_LABEL_FILTER to manage what tests run where. * Move controller validatation into BeforeSuite since the controller needs to be up and ready for each provider test, this will also enable us to add controller specific test cases later and make those run without the "test e2e" flag. * Seperate self-hosted and hosted test concurrency groups * Update docs with test filtering instructions * Ensure a Release exists for the custom build.version we deploy * Move all e2e related helpers into e2e dir * Add new clusteridentity package for creating ClusterIdentity kind's and associated Secret's. * Merge PR workflows together * Make sure VERSION gets passed across jobs * Ensure uniqueness among deployed ManagedClusters, simplify MANAGED_CLUSTER_NAME in CI to prevent Azure availabilitySetName validation error. * Default Azure test templates to uswest2 to prevent issues with AvailabilityZone. * Use the same concurrency-group across all jobs, except Cleanup which intentionally does not belong to a concurrency-group. * Use Setup Go across jobs for caching. * Support patching other hosted clusters to status.ready with a common patching helper. * Move VSphere delete into AfterEach to serve as cleanup. * Add support for cleaning Azure resources. * Prevent ginkgo from timing out tests. * Use azure-disk CSI driver we deploy via templates. Signed-off-by: Kyle Squizzato <[email protected]>
Signed-off-by: Kyle Squizzato <[email protected]>
Signed-off-by: Kyle Squizzato <[email protected]>
Signed-off-by: Kyle Squizzato <[email protected]>
squizzi
force-pushed
the
vsphere-tests-on-self-hosted
branch
from
eb66984 to
24603f7
Compare
|
@a13x5 rebased |
a13x5
approved these changes
Oct 17, 2024
slysunkin
approved these changes
Oct 17, 2024
|
Will be merging w/ admin rights to bypass the checks Job will not run on this patch because of @squizzi will make a follow-up after this PR merge to ensure no issues are present. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR refactors our existing e2e test spec into a more conventional e2e test run. We now have a
BeforeSuitethat runs and installs/validates the controller. Prior to this we had all of our e2e tests under a singleControllerspec, but this breaks it into:Label(controller))Label(provider))Label(provider:aws, provider:cloud)) -test/e2e/provider_aws_test.goLabel(provider:azure, provider:cloud)) -test/e2e/provider_azure_test.goLabel(provider:vsphere, provider:onprem)) -test/e2e/provider_vsphere_test.goThe labels for these tests have been documented here: https://github.com/Mirantis/hmc/pull/360/files#diff-3a6cab76ecce19612a704fe0ef89dc83f1ed2df44f7bfa28d47d2529a7949fafR139
Each of the umbrella labels correspond to a GitHub job that is dependent on the
Build and Unit Testjob which provides setup. TheControllertests always run, no matter the label, right now theControllertest is disabled because there aren't any tests other than Before/AfterSuite, but we can enable it because that sort of is a test of the controller in itself.The provider tests are still attached to the
test e2elabel, but I think we can break these out further withproviderspecific GitHub labels at some point.provider:onpremtests run on a self-hosted GitHub runner that has access to Mirantis' network, this is to support providers that do not necessarily have cloud infrastructure we can get to from a GitHub hosted runner -- like VSphere.This PR also adds support for creating
ClusterIdentity,Credentialand their associatedSecretresources with a newclusteridentitypackage.And of course, with testing comes bug fixing, so this PR fixes several different issues discovered along the way.
Closes: #210, Closes: #323
Note: I've opened two other issues to further simplify these tests, we could make these a lot easier to write in the future with: