Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use Pytest for testing, enable step version tests #342

Merged
merged 1 commit into from
Oct 10, 2023

Conversation

maxhoesel
Copy link
Collaborator

This PR moves testing from tox + podman to Pytest and Docker and enables testing multiple versions of the smallstep tools in CI

Rationale:
With the planned release of the 1.0 version of this collection, we are getting rid of the lockstepped version requirement for smallstep tools. This requires that we specify a minimum supported smallstep version and thus also test it in CI.
Instead of implementing this for the old tox-based setup, this PR instead runs ansible tests with a Pytest-based approach.
This is done for the following reasons:

  • Molecule tox environments were created by the tox-ansible plugin, which does not allow for easy parametrization of external tools. We most likely would have had to pass in the step versions as posargs in CI, invalidating the principle of having one tox environment for one compatibility test
  • The tox-ansible plugin recently released a rewritten version 2, which does not support molecule natively, so we would have been stuck with an old unmaintained version
  • By using pytest, we can offer much better integration into existing tooling - test cases show up in IDEs/Editors, and running them is as easy as typing pytest once the dependencies are installed
  • Instead of using tox for compatibility scenarios, we disable them by default for local tests (faster, about 30 minutes), but also include easy-to-use pytest args that allow users to test any combination of versions themselves
  • We also use this opportunity to move away from podman, back to Docker (previous issue: About molecule-docker+systemd issues and migrating Molecule tests to Podman... #105). Turns out that there is a workaround for getting systemd working in docker under cgroups2 (though it requires rw on the cgroups directory). This has the nice effect of making the setup of a local dev env pretty much painless (only python + docker) and doesn't seem to affect runtime negatively
  • By using a full language like python for our tests instead of tox, we can get rid of the docker-compose files in tests and instead access the docker CLI directly, as done in the test fixtures

The collection-testing orb has already released a new version with the required pytest command and with parallelism support on a per-test basis

Partially addresses #335. The README has intentionally been left untouched as the step-cli versions used in CI are provisional. We should decide on what minimum version to support with the full 1.x release once all other features are implemented

@maxhoesel maxhoesel added the chore Backend change that doesn't affect users label Oct 10, 2023
@maxhoesel maxhoesel force-pushed the step_multiversion_tests branch 2 times, most recently from da4211a to a7ba74d Compare October 10, 2023 16:03
@maxhoesel maxhoesel added this to the version-1 milestone Oct 10, 2023
@maxhoesel maxhoesel force-pushed the step_multiversion_tests branch from a7ba74d to 6a0b7b1 Compare October 10, 2023 16:06
@maxhoesel maxhoesel merged commit 806a5f8 into main Oct 10, 2023
1 check passed
@maxhoesel maxhoesel deleted the step_multiversion_tests branch October 10, 2023 17:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
chore Backend change that doesn't affect users
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant