Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nk3 test: Add --deny-skipped option #375

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

robin-nitrokey
Copy link
Member

This patch adds an option to the nk3 test subcommand that causes one or more skipped tests to fail the test run. This can be used in production to make sure that all selected tests are actually executed.

Open questions:

  • Should --all automatically activate this flag?

Changes

  • Add --deny-skipped flag to nk3 test.

Checklist

  • tested with Python3.9
  • signed commits
  • updated documentation (e.g. parameter description, inline doc, docs.nitrokey)
  • added labels

Test Environment and Execution

  • OS: Debian 11
  • device's model: NK3AM
  • device's firmware version: v1.3.1

This patch adds an option to the nk3 test subcommand that causes one or
more skipped tests to fail the test run.  This can be used in production
to make sure that all selected tests are actually executed.
@jans23
Copy link
Member

jans23 commented Apr 13, 2023

Should --all automatically activate this flag?

IMO yes, this is what I expect from it.

Why to introduce --deny-skipped additionally?

@robin-nitrokey
Copy link
Member Author

--all decides which tests are executed, and --deny-skipped how the results are interpreted. The reasoning behind the skipped state is that we don’t want to indicate failures to end users that are not really a failure.

@robin-nitrokey robin-nitrokey marked this pull request as draft April 23, 2023 17:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants