Switch generate_psa_test.py to automatic dependencies for positive test cases #83
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
gilles-peskine-arm
added
needs-work
needs-ci
5 tasks
The new class `psa_test_case.TestCase` will automatically infer dependencies from the test data. The dependency inference is not done yet, this will be implemented in subsequent commits. No change to any generated file since this new module is not used yet. Signed-off-by: Gilles Peskine <[email protected]>
Allow "skipping" a test case, meaning that the test case is generated commented out. This is useful when systematically generating test cases according to certain rules, where some generated tests cannot be executed but we still want them to be visible when auditing the generation output. Signed-off-by: Gilles Peskine <[email protected]>
But for now, fully override its automatic dependency inference. We will switch to using the automatic dependencies in future commits. No change to the generated files. Signed-off-by: Gilles Peskine <[email protected]>
No change to the generated files. Signed-off-by: Gilles Peskine <[email protected]>
…anisms No change to the generated output. Signed-off-by: Gilles Peskine <[email protected]>
Signed-off-by: Gilles Peskine <[email protected]>
To determine PSA mechanisms that are not implemented, also read `PSA_WANT_` symbols that cannot (or are not intended to) be configured independently, and thus are not listed in `psa/crypto_config.h`. Find those symbols in the config adjustment header `psa/crypto_adjust_config_synonyms.h`. No impact on generated files yet, because `find_dependencies_not_implemented` is currently only used on key types that have explicit dependencies. This will allow using hack_dependencies_not_implemented in other places, for example to handle algorithm variants like `PSA_WANT_ALG_ECDSA_ANY` which is inferred from `PSA_WANT_ALG_ECDSA`. Signed-off-by: Gilles Peskine <[email protected]>
We can't even attempt to generate DSA test cases because `asymmetric_key_data.py` doesn't have test data for DSA. Signed-off-by: Gilles Peskine <[email protected]>
When we generate a test case for a mechanism that is not implemented, comment out the test case rather than giving it a never-fulfilled dependency. That way we don't create test cases that cannot be executed. This changes the generated output in the following ways: * No longer emit test cases with a dependency on `DEPENDENCY_NOT_IMPLEMENTED_YET`. All removed lines that start with `depends_on:` contain `DEPENDENCY_NOT_IMPLEMENTED_YET. * Emit commented-out test cases instead: all the new lines are comment lines. There is no change in which test cases actually get executed. This removes many test cases from the list of available test cases, which causes some of the exceptions in `analyze_outcomes.py` to no longer be useful. Signed-off-by: Gilles Peskine <[email protected]>
No semantic change. In the generated files, `depends_on:` lines have entries that are reordered. Signed-off-by: Gilles Peskine <[email protected]>
No change to the generated files (the new code isn't used yet). Signed-off-by: Gilles Peskine <[email protected]>
In `psa_test_cases.TestCase`: * Implement basic support for automatic dependencies, by calling `psa_information.automatic_dependencies`. * Support an alternative dependency prefix. No changes to the generated file. Signed-off-by: Gilles Peskine <[email protected]>
This fixes the dependencies for DH group and elliptic curve families. No changes to the generated output (the new functionality isn't used yet). Signed-off-by: Gilles Peskine <[email protected]>
This fixes the dependencies for key pair types, which have finer-grained dependencies for different operations (BASIC, GENERATE, EXPORT, ...). No changes to the generated output (the new functionality isn't used yet). Signed-off-by: Gilles Peskine <[email protected]>
No changes to the generated output (the new functionality isn't used yet). Signed-off-by: Gilles Peskine <[email protected]>
Automatically skip test cases with not-implemented automatic dependencies. No changes to the generated output. Signed-off-by: Gilles Peskine <[email protected]>
This causes more test cases to be commented out due to mechanisms that are not implemented, because the code `generate_psa_tests.StorageFormat` was not trying to skip never-supported dependencies. To review for correctness, filter the diff of the generated files as follows to find new skip reasons: ``` grep -E '^\+## # skipped because' | sort -u ``` And check that none of the appearing mechanisms are implemented. Signed-off-by: Gilles Peskine <[email protected]>
gilles-peskine-arm
force-pushed
the
dev/gilles-peskine-arm/psa-storage-test-cases-never-supported-positive-framework
branch
from
18e5920
to
b2ea59e
Compare
|
Needs rebase atop #67 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is mostly refactoring around
generate_psa_tests.pyand supporting Python libraries. Create a class for test case generation that can automatically determine test case dependencies. Use this forgenerate_psa_tests.pyfor positive test cases.Practical effect:
This is a step on the forward port of Mbed-TLS/mbedtls#9025. Here, I handle the positive test cases of
generate_psa_tests.py, which brings me. to a natural cutoff point.Follow-up: handling the negative test cases (
KeyTypeNotSupported,OpFail).Status: work in progress. Only tested on 3.6 so far. I'll probably make some changes to the history before this is finalized.
PR checklist