Support v2 test commands #32
Conversation
src/data/types.js
Outdated
| @@ -41,7 +41,8 @@ | |||
| * @property {string} [commands[].extraInstruction] human-readable additional instruction to follow | |||
| * @property {object[]} assertions[] | |||
| * @property {1 | 2} assertions[].priority | |||
| * @property {string} assertions[].expectation | |||
| * @property {string} [assertions[].expectation] - Expected result, required if assertionStatement is not provided. | |||
| * @property {string} [assertions[].assertionStatement] - Assertion statement, required if expectation is not provided. | |||
There was a problem hiding this comment.
This is not reflected in the canonical definition of the type, contradicting the single-line comment which precedes this multi-line comment. @howard-e can you see about updating ARIA-AT as necessary? Does this need to be reflected in the Test Format V2 design document?
There was a problem hiding this comment.
contradicting the single-line comment which precedes this multi-line comment
expectation is a derived value that came from the assertions found in a v1 tests.csv files format. expectation is synonymous with assertionStatement in v2 and could technically still be referenced through the expectation value but keeping them separate is preferred (to me) since it will line up with a known *.csv property. The comment indicates that this is the case, kinda. But it also makes it seem like there could be a case where expectation and assertionStatement could exist simultaneously for a version of a single test plan.
@howard-e can you see about updating ARIA-AT as necessary?
Can do!
Does this need to be reflected in the Test Format V2 design document?
I think my above comment might not require this but just to be clear, on reflecting assertionStatement?
There was a problem hiding this comment.
I think my above comment might not require this but just to be clear, on reflecting assertionStatement?
Yup, thanks!
There was a problem hiding this comment.
I attempted to clarify the comments to make it clear when to expect which and that they exist exclusive of one another. Let me know if there is any other change needed in this repo.
There was a problem hiding this comment.
Follow-up: w3c/aria-at#1025
There was a problem hiding this comment.
LGTM - Merging
(nevermind, saw change request from @jugglinmike , gonna let him review and merge)
This change was originally proposed and accepted in the aria-at-automation-harness repository (which maintains a copy of the type definition) [1]. [1] w3c/aria-at-automation-harness#32 (comment)
This change was originally proposed and accepted in the aria-at-automation-harness repository (which maintains a copy of the type definition) [1]. [1] w3c/aria-at-automation-harness#32 (comment)
This work adds support to the harness for v2 test output. My work here is limited by my lack of access to a Windows machine so I will lean on my reviewer(s) for verification that this works.
To test my work locally, I ran a mock agent. I was able to get identical output from identical v2/v2 tests:
As part of this work, I added validation to the mock agent as well which necessitates an update to the testing snapshots as the validation writes logs around the PAGE_UP commands built into the earlier test files. The test coverage remains the same after these updates.