Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: add ability to run additional test tags against PRs #5658

Merged
merged 75 commits into from
Dec 11, 2024

Conversation

midleman
Copy link
Contributor

@midleman midleman commented Dec 7, 2024

Intent

This PR enhances our test-based workflows by making them more modular, composable, and dispatchable (is that a word?). In addition, we can now specify additional tags with an ampersand (&critical, etc) anywhere in PR text to run additional tests against a pull request.

Changes

Modular Workflows

  • Split core tests (e2e-linux, e2e-windows, unit-tests, integration-tests) into standalone workflows.
    • Parent workflows can now combine these tests as needed.
    • Added parameters/inputs to e2e workflows for tagging and filtering e2e tests.
  • Added bash script to handle combining Playwright tags to run a subset of tests:
    • Enables AND logic for tags where needed.
    • Handles “base tags” like @win or @web

Dynamic Tag Parsing

  • Allows tags (&critical, etc) in PR descriptions to be parsed and included dynamically.
  • Ensures at minimum that critical tests always run (even if not tagged)
  • Added a bash script to handle parsing PR text to pass along to e2e test runner

QA Notes

  • Ran the full test suite and confirmed it passed and reports were available as expected.
  • Once merged, I plan to manually verify each of the workflow dispatches works as expected, but I did try to test this as much as I could already.

Screenshots

test-full-suite.yml dispatch
full-suite

test-e2e-linux.yml dispatch
linux

test-e2e-windows.yml dispatch
windows

test-e2e-release.yml dispatch
latest-release

@midleman midleman force-pushed the mi/update-e2e-workflow branch from c4de42c to 55842e8 Compare December 8, 2024 18:26
@midleman midleman force-pushed the mi/update-e2e-workflow branch from 8a0b562 to 99976ca Compare December 9, 2024 15:31
@midleman midleman force-pushed the mi/update-e2e-workflow branch from f955bf3 to 6f1c83b Compare December 9, 2024 17:01
@midleman midleman changed the title wip ci: update test workflows Dec 9, 2024
@midleman midleman force-pushed the mi/update-e2e-workflow branch from e957bf5 to e85b7db Compare December 10, 2024 16:17
@midleman midleman force-pushed the mi/update-e2e-workflow branch 3 times, most recently from 1450af9 to a7d0727 Compare December 10, 2024 16:33
@midleman midleman force-pushed the mi/update-e2e-workflow branch from a7d0727 to 3b98832 Compare December 10, 2024 16:36
@midleman midleman force-pushed the mi/update-e2e-workflow branch from abc3310 to 7e13e8c Compare December 10, 2024 16:41
@midleman midleman changed the title ci: add ability to run additional test tags in PRs ci: add ability to run additional test tags against PRs Dec 10, 2024
testlabauto
testlabauto previously approved these changes Dec 10, 2024
Copy link
Contributor

@testlabauto testlabauto left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!
I think the best way to really review this properly is to start using it and I am going to that post merge in a minute.

@testlabauto
Copy link
Contributor

LGTM! I think the best way to really review this properly is to start using it and I am going to that post merge in a minute.

Oh, actually I think we are waiting for the upstream merge, but excited to try this out.

@midleman midleman merged commit c06aa01 into main Dec 11, 2024
5 checks passed
@midleman midleman deleted the mi/update-e2e-workflow branch December 11, 2024 18:55
@github-actions github-actions bot locked and limited conversation to collaborators Dec 11, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants