-
Notifications
You must be signed in to change notification settings - Fork 13
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
build: implement visual regression testing with @web/test-runner #2624
Conversation
47cb60a
to
0ac1261
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some elements are still obscure but I have a general idea of the implementation.
I'm not sure how the remote baseline is used and why we need to use a container to run the diff locally (but I'm sure there is a reason)
Some discussion topics:
- I'm a bit scared of the exponential nature of the number of
cases
(P1 * P2 * ... * Viewport * browsers * 3 (default, focus & hover)
)- Pro: covers almost every use case
- Con: grows very quickly and requires an "implementation" for each component
- Alternatively, we could reuse the stories template?
- Pro: we would avoid most of the
.snapshot.spec
code - Con: not exhaustive like combining each case
- Pro: we would avoid most of the
Codecov ReportAttention: Patch coverage is
❗ Your organization needs to install the Codecov GitHub app to enable full functionality. Additional details and impacted files@@ Coverage Diff @@
## main #2624 +/- ##
==========================================
- Coverage 93.18% 93.13% -0.06%
==========================================
Files 316 315 -1
Lines 25389 25323 -66
Branches 2063 2071 +8
==========================================
- Hits 23660 23585 -75
- Misses 1696 1705 +9
Partials 33 33 ☔ View full report in Codecov by Sentry. |
f5256ad
to
42cf6c2
Compare
…r-setup' into build-visual-regression-container-setup # Conflicts: # package.json # src/components/button/button/button.snapshot.spec.ts # src/components/core/testing/private/describe-each.ts # src/components/core/testing/private/describe-viewports.ts # src/components/core/testing/private/visual-regression-snapshot.ts # src/components/core/testing/test-setup.ts # yarn.lock
Signed-off-by: Jeremias Peier <[email protected]>
Signed-off-by: Jeremias Peier <[email protected]>
Signed-off-by: Jeremias Peier <[email protected]>
Signed-off-by: Jeremias Peier <[email protected]>
Signed-off-by: Jeremias Peier <[email protected]>
Signed-off-by: Jeremias Peier <[email protected]>
Signed-off-by: Jeremias Peier <[email protected]>
Signed-off-by: Jeremias Peier <[email protected]>
Thanks for the review and feedback! |
This PR sets up the basic configuration for visual regression testing with
@web/test-runner
.web-test-runner.config.js
contains a new test group, which is only activated when explicitly called withyarn test:visual-regression
. This test group introduces a new file pattern*.snapshot.spec.ts
, which contains the visual regression tests.Additionally, there is a new GitHub Actions workflow, which runs the visual regression tests and saves failed screenshots in an artifact. This artifact is then downloaded in the secure workflow, which adds an
in_progress
check, if any failed screenshots exist.We do not want to save the snapshot png in the repository, as this would bloat it considerably. Due to this, we use containers as an external storage of the png files. The process works as follows:
Each continuous integration run on our main branch creates/updates a container
baseline
(https://github.com/lyne-design-system/lyne-components/pkgs/container/lyne-components%2Fvisual-regression). For any other CI run (in the secure context) this container is used as a service and the png files are downloaded when needed.Additionally, the baseline image is hosted at
https://lyne-visual-regression-baseline.app.sbb.ch/
in order to enable local visual regression testing.As for local visual regression testing; To avoid diffs between operating system browser differences, visual regression testing is only directly run in a Linux environment. If it is executed in any other OS, it will try to run it in a container to prevent OS differences showing up in the snapshots.