Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce "breakage testing" #1233

Merged
merged 4 commits into from
Oct 21, 2023
Merged

Introduce "breakage testing" #1233

merged 4 commits into from
Oct 21, 2023

Conversation

ericcornelissen
Copy link
Owner

@ericcornelissen ericcornelissen commented Oct 14, 2023

Summary

Add a new type of testing including a new test suite that's aimed solely at preventing the introduction of breaking changes. To this end the suite runs both the current version of the library and the latest released version and compares their behavior.

TODO

  • Add breakage test suite.
  • Add command to run breakage tests (w/ and w/o coverage).
  • Continuously run breakage tests in CI.
  • Document breakage testing in the Contributor Guidelines.

Add a new type of testing including a new test suite that's aimed solely
at preventing the introduction of breaking changes. To this end the
suite runs both the current version of the library and the latest
released version and compares their behaviour.
@github-actions github-actions bot added ci/cd Relates to ci/cd dependencies Changes to the project's dependencies meta Relates to the project or repository itself labels Oct 14, 2023
Expand the breakage test suite with a test file for the shescape/testing
module. This suite is a bit interesting because it contains some skipped
tests. These tests are skipped because of bugfixes and an API extension
which both occurred in [1].

The breakage coverage configuration is also updated to include the
testing module in the coverage report.

--
1. 4f03fd8
@ericcornelissen ericcornelissen marked this pull request as ready for review October 21, 2023 16:03
@ericcornelissen ericcornelissen merged commit 70a199d into main Oct 21, 2023
33 checks passed
@ericcornelissen ericcornelissen deleted the breakage-testing branch October 21, 2023 16:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ci/cd Relates to ci/cd dependencies Changes to the project's dependencies meta Relates to the project or repository itself
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant