Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run the integration test in a random order #19065

Open
timvandermeij opened this issue Nov 17, 2024 · 2 comments
Open

Run the integration test in a random order #19065

timvandermeij opened this issue Nov 17, 2024 · 2 comments
Assignees
Labels

Comments

@timvandermeij
Copy link
Contributor

timvandermeij commented Nov 17, 2024

The integration tests currently run in a fixed order, see https://github.com/mozilla/pdf.js/blob/master/test/integration-boot.mjs#L24, which is in contrast with e.g. the unit test that deliberately run in a random order to bring any isolation issues (dependencies between tests) to the surface. Most likely this wasn't done for the integration tests yet because we used to have quite a lot of intermittent issues and dependencies between tests when we started with the integration tests, but over time we have fixed most of them so hopefully running the integration tests in a random order is more feasible nowadays.

We should try this out and make a list of any issues we encounter. This could involve closing the document between tests in the same describe block to avoid state leaking through to subsequent tests; see e.g. PR #19064 for which an alternative solution was implemented.

The objective is to reduce the number of test failures (currently if there is a failure in one test it'll currently cause all subsequent dependent tests to fail too) and to improve debugging of the remaining intermittents (currently we sometimes need to run a full describe block only to debug one test which is time-consuming).

@calixteman
Copy link
Contributor

See #19064 (comment).

@timvandermeij timvandermeij self-assigned this Dec 22, 2024
@timvandermeij
Copy link
Contributor Author

timvandermeij commented Dec 22, 2024

I have tested this locally today and that has resulted in a relatively good overview of what needs to be done to achieve the goal of independent tests that can be run in a random order.

Some spec files already work in random mode, but others are problematic because they contains tests that build upon the result of previous tests and therefore fail immediately if the document is closed between tests. Fortunately this process does bring dependencies between subsequent tests to the surface so we can fix them, and from what I can tell enabling random order should directly be possible then because I haven't found dependencies between non-subsequent tests.

This work roughly involves closing the document between all tests, fixing every failing test and enabling random mode. This is the result of the test:

File Passes if document is closed Passes (three times) if random mode is enabled
accessibility_spec.mjs
annotation_spec.mjs
caret_browsing_spec.mjs
copy_paste_spec.mjs
find_spec.mjs
freetext_editor_spec.mjs ❌ (8 of 53 failed)
highlight_editor_spec.mjs
ink_editor_spec.mjs ❌ (1 of 19 failed)
scripting_spec.mjs ❌ (3 of 63 failed)
stamp_editor_spec.mjs
text_field_spec.mjs
text_layer_spec.mjs
viewer_spec.mjs

I have a local branch with fixes and refactorings, and once the work is complete I'll create PRs for each self-contained chunk.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants