-
Notifications
You must be signed in to change notification settings - Fork 20.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Migrate testing infrastructure to minimal dependencies #5418
Conversation
New npm scripts include: - test:unit => run tests using a yargs command. See `npm run test:unit -- --help`. - test:unit:all => bypass yargs and run all tests in chrome headless without isolation. - test:server => start up a local test server to run tests manually Notes: - add helpers for the browserstack REST api (node-browserstack is out-of-date and not maintained) - add browserstack-compatible report API with progress updates - allow different browsers to run in parallel - add retries for workers with broken tunnels and refreshes for acknowledged workers that haven't responded in a while - continually checks maximum allowed browserstack sessions by default when adding new workers to a test run - bs-local.com used for the testing URLs, which is required for iOS (and for some reason desktop Safari), but works in all browsers. - IE11 required an old-fashioned handler for circular references before JSON.stringify on objects containing elements
- update supportjQuery to 3.7.1 - add debug option for browserstack - upgrade QUnit - upgrade sinon to version 9.x, which required changes to animation tests to account for 0 being falsey in the second argument. - browserstack can take around 3min to spin up safari on iOS 15
- replaced JSDOM grunt task with jsdom browser support in the task runner - ensured support for Firefox ESR in its own task - added task for running in Safari on macos, separated to allow other tasks to continue running on ubuntu, which is significantly faster. - improved job names - always log selenium driver's browser name and version
- improved gzipped size from 5-47 bytes depending on the file
c19df50
to
c029790
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I managed to only review the first commit so far but I'm posting it for review so that you don't need to wait for me to finish with the rest.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK, I managed to review the whole thing in the end; the first commit contained the majority of the work. Nice job! 🎉
For now I looked at the code, I haven't played with it locally yet.
Added back |
This reverts commit 5253ab9.
Passing BrowserStack run after all the changes: https://github.com/timmywil/jquery/actions/runs/8033218053 I'd like to get this in and iterate more afterwards as needed, unless there's anything else glaring that needs changing. Then I can start on a 3.x PR and add a workflow for updating the git versions. Then, we can finally decommission old Jenkins & Testswarm servers. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One last comment in #5418 (comment). Otherwise, LGTM. Great work!
This is a complete rework of our testing infrastructure. The main goal is to modernize and drop deprecated or undermaintained dependencies (specifically, grunt, karma, and testswarm). We've achieved that by limiting our dependency list to ones that are unlikely to drop support any time soon. The new dependency list includes: - `qunit` (our trusty unit testing library) - `selenium-webdriver` (for spinning up local browsers) - `express` (for starting a test server and adding middleware) - express middleware includes uses of `body-parser` and `raw-body` - `yargs` (for constructing a CLI with pretty help text) - BrowserStack (for running each of our QUnit modules separately in all of our supported browsers) - `browserstack-local` (for opening a local tunnel. This is the same package still currently used in the new Browserstack SDK) - We are not using any other BrowserStack library. The newest BrowserStack SDK does not fit our needs (and isn't open source). Existing libraries, such as `node-browserstack` or `browserstack-runner`, either do not quite fit our needs, are under-maintained and out-of-date, or are not robust enough to meet all of our requirements. We instead call the [BrowserStack REST API](https://github.com/browserstack/api) directly. - automatically retries individual modules in case of test failure(s) - automatically attempts to re-establish broken tunnels - automatically refreshes the page in case a test run has stalled - runs all browsers concurrently and uses as many sessions as are available under the BrowserStack plan. It will wait for available sessions if there are none. - supports filtering the available list of browsers by browser name, browser version, device, OS, and OS version (see `npm run test:unit -- --list-browsers` for more info). It will retrieve the latest matching browser available if any of those parameters are not specified. - cleans up after itself (closes the local tunnel, stops the test server, etc.) - Requires `BROWSERSTACK_USERNAME` and `BROWSERSTACK_ACCESS_KEY` environment variables. - supports running any local browser as long as the driver is installed, including support for headless mode in Chrome, FF, and Edge - supports running `basic` tests on the latest [jsdom](https://github.com/jsdom/jsdom#readme), which can be seen in action in this PR (see `test:browserless`) - Node tests will run as before in PRs and all non-dependabot branches, but now includes tests on real Safari in a GH actions macos image instead of playwright-webkit. - can run multiple browsers and multiple modules concurrently Other notes: - Stale dependencies have been removed and all remaining dependencies have been upgraded with a few exceptions: - `sinon`: stopped supporting IE in version 10. But, `sinon` has been updated to 9.x. - `husky`: latest does not support Node 10 and runs on `npm install`. Needed for now until git builds are migrated to GitHub Actions. - `rollup`: latest does not support Node 10. Needed for now until git builds are migrated to GitHub Actions. - BrowserStack tests are set to run on each `main` branch commit - `debug` mode leaves Selenium browsers open whether they pass or fail and leaves browsers with test failures open on BrowserStack. The latter is to avoid leaving open too many sessions. - This PR includes a workflow to dispatch BrowserStack runs on-demand - The Node version used for most workflow tests has been upgraded to 20.x - updated supportjQuery to 3.7.1 Run `npm run test:unit -- --help` for CLI documentation Close jquerygh-5418
Summary
This is a complete rework of our testing infrastructure. The main goal is to modernize and drop deprecated or undermaintained dependencies (specifically, grunt, karma, and testswarm). We've achieved that by limiting our dependency list to ones that are unlikely to drop support any time soon. The new dependency list includes:
qunit
(our trusty unit testing library)selenium-webdriver
(for spinning up local browsers)express
(for starting a test server and adding middleware)body-parser
andraw-body
yargs
(for constructing a CLI with pretty help text)browserstack-local
(for opening a local tunnel. This is the same package still currently used in the new Browserstack SDK)node-browserstack
orbrowserstack-runner
, either do not quite fit our needs, are under-maintained and out-of-date, or are not robust enough to meet all of our requirements. We instead call the BrowserStack REST API directly.BrowserStack Runner
npm run test:unit -- --list-browsers
for more info). It will retrieve the latest matching browser available if any of those parameters are not specified.BROWSERSTACK_USERNAME
andBROWSERSTACK_ACCESS_KEY
environment variables.Selenium Runner
basic
tests on the latest jsdom, which can be seen in action in this PR (seetest:browserless
)Other notes:
sinon
: stopped supporting IE in version 10. But,sinon
has been updated to 9.x.husky
: latest does not support Node 10 and runs onnpm install
. Needed for now until git builds are migrated to GitHub Actions.rollup
: latest does not support Node 10. Needed for now until git builds are migrated to GitHub Actions.main
branch commitdebug
mode leaves Selenium browsers open whether they pass or fail and leaves browsers with test failures open on BrowserStack. The latter is to avoid leaving open too many sessions.Run
npm run test:unit -- --help
for CLI documentation or view the command.js file directly.Checklist