Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for advanced test lifecycle callbacks - setUpAll #1751

Merged
merged 82 commits into from
Oct 25, 2023

Conversation

bartekpacia
Copy link
Contributor

Targeted at #1503.

See investigation conducted in #1741.

@github-actions github-actions bot added the package: patrol Related to the patrol package (native automation, test bundling) label Sep 28, 2023
@bartekpacia bartekpacia changed the base branch from develop to lifecycle_callbacks September 28, 2023 10:54
@bartekpacia bartekpacia changed the title Lifecycle callbacks - support setUpAll Add support for advanced lifecycle callbacks - setUpAll Oct 2, 2023
@bartekpacia bartekpacia changed the title Add support for advanced lifecycle callbacks - setUpAll Add support for advanced test lifecycle callbacks - setUpAll Oct 2, 2023
@bartekpacia bartekpacia force-pushed the lifecycle_callbacks_setupall branch from f879189 to a84464f Compare October 2, 2023 18:30
@bartekpacia
Copy link
Contributor Author

bartekpacia commented Oct 2, 2023

Patrol's test bundling feature offloads scheduling&execution of tests to package:test. How it works is that we always execute all the tests, but only the one is requested and runs, and the rest is skipped (i.e. their body is executed but exits early). This behavior implies being able to access currentTest from within testWidgets, setUp and tearDown body, so we can if currentTest == requestedTest and decide whether to skip or execute it.

However, currentTest is not available inside setUpAll and tearDownAll body callbacks, so this exact approach won't work.

I think a workaround based on using parentGroup should work. See gist.


currentTest = Invoker.current!.liveTest.individualName
parentGroup = Invoker.current!.liveTest.groups.last.name

@bartekpacia
Copy link
Contributor Author

bartekpacia commented Oct 9, 2023

When should the setUpAll callbacks be run? The obvious answer is "before its tests start running". This means that Dart side has to know which Dart setUpAll callbacks have already run, and which not.

This is a problem, because currently, the only synchronization point between instrumentation app and app under test is markPatrolAppServiceReady() and listDartTest(). Both of them happen too late, after all tests have been executed, after all setUpAlls have been called.


I made a playground where I'm testing how to bend package:test API to our use case.

@bartekpacia
Copy link
Contributor Author

bartekpacia commented Oct 10, 2023

Plan

To discover setUpAlls, all tests must run. This currently cannot be done during the inital run, because Dart tests only run after runDartTest() is called.

To fix that, behavior of Dart tests must differ between the initial run and normal runs. Currently this behavior is mostly the same:

  1. App under test starts
  2. patrol_test_explorer test runs, subsequent tests DO NOT RUN
  3. Once patrol_test_explorer finishes, instrumentation calls listDartTests()
  4. [Only in normal run] Once listDartTests() finishes, instrumentation calls runDartTest()

But to collect setUpAlls, this would have to happen:

  1. App under test starts
  2. patrol_test_explorer test runs, subsequent tests RUN but are SKIPPED
  3. Once patrol_test_explorer finishes, instrumentation calls listDartTests()
  4. [Only in initial run] Once listDartTests() finishes, instrumentation calls listDartLifecycleCallbacks()
  5. [Only in normal run] Once listDartTests() finishes, instrumentation calls runDartTest()

@github-actions github-actions bot added the package: patrol_cli Related to the patrol_cli package label Oct 10, 2023
@bartekpacia bartekpacia force-pushed the lifecycle_callbacks_setupall branch from 21d4862 to 9edac16 Compare October 10, 2023 21:24
@bartekpacia bartekpacia force-pushed the lifecycle_callbacks_setupall branch from 5b742a3 to 56dca3b Compare October 10, 2023 22:26
@bartekpacia bartekpacia force-pushed the lifecycle_callbacks_setupall branch from 86715a6 to d205d7a Compare October 24, 2023 18:56
@bartekpacia bartekpacia marked this pull request as ready for review October 24, 2023 19:33
@bartekpacia bartekpacia force-pushed the lifecycle_callbacks_setupall branch from 21aafa2 to b8f5bea Compare October 25, 2023 08:15
@bartekpacia bartekpacia force-pushed the lifecycle_callbacks_setupall branch from b8f5bea to 6a6615e Compare October 25, 2023 08:42
@bartekpacia bartekpacia merged commit c079ad8 into develop Oct 25, 2023
18 of 20 checks passed
@bartekpacia bartekpacia deleted the lifecycle_callbacks_setupall branch October 25, 2023 09:00
@bartekpacia bartekpacia mentioned this pull request Oct 25, 2023
12 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
package: patrol_cli Related to the patrol_cli package package: patrol Related to the patrol package (native automation, test bundling)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants