Patrol 2.0 shortcomings

[bartekpacia]

In this issue, we gather a list of problems, bugs, and lacks related to the test bundling feature. This is mostly an extract of problems mentioned in this comment.

Individual issues might be created later.

---

• preserve the tree structure of Dart tests when translated to native tests (update: see #1502)

  Given the following hierarchy of the integration_test directory:

  integration_test
  ├── feature_bravo
  │   ├── 1_test.dart
  │   └── 2_test.dart
  └── feature_delta
      ├── 1_test.dart
      └── 2_test.dart
  

  the results are presented like this on a native platform (in the example below: Android:)

  test feature_bravo.1_test.dart ✅
  test feature_bravo.2_test.dart ✅
  test feature_delta.1_test.dart ✅
  test feature_delta.2_test.dart ✅
  

  but should be presented like this instead:

  group feature_bravo
  ├── test 1_test.dart ✅
  └── test 2_test.dart ✅
  group feature_delta
  ├── test 1_test.dart ✅
  └── test 2_test.dart ✅
  

  In other words, the hierarchy is removed, but it's emulated by . (a dot).

---

• group() cannot be used (update: see #1619)

  This stems from how tests are registered for execution by patrolTest().

  See also:

  • Related code with a comment containing more details.

---

• set-up and tear-down callbacks cannot be used (update: see #1503)

  Or rather: they can be used, but they execute in a very different order.
  This applies to setUp(), setUpAll(), tearDown(), tearDownAll(), and similar.

  Assume there are 3 tests in the integration_test director:

  Source code

  import 'common.dart';
  
  // integration_test/alpha_test.dart
  
  void main() {
    print('PATROL_DEBUG: alpha_test file loaded');
  
    setUp(() {
      print('PATROL_DEBUG: alpha_test setup');
    });
  
    tearDown(() {
      print('PATROL_DEBUG: alpha_test teardown');
    });
  
    patrolTest('alpha_test', nativeAutomation: true, ($) async {
      print('PATROL_DEBUG: alpha_test body');
      await $.pumpWidget(ExampleApp());
    });
  }

  import 'common.dart';
  
  // integration_test/bravo_test.dart
  
  void main() {
    print('PATROL_DEBUG: bravo_test file loaded');
  
    setUp(() {
      print('PATROL_DEBUG: bravo_test setup');
    });
  
    tearDown(() {
      print('PATROL_DEBUG: bravo_test teardown');
    });
  
    patrolTest('bravo_test', nativeAutomation: true, ($) async {
      print('PATROL_DEBUG: bravo_test body');
      await $.pumpWidget(ExampleApp());
    });
  }

  import 'common.dart';
  
  // integration_test/charlie_test.dart
  
  void main() {
    print('PATROL_DEBUG: charlie_test file loaded');
  
    setUp(() {
      print('PATROL_DEBUG: charlie_test setup');
    });
  
    tearDown(() {
      print('PATROL_DEBUG: charlie_test teardown');
    });
  
    patrolTest('charlie test', nativeAutomation: true, ($) async {
      print('PATROL_DEBUG: charlie_test body');
      await $.pumpWidget(ExampleApp());
    });
  }

  Here's how it'll end up being:

  Logs

  $ adb logcat -v color -d | grep -E 'PATROL_DEBUG' | cut -d ':' -f 4,5
   PATROL_DEBUG: alpha_test file loaded
   PATROL_DEBUG: bravo_test file loaded
   PATROL_DEBUG: charlie_test file loaded
   PATROL_DEBUG: alpha_test setup
   PATROL_DEBUG: alpha_test file loaded
   PATROL_DEBUG: bravo_test file loaded
   PATROL_DEBUG: charlie_test file loaded
   PATROL_DEBUG: alpha_test setup
   PATROL_DEBUG: alpha_test body
   PATROL_DEBUG: alpha_test teardown
   PATROL_DEBUG: bravo_test setup
   PATROL_DEBUG: alpha_test file loaded
   PATROL_DEBUG: bravo_test file loaded
   PATROL_DEBUG: charlie_test file loaded
   PATROL_DEBUG: alpha_test setup
   PATROL_DEBUG: alpha_test teardown
   PATROL_DEBUG: bravo_test setup
   PATROL_DEBUG: bravo_test body
   PATROL_DEBUG: bravo_test teardown
   PATROL_DEBUG: charlie_test setup
   PATROL_DEBUG: alpha_test file loaded
   PATROL_DEBUG: bravo_test file loaded
   PATROL_DEBUG: charlie_test file loaded
   PATROL_DEBUG: alpha_test setup
   PATROL_DEBUG: alpha_test teardown
   PATROL_DEBUG: bravo_test setup
   PATROL_DEBUG: bravo_test teardown
   PATROL_DEBUG: charlie_test setup
   PATROL_DEBUG: charlie_test body
   PATROL_DEBUG: charlie_test teardown

  This is because of the way we implemented running a particular test. Tests that appear before it aren't actually skipped – they are run but finish immediately. Think: fallthrough.

---

• The main() function must not be asynchronous.

  Example of incorrect test:

  void main() async {
    patrolTest(...);
  }

  Example of correct test:

  void main() {
    patrolTest(...);
  }

[mzdm]

Q: Test bundling is optional or it will be the only option in 2.0? setUp, tearDown etc. are quite crucial in our codebase and I am wondering what the workarounds will be

[bartekpacia]

A: It'll be the only way of running tests in 2.0.

[bartekpacia]

I am wondering what the workarounds will be

You can run set-ups manually at the top of the callback you pass to patrolTest().

Using tear-downs is not recommended since there's no guarantee they'll execute, due to e.g. fatal native crash.

I'm curios to hear what you think about this. I'd also like to bring back support for set-ups and tear-downs but it was decided not needed for 2.0.

[bartekpacia]

Please also note that MainActivityTest.java was changed in 2.0.0-dev.3, here's how it must look like now.

[findms]

@bartekpacia In my case I don't have this MainActivityTest.java should I add this file and code according to patrol_cli_2.0.0 documentation as getting some errors in test_bundle.dart and build.gradle on activating patrol_cli to 2.0.0

Is there any way to go back on patrol_cli_1.1.11?

[bartekpacia]

Sure, you can easily downgrade to v1:

dart pub global activate patrol_cli ^1.0.0

[jBorkowska]

There are separate issues for this topics, closing

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar problem, please file a new issue. Make sure to follow the template and provide all the information necessary to reproduce the issue.