Failing UI tests can now be retried

CHANGELOG.md

@@ -1,20 +1,25 @@
 # CHANGELOG
 
+## [6.0.0](https://github.com/theappbusiness/MasterFastfile/releases/tag/6.0.0)
+
+- UI tests are now executed using `fastlane-plugin-test_center` - https://github.com/lyndsey-ferguson/fastlane-plugin-test_center.
+- Added new (optional) environment variable `MULTI_SCAN_TRY_COUNT` to set the number of times a test is retried before it's considered a fail.
+- `TAB_UI_TEST_DEVICES` now supports the setting of multiple devices, as the name implies. If setting multiple, these should be ':' separated.
+- Added new (optional) environment variable `MULTI_SCAN_PARALLEL_WORKER_COUNT` to set the number of simulators for batched UI tests.
+- Added new (optional) environment variable `TAB_UNIT_TEST_DEVICE` to be able to set a unit test device independently. The `device` scan parameter was used previously which conflicted with the use of `devices` by `multi-scan`.
+
 ## [5.0.1](https://github.com/theappbusiness/MasterFastfile/releases/tag/5.0.1)
 
-Fixes an issue that was causing hockey uploads to fail.
+- Fixes an issue that was causing hockey uploads to fail.
 
 ## [5.0.0](https://github.com/theappbusiness/MasterFastfile/releases/tag/5.0.0)
 
-`TAB_PRIMARY_TARGET` environment variable is now mandatory for projects using Icon Overlay.
-
-As of v 2.86.0, fastlane requires a target for get_version_number, used for icon badging.
+- `TAB_PRIMARY_TARGET` environment variable is now mandatory for projects using Icon Overlay.
 
+- As of v 2.86.0, fastlane requires a target for get_version_number, used for icon badging.
 See discussion here: fastlane/fastlane#12177. See PR #64 for details.
-
-Use intended Bundle ID environment setting for each action
-
-Distinguishing between `FL_UPDATE_PLIST_APP_IDENTIFIER` and `FL_UPDATE_APP_IDENTIFIER`. See PR #63 for details.
+- Use intended Bundle ID environment setting for each action
+- Distinguishing between `FL_UPDATE_PLIST_APP_IDENTIFIER` and `FL_UPDATE_APP_IDENTIFIER`. See PR #63 for details.
 
 ## [4.0.0](https://github.com/theappbusiness/MasterFastfile/releases/tag/4.0.0)
 ### Changed

Fastfile

@@ -14,18 +14,47 @@ desc 'It\'s not recommended to include UI tests in this scheme, instead run the
 lane :test do
   _setup
   skip_slack = ENV['SCAN_SLACK_CHANNEL'].to_s.strip.empty?
-  scan(skip_slack: skip_slack)
+  scan(
+    skip_slack: skip_slack,
+    devices: ENV['TAB_UNIT_TEST_DEVICE'] || ['iPhone 8']
+  )
 end
 
 desc 'Runs UI tests that are included in the scheme.'
 desc 'Environment variables to use: `TAB_UI_TEST_DEVICES`, `TAB_REPORT_FORMATS`, `TAB_UI_TEST_SCHEME`'
 lane :ui_test do
   _setup
   skip_slack = ENV['SCAN_SLACK_CHANNEL'].to_s.strip.empty?
-  scan(skip_slack: skip_slack,
-       devices: ENV['TAB_UI_TEST_DEVICES'],
-       output_types: ENV['TAB_REPORT_FORMATS'],
-       scheme: ENV['TAB_UI_TEST_SCHEME'])
+
+  max_test_try_count = (ENV['MULTI_SCAN_TRY_COUNT'] || "1").to_i
+  test_run_block = lambda do |testrun_info|
+    failed_test_count = testrun_info[:failed].size
+
+    if failed_test_count > 0
+      try_attempt = testrun_info[:try_count]
+      if try_attempt < max_test_try_count
+        UI.header('Not all UI tests have passed. Attempting re-run of failed...')
+      else
+        UI.important('Not all UI tests have passed and the maximum try count has been reached.')
+      end
+    end
+    UI.message("Attempt #{testrun_info[:try_count]} out of #{max_test_try_count} run info: #{testrun_info}")
+
+  end
+
+  result = multi_scan(
+    skip_slack: skip_slack,
+    devices: ENV['TAB_UI_TEST_DEVICES'].split(':'),
+    output_types: ENV['TAB_REPORT_FORMATS'],
+    scheme: ENV['TAB_UI_TEST_SCHEME'],
+    try_count: max_test_try_count,
+    fail_build: true,
+    testrun_completed_block: test_run_block,
+    parallel_testrun_count: (ENV['MULTI_SCAN_PARALLEL_WORKER_COUNT'] || "4").to_i
+  )
+  unless result[:failed_testcount].zero?
+    UI.message("There are #{result[:failed_testcount]} legitimate failing tests")
+  end
 end
 
 desc 'Runs all unit tests before deploying to App Center.'
@@ -75,12 +104,11 @@ end
 
 def _setup
   ENV['SCAN_SCHEME'] = ENV['GYM_SCHEME']
-  ENV['SCAN_DEVICE'] ||= 'iPhone 6 (12.0)'
   xcode_select(ENV['TAB_XCODE_PATH']) if is_ci && !ENV['TAB_XCODE_PATH'].nil?
 
   unless ENV['TAB_UI_TEST_SCHEME'].nil? # rubocop:disable Style/GuardClause
-    ENV['TAB_REPORT_FORMATS'] = 'html' if ENV['TAB_OUTPUT_TYPES'].nil?
-    ENV['TAB_UI_TEST_DEVICES'] ||= 'iPhone 8'
+    ENV['TAB_REPORT_FORMATS'] = 'html' if ENV['TAB_REPORT_FORMATS'].nil?
+    ENV['TAB_UI_TEST_DEVICES'] ||= ['iPhone 8']
   end
 end
 
@@ -119,10 +147,21 @@ end
 def _build_with_gym
   install_provisioning_profiles
   _update_team_id_if_necessary
+  _update_code_signing_type_if_necessary
   export_method = _get_export_method
   xcconfig_filename = Dir.pwd + '/TAB.release.xcconfig'
   create_xcconfig(filename: xcconfig_filename)
-  gym(export_method: export_method, xcconfig: xcconfig_filename)
+
+  gym(
+    configuration: _get_build_config,
+    export_method: export_method,
+    xcconfig: xcconfig_filename,
+    xcargs: ENV['GYM_XCARGS'] || ''
+  )
+end
+
+def _get_build_config
+  return ENV['FL_BUILD_CONFIGURATION'].nil? ? "Release" : ENV['FL_BUILD_CONFIGURATION']
 end
 
 def _get_export_method
@@ -149,6 +188,16 @@ def _update_team_id_if_necessary
   end
 end
 
+def _update_code_signing_type_if_necessary
+  automatic_code_signing = ENV['FL_USE_AUTOMATIC_CODE_SIGNING']
+  if automatic_code_signing
+    UI.message("Updating code signing to '#{automatic_code_signing == 'true' ? 'automatic' : 'manual'}'")
+    update_code_signing_settings(
+      use_automatic_signing: automatic_code_signing == 'true',
+    )
+  end
+end
+
 def _get_team_id
   team_id = ENV['FL_PROJECT_TEAM_ID']
   unless team_id

README.md

@@ -84,6 +84,9 @@ The MasterFastfile uses whichever export options you have set for the `GYM_EXPOR
 
 ## Dependencies
 
+### Test Center plugin
+Add the test center plugin to your fastlane setup for the `ui_test` lane. Follow the guide [here](https://github.com/lyndsey-ferguson/fastlane-plugin-test_center)
+
 ### App Center plugin
 Add the app center plugin to your fastlane setup for the app center lanes. Follow the guide [here](https://github.com/microsoft/fastlane-plugin-appcenter)