Failing UI tests can now be retried #71
Conversation
…ub.com/lyndsey-ferguson/fastlane-plugin-test_center * Added new (optional) environment variable `MULTI_SCAN_TRY_COUNT` to set the number of times a test is retried before it's considered a fail. * `TAB_UI_TEST_DEVICES` now supports the setting of multiple devices, as the name implies. If setting multiple, these should be ':' separated. * Added new (optional) environment variable `MULTI_SCAN_PARALLEL_WORKER_COUNT` to set the number of simulators for batched UI tests. * Added new (optional) environment variable `TAB_UNIT_TEST_DEVICE` to be able to set a unit test device independently. The `device` scan parameter was used previously which conflicted with the use of `devices` by `multi-scan`. * Updated `changelog` for version 6.0.0 changes. * Updated `readme` for version 6.0.0 changes.
|
Hey! Thanks for taking the time to contribute. This seems like something that could be added locally to your fastfile, rather than complicating the MasterFastfile? Ideally this would be great to raise as an Issue first so we can have a discussion around it, which also means less wasted time on your part if we decide not to go ahead with these changes. In a nutshell I'm not sure if we want to add another dependency to our shared MasterFastfile, we use this across multiple projects here at TAB and every dependency needs to be signed off individually by the owners of each project. |
|
Hey @KaneCheshire thanks for the feedback. The reason for adding to the MasterFastfile was just that's where the The motivations behind making this change were as a result of flaky UI tests since migrating to Xcode 11 with parallel UI tests. Therefore I imagined this might be of value across other TAB projects and wasn't too keen on forking etc. Whether we merge the change or not I'd have still needed to make the same changes, albeit left on a fork so no worries about the time part ;) I'm more than happy to raise an issue for this though. With those points in mind, let me know. Thanks |
|
That's kind of the point of the MasterFastfile though, it comes with some basic lanes that anyone can use without many dependencies and then extend it as each project needs individually. We've been on a bit of a drive to simplify the MasterFastfile recently, I'm not sure this is in line with that really. If it was a case of you couldn't have this functionality at all without it being part of the MasterFastfile (or if something was hugely broken with the current setup) then it would make sense to add it here, but since this is just something you need because you've decided to move to Xcode 11 (not something I'd advise with all the issues!) then it's something your project will just need to handle locally. Internally at TAB we're advising people not to move to Xcode 11 wherever possible because there's just been too many issues, this is just one example but there's plenty of others like the recent issue with Xcode 11.2 crashing anything older than iOS 13.2 using UITextViews in a storyboard. Instead I'd recommend filing as many bug reports with Apple as you can (like we are here) and then wait for Apple to address them. |
|
I take your point about the dependencies, I guess on that basis the MS App Center shouldn't be in there either 🤷♂️ Technically all functionality is achievable without the Masterfastfile, so on that basis nothing would be in there :) I understand the desire behind simplifying it though. For us moving over to Xcode 11 isn't really an option as Apple will enforce new submissions be done with 11 in April next year and we've not got capacity early next year to be doing the migration then. Also, waiting on Apple in the hopes they improve the situation with running of parallel tests doesn't feel like a risk we can take. However, the |
… leading to `TAB_REPORT_FORMATS` being set (overridden) to `html`. This resulted in log warning during a UI test run due `junit` not being set
Gym can now be configured for either a `release` or `debug` build
* Added `FL_USE_AUTOMATIC_CODE_SIGNING` which can be used to optionally control automatic/manual code signing.
…ode_signing Code signing type (auto/manual) can now be configured
…being able to set additional build arguments - https://docs.fastlane.tools/actions/build_app/
…g_support Added support for settings of gym `xcargs` parameter for purposes of …
Add intermediate logging for each attempt of multi_scan run.
fastlane-plugin-test_centerfor UI tests - https://github.com/lyndsey-ferguson/fastlane-plugin-test_centerMULTI_SCAN_TRY_COUNTto set the number of times a test is retried before it's considered a fail.TAB_UI_TEST_DEVICESnow supports the setting of multiple devices, as the name implies. If setting multiple, these should be ':' separated.MULTI_SCAN_PARALLEL_WORKER_COUNTto set the number of simulators for batched UI tests.TAB_UNIT_TEST_DEVICEto be able to set a unit test device independently. Thedevicescan parameter was used previously which conflicted with the use ofdevicesbymulti-scan.changelogfor version 6.0.0 changes.readmefor version 6.0.0 changes.