Improve CI workflows to use even more test bundling advantages

[fylyppo]

Overview

As we're getting soon to fully implement test bundling and #1240 is done, I want to improve our CI workflows to:

• Have test sharding (running tests in parallel) for iOS simulators (low priority):

  I tested bluepill but I found two issues:

  • I need to set location on all simulators before running tests, because I don't know which simulator will receive the permission_location_test (it requires location to be set, otherwise it fails)
  • I need to standarize test outputs - after tests with bluepill I couldn't find XCRESULT file which I provide to job summary as a MD report. There were some JUnit XML and HTML files but they don't fit Action for detailed report I use right now. Check here example JUnit XML parsed by Test Reporter. Maybe if I exclude RunnerTests.xctest this table will work, Idk.

---

• Have test sharding for Android devices in Firebase Test Lab (mid priority):

  • I can try Flank
  • Still remember about standarizing test outputs. They provide HTML report and JUnit XML report

---

• Manage signing for Xcode release builds (we need release mode to run tests on FTL) (high priority):

  There are few solutions, examples:

  • Fastlane solution to automatically manage signings
  • Github manual solution

---

• Have test sharding for iOS devices in Firebase Test Lab (mid priority):

  Firstly, I need to configure code signing

  • I can use also Flank so test outputs could be more standarized as for Android devices

---

• Upload artifact per shard and set artifact name to one that points out what tests have been run in specific artifact (mid/high priority):

  Firstly, configure test sharding for all workflows

  • FTL and emulator.wtf provide test outputs with dirs called like "0,1,2,3,4,[...]" so I don't know which tests are in which directory
  • It'll be handy if I somehow name artifacts by test names which are included in specific shard. Then I can upload artifact per shard

---

• Improve job summaries so as to have good-looking, detailed and similar for both platforms (Android, iOS) markdown summaries (high priority):

  Firstly, I have to standarize (if possible) all test outputs (from each workflow) then I can think of improving test reports which are provided to job summary as MD**

  • It'll be good to have the same format for table with results for every workflow. The idea is to have parser for XCRESULT and JUnit XML to one MD format. Then I can provide this report to run summary
  • Maybe I can provide links to videos from each test?

---

One obvious thing to do:

• Remove Github Actions job or step duplication
  • With this doc

---

And I have also one tough nut to crack in our minds (but it could be done in the further future):

• Speed up Xcode builds:
  • I could investigate solution from this article https://dev.sweatco.in/speeding-up-xcode-archive-builds/

[bartekpacia]

@fylyppo What's the status here?

[bartekpacia]

@fylyppo What's the status here?