🧪 Considerations for Integration and End-to-End Testing Strategies

[JGreenlee]

e-mission/e-mission-phone#1040 (comment) suggests that since Cordova plugins are able to be tested, we should be able to use them in our UI tests.
I have found documentation (https://github.com/apache/cordova-plugin-test-framework) for how to write tests for Cordova plugins (ie. unit testing the Cordova plugins themselves, independent of e-mission-phone), but I have not found any way to use the plugins in a test for the project (ie. integration testing of how plugins are used in e-mission-phone.)

Appium has been suggested as an E2E testing tool for Cordova projects, allowing testing of the app in a context where the Cordova plugins are defined. Appium can also be used for React Native projects, so it may be a promising long-term option.
An example Cordova project with Appium testing is here: https://github.com/asialgearoid/cordova-appium-example, but it's unclear if this actually provides the plugins or is just for testing the content inside the WebView.

[jiji14]

I will search more about Appium and share what I've found.

[jiji14]

Cordova-plugin-test-framework vs Appium

What I've found is that Cordova-plugin-test-framework and Appium serve different purposes. Cordova-plugin-test-framework is for integration testing, while Appium is for end-to-end testing.

You can read more about about the difference between integration testing and end-to-end testing in this blog post:
https://www.onpathtesting.com/blog/end-to-end-vs-integration-testing

We need to decide which one is more suitable for our scenario.

Appium Requirements

Appium can be used for hybrid apps (including Cordova project) and our project meets the requirements for the testing environment.
You can check the requirements here: https://appium.io/docs/en/2.1/intro/requirements/

How Appium works

1. Appium server developed using Node.js implements selenium WebDriver.
2. The connection request is sent by WebDriver script to Appium server via JSON Wire protocol.
3. Automation session and desired capabilities are set on an Android / IOS mobile device or emulator
   (One advantage of Appium is that it allows testing on both emulators and real devices.)
4. After execution of the request sent, the message is sent back to the Appium server, and we can check the logs for the testing results.

Appium workflow

@shankari @JGreenlee

Based on my research, setting up Appium can be somewhat complex, but it appears to be the best choice for end-to-end testing of hybrid apps. If you believe it's worth trying, I can set up an Appium server locally and run some basic tests."

[JGreenlee]

Based on what Jiji found about Appium, I have been experimenting with integrating Appium and I want to give an update on what I've managed.

Although Appium itself is well-documented, there is not much on using it for Cordova apps. The one example I did find, https://medium.com/the-web-tub/testing-cordova-apps-with-appium-2b3b236e026b (and the corresponding repo https://github.com/asialgearoid/cordova-appium-example) is very outdated and I don't think it would even work with newer versions of Cordova, Node 19, modern OS versions, etc.

However, I think these tools will still be compatible if I just find out how to use the newer versions. To my understanding, the above guide leverages Appium to interface with an emulator or physical device, but it needs a different library, WebdriverIO, to interface with the underlying Webview (where all our JS runs).
The tests are written in Jasmine (similar syntax to Jest) and we should be able to query elements, simulate clicking them, and eventually map out the crucial interaction flows we want to test.

Because there is a lack of documentation for doing this on Cordova apps, it has taken a lot of trial and error to get this configured. Luckily I found this example project https://github.com/webdriverio/appium-boilerplate, which is more up-to-date and describes how to use WebdriverIO 7 with Appium 1.22.
I think I am close to being able to run tests on a physical Android device. After that, the next challenge will be to run them in an emulator by terminal (so that we can eventually do this testing in Github actions).

[shankari]

@JGreenlee @jiji14 all this is super cool, and I'm really looking forward to be able to run end-to-end testing.

I have a couple of high level questions on the setup, since I haven't poked around with Appium yet:

• given that Appium is an Apache project, I assume you can run the server locally - I'm a bit confused why there are two Appium servers in the diagram above
• how does the Appium server communicate with the android and iOS devices? Is there a separate appium app that we need to install? How does the appium app communicate with our app? In general, apps on phones run in sandboxes and cannot communicate with each other.
  • I want to make sure that the appium framework will be able to use our plugins. A lot of other frameworks, such as Expo, come with a "pre-built" app that includes common plugins like camera and auth. When we use expo, we will need to use a development build instead of Expo Go. Just wanted to check that there is a similar approach available for appium.

wrt the cordova test framework, https://github.com/apache/cordova-plugin-test-framework, while it is primarily used for integration testing, it allows you to make calls to native code from test javascript. I don't see any reason why it cannot be used for integration testing, and even end-to-end testing.

To answer the questions in the call today:

• it looks like you add a plugin which creates a cdvtests/index.html file
• you write tests in jasmine format
• when the file is launched, it runs all the tests
• if you include the file in your app, you can also run all the tests in the app

Then your app could even embed a link to the test page (cdvtests/index.html) from a help section of your app, to give end users a way to run your test suite out in the field. That may help diagnose causes of issues within your app. Maybe.

Here's an example of how a plugin using the plugin-test-framework runs the tests in the emulator.
https://github.com/apache/cordova-plugin-device/blob/master/.github/workflows/android.yml

It uses paramedic,
https://github.com/apache/cordova-paramedic

which basically automates the steps required for testing
https://kerrishotts.github.io/pgday/workshops/2017/campp/testing.html#cordova-paramedic

It seems like as long as we can launch the app in the emulator, we can write tests (either in the jasmine format, or potentially even in jest) that can just call the plugins directly instead of calling their mocks.

If we wanted to go down this route, I would suggest:

• trying to use paramedic with the device plugin to see what the cdvtests/index.html looks like
• try to make similar calls from jest/jasmine test code in our app

Let me know what you find!

[JGreenlee]

• how does the Appium server communicate with the android and iOS devices? Is there a separate appium app that we need to install? How does the appium app communicate with our app? In general, apps on phones run in sandboxes and cannot communicate with each other.

So far I've only tried it on Android. I have observed that it uses adb behind the scenes to install an app called "Appium Settings", as well as the app being tested. You can see both of these being opened and manipulated in the foreground during runtime.

• I want to make sure that the appium framework will be able to use our plugins. A lot of other frameworks, such as Expo, come with a "pre-built" app that includes common plugins like camera and auth. When we use expo, we will need to use a development build instead of Expo Go. Just wanted to check that there is a similar approach available for appium.

Yeah it installs the entire built app, plugins and all

[jiji14]

@shankari

Apologies for the confusion. The one on the very left is the Appium Client, not the Appium Server. The Appium Client sends REST API requests, which are derived from user written commands. And theses requests are sent to the server using the Mobile Json Wire Protocol.

Here is a better flowchart for iOS testing.

And yes, you can run the appium server locally.

This is how server and IOS communicate:

1. Appium server forward requests to the target iOS device/simulator.
2. These commands are interpreted by WebDriverAgent.app which converts them into mobile understandable format by calling Apple’s XCUITest API.
3. The commands are performed on the device/simulator.
4. Device/simulator then reverts the outcome of the performed command to the Appium server via WebDriverAgent.app.

Appium Architecture blog

[shankari]

@jiji14 where does the Appium Server live? Does it run locally on your laptop or in the cloud?
Our code is open source, so I don't care that much if it is in the cloud, but what happens if it goes away?

[jiji14]

@shankari
So far, I have only run the server locally.

Here is the video for testing Appium. I launched my emulator, ran the local server, and then executed the script.
For the test, I created a script to connect to the Appium server (in this case, my local server), open the app, and write the server address.

AppiumDriver driver = new IOSDriver(new URL("http://0.0.0.0:4723"), options);

I'm not sure what "what happens if it goes away?" means exactly, but the script fails after the server stops.
I believe we can set up the server in the cloud and run the script regularly.

test-video3.mov

[shankari]

I'm not sure what "what happens if it goes away?" means exactly, but the script fails after the server stops.

If it were only running on the cloud as a commercial product (like PhoneGap build), then I would worry about what would happen if the company went out of business and the cloud service went away. But given that you are running this on your laptop, I think it is not an issue 😄

[jiji14]

I'm not sure what "what happens if it goes away?" means exactly, but the script fails after the server stops.

If it were only running on the cloud as a commercial product (like PhoneGap build), then I would worry about what would happen if the company went out of business and the cloud service went away. But given that you are running this on your laptop, I think it is not an issue 😄

Oh Thanks for clarifying! Now I understand what you meant 😃