[test:e2e] Check if playwright offers a quicker/more reliable framework for web3 testing

[0xJem]

No description provided.

[ivelin]

Docs on using playwright with chrome extensions:
https://playwright.dev/docs/chrome-extensions

Online monitoring service for playwritght and puppeteer in case we want to consider running our e2e tests periodically from multiple locations around the world:
https://www.checklyhq.com/

[0xJem]

@ivelin Were you able to test whether playwright and metamask would play nicely together?

[ivelin]

@ivelin Were you able to test whether playwright and metamask would play nicely together?

@0xJem Not yet. Marking this as a Good First Issue.

BTW, I am waiting on feedback for this test/base PR:
#1026

[ivelin]

@0xJem FYI, playing with this Playwright setup for Gitpod today:
https://github.com/4DvAnCeBoY/pw-test

[ivelin]

0xJem

UPDATE: I found a gitpod image that works well with both puppeteer and playwright. Also includes a debug tool for both:
https://github.com/testimio/root-cause-interactive-demo

When it comes to MetaMask, I think we have two options with pros and cons:

1. Use Puppeteer with dappeteer on chome only. Lock in MetaMask version to avoid brittleness. Downside here is lack of multi-browser testing.
2. Use Playwright and stub-out/mock wallet interactions. This will allow us to test on multiple browser targets. Downside is we can't replay e2e user paths through wallet interactions.

Since we use the WalletConnect API as a wallet connector on mobile apps and direct MetaMask conncetion on Desktop, it might be better to work with wallet abstractions and mockups for now until we get a stable e2e API from WalletConnect and/or MetaMask. Looks like both projects are interested in working on this in 2022.

Thoughts?

[0xJem]

Puppeteer supports Firefox, Chromium and Chrome as testing targets, by the way. But that does exclude others.

If we can abstract the wallet client out and have specific tests to ensure that wallet X works, I think that would be ideal. I assume that mocking the WalletConnect API is non-trivial, so is this something that we can tackle now?

Assuming the mocking is trivial, how much easier is it to use playwright?

[ivelin]

Puppeteer supports Firefox, Chromium and Chrome as testing targets, by the way. But that does exclude others.

Correct. Does not support webkit based browsers like Safari. Playwright does.

If we can abstract the wallet client out and have specific tests to ensure that wallet X works, I think that would be ideal. I assume that mocking the WalletConnect API is non-trivial, so is this something that we can tackle now?

Yes, I agree. Even though wallet vendors responded positively to the idea, none committed to an action plan so far. Stubbing out the WalletConnect API seems like our best bet for the time being. I believe its possible and am happy to work on that.

Assuming the mocking is trivial, how much easier is it to use playwright?

Most of my experience is with cypress. From what I can tell, puppeteer and playwright are very similar and shouldn't be too hard to use either.

Related note:
We just got yarn test:unit in develop. I would suggest we get unit test coverage to 80% or above before we move on to e2e tests. Otherwise I am concerned we will end up with several dangling test branches for awhile longer.

[ivelin]

Just for historical context. Listing below the ongoing threads with wallet providers for E2E Test APIs:

• Metamask: Wallet E2E Test API MetaMask/test-dapp#147
• Brave: https://community.brave.com/t/e2e-testing-for-web3-app-interactions-with-brave-wallet/318181
• WalletConnect: E2E Test API WalletConnect/walletconnect-monorepo#683
• WalletLink: e2e test API? coinbase/coinbase-wallet-sdk#256
• Safari Wallet: https://ricburton.substack.com/p/what-the-browser-wars-can-teach-us/comment/4326563

[0xJem]

We just got yarn test:unit in develop. I would suggest we get unit test coverage to 80% or above before we move on to e2e tests. Otherwise I am concerned we will end up with several dangling test branches for awhile longer.

That's fair. I would like to get the work I'm doing with the v2 contracts deployment and docker done (#655 ), even if we don't populate the e2e tests immediately afterwards, as it will be difficult to context-switch back.