E2E Test API

[ivelin]

Transferring a discussion from the Brave public forum per @bbondy 's request.

https://community.brave.com/t/e2e-testing-for-web3-app-interactions-with-brave-wallet/318181/4

The Web3 Frontend Brittleness Problem

Many of today's web3 apps feel brittle, partially because there is no robust e2e test framework API officially supported by MetaMask.

There are frequent ongoing UI flow breaks and misplacement or misrepresentation of user's crypto funds due to subpar UX. This is true even for top web3 apps like Uniswap, Axie and Sandbox. As users navigate between different web3 apps and interact with multiple blockchain networks and wallets, these web3 apps often get out of sync with the state of the current wallet and as a result, the user ends up looking at unexpected content.

Given web3's promise to better protect user's identity, privacy and provide more control over user's own assets, it is imperative to offer superior, trustworthy UX. That in turn requires improving on test automation practices and infrastructure.

Current Workarounds

Test frameworks like dappeteer and synpress are trying to bridge the gap, but they feel brittle as they are constantly chasing MetaMask's changing front end code. This test framework brittleness frustrates web3 developers who are used to sophisticated web2 tooling. Writing web3 e2e tests that frequently break due to underlying base layer problems, results in postponing these test automation initiatives, which then results in brittle web3 frontend apps and subpar web3 UX.

The following snippet is an example of web3 e2e test frameworks "screen scraping" metamask code.

const firstTimeFlowFormPage = '.first-time-flow__form';
const secretWordsInput = `${firstTimeFlowFormPage} .first-time-flow__seedphrase input`;
const passwordInput = `${firstTimeFlowFormPage} #password`;
const confirmPasswordInput = `${firstTimeFlowFormPage} #confirm-password`;
const termsCheckbox = `${firstTimeFlowFormPage} .first-time-flow__terms`;
const importButton = `${firstTimeFlowFormPage} .first-time-flow__button`;
const newPasswordInput = `${firstTimeFlowFormPage} #create-password`;
const newSignupCheckbox = `${firstTimeFlowFormPage} .first-time-flow__checkbox`;

and another example:

  const addressInput = await page.waitForSelector('#custom-address');
  addressInput.type(tokenAddress);

  await page.waitForTimeout(4000);

  const nextButton = await page.waitForSelector(`button[data-testid='page-container-footer-next']:not([disabled])`);
  await nextButton.click();

  const footerButtons = await page.$$('footer > button');

While web3 app test developers are looking for reliable API that looks like this:

  const browser = await dappeteer.launch(puppeteer, { metamaskVersion: 'v10.1.1' });
  const metamask = await dappeteer.setupMetamask(browser);

  // you can change the network if you want
  await metamask.switchNetwork('ropsten');

  // you can import a token
  await metamask.addToken('0x4f96fe3b7a6cf9725f59d353f723c1bdb64ca6aa');

  // go to a dapp and do something that prompts MetaMask to confirm a transaction
  const page = await browser.newPage();
  await page.goto('http://my-dapp.com');
  const payButton = await page.$('#pay-with-eth');
  await payButton.click();

  // 🏌
  await metamask.confirmTransaction();

This MetaMask test-dapp repo could be a starting point for standardizing web3 e2e testing by documenting and providing test-spec examples for the interaction between its page components and the metamask wallet in various browsers through a cross browser e2e framework such as playwright.

This issue was also reported in MetaMask's repo:
MetaMask/test-dapp#147

[briareus-the-titan]

@bbondy something like this would really improve most web3 projects since getting e2e test working on most web3 projects is near impossible since interactions with web3 wallets are such an issue