test: improve logs for e2e errors #28479

seaona · 2024-11-15T07:56:56Z

Description

This PR adds an improvement on our logs when the errors do not have the expected form of a.value, leading to displaying empty errors and not failing the test. Those are happening for RPC and some snap errors types, which currently are displayed as empty (see below screenshots):

The RPC errors doesn't have a value property but a description, so we were seeing empty errors in the logs
In the snaps errors, the a.value property is not directly present, instead, the relevant information is nested within the preview.properties array
Other error structures, which doesn't fall under the 3 error categories, will be captured in a fallback, which will stringify the complete error

With this change we are now able to see better error logs in our e2e.

Related issues

Fixes: https://github.com/MetaMask/MetaMask-planning/issues/3648

Manual testing steps

Run a test which triggers an RPC error like: yarn test:e2e:single test/e2e/tests/request-queuing/ui.spec.js --browser=chrome
Check console errors before and after this change

Screenshots/Recordings

Before

See empty RPC error logs:

See empty snap error logs (the 1st one type is logged but the 2nd one is empty):

After

See complete RPC error logs

See complete snaps error logs

Pre-merge author checklist

I've followed MetaMask Contributor Docs and MetaMask Extension Coding Standards.
I've completed the PR template to the best of my ability
I’ve included tests if applicable
I’ve documented my code using JSDoc format if applicable
I’ve applied the right labels on the PR (see labeling guidelines). Not required for external contributors.

Pre-merge reviewer checklist

I've manually tested the PR (e.g. pull and build branch, run the app, test code being changed).
I confirm that this PR addresses all acceptance criteria described in the ticket it closes and includes the necessary testing evidence such as recordings and or screenshots.

github-actions · 2024-11-15T07:57:10Z

CLA Signature Action: All authors have signed the CLA. You may need to manually re-run the blocking PR check if it doesn't pass in a few minutes.

test/e2e/webdriver/driver.js

seaona · 2024-11-15T11:01:19Z

test/e2e/snaps/test-snap-metrics.spec.js

@@ -900,6 +900,7 @@ describe('Test Snap Metrics', function () {
        testSpecificMock: mockSegment,
        ignoredConsoleErrors: [
          'MetaMask - RPC Error: Failed to fetch snap "npm:@metamask/bip32-example-snap": Failed to fetch tarball for package "@metamask/bip32-example-snap"..',
+          'Failed to fetch snap "npm:@metamask/bip32-example-…ball for package "@metamask/bip32-example-snap"..',


this error is expected in this test. It was a blank empty error before (see screenshot in dscription) and didn't cause the test to fail. Now this is needed, as we are logging also this error message instead of an empty string and makes the test fail

seaona · 2024-11-15T11:01:46Z

test/e2e/tests/account/lockdown.spec.ts

@@ -86,6 +86,9 @@ describe('lockdown', function (this: Mocha.Suite) {
      {
        fixtures: new FixtureBuilder().build(),
        ganacheOptions,
+        ignoredConsoleErrors: [
+          'Error: Could not establish connection.',


this error is expected in this test. It was a blank empty error before and didn't cause the test to fail. Now this is needed, as we are logging also this error message instead of an empty string and makes the test fail

metamaskbot · 2024-11-15T12:01:44Z

Builds ready [d839f0e]

builds: chrome, firefox
builds (beta): chrome
builds (flask): chrome, firefox
builds (MMI): chrome, firefox
builds (test): chrome, firefox
builds (test-flask): chrome, firefox
build viz: Build System
mv3: Background Module Init Stats
mv3: UI Init Stats
mv3: Module Load Stats
mv3: Bundle Size Stats
mv2: E2e Actions Stats
code coverage: Report
storybook: Storybook
typescript migration: Dashboard
all artifacts
bundle viz:
- background: 0, 1, 2, 3, 4, 5, 6, 7
- common: 0, 1, 10, 11, 12, 2, 3, 4, 5, 6, 7, 8, 9
- content-script: 0
- offscreen: 0
- ui: 0, 1, 10, 11, 12, 2, 3, 4, 5, 6, 7, 8, 9

Page Load Metrics (1766 ± 55 ms)

Platform	Page	Metric	Min (ms)	Max (ms)	Average (ms)	StandardDeviation (ms)	MarginOfError (ms)
Chrome	Home	firstPaint	294	2008	1705	337	162
		domContentLoaded	1545	1934	1740	98	47
		load	1552	2015	1766	114	55
		domInteractive	28	245	48	46	22
		backgroundConnect	10	79	23	17	8
		firstReactRender	54	296	124	58	28
		getState	4	74	19	22	10
		initialActions	0	0	0	0	0
		loadScripts	1126	1438	1274	79	38
		setupStore	5	74	17	21	10
		uiStartup	1745	2271	2026	140	67

Bundle size diffs

background: 0 Bytes (0.00%)
ui: 0 Bytes (0.00%)
common: 0 Bytes (0.00%)

seaona · 2024-11-15T12:05:00Z

test/e2e/tests/account/lockdown.spec.ts

@@ -86,6 +86,7 @@ describe('lockdown', function (this: Mocha.Suite) {
      {
        fixtures: new FixtureBuilder().build(),
        ganacheOptions,
+        ignoredConsoleErrors: ['Error: Could not establish connection.'],


this error is expected in this test. It was a blank empty error before (see screenshot in dscription) and didn't cause the test to fail. Now this is needed, as we are logging also this error message instead of an empty string and makes the test fail

chloeYue

LGTM

console errors fix

4bfb553

seaona requested review from a team as code owners November 15, 2024 07:56

seaona self-assigned this Nov 15, 2024

seaona added the area-qa Relating to QA work (Quality Assurance) label Nov 15, 2024

seaona added the flaky tests label Nov 15, 2024

metamaskbot added the team-extension-platform label Nov 15, 2024

seaona changed the title ~~tests: improve logs for RPC errors and remove old references~~ test: improve logs for RPC errors and remove old references Nov 15, 2024

add snap error log type

c958fd8

seaona commented Nov 15, 2024

View reviewed changes

test/e2e/webdriver/driver.js Outdated Show resolved Hide resolved

seaona and others added 5 commits November 15, 2024 10:00

Update test/e2e/webdriver/driver.js

77c7230

add new snaps errorr ignore

651abcb

remov eonly

7f4a8a6

fix lint

3586284

could not establish

748a167

seaona commented Nov 15, 2024

View reviewed changes

seaona added 2 commits November 15, 2024 12:04

fallback

759e47e

lint fix

d839f0e

seaona changed the title ~~test: improve logs for RPC errors and remove old references~~ test: improve logs for e2e errors Nov 15, 2024

seaona commented Nov 15, 2024

View reviewed changes

chloeYue approved these changes Nov 15, 2024

View reviewed changes

DDDDDanica approved these changes Nov 15, 2024

View reviewed changes

seaona enabled auto-merge November 15, 2024 16:15

FrederikBolding approved these changes Nov 19, 2024

View reviewed changes

seaona added this pull request to the merge queue Nov 19, 2024

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Nov 19, 2024

seaona added this pull request to the merge queue Nov 19, 2024

Merged via the queue into develop with commit 3c51786 Nov 19, 2024
89 checks passed

seaona deleted the test-chore-errors-console branch November 19, 2024 10:55

github-actions bot locked and limited conversation to collaborators Nov 19, 2024

metamaskbot added the release-12.9.0 Issue or pull request that will be included in release 12.9.0 label Nov 19, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test: improve logs for e2e errors #28479

test: improve logs for e2e errors #28479

seaona commented Nov 15, 2024 •

edited

Loading

github-actions bot commented Nov 15, 2024

seaona Nov 15, 2024 •

edited

Loading

seaona Nov 15, 2024

metamaskbot commented Nov 15, 2024

seaona Nov 15, 2024

chloeYue left a comment

test: improve logs for e2e errors #28479

test: improve logs for e2e errors #28479

Conversation

seaona commented Nov 15, 2024 • edited Loading

Description

Related issues

Manual testing steps

Screenshots/Recordings

Before

After

Pre-merge author checklist

Pre-merge reviewer checklist

github-actions bot commented Nov 15, 2024

seaona Nov 15, 2024 • edited Loading

Choose a reason for hiding this comment

seaona Nov 15, 2024

Choose a reason for hiding this comment

metamaskbot commented Nov 15, 2024

seaona Nov 15, 2024

Choose a reason for hiding this comment

chloeYue left a comment

Choose a reason for hiding this comment

seaona commented Nov 15, 2024 •

edited

Loading

seaona Nov 15, 2024 •

edited

Loading