Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf: add Sentry tracing to CircleCI runs #26588

Merged
merged 15 commits into from
Sep 11, 2024
Merged

Conversation

HowardBraham
Copy link
Contributor

@HowardBraham HowardBraham commented Aug 21, 2024

Description

This will allow CircleCI runs to report information to Sentry. It was very difficult to achieve, because there were specific E2E tests in test/e2e/tests/metrics/errors.spec.js that could never pass with this code added.

Before this PR, there was no way to pass runtime flags to the extension during tests. (Well okay, more precisely, you could change certain things through Fixtures, and you could change certain things through Ganache parameters, but both of these didn't load until late in the app load. Sentry is one of the first things to load, and it needed a way to get runtime flags very early in the app load.)

The way this works is a bit of a hack, but it's the only solution we could come up with that would load early enough:

  1. A global beforeEach hook in the E2E tests backs up the manifest.json file
  2. The helpers.withFixtures() function calls setManifestFlags(), which reads in the manifest file and parses it
  3. We alter the manifest with CircleCI environment variables and custom flags, then write it back to manifest.json
  4. The test runs, and the built extension can call getManifestFlags() to get the custom flags
  5. A global afterEach hook restores the backup copy of the manifest, so that the next test gets the normal manifest

Open in GitHub Codespaces

Related issues

Manual testing steps

Screenshots/Recordings

Before

After

Pre-merge author checklist

Pre-merge reviewer checklist

  • I've manually tested the PR (e.g. pull and build branch, run the app, test code being changed).
  • I confirm that this PR addresses all acceptance criteria described in the ticket it closes and includes the necessary testing evidence such as recordings and or screenshots.

@HowardBraham HowardBraham added the team-tiger Tiger team (for tech debt reduction + performance improvements) label Aug 21, 2024
@HowardBraham HowardBraham self-assigned this Aug 21, 2024
Copy link
Contributor

CLA Signature Action: All authors have signed the CLA. You may need to manually re-run the blocking PR check if it doesn't pass in a few minutes.

@HowardBraham HowardBraham force-pushed the tiger-sentry-circleci branch 2 times, most recently from d23477f to cb0212d Compare August 22, 2024 02:19
"env:e2e": "SEGMENT_HOST='https://api.segment.io' SEGMENT_WRITE_KEY='FAKE' SENTRY_DSN_DEV=https://[email protected]/0000000 yarn",
"env:e2e": "SEGMENT_HOST='https://api.segment.io' SEGMENT_WRITE_KEY='FAKE' yarn",
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@matthewwalsh0 I need it to do real Sentry reporting, is this an okay change to make? I defined SENTRY_DSN_DEV in the CircleCI env vars. Is there a better way to accomplish this?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be safer to set this inside .circleci/config.yml on the relevant jobs so devs can still run the tests locally without any Sentry overhead?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wrote that initial message 2 weeks ago and a few things have changed, but I think we have to decide -- do we want to respect what the developer set in their .metamaskrc for SENTRY_DSN_DEV, or do we want to override it for E2E tests? If they do not set SENTRY_DSN_DEV at all, it will go down the log('Skipped initialization'); path.

Copy link
Member

@matthewwalsh0 matthewwalsh0 Sep 9, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point.

Maybe the safest compromise is update getSentryTarget to use the fake DSN if no SENTRY_DSN_DEV is defined and process.env.IN_TEST is set?

Meaning all the tests by default still use the fake DSN as before, but now the developer can also use their own as desired by these changes?

app/scripts/lib/setupSentry.js Outdated Show resolved Hide resolved
@HowardBraham HowardBraham force-pushed the tiger-sentry-circleci branch 9 times, most recently from 9eb4e3c to 5d93032 Compare August 29, 2024 02:45
@HowardBraham HowardBraham force-pushed the tiger-sentry-circleci branch 4 times, most recently from c9418ae to 969f352 Compare September 4, 2024 03:25
@metamaskbot
Copy link
Collaborator

Builds ready [969f352]
Page Load Metrics (1794 ± 83 ms)
PlatformPageMetricMin (ms)Max (ms)Average (ms)StandardDeviation (ms)MarginOfError (ms)
ChromeHomefirstPaint21823031640501240
domContentLoaded15742204177415876
load15842313179417483
domInteractive139538199
Bundle size diffs [🚨 Warning! Bundle size has increased!]
  • background: 0 Bytes (0.00%)
  • ui: 192 Bytes (0.00%)
  • common: 1.06 KiB (0.01%)

@HowardBraham HowardBraham marked this pull request as ready for review September 4, 2024 05:48
@HowardBraham HowardBraham requested review from kumavis and a team as code owners September 4, 2024 05:49
@metamaskbot
Copy link
Collaborator

Builds ready [4718657]
Page Load Metrics (1769 ± 104 ms)
PlatformPageMetricMin (ms)Max (ms)Average (ms)StandardDeviation (ms)MarginOfError (ms)
ChromeHomefirstPaint21226561630518249
domContentLoaded15372442174919996
load155025491769216104
domInteractive13105382010
Bundle size diffs [🚨 Warning! Bundle size has increased!]
  • background: 0 Bytes (0.00%)
  • ui: 192 Bytes (0.00%)
  • common: 1.06 KiB (0.01%)

@metamaskbot
Copy link
Collaborator

Builds ready [1f0f5db]
Page Load Metrics (1763 ± 124 ms)
PlatformPageMetricMin (ms)Max (ms)Average (ms)StandardDeviation (ms)MarginOfError (ms)
ChromeHomefirstPaint51326511701381183
domContentLoaded149524891740244117
load154125931763259124
domInteractive129433189
Bundle size diffs [🚨 Warning! Bundle size has increased!]
  • background: 0 Bytes (0.00%)
  • ui: 192 Bytes (0.00%)
  • common: 1.06 KiB (0.01%)

Copy link

codecov bot commented Sep 4, 2024

Codecov Report

Attention: Patch coverage is 3.12500% with 31 lines in your changes missing coverage. Please review.

Project coverage is 70.11%. Comparing base (7dc6362) to head (c48a4c8).
Report is 8 commits behind head on develop.

Files with missing lines Patch % Lines
app/scripts/lib/setupSentry.js 3.57% 27 Missing ⚠️
app/scripts/lib/manifestFlags.ts 0.00% 4 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop   #26588      +/-   ##
===========================================
- Coverage    70.15%   70.11%   -0.04%     
===========================================
  Files         1425     1426       +1     
  Lines        49653    49683      +30     
  Branches     13891    13901      +10     
===========================================
+ Hits         34833    34834       +1     
- Misses       14820    14849      +29     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
…es, changed `circleci.enabled` to `true` or unset
@metamaskbot
Copy link
Collaborator

Builds ready [bd44443]
Page Load Metrics (1844 ± 74 ms)
PlatformPageMetricMin (ms)Max (ms)Average (ms)StandardDeviation (ms)MarginOfError (ms)
ChromeHomefirstPaint24222571787386186
domContentLoaded16472238182314670
load16572250184415574
domInteractive196836136
Bundle size diffs [🚨 Warning! Bundle size has increased!]
  • background: 0 Bytes (0.00%)
  • ui: 0 Bytes (0.00%)
  • common: 1.22 KiB (0.02%)

davidmurdoch
davidmurdoch previously approved these changes Sep 10, 2024
Copy link
Contributor

@davidmurdoch davidmurdoch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This look good. I've Approved, but left a question and one more nit/suggestion.

We should think and talk about following this PR up with a way to ensure that all this additional runtime logic gets compiled out when built for production.

doNotForceSentryForThisTest?: boolean;
};

// eslint-disable-next-line @typescript-eslint/consistent-type-definitions -- you can't extend a type, we want this to be an interface
Copy link
Contributor

@davidmurdoch davidmurdoch Sep 10, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I think you can do it with something like this placed in a browser.d.ts file in our /types directory.

// in app/scripts/lib/manifestFlags.ts
// use
browser.runtime.getManifest<ManifestFlags>()._flags

or 

browser.runtime.getManifest<ManifestFlags | undefined>()._flags

then in /types/browser.d.ts:

import 'webextension-polyfill';

declare module 'webextension-polyfill' {
  // Extend the existing WebExtensionManifest to include _flags
  namespace Manifest {
    interface WebExtensionManifestWithFlags<Flags>
      extends WebExtensionManifest {
      _flags: Flags;
    }
  }

  namespace Runtime {
    interface Static {
      // extend `getManifest` to allow for expecting a flags property
      getManifest<Flags = never>(): [Flags] extends [never]
        ? Manifest.WebExtensionManifest
        : Manifest.WebExtensionManifestWithFlags<Flags>;
    }
  }
}

You wouldn't have to make it generic, you could also hard code the types in the /types/browser.d.ts, or just make it return something like Record<string, unknown> then cast _flags like you already are.

If you don't want to include | undefined in the generic you can set the _flags: Flags type in browser.d.ts to _flags?: Flags instead.

Again, just a nit.

const { circleci } = getManifestFlags();

if (circleci?.enabled) {
Sentry.setTag('circleci.enabled', Boolean(circleci.enabled));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do you need to use Boolean here?

davidmurdoch
davidmurdoch previously approved these changes Sep 10, 2024
Copy link

Quality Gate Failed Quality Gate failed

Failed conditions
0.0% Coverage on New Code (required ≥ 80%)

See analysis details on SonarCloud

@metamaskbot
Copy link
Collaborator

Builds ready [c48a4c8]
Page Load Metrics (1784 ± 94 ms)
PlatformPageMetricMin (ms)Max (ms)Average (ms)StandardDeviation (ms)MarginOfError (ms)
ChromeHomefirstPaint23221981715392188
domContentLoaded14952188176118790
load15032200178419794
domInteractive126326105
Bundle size diffs [🚨 Warning! Bundle size has increased!]
  • background: 0 Bytes (0.00%)
  • ui: 0 Bytes (0.00%)
  • common: 1.21 KiB (0.02%)

@HowardBraham HowardBraham merged commit 5280151 into develop Sep 11, 2024
77 of 78 checks passed
@HowardBraham HowardBraham deleted the tiger-sentry-circleci branch September 11, 2024 15:06
@github-actions github-actions bot locked and limited conversation to collaborators Sep 11, 2024
@metamaskbot metamaskbot added the release-12.6.0 Issue or pull request that will be included in release 12.6.0 label Sep 11, 2024
@metamaskbot metamaskbot added release-12.5.0 Issue or pull request that will be included in release 12.5.0 and removed release-12.6.0 Issue or pull request that will be included in release 12.6.0 labels Sep 29, 2024
@HowardBraham HowardBraham added contributor experience An issue that impacts, or planned improvement to, the contributor experience. and removed team-contributor-experience labels Oct 18, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
contributor experience An issue that impacts, or planned improvement to, the contributor experience. release-12.5.0 Issue or pull request that will be included in release 12.5.0 team-tiger Tiger team (for tech debt reduction + performance improvements)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants