Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Performance] Add minimal performance test to detect telemetry eviction #7312

Open
unlikelyzero opened this issue Dec 19, 2023 · 8 comments · May be fixed by akhenry/openmct-yamcs#425
Open

[Performance] Add minimal performance test to detect telemetry eviction #7312

unlikelyzero opened this issue Dec 19, 2023 · 8 comments · May be fixed by akhenry/openmct-yamcs#425
Assignees
@ozyx @scottbell
Labels
type:maintenance tests, chores, or project maintenance

Comments

@unlikelyzero
Copy link
Collaborator

Summary
@unlikelyzero unlikelyzero added the type:maintenance tests, chores, or project maintenance label Dec 19, 2023
@unlikelyzero unlikelyzero added this to the Target:3.3.0 milestone Dec 19, 2023
@scottbell scottbell self-assigned this Dec 20, 2023
@scottbell
Copy link
Contributor

scottbell commented Dec 20, 2023 •
edited
Loading

Getting 7 memory leaks, though none look related to imagery telemetry:

⏲️  Waiting 2000ms...
🚮 Running garbage collection...
📸 Capturing heap snapshot to /Users/scott/workspace/openmct/e2e/test-data/snapshots/data/cur/s1.heapsnapshot
⏲️  Waiting 2000ms...
🚮 Running garbage collection...
📸 Capturing heap snapshot to /Users/scott/workspace/openmct/e2e/test-data/snapshots/data/cur/s2.heapsnapshot
⏲️  Waiting 2000ms...
🚮 Running garbage collection...
📸 Capturing heap snapshot to /Users/scott/workspace/openmct/e2e/test-data/snapshots/data/cur/s3.heapsnapshot
MemLab found 7 leak(s)
Number of clusters loaded: 0

--Similar leaks in this run: 17--
--Retained size of leaked objects: 10KB--
[<synthetic>] (synthetic) @1 [13.6MB]
  --3 (shortcut)--->  [Window / http://localhost:8080] (object) @6211 [154.3KB]
  --__VUE_INSTANCE_SETTERS__ (property)--->  [Array] (object) @254719 [164 bytes]
  --0 (element)--->  [<closure>] (closure) @254721 [72 bytes]
  --context (internal)--->  [<function scope>] (object) @65935 [13KB]
  --n (variable)--->  [Object] (object) @224927 [4.4KB]
  --get SU (property)--->  [SU] (closure) @252267 [72 bytes]
  --context (internal)--->  [<function scope>] (object) @71073 [24.5KB]
  --Be (variable)--->  [WeakMap] (object) @259471 [8.4KB]
  --table (internal)--->  [<array>] (array) @259787 [8.4KB]
  --36 / part of key (Object @319849) -> value (system / JSProxy @319851) pair in WeakMap (table @259787) (internal)--->  [system / JSProxy] (hidden) @319851 [268 bytes]
  --1 (hidden)--->  [Object] (object) @319849 [252 bytes]
  --onChange (property)--->  [p] (closure) @354829 [76 bytes]
  --context (internal)--->  [<function scope>] (object) @354849 [312 bytes]
  --l (variable)--->  [destroy] (closure) @354933 [48 bytes]
  --context (internal)--->  [<function scope>] (object) @355163 [20 bytes]
  --c (variable)--->  [Object] (object) @354143 [10.9KB]
  --_container (property)--->  [Detached HTMLDivElement] (native) @60335 [10KB]
  --5 (element)--->  [Detached HTMLDivElement] (native) @60361 [9.8KB]
  --5 (element)--->  [Detached HTMLDivElement] (native) @60353 [196 bytes]
  --6 (element)--->  [Detached Text] (native) @60351 [108 bytes]
  --4 (element)--->  [Detached HTMLDivElement] (native) @60347 [368 bytes]
  --9 (element)--->  [Detached InternalNode] (native) @12940 [112 bytes]
  --1 (element)--->  [Detached InternalNode] (native) @12942 [112 bytes]
  --1 (element)--->  [Detached InternalNode] (native) @14560 [112 bytes]
  --1 (element)--->  [Detached EventListener] (native) @10192 [112 bytes]


--Similar leaks in this run: 48--
--Retained size of leaked objects: 5.5KB--
[<synthetic>] (synthetic) @1 [13.6MB]
  --3 (shortcut)--->  [Window / http://localhost:8080] (object) @6211 [154.3KB]
  --openmct (property)--->  [v] (object) @69049 [63.4KB]
  --forms (property)--->  [I] (object) @373367 [2.8KB]
  --formController (property)--->  [E] (object) @373573 [2.6KB]
  --controls (property)--->  [Object] (object) @491191 [2.4KB]
  --file-input (property)--->  [Object] (object) @319867 [2.1KB]
  --destroy (property)--->  [destroy] (closure) @491245 [28 bytes]
  --context (internal)--->  [<function scope>] (object) @349391 [2.1KB]
  --i (variable)--->  [destroy] (closure) @349925 [2KB]
  --context (internal)--->  [<function scope>] (object) @349927 [2KB]
  --c (variable)--->  [Object] (object) @344171 [2KB]
  --_container (property)--->  [Detached HTMLDivElement] (native) @60435 [144 bytes]
  --5 (element)--->  [Detached HTMLDivElement] (native) @60421 [576 bytes]
  --7 (element)--->  [Detached HTMLDivElement] (native) @60411 [124 bytes]
  --5 (element)--->  [Detached HTMLFormElement] (native) @60389 [888 bytes]
  --14 (element)--->  [Detached InternalNode] (native) @6344 [112 bytes]
  --1 (element)--->  [Detached InternalNode] (native) @3910 [112 bytes]
  --1 (element)--->  [Detached InternalNode] (native) @3912 [112 bytes]
  --1 (element)--->  [Detached EventListener] (native) @14640 [112 bytes]


--Similar leaks in this run: 46--
--Retained size of leaked objects: 4.8KB--
[<synthetic>] (synthetic) @1 [13.6MB]
  --3 (shortcut)--->  [Window / http://localhost:8080] (object) @6211 [154.3KB]
  --openmct (property)--->  [v] (object) @69049 [63.4KB]
  --objects (property)--->  [g] (object) @75753 [5.4KB]
  --eventEmitter (property)--->  [o] (object) @68377 [16 bytes]
  --_events (property)--->  [Object] (object) @68181 [9.2KB]
  --mine:name (property)--->  [Array] (object) @68955 [164 bytes]
  --4 (element)--->  [n] (object) @473673 [24 bytes]
  --fn (property)--->  [<closure>] (closure) @473553 [27.3KB]
  --context (internal)--->  [<function scope>] (object) @473233 [27.3KB]
  --this (variable)--->  [system / JSProxy] (hidden) @370163 [27.2KB]
  --1 (hidden)--->  [Object] (object) @370177 [27.2KB]
  --_ (property)--->  [Object] (object) @370187 [21.6KB]
  --subTree (property)--->  [Object] (object) @370437 [12.2KB]
  --el (property)--->  [Detached HTMLDivElement] (native) @61617 [124 bytes]
  --3 (element)--->  [Detached HTMLDivElement] (native) @180 [96 bytes]
  --2 (element)--->  [Detached HTMLUListElement] (native) @61641 [124 bytes]
  --4 (element)--->  [Detached Text] (native) @61645 [108 bytes]
  --4 (element)--->  [Detached HTMLLIElement] (native) @61691 [124 bytes]
  --6 (element)--->  [Detached HTMLLIElement] (native) @61683 [124 bytes]
  --6 (element)--->  [Detached HTMLLIElement] (native) @61675 [124 bytes]
  --4 (element)--->  [Detached HTMLDivElement] (native) @61681 [204 bytes]
  --3 (element)--->  [Detached Text] (native) @14986 [80 bytes]


--Similar leaks in this run: 3--
--Retained size of leaked objects: 432 bytes--
[<synthetic>] (synthetic) @1 [13.6MB]
  --3 (shortcut)--->  [Window / http://localhost:8080] (object) @6211 [154.3KB]
  --openmct (property)--->  [v] (object) @69049 [63.4KB]
  --indicators (property)--->  [c] (object) @373351 [1.1KB]
  --indicatorObjects (property)--->  [Array] (object) @373545 [820 bytes]
  --2 (element)--->  [Object] (object) @373815 [76 bytes]
  --destroy (property)--->  [destroy] (closure) @374087 [48 bytes]
  --context (internal)--->  [<function scope>] (object) @374291 [20 bytes]
  --c (variable)--->  [Object] (object) @75675 [1KB]
  --_container (property)--->  [Detached HTMLDivElement] (native) @62011 [144 bytes]


  ✘  1 [chrome-memory] › memory/eviction.memory.perf.spec.js:91:8 › Telemetry eviction for › imagery (27.7s)
--Retained size of leaked objects: 324 bytes--
[<synthetic>] (synthetic) @1 [13.6MB]
  --3 (shortcut)--->  [Window / http://localhost:8080] (object) @6211 [154.3KB]
  --webpackHotUpdateopenmct (property)--->  [<closure>] (closure) @315253 [416 bytes]
  --context (internal)--->  [<function scope>] (object) @315255 [344 bytes]
  --previous (internal)--->  [<function scope>] (object) @69777 [140 bytes]
  --n (variable)--->  [Object] (object) @381347 [386.2KB]
  --61120 (element)--->  [Object] (object) @258701 [640 bytes]
  --exports (property)--->  [<closure>] (closure) @257753 [114KB]
  --context (internal)--->  [<function scope>] (object) @207947 [113.9KB]
  --_r (variable)--->  [e] (closure) @257767 [1KB]
  --_link (property)--->  [Detached HTMLAnchorElement] (native) @61879 [324 bytes]
  --3 (element)--->  [Detached DOMTokenList] (native) @6078 [56 bytes]


--Similar leaks in this run: 1--
--Retained size of leaked objects: 144 bytes--
[<synthetic>] (synthetic) @1 [13.6MB]
  --3 (shortcut)--->  [Window / http://localhost:8080] (object) @6211 [154.3KB]
  --openmct (property)--->  [v] (object) @69049 [63.4KB]
  --time (property)--->  [c] (object) @470183 [3.4KB]
  --_events (property)--->  [Object] (object) @470221 [1.8KB]
  --tick (property)--->  [Array] (object) @471837 [80 bytes]
  --0 (element)--->  [n] (object) @472099 [24 bytes]
  --fn (property)--->  [<closure>] (closure) @370015 [28 bytes]
  --context (internal)--->  [<function scope>] (object) @369905 [5.8KB]
  --e (variable)--->  [native_bind] (closure) @271541 [5.8KB]
  --bound_this (internal)--->  [system / JSProxy] (hidden) @369907 [5.7KB]
  --1 (hidden)--->  [Object] (object) @370013 [5.6KB]
  --_ (property)--->  [Object] (object) @66385 [5.5KB]
  --appContext (property)--->  [Object] (object) @369917 [1.4KB]
  --app (property)--->  [Object] (object) @369801 [1KB]
  --_container (property)--->  [Detached HTMLDivElement] (native) @62021 [144 bytes]


--Similar leaks in this run: 1--
--Retained size of leaked objects: 132 bytes--
[<synthetic>] (synthetic) @1 [13.6MB]
  --6 (element)--->  [C++ Persistent roots] (synthetic) @15448 [25.7KB]
  --5 (element)--->  [HTMLInputElement] (native) @62085 [1.2KB]
  --<symbol _vei> (property)--->  [Object] (object) @402719 [28 bytes]
  --onClick (property)--->  [i] (closure) @62087 [84 bytes]
  --context (internal)--->  [<function scope>] (object) @402715 [24 bytes]
  --previous (internal)--->  [<function scope>] (object) @67085 [10.2KB]
  --B (variable)--->  [Detached HTMLTemplateElement] (native) @61867 [132 bytes]

@scottbell
Copy link
Contributor

Running against YAMCS yield 4 unrelated memory leaks:

     1 [chromium] › yamcs/eviction.e2e.spec.js:43:5 › Telemetry eviction for @yamcs › gyro plot
[WebServer] <i> [webpack-dev-server] [HPM] Upgrading to WebSocket
⏲️  Waiting 2000ms...
🚮 Running garbage collection...
📸 Capturing heap snapshot to /Users/scott/workspace/openmct-yamcs/tests/e2e/test-data/snapshots/data/cur/s1.heapsnapshot
⏲️  Waiting 2000ms...
🚮 Running garbage collection...
📸 Capturing heap snapshot to /Users/scott/workspace/openmct-yamcs/tests/e2e/test-data/snapshots/data/cur/s2.heapsnapshot
⏲️  Waiting 2000ms...
🚮 Running garbage collection...
📸 Capturing heap snapshot to /Users/scott/workspace/openmct-yamcs/tests/e2e/test-data/snapshots/data/cur/s3.heapsnapshot
MemLab found 4 leak(s)
Number of clusters loaded: 0

--Similar leaks in this run: 3--
--Retained size of leaked objects: 420 bytes--
[<synthetic>] (synthetic) @1 [12.9MB]
  --2 (shortcut)--->  [Window / http://localhost:9000] (object) @6207 [37.7KB]
  --openmct (property)--->  [v] (object) @107339 [45.4KB]
  --set _assetPath (property)--->  [set] (closure) @335437 [32 bytes]
  --context (internal)--->  [<function scope>] (object) @348613 [80 bytes]
  --previous (internal)--->  [<function scope>] (object) @119391 [51.1KB]
  --$s (variable)--->  [Detached HTMLDivElement] (native) @59129 [420 bytes]
  --3 (element)--->  [Detached HTMLAnchorElement] (native) @3994 [296 bytes]


--Similar leaks in this run: 2--
--Retained size of leaked objects: 324 bytes--
[<synthetic>] (synthetic) @1 [12.9MB]
  --2 (shortcut)--->  [Window / http://localhost:9000] (object) @6207 [37.7KB]
  ✘  1 [chromium] › yamcs/eviction.e2e.spec.js:43:5 › Telemetry eviction for @yamcs › gyro plot (22.1s)
  --context (internal)--->  [<function scope>] (object) @196197 [356 bytes]
  --previous (internal)--->  [<function scope>] (object) @75671 [103.4KB]
  --e (variable)--->  [Object] (object) @240881 [50.9KB]
  --61120 (element)--->  [Object] (object) @163641 [24 bytes]
  --exports (property)--->  [<closure>] (closure) @259621 [113.9KB]
  --context (internal)--->  [<function scope>] (object) @192971 [113.9KB]
  --wr (variable)--->  [t] (closure) @259657 [1KB]
  --_link (property)--->  [Detached HTMLAnchorElement] (native) @59107 [324 bytes]
  --3 (element)--->  [Detached DOMTokenList] (native) @108 [56 bytes]


--Similar leaks in this run: 2--
--Retained size of leaked objects: 200 bytes--
[<synthetic>] (synthetic) @1 [12.9MB]
  --2 (shortcut)--->  [Window / http://localhost:9000] (object) @6207 [37.7KB]
  --openmct (property)--->  [v] (object) @107339 [45.4KB]
  --layout (property)--->  [s] (object) @286411 [8.9KB]
  --$refs (property)--->  [Object] (object) @389901 [376 bytes]
  --browseObject (property)--->  [s] (object) @287429 [7.3KB]
  --viewContainer (property)--->  [Detached HTMLDivElement] (native) @58145 [200 bytes]
  --5 (element)--->  [Detached InternalNode] (native) @6708 [56 bytes]
  --1 (element)--->  [Detached InternalNode] (native) @992 [56 bytes]
  --1 (element)--->  [Detached DOMTokenList] (native) @994 [56 bytes]


--Similar leaks in this run: 1--
--Retained size of leaked objects: 144 bytes--
[<synthetic>] (synthetic) @1 [12.9MB]
  --2 (shortcut)--->  [Window / http://localhost:9000] (object) @6207 [37.7KB]
  --openmct (property)--->  [v] (object) @107339 [45.4KB]
  --__ob__ (property)--->  [vt] (object) @324249 [24 bytes]
  --dep (property)--->  [lt] (object) @288229 [216 bytes]
  --subs (property)--->  [Array] (object) @391569 [196 bytes]
  --6 (element)--->  [di] (object) @352463 [548 bytes]
  --vm (property)--->  [s] (object) @316495 [3.9KB]
  --$parent (property)--->  [yi] (object) @344783 [2.6KB]
  --$options (property)--->  [Object] (object) @353327 [672 bytes]
  --el (property)--->  [Detached HTMLDivElement] (native) @58223 [144 bytes]

@scottbell
Copy link
Contributor

scottbell commented Dec 21, 2023 •
edited
Loading

Last state of this is we've got a test in the openmct-yamcs repo, and it works if the telemetry is being emitted at ~100Hz. Below that, memlab doesn't seem to see that unbounded object growth of the plot series.

The thing I'd like to add next is to get a trace of the object, find if they're openmct objects, and collect only those for the test instead of relying on hardcoded object types.

@scottbell
Copy link
Contributor

Some open questions @akhenry @unlikelyzero @ozyx

  1. Do we want this in the openmct-yamcs branch? I think this approach should also work with SWGs. Though maybe we’re also worried about the adapter itself? Do we want to test every object that can take telemetry for memory growth over time?
  2. I set a threshold for memory growth at 500kb. Is that reasonable? Should we set it higher/lower?
  3. How long do we want to look at these objects? I’m currently looking at them for 30s per snapshot, so 150s total.

@scottbell
Copy link
Contributor

note i’ve also synthetically tested this approach by just adding a quick return in the PlotSeries’s purge method. the test correctly identifies that telemetry is stacking up.

@scottbell
Copy link
Contributor

Approach after conferring with @akhenry:
When we receive the first telemetry value for both request & subscribe (in the Telemetry API), register the telemetry javascript objects with the finalization method, run view for a minute, then check to see if both telemetry javascript objects are gone.

  1. want to test every view that can take telemetry (one big display?)
  2. check to see if we can use telemetry api low level
  3. do it in openmct-yamcs

@unlikelyzero unlikelyzero modified the milestones: Target:3.3.0, Target:4.0.0 Jan 17, 2024
@scottbell
Copy link
Contributor

scottbell commented Jan 19, 2024 •
edited
Loading

After talking with @akhenry, putting this one on hold for a bit. Current status:

  1. Interception of all TelemetryAPI request calls is working. At page load + 5s, all returned telemetry objects from the request calls are registered with the FinalizationRegistry.
  2. Interception of all TelemetryAPI subscribe calls is working.At page load + 5s, all telemetry objects from the subscribe callbacks are registered with the FinalizationRegistry.
  3. The test passes once all telemetry objects are garbage collected, and thus "evicted".

The reason we're putting this on hold is that the telemetry objects themselves, at least within the Plot View, are almost immediately copied into a new array.. This means the telemetry objects are garbage collected, but also reduces the utility of this test. We should consider:

  1. Changing test to directly hook into PlotSeries to track the newly copied data (i.e., in the add method). To do this we'd need to allow a function to be added to the in PlotSeries via a $ref that feed references into the FinalizationRegistry. This seems really ugly!
  2. Longer term, stop copying telemetry data into a new array.

@scottbell scottbell linked a pull request Feb 6, 2024 that will close this issue
Add minimal performance test to detect telemetry eviction handover akhenry/openmct-yamcs#425
Open
15 tasks
This was referenced Feb 6, 2024
Open
Closed
@scottbell scottbell assigned scottbell and unassigned scottbell Feb 15, 2024
@scottbell
Copy link
Contributor

Digging into this more, I think I'm wrong about this. I did a bunch of testing in this branch adding finalization registers inside of the Telemetry API and the actual PlotSeries, and found the data being garbage collected at the same rate, and the same number of items:

Screen.Recording.2024-02-16.at.12.02.44.PM.mov

So I think this PR is probably fine as is.

@akhenry akhenry removed this from the Target:4.0.0 milestone Mar 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ozyx ozyx

@scottbell scottbell

Labels
type:maintenance tests, chores, or project maintenance
Projects
None yet
Milestone
No milestone
4 participants
@akhenry @ozyx @scottbell @unlikelyzero