Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Disable MockTelemetryPlugin by default for integ tests #12877

Merged
merged 1 commit into from
Mar 25, 2024

Conversation

mch2
Copy link
Member

@mch2 mch2 commented Mar 22, 2024

Description

With this plugin enabled I see significant increase in memory usage during ITs. This is due to a static map that collects Span entries per write/replication request and is not cleared until after the test execution. Tests with higher counts of independent writes are at risk of running oom or experiencing slow down / warnings related to this.

I've run two reported flaky tests on repeat with this disabled #12801 and #12836 and do not see failures.

Disabled the plugin by default until this is resolved given it is still under FeatureFlag.

Related Issues

related to #12615
Resolves #12801
Resolves #12836

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
  • Commits are signed per the DCO using --signoff
  • Commit changes are listed out in CHANGELOG.md file (See: Changelog)
  • Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

With this plugin enabled we see significant increase in memory usage.
This is due to a static map that collects Span entries per write/replication request.
Tests with higher counts of independent writes are at risk of running oom or experiencing
slow down / warnings related to this.

Disabled the plugin by default until this is resolved given it is still under FeatureFlag.

Signed-off-by: Marc Handalian <[email protected]>
Copy link
Contributor

Compatibility status:

Checks if related components are compatible with change 6a8dbd7

Incompatible components

Incompatible components: [https://github.com/opensearch-project/security.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/flow-framework.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/performance-analyzer.git]

Copy link
Contributor

❌ Gradle check result for 6a8dbd7: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Copy link
Contributor

❕ Gradle check result for 6a8dbd7: UNSTABLE

  • TEST FAILURES:
      1 org.opensearch.snapshots.CloneSnapshotIT.testShallowCloneNameAvailability

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

Copy link

codecov bot commented Mar 22, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 71.31%. Comparing base (b15cb0c) to head (6a8dbd7).
Report is 86 commits behind head on main.

Additional details and impacted files
@@             Coverage Diff              @@
##               main   #12877      +/-   ##
============================================
- Coverage     71.42%   71.31%   -0.11%     
- Complexity    59978    60183     +205     
============================================
  Files          4985     5011      +26     
  Lines        282275   283675    +1400     
  Branches      40946    41117     +171     
============================================
+ Hits         201603   202294     +691     
- Misses        63999    64597     +598     
- Partials      16673    16784     +111     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@reta
Copy link
Collaborator

reta commented Mar 25, 2024

@Gaganjuneja fyi

@Gaganjuneja
Copy link
Contributor

Can we hold on this PR. I will try to fix within this week only. Otherwise we can go ahead with this PR.

@mch2
Copy link
Member Author

mch2 commented Mar 25, 2024

Can we hold on this PR. I will try to fix within this week only. Otherwise we can go ahead with this PR.

I would prefer not to wait. This could incorrectly flag a lot of tests as flaky that would take more time to analyze. We can re-enable once the fix is ready.

@mch2
Copy link
Member Author

mch2 commented Mar 25, 2024

UNSTABLE

#10155

@mch2 mch2 added the backport 2.x Backport to 2.x branch label Mar 25, 2024
@mch2 mch2 merged commit 52ae79d into opensearch-project:main Mar 25, 2024
81 of 137 checks passed
opensearch-trigger-bot bot pushed a commit that referenced this pull request Mar 25, 2024
With this plugin enabled we see significant increase in memory usage.
This is due to a static map that collects Span entries per write/replication request.
Tests with higher counts of independent writes are at risk of running oom or experiencing
slow down / warnings related to this.

Disabled the plugin by default until this is resolved given it is still under FeatureFlag.

Signed-off-by: Marc Handalian <[email protected]>
(cherry picked from commit 52ae79d)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
reta pushed a commit that referenced this pull request Mar 25, 2024
With this plugin enabled we see significant increase in memory usage.
This is due to a static map that collects Span entries per write/replication request.
Tests with higher counts of independent writes are at risk of running oom or experiencing
slow down / warnings related to this.

Disabled the plugin by default until this is resolved given it is still under FeatureFlag.


(cherry picked from commit 52ae79d)

Signed-off-by: Marc Handalian <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@mch2 mch2 deleted the 12801 branch March 25, 2024 19:24
shiv0408 pushed a commit to Gaurav614/OpenSearch that referenced this pull request Apr 25, 2024
…oject#12877)

With this plugin enabled we see significant increase in memory usage.
This is due to a static map that collects Span entries per write/replication request.
Tests with higher counts of independent writes are at risk of running oom or experiencing
slow down / warnings related to this.

Disabled the plugin by default until this is resolved given it is still under FeatureFlag.

Signed-off-by: Marc Handalian <[email protected]>
Signed-off-by: Shivansh Arora <[email protected]>
harshavamsi pushed a commit to harshavamsi/OpenSearch that referenced this pull request Apr 29, 2024
…oject#12877)

With this plugin enabled we see significant increase in memory usage.
This is due to a static map that collects Span entries per write/replication request.
Tests with higher counts of independent writes are at risk of running oom or experiencing
slow down / warnings related to this.

Disabled the plugin by default until this is resolved given it is still under FeatureFlag.

Signed-off-by: Marc Handalian <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 2.x Backport to 2.x branch bug Something isn't working Build Build Tasks/Gradle Plugin, groovy scripts, build tools, Javadoc enforcement. flaky-test Random test failure that succeeds on second run skip-changelog Storage:Snapshots
Projects
Status: ✅ Done
6 participants