Fix Flaky - testForceMergeWithSoftDeletesRetentionAndRecoverySource #5364 #11494

sarthakaggarwal97 · 2023-12-06T19:26:03Z

Description

The document that is deleted is the one that would be indexed again while we check for the useRecoverySource flag to populate the liveDocsWithSource set

The tests are also passing with the failed seeds

C75D09DC5A0C3458

1766E6A3B733A7AA

10K Iterations Passed

Related Issues

Resolves #5364

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)
Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-12-06T19:48:21Z

Compatibility status:

Checks if related components are compatible with change 83938fb

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/performance-analyzer.git]

Signed-off-by: Sarthak Aggarwal <[email protected]>

github-actions · 2023-12-06T20:12:36Z

❌ Gradle check result for f68b0f1: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2023-12-06T20:45:48Z

✅ Gradle check result for f3f1c76: SUCCESS

codecov · 2023-12-06T20:47:12Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 71.41%. Comparing base (f7f3500) to head (83938fb).
Report is 411 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #11494      +/-   ##
============================================
+ Coverage     71.37%   71.41%   +0.04%     
- Complexity    59113    59143      +30     
============================================
  Files          4893     4903      +10     
  Lines        277831   277983     +152     
  Branches      40367    40382      +15     
============================================
+ Hits         198288   198518     +230     
+ Misses        63042    62969      -73     
+ Partials      16501    16496       -5

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

github-actions · 2023-12-06T20:58:38Z

❕ Gradle check result for 83938fb: UNSTABLE

TEST FAILURES:

      1 org.opensearch.remotestore.RemoteIndexPrimaryRelocationIT.testPrimaryRelocationWhileIndexing

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

opensearch-trigger-bot · 2024-01-25T15:21:18Z

This PR is stalled because it has been open for 30 days with no activity.

opensearch-trigger-bot · 2024-02-27T15:20:14Z

This PR is stalled because it has been open for 30 days with no activity.

opensearch-trigger-bot · 2024-04-07T15:19:57Z

This PR is stalled because it has been open for 30 days with no activity.

gaobinlong · 2024-05-11T13:32:16Z

server/src/test/java/org/opensearch/index/engine/InternalEngineTests.java

@@ -1822,8 +1822,8 @@ public void testForceMergeWithSoftDeletesRetentionAndRecoverySource() throws Exc
            )
        ) {
            int numDocs = scaledRandomIntBetween(10, 100);
+            boolean useRecoverySource = randomBoolean() || omitSourceAllTheTime;


Move the useRecoverySource flag out of the for loop makes every document has same flag, the randomness is lost, I think we need to keep the randomness in order to cover different cases.

gaobinlong · 2024-05-11T13:34:17Z

server/src/test/java/org/opensearch/index/engine/InternalEngineTests.java

                ParsedDocument doc = testParsedDocument(Integer.toString(i), null, testDocument(), B_1, null, useRecoverySource);
-                if (randomBoolean()) {
+                boolean isDeleted = randomBoolean();
+                if (isDeleted) {


Does this mean that if isDeleted is true, for each document, we delete it firstly, and then add them back? If so the randomness is also lost.

gaobinlong · 2024-05-11T13:35:10Z

server/src/test/java/org/opensearch/index/engine/InternalEngineTests.java

@@ -1848,9 +1848,8 @@ public void testForceMergeWithSoftDeletesRetentionAndRecoverySource() throws Exc
                        liveDocsWithSource.remove(doc.id());
                    }
                }
-                if (randomBoolean()) {


Why remove the if condition?

gaobinlong · 2024-05-11T13:38:08Z

server/src/test/java/org/opensearch/index/engine/InternalEngineTests.java

@@ -1896,7 +1895,6 @@ public void testForceMergeWithSoftDeletesRetentionAndRecoverySource() throws Exc
                numSegments = searcher.getDirectoryReader().leaves().size();
            }
            if (numSegments == 1) {
-                boolean useRecoverySource = randomBoolean() || omitSourceAllTheTime;


If numSegments = 1, we add a new document here, so the flag should be generated randomly.

sarthakaggarwal97 added the skip-changelog label Dec 6, 2023

github-actions bot added bug Something isn't working flaky-test Random test failure that succeeds on second run Indexing Indexing, Bulk Indexing and anything related to indexing labels Dec 6, 2023

sarthakaggarwal97 changed the title ~~Fix Flaky #5364~~ Fix Flaky - testForceMergeWithSoftDeletesRetentionAndRecoverySource #5364 Dec 6, 2023

fix flaky opensearch-project#5364

f3f1c76

Signed-off-by: Sarthak Aggarwal <[email protected]>

sarthakaggarwal97 force-pushed the fix-flaky-5364 branch from f68b0f1 to f3f1c76 Compare December 6, 2023 19:52

fix flaky opensearch-project#5364

83938fb

Signed-off-by: Sarthak Aggarwal <[email protected]>

sarthakaggarwal97 marked this pull request as ready for review December 26, 2023 08:41

sarthakaggarwal97 requested review from abbashus, adnapibar, anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, dbwiddis, dreamer-89, gbbafna, kartg, kotwanikunal, mch2, msfroh, nknize and owaiskazi19 as code owners December 26, 2023 08:41

sarthakaggarwal97 requested review from reta, Rishikesh1159, ryanbogan, sachinpkale, saratvemulapalli, setiah, shwetathareja, sohami, tlfeng and VachaShah as code owners December 26, 2023 08:41

sarthakaggarwal97 marked this pull request as draft December 26, 2023 08:50

opensearch-trigger-bot bot added the stalled Issues that have stalled label Jan 25, 2024

sarthakaggarwal97 removed the stalled Issues that have stalled label Jan 26, 2024

sarthakaggarwal97 marked this pull request as ready for review January 26, 2024 20:48

opensearch-trigger-bot bot added the stalled Issues that have stalled label Feb 27, 2024

opensearch-trigger-bot bot removed the stalled Issues that have stalled label Mar 6, 2024

opensearch-trigger-bot bot added stalled Issues that have stalled and removed stalled Issues that have stalled labels Apr 7, 2024

gaobinlong reviewed May 11, 2024

View reviewed changes

sarthakaggarwal97 closed this May 31, 2024

sarthakaggarwal97 mentioned this pull request May 31, 2024

Fix Flaky - testForceMergeWithSoftDeletesRetentionAndRecoverySource #5364 #13910

Merged

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix Flaky - testForceMergeWithSoftDeletesRetentionAndRecoverySource #5364 #11494

Fix Flaky - testForceMergeWithSoftDeletesRetentionAndRecoverySource #5364 #11494

sarthakaggarwal97 commented Dec 6, 2023 •

edited

Loading

github-actions bot commented Dec 6, 2023 •

edited

Loading

github-actions bot commented Dec 6, 2023

github-actions bot commented Dec 6, 2023

codecov bot commented Dec 6, 2023 •

edited

Loading

github-actions bot commented Dec 6, 2023

opensearch-trigger-bot bot commented Jan 25, 2024

opensearch-trigger-bot bot commented Feb 27, 2024

opensearch-trigger-bot bot commented Apr 7, 2024

gaobinlong May 11, 2024

gaobinlong May 11, 2024

gaobinlong May 11, 2024

gaobinlong May 11, 2024

Fix Flaky - testForceMergeWithSoftDeletesRetentionAndRecoverySource #5364 #11494

Fix Flaky - testForceMergeWithSoftDeletesRetentionAndRecoverySource #5364 #11494

Conversation

sarthakaggarwal97 commented Dec 6, 2023 • edited Loading

Description

Related Issues

Check List

github-actions bot commented Dec 6, 2023 • edited Loading

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Dec 6, 2023

github-actions bot commented Dec 6, 2023

codecov bot commented Dec 6, 2023 • edited Loading

Codecov Report

github-actions bot commented Dec 6, 2023

opensearch-trigger-bot bot commented Jan 25, 2024

opensearch-trigger-bot bot commented Feb 27, 2024

opensearch-trigger-bot bot commented Apr 7, 2024

gaobinlong May 11, 2024

Choose a reason for hiding this comment

gaobinlong May 11, 2024

Choose a reason for hiding this comment

gaobinlong May 11, 2024

Choose a reason for hiding this comment

gaobinlong May 11, 2024

Choose a reason for hiding this comment

sarthakaggarwal97 commented Dec 6, 2023 •

edited

Loading

github-actions bot commented Dec 6, 2023 •

edited

Loading

codecov bot commented Dec 6, 2023 •

edited

Loading