Fix flaky test RemoteIndexPrimaryRelocationIT #11614

sachinpkale · 2023-12-14T06:55:43Z

Description

During primary relocation, once new primary is added to the replication group, if the old primary gets the indexing requests, these requests are also forwarded to the new primary.
New primary, post processing these indexing requests, do not update the local checkpoint and sends the older local checkpoint in the replica response.
Code reference where RemoteFsTranslog does not update local checkpoint in ensureSynced: https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/index/translog/RemoteFsTranslog.java#L259
The primaryContext sent from old primary to new primary in the handoff process contains map of allocation ID to local checkpoint info. This map has older checkpoint for the new primary.
Local checkpoint of the new primary is updated during performSegRep step of the relocation.
So, when the new primary receives the primaryContext, the local checkpoint in the context vs actual checkpoint differs, failing this assertion.
In this bugfix, we compare the local checkpoint of new primary with that of old primary for remote backed indexes.

Related Issues

[BUG] org.opensearch.remotestore.RemoteIndexPrimaryRelocationIT.testPrimaryRelocationWhileIndexing flaky failure #9191

Check List

~~New functionality includes testing.~~
- ~~All tests pass~~
~~New functionality has been documented.~~
- ~~New functionality has javadoc added~~
~~Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)~~
Commits are signed per the DCO using --signoff
~~Commit changes are listed out in CHANGELOG.md file (See: Changelog)~~
~~Public documentation issue/PR created~~

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-12-14T07:14:20Z

❌ Gradle check result for 833c1b6: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2023-12-14T07:17:25Z

Compatibility status:

Checks if related components are compatible with change 9362e4f

Incompatible components

Incompatible components: [https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/performance-analyzer-rca.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/k-nn.git]

github-actions · 2023-12-14T07:47:16Z

✅ Gradle check result for 303eff0: SUCCESS

codecov · 2023-12-14T07:48:46Z

Codecov Report

Attention: Patch coverage is 25.00000% with 3 lines in your changes are missing coverage. Please review.

Project coverage is 71.38%. Comparing base (904c9a9) to head (9362e4f).
Report is 155 commits behind head on main.

Files	Patch %	Lines
...in/java/org/opensearch/index/shard/IndexShard.java	25.00%	1 Missing and 2 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #11614      +/-   ##
============================================
- Coverage     71.44%   71.38%   -0.07%     
+ Complexity    59397    59362      -35     
============================================
  Files          4923     4923              
  Lines        279178   279181       +3     
  Branches      40581    40582       +1     
============================================
- Hits         199470   199283     -187     
- Misses        63064    63315     +251     
+ Partials      16644    16583      -61

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

github-actions · 2023-12-14T08:52:46Z

❕ Gradle check result for dfd8f7a: UNSTABLE

TEST FAILURES:

      1 org.opensearch.remotestore.multipart.RemoteStoreMultipartIT.testNoSearchIdleForAnyReplicaCount

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

peternied · 2023-12-14T22:53:29Z

server/src/main/java/org/opensearch/index/shard/IndexShard.java

-                + "] does not match checkpoint from primary context ["
-                + primaryContext
-                + "]";
+        if (isRemoteStoreEnabled()) {


Lets add in some tests cases to confirm this behavior

There already a test exists: org.opensearch.remotestore.RemoteIndexPrimaryRelocationIT.testPrimaryRelocationWhileIndexing but it is flaky without this change

From the annotation below this codepath isn't hit, is the coverage tool incorrect?

The problem is: coverage tool does not consider code covered by integ tests

OpenSearch/gradle/code-coverage.gradle

Lines 37 to 44 in 4323af1

if (System.getProperty("tests.coverage")) {

reporting {

reports {

testCodeCoverageReport(JacocoCoverageReport) {

testType = TestSuiteType.UNIT_TEST

}

}

}

Let me add unit test around this change.

Thanks @sachinpkale !

server/src/main/java/org/opensearch/index/shard/IndexShard.java

opensearch-trigger-bot · 2024-01-17T15:20:44Z

This PR is stalled because it has been open for 30 days with no activity.

Signed-off-by: Sachin Kale <[email protected]>

github-actions · 2024-01-18T12:17:23Z

❌ Gradle check result for 9362e4f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-01-18T14:57:08Z

❌ Gradle check result for 9362e4f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-01-19T04:50:45Z

❌ Gradle check result for 9362e4f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-01-19T05:45:34Z

❌ Gradle check result for 9362e4f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-01-19T06:37:07Z

✅ Gradle check result for 9362e4f: SUCCESS

opensearch-trigger-bot · 2024-03-06T15:20:46Z

This PR is stalled because it has been open for 30 days with no activity.

sachinpkale · 2024-03-21T05:39:29Z

The test is fixed as part of #12494. Closing this PR.

sachinpkale added the skip-changelog label Dec 14, 2023

peternied requested changes Dec 14, 2023

View reviewed changes

peternied self-assigned this Dec 14, 2023

opensearch-trigger-bot bot added the stalled Issues that have stalled label Jan 17, 2024

Sachin Kale added 4 commits January 18, 2024 17:17

Fix flaky test RemoteIndexPrimaryRelocationIT

154b963

Signed-off-by: Sachin Kale <[email protected]>

Keep the old assertion same for non remote backed indexes

6664012

Signed-off-by: Sachin Kale <[email protected]>

Fix spotless errors

caf7498

Signed-off-by: Sachin Kale <[email protected]>

Address PR comments

9362e4f

Signed-off-by: Sachin Kale <[email protected]>

sachinpkale force-pushed the handoff-failure-flaky-fix branch from dfd8f7a to 9362e4f Compare January 18, 2024 11:48

sachinpkale requested a review from peternied January 18, 2024 11:50

opensearch-trigger-bot bot removed the stalled Issues that have stalled label Jan 18, 2024

opensearch-trigger-bot bot added stalled Issues that have stalled and removed stalled Issues that have stalled labels Mar 6, 2024

sachinpkale closed this Mar 21, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix flaky test RemoteIndexPrimaryRelocationIT #11614

Fix flaky test RemoteIndexPrimaryRelocationIT #11614

sachinpkale commented Dec 14, 2023 •

edited

Loading

github-actions bot commented Dec 14, 2023

github-actions bot commented Dec 14, 2023 •

edited

Loading

github-actions bot commented Dec 14, 2023

codecov bot commented Dec 14, 2023 •

edited

Loading

github-actions bot commented Dec 14, 2023

peternied Dec 14, 2023

sachinpkale Jan 18, 2024

peternied Jan 19, 2024

sachinpkale Jan 29, 2024

sachinpkale Jan 29, 2024

peternied Feb 1, 2024

opensearch-trigger-bot bot commented Jan 17, 2024

github-actions bot commented Jan 18, 2024

github-actions bot commented Jan 18, 2024

github-actions bot commented Jan 19, 2024

github-actions bot commented Jan 19, 2024

github-actions bot commented Jan 19, 2024

opensearch-trigger-bot bot commented Mar 6, 2024

sachinpkale commented Mar 21, 2024

	if (System.getProperty("tests.coverage")) {
	reporting {
	reports {
	testCodeCoverageReport(JacocoCoverageReport) {
	testType = TestSuiteType.UNIT_TEST
	}
	}
	}

Fix flaky test RemoteIndexPrimaryRelocationIT #11614

Fix flaky test RemoteIndexPrimaryRelocationIT #11614

Conversation

sachinpkale commented Dec 14, 2023 • edited Loading

Description

Related Issues

Check List

github-actions bot commented Dec 14, 2023

github-actions bot commented Dec 14, 2023 • edited Loading

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Dec 14, 2023

codecov bot commented Dec 14, 2023 • edited Loading

Codecov Report

github-actions bot commented Dec 14, 2023

peternied Dec 14, 2023

Choose a reason for hiding this comment

sachinpkale Jan 18, 2024

Choose a reason for hiding this comment

peternied Jan 19, 2024

Choose a reason for hiding this comment

sachinpkale Jan 29, 2024

Choose a reason for hiding this comment

sachinpkale Jan 29, 2024

Choose a reason for hiding this comment

peternied Feb 1, 2024

Choose a reason for hiding this comment

opensearch-trigger-bot bot commented Jan 17, 2024

github-actions bot commented Jan 18, 2024

github-actions bot commented Jan 18, 2024

github-actions bot commented Jan 19, 2024

github-actions bot commented Jan 19, 2024

github-actions bot commented Jan 19, 2024

opensearch-trigger-bot bot commented Mar 6, 2024

sachinpkale commented Mar 21, 2024

sachinpkale commented Dec 14, 2023 •

edited

Loading

github-actions bot commented Dec 14, 2023 •

edited

Loading

codecov bot commented Dec 14, 2023 •

edited

Loading