Segment Replication - Allow search idle with no replicas #8173

mch2 · 2023-06-20T15:59:31Z

This is a re-open of #7736.

Description
With Segment Replication enabled we generally disable shard idle. This change ensures there are replicas in the index before disabling shard idle with SR. It will warn if a user attempts to update the search.idle.after interval with replicas configured.

Related Issues
#7761

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-06-20T16:32:38Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.search.backpressure.SearchBackpressureIT.testSearchTaskCancellationWithHighCpu

URL: https://build.ci.opensearch.org/job/gradle-check/18004/
CommitID: 5d11cdd
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

codecov · 2023-06-20T16:33:36Z

Codecov Report

Merging #8173 (fde8361) into main (6a19660) will decrease coverage by 0.02%.
The diff coverage is 55.55%.

@@             Coverage Diff              @@
##               main    #8173      +/-   ##
============================================
- Coverage     70.88%   70.87%   -0.02%     
- Complexity    57196    57200       +4     
============================================
  Files          4771     4771              
  Lines        270312   270323      +11     
  Branches      39505    39509       +4     
============================================
- Hits         191609   191585      -24     
- Misses        62600    62616      +16     
- Partials      16103    16122      +19

Impacted Files	Coverage Δ
.../main/java/org/opensearch/index/IndexSettings.java	`85.25% <0.00%> (-0.43%)`	⬇️
...earch/action/search/AbstractSearchAsyncAction.java	`77.03% <25.00%> (-1.50%)`	⬇️
...in/java/org/opensearch/index/shard/IndexShard.java	`69.65% <100.00%> (+0.44%)`	⬆️
...ices/replication/RemoteStoreReplicationSource.java	`94.28% <100.00%> (+0.73%)`	⬆️

... and 476 files with indirect coverage changes

github-actions · 2023-06-20T16:37:38Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.snapshots.DedicatedClusterSnapshotRestoreIT.testIndexDeletionDuringSnapshotCreationInQueue
      1 org.opensearch.search.backpressure.SearchBackpressureIT.testSearchTaskCancellationWithHighCpu

URL: https://build.ci.opensearch.org/job/gradle-check/18005/
CommitID: 455ff46
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

server/src/main/java/org/opensearch/index/shard/IndexShard.java

server/src/main/java/org/opensearch/index/IndexSettings.java

github-actions · 2023-06-20T19:27:45Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.indices.replication.SegmentReplicationRelocationIT.testPrimaryRelocationWithSegRepFailure
      1 org.opensearch.index.IndexServiceTests.testAsyncTranslogTrimTaskOnClosedIndex

URL: https://build.ci.opensearch.org/job/gradle-check/18027/
CommitID: dbdcff7
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2023-06-20T23:17:18Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.search.backpressure.SearchBackpressureIT.testSearchShardTaskCancellationWithHighCpu

URL: https://build.ci.opensearch.org/job/gradle-check/18051/
CommitID: 85343e9
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

mch2 · 2023-07-14T00:18:56Z

@andrross @Bukhtawar Any concerns on this? I think this is still valuable particularly with remote store and we can explore enabling search idle across the board with SegRep in a separate effort.

andrross · 2023-07-14T16:14:38Z

Since "search idle" is a performance optimization I think we have a lot of options to change and improve the behavior going forward, so I don't see any one-way doors here. I also think this is an improvement over the current behavior of disabling search idle when there are no replicas, so I'm good with this change even as we explore improving the search idle behavior in all cases.

server/src/internalClusterTest/java/org/opensearch/action/search/TransportSearchIT.java

github-actions · 2023-07-14T18:57:11Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/20209/
CommitID: 9ab9fb5
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-07-14T19:35:43Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/20210/
CommitID: 0a539ac
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

mch2 · 2023-07-14T20:23:59Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌

URL: https://build.ci.opensearch.org/job/gradle-check/20210/

CommitID: 0a539ac
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

#8699

…index. Signed-off-by: Marc Handalian <[email protected]>

…enabled. Signed-off-by: Marc Handalian <[email protected]>

Signed-off-by: Marc Handalian <[email protected]>

github-actions · 2023-07-14T22:45:17Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/20224/
CommitID: fde8361

* Segment Replication - Allow shard idle when there are no replicas an index. Signed-off-by: Marc Handalian <[email protected]> * Add warning when updating search.idle.after with Segment Replication enabled. Signed-off-by: Marc Handalian <[email protected]> * PR cleanup. Signed-off-by: Marc Handalian <[email protected]> * Add more tests. Signed-off-by: Marc Handalian <[email protected]> * Remove unnecessary assertBusy Signed-off-by: Marc Handalian <[email protected]> --------- Signed-off-by: Marc Handalian <[email protected]> (cherry picked from commit ca74aac) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Segment Replication - Allow shard idle when there are no replicas an index. * Add warning when updating search.idle.after with Segment Replication enabled. * PR cleanup. * Add more tests. * Remove unnecessary assertBusy --------- (cherry picked from commit ca74aac) Signed-off-by: Marc Handalian <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…project#8173) * Segment Replication - Allow shard idle when there are no replicas an index. Signed-off-by: Marc Handalian <[email protected]> * Add warning when updating search.idle.after with Segment Replication enabled. Signed-off-by: Marc Handalian <[email protected]> * PR cleanup. Signed-off-by: Marc Handalian <[email protected]> * Add more tests. Signed-off-by: Marc Handalian <[email protected]> * Remove unnecessary assertBusy Signed-off-by: Marc Handalian <[email protected]> --------- Signed-off-by: Marc Handalian <[email protected]>

…project#8173) * Segment Replication - Allow shard idle when there are no replicas an index. Signed-off-by: Marc Handalian <[email protected]> * Add warning when updating search.idle.after with Segment Replication enabled. Signed-off-by: Marc Handalian <[email protected]> * PR cleanup. Signed-off-by: Marc Handalian <[email protected]> * Add more tests. Signed-off-by: Marc Handalian <[email protected]> * Remove unnecessary assertBusy Signed-off-by: Marc Handalian <[email protected]> --------- Signed-off-by: Marc Handalian <[email protected]> Signed-off-by: Kaushal Kumar <[email protected]>

…project#8173) * Segment Replication - Allow shard idle when there are no replicas an index. Signed-off-by: Marc Handalian <[email protected]> * Add warning when updating search.idle.after with Segment Replication enabled. Signed-off-by: Marc Handalian <[email protected]> * PR cleanup. Signed-off-by: Marc Handalian <[email protected]> * Add more tests. Signed-off-by: Marc Handalian <[email protected]> * Remove unnecessary assertBusy Signed-off-by: Marc Handalian <[email protected]> --------- Signed-off-by: Marc Handalian <[email protected]> Signed-off-by: Shivansh Arora <[email protected]>

mch2 force-pushed the disableSI branch from 5d11cdd to 455ff46 Compare June 20, 2023 16:00

This was referenced Jun 20, 2023

Segment Replication - Allow shard idle when there are no replicas in index #7736

Closed

[Segment Replication] Allow shard idle on indices with zero replicas #7761

Closed

andrross reviewed Jun 20, 2023

View reviewed changes

server/src/main/java/org/opensearch/index/shard/IndexShard.java Outdated Show resolved Hide resolved

andrross reviewed Jun 20, 2023

View reviewed changes

server/src/main/java/org/opensearch/index/IndexSettings.java Outdated Show resolved Hide resolved

mch2 marked this pull request as ready for review June 20, 2023 18:57

mch2 requested review from reta, anasalkouz, Bukhtawar, CEHENKLE, dblock, gbbafna, setiah, kartg, kotwanikunal, nknize, owaiskazi19, Rishikesh1159, ryanbogan, saratvemulapalli, shwetathareja, dreamer-89, tlfeng, VachaShah, dbwiddis and sachinpkale as code owners June 20, 2023 18:57

mch2 force-pushed the disableSI branch from 856f344 to 85343e9 Compare June 20, 2023 22:45

This comment was marked as outdated.

Sign in to view

andrross reviewed Jul 14, 2023

View reviewed changes

server/src/internalClusterTest/java/org/opensearch/action/search/TransportSearchIT.java Outdated Show resolved Hide resolved

mch2 force-pushed the disableSI branch from 85343e9 to 9ab9fb5 Compare July 14, 2023 18:29

mch2 added the skip-changelog label Jul 14, 2023

mch2 added 5 commits July 14, 2023 15:11

Segment Replication - Allow shard idle when there are no replicas an …

7a6c2fa

…index. Signed-off-by: Marc Handalian <[email protected]>

Add warning when updating search.idle.after with Segment Replication …

127cf87

…enabled. Signed-off-by: Marc Handalian <[email protected]>

PR cleanup.

84b7e5b

Signed-off-by: Marc Handalian <[email protected]>

Add more tests.

dcbc2e5

Signed-off-by: Marc Handalian <[email protected]>

Remove unnecessary assertBusy

fde8361

Signed-off-by: Marc Handalian <[email protected]>

mch2 force-pushed the disableSI branch from 0a539ac to fde8361 Compare July 14, 2023 22:11

andrross approved these changes Jul 14, 2023

View reviewed changes

andrross added backport 2.x Backport to 2.x branch v2.10.0 labels Jul 14, 2023

andrross merged commit ca74aac into opensearch-project:main Jul 14, 2023

opensearch-trigger-bot bot mentioned this pull request Jul 14, 2023

[Backport 2.x] Segment Replication - Allow search idle with no replicas #8709

Merged

mch2 deleted the disableSI branch July 15, 2023 00:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Segment Replication - Allow search idle with no replicas #8173

Segment Replication - Allow search idle with no replicas #8173

mch2 commented Jun 20, 2023 •

edited

Loading

github-actions bot commented Jun 20, 2023

codecov bot commented Jun 20, 2023 •

edited

Loading

github-actions bot commented Jun 20, 2023

github-actions bot commented Jun 20, 2023

This comment was marked as outdated.

github-actions bot commented Jun 20, 2023

mch2 commented Jul 14, 2023

andrross commented Jul 14, 2023

github-actions bot commented Jul 14, 2023

github-actions bot commented Jul 14, 2023

mch2 commented Jul 14, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jul 14, 2023

Segment Replication - Allow search idle with no replicas #8173

Segment Replication - Allow search idle with no replicas #8173

Conversation

mch2 commented Jun 20, 2023 • edited Loading

Check List

github-actions bot commented Jun 20, 2023

Gradle Check (Jenkins) Run Completed with:

codecov bot commented Jun 20, 2023 • edited Loading

Codecov Report

github-actions bot commented Jun 20, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jun 20, 2023

Gradle Check (Jenkins) Run Completed with:

This comment was marked as outdated.

github-actions bot commented Jun 20, 2023

Gradle Check (Jenkins) Run Completed with:

mch2 commented Jul 14, 2023

andrross commented Jul 14, 2023

github-actions bot commented Jul 14, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jul 14, 2023

Gradle Check (Jenkins) Run Completed with:

mch2 commented Jul 14, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jul 14, 2023

Gradle Check (Jenkins) Run Completed with:

mch2 commented Jun 20, 2023 •

edited

Loading

codecov bot commented Jun 20, 2023 •

edited

Loading