[Backport 2.x] Honor max segment size during only_expunge_deletes #10603
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
msfroh
changed the title
Gradle Check (Jenkins) Run Completed with:
|
Compatibility status:Checks if related components are compatible with change f052d94 Incompatible componentsIncompatible components: [https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/performance-analyzer.git] Skipped componentsCompatible componentsCompatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/ml-commons.git] |
Gradle Check (Jenkins) Run Completed with:
|
Gradle Check (Jenkins) Run Completed with:
|
msfroh
force-pushed
the
backport/backport-10036-to-2.x
branch
from
abb99b5 to
c13ab45
Compare
Gradle Check (Jenkins) Run Completed with:
|
msfroh
force-pushed
the
backport/backport-10036-to-2.x
branch
from
c13ab45 to
85118f4
Compare
Gradle Check (Jenkins) Run Completed with:
|
msfroh
force-pushed
the
backport/backport-10036-to-2.x
branch
from
85118f4 to
d84c5c7
Compare
Gradle Check (Jenkins) Run Completed with:
|
Gradle Check (Jenkins) Run Completed with:
|
Gradle Check (Jenkins) Run Completed with:
|
Gradle Check (Jenkins) Run Completed with:
|
msfroh
requested review from
reta,
anasalkouz,
andrross,
Bukhtawar,
CEHENKLE,
dblock,
gbbafna,
setiah,
kartg,
kotwanikunal,
mch2,
nknize,
owaiskazi19,
peternied and
Rishikesh1159
as code owners
msfroh
requested review from
ryanbogan,
saratvemulapalli,
shwetathareja,
dreamer-89,
VachaShah,
dbwiddis,
sachinpkale and
sohami
as code owners
|
Sigh... on the last run, Gradle Check succeeded, but GitHub didn't see it. I'm going to retry yet again. |
Gradle Check (Jenkins) Run Completed with:
|
Gradle Check (Jenkins) Run Completed with:
|
Gradle Check (Jenkins) Run Completed with:
|
msfroh
force-pushed
the
backport/backport-10036-to-2.x
branch
from
d84c5c7 to
e0fec54
Compare
reta
approved these changes
Oct 23, 2023
Gradle Check (Jenkins) Run Completed with:
|
Gradle Check (Jenkins) Run Completed with:
|
…t#10036) (cherry picked from commit 1dde018) Signed-off-by: Michael Froh <[email protected]>
msfroh
force-pushed
the
backport/backport-10036-to-2.x
branch
from
e0fec54 to
f052d94
Compare
Gradle Check (Jenkins) Run Completed with:
|
Gradle Check (Jenkins) Run Completed with:
|
Codecov Report
@@ Coverage Diff @@
## 2.x #10603 +/- ##
============================================
- Coverage 70.93% 70.92% -0.02%
- Complexity 58731 58739 +8
============================================
Files 4844 4844
Lines 277472 277472
Branches 40705 40705
============================================
- Hits 196828 196799 -29
- Misses 63860 63961 +101
+ Partials 16784 16712 -72
|
4 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Manual backport of #10036
There are latency-sensitive users for whom the default 20% allowable accumulation of deletes is too high, so they force merge with "only_expunge_deletes".
Unfortunately, this has ignored the max segment size, so despite the word "only" in there, they get the terrible side-effect of producing segments larger than the max segment size. These oversized segments are unlikely to participate in future merges, until they have so many deletes that they could drop back below the max segment size, likely making the whole "too many deletes" problem even worse.
This commit fixes that so that "only_expunge_deletes" ONLY EXPUNGES DELETES without creating oversized segments.
Related Issues
N/A
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.