Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make index.soft_deletes.enabled a static setting #3054

Open
saikaranam-amazon opened this issue Apr 25, 2022 · 1 comment
Open

Make index.soft_deletes.enabled a static setting #3054

saikaranam-amazon opened this issue Apr 25, 2022 · 1 comment
Labels
enhancement Enhancement or improvement to existing feature or request Indexing & Search

Comments

@saikaranam-amazon
Copy link
Member

saikaranam-amazon commented Apr 25, 2022

Is your feature request related to a problem? Please describe.
Currently, this(index.soft_deletes.enabled) is a static setting and is set during the index creation time. In addition to being static setting, It is marked as “Final” ( and OS will not allow any changes over the lifecycle of the index within the cluster).

Describe the solution you'd like
Make this as a static setting and allow setting update on closed index (like other static index settings).

@saikaranam-amazon saikaranam-amazon added enhancement Enhancement or improvement to existing feature or request untriaged labels Apr 25, 2022
@mgodwan
Copy link
Member

mgodwan commented May 20, 2024

@saikaranam-amazon We may need a safe state where this can be enabled/disabled on an index. Will there be any prerequisites to ensure that this can be changed on a close index itself (e.g. recovery flow, flushed ops, cross cluster replication enablement, etc)?

As an OpenSearch operator, I've seen multiple issues with soft deletes in general, and have seen the need to disable it at times (which required re-indexing the data due to requirement of creating new index)

  1. Handling DV Updates has been costly for segments with high amount of deletes [Performance] Possible 2.x search regression related to mandatory soft deletes  #7621 (comment) (Workaround has been to reduce the deletes percentage for segments or reducing Translog size to reduced retained documents leading to more background activities)
  2. Change in behavior of deleted document count due to tombstone entries [BUG] Incorrect docs.deleted count with Soft Delete Enabled #13725
  3. Performance and slight storage impact due to _recovery_source as highlighted here [Feature Request] Capability to remove _recovery_source per field #13490

Adding @backslasht @shwetathareja @msfroh @nknize for thoughts on this.
I'm wondering if we should evaluate to disable this by default in general in the next major version at least, unless we've a way to address the known issues.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Enhancement or improvement to existing feature or request Indexing & Search
Projects
Status: 🆕 New
Development

No branches or pull requests

3 participants