You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Opensearch indexes merge Lucene segments in the background or allowing the user to force merge the segments.
If a user tries to delete an index when a merge is ongoing, it fails to delete the index and causes a node drop as the delete operation could not be performed on the node.
Run an ingestion load using opensearch-benchmark workloads
Sample command: opensearch-benchmark execute-test \ --target-hosts='http://<host>:9200' \ --workload=nyc_taxis \ --include-tasks=create-index,index \ --pipeline=benchmark-only \ --kill-running-processes &
Start a force merge for the index using the following command - curl -X POST 'http://localhost:9200/nyc_taxis/_forcemerge?max_num_segments=1'
When the merge is ongoing, try to delete the index using - curl -X DELETE 'http://localhost:9200/nyc_taxis'
The delete operation fails.
Expected behavior
Delete operation should stop any merges and should be able to take precedence over the merge operation, ideally cancel the merge operation
Merge operations can be wrapped with CancellableThreads or executed as CancellableTask instances to interrupt the merge
Possible solutions
Short term solution
This problem is very similar to the long running operation of snapshotting an index - which has protections added in within the core code base here.
Similar to this approach, the proposed solution is to prevent index deletions in case of an ongoing merge operation.
This will introduce safeguarding against node drops by adding in delete restrictions on the index.
/**
* Delete some indices from the cluster state.
*/
public ClusterState deleteIndices(ClusterState currentState, Set<Index> indices) {
final Metadata meta = currentState.metadata();
final Set<Index> indicesToDelete = new HashSet<>();
final Map<Index, DataStream> backingIndices = new HashMap<>();
for (Index index : indices) {
...
}
// Check if index deletion conflicts with any running snapshots
Set<Index> snapshottingIndices = SnapshotsService.snapshottingIndices(currentState, indicesToDelete);
if (snapshottingIndices.isEmpty() == false) {
throw new SnapshotInProgressException(
"Cannot delete indices that are being snapshotted: "
+ snapshottingIndices
+ ". Try again after snapshot finishes or cancel the currently running snapshot."
);
}
Set<Index> mergingIndices = clusterService.mergingIndices(currentState, indicesToDelete);
if (mergingIndices.isEmpty() == false) {
throw new MergeInProgressException(
"Cannot delete indices that are being merged: "
+ mergingIndices
+ ". Try again after merge finishes."
);
}
...
...
}
Long term solution
The long term solution would require the merge operation to follow the constructs of a CancellableTask , where in whenever a request for deletion comes in, we should be able to cancel the ongoing merge operation as deletion takes precedence over a merge operation.
Additional Details
Plugins
Please list all plugins currently enabled.
Describe the bug
Opensearch indexes merge Lucene segments in the background or allowing the user to force merge the segments.
If a user tries to delete an index when a merge is ongoing, it fails to delete the index and causes a node drop as the delete operation could not be performed on the node.
Related component
Indexing
To Reproduce
Sample command:
opensearch-benchmark execute-test \ --target-hosts='http://<host>:9200' \ --workload=nyc_taxis \ --include-tasks=create-index,index \ --pipeline=benchmark-only \ --kill-running-processes &curl -X POST 'http://localhost:9200/nyc_taxis/_forcemerge?max_num_segments=1'curl -X DELETE 'http://localhost:9200/nyc_taxis'The delete operation fails.
Expected behavior
CancellableThreadsor executed asCancellableTaskinstances to interrupt the mergePossible solutions
Short term solution
This problem is very similar to the long running operation of snapshotting an index - which has protections added in within the core code base here.
Similar to this approach, the proposed solution is to prevent index deletions in case of an ongoing merge operation.
This will introduce safeguarding against node drops by adding in delete restrictions on the index.
Long term solution
The long term solution would require the merge operation to follow the constructs of a
CancellableTask, where in whenever a request for deletion comes in, we should be able to cancel the ongoing merge operation as deletion takes precedence over a merge operation.Additional Details
Plugins
Please list all plugins currently enabled.
Screenshots
Logs
Host/Environment (please complete the following information):
Additional context
Add any other context about the problem here.
The text was updated successfully, but these errors were encountered: