-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Remote Store] Permit backed futures to prevent timeouts during upload bursts #12159
[Remote Store] Permit backed futures to prevent timeouts during upload bursts #12159
Conversation
d22dbe1
to
8005bdf
Compare
❌ Gradle check result for d22dbe1: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
2a0e26c
to
3defb7b
Compare
…transfers Signed-off-by: vikasvb90 <[email protected]>
3defb7b
to
58d730b
Compare
❌ Gradle check result for 3defb7b: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
❌ Gradle check result for 58d730b: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
.../repository-s3/src/main/java/org/opensearch/repositories/s3/GenericStatsMetricPublisher.java
Show resolved
Hide resolved
.../repository-s3/src/main/java/org/opensearch/repositories/s3/GenericStatsMetricPublisher.java
Show resolved
Hide resolved
server/src/main/java/org/opensearch/common/blobstore/BlobStore.java
Outdated
Show resolved
Hide resolved
.../repository-s3/src/main/java/org/opensearch/repositories/s3/GenericStatsMetricPublisher.java
Show resolved
Hide resolved
plugins/repository-s3/src/main/java/org/opensearch/repositories/s3/S3BlobStore.java
Show resolved
Hide resolved
plugins/repository-s3/src/main/java/org/opensearch/repositories/s3/S3RepositoryPlugin.java
Show resolved
Hide resolved
…on in repository-s3 Signed-off-by: vikasvb90 <[email protected]>
❌ Gradle check result for 8c54c68: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
❌ Gradle check result for 8c54c68: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
❌ Gradle check result for 8c54c68: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
The backport to
To backport manually, run these commands in your terminal: # Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-12159-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 c328c18f69c08b21aa8a76af270b04a70c5a8069
# Push it to GitHub
git push --set-upstream origin backport/backport-12159-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x Then, create a pull request where the |
…d bursts (opensearch-project#12159) Signed-off-by: vikasvb90 <[email protected]> (cherry picked from commit c328c18)
…d bursts (opensearch-project#12159) Signed-off-by: vikasvb90 <[email protected]>
…d bursts (#12159) (#13645) Signed-off-by: vikasvb90 <[email protected]>
…d bursts (opensearch-project#12159) Signed-off-by: vikasvb90 <[email protected]>
…d bursts (opensearch-project#12159) Signed-off-by: vikasvb90 <[email protected]>
Description
During burst of uploads happening typically in finalize recovery in cases like shrink, split and force merge where lot of segments or large segments become available for upload, large number of requests get queued up behind connection pool. This either results in timeouts due to failure in acquiring a connection or idle connection timeout where an ongoing request takes too long to read and compute data for upload which is because of high wait time for acquiring a thread in stream reader pool. It is more prevalent in async flow since main thread doesn't wait for the response and everything ends up getting submitted for upload. Both sync and async S3 SDK apis do not have a way today to handle such bursts.
This PR resolves these problems by applying natural backpressure on main thread with the help of backing permits. It also adds retries on future in case of a SDK exception or failure in acquisition of a permit. This means that in case of multi-part upload, a failing part can be independently retried.
Testing
Related Issues
Resolves #[Issue number to be closed when this PR is merged]
Check List
New functionality has been documented.New functionality has javadoc addedCommit changes are listed out in CHANGELOG.md file (See: Changelog)Public documentation issue/PR createdBy submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.