Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

S3 uploader: switch to aws-sdk-go-v2 #39248

Closed
zmb3 opened this issue Mar 12, 2024 · 3 comments
Closed

S3 uploader: switch to aws-sdk-go-v2 #39248

zmb3 opened this issue Mar 12, 2024 · 3 comments
Assignees
Labels
aws Used for AWS Related Issues. bug c-ju Internal Customer Reference

Comments

@zmb3
Copy link
Collaborator

zmb3 commented Mar 12, 2024

We have observed 503 errors in the auth uploader's CompleteMultipartUpload calls.

As per https://repost.aws/knowledge-center/http-5xx-errors-s3:

The error code 500 Internal Error indicates that Amazon S3 can't handle the request at that time. The error code 503 Slow Down typically indicates that the number of requests to your S3 bucket is high.

This same post says that:

All AWS SDKs have a built-in retry mechanism with an algorithm that uses exponential backoff.

However, it looks like there's a long-standing issue in aws-sdk-go where the SDK does not throttle S3 requests properly: aws/aws-sdk-go#3977

Edit: aws-sdk-go-v2 correctly implements throttled retries in this case. We should use it here.

@zmb3 zmb3 added bug c-ju Internal Customer Reference aws Used for AWS Related Issues. labels Mar 12, 2024
@rosstimothy
Copy link
Contributor

Does aws-sdk-go-v2 suffer from the same problem?

@zmb3
Copy link
Collaborator Author

zmb3 commented Mar 14, 2024

Good thought, @rosstimothy.

No, it looks like aws-sdk-go-v2 does implement retries and throttling for this case: https://github.com/aws/aws-sdk-go-v2/blob/49b368e9d7a38a2373c833be135270e5390c2b41/aws/retry/standard.go#L72

V1 for reference: https://github.com/aws/aws-sdk-go/blob/53e4759915361d72a33be783d5e878a63d85f807/aws/request/retryer.go#L84-L96

@zmb3 zmb3 changed the title S3 uploader: add throttling S3 uploader: switch to aws-sdk-go-v2 Mar 14, 2024
@rudream rudream self-assigned this Jun 21, 2024
@zmb3
Copy link
Collaborator Author

zmb3 commented Aug 24, 2024

Closed by #44728

@zmb3 zmb3 closed this as completed Aug 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
aws Used for AWS Related Issues. bug c-ju Internal Customer Reference
Projects
None yet
Development

No branches or pull requests

3 participants