Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add UploadedBytes to UploadOutput in feature/s3/manager #1397

Closed
shawnHartsell opened this issue Aug 30, 2021 · 4 comments
Closed

Add UploadedBytes to UploadOutput in feature/s3/manager #1397

shawnHartsell opened this issue Aug 30, 2021 · 4 comments
Labels
feature/s3/manager Pertains to S3 transfer manager HLL (feature/s3/manager). feature-request A feature should be added or improved. m Effort estimation: medium p3 This is a minor priority issue

Comments

@shawnHartsell
Copy link

shawnHartsell commented Aug 30, 2021

Is your feature request related to a problem? Please describe.

I'm migrating a service from https://github.com/minio/minio-go to the V2 SDK and using the s3Manager to handle uploads (the service is essentially an HTTP proxy that sites in front S3 where clients can stream uploads). Since we're uploading using an io.Reader (from the http request body) we don't know the size of the content being uploaded. For auditing and monitoring purposes, it would be useful if the UploadOutput struct returned this data.

Describe the solution you'd like

A field is added to UploadOutput that shows the total upload size of content

Describe alternatives you've considered

Currently, I'm working around the issue by using a custom Build middleware that's injected for each call to Upload that tracks the content-length for single and multi-part uploads (by inspecting the x-id query parameter in the http.Request).

Long term, we've also considered event based solutions such as creating a Lambda to trigger via s3:ObjectCreated:* events from S3, or creating a DynamoDB stream (our content meta-data index) that preforms a HeadObject call when an item is marked as uploaded.

@shawnHartsell shawnHartsell added feature-request A feature should be added or improved. needs-triage This issue or PR still needs to be triaged. labels Aug 30, 2021
@shawnHartsell
Copy link
Author

I imagine we could also do what we needed if #941 is implemented as well

@KaibaLopez
Copy link
Contributor

Hi @shawnHartsell ,
Sorry for the lack of response, but yea this is something we are thinking of implementing just haven't got to it yet.

@KaibaLopez KaibaLopez removed the needs-triage This issue or PR still needs to be triaged. label Dec 17, 2021
@RanVaknin RanVaknin added p3 This is a minor priority issue m Effort estimation: medium labels Nov 7, 2022
@lucix-aws lucix-aws added the feature/s3/manager Pertains to S3 transfer manager HLL (feature/s3/manager). label Oct 17, 2023
@lucix-aws
Copy link
Contributor

@shawnHartsell -- as you've indicated, #941 will cover this and that's the route we intend to take. Closing as duplicate.

@lucix-aws lucix-aws closed this as not planned Won't fix, can't repro, duplicate, stale Oct 18, 2023
@github-actions
Copy link

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature/s3/manager Pertains to S3 transfer manager HLL (feature/s3/manager). feature-request A feature should be added or improved. m Effort estimation: medium p3 This is a minor priority issue
Projects
None yet
Development

No branches or pull requests

4 participants