Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

S3: Expose both part size and full size in GET/HEAD requests #4169

Closed
vincer opened this issue Jun 18, 2024 · 2 comments
Closed

S3: Expose both part size and full size in GET/HEAD requests #4169

vincer opened this issue Jun 18, 2024 · 2 comments
Assignees
Labels
feature-request This issue requests a feature. s3

Comments

@vincer
Copy link

vincer commented Jun 18, 2024

Describe the feature

I'd like to inspect an object and get its part size as well as its total size in a single request. The API does expose this information. If a HEAD request (or GET) is made with partNumber=1, then the response will have a content-length header that has the size of the part as well as a content-range header that also has the size of the whole object. e.g.

'content-range': 'bytes 0-400/5000'

In this example the full size of the object is 5000.

Currently boto3 doesn't have a way to get to this information easily with a single request. It only exposes the part size and not the whole object size. My workaround looks like this:

def get_full_size(parsed: dict, **kwargs):
    content_range = parsed["ResponseMetadata"]["HTTPHeaders"]["content-range"]
    full_size = int(content_range.split("/")[1])
    print(f"full size: {full_size}")

# The handler needs to be registered/unregistered every time
# to be able to access the full_size from the correct context each time.
client.meta.events.register(
    "after-call.s3.HeadObject",
    get_full_size,
    unique_id="get_full_size",
)
info = client.head_object(
    Bucket=bucket, Key=key, PartNumber=1
)
client.meta.events.unregister(
    "after-call.s3.HeadObject",
    unique_id="get_full_size",
)```

### Use Case

Getting both pieces of info in a single request.

### Proposed Solution

Expose part size as well as full object size when making `head_object` calls as well as in the `Object` resource so that only a single HEAD request is made for that information.

### Other Information

_No response_

### Acknowledgements

- [ ] I may be able to implement this feature request
- [ ] This feature might incur a breaking change

### SDK version used

1.34.69

### Environment details (OS name and version, etc.)

MacOS 14.5
@vincer vincer added feature-request This issue requests a feature. needs-triage This issue or PR still needs to be triaged. labels Jun 18, 2024
@tim-finnigan tim-finnigan self-assigned this Jun 21, 2024
@tim-finnigan tim-finnigan added the investigating This issue is being investigated and/or work is in progress to resolve the issue. label Jun 21, 2024
@tim-finnigan
Copy link
Contributor

Thanks for the feature request. There is a feature freeze on resources so your request involving the Object resource would not be considered. Changes involving the HeadObject API would need to be implemented by the S3 team. Please consider opening an issue in our cross-SDK repository if you'd like someone to reach out to S3 on your behalf. The content-length and content-range is already returned in the HeadObject response so if you're request is for something beyond that, then please explain more about what that involves and what use case you are trying to address.

@tim-finnigan tim-finnigan closed this as not planned Won't fix, can't repro, duplicate, stale Jun 21, 2024
@tim-finnigan tim-finnigan added s3 and removed investigating This issue is being investigated and/or work is in progress to resolve the issue. needs-triage This issue or PR still needs to be triaged. labels Jun 21, 2024
Copy link

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature-request This issue requests a feature. s3
Projects
None yet
Development

No branches or pull requests

2 participants