Feature request: be able to pass an iterable of bytes to upload_fileobj (or other function)

[michalc]

Hi 👋

This is about the upload_fileobj function that accepts file-like objects for upload to S3.

A few Python libraries seem to prefer as output iterables of bytes rather than file-like objects. For example:

• HTTPX - some discussion at Why does stream not return a file-like object (this breaks compatibility) encode/httpx#2296
• Requests - does give access to a file-like object referred to as "raw" as https://requests.readthedocs.io/en/latest/user/quickstart/#raw-response-content, but I suspect this is very raw - so say if the response uses chunked transfer encoding, this is before de-chunking, and so probably not what is wanted to be saved to S3. Otherwise, it offers an "iter_content" function https://requests.readthedocs.io/en/latest/api/#requests.Response.iter_content that gives an iterable of bytes
• stream-zip - some discussion at Add file like object interface uktrade/stream-zip#84 (full disclosure: I maintain this)

Iterables of bytes can always be converted to file-like objects, like at https://stream-zip.docs.trade.gov.uk/output-examples/#file-like-object. But I wonder, should there be an upload function in boto3 itself that accepts iterables of bytes?

[michalc]

In case helpful: have now published https://github.com/uktrade/to-file-like-obj that converts iterables of bytes to file-like objects, suitable for upload_fileobj