Recommended way to connect an io.Writer to the S3 download manger #1735
-
In order to maximise download speeds, I'm trying to use the S3 download manager to download tar files from S3, and pipe them into a Standard input is clearly an Because of this, at present I'm using the plain Are there any utility classes or something I've missed in the SDK to make this possible? ( |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 3 replies
-
It sounds like you're looking to download an object from S3, then stream the result to the standard output. If so, this is not a feature that is currently supported by the Downloader. The reasoning behind the Downloader requiring a In addition, when you mention "standard input" did you mean to use "standard output"? Could you go into more detail about how you'd like to use the Downloader to stream the result to a standard output? Since the download could download parts of an object in any order, the full object is not available to stream until all parts are downloaded, and the Downloader's Download method returns. |
Beta Was this translation helpful? Give feedback.
-
Hello! Reopening this discussion to make it searchable. |
Beta Was this translation helpful? Give feedback.
It sounds like you're looking to download an object from S3, then stream the result to the standard output. If so, this is not a feature that is currently supported by the Downloader.
GetObject
operation method is the best way to do that at the moment. I think to support this the Downloader would need to have a "SequentialDownload" method/behavior that would still download object parts, but one at a time in order instead of out-of-order concurrently. If so a Feature Request would be a great to capture this use case.The reasoning behind the Downloader requiring a
io.WriterAt
is that it is the only interface that allows concurrent writes to the underlying value. Aio.Writer
that does not s…