Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stream uploads from http.ResponseWriter directly to host without buffering #1554

Open
ChrisSchinnerl opened this issue Sep 19, 2024 · 0 comments
Milestone

Comments

@ChrisSchinnerl
Copy link
Member

The design of the RHP4 client allows us to directly pipe the body of the http.Request all the way to the Stream we use for uploading to a host.

This will only work if the client specifies the Content-Length header but that should be the case for most uploads.
Uploads that don't will need to buffer the whole sector in memory.

As a result, we will be able to get rid of the latency before an upload actually starts since renterd no longer needs to read 40MB per chunk into memory before starting the upload. We also don't lose precious time performing erasure coding and encryption before we start uploading since that will happen incrementally as we upload to the hosts.

@ChrisSchinnerl ChrisSchinnerl converted this from a draft issue Sep 19, 2024
@ChrisSchinnerl ChrisSchinnerl added this to the v2.1.0 milestone Sep 19, 2024
@SiaFoundation SiaFoundation deleted a comment Sep 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Backlog
Development

No branches or pull requests

1 participant