Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

create zip/tgz on special dataset's endpoint #35

Open
TomasKulhanek opened this issue Mar 8, 2018 · 0 comments
Open

create zip/tgz on special dataset's endpoint #35

TomasKulhanek opened this issue Mar 8, 2018 · 0 comments

Comments

@TomasKulhanek
Copy link
Member

I’d add some my ideas about zip/tgz of RAW data of repository.
I propose to consider streaming the data instead of ZIP/TGZ them in batch thus a zip/tgz is made on demand
Reason to do that

  1. User can select specific folders/files to be included in the zip stream.
  2. No need to store extremely big zip on server, no need to wait until big archive is made
  3. If you need to save space on repository storage, there can be configured compression on filesystem transparently – there are tools like BTRFS, no need to implement it on level of D6.2 code base
  4. Streaming is simple – made prototype – cgi script which compress the user’s directory on demand and sends stream directly to http session channel.
  5. Such stream can be redirected directly to other service: virtual folder storage.
@TomasKulhanek TomasKulhanek added this to the D6.2 Repository Instance milestone Mar 8, 2018
@TomasKulhanek TomasKulhanek changed the title zip/tgz of dataset's RAW data files create zip/tgz on special dataset's endpoint Mar 20, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants