Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a compressed output format for rethinkdb export / import #249

Open
iantocristian opened this issue Feb 14, 2021 · 6 comments
Open

Add a compressed output format for rethinkdb export / import #249

iantocristian opened this issue Feb 14, 2021 · 6 comments
Labels
enhancement New feature or request not qualified The issue is not checked yet by the owners question Further information is requested

Comments

@iantocristian
Copy link

Is your feature request related to a problem? Please describe.
Dumping large databases / tables involves writing very large files to the disk, is expensive in terms of disk io.

Describe the solution you'd like
Add a new compressed output format json.gz to rethinkdb export.
Enable on the fly streaming compression of data files using zlib.

@iantocristian iantocristian added enhancement New feature or request not qualified The issue is not checked yet by the owners question Further information is requested labels Feb 14, 2021
@lsabi
Copy link
Contributor

lsabi commented Feb 14, 2021

This is an interesting (and useful) feature that surely deserves to be implemented.

As of now, we're in the middle of a revamp of this repo, because the project is flooded with old code and bad mixes of python 2 and 3. We're already working on it, in a separate repo, where we're cleaning up the messy code and start supporting only python 3 since the EOL of python 2.

Our first goal is to not break any existing code. We will then start adding features as well, like the one requested in this issue.

@iantocristian
Copy link
Author

This is something I'm experimenting with at the moment. Does it make sense to open a PR with a proposed implementation to get some feedback?

@lsabi
Copy link
Contributor

lsabi commented Feb 16, 2021

If there's already the implementation, we could try to make room for it in the new repo (temporarily private since early WIP).

You can use the python-2-to-3 branch (or master if already started working from that one). Please try to keep your code as separate as possible from the existing code, so we can extract it easily and then port it to the new codebase.

@iantocristian
Copy link
Author

Cool, I opened a PR against master for now, that's what I was working on. If it's a go I can switch to python-2-to-3.

@lsabi
Copy link
Contributor

lsabi commented Feb 19, 2021

I checked the PR rapidly, but it looks really neat. Thanks!

It will need a proper code review and we will then merge it into the new repo (as soon as we're done with the basis of it). I would keep the PR open until then, if you don't mind, do you?

@iantocristian
Copy link
Author

Sure no problem 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request not qualified The issue is not checked yet by the owners question Further information is requested
Projects
None yet
Development

No branches or pull requests

2 participants