Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow not copying the dictionary #88

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Conversation

JeWe37
Copy link

@JeWe37 JeWe37 commented Aug 25, 2023

For large dictionary files, the documentation recommends simply using mmap to open them. This works, however as internally open-vcdiff always copies this data into the VCDiffEngine immediately, the full file has to be present in memory, without being able to take advantage of the kernel's smart caching. This can easily lead to OOM for large files.

The change here is fairly minor, essentially it just allows to explicitly specify that the data does not need to be copied and to assume that its lifetime is sufficient.

@google-cla
Copy link

google-cla bot commented Aug 25, 2023

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

@JeWe37
Copy link
Author

JeWe37 commented Aug 26, 2023

If anyone is even looking at this anymore, the memory requirements can be further reduced by increasing the kBlockSize and by decreasing the hash table size in this line where using a collision probability of 1/4 seems to work fine. This should probably be configurable dynamically.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant