Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compression #112

Merged
merged 1 commit into from
Feb 15, 2024
Merged

Compression #112

merged 1 commit into from
Feb 15, 2024

Conversation

fulmicoton
Copy link
Contributor

@fulmicoton fulmicoton commented Feb 12, 2024

Adding ZSTD compression to chitchat's Deltas.

The difficulty in this PR is coming from the fact that
we need to observe the maximum transmissible unit,
and due to the fact that we want to keep the simplficity
of using the same `MessageDigest` object when
sending and emitting messages.

The solution I went for requires to compress things twice.
When building our message object, we rely on a stream
compressor that makes it possible to get an upperbound of the
resulting payload after we insert an arbitrary serializable object.

We then decompose the delta into as many mutations.

I had to refactor the delta object, so that we have a proper bijection
between a delta object and the list of mutations that were used to
build it.

The delta object hence encodes the order in which nodes, and values
were inserted.
The point there is to ensure that the actual serialization will
yield the same size as the "simulated one" and hence respect the mtu.

The code is a bit simpler that way too.

@fulmicoton fulmicoton force-pushed the compression2 branch 3 times, most recently from 167c913 to ce8b913 Compare February 12, 2024 07:14
chitchat/src/lib.rs Outdated Show resolved Hide resolved
chitchat/src/lib.rs Outdated Show resolved Hide resolved
@fulmicoton fulmicoton force-pushed the compression2 branch 2 times, most recently from db47399 to eaff8b9 Compare February 13, 2024 09:12
@fulmicoton fulmicoton changed the title Compression2 Compression Feb 13, 2024
@fulmicoton fulmicoton force-pushed the compression2 branch 11 times, most recently from 4dab87d to e7c5728 Compare February 13, 2024 13:17
@fulmicoton fulmicoton marked this pull request as ready for review February 13, 2024 13:18
@fulmicoton fulmicoton requested a review from guilload February 13, 2024 13:18
The difficulty in this PR is coming from the fact that
we need to observe the maximum transmissible unit,
and due to the fact that we want to keep the simplficity
of using the same `MessageDigest` object when
sending and emitting messages.

The solution I went for requires to compress things twice.
When building our message object, we rely on a stream
compressor that makes it possible to get an upperbound of the
resulting payload after we insert an arbitrary serializable object.

We then decompose the delta into as many mutations.

I had to refactor the delta object, so that we have a proper bijection
between a delta object and the list of mutations that were used to
build it.

The delta object hence encodes the order in which nodes, and values
were inserted.
The point there is to ensure that the actual serialization will
yield the same size as the "simulated one" and hence respect the mtu.

The code is a bit simpler that way too.

Closes #72
@fulmicoton fulmicoton merged commit 15f3641 into main Feb 15, 2024
2 checks passed
fulmicoton added a commit that referenced this pull request Feb 15, 2024
The difficulty in this PR is coming from the fact that
we need to observe the maximum transmissible unit,
and due to the fact that we want to keep the simplficity
of using the same `MessageDigest` object when
sending and emitting messages.

The solution I went for requires to compress things twice.
When building our message object, we rely on a stream
compressor that makes it possible to get an upperbound of the
resulting payload after we insert an arbitrary serializable object.

We then decompose the delta into as many mutations.

I had to refactor the delta object, so that we have a proper bijection
between a delta object and the list of mutations that were used to
build it.

The delta object hence encodes the order in which nodes, and values
were inserted.
The point there is to ensure that the actual serialization will
yield the same size as the "simulated one" and hence respect the mtu.

The code is a bit simpler that way too.

Closes #72
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants