Skip to content

Commit

Permalink
Adding ZSTD compression to chitchat's Deltas.
Browse files Browse the repository at this point in the history
The difficulty in this PR is coming from the fact that
we need to observe the maximum transmissible unit,
and due to the fact that we want to keep the simplficity
of using the same `MessageDigest` object when
sending and emitting messages.

The solution I went for requires to compress things twice.
When building our message object, we rely on a stream
compressor that makes it possible to get an upperbound of the
resulting payload after we insert an arbitrary serializable object.

We then decompose the delta into as many mutations.

I had to refactor the delta object, so that we have a proper bijection
between a delta object and the list of mutations that were used to
build it.

The delta object hence encodes the order in which nodes, and values
were inserted.
The point there is to ensure that the actual serialization will
yield the same size as the "simulated one" and hence respect the mtu.

The code is a bit simpler that way too.
  • Loading branch information
fulmicoton committed Feb 13, 2024
1 parent c3a5baf commit e2b0ec3
Show file tree
Hide file tree
Showing 11 changed files with 918 additions and 345 deletions.
10 changes: 9 additions & 1 deletion chitchat/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -16,14 +16,22 @@ bytes = "1"
itertools = "0.12"
rand = { version = "0.8", features = ["small_rng"] }
serde = { version = "1", features = ["derive"] }
tokio = { version = "1.28.0", features = ["net", "sync", "rt-multi-thread", "macros", "time"] }
tokio = { version = "1.28.0", features = [
"net",
"sync",
"rt-multi-thread",
"macros",
"time",
] }
tokio-stream = { version = "0.1", features = ["sync"] }
tracing = "0.1"
zstd = "0.13"

[dev-dependencies]
assert-json-diff = "2"
mock_instant = "0.3"
tracing-subscriber = "0.3"
proptest = "1.4"

[features]
testsuite = []
Loading

0 comments on commit e2b0ec3

Please sign in to comment.