[NEW] Support network compression on valkey, with possible algorithms like lz4, zstd, gzip.

[rohan-changejar]

Significant improvement when dealing with huge payloads and reduce network bandwidth costs

With current case, the network usage in cases of huge payload sizes will eventually end up costing significant amounts of money and possible hit on performance too

Configurable network compression parameters

Give an option with parameters which can be enabled and clients like redission which promise support for valkey can leverage this.

Any additional information that is relevant to the feature request.

Although not directly related, posting here some references to network compression being implemented on DBs like mongo
https://www.mongodb.com/developer/products/mongodb/mongodb-network-compression/

[msnaru21]

Was looking for exactly the same for my project. Would be great to have this

[madolson]

Love this idea!

@hpatro investigated this a bit in Redis with redis/redis#6809, but that issue was originally more focused on moving away from lzf.

[zuiderkwast]

I like this idea too. For the related topics, we already have #1223.

Some questions to discuss:

1. How can clients negotiate this? A command like CLIENT COMPRESSION LZ4 to switch the entire command and reply stream to be compressed?
2. How to handle the dependency if we add more compression algorithms. So far, we have included small deps inside our code base. For an algorithm like lz4 we'd only need a sincle .c and .h, which I think is fine. (An alternative is to require it to be installed in the system when compiling, but then we may want to make it optional at compile time.)

[asafpamzn]

@rohan-changejar , why cannot you compress the payload at the application side?
For simple GET/SET workload it will work great. The main problem is read modify write commands like SETBIT that won't work and for them the server has to decompress and recompress the data.
The main advantage of compressing at the application side is that you also save memory at the server.

Can you please share if this blocks you from using compression/decompression at the application level. And I think that we should come up with solution that does the compression/decompression by the application and the server has to perform decompression/compression at the server side for read/write commands. I made a design for this problem at the past, if this is the direction I can share it.

[rohan-changejar]

@asafpamzn having network compression eliminates the need for complexity of changing application code and also makes the same data available with and without compression to multiple clients. Considering compression being a task which could impair performance on some systems, having the choice not to use compression will be a major advantage.
Having this on the network layer will also eliminate the need to build application specific logic and also spend time to standardise protocols and methods across languages.
From a user perspective, this will become a simple optional turnkey change, which will not need internal changes in particular.

[asafpamzn]

@rohan-changejar ,
That makes sense. I understand the use case now. The other option is for the client and server to handle it and make it transparent for the application and then the application does not care how the data is stored and can get the benefit from memory saving if we want to compress only the values. But, as said it is more complex and requires additional logic at the server side.

[sarthakaggarwal97]

How to handle the dependency if we add more compression algorithms.

It's a good point! Also, how would we maintain these dependencies with time if the dependencies are inside our code base?