Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

VStreamer: For larger compressed transaction payloads, stream the internal contents #17239

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

VStreamer: For larger compressed transaction payloads, stream the internal contents #17239

wants to merge 5 commits into from

Commits on Nov 15, 2024

  1. For larger payloads, stream the internal contents 

    For larger payloads (> ZstdInMemoryDecompressorMaxSize) we were already
streaming the internal events as we decompressed the payload, but in
the vstreamer we were still reading the entire contents into memory
before sending them to the consumer (vplayer).

With this we stream the internal contents all the way from the binlog
consumer to the vstream consumer so that we do not need to hold the
entire contents, which can be 10s of GiBs, in memory all at once.

Signed-off-by: Matt Lord <[email protected]>
    @mattlord
    mattlord committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    92cf593 View commit details
    Browse the repository at this point in the history

  2. Check for nil bufferAndTransmit function 

    Signed-off-by: Matt Lord <[email protected]>
    @mattlord
    mattlord committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    c705498 View commit details
    Browse the repository at this point in the history

  3. Remove debug logs 

    Signed-off-by: Matt Lord <[email protected]>
    @mattlord
    mattlord committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    4b96ef8 View commit details
    Browse the repository at this point in the history

  4. Minor changes from self review 

    Signed-off-by: Matt Lord <[email protected]>
    @mattlord
    mattlord committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    71e2f5b View commit details
    Browse the repository at this point in the history

Commits on Nov 16, 2024

  1. Merge remote-tracking branch 'origin/main' into vstream_stream_large_… 

    …cmp_trx_payload_contents

Signed-off-by: Matt Lord <[email protected]>
    @mattlord
    mattlord committed Nov 16, 2024
    Configuration menu
    Copy the full SHA
    0bc59bc View commit details
    Browse the repository at this point in the history