-
Notifications
You must be signed in to change notification settings - Fork 182
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
modules/zstd: Add support for decoding compressed blocks #1857
Open
lpawelcz
wants to merge
62
commits into
google:main
Choose a base branch
from
antmicro:zstd_compressed_block_dec
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- XLSStruct for easier handling and serializing/deserializing XLS structs - XLSChannel that serves as a dummy receiving channel - XLSMonitor that monitors transactions on an XLS channel - XLSDriver that can send data on an XLS channel - LatencyScoreboard that can measure latency between corresponding transactions on input and output buses - File-backed AXI memory python model Internal-tag: [#64075] Signed-off-by: Krzysztof Obłonczek <[email protected]>
Signed-off-by: Pawel Czarnecki <[email protected]>
Co-authred-by: Pawel Czarnecki <[email protected]> Co-authred-by: Robert Winkler <[email protected]> Signed-off-by: Michal Czyz <[email protected]> Signed-off-by: Pawel Czarnecki <[email protected]> Signed-off-by: Robert Winkler <[email protected]>
Signed-off-by: Pawel Czarnecki <[email protected]>
Signed-off-by: Pawel Czarnecki <[email protected]>
Signed-off-by: Pawel Czarnecki <[email protected]>
Signed-off-by: Pawel Czarnecki <[email protected]>
Signed-off-by: Maciej Torhan <[email protected]>
Signed-off-by: Pawel Czarnecki <[email protected]>
Signed-off-by: Pawel Czarnecki <[email protected]>
Signed-off-by: Pawel Czarnecki <[email protected]>
Signed-off-by: Maciej Torhan <[email protected]>
Remove references to buffer structs as those are not used anywhere Signed-off-by: Pawel Czarnecki <[email protected]>
Signed-off-by: Pawel Czarnecki <[email protected]>
Signed-off-by: Pawel Czarnecki <[email protected]>
Co-authored-by: Pawel Czarnecki <[email protected]> Co-authored-by: Robert Winkler <[email protected]> Signed-off-by: Maciej Torhan <[email protected]> Signed-off-by: Pawel Czarnecki <[email protected]> Signed-off-by: Robert Winkler <[email protected]>
Signed-off-by: Krzysztof Oblonczek <[email protected]>
Signed-off-by: Maciej Torhan <[email protected]>
Co-authored-by: Pawel Czarnecki <[email protected]> Signed-off-by: Maciej Torhan <[email protected]> Signed-off-by: Pawel Czarnecki <[email protected]>
Co-authored-by: Pawel Czarnecki <[email protected]> Co-authored-by: Robert Winkler <[email protected]> Signed-off-by: Maciej Torhan <[email protected]> Signed-off-by: Pawel Czarnecki <[email protected]> Signed-off-by: Robert Winkler <[email protected]>
Co-authored-by: Pawel Czarnecki <[email protected]> Signed-off-by: Robert Winkler <[email protected]> Signed-off-by: Pawel Czarnecki <[email protected]>
This reverts commit 04ad379225b706ddf492d440c673e77348d7a409.
Signed-off-by: Pawel Czarnecki <[email protected]>
Signed-off-by: Pawel Czarnecki <[email protected]>
Internal-tag: [#67096] Signed-off-by: Pawel Czarnecki <[email protected]>
Signed-off-by: Maciej Torhan <[email protected]> Signed-off-by: Krzysztof Obłonczek <[email protected]>
Signed-off-by: Krzysztof Obłonczek <[email protected]>
Internal-tag: [#66955] Signed-off-by: Pawel Czarnecki <[email protected]>
Signed-off-by: Pawel Czarnecki <[email protected]>
* Fix byte ordering when receiving a series of non-empty packets * Adjust MemReader DSLX tests Internal-tag: [#67272] Signed-off-by: Pawel Czarnecki <[email protected]>
Internal-tag: [#67272] Signed-off-by: Pawel Czarnecki <[email protected]>
Signed-off-by: Pawel Czarnecki <[email protected]>
Add a buffering proc that implements the desired API of the final buffering mechanism. A more efficient implementation will be provided later. Internal-tag: [#55149] Signed-off-by: Robert Winkler <[email protected]>
Internal-tag: [#57353] Signed-off-by: Robert Winkler <[email protected]>
Internal-tag: [#57353] Signed-off-by: Robert Winkler <[email protected]>
Co-authored-by: Krzysztof Obłonczek <[email protected]> Signed-off-by: Robert Winkler <[email protected]> Signed-off-by: Krzysztof Obłonczek <[email protected]>
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
Co-authored-by: Robert Winkler <[email protected]> Co-authored-by: Pawel Czarnecki <[email protected]> Signed-off-by: Robert Winkler <[email protected]> Signed-off-by: Pawel Czarnecki <[email protected]> Signed-off-by: Krzysztof Obłonczek <[email protected]>
Internal-tag: [#68052] Signed-off-by: Krzysztof Obłonczek <[email protected]>
Internal-tag: [#58422] Signed-off-by: Ryszard Rozak <[email protected]>
Internal-tag: [#58557] Co-authored-by: Ryszard Rozak <[email protected]> Co-authored-by: Maciej Torhan <[email protected]> Signed-off-by: Robert Winkler <[email protected]>
Co-authored-by: Pawel Czarnecki <[email protected]> Co-authored-by: Ryszard Rozak <[email protected]> Signed-off-by: Maciej Torhan <[email protected]> Signed-off-by: Pawel Czarnecki <[email protected]> Signed-off-by: Ryszard Rozak <[email protected]>
Signed-off-by: Maciej Torhan <[email protected]> Co-authored-by: Maciej Dudek <[email protected]>
Add 3:1 demultiplexer for RamModel Signed-off-by: Robert Winkler <[email protected]>
Signed-off-by: Robert Winkler <[email protected]>
Internal-tag: [#69203] Signed-off-by: Pawel Czarnecki <[email protected]>
Signed-off-by: Robert Winkler <[email protected]>
Signed-off-by: Robert Winkler <[email protected]>
Signed-off-by: Pawel Czarnecki <[email protected]>
Co-authored-by: Robert Winkler <[email protected]> Signed-off-by: Robert Winkler <[email protected]> Signed-off-by: Krzysztof Obłonczek <[email protected]>
Co-authored-by: Robert Winkler <[email protected]> Signed-off-by: Robert Winkler <[email protected]> Signed-off-by: Krzysztof Obłonczek <[email protected]>
Signed-off-by: Robert Winkler <[email protected]>
Signed-off-by: Robert Winkler <[email protected]>
Signed-off-by: Maciej Torhan <[email protected]>
Signed-off-by: Robert Winkler <[email protected]>
Signed-off-by: Robert Winkler <[email protected]>
tmichalak
force-pushed
the
zstd_compressed_block_dec
branch
from
January 15, 2025 20:51
47ab08c
to
50f6140
Compare
lpawelcz
force-pushed
the
zstd_compressed_block_dec
branch
from
January 16, 2025 07:19
50f6140
to
36bc838
Compare
This was referenced Jan 20, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR extends the
ZstdDecoder
with support for decoding compressed blocks.It is based on the following two PRs that must be merged before this one:
It also supersedes PRs:
The decoder is capable of decoding RAW and RLE literals as well as sequences with predefined FSE tables.
A suite of DSLX tests comprising unit tests of all underlying procs and an integration test was prepared.
The integration test, similarly as in #1654, first generates a random valid ZSTD frame with compressed blocks and expected decoded output. Test data is then converted to a DSLX file (example) that is imported by the integration tests file.
At the beginning of the test, the default FSE decoding tables are filled with default distributions taken from RFC 8878 section 3.1.1.3.2.2. Default Distributions . Next, the encoded frame is loaded to the system memory and the decoder is configured through a set of CSRs to start the decoding process. The decoder starts the operation and writes the decoded frame back into the output buffer in the system memory. Once it finishes, it sends a pulse on the
notify
channel signaling the end of the decoding. The output of the decoder is compared against the decoding result from the reference library.The PR introduces among others:
CompressedBlockDecoder
- manages bothSequenceDecoder
andLiteralsDecoder
to enable compress block decoding. Integrated with the top-levelZstdDecoder
SequenceDecoder
- responsible for decoding sequence sections of the compressed blocksFseDecoder
- introduced as the core part of theSequenceDecoder
RefillingShiftBuffer
- used for storing and outputting in forward and backward fashion an arbitrary amount of bits required by the FSE decoderLiteralsDecoder
- capable of decoding RAW, RLE and Huffman-coded literalsHuffmanDecoder
- used in decoding huffman-coded literals. Decoded Huffman trees are then used to decode one or four Huffman-coded streams.CommandConstructor
- this proc is responsible for sending packets with decoded sequences and literals to theSequenceExecutor
procRamMux
andRamDemux
- procs used for handling requests/responses to multiple memory models. The procs interface with 3 separate memory buffers for FSE decoding tables.