Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🎨 RLE indices no longer materialise bitsets #434

Merged

Conversation

AshleyRoll
Copy link
Contributor

@AshleyRoll AshleyRoll commented Dec 1, 2023

The previous interim implementation of RLE encoded message indices would decode data into fully materialised bitsets in order to fit into the existing infrastructure easily.

This commit now eliminates the need to materialise a bitset to compute which callbacks are to be executed for a given message.

The rle_intersect type is a proxy object that collects together multiple rle_decoders created from the indexed RLE data during message matching. This proxy object can then be used to virtually combine the run length encoded bitset data and enumerate the resulting 1 bits.

It is still possible to materialise the bitsets if needed.

Implements #396

@AshleyRoll
Copy link
Contributor Author

I'm going to write up a document describing how this and the RLE encoding process works, so it might be a good idea to wait for that to assist with reviews.

@AshleyRoll AshleyRoll force-pushed the feature/efficient-rle-bitset-intersection branch 2 times, most recently from bedc897 to bdde90f Compare December 15, 2023 12:47
docs/message.adoc Outdated Show resolved Hide resolved
@AshleyRoll AshleyRoll force-pushed the feature/efficient-rle-bitset-intersection branch from 472142d to 70c5ad3 Compare December 19, 2023 07:14
docs/implementation_details.adoc Outdated Show resolved Hide resolved
docs/implementation_details.adoc Outdated Show resolved Hide resolved
docs/implementation_details.adoc Outdated Show resolved Hide resolved
docs/implementation_details.adoc Outdated Show resolved Hide resolved
docs/implementation_details.adoc Outdated Show resolved Hide resolved
include/msg/rle_indexed_handler.hpp Outdated Show resolved Hide resolved
test/msg/detail/rle_codec.cpp Outdated Show resolved Hide resolved
test/msg/detail/rle_codec.cpp Outdated Show resolved Hide resolved
test/msg/detail/rle_codec.cpp Outdated Show resolved Hide resolved
test/msg/detail/rle_codec.cpp Show resolved Hide resolved
@AshleyRoll AshleyRoll force-pushed the feature/efficient-rle-bitset-intersection branch 2 times, most recently from efc5bd9 to 15d283d Compare January 3, 2024 11:26
@AshleyRoll
Copy link
Contributor Author

I'll wait until you get the changes in intel/cpp-std-extensions#45 merged, and then fix both the FWD() implementation and the way I set multiple bits when decoding the RLE data.

The previous interim implementation of RLE encoded message indices would
decode data into fully materialised `bitset`s in order to fit into the
existing infrastructure easily.

This commit now eliminates the need to materialise a `bitset` to compute
which callbacks are to be executed for a given message.

The `rle_intersect` type is a proxy object that collects together
multiple `rle_decoder`s created from the indexed RLE data during message
matching. This proxy object can then be used to virtually combine the
run length encoded bitset data and enumerate the resulting `1` bits.

It is still possible to materialise the `bitset`s if needed.
@AshleyRoll AshleyRoll force-pushed the feature/efficient-rle-bitset-intersection branch from 84c6eff to cb096cc Compare January 5, 2024 03:02
Copy link
Contributor

@lukevalenty lukevalenty left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Look's great! Had a chance today to finish my review. I may post an issue/enhancemenr later for additional property based tests on the RLE encode/decode functions.

@lukevalenty lukevalenty merged commit 6fddb9e into intel:main Jan 6, 2024
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants