Bloom/fuse queries #11088
Merged
Bloom/fuse queries #11088
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Owen Diehl <[email protected]>
Signed-off-by: Owen Diehl <[email protected]>
Signed-off-by: Owen Diehl <[email protected]>
Signed-off-by: Owen Diehl <[email protected]>
Signed-off-by: Owen Diehl <[email protected]>
Signed-off-by: Owen Diehl <[email protected]>
Signed-off-by: Owen Diehl <[email protected]>
Signed-off-by: Owen Diehl <[email protected]>
Signed-off-by: Owen Diehl <[email protected]>
Signed-off-by: Owen Diehl <[email protected]>
Signed-off-by: Owen Diehl <[email protected]>
rhnasc
pushed a commit
to inloco/loki
that referenced
this pull request
Apr 12, 2024
Introduces a "Fuse" operation on a single block querier which multiplexes queries against a single block. This is advantageous for a few reasons: * A "query" contains one of more series, with each series needing to test one or more chunks for some search string(s). * Blocks are sorted by fingerprint, which are the result of a hash function. This means series are ~uniformly distributed across the hash keyspace (`uint64` in our case). Because of this, a query for some number of series are likely to be spread out across the block itself. * Blocks are organized into individually compressed "pages". In essence, querying them requires Seeking to the correct offset, decompressing the page, then seeking within the page to the actual offset for a particular series. * Since the series for any query are likely spread out across the block, performing `n` queries decompresses each page `n` times. * By multiplexing many queries into a single pass iteration of the block, we amortize the cost of loading & decompressing Misc: Also introduces a bunch of generic-powered iterators. These can probably be optimized (I bet they use dynamic dispatch under the hood), but they're very convenient to prototype with and we can improve them in the future if it proves warranted.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Introduces a "Fuse" operation on a single block querier which multiplexes queries against a single block. This is advantageous for a few reasons:
uint64in our case). Because of this, a query for some number of series are likely to be spread out across the block itself.nqueries decompresses each pagentimes.Misc:
Also introduces a bunch of generic-powered iterators. These can probably be optimized (I bet they use dynamic dispatch under the hood), but they're very convenient to prototype with and we can improve them in the future if it proves warranted.