Add e2e tests for bloom filtering #11645
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Christian Haudum <[email protected]>
Signed-off-by: Christian Haudum <[email protected]>
Signed-off-by: Christian Haudum <[email protected]>
Signed-off-by: Christian Haudum <[email protected]>
A test that both builds bloom filters as well as uses them for filtering chunk refs when querying. Signed-off-by: Christian Haudum <[email protected]>
Signed-off-by: Christian Haudum <[email protected]>
Signed-off-by: Christian Haudum <[email protected]>
Signed-off-by: Christian Haudum <[email protected]>
Signed-off-by: Christian Haudum <[email protected]>
Signed-off-by: Christian Haudum <[email protected]>
Signed-off-by: Christian Haudum <[email protected]>
chaudum
force-pushed
the
chaudum/bloom-filter-e2e-test
branch
from
f408ceb to
f55d79b
Compare
chaudum
commented
Jan 11, 2024
Signed-off-by: Christian Haudum <[email protected]>
Signed-off-by: Christian Haudum <[email protected]>
vlad-diachenko
approved these changes
Jan 11, 2024
| // isOutsideRange tests if a given BlockRef b is outside of search boundaries | ||
| // defined by min/max timestamp and min/max fingerprint. | ||
| // Fingerprint ranges must be sorted in ascending order. | ||
| func isOutsideRange(b *BlockRef, startTimestamp, endTimestamp model.Time, fingerprints [][2]uint64) bool { |
There was a problem hiding this comment.
I would create a struct, maybe FpRange and pass []FpRange into this function. it would make the code more clear.
Signed-off-by: Christian Haudum <[email protected]>
Signed-off-by: Christian Haudum <[email protected]>
chaudum
commented
Jan 11, 2024
| // this is already covered in the range check above, but I keep it as a second gate | ||
| if idx > len(fingerprints)-1 { | ||
| return true | ||
| prev := fpRange{0, 0} |
There was a problem hiding this comment.
Could be written as
Suggested change
| prev := fpRange{0, 0} | |
| var prev fpRange |
rhnasc
pushed a commit
to inloco/loki
that referenced
this pull request
Apr 12, 2024
**What?** This PR adds an end-to-end integration test for creating bloom filters and using them in the gateway to filter chunks. **Why?** This test helped to identify bugs in the bloom shipper code that would have taken a very long time to discover in other ways, such as deploying to a dev environment or running a long-running docker compose setup. **Notes** The following commits of this PR are actual changes to the compactor/gateway code to make e2e test work: * grafana@2dcc761 * grafana@84052dd * grafana@3ba44f8 The bloom gateway code path for processing blocks was cleaned up, because it still contained the unused "sequention processing" path of blocks, which was initially kept to verify the callback based processing works alike: * grafana@f55d79b --------- Signed-off-by: Christian Haudum <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What?
This PR adds an end-to-end integration test for creating bloom filters and using them in the gateway to filter chunks.
Why?
This test helped to identify bugs in the bloom shipper code that would have taken a very long time to discover in other ways, such as deploying to a dev environment or running a long-running docker compose setup.
Notes
The following commits of this PR are actual changes to the compactor/gateway code to make e2e test work:
The bloom gateway code path for processing blocks was cleaned up, because it still contained the unused "sequention processing" path of blocks, which was initially kept to verify the callback based processing works alike: