Blooms/integration fixes #11979
Merged
Blooms/integration fixes #11979
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Owen Diehl <[email protected]>
Signed-off-by: Owen Diehl <[email protected]>
Signed-off-by: Owen Diehl <[email protected]>
avoids race conditions in storage Signed-off-by: Owen Diehl <[email protected]>
Signed-off-by: Owen Diehl <[email protected]>
Signed-off-by: Owen Diehl <[email protected]>
Signed-off-by: Owen Diehl <[email protected]>
Signed-off-by: Owen Diehl <[email protected]>
Signed-off-by: Owen Diehl <[email protected]>
owen-d
force-pushed
the
blooms/integration-fixes
branch
from
62c6cd2 to
5ee3d5a
Compare
Signed-off-by: Owen Diehl <[email protected]>
owen-d
force-pushed
the
blooms/integration-fixes
branch
from
5ee3d5a to
0f9920c
Compare
Signed-off-by: Owen Diehl <[email protected]>
owen-d
force-pushed
the
blooms/integration-fixes
branch
from
9fd015e to
2d09d20
Compare
Signed-off-by: Owen Diehl <[email protected]>
Signed-off-by: Owen Diehl <[email protected]>
chaudum
reviewed
Feb 19, 2024
pkg/bloomcompactor/controller.go
Outdated
| @@ -92,6 +92,15 @@ func (s *SimpleBloomController) compactTenant( | |||
| return errors.Wrap(err, "failed to get metas") | |||
| } | |||
|
|
|||
| level.Debug(logger).Log("msg", "found relevant metas", "metas", len(metas)) | |||
|
|
|||
| // fetch all metas overlapping all metas overlapping our ownership range so we can safely | |||
There was a problem hiding this comment.
Suggested change
| // fetch all metas overlapping all metas overlapping our ownership range so we can safely | |
| // fetch all metas overlapping our ownership range so we can safely |
| @@ -226,7 +226,7 @@ func TestMergeBuilder(t *testing.T) { | |||
| ) | |||
|
|
|||
| // Ensure that the merge builder combines all the blocks correctly | |||
| mergeBuilder := NewMergeBuilder(dedupedBlocks(blocks), storeItr, pop) | |||
| mergeBuilder := NewMergeBuilder(dedupedBlocks(blocks), storeItr, pop, NewMetrics(nil)) | |||
There was a problem hiding this comment.
Shouldn't a registerer be passed to NewMetrics()?
There was a problem hiding this comment.
it short circuits with nil
| @@ -77,7 +81,7 @@ func (defaultKeyResolver) Block(ref BlockRef) Location { | |||
| ref.TenantID, | |||
| BlocksPrefix, | |||
| ref.Bounds.String(), | |||
| fmt.Sprintf("%d-%d-%x", ref.StartTimestamp, ref.EndTimestamp, ref.Checksum), | |||
| fmt.Sprintf("%d-%d-%x%s", ref.StartTimestamp, ref.EndTimestamp, ref.Checksum, extTarGz), | |||
salvacorts
approved these changes
Feb 19, 2024
| level.Debug(logger).Log("msg", "found relevant metas", "metas", len(metas)) | ||
|
|
||
| // fetch all metas overlapping all metas overlapping our ownership range so we can safely | ||
| // check which metas can be deleted even if they only partially overlap out ownership range |
There was a problem hiding this comment.
Suggested change
| // check which metas can be deleted even if they only partially overlap out ownership range | |
| // check which metas can be deleted even if they only partially overlap our ownership range |
| return errors.Wrap(err, "failed to delete block") | ||
| } | ||
| } | ||
| level.Debug(logger).Log("msg", "removed outdated block", "block", block.String()) |
There was a problem hiding this comment.
I think it would be nice to have a metric to track how many blocks/metas are deleted
pkg/bloomcompactor/tsdb.go
Outdated
| @@ -236,8 +236,8 @@ func NewTSDBStores( | |||
| if err != nil { | |||
| return nil, errors.Wrap(err, "failed to create object client") | |||
| } | |||
| prefix := path.Join(cfg.IndexTables.PathPrefix, cfg.IndexTables.Prefix) | |||
| res.stores[i] = NewBloomTSDBStore(storage.NewIndexStorageClient(c, prefix)) | |||
| // prefix := path.Join(cfg.IndexTables.PathPrefix, cfg.IndexTables.Prefix) | |||
| } | ||
|
|
||
| const chunkIndexedTypeIterated = "iterated" | ||
| const chunkIndexedTypeCopied = "copied" | ||
|
|
||
| func NewMetrics(r prometheus.Registerer) *Metrics { | ||
| return &Metrics{ | ||
| sbfCreationTime: promauto.With(r).NewCounter(prometheus.CounterOpts{ |
There was a problem hiding this comment.
I think this should probably be a histogram
Signed-off-by: Owen Diehl <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
A few more fixes/improvements while running & debugging blooms