-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Block deletion endpoints #1
base: master
Are you sure you want to change the base?
Conversation
Signed-off-by: ilangofman <[email protected]>
pkg/api/api.go
Outdated
a.RegisterRoute("/purger/delete_tenant_status", http.HandlerFunc(api.DeleteTenantStatus), true, "GET") | ||
func (a *API) RegisterBlocksPurgerAPI(blocksPurger *purger.BlocksPurgerAPI) { | ||
|
||
a.RegisterRoute(path.Join(a.cfg.PrometheusHTTPPrefix, "/api/v1/admin/tsdb/delete_series"), http.HandlerFunc(blocksPurger.V2AddDeleteRequestHandler), true, "PUT", "POST") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This isn't really V2 right? Should there be V2 in the name?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These functions exist for the chunks purger. I thought adding the V2 like in the ingester functions would make things more clear. They are in the blocksPurger
so that might be enough of information to tell them apart. I will remove it.
Signed-off-by: ilangofman <[email protected]>
0166c79
to
220a47e
Compare
Signed-off-by: ilangofman <[email protected]>
pkg/storage/tsdb/tombstones.go
Outdated
// add all the tombstones to a map and check for duplicates, | ||
// if a key exists with the same request ID (but two different states) | ||
tombstoneMap := make(map[string]*Tombstone) | ||
err := userBucket.Iter(ctx, "tombstones/", func(s string) error { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this too can be simplified as follows:
- Initialize the tombstone map
- Sort all the tombstones in the bucket by state. https://gobyexample.com/sorting-by-functions
- Iterate over the sorted tombstones. set
tombstonesMap[id] = tombstoneFile
- What's left in the map would be the most recent state.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As per your other comment, I removed the removeDuplicateTombstone
call from the get functions and add it during the cleanup in the compactor. This should simplify it a bit.
I think with the suggestion you made, it might be less efficient if I interpreted it correctly:
- Iterate over the tombstones in the bucket and add them all to a slice.
- Sort the slice
- Iterate over the slice and add it to the map
- iterate over the map and add it to a new slice
- return the slice
This will add another pass over the slice and a sort operation.
Let me know if I misinterpreted your suggestion.
pkg/storage/tsdb/tombstones.go
Outdated
// If there are multiple tombstones with the same request id but different state, should delete the files with the lower state | ||
for _, ts := range found[0 : len(found)-1] { | ||
level.Info(userLogger).Log("msg", "Found extra tombstone file with outdated state. Will delete it.", "requestID", ts.RequestID, "state", ts.State) | ||
if err := DeleteTombstoneFile(ctx, bkt, cfgProvider, ts); err != nil { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't like the fact that we are deleting tombstones during a GET operation. Can't we just perform clean up in the compactor after deletion is complete?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah I think that makes more sense. I removed the deletion operations in the get functions.
Signed-off-by: ilangofman <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Signed-off-by: ilangofman <[email protected]>
Signed-off-by: ilangofman <[email protected]>
Signed-off-by: ilangofman <[email protected]>
Signed-off-by: ilangofman <[email protected]>
Signed-off-by: ilangofman <[email protected]>
Signed-off-by: ilangofman <[email protected]>
Signed-off-by: ilangofman <[email protected]>
Signed-off-by: ilangofman <[email protected]>
Signed-off-by: ilangofman <[email protected]>
Signed-off-by: ilangofman <[email protected]>
Signed-off-by: ilangofman <[email protected]>
Signed-off-by: ilangofman <[email protected]>
Signed-off-by: ilangofman <[email protected]>
Signed-off-by: ilangofman <[email protected]>
…man/cortex into block_deletion_endpoints
Signed-off-by: ilangofman <[email protected]>
Signed-off-by: ilangofman <[email protected]>
Signed-off-by: ilangofman <[email protected]>
reverse change of combining blocks purger and tenant deletion
Signed-off-by: Ilan Gofman <[email protected]>
Signed-off-by: Ilan Gofman <[email protected]>
What this PR does:
This PR is part of the proposed addition of the time series deletion API for block storage.
The current plan is to split the feature into 4 smaller PR's:
Which issue(s) this PR fixes:
Fixes #
Checklist
CHANGELOG.md
updated - the order of entries should be[CHANGE]
,[FEATURE]
,[ENHANCEMENT]
,[BUGFIX]