Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support annotating pods on change for thanos receiver router and ingestor setup #125

Open
rtnpro opened this issue Nov 12, 2023 · 4 comments · May be fixed by #126
Open

Support annotating pods on change for thanos receiver router and ingestor setup #125

rtnpro opened this issue Nov 12, 2023 · 4 comments · May be fixed by #126

Comments

@rtnpro
Copy link

rtnpro commented Nov 12, 2023

Currently, thanos-receive-controller uses the same label to watch receiver pods and update them on hashring changes.

However, in a Thanos receiver router and ingestor setup (https://thanos.io/tip/proposals-accepted/202012-receive-split.md/), while we need to watch the ingestor pods to update the hashring configmap, the hashring configmap gets used in the router pods. In this scenario, the router pods need to be annotated on hashring change instead of the ingestor pods.

For thanos-receive-controller to support this Thanos receiver router+receiver setup, it needs to support an option to specify a separate pod label (different from the pod label used to watch the receiver ingestor pods) to use for annotating pods on hashring change.
@rtnpro rtnpro linked a pull request Nov 12, 2023 that will close this issue
Open
@rtnpro
Copy link
Author

rtnpro commented Nov 12, 2023

Here's a draft PR: #126

@rtnpro
Copy link
Author

rtnpro commented Dec 23, 2023

@matej-g I need your thoughts on this issue.

We recently implemented the proposal to setup Thanos Receiver as Router and Ingestor for making Thanos Receiver highly available.

Currently, we are using reloader controller to rotate the receiver router pods on hashring change. It works, but has it's delay.

We have tested the changes in #126 and it works faster and updates the hashring config in the router pods without rotating them.

It will be great if this changes could be implemented in upstream.

Thanks.

@miguel-cardoso-mindera
Copy link

I'm also interested in this, has there been any news?

@rtnpro appreciate your good work!

@rtnpro
Copy link
Author

rtnpro commented May 15, 2024

I am still waiting on a response and review from the upstream.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Allow to annotate pods on change using a different label rtnpro/thanos-receive-controller
2 participants
@rtnpro @miguel-cardoso-mindera