kapp-controller leader election should provide safety during update operations #838
Assignees
Labels
carvel-accepted
This issue should be considered for future work and that the triage process has been completed
enhancement
This issue is a feature request
good first issue
An issue that will be a good candidate for a new contributor
priority/important-longterm
Important over the long term, but may not be staffed and/or may need multiple releases to complete.
Comments
jdef
added
carvel-triage
|
@jdef Thanks for filing this issue. Can you share more about how you're using kapp-controller? Right now there's not really a way to run multiple kapp-controllers at once |
joe-kimmel-vmw
added
dk-think-more
and removed
carvel-triage
|
we're just getting started with it. it looks to be running an apiservice
(at least, there's an apiservice resource included in resources.yml) but
there's only a single replica. our concerns:
(a) if it's in ANY kind of api/controlplane hot-path, it should be HA (for
us, that translates to more than a single replica); which leads to ...
(b) added insurance that no more than 1 replica is attempting to reconcile
applications/packages (no split-brain, ever, please)
if i trusted Deployment more to guard against split-brain, maybe i'd feel
better. i don't, because i've seen meltdowns from such assumptions. leader
election has proven much more reliable along these lines. if we're going to
run this in prod envs, we'd like it to meet our prod standards.
HTH
…On Mon, Aug 22, 2022 at 11:16 AM Joe Kimmel ***@***.***> wrote:
@jdef <https://github.com/jdef> Thanks for filing this issue. Can you
share more about how you're using kapp-controller? Right now there's not
really a way to run multiple kapp-controllers at once
—
Reply to this email directly, view it on GitHub
<#838 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAR5KLD447FAH4JUTIVOO4DV2OKWXANCNFSM57HXYTOQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
--
James DeFelice
|
|
Any other thoughts here from the KC team?
On Mon, Aug 22, 2022, 5:33 PM James DeFelice ***@***.***>
wrote:
… we're just getting started with it. it looks to be running an apiservice
(at least, there's an apiservice resource included in resources.yml) but
there's only a single replica. our concerns:
(a) if it's in ANY kind of api/controlplane hot-path, it should be HA (for
us, that translates to more than a single replica); which leads to ...
(b) added insurance that no more than 1 replica is attempting to reconcile
applications/packages (no split-brain, ever, please)
if i trusted Deployment more to guard against split-brain, maybe i'd feel
better. i don't, because i've seen meltdowns from such assumptions. leader
election has proven much more reliable along these lines. if we're going to
run this in prod envs, we'd like it to meet our prod standards.
HTH
On Mon, Aug 22, 2022 at 11:16 AM Joe Kimmel ***@***.***>
wrote:
> @jdef <https://github.com/jdef> Thanks for filing this issue. Can you
> share more about how you're using kapp-controller? Right now there's not
> really a way to run multiple kapp-controllers at once
>
> —
> Reply to this email directly, view it on GitHub
> <#838 (comment)>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAR5KLD447FAH4JUTIVOO4DV2OKWXANCNFSM57HXYTOQ>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
--
James DeFelice
|
|
hi @jdef - I just realized the maintainers had a sort of unfinished exchange that we hadn't percolated back out to you, sorry about that! It seems like we'd be open to adding a leader/follower lease setup, similar to the k8s core controllers- would that satisfy your concerns? I think this is something we'd be happy to do eventually and/or accept contributions for. |
joe-kimmel-vmw
added
good first issue
|
Yep that would work fine, thanks.
…On Sat, Sep 3, 2022, 5:03 PM Joe Kimmel ***@***.***> wrote:
hi @jdef <https://github.com/jdef> - I just realized the maintainers had
a sort of unfinished exchange that we hadn't percolated back out to you,
sorry about that!
It seems like we'd be open to adding a leader/follower lease setup,
similar to the k8s core controllers- would that satisfy your concerns? I
think this is something we'd be happy to do eventually and/or accept
contributions for.
—
Reply to this email directly, view it on GitHub
<#838 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAR5KLB2S7I3AHJEO5NZYHLV4O4J5ANCNFSM57HXYTOQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
Hi @joe-kimmel-vmw, you mean using something like the leaderelection library from client-go, right? If that's the case, I'd be willing to work on this. |
|
That's what I had in mind
…On Wed, Oct 5, 2022, 2:40 AM Víctor Martínez Bevià ***@***.***> wrote:
Hi @joe-kimmel-vmw <https://github.com/joe-kimmel-vmw>, you mean using
something like the leaderelection
<https://github.com/kubernetes/client-go/tree/master/tools/leaderelection>
library from client-go, right? If that's the case, I'd be willing to work
on this.
—
Reply to this email directly, view it on GitHub
<#838 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAR5KLFJKL7SGCN5ENQC7ITWBUPEFANCNFSM57HXYTOQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
5 tasks
|
I will look into this assign me this issue! |
|
/assign |
|
@basit9958 Thank you for showing interest in working on the issue ❤️ |
5 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the problem/challenge you have
Describe the solution you'd like
Anything else you would like to add:
[Additional information that will assist in solving the issue.]
Vote on this request
This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.
👍 "I would like to see this addressed as soon as possible"
👎 "There are other more important things to focus on right now"
We are also happy to receive and review Pull Requests if you want to help working on this issue.
The text was updated successfully, but these errors were encountered: