Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Lower ASM follower sync error log level for expected initial case #2133

Closed
matt2e opened this issue Jul 22, 2024 · 0 comments · Fixed by #2151
Closed

Lower ASM follower sync error log level for expected initial case #2133

matt2e opened this issue Jul 22, 2024 · 0 comments · Fixed by #2151
Assignees
@stuartwdouglas
Labels
good first issue Good for newcomers

Comments

@matt2e
Copy link
Collaborator

matt2e commented Jul 22, 2024 •
edited
Loading

This is expected when new controllers are brought up but for 10s or so the previous controller still has a lease.
The initial errors are confusing as they make it look like something unexpected is happening. We should only error after a certain number of errors or after a certain amount of time.

Error syncing asm: asm/follower/http://10.2.110.252:8892: error getting secrets list from leader: permission_denied: RBAC: access denied
@matt2e matt2e self-assigned this Jul 22, 2024
@ftl-robot ftl-robot mentioned this issue Jul 22, 2024
Open
@matt2e matt2e assigned stuartwdouglas and unassigned matt2e Jul 23, 2024
@matt2e matt2e added the good first issue Good for newcomers label Jul 24, 2024
stuartwdouglas added a commit that referenced this issue Jul 24, 2024
@stuartwdouglas
fix: don't log at error level in lease timeout
71c1657 
We don't want error alerts if a controller is just failing over, this PR adds some infrasturcture to allow for a lower level of error reporting when the errors all occur within the lease TTL.

fixes #2133
stuartwdouglas added a commit that referenced this issue Jul 24, 2024
@stuartwdouglas
fix: don't log at error level in lease timeout
bf42703 
We don't want error alerts if a controller is just failing over, this PR adds some infrasturcture to allow for a lower level of error reporting when the errors all occur within the lease TTL.

fixes #2133
@stuartwdouglas stuartwdouglas mentioned this issue Jul 24, 2024
Merged
stuartwdouglas added a commit that referenced this issue Jul 24, 2024
@stuartwdouglas
fix: don't log at error level in lease timeout
a0dfc60 
We don't want error alerts if a controller is just failing over, this PR adds some infrasturcture to allow for a lower level of error reporting when the errors all occur within the lease TTL.

fixes #2133
stuartwdouglas added a commit that referenced this issue Jul 24, 2024
@stuartwdouglas
fix: don't log at error level in lease timeout
cfd52fc 
We don't want error alerts if a controller is just failing over, this PR adds some infrasturcture to allow for a lower level of error reporting when the errors all occur within the lease TTL.

fixes #2133
stuartwdouglas added a commit that referenced this issue Jul 25, 2024
@stuartwdouglas
fix: don't log at error level in lease timeout
35bea4a 
We don't want error alerts if a controller is just failing over, this PR adds some infrasturcture to allow for a lower level of error reporting when the errors all occur within the lease TTL.

fixes #2133
stuartwdouglas added a commit that referenced this issue Jul 25, 2024
@stuartwdouglas
fix: don't log at error level in lease timeout
61b5ddb 
We don't want error alerts if a controller is just failing over, this PR adds some infrasturcture to allow for a lower level of error reporting when the errors all occur within the lease TTL.

fixes #2133
stuartwdouglas added a commit that referenced this issue Jul 25, 2024
@stuartwdouglas
fix: don't log at error level in lease timeout
15ff6ed 
We don't want error alerts if a controller is just failing over, this PR adds some infrasturcture to allow for a lower level of error reporting when the errors all occur within the lease TTL.

fixes #2133
stuartwdouglas added a commit that referenced this issue Jul 26, 2024
@stuartwdouglas
fix: don't log at error level in lease timeout
2c13873 
We don't want error alerts if a controller is just failing over, this PR adds some infrasturcture to allow for a lower level of error reporting when the errors all occur within the lease TTL.

fixes #2133
stuartwdouglas added a commit that referenced this issue Jul 28, 2024
@stuartwdouglas
fix: don't log at error level in lease timeout
2584ca6 
We don't want error alerts if a controller is just failing over, this PR adds some infrasturcture to allow for a lower level of error reporting when the errors all occur within the lease TTL.

fixes #2133
stuartwdouglas added a commit that referenced this issue Jul 29, 2024
@stuartwdouglas
fix: don't log at error level in lease timeout
8dd27a1 
We don't want error alerts if a controller is just failing over, this PR adds some infrasturcture to allow for a lower level of error reporting when the errors all occur within the lease TTL.

fixes #2133
stuartwdouglas added a commit that referenced this issue Aug 6, 2024
@stuartwdouglas
fix: don't log at error level in lease timeout
dc6903d 
We don't want error alerts if a controller is just failing over, this PR adds some infrasturcture to allow for a lower level of error reporting when the errors all occur within the lease TTL.

fixes #2133
github-merge-queue bot pushed a commit that referenced this issue Aug 12, 2024
@stuartwdouglas
fix: don't log at error level in lease timeout (#2151)
1fa898f 
We don't want error alerts if a controller is just failing over, this PR
adds some infrasturcture to allow for a lower level of error reporting
when the errors all occur within the lease TTL.

fixes #2133
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@stuartwdouglas stuartwdouglas

Labels
good first issue Good for newcomers
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: don't log at error level in lease timeout block/ftl
2 participants
@stuartwdouglas @matt2e