Upgrading FTL can lead to ASM followers failing to sync #2011
Assignees
Comments
Open
matt2e
added a commit
that referenced
this issue
Jul 10, 2024
fixes #2011 Cause: - Coordinator only does it coordination logic when something calls `Get()` - If leader/follower coordinator's `Get()` function was not called then an expired leader/follower would not change - ASM follower syncs with what it expects to be the leader even if that leader is not alive. Even if we get a new leader, the follower would not be swapped out for a valid one until `Get()` was called. So it would churn away failing to sync. Fix: - Proactively call `Get()` periodically so that coordination occurs even if no calls to Get() would otherwise happen.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Upgrade happens, 3 leases get expired
Unable to sync asm-follower: error getting secrets list from leader: unavailable: read tcp 10.1.141.197:58350->10.1.104.37:8892: read: connection reset by peerThis happens for about 4 minutes after upgrade
Then the controller was manually restarted again to try and solve it
4 leases get expired
Errors stop happening
The text was updated successfully, but these errors were encountered: