Fix node-validator's state becoming stale due to disconnected leaf stream #2051
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Ayiga
requested review from
nomaxg,
philippecamacho,
ImJeremyHe,
sveitser,
jbearer,
tbro and
imabdulbasit
as code owners
Ayiga
force-pushed
the
ts/fix/node-validator-leaf-stream-disconnect
branch
3 times, most recently
from
f76a249 to
2eb46dc
Compare
imabdulbasit
reviewed
Oct 3, 2024
imabdulbasit
reviewed
Oct 3, 2024
…ream The Leaf Stream that is sourced from the HotShot Query Service can becomes disconnected due to a variety of issues that are not accounted for within the node-validator API. These reasons can be varied, but they all have the same result, the failure to retrieve a Stream, or the end of an existing Stream. In either of these cases the node-validator will not attempt to re-establish the Leaf Stream, resulting in the validator's state becoming stale over time. In order to address this issue effectively, and ideally, changes have been made so attempts to re-acquire the initial Leaf Stream state. If this retrieval fails then it will attempt again and again with an exponential backoff. If the failure persists for too long, the process will eventually give up and panic instead.
Ayiga
force-pushed
the
ts/fix/node-validator-leaf-stream-disconnect
branch
from
be1229a to
6ff63cf
Compare
imabdulbasit
approved these changes
Oct 3, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #2002
This PR:
Addresses an issue that occurs when the source Leaf Stream HotShot Query service becomes unavailable, resulting in the node-validator service ending up in a Stale state with no future updates being delivered.
This is implemented with connection re-attempts to acquire a new Leaf Stream should the previous one complete. The re-attempts themselves will loop to try and ensure a new Leaf Stream is returned with an exponential backkoff up to a maximum limit of 100 times, at which point it will panic as it seems that the service is not going to be coming back immediately.