Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Revert wait for activation polling back to stream #14661

Draft
wants to merge 8 commits into
base: develop
Choose a base branch
from
Draft

Revert wait for activation polling back to stream #14661

wants to merge 8 commits into from

Conversation

james-prysm
Copy link
Contributor

@james-prysm james-prysm commented Nov 22, 2024 •
edited
Loading

What type of PR is this?

Bug

What does this PR do? Why is it needed?

originally triggered due to testing a newly deposited validator and its journey from Deposited status -> Pending status -> Active status

the following context deadline errors occurred during this time

[2024-11-22 03:26:26] ERROR client: error getting validator duties error=rpc error: code = DeadlineExceeded desc = context deadline exceeded
[2024-11-22 03:26:26] ERROR client: Failed to update assignments error=rpc error: code = DeadlineExceeded desc = context deadline exceeded
[2024-11-22 03:26:26] ERROR client: error getting validator duties error=rpc error: code = DeadlineExceeded desc = context deadline exceeded
[2024-11-22 03:26:26] ERROR client: Failed to update assignments error=rpc error: code = DeadlineExceeded desc = context deadline exceeded
[2024-11-22 03:26:26] ERROR client: error getting validator duties error=rpc error: code = DeadlineExceeded desc = context deadline exceeded
[2024-11-22 03:26:26] ERROR client: Failed to update assignments error=rpc error: code = DeadlineExceeded desc = context deadline exceeded
[2024-11-22 03:26:26] ERROR client: error getting validator duties error=rpc error: code = DeadlineExceeded desc = context deadline exceeded
[2024-11-22 03:26:26] ERROR client: Failed to update assignments error=rpc error: code = DeadlineExceeded desc = context deadline exceeded
[2024-11-22 03:26:26] ERROR client: error getting validator duties error=rpc error: code = DeadlineExceeded desc = context deadline exceeded
[2024-11-22 03:26:26] ERROR client: Failed to update assignments error=rpc error: code = DeadlineExceeded desc = context deadline exceeded
[2024-11-22 03:26:26]  INFO client: Schedule for epoch 94700 attesterCount=2 proposerCount=0
[2024-11-22 03:26:26]  INFO client: Duties schedule attesterCount=1 attesterPubkeys=[0x92dc3bb43fd0] slot=3030404 slotInEpoch=4
[2024-11-22 03:26:26]  INFO client: Duties schedule attesterCount=1 attesterPubkeys=[0x8e4e5b65e7e0] slot=3030414 slotInEpoch=14
[2024-11-22 03:26:26] ERROR client: Could not request attestation to sign at slot error=rpc error: code = DeadlineExceeded desc = context deadline exceeded pubkey=0x92dc3bb43fd0 slot=3030404
[2024-11-22 03:26:26] ERROR client: Could not request attestation to sign at slot error=rpc error: code = DeadlineExceeded desc = context deadline exceeded pubkey=0x8e4e5b65e7e0 slot=3030414
[2024-11-22 03:26:26] ERROR client: Could not report validator's rewards/penalties error=rpc error: code = DeadlineExceeded desc = context deadline exceeded

with the validator recovering in the following slot.

note: it's unclear if this change was what caused the context deadline above, this will be in draft mode until tested again with roll back

Which issues(s) does this PR fix?

rolls back #14514
Other notes for review

Acknowledgements

  • I have read CONTRIBUTING.md.
  • I have made an appropriate entry to CHANGELOG.md.
  • I have added a description to this PR with sufficient context for reviewers to understand this PR.

james-prysm
james-prysm commented Nov 23, 2024
View reviewed changes
validator/client/wait_for_activation.go
return ctx.Err()
case <-accountsChangedChan:
// Accounts (keys) changed, restart the process.
return v.internalWaitForActivation(ctx, accountsChangedChan)
default:
if err := v.waitForNextEpoch(ctx, v.genesisTime, accountsChangedChan); err != nil {
return v.retryWaitForActivation(ctx, span, err, "Failed to wait for next epoch. Reconnecting...", accountsChangedChan)
res, err := (stream).Recv() // retrieve from stream one loop at a time
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should keep investigating ways to get rid of this, and switch with a polling structure

@james-prysm
Copy link
Contributor Author

I don't think this revert improves it

[2024-11-22 23:55:17] ERROR client: error getting validator duties error=rpc error: code = DeadlineExceeded desc = context deadline exceeded
[2024-11-22 23:55:17] ERROR client: Failed to update assignments error=rpc error: code = DeadlineExceeded desc = context deadline exceeded
[2024-11-22 23:55:17] ERROR client: Failed to update assignments error=rpc error: code = DeadlineExceeded desc = context deadline exceeded
[2024-11-22 23:55:17] ERROR client: error getting validator duties error=rpc error: code = DeadlineExceeded desc = context deadline exceeded
[2024-11-22 23:55:17] ERROR client: Failed to update assignments error=rpc error: code = DeadlineExceeded desc = context deadline exceeded
[2024-11-22 23:55:17] ERROR client: error getting validator duties error=rpc error: code = DeadlineExceeded desc = context deadline exceeded
[2024-11-22 23:55:17] ERROR client: Failed to update assignments error=rpc error: code = DeadlineExceeded desc = context deadline exceeded
[2024-11-22 23:55:18]  WARN client: Invalid attester slot duty=public_key:"\x92\xf9Sц\xa0\xb1\x89\xa8\xb7\x18ق@%\xcc\xfeC\x80\xf2'\xb6\xa9\xa0\xd8m\"\x8f.\xcd+W\x1cni\xffl\x08H\xad\xefwјp \x94\x88"  status:ACTIVE  validator_index:1890457
[2024-11-22 23:55:18]  INFO client: Schedule for epoch 94892 attesterCount=0 proposerCount=0
[2024-11-22 23:55:18] ERROR client: Could not report validator's rewards/penalties error=rpc error: code = DeadlineExceeded desc = context deadline exceeded
[2024-11-22 23:55:18]  INFO client: Schedule for epoch 94893 attesterCount=1 proposerCount=0
[2024-11-22 23:55:18]  INFO client: Duties schedule attesterCount=1 attesterPubkeys=[0x92f953d186a0] slot=3036595 slotInEpoch=19 timeUntilDuty=3m42s

this was still seen

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@james-prysm