Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

region/client: re-establish connection on ServerNotRunningYetException #266

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

dethi
Copy link
Collaborator

@dethi dethi commented Jul 13, 2024

When receiving ServerNotRunningYetException, a client shouldn't retry to send the request to the same server. Instead, the client should be closed and the region lookup should happen again.

There is two cases when ServerNotRunningYetException is returned:

  • when the RegionServer is listening but not online yet: in that case, retrying the RPC on the same server may succeed if the Regionserver become ready and if the region is indeed assigned to it. But most likely the region would have been reassigned to another Regionserver and thus it will return NotServingRegionException in the following request. If the Regionserver is stuck in startup phase, it could also cause the client to be stuck in retry loop whereas HBasemaster may have detected the issue and correctly moved the region to another Regionserver already.

  • when the HBasemaster server is currently not active: in that case, retrying the RPC on the same server is guaranteed to fail until a failover. The client will be stuck in a forever retrying loop.

If we receive multiple ServerError for the same RPC, we will backoff before retrying. This is to avoid overwhelming HBase. Scenario where this could happen is a cluster that is recovering from catastrophic failure, with all HBasemaster still trying to start (like recovering WALs or what not).

Also add MasterStoppedException and PleaseHoldException to the list of known exception that can be returned by HBase.

Fix #265

When receiving ServerNotRunningYetException, a client shouldn't retry to
send the request to the same server. Instead, the client should be
closed and the region lookup should happen again.

There is two cases when ServerNotRunningYetException is returned:
- when the RegionServer is listening but not online yet: in that case,
  retrying the RPC on the same server may succeed if the Regionserver
  become ready and if the region is indeed assigned to it. But most
  likely the region would have been reassigned to another Regionserver
  and thus it will return NotServingRegionException in the following
  request. If the Regionserver is stuck in startup phase, it could also
  cause the client to be stuck in retry loop whereas HBasemaster may
  have detected the issue and correctly moved the region to another
  Regionserver already.

- when the HBasemaster server is currently not active: in that case,
  retrying the RPC on the same server is guaranteed to fail until a
  failover. The client will be stuck in a forever retrying loop.

If we receive multiple ServerError for the same RPC, we will backoff
before retrying. This is to avoid overwhelming HBase. Scenario where
this could happen is a cluster that is recovering from catastrophic
failure, with all HBasemaster still trying to start (like recovering
WALs or what not).

Also add MasterStoppedException and PleaseHoldException to the list of
known exception that can be returned by HBase.

Fix #265
@dethi
Copy link
Collaborator Author

dethi commented Jul 13, 2024

Ready for review but not for merging, I want to add to try to add a test for this before (which is a bit painful because it needs hadoop hdfs to run to be able to have multiple hbasemaster)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

gohbase doesn't handle active hbasemaster going away/changing
2 participants