region/client: re-establish connection on ServerNotRunningYetException #266
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When receiving ServerNotRunningYetException, a client shouldn't retry to send the request to the same server. Instead, the client should be closed and the region lookup should happen again.
There is two cases when ServerNotRunningYetException is returned:
when the RegionServer is listening but not online yet: in that case, retrying the RPC on the same server may succeed if the Regionserver become ready and if the region is indeed assigned to it. But most likely the region would have been reassigned to another Regionserver and thus it will return NotServingRegionException in the following request. If the Regionserver is stuck in startup phase, it could also cause the client to be stuck in retry loop whereas HBasemaster may have detected the issue and correctly moved the region to another Regionserver already.
when the HBasemaster server is currently not active: in that case, retrying the RPC on the same server is guaranteed to fail until a failover. The client will be stuck in a forever retrying loop.
If we receive multiple ServerError for the same RPC, we will backoff before retrying. This is to avoid overwhelming HBase. Scenario where this could happen is a cluster that is recovering from catastrophic failure, with all HBasemaster still trying to start (like recovering WALs or what not).
Also add MasterStoppedException and PleaseHoldException to the list of known exception that can be returned by HBase.
Fix #265