Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enabling the CircuitBreaker for remote HTTP cache #24685

Draft
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

AlexanderGolovlev
Copy link
Contributor

@AlexanderGolovlev AlexanderGolovlev commented Dec 13, 2024

This PR implements the feature to stop using the remote HTTP cache when it became unreachable.

The implementation is based on already existing CircuitBreaker logic. It includes the following changes:

  • All non-retriable errors were counted as successful API calls. However, they include not only the cache miss errors (which should be considered as a successful call), but also the different connectivity errors (which should be considered as failures). To distinguish these errors, we have added the isSuccess predicate. HTTP_SUCCESS_CODES defines success codes for HTTP, and RemoteRetrier.GRPC_SUCCESS_CODES defines success codes for grpc.
  • Current implementation of FailureCircuitBreaker defines the threshold required to calculate the failure rate: minCallCountToComputeFailureRate=100. In case of connection timeout errors this requires a very large time interval to reach the threshold, larger that the default experimental_remote_failure_window_interval=60s. To solve the problem, we have added one more threshold: DEFAULT_MIN_FAIL_COUNT_TO_COMPUTE_FAILURE_RATE=12. The value is chosen to be slightly more than number of failures within window interval with default experimental_remote_failure_rate_threshold=10.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant