Added custom retry with backoff for error not covered in BQ client (4… #1174
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sechegaray
force-pushed
the
fix/bigquery-retry-batch-job
branch
from
4e7ef71 to
2a4dcd4
Compare
sechegaray
force-pushed
the
fix/bigquery-retry-batch-job
branch
from
2a4dcd4 to
7e7298a
Compare
…00, please retry with backoff)
sechegaray
force-pushed
the
fix/bigquery-retry-batch-job
branch
from
7e7298a to
de02031
Compare
|
|
||
| // Wait for the query to complete | ||
| queryJob.waitFor(); | ||
| final String retryableStringPattern = "Retrying the job with back-off"; |
There was a problem hiding this comment.
nit: make this a constant
| List<Function<BigQueryException, Boolean>> retryRules = new ArrayList<>(); | ||
| retryRules.add( | ||
| (BigQueryException e) -> e.getCode() == 400 | ||
| && (e.getMessage().contains(retryableStringPattern) || e.getReason().contains(retryableStringPattern)) |
There was a problem hiding this comment.
Are the methods on BigQueryException like getMessage() and getReason() guaranteed to be non null?
| @@ -169,6 +173,46 @@ public void run(ActionContext context) throws Exception { | |||
| context.getMetrics().gauge(RECORDS_PROCESSED, rows); | |||
| } | |||
|
|
|||
| /** | |||
| * Executes Query with added retry rules following: | |||
| * https://cloud.google.com/bigquery/sla | |||
There was a problem hiding this comment.
This should also apply to BigQuery Sink plugin and may be even replication BQ plugins? Or for the time being it is only done for execute? It would be good to put this in a different class if it is applicable for other plugins.
| */ | ||
| private Job executeQueryJobWithCustomRetry(BigQuery bigQuery, QueryJobConfiguration queryConfig, | ||
| List<Function<BigQueryException, Boolean>> retryRules) throws Exception { | ||
| // The longest amount of time to wait in-between retries. |
There was a problem hiding this comment.
nit: mention the timeunit
| return queryJob; | ||
| } catch (BigQueryException bigQueryException) { | ||
| if (retries >= maxRetries) { | ||
| LOG.error("Run out of retries while executing query with backoff."); |
There was a problem hiding this comment.
nit: Ran out of...
|
|
||
| int retries = 0; | ||
|
|
||
| while (true) { |
There was a problem hiding this comment.
Please add a unit test.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
…00, please retry with backoff)