[indexer_autoscaler_lambda] pause indexing if the available storage drops below a threshold #14
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
akumar1214
force-pushed
the
akumar/autoscaler-tune-lambda-concurrency
branch
from
d6a04f1 to
303f029
Compare
akumar1214
force-pushed
the
akumar/autoscaler-min-free-storage
branch
11 times, most recently
from
f703aff to
48f6a76
Compare
akumar1214
requested review from
myronmarston and
BrianSigafoos-SQ
as code owners
akumar1214
changed the title
akumar1214
force-pushed
the
akumar/autoscaler-min-free-storage
branch
from
48f6a76 to
e446554
Compare
akumar1214
force-pushed
the
akumar/autoscaler-tune-lambda-concurrency
branch
from
303f029 to
e032ddc
Compare
myronmarston
requested changes
Nov 6, 2024
elasticgraph-indexer_autoscaler_lambda/elasticgraph-indexer_autoscaler_lambda.gemspec
Outdated
Show resolved
Hide resolved
...-indexer_autoscaler_lambda/lib/elastic_graph/indexer_autoscaler_lambda/concurrency_scaler.rb
Outdated
Show resolved
Hide resolved
...-indexer_autoscaler_lambda/lib/elastic_graph/indexer_autoscaler_lambda/concurrency_scaler.rb
Show resolved
Hide resolved
...-indexer_autoscaler_lambda/lib/elastic_graph/indexer_autoscaler_lambda/concurrency_scaler.rb
Show resolved
Hide resolved
...-indexer_autoscaler_lambda/lib/elastic_graph/indexer_autoscaler_lambda/concurrency_scaler.rb
Outdated
Show resolved
Hide resolved
...raph-indexer_autoscaler_lambda/lib/elastic_graph/indexer_autoscaler_lambda/details_logger.rb
Outdated
Show resolved
Hide resolved
elasticgraph-indexer_autoscaler_lambda/sig/elastic_graph/cloudwatch_client.rbs
Outdated
Show resolved
Hide resolved
...toscaler_lambda/spec/unit/elastic_graph/indexer_autoscaler_lambda/concurrency_scaler_spec.rb
Outdated
Show resolved
Hide resolved
akumar1214
force-pushed
the
akumar/autoscaler-min-free-storage
branch
5 times, most recently
from
3ee57c3 to
4f002b7
Compare
myronmarston
requested changes
Nov 7, 2024
| @@ -30,32 +30,43 @@ def initialize( | |||
| } | |||
| end | |||
|
|
|||
| def log_increase(cpu_utilization:, current_concurrency:, new_concurrency:) | |||
| def log_increase(cpu_utilization:, min_free_storage_in_mb:, current_concurrency:, new_concurrency:) | |||
There was a problem hiding this comment.
I still think that min is overloaded (since it can refer both to the threshold at which we pause indexing or the value we compare to that threshold). It's not very clear which this is. Can you include both (they both seem useful to log) and name them appropriately, similar to the renaming you did in the lambda logic? e.g. lowest_node_free_storage_in_mb and required_free_storage_in_mb.
(And please apply that throughout this file.)
| @@ -26,7 +26,9 @@ def handle_request(event:, context:) | |||
| min_cpu_target: event.fetch("min_cpu_target"), | |||
| max_cpu_target: event.fetch("max_cpu_target"), | |||
| maximum_concurrency: event.fetch("maximum_concurrency"), | |||
| indexer_function_name: event.fetch("indexer_function_name") | |||
| required_free_storage_in_mb: event.fetch("min_free_storage_in_mb"), | |||
There was a problem hiding this comment.
/nit it feels inconsistent to call it required_free_storage_in_mb internally but call it min_free_storage_in_mb in the event payload. Can you call it required_free_storage_in_mb in both spots? Every other event key gets passed through as-is instead of getting renamed...
There was a problem hiding this comment.
ah yeah good point, that should match to be consistent
akumar1214
force-pushed
the
akumar/autoscaler-min-free-storage
branch
from
4f002b7 to
8451f65
Compare
Base automatically changed from
akumar/autoscaler-tune-lambda-concurrency
to
main
November 12, 2024 19:49
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The amount of free space declines much faster during a backfill than live indexing and if there is a node that runs out of space, the node will crash and lose data if not caught in time. This adds an extra layer of protection during backfills to set the concurrency to the minimum to pause indexing if the node with the least amount of free storage drops below a client-set threshold.
This approach uses the CloudWatch client to retrieve the free storage metrics instead of directly from Elasticsearch. I compared the Elasticsearch disk space metrics to CloudWatch and they were ~10,000 MB higher which seemed high enough to prefer CloudWatch. CloudWatch metrics can be delayed ~10-15 minutes but this is fine enough for a backfill. Getting the metric directly from Opensearch also wouldn't be a minimum over time and could be prone to spikes.
How tested?