Expose platform properties in scheduler metrics #151
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently, prometheus metrics on the scheduler report all platform properties as a json-serialized string under the "properties" label. This is useful for bb-autoscaler, which knows how to parse this value, but is very challenging to use with tooling which might already be available for autoscaling in your setup, like keda. It also makes it challenging to dashboard or analyze queue metrics.
This change modifies the metric reporting behavior in the scheduler, adding all entries found in the platform to the labels. Collisions are resolved with comma-delimiting the values, which will act as a consistent label due to the strict ordering on properties accepted by buildbarn. Existing labels (platform, instance_name_prefix, and size_class) are prioritized, and cannot be renamed to avoid breaking existing alerts, and to avoid interfering with bb-autoscaler.
Implementation-wise, the individual metrics have been converted to templates which are instantiated whenever a new size class queue is created. The size class queue provisions and registers a registry for its metrics, which it deregisters in its
remove
method.