Kubernetes Custom Metrics

[pinkfloydx33]

Kubernetes offers a custom metric API by which any application can expose an endpoint of a particular shape and provide metrics of any type to the cluster. These metrics can then be used for horizontal pod autoscaling. Medium Article

A useful metric to have would be requests/second, tracked per Ingress. The HPA could then use this to scale the backends if the traffic begins to increase over a certain amount. This k8s walk through includes an example of leveraging a hypothetical requests-per-second metric on an Ingress to scale a Deployment.

Any idea what might be the best way to implement this? I was thinking that the new metrics APIs could be used, perhaps alongside a custom Middleware, to track the load. Of course I'd also need to expose the endpoint for retrieving the metrics... The latter seems like the easier part, but I'm a little lost on where to start with the former.

Has anyone implemented something like this for their YARP installation in Kubernetes? Or does anyone have any general thoughts/ideas? This could probably be baked in as part of YARP's Kubernetes Controller in some form, but I figured I'd see if I could get something working on my own before proposing it outright.

(Aside: supposedly Meter.CreateCounter can be used to track/retrieve such information, though I'm not sure if it can handle an arbitrary number of endpoints, nor what happens when the MaxValue of the underlying storage type is hit... so I'm not sure if it's even a feasible approach).

[MihaZupan]

This sounds similar to #124.

This sort of information (e.g. RPS per cluster) may be interesting even outside of the k8s controller scenario.

As for what API would be most appropriate, we're still working on that - dotnet/runtime#79371. Right now the majority of networking telemetry is implemented via EventCounters, but we are looking into potentially exposing Meter-like equivalents.

[pinkfloydx33]

I was thinking high-level: create a Meter, add a Middleware to the pipeline, have that Middleware inspect the proxy destination/cluster and increment the meter accordingly. And then use same meter for the api endpoint.

Haven't put pen to paper yet, not sure if/how well that can work with an arbitrary number of clusters though. Haven't had free time yet, but hopefully soon I'll try my hand at something

[adityamandaleeka]

@samsp-msft FYI

[pinkfloydx33]

Hrmm I just realized another gotcha. We'd need to be able to somehow Aggregate the per-cluster metrics across all instances of YARP, possibly using redis or some other means. You'd then either need a second dedicated metrics exporter app (similar to my links above) or a way for the individual instances to Aggregate said data on their own (i think?). Hrmm.... Need to think this one out a bit more

[samsp-msft]

Having a meter in YARP for request count dimensioned by route would be a good thing to have - hint hint @MihaZupan - that would provide the basic information, and would be the right thing for all YARP customers.
The data for that can be collected in-proc, or using dotnet-monitor which will expose it as a json & prometheus endpoint. I haven't fully parsed the medium post to determine what format k8s wants the data in?

[Tratcher]

@samsp-msft why limit it to YARP? Per our earlier discussion, what if we had a request rate/count metric for every route in AspNet?

[pinkfloydx33]

The format is hidden in the article inside of an image... I'm on mobile so can't really transcribe it.

The endpoint outputs a MetricValueList resource. Each item in the list consists of a metric name, a metric value, timestamp, and the resource it's describing (ie. which ingresses and/or services the routes are associated to)

Here's the image (for now)

The k8s apiserver uses a Service to locate the application producing the metrics. In a scenario where you have multiple YARP instances you'd need a way to. aggregate the metrics across all instances... Without it, k8s could hit any single one of them and only get a partial view. In the medium article they use a dedicated, single-instanced "metrics exporter" application that in theory would aggregate the data from the instances that are actually producing the metrics. Though, if you found a way to aggregate there's no reason it couldn't be part of that main application.

[Tratcher]

I filed dotnet/aspnetcore#46361 as a proposal for adding metering to all AspNetCore routes, not just YARP.