Extremely Weird/Bad Metrics Consumer Behavior with Prometheus.NET and/or AppInsights

[pinkfloydx33]

I recently added Prometheus metrics to my YARP proxy, essentially copying the samples in this repo. The only difference is that I'm using the latest versions of the Prometheus.NET libraries (8.0.0 vs 4.1.0)

I noticed something very odd about the metrics being produced. I saw all the yarp_-prefixed metrics from the sample project, but also a bunch of yarp_reverseproxy_-prefixed ones that I couldn't quite track down at first. What's even weirder is that all the metrics were slowly increasing without any interaction with the application!

For example, I'd make a single request through the proxy to a page that also requests 2 additional resources. As expected yarp_proxy_requests_started would initially return 3. However, refreshing my metrics endpoint would show it climbing higher and higher. Meanwhile, the mystery yarp_reverseproxy_requests_started metric would stay at 3.

I stuck a breakpoint in PrometheusForwarderMetrics.OnMetrics and made no requests to the application. I noticed that on each (almost constant) invocation of the method that current.RequestsStarted would alternate between the real value and an ever-increasing alternate value. For example, here's the value for 10 consecutive hits of the breakpoint:

• 3
• 3
• 17
• 3
• 3
• 17
• 3
• 3
• 18
• 19

On each hit that wasn't 3 , the IncTo method would increment and distort the metrics.

So here's what I've now learned. The yarp_reverseproxy_ metrics are coming from Prometheus.NET's default metrics factory and it's inclusion of EventCounters. It's translating the the name of the event source from "Yarp.ReverseProxy" to "yarp_reverseproxy_". This also seems to somewhat be the culprit with the weird metric values being reported by YARP into my metrics consumer. I think the fact the Metric Producers/Services are implemented as EventSource/ EventListener is somehow conflicting with the default Prometheus behavior.

To fix the "jumping around" of the data (3-17-3), I had to disable built-in event counters tracking with a suppression at startup:

Metrics.SuppressDefaultMetrics(new SuppressDefaultMetricOptions
{
    SuppressEventCounters = true
});

Even with this "fix", the value of current.RequestsStarted still constantly increases! I can close all browsers and set a breakpoint and the value still changes! At least now it only increases and doesn't jump around, but that still isn't correct.

It turns out that I also have Application Insights installed. After a bunch of guess-and-check I isolated it to the on-by-default EnableQuickPulseMetricStream option. So long as that is true, YARP reports metrics value increases when the application is not receiving any traffic. If I set it to false then suddenly YARP stops reporting changes for no reason. This isn't limited to RequestsStarted... it includes BytesSent and all the various other metrics.

Has anyone experienced any of this before? Does it sound like there could be a bug on the YARP side? Or with Prometheus.NET, AppInsights, or all of the above?

Aside: should the samples be updated to use the most recent version of Prometheus.NET and potentially call-out the above issue?

Edit: I thought I had completely worked-around the issue. I had been validating the last bits without hitting a proxy route as the behavior would trigger on initial load. But I found one more issue once I actually hit a proxy route. If the AppInsights DependencyTrackingTelemetryModule is enabled, then once I actually navigate to a page the Requests Started value chain alternates between two values: ie 3-4-3-4-3-4-etc

[MihaZupan]

This is likely related to #2102. There is a bug in EventListener that YARP's telemetry consumption implementation runs into when you listen to both HttpMetrics and ForwarderMetrics.
We've recently worked around this issue in YARP: #2104. Are you able to reproduce the issue using a daily build of YARP?

[pinkfloydx33]

Doh! I don't know how I missed that.... And I wish I saw your response an hour ago as I've been chasing down yet another thing that was still off!

I've stepped away from my desk for the evening but will try the daily feed first thing tomorrow assuming I can get connected up to it.

[pinkfloydx33]

I tried installing latest previews 2.1.0-preview.1.23253.1 but I'm missing Microsoft.Bcl.TimeProvider package and can't find it anywhere. Any idea where I might acquire it?

[MihaZupan]

If you target net8.0 in your project, it will already be included in the base libraries.

Alternatively, you can also add the dotnet8 NuGet feed

<add key="dotnet8" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet8/nuget/v3/index.json" />

I'll update our DailyBuilds docs to include this.

[pinkfloydx33]

If the daily build fix works, the samples should probably still be updated to use the latest prometheus libraries. They're several major versions behind. As is, you're still going to get double metrics for yarp-related things (under different names) unless you either:

A. opt-out of EventCounter instrumentation completely (as in my OP), or
B. pass a predicate to ConfigureEventCounterAdapter that excludes any sources that Yarp is using for it's metrics (based on "Yarp.ReverseProxy" prefix maybe?)

[pinkfloydx33]

Daily build worked! The socket bytes sent metrics constantly increases if I leave the App Insights Quick Pulse stream enabled, but that makes sense since it's constantly using sockets to communicate with Azure. I don't think there's any way for that to be excluded (by me, or generally).

I still have an issue that I believe is a bug in prometheus.net where I'm getting the event source metrics when they shouldn't be included by default (and filtering options don't seem to work). I opened an issue with them and crossing-fingers YARP isn't somehow to blame.

Regarding the fix: is it possible to get that back-ported? I'm more than happy to use preview/daily feeds to debug and validate but I can't push that into production... especially with the BCL preview package (we cannot update to net8 until at least GA). It seems like it can have a more general impact for anyone wishing to trust metrics coming out of YARP.