feat: tune latency attribute buckets to reduce cardinality

[deniseli]

Fixes #2286

Log Buckets

This PR introduces the ability to constrain logBucket with a max and min bucket. Log buckets just bucket numbers (in this case, ms latency):

Example for base 4:

<1, [1,4), [4,16), [16,64), [64,256), [256,1024), [1024,4096), ...

In DataDog, looking across the last week, we see several latency buckets displaying low counts in the noise, especially on the buckets closer to 1. This makes sense given the intrinsic nature of log buckets - they are densest close to 1 and sparser the further up you go. To reduce cardinality (thus, cost), we can chop away that noise at the low end.

It's similarly worth capping the buckets at the high end because although the buckets get exponentially larger, you can still have infinitely more buckets. We haven't been hit with any major issues yet, but you can imagine how this could impact cardinality during a latency spike.

Call Latency

DataDog links: bucketed counts, avg latency rollup

Currently set to base 2 without min/max.

• <1, 1-2 .. 8-16: noise
• 16-32 .. 1024-2048: healthy number of data points
• 2048-4096, 4096-8192: some spikes, but low data
• 8192-16384: healthy number of data points (worth investigating separately from this change)
• 15 buckets total

Proposed change: base 4, min 4^3 (i.e. <64 is the smallest bucket), max 4^7 (i.e. >=16384 is the highest bucket, accounting for outliers).

• => Max cardinality = 6 buckets

Async Call Latency

DataDog links: bucketed counts, avg latency rollup

Currently set to base 8 without min/max.

• 8-64: noise. Note that <1 and 1-8 buckets have never occurred, though they are possible in the current setup.
• 64-512, 512-4096: healthy number of data points
• 4096-32768: noise
• 4 buckets total

Proposed change: base 4, min 4^4 (i.e. <256 is the smallest bucket), max 4^6 (i.e. >=4096 is the highest bucket)

• => Max cardinality = 4 buckets

Ingress Latency

DataDog links: bucketed counts, avg latency rollup
Currently set to base 2 without min/max.

• <1, 1-2 .. 16-32: noise.
• 32-64 .. 1024-2048: healthy number of data points
• 2048-4096, 4096-8192 : noise
• 8192-16384: healthy number of data points (worth investigating separately from this change)
• 15 buckets total

Proposed change: base 4, min 4^3 (i.e. <64 is the smallest bucket), max 4^7 (i.e. >=16384 is the highest bucket).

• => Max cardinality = 6 buckets

[safeer]

Nice! Might we want to have the largest bucket specifically for outliers? I.e. > 4^7 for the call latency and ingress latency buckets

[deniseli]

Nice! Might we want to have the largest bucket specifically for outliers? I.e. > 4^7 for the call latency and ingress latency buckets

Great idea! I'll add that in now.