Bring back RED metrics for querier when processing scheduler requests.

[jeschkies]

What this PR does / why we need it:
A previous change removed the RED metrics for the querier. This adds them back as part of a middleware.

Checklist

• Reviewed the CONTRIBUTING.md guide (required)
• Documentation added
• Tests updated
• CHANGELOG.md updated
  • If the change is worth mentioning in the release notes, add add-to-release-notes label
• Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
• For Helm chart changes bump the Helm chart version in production/helm/loki/Chart.yaml and update production/helm/loki/CHANGELOG.md and production/helm/loki/README.md. Example PR
• If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

[jeschkies]

So this could be used by EncodePath as well.

[cstyan]

LGTM, tested in dev as well. We get method=GET for /metrics, /ready, and /tail, method=grpc for the rest. Route label looks fine /loki/api/v1/... becomes loki_api_v1_... as expected.

[dannykopping]

Looks reasonable 👍
Have we compared the metrics produced with the previous ones?
Showing a clean diff would be tablestakes for merging this I think.

[jeschkies]

Callum verified that the metrics are present. I can create a diff

[jeschkies]

I've Loki locally with k172 and this patch.

curl -v -H "X-Scope-OrgID: 1" -G http://localhost:8004/metrics | cut -f 1 -d " " | sort | uniq > main.log
curl -v -H "X-Scope-OrgID: 1" -G http://localhost:8004/metrics | cut -f 1 -d " " | sort | uniq > k172.log
diff k172.log main.log > diff.log

Here's the diff.log. We are missing loki_request_message_bytes_bucket and loki_response_message_bytes_bucket. However, this patch is not addressing these.

[dannykopping]

I've Loki locally with k172 and this patch.

curl -v -H "X-Scope-OrgID: 1" -G http://localhost:8004/metrics | cut -f 1 -d " " | sort | uniq > main.log
curl -v -H "X-Scope-OrgID: 1" -G http://localhost:8004/metrics | cut -f 1 -d " " | sort | uniq > k172.log
diff k172.log main.log > diff.log

Here's the diff.log. We are missing loki_request_message_bytes_bucket and loki_response_message_bytes_bucket. However, this patch is not addressing these.

Can you attach the raw logs as well pls?

[jeschkies]

Here is the raw output from the /metrics endpoint, @dannykopping
metrics_k172.log
metrics_patch.log

[dannykopping]

I dropped the sample values from all the metrics to make the comparison purely on the series themselves.

$  awk 'NF{NF-=1};1' <metrics_k172.log >before.log
$  awk 'NF{NF-=1};1' <metrics_patch.log >after.log
$ diff -u {before,after}.log > diff.txt

I've attached the diff here. Most of it looks right but I'm between a few threads this morning and you have most context.
I'll approve the PR and if the diff looks good to you then merge it.

diff.txt