fix: sanatize structured metadata at query time

[MasslessParticle]

There's a bug in structured metadata where Loki can accept characters that are invalid in prometheus label names. A subsequent PR will reject inputs but the data that's already been ingested is not queryable.

As a workaround, this PR sanatizes structured metadata label names at query time. In the case where no bad inputs exist, there isn't a meaningful performance difference. Otherwise, there is 1 alloc per bad input. We could do this with byte slices to avoid incurring allocs, but doing it this way is the most straightforward way to ensure we aren't missing multi-byte characters that may exist in label names.

benchmark before:

goos: darwin
goarch: arm64
pkg: github.com/grafana/loki/v3/pkg/logql/log
Benchmark_Pipeline/pipeline_bytes-12              270484              4416 ns/op            1795 B/op         34 allocs/op
Benchmark_Pipeline/pipeline_string-12             266888              4432 ns/op            1859 B/op         35 allocs/op
Benchmark_Pipeline/line_extractor_bytes-12                                        241034              4940 ns/op            1489 B/op         33 allocs/op
Benchmark_Pipeline/line_extractor_string-12                                       243536              4933 ns/op            1489 B/op         33 allocs/op
Benchmark_Pipeline/label_extractor_bytes-12                                       236311              5224 ns/op            1489 B/op         33 allocs/op
Benchmark_Pipeline/label_extractor_string-12                                      239662              5015 ns/op            1489 B/op         33 allocs/op

benchmark after

goos: darwin
goarch: arm64
pkg: github.com/grafana/loki/v3/pkg/logql/log
Benchmark_Pipeline/pipeline_bytes-12              244771              5156 ns/op            1795 B/op         34 allocs/op
Benchmark_Pipeline/pipeline_string-12             244720              4869 ns/op            1859 B/op         35 allocs/op
Benchmark_Pipeline/pipeline_bytes_no_invalid_structured_metadata-12               238662              5326 ns/op            1555 B/op         35 allocs/op
Benchmark_Pipeline/pipeline_string_with_invalid_structured_metadata-12            228170              5219 ns/op            1664 B/op         37 allocs/op
Benchmark_Pipeline/line_extractor_bytes-12                                        206654              5378 ns/op            1490 B/op         33 allocs/op
Benchmark_Pipeline/line_extractor_string-12                                       222890              5429 ns/op            1490 B/op         33 allocs/op
Benchmark_Pipeline/label_extractor_bytes-12                                       218815              5601 ns/op            1490 B/op         33 allocs/op
Benchmark_Pipeline/label_extractor_string-12                                      208882              5470 ns/op            1490 B/op         33 allocs/op

[rfratto]

Will this break anything for keys coming from OTel attributes, since (IIRC) periods are common there?

[MasslessParticle]

@rfratto yes and no. Yes because Loki will happily ingest periods so users will be querying for something unexpected. No because this data is unqueryable right now because the invalid labels will break here.

[rfratto]

Makes sense to me.

I have some concerns about the behaviour at query time if two different keys sanitize to the same value (i.e., hello.world and hello!world both sanitize to hello_world), but I don't know enough about the LogQL engine yet to know if this would be a real issue.

[ashwanthgoli]

looking at prometheus NormalizeLabel, we also need to handle labels starting with a number

[MasslessParticle]

great find. I didn't realize NormalizeLabel was a thing!

[cyriltovena]

I don't think we should validate structured metadata in the future at ingest time. I would vote instead to disable validation for returned labels and structured metadata. This seems trickier to implement but more future-proof.

Only labels should be validate at ingest time.

EDIT: we should avoid query-time validation/normalization.