From 469b54e3d58807d4bc24e4d247a2a205036b0954 Mon Sep 17 00:00:00 2001 From: J Stickler Date: Wed, 20 Nov 2024 11:10:39 -0500 Subject: [PATCH] docs: update topics for Blooms (#15028) --- .../get-started/labels/structured-metadata.md | 12 +++++--- docs/sources/query/query_accceleration.md | 30 ++++++++++++------- 2 files changed, 27 insertions(+), 15 deletions(-) diff --git a/docs/sources/get-started/labels/structured-metadata.md b/docs/sources/get-started/labels/structured-metadata.md index f1877139fef70..d76d1129471f9 100644 --- a/docs/sources/get-started/labels/structured-metadata.md +++ b/docs/sources/get-started/labels/structured-metadata.md @@ -23,8 +23,10 @@ You should only use structured metadata in the following situations: - If you are ingesting data in OpenTelemetry format, using Grafana Alloy or an OpenTelemetry Collector. Structured metadata was designed to support native ingestion of OpenTelemetry data. - If you have high cardinality metadata that should not be used as a label and does not exist in the log line. Some examples might include `process_id` or `thread_id` or Kubernetes pod names. - -It is an antipattern to extract information that already exists in your log lines and put it into structured metadata. +- If you are using [Explore Logs](https://grafana.com/docs/grafana-cloud/visualizations/simplified-exploration/logs/) to visualize and explore your Loki logs. +- If you are a large-scale customer, who is ingesting more than 75TB of logs a month and are using [Bloom filters](https://grafana.com/docs/loki//operations/bloom-filters/) + +We do not recommend extracting information that already exists in your log lines and putting it into structured metadata. ## Attaching structured metadata to log lines @@ -37,9 +39,10 @@ See the [Promtail: Structured metadata stage](https://grafana.com/docs/loki//send-data/logstash/). {{< admonition type="warning" >}} -Structured metadata size is taken into account while asserting ingestion rate limiting. +Structured metadata size is taken into account while asserting ingestion rate limiting. Along with that, there are separate limits on how much structured metadata can be attached per log line. -``` + +```yaml # Maximum size accepted for structured metadata per log line. # CLI flag: -limits.max-structured-metadata-size [max_structured_metadata_size: | default = 64KB] @@ -48,6 +51,7 @@ Along with that, there are separate limits on how much structured metadata can b # CLI flag: -limits.max-structured-metadata-entries-count [max_structured_metadata_entries_count: | default = 128] ``` + {{< /admonition >}} ## Querying structured metadata diff --git a/docs/sources/query/query_accceleration.md b/docs/sources/query/query_accceleration.md index 9117ecb209f23..a760b9e66faa2 100644 --- a/docs/sources/query/query_accceleration.md +++ b/docs/sources/query/query_accceleration.md @@ -1,5 +1,5 @@ --- -title: Query acceleration (Experimental) +title: Query acceleration menuTitle: Query acceleration description: Provides instructions on how to write LogQL queries to benefit from query acceleration. weight: 900 @@ -8,10 +8,11 @@ keywords: - query acceleration --- -# Query acceleration (Experimental) +# Query acceleration {{< admonition type="warning" >}} -Query acceleration using blooms is an [experimental feature](/docs/release-life-cycle/). Engineering and on-call support is not available. No SLA is provided. +In Loki and Grafana Enterprise Logs (GEL), Query acceleration using blooms is an [experimental feature](/docs/release-life-cycle/). Engineering and on-call support is not available. No SLA is provided. +In Grafana Cloud, Query acceleration using blooms is enabled for large-scale customers that send more than 75TB of logs a month as a [public preview](/docs/release-life-cycle/) with limited support and no SLA. {{< /admonition >}} If [bloom filters][] are enabled, you can write LogQL queries using [structured metadata][] to benefit from query acceleration. @@ -26,19 +27,26 @@ If [bloom filters][] are enabled, you can write LogQL queries using [structured Queries will be accelerated for any [label filter expression][] that satisfies _all_ of the following criteria: * The label filter expression using **string equality**, such as `| key="value"`. - * `or` and `and` operators can be used to match multiple values, such as `| detected_level="error" or detected_level="warn"`. - * _Basic_ regular expressions are automatically simplified into a supported expression: - * `| key=~"value"` is converted to `| key="value"`. - * `| key=~"value1|value2"` is converted to `| key="value1" or key="value2"`. - * `| key=~".+"` checks for existence of `key`. `.*` is not supported. + * `or` and `and` operators can be used to match multiple values, such as `| detected_level="error" or detected_level="warn"`. + * _Basic_ regular expressions are automatically simplified into a supported expression: + * `| key=~"value"` is converted to `| key="value"`. + * `| key=~"value1|value2"` is converted to `| key="value1" or key="value2"`. + * `| key=~".+"` checks for existence of `key`. `.*` is not supported. * The label filter expression is querying for structured metadata and not a stream label. * The label filter expression is placed before any [parser expression][], [labels format expression][], [drop labels expression][], or [keep labels expression][]. -To take full advantage of query acceleration with blooms, ensure that filtering structured metadata is done before any parse expression: +To take full advantage of query acceleration with blooms, ensure that filtering structured metadata is done before any parser expression: + +In the following example, the query is not accelerated because the structured metadata filter, `detected_level="error"`, is after a parser stage, `json`. + +```logql +{cluster="prod"} | logfmt | json | detected_level="error" +``` + +In the following example, the query is accelerated because the structured metadata filter is before any parser stage. ```logql -{cluster="prod"} | logfmt | json | detected_level="error" # NOT ACCELERATED: structured metadata filter is after a parse stage -{cluster="prod"} | detected_level="error" | logfmt | json # ACCELERATED: structured metadata filter is before any parse stage +{cluster="prod"} | detected_level="error" | logfmt | json ``` [bloom filters]: https://grafana.com/docs/loki//operations/bloom-filters/