Skip to content

Commit

Permalink
docs: update topics for Blooms (grafana#15028)
Browse files Browse the repository at this point in the history
  • Loading branch information
JStickler authored Nov 20, 2024
1 parent be24862 commit 469b54e
Show file tree
Hide file tree
Showing 2 changed files with 27 additions and 15 deletions.
12 changes: 8 additions & 4 deletions docs/sources/get-started/labels/structured-metadata.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,10 @@ You should only use structured metadata in the following situations:

- If you are ingesting data in OpenTelemetry format, using Grafana Alloy or an OpenTelemetry Collector. Structured metadata was designed to support native ingestion of OpenTelemetry data.
- If you have high cardinality metadata that should not be used as a label and does not exist in the log line. Some examples might include `process_id` or `thread_id` or Kubernetes pod names.

It is an antipattern to extract information that already exists in your log lines and put it into structured metadata.
- If you are using [Explore Logs](https://grafana.com/docs/grafana-cloud/visualizations/simplified-exploration/logs/) to visualize and explore your Loki logs.
- If you are a large-scale customer, who is ingesting more than 75TB of logs a month and are using [Bloom filters](https://grafana.com/docs/loki/<LOKI_VERSION>/operations/bloom-filters/)

We do not recommend extracting information that already exists in your log lines and putting it into structured metadata.

## Attaching structured metadata to log lines

Expand All @@ -37,9 +39,10 @@ See the [Promtail: Structured metadata stage](https://grafana.com/docs/loki/<LOK
With Loki version 1.2.0, support for structured metadata has been added to the Logstash output plugin. For more information, see [logstash](https://grafana.com/docs/loki/<LOKI_VERSION>/send-data/logstash/).

{{< admonition type="warning" >}}
Structured metadata size is taken into account while asserting ingestion rate limiting.
Structured metadata size is taken into account while asserting ingestion rate limiting.
Along with that, there are separate limits on how much structured metadata can be attached per log line.
```

```yaml
# Maximum size accepted for structured metadata per log line.
# CLI flag: -limits.max-structured-metadata-size
[max_structured_metadata_size: <int> | default = 64KB]
Expand All @@ -48,6 +51,7 @@ Along with that, there are separate limits on how much structured metadata can b
# CLI flag: -limits.max-structured-metadata-entries-count
[max_structured_metadata_entries_count: <int> | default = 128]
```
{{< /admonition >}}
## Querying structured metadata
Expand Down
30 changes: 19 additions & 11 deletions docs/sources/query/query_accceleration.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
title: Query acceleration (Experimental)
title: Query acceleration
menuTitle: Query acceleration
description: Provides instructions on how to write LogQL queries to benefit from query acceleration.
weight: 900
Expand All @@ -8,10 +8,11 @@ keywords:
- query acceleration
---

# Query acceleration (Experimental)
# Query acceleration

{{< admonition type="warning" >}}
Query acceleration using blooms is an [experimental feature](/docs/release-life-cycle/). Engineering and on-call support is not available. No SLA is provided.
In Loki and Grafana Enterprise Logs (GEL), Query acceleration using blooms is an [experimental feature](/docs/release-life-cycle/). Engineering and on-call support is not available. No SLA is provided.
In Grafana Cloud, Query acceleration using blooms is enabled for large-scale customers that send more than 75TB of logs a month as a [public preview](/docs/release-life-cycle/) with limited support and no SLA.
{{< /admonition >}}

If [bloom filters][] are enabled, you can write LogQL queries using [structured metadata][] to benefit from query acceleration.
Expand All @@ -26,19 +27,26 @@ If [bloom filters][] are enabled, you can write LogQL queries using [structured
Queries will be accelerated for any [label filter expression][] that satisfies _all_ of the following criteria:

* The label filter expression using **string equality**, such as `| key="value"`.
* `or` and `and` operators can be used to match multiple values, such as `| detected_level="error" or detected_level="warn"`.
* _Basic_ regular expressions are automatically simplified into a supported expression:
* `| key=~"value"` is converted to `| key="value"`.
* `| key=~"value1|value2"` is converted to `| key="value1" or key="value2"`.
* `| key=~".+"` checks for existence of `key`. `.*` is not supported.
* `or` and `and` operators can be used to match multiple values, such as `| detected_level="error" or detected_level="warn"`.
* _Basic_ regular expressions are automatically simplified into a supported expression:
* `| key=~"value"` is converted to `| key="value"`.
* `| key=~"value1|value2"` is converted to `| key="value1" or key="value2"`.
* `| key=~".+"` checks for existence of `key`. `.*` is not supported.
* The label filter expression is querying for structured metadata and not a stream label.
* The label filter expression is placed before any [parser expression][], [labels format expression][], [drop labels expression][], or [keep labels expression][].

To take full advantage of query acceleration with blooms, ensure that filtering structured metadata is done before any parse expression:
To take full advantage of query acceleration with blooms, ensure that filtering structured metadata is done before any parser expression:

In the following example, the query is not accelerated because the structured metadata filter, `detected_level="error"`, is after a parser stage, `json`.

```logql
{cluster="prod"} | logfmt | json | detected_level="error"
```

In the following example, the query is accelerated because the structured metadata filter is before any parser stage.

```logql
{cluster="prod"} | logfmt | json | detected_level="error" # NOT ACCELERATED: structured metadata filter is after a parse stage
{cluster="prod"} | detected_level="error" | logfmt | json # ACCELERATED: structured metadata filter is before any parse stage
{cluster="prod"} | detected_level="error" | logfmt | json
```

[bloom filters]: https://grafana.com/docs/loki/<LOKI_VERSION>/operations/bloom-filters/
Expand Down

0 comments on commit 469b54e

Please sign in to comment.