Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: update topics for Blooms #15028

Merged
merged 1 commit into from
Nov 20, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 8 additions & 4 deletions docs/sources/get-started/labels/structured-metadata.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,10 @@ You should only use structured metadata in the following situations:

- If you are ingesting data in OpenTelemetry format, using Grafana Alloy or an OpenTelemetry Collector. Structured metadata was designed to support native ingestion of OpenTelemetry data.
- If you have high cardinality metadata that should not be used as a label and does not exist in the log line. Some examples might include `process_id` or `thread_id` or Kubernetes pod names.

It is an antipattern to extract information that already exists in your log lines and put it into structured metadata.
- If you are using [Explore Logs](https://grafana.com/docs/grafana-cloud/visualizations/simplified-exploration/logs/) to visualize and explore your Loki logs.
- If you are a large-scale customer, who is ingesting more than 75TB of logs a month and are using [Bloom filters](https://grafana.com/docs/loki/<LOKI_VERSION>/operations/bloom-filters/)

We do not recommend extracting information that already exists in your log lines and putting it into structured metadata.

## Attaching structured metadata to log lines

Expand All @@ -37,9 +39,10 @@ See the [Promtail: Structured metadata stage](https://grafana.com/docs/loki/<LOK
With Loki version 1.2.0, support for structured metadata has been added to the Logstash output plugin. For more information, see [logstash](https://grafana.com/docs/loki/<LOKI_VERSION>/send-data/logstash/).

{{< admonition type="warning" >}}
Structured metadata size is taken into account while asserting ingestion rate limiting.
Structured metadata size is taken into account while asserting ingestion rate limiting.
Along with that, there are separate limits on how much structured metadata can be attached per log line.
```

```yaml
# Maximum size accepted for structured metadata per log line.
# CLI flag: -limits.max-structured-metadata-size
[max_structured_metadata_size: <int> | default = 64KB]
Expand All @@ -48,6 +51,7 @@ Along with that, there are separate limits on how much structured metadata can b
# CLI flag: -limits.max-structured-metadata-entries-count
[max_structured_metadata_entries_count: <int> | default = 128]
```

{{< /admonition >}}

## Querying structured metadata
Expand Down
30 changes: 19 additions & 11 deletions docs/sources/query/query_accceleration.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
title: Query acceleration (Experimental)
title: Query acceleration
menuTitle: Query acceleration
description: Provides instructions on how to write LogQL queries to benefit from query acceleration.
weight: 900
Expand All @@ -8,10 +8,11 @@ keywords:
- query acceleration
---

# Query acceleration (Experimental)
# Query acceleration

{{< admonition type="warning" >}}
Query acceleration using blooms is an [experimental feature](/docs/release-life-cycle/). Engineering and on-call support is not available. No SLA is provided.
In Loki and Grafana Enterprise Logs (GEL), Query acceleration using blooms is an [experimental feature](/docs/release-life-cycle/). Engineering and on-call support is not available. No SLA is provided.
In Grafana Cloud, Query acceleration using blooms is enabled for large-scale customers that send more than 75TB of logs a month as a [public preview](/docs/release-life-cycle/) with limited support and no SLA.
{{< /admonition >}}

If [bloom filters][] are enabled, you can write LogQL queries using [structured metadata][] to benefit from query acceleration.
Expand All @@ -26,19 +27,26 @@ If [bloom filters][] are enabled, you can write LogQL queries using [structured
Queries will be accelerated for any [label filter expression][] that satisfies _all_ of the following criteria:

* The label filter expression using **string equality**, such as `| key="value"`.
* `or` and `and` operators can be used to match multiple values, such as `| detected_level="error" or detected_level="warn"`.
* _Basic_ regular expressions are automatically simplified into a supported expression:
* `| key=~"value"` is converted to `| key="value"`.
* `| key=~"value1|value2"` is converted to `| key="value1" or key="value2"`.
* `| key=~".+"` checks for existence of `key`. `.*` is not supported.
* `or` and `and` operators can be used to match multiple values, such as `| detected_level="error" or detected_level="warn"`.
* _Basic_ regular expressions are automatically simplified into a supported expression:
* `| key=~"value"` is converted to `| key="value"`.
* `| key=~"value1|value2"` is converted to `| key="value1" or key="value2"`.
* `| key=~".+"` checks for existence of `key`. `.*` is not supported.
* The label filter expression is querying for structured metadata and not a stream label.
* The label filter expression is placed before any [parser expression][], [labels format expression][], [drop labels expression][], or [keep labels expression][].

To take full advantage of query acceleration with blooms, ensure that filtering structured metadata is done before any parse expression:
To take full advantage of query acceleration with blooms, ensure that filtering structured metadata is done before any parser expression:

In the following example, the query is not accelerated because the structured metadata filter, `detected_level="error"`, is after a parser stage, `json`.

```logql
{cluster="prod"} | logfmt | json | detected_level="error"
```

In the following example, the query is accelerated because the structured metadata filter is before any parser stage.

```logql
{cluster="prod"} | logfmt | json | detected_level="error" # NOT ACCELERATED: structured metadata filter is after a parse stage
{cluster="prod"} | detected_level="error" | logfmt | json # ACCELERATED: structured metadata filter is before any parse stage
{cluster="prod"} | detected_level="error" | logfmt | json
```

[bloom filters]: https://grafana.com/docs/loki/<LOKI_VERSION>/operations/bloom-filters/
Expand Down
Loading