Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Structured metadata stream stats #15427

Draft
wants to merge 21 commits into
base: main
Choose a base branch
from

Conversation

salvacorts
Copy link
Contributor

@salvacorts salvacorts commented Dec 16, 2024

What this PR does / why we need it:

This PR adds structured metadata (SM) stats to the TSDB index. For each stream we save a list of available structured metadata field names. We then use these stats to:

  • Return a list of available SM fields for the /api/labels endpoint (both on the ingester and the index).
  • On the query path, skip all series not having a given SM field we are filtering by (here).

On ingesters, for each stream we keep (here):

  • A closed set of stats: has the stats up to the most recent closed chunk.
  • An open set of stats: these are updated as we receive push requests (closed set + newer data).
    On the flush loop, the Ingester updates the TSDB with the closed set of stats.

We store these stats next to the label set of each stream on TSDB. As for regular labels, SM names are referenced using the symbols table.
image

Special notes for your reviewer:

Checklist

  • Reviewed the CONTRIBUTING.md guide (required)
  • Documentation added
  • Tests updated
  • Title matches the required conventional commits format, see here
    • Note that Promtail is considered to be feature complete, and future development for logs collection will be in Grafana Alloy. As such, feat PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.
  • Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
  • If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

@salvacorts salvacorts changed the title Structured metadata indexing feat: Structured metadata stream stats Dec 16, 2024
@github-actions github-actions bot added the type/docs Issues related to technical documentation; the Docs Squad uses this label across many repositories label Dec 16, 2024
Copy link
Contributor

github-actions bot commented Dec 16, 2024

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
size/XXL type/docs Issues related to technical documentation; the Docs Squad uses this label across many repositories
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants