feat: move detected field logic to query frontend #14248
Merged
+1,653
−1,049
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport 36ace66 from #14212
What this PR does / why we need it:
/detected_fields
queries in ops are failing over bigger time ranges in Explore Logs, whereas logs queries for the same time period are not. I think this is because of all the optimizations/protections we have in place for logs queries (result cache, sharding, limits, etc.) that have not been implemented for/detected_fields
. However, since/detected_fields
is just doing aSelectLogs
under the hood in the querier, if we move the detected fields logic up to the Query Frontend we can use the existing limited and filter tripperwares to get the underlying logs. Due to the limits already in place in those tripperwares, they will never return too many lines for the detected fields logic to reasonably handle in the Query Frontend, which allows us to move this logic there without fear of knocking over QFs.This replaces #14210 as it works much better. I have tested in in Ops and it solves the problem without overloading QFs.
Special notes for your reviewer:
Checklist
CONTRIBUTING.md
guide (required)feat
PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.docs/sources/setup/upgrade/_index.md
production/helm/loki/Chart.yaml
and updateproduction/helm/loki/CHANGELOG.md
andproduction/helm/loki/README.md
. Example PRdeprecated-config.yaml
anddeleted-config.yaml
files respectively in thetools/deprecated-config-checker
directory. Example PR