feat: add pattern match line filter #12398
Conversation
| ErrNoCapture = errors.New("at least one capture is required") | ||
| ErrInvalidExpr = errors.New("invalid expression") | ||
| ErrNoCapture = errors.New("at least one capture is required") | ||
| ErrCaptureNotAllowed = errors.New("named captures are not allowed") |
There was a problem hiding this comment.
This is debatable, but I think it makes sense: if a user tries to parse fields at the filter stage, it's better to return an explicit error rather than silently ignore the user's intent. In the future, the restriction can be removed.
Note that we're using an unnamed placeholder <_>. I think we could use a new capture identifier (such as <*>) to emphasize the difference with the pattern parse stage. However, it feels like a new syntax for pattern matching, which may confuse users. I would like to hear others' thoughts on this.
There was a problem hiding this comment.
I don't have strong opinitions using _ vs * but I definitively agree we should fail fast.
There was a problem hiding this comment.
I would vote for <_> as the behavior between these two examples feels consistent to me:
|> "dur=<_> err=<_>"
| pattern "dur=<_> err=<_>"
I think of that as telling Loki to ignore any of the content in <_>
I'm not sure I see the reason to use <*> or maybe I'm misunderstanding something?
There was a problem hiding this comment.
This popped up in the Slack conversation, so I brought it here.
Thank you, guys, for sharing your thoughts. I keep <_>.
| @@ -539,6 +581,10 @@ func TestStringer(t *testing.T) { | |||
| in: `{app="foo"} !~ ip("127.0.0.1") or "foo"`, | |||
| out: `{app="foo"} !~ ip("127.0.0.1") !~ "foo"`, | |||
| }, | |||
| { | |||
| in: `{app="foo"} !> "<_> foo <_>" or "foo <_>" !> "foo <_> baz"`, | |||
There was a problem hiding this comment.
I think the or is a bug here and should not be support but I see it elsewhere. I didn't know about this cc @kavirajk
There was a problem hiding this comment.
It looks legit to me: note the De Morgan's law a few lines above (!(A || B) == !A && !B):
not match ("<_> foo <_>" or "foo <_>") is the same as
not match "<_> foo <_>" and not match "foo <_>"
|
please add a changelog item btw ! |
The PR introduces new line filter operators for pattern matching,
|>and!>. Despite the fact that filtering by patterns is a subset of the regex filter scope, we decided to extend the syntax explicitly.The filter presumes that placeholders mask arbitrary non-empty literals. Thus, the
"<_>"pattern matches any non-empty line, and the""pattern only matches empty lines. The semantics are identical to thepatternparse stage, with an exception that named captures are not allowed: fields are never extracted at filtering.The PR does not include documentation. At this point, the new feature is somewhat experimental and should be used with caution. The same applies to the frontend query builder and syntax checker.
Fixes #12347
Checklist
CONTRIBUTING.mdguide (required)CHANGELOG.mdupdatedadd-to-release-noteslabeldocs/sources/setup/upgrade/_index.mdproduction/helm/loki/Chart.yamland updateproduction/helm/loki/CHANGELOG.mdandproduction/helm/loki/README.md. Example PRdeprecated-config.yamlanddeleted-config.yamlfiles respectively in thetools/deprecated-config-checkerdirectory. Example PR