Misunderstanding query_key and alerting

[Saintel]

I have documents in ElasticSearch which contains at least 5 fields: solution, source, environment, region, level
And i have to get alert, when one of resulting group solution*source*environment*region contains 10+ documents with level:<4

My config:

type: frequency

timeframe:
  minutes: 5

query_key:
 - solution
 - source
 - environment
 - region

num_events: 10

filter:
- query:
    query_string: {query: 'solution: ("mySolution1","mySolution2) AND level:<4'}

alertmanager_fields:
  solution: solution
  source: source
  environment: environment
  region: region

alert_text: "Too many exceptions: {0}"
alert_text_type: alert_text_only
alert_text_args:
- num_matches
include: ["solution","environment","source","region"]

But in alert i recieved the same num_matches (in test cases):

Alerts, [13.11.2024 11:41]
🔥 Alerts Firing 🔥


🪪 Exceptions
📖 Too many exceptions: 18

environment: Production
region: Region2
solution: mySolution1
source: RouterApi

Alerts, [13.11.2024 11:41]
🔥 Alerts Firing 🔥

🪪 Exceptions
📖 Too many exceptions: 18

environment: Production
region: Region2
solution: mySolution2
source: Jobs.ActivationReminder

🔥 Alerts Firing 🔥


🪪 Exceptions
📖 Too many exceptions: 18

environment: Test
region: Region1
solution: mySolution1
source: Platform.Backend

...
etc.

And when i manualy check documents, the counters per group not match with alerts.

What i missing? Can anyone help me to achieve results i need?

[jertel]

In Frequency rules, num_matches represents the recent count of matches across all query keys. You can either split this rule into four separate Frequency rules, one per each field you're using for query_key, or you can write an enhancement to grab the len(rule.occurrences[query_key]) value and push that back into the match dict for use by your alerter.

[Saintel]

@jertel Thank you for your reply.
Unfortunately, i need to check count after grouping.

For example:
I have 3 service in my solution: serviceA, serviceB, serviceC
Each of them is located in 3 regions: south,east,west
And each of them has 2 environments: production,test

There are 3*3*2=18 independent services

And i need to know per solution which service has problems (level:<4) and %doc_count% > 10
May be i need to change type? Or something else?

[jertel]

The query_key group count is compared to the num_events to determine if an alert needs to be sent. I believe that part is as you are expecting. However, the num_matches you are wanting to include in your alert message is not showing you that query_key group count, and to get that would require splitting up the rule into multiple, which perhaps would be 18 rules based on your math above. Not ideal. Alternatively, write an enhancement to attempt to retrieve this true count. There's also the option of submitting a PR to adjust the frequency rule to include the query_key count at the time that the match is detected (ruletypes.py::FrequencyRule::check_for_match).