Getting number of hits for each unique query_key

[RezaBagheri3]

hello everyone,

I have following rule :

name: error-DeactivateSupplier

type: frequency

index: my_index

category: error-frequency

num_events: 35

timeframe:
  minutes: 10
  
realert:
  minutes: 10

buffer_time:
  minutes: 10

filter:
  - query:
      query_string:
        query: 'level: "Error"'
      
query_key: fields.supplierName
  
kibana_discover_to_timedelta: 
  minutes: 0

alert:
  - "ms_teams"

alert_subject: 'Deactivate Supplier'

ms_teams_alert_summary: "Supplier must be Deactivated."

alert_text: 'error more than 35 times per 10 minute <hr style="height: 2px; background-color: #FDB713; border: none;" />
Supplier Name: {0}<br>
Error Count: {1}<br>
Kibana link: [kibana]({2})'

alert_text_args:
  - fields.supplierName
  - num_hits
  - kibana_discover_url

alert_text_type: alert_text_only

ms_teams_webhook_url: my URL

I want to alert when each unique supplierName has more than 35 logs with "Error" level in last 10 minute from current time
also run_every is set to 30 second in config.yaml

at the moment I used num_hits to get error count for each unique supplierName, when an alert sends to Microsoft teams channel I open the kibana dashboard and write a query for past 10 minute for that specific supplierName which is specified in the alert message with level : Error, then the number of hits returned from query is different from Error count that specified in Microsoft teams alert message

My question is how to send the count of errors(currently I am using num_hits) of each unique supplier in the last 10 minutes in the alert message?
I am also struggling with the buffer time and I don't know what to set the value of the buffer time so that the query is executed only in the last 10 minutes, not more and no less.

[jertel]

Unfortunately, the counts per query_key are not currently made available to the alerters.

timeframe: Setting to 10 minutes instructs the rules to only include records that fall within the past 10 minutes.
buffer_time: Setting to 10 minutes instructs ElastAlert 2 to query from 10m ago till present.
run_every: Setting to 10 seconds instructs ElastAlert 2 to run the rule every 10 seconds.

Therefore, your rule will query Elasticsearch every 10 seconds and each query will overlap the previous query time window by 9m 50s. For each result in the query, ElastAlert 2 will compare the @timestamp on the result with the timerange. If it's within 10m then it will add it to the match set, which is remembered across queries.

So it would appear your buffer_time of 10m is sufficient. If you wanted to make sure you don't miss any data you could extend it to a larger window, perhaps 15m. But then this would cause your query to have a larger time window, and you stated above you only want the query to execute for the last 10 minutes, however, you didn't explain why you want this. Your timeframe of 10m is also correct since you stated you only want to alert on events occurring in the past 10 minutes.

[RezaBagheri3]

I appreciate your help. Thank you!
But unfortunately, I still think that I haven't fully understood the difference between timeframe and buffer_time.
According to your answer, I think they do the same thing. If I am wrong please let me know.
If I want to query and filter only the logs related to the last 10 minutes, what should I use? both of them? or just one of them?
thanks again

[david-sykora]

@jertel would it be a problem to make counts per query_key available to alerters when elastalert2 has these numbers available in the result of a search query from elasticsearch?

or please correct me, I tried using es_debug_trace to see what the elasticsearch queries look like, part of the _search responses include the numbers we are looking for:

"aggregations": {
    "counts": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [
        {
          "key": "Some value",
          "doc_count": 5
        },
        {
          "key": "Other value",
          "doc_count": 1
        }
      ]
    }
  }

[jertel]

I think I looked into the code a while ago and came away thinking that it is possible. So it's likely just a matter of getting someone motivated enough to submit a PR that adds the counts to the match data, or something like that. If you're interested in submitting a PR please review the contributing guidelines in the root of this project, which will help get you started.

[nsano-rururu]

run_every

Monitoring interval

buffer_time

Data acquisition period

num_events

threshold. Alert if threshold is exceeded within period

timeframe

monitoring period

example

If the monitoring rule conditions are not matched within the period specified by buffer_time in config.yaml, nothing will happen. With this setting, you will not be notified unless a request with status code 500 has occurred 10 times in 5 minutes in the past hour.

config.yaml

es_host: 10.255.0.100
es_port: 9200
rules_folder: rules
buffer_time:
  hours: 1
run_every:
  minutes: 1
writeback_index: elastalert_status

This time I'm using the minimum settings.

The Elasticsearch indexes you want to monitor are all starting with logstash-. The condition of 10 times in 5 minutes is specified by num_events and timeframe. This time, we want to count only the values in the response field of 500, so we will filter using the term query.

status500-moniroting.yaml

name: status500-moniroting
type: frequency
index: logstash-*
num_events: 10
timeframe:
    minutes: 5
filter:
  - term:
      response: 500
alert:
  - slack
slack_webhook_url: "https://hooks.slack.com/services/xxxxxxxxx/xxxxxxxx/xxxxxxxxxxxxxxxxxxx"