Rule not triggering alert.

[manishparmar5751]

Below is the rule, which was working before but does not seem to be now. Nothing has changes in our environment here.

Index switches is a test index and has no data coming in i.e always zero hits for this index. Jira credentials are good.

name: "Nodata flatline - switches"
type: flatline
index: "switches*"
realert:
  minutes: 2
threshold: 1
timeframe:
  minutes: 2
filter:
- query:
    match_all: {}
doc_type: _doc
alert:
- "jira"
jira_server: "https://XYZ.atlassian.net/"
jira_project: "CASE"
jira_issuetype: "Incident"
jira_account_file: "XYZ/examples/rules/jira_acct.txt"
jira_priority: 0
jira_description: "Index swtiches* has no data flowing in Prod ELK cluster. \n"

Jun 25 04:09:33 prod-elk-monitor python[3041]: INFO:elastalert:Ran Nodata flatline - switches from 2024-06-25 04:02 PDT to 2024-06-25 04:09 PDT: 0 query hits (0 already seen), 0 matches, 0 alerts sent
Jun 25 04:09:33 prod-elk-monitor python[3041]: INFO:elastalert:Nodata flatline - switches range 442

[jertel]

Enable debug logging and see what additional info the logs tell you.

[manishparmar5751]

Well that is the question, why is it been silenced?

[jertel]

It's impossible for anyone here to answer that given the tiny snippet of logs you provided. I suggest shutting down ElastAlert 2. Rename the rule to something new to get a fresh start. Restart ElastAlert 2 with debug logs enabled. Then watch the logs to find out when the rule is first alerted, when it's silenced, etc.

[manishparmar5751]

This are the relevant logs after running above steps. Same set of logs are repeated after subsequent runs. It longer says about the alerts being silenced but does not generate the alerts either.

Jun 27 14:25:29 prod-elk-monitor python[10204]: INFO:elastalert:Queried rule Test Rule logs from 2024-06-27 14:10 PDT to 2024-06-27 14:25 PDT: 0 / 0 hits
Jun 27 14:25:29 prod-elk-monitor python[10204]: INFO:elastalert:Skipping writing to ES: {'rule_name': 'Test Rule logs', 'endtime': '2024-06-27T21:25:28.980764Z', 'starttime': '2024-06-27T21:10:28.980764Z', 'matches': 0, 'hits': 0, '@timestamp': '2024-06-27T21:25:29.248980Z', 'time_taken': 0.2682020664215088}
Jun 27 14:25:29 prod-elk-monitor python[10204]: INFO:elastalert:Ran Test Rule logs from 2024-06-27 14:10 PDT to 2024-06-27 14:25 PDT: 0 query hits (0 already seen), 0 matches, 0 alerts sent
Jun 27 14:25:29 prod-elk-monitor python[10204]: INFO:elastalert:Test Rule logs range 900

[jertel]

I just started up a fresh instance of ElastAlert 2 and loaded this single rule:

name: flatline15
enabled: true
type: flatline
index: "this_index_does_not_exist"
filter:
- query:
    match_all: {}
threshold: 1
timeframe:
  seconds: 20
alert:
- debug
realert:
  minutes: 1

After about 25 or so seconds I saw the alert fire. So I know that both ElastAlert 2 version 2.18.0, and the rule I pasted above both work. I suggest using that as a template and verifying you are seeing the same behavior, then move on to customizing your rule.

[manishparmar5751]

Was running 2.15 version, recreated the environment, cloned latest version. Running OK!