Rule not triggering alerts but does with elastalert-test-rule --alert #1363

iliketurtlesyep · 2024-01-25T01:36:22Z

iliketurtlesyep
Jan 25, 2024

Hi there. I have the following rule that triggers when run against elastalert-test-rule --alert but doesn't otherwise:

name: root login
type: any
index: .ds-logs-system.auth-default-*
use_strftime_index: true
is_enabled: true

filter:
- query:
    query_string:
      query: "host.hostname: myhostname AND user.name: root AND event.action: ssh_login AND system.auth.ssh.event: Accepted"

realert:
  minutes: 0

alert:
  - "slack"

alert_subject: "root login at {} from {}"
alert_subject_args:
  - "@timestamp"
  - source.ip
alert_text: |
  Timestamp: {0}
  Source host: {1}
alert_text_args: ["@timestamp", "source.ip"]

slack_webhook_url: REDACTED

The logs show 0 hits and 0 seen:

INFO:elastalert:Queried rule root login from 2024-01-25 01:15 UTC to 2024-01-25 01:24 UTC: 0 / 0 hits
INFO:elastalert:Ran root login from 2024-01-25 01:15 UTC to 2024-01-25 01:24 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:root login range 592

But elastalert-test-rule shows the correct result of 13 and with '--alert' passed to it, it sends the alerts to my slack channel as you'd expect.

elastalert_status - {'rule_name': 'root login', 'endtime': datetime.datetime(2024, 1, 25, 1, 21, 5, 194603, tzinfo=tzutc()), 'starttime': datetime.datetime(2024, 1, 24, 1, 21, 5, 194603, tzinfo=tzutc()), 'matches': 13, 'hits': 13, '@timestamp': datetime.datetime(2024, 1, 25, 1, 21, 9, 295511, tzinfo=tzutc()), 'time_taken': 3.8375191688537598}

I'm just not sure why it's working properly with elastalert-test-rule but not otherwise. Does anyone have any ideas?

Thanks!

Answered by jertel

Jan 29, 2024

The queries look identical, and assuming you are redacting the identical Elastic hostname, using the same ElastAlert 2 config (Elast user/pass), and using the same ElastAlert 2 code (both modes are running from the same server, using the same ElastAlert 2 version) then it doesn't seem possible for one query by host.hostname to work, but the other not to work.

I think my next step would be to start narrowing down if I can get any hits via a hostname query. Ex: host.hostname: ip*, host.hostname: *, _exists_: host.hostname, etc.

View full answer

jertel · 2024-01-25T15:08:33Z

jertel
Jan 25, 2024
Maintainer

Hello! The two logs are showing different time ranges: The elastalert-test-rule shows a 24h time range, while the normal mode log shows a 9 minute time range. If those 13 records were located prior to the 9 minute time range then it would explain why they aren't showing up.

9 replies

jertel Jan 26, 2024
Maintainer

Add this to your config, where loggers: should be at the same indentation level as handlers:.

 loggers:
   elastalert:
     level: DEBUG
     handlers: []
     propagate: true

   elasticsearch:
     level: DEBUG
     handlers: []
     propagate: true

   elasticsearch.trace:
     level: DEBUG
     handlers: []
     propagate: true

   '':  # root logger
     level: DEBUG
     handlers:
       - console
     propagate: false

The rest of the info you just provided is very helpful. The times appear to match up, assuming that @timestamp field is being store/shown as UTC time. You could also try simplifying your query filter to have it look for "*" as the query: value, and see if you get hits.

jertel Jan 26, 2024
Maintainer

Also, the file: handler should be at the same level as console:.

iliketurtlesyep Jan 26, 2024
Author

Great, thank you very much! I ran it again in debug mode, and let it settle:

# docker ps
CONTAINER ID   IMAGE                COMMAND                  CREATED         STATUS         PORTS     NAMES
f71548371a80   jertel/elastalert2   "/opt/elastalert/run…"   6 minutes ago   Up 6 minutes             elastalert

Kicked off an SSH session to trigger the alert at 20:04:04 UTC (looks like there might be a couple second drift on the remote host):

root@mybox [/opt/elastalert2]
# ssh root@remotehost
$ date
Fri Jan 26 15:04:04 EST 2024

ES indexed it right away:

@timestamp Jan 26, 2024 @ 08:04:01.000

And the debug logs for that time period:

2024-01-26 20:06:00,918    DEBUG apscheduler.scheduler Looking for jobs to run
2024-01-26 20:06:00,918    DEBUG apscheduler.scheduler Next wakeup is due at 2024-01-26 20:06:32.072598+00:00 (in 31.154201 seconds)
2024-01-26 20:06:00,918     INFO apscheduler.executors.default Running job "Rule: root login (trigger: interval[0:01:00], next run at: 2024-01-26 20:07:03 UTC)" (scheduled at 2024-01-26 20:06:00.917618+00:00)
2024-01-26 20:06:00,920    DEBUG urllib3.connectionpool Resetting dropped connection: myeshost.mydomain.com
2024-01-26 20:06:00,975    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-26 20:06:00,975     INFO        elasticsearch POST https://myeshost.mydomain.com:9200/.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 [status:200 request:0.057s]
2024-01-26 20:06:00,975    DEBUG        elasticsearch > {"query":{"bool":{"filter":{"bool":{"must":[{"range":{"@timestamp":{"gt":"2024-01-26T19:51:00.918774Z","lte":"2024-01-26T20:06:00.918774Z"}}},{"query_string":{"query":"host.hostname: myhostname AND user.name: root AND event.action: ssh_login AND system.auth.ssh.event: Accepted"}}]}}}},"sort":[{"@timestamp":{"order":"asc"}}]}
2024-01-26 20:06:00,976    DEBUG        elasticsearch < {"_scroll_id":"FGluY2x1ZGVfY29udGV4dF91dWlkDnF1ZXJ5VGhlbkZldGNoAxZleFlhTDhyVlNGYU1VbDNzRU9UUjVBAAAAAABrESQWcVdnMEtVLWlUd2FWVm1VZHdtaHIwdxY3U1dOaTNzc1FweVlIQVdtZkt3U1lRAAAAAAAGLeQWc0hkYkQ0VnFTRWlDdndHMWNfV1VUZxZleFlhTDhyVlNGYU1VbDNzRU9UU
jVBAAAAAABrESUWcVdnMEtVLWlUd2FWVm1VZHdtaHIwdw==","took":7,"timed_out":false,"_shards":{"total":5,"successful":5,"skipped":2,"failed":0},"hits":{"total":{"value":0,"relation":"eq"},"max_score":null,"hits":[]}}
2024-01-26 20:06:00,976    DEBUG           elastalert {'_scroll_id': 'FGluY2x1ZGVfY29udGV4dF91dWlkDnF1ZXJ5VGhlbkZldGNoAxZleFlhTDhyVlNGYU1VbDNzRU9UUjVBAAAAAABrESQWcVdnMEtVLWlUd2FWVm1VZHdtaHIwdxY3U1dOaTNzc1FweVlIQVdtZkt3U1lRAAAAAAAGLeQWc0hkYkQ0VnFTRWlDdndHMWNfV1VUZxZleFlhTDhyVlNGYU1VbDNzRU9UUj
VBAAAAAABrESUWcVdnMEtVLWlUd2FWVm1VZHdtaHIwdw==', 'took': 7, 'timed_out': False, '_shards': {'total': 5, 'successful': 5, 'skipped': 2, 'failed': 0}, 'hits': {'total': {'value': 0, 'relation': 'eq'}, 'max_score': None, 'hits': []}}
2024-01-26 20:06:00,976     INFO           elastalert Queried rule root login from 2024-01-26 19:51 UTC to 2024-01-26 20:06 UTC: 0 / 0 hits
2024-01-26 20:06:00,980    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-26 20:06:00,981     INFO        elasticsearch DELETE https://myeshost.mydomain.com:9200/_search/scroll [status:200 request:0.004s]
2024-01-26 20:06:00,981    DEBUG        elasticsearch > {"scroll_id":["FGluY2x1ZGVfY29udGV4dF91dWlkDnF1ZXJ5VGhlbkZldGNoAxZleFlhTDhyVlNGYU1VbDNzRU9UUjVBAAAAAABrESQWcVdnMEtVLWlUd2FWVm1VZHdtaHIwdxY3U1dOaTNzc1FweVlIQVdtZkt3U1lRAAAAAAAGLeQWc0hkYkQ0VnFTRWlDdndHMWNfV1VUZxZleFlhTDhyVlNGYU1VbDNzRU9UU
jVBAAAAAABrESUWcVdnMEtVLWlUd2FWVm1VZHdtaHIwdw=="]}
2024-01-26 20:06:00,981    DEBUG        elasticsearch < {"succeeded":true,"num_freed":3}
2024-01-26 20:06:00,981     INFO           elastalert Skipping writing to ES: {'rule_name': 'root login', 'endtime': '2024-01-26T20:06:00.918774Z', 'starttime': '2024-01-26T19:51:00.918774Z', 'matches': 0, 'hits': 0, '@timestamp': '2024-01-26T20:06:00.981330Z', 'time_taken': 0.06253576278686523}
2024-01-26 20:06:00,981     INFO           elastalert Ran root login from 2024-01-26 19:51 UTC to 2024-01-26 20:06 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent 
2024-01-26 20:06:00,981     INFO           elastalert root login range 900
2024-01-26 20:06:00,981     INFO apscheduler.executors.default Job "Rule: root login (trigger: interval[0:01:00], next run at: 2024-01-26 20:07:03 UTC)" executed successfully

There should be four results which is exactly what elastalert-test-rule finds:

elastalert_status - {'rule_name': 'root login', 'endtime': datetime.datetime(2024, 1, 26, 20, 31, 10, 130700, tzinfo=tzutc()), 'starttime': datetime.datetime(2024, 1, 25, 20, 31, 10, 130700, tzinfo=tzutc()), 'matches': 4, 'hits': 4, '@timestamp': datetime.datetime(2024, 1, 26, 20, 31, 13, 119946, tzinfo=tzutc()), 'time_taken': 2.667508363723755}

Thanks again for the help!

iliketurtlesyep Jan 26, 2024
Author

Interesting -- I was playing with the query as you suggested and changing it to "*" matched so may, it brought the box down :)

Looks like the issue is around the hostname. By stripping just the host.hostname field and leaving the rest ("user.name: root AND event.action: ssh_login AND system.auth.ssh.event: Accepted"), it returned 482 results:

INFO:elastalert:Queried rule root login from 2024-01-26 21:00 UTC to 2024-01-26 21:06 UTC: 482 / 482 hits

The host is in AWS so actually has a hostname in the ip-10-1-1-1 naming convention, and that seems to be the issue.

Having the query be just the host.hostname field, it doesn't return anything:

query: "host.hostname: ip-10-0-1-156"

INFO:elastalert:Ran root login from 2024-01-26 21:00 UTC to 2024-01-26 21:13 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent

But as you can see, it does return 164 results when queried in Kibana:

Interesting that elastalert-test-rule seems to be fine with the hostname. Can you think of anything else we can try at this point?

Thanks again for all the help -- I really appreciate it!

jertel Jan 27, 2024
Maintainer

That is interesting. Did you get debug logs from the elastalert-test-rule run? I'd like to compare the Elastic query and response of that one to the non-test one you pasted above. Something must be different.

iliketurtlesyep · 2024-01-29T17:26:03Z

iliketurtlesyep
Jan 29, 2024
Author

Hey there and thank you so much for your continued assistance.

Here are the elastalert-test-rule debug logs. I kicked off a couple SSH sessions to trigger the rule:

root@mybox [/opt/elastalert2]
# ssh root@remotehost date
Mon Jan 29 11:58:27 EST 2024

elastalert@d0e04800e812:~$ elastalert-test-rule rules/root_login.yaml 
2024-01-29 16:59:34,915     INFO           elastalert Note: In debug mode, alerts will be logged to console but NOT actually sent.
            To send them but remain verbose, use --verbose instead.
2024-01-29 16:59:34,915    DEBUG /usr/local/lib/python3.12/site-packages/envparse.py Get 'ES_USE_SSL' casted as 'None'/'None' with default 'None'
2024-01-29 16:59:34,915    DEBUG /usr/local/lib/python3.12/site-packages/envparse.py Get 'ES_BEARER' casted as 'None'/'None' with default 'None'
2024-01-29 16:59:34,916    DEBUG /usr/local/lib/python3.12/site-packages/envparse.py Get 'ES_PASSWORD' casted as 'None'/'None' with default 'None'
2024-01-29 16:59:34,916    DEBUG /usr/local/lib/python3.12/site-packages/envparse.py Get 'ES_USERNAME' casted as 'None'/'None' with default 'None'
2024-01-29 16:59:34,916    DEBUG /usr/local/lib/python3.12/site-packages/envparse.py Get 'ES_API_KEY' casted as 'None'/'None' with default 'None'
2024-01-29 16:59:34,916    DEBUG /usr/local/lib/python3.12/site-packages/envparse.py Get 'ES_HOST' casted as 'None'/'None' with default 'None'
2024-01-29 16:59:34,916    DEBUG /usr/local/lib/python3.12/site-packages/envparse.py Get 'ES_HOSTS' casted as 'None'/'None' with default 'None'
2024-01-29 16:59:34,916    DEBUG /usr/local/lib/python3.12/site-packages/envparse.py Get 'ES_PORT' casted as 'None'/'None' with default 'None'
2024-01-29 16:59:34,916    DEBUG /usr/local/lib/python3.12/site-packages/envparse.py Get 'ES_URL_PREFIX' casted as 'None'/'None' with default 'None'
2024-01-29 16:59:34,916    DEBUG /usr/local/lib/python3.12/site-packages/envparse.py Get 'STATSD_INSTANCE_TAG' casted as 'None'/'None' with default 'None'
2024-01-29 16:59:34,916    DEBUG /usr/local/lib/python3.12/site-packages/envparse.py Get 'STATSD_HOST' casted as 'None'/'None' with default 'None'
2024-01-29 16:59:35,065    DEBUG urllib3.connectionpool Starting new HTTPS connection (1): myeshost.mydomain.com:9200
2024-01-29 16:59:35,146    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?ignore_unavailable=true&size=1 HTTP/1.1" 200 None
2024-01-29 16:59:35,147     INFO        elasticsearch POST https://myeshost.mydomain.com:9200/.ds-logs-system.auth-default-*/_search?ignore_unavailable=true&size=1 [status:200 request:0.084s]
2024-01-29 16:59:35,147    DEBUG        elasticsearch > {"query":{"bool":{"filter":{"bool":{"must":[{"range":{"@timestamp":{"gt":"2024-01-28T16:59:34.907319Z","lte":"2024-01-29T16:59:34.907319Z"}}},{"query_string":{"query":"host.hostname: ip-10-0-1-156 AND user.name: root AND event.action: ssh_login AND system.auth.ssh.event: Accepted"}}]}}}},"sort":[{"@timestamp":{"order":"asc"}}]}
2024-01-29 16:59:35,148    DEBUG        elasticsearch < {"took":31,"timed_out":false,"_shards":{"total":5,"successful":5,"skipped":1,"failed":0},"hits":{"total":{"value":2,"relation":"eq"},"max_score":null,"hits":[{"_index":".ds-logs-system.auth-default-2024.01.29-000075","_id":"RW0UVo0BW5fCFjgd0zK4","_score":null,"_source":{"agent":{"name":"mybox","id":"e6067e8d-1ca4-43cb-a178-d17f2ec37a40","ephemeral_id":"4be880fc-f4a6-4fb8-a386-a0510966a233","type":"filebeat","version":"8.10.3"},"process":{"name":"sshd","pid":77967},"log":{"file":{"path":"/var/log/remote_auth.log"},"offset":729673249},"elastic_agent":{"id":"e6067e8d-1ca4-43cb-a178-d17f2ec37a40","version":"8.10.3","snapshot":false},"source":{"address":"10.1.1.1","port":44810,"ip":"10.1.1.1"},"tags":["system-auth"],"cloud":{"availability_zone":"us-east-1a","instance":{"name":"mybox","id":"myinstance"},"provider":"openstack","machine":{"type":"m7a.large"},"service":{"name":"Nova"}},"input":{"type":"log"},"@timestamp":"2024-01-29T11:36:52.000Z","system":{"auth":{"ssh":{"method":"publickey","event":"Accepted"}}},"ecs":{"version":"8.0.0"},"related":{"hosts":["ip-10-0-1-156"],"ip":["10.1.1.1"],"user":["root"]},"data_stream":{"namespace":"default","type":"logs","dataset":"system.auth"},"host":{"hostname":"ip-10-0-1-156","os":{"kernel":"6.2.0-1017-aws","codename":"jammy","name":"Ubuntu","type":"linux","family":"debian","version":"22.04.3 LTS (Jammy Jellyfish)","platform":"ubuntu"},"containerized":false,"ip":["10.1.1.1","fe80::105b:d3ff:fefd:2a65"],"name":"mybox","id":"ec2f7984f126942d831b2cfb0726608d","mac":["12-5B-D3-FD-2A-65"],"architecture":"x86_64"},"event":{"agent_id_status":"verified","ingested":"2024-01-29T16:37:02Z","timezone":"+00:00","kind":"event","action":"ssh_login","type":["info"],"category":["authentication","session"],"dataset":"system.auth","outcome":"success"},"user":{"name":"root"}},"sort":[1706528212000]}]}}
2024-01-29 16:59:35,161    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_count?ignore_unavailable=true HTTP/1.1" 200 71
2024-01-29 16:59:35,161     INFO        elasticsearch POST https://myeshost.mydomain.com:9200/.ds-logs-system.auth-default-*/_count?ignore_unavailable=true [status:200 request:0.013s]
2024-01-29 16:59:35,161    DEBUG        elasticsearch > {"query":{"bool":{"filter":{"bool":{"must":[{"range":{"@timestamp":{"gt":"2024-01-28T16:59:34.907319Z","lte":"2024-01-29T16:59:34.907319Z"}}},{"query_string":{"query":"host.hostname: ip-10-0-1-156 AND user.name: root AND event.action: ssh_login AND system.auth.ssh.event: Accepted"}}]}}}}}
2024-01-29 16:59:35,161    DEBUG        elasticsearch < {"count":2,"_shards":{"total":5,"successful":5,"skipped":0,"failed":0}}
Got 2 hits from the last 0 day

Available terms in first hit:
	agent.name
	agent.id
	agent.ephemeral_id
	agent.type
	agent.version
	process.name
	process.pid
	log.file.path
	log.offset
	elastic_agent.id
	elastic_agent.version
	elastic_agent.snapshot
	source.address
	source.port
	source.ip
	tags
	cloud.availability_zone
	cloud.instance.name
	cloud.instance.id
	cloud.provider
	cloud.machine.type
	cloud.service.name
	input.type
	@timestamp
	system.auth.ssh.method
	system.auth.ssh.event
	ecs.version
	related.hosts
	related.ip
	related.user
	data_stream.namespace
	data_stream.type
	data_stream.dataset
	host.hostname
	host.os.kernel
	host.os.codename
	host.os.name
	host.os.type
	host.os.family
	host.os.version
	host.os.platform
	host.containerized
	host.ip
	host.name
	host.id
	host.mac
	host.architecture
	event.agent_id_status
	event.ingested
	event.timezone
	event.kind
	event.action
	event.type
	event.category
	event.dataset
	event.outcome
	user.name

2024-01-29 16:59:35,167     INFO           elastalert Note: In debug mode, alerts will be logged to console but NOT actually sent.
                To send them but remain verbose, use --verbose instead.
2024-01-29 16:59:35,167     INFO           elastalert 1 rules loaded
2024-01-29 16:59:35,174     INFO apscheduler.scheduler Adding job tentatively -- it will be properly scheduled when the scheduler starts
2024-01-29 16:59:35,177    DEBUG urllib3.connectionpool Starting new HTTPS connection (1): myeshost.mydomain.com:9200
2024-01-29 16:59:35,291    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:35,292     INFO           elastalert Queried rule root login from 2024-01-28 16:59 UTC to 2024-01-28 17:14 UTC: 0 / 0 hits
2024-01-29 16:59:35,297    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:35,308    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:35,309     INFO           elastalert Queried rule root login from 2024-01-28 17:14 UTC to 2024-01-28 17:29 UTC: 0 / 0 hits
2024-01-29 16:59:35,313    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:35,343    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:35,344     INFO           elastalert Queried rule root login from 2024-01-28 17:29 UTC to 2024-01-28 17:44 UTC: 0 / 0 hits
2024-01-29 16:59:35,349    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:35,362    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:35,363     INFO           elastalert Queried rule root login from 2024-01-28 17:44 UTC to 2024-01-28 17:59 UTC: 0 / 0 hits
2024-01-29 16:59:35,367    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:35,379    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:35,380     INFO           elastalert Queried rule root login from 2024-01-28 17:59 UTC to 2024-01-28 18:14 UTC: 0 / 0 hits
2024-01-29 16:59:35,384    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:35,396    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:35,396     INFO           elastalert Queried rule root login from 2024-01-28 18:14 UTC to 2024-01-28 18:29 UTC: 0 / 0 hits
2024-01-29 16:59:35,400    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:35,419    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:35,420     INFO           elastalert Queried rule root login from 2024-01-28 18:29 UTC to 2024-01-28 18:44 UTC: 0 / 0 hits
2024-01-29 16:59:35,424    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:35,437    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:35,437     INFO           elastalert Queried rule root login from 2024-01-28 18:44 UTC to 2024-01-28 18:59 UTC: 0 / 0 hits
2024-01-29 16:59:35,453    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:35,474    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:35,475     INFO           elastalert Queried rule root login from 2024-01-28 18:59 UTC to 2024-01-28 19:14 UTC: 0 / 0 hits
2024-01-29 16:59:35,480    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:35,501    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:35,502     INFO           elastalert Queried rule root login from 2024-01-28 19:14 UTC to 2024-01-28 19:29 UTC: 0 / 0 hits
2024-01-29 16:59:35,513    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:35,524    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:35,525     INFO           elastalert Queried rule root login from 2024-01-28 19:29 UTC to 2024-01-28 19:44 UTC: 0 / 0 hits
2024-01-29 16:59:35,529    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:35,543    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:35,543     INFO           elastalert Queried rule root login from 2024-01-28 19:44 UTC to 2024-01-28 19:59 UTC: 0 / 0 hits
2024-01-29 16:59:35,548    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:35,559    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:35,560     INFO           elastalert Queried rule root login from 2024-01-28 19:59 UTC to 2024-01-28 20:14 UTC: 0 / 0 hits
2024-01-29 16:59:35,567    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:35,580    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:35,581     INFO           elastalert Queried rule root login from 2024-01-28 20:14 UTC to 2024-01-28 20:29 UTC: 0 / 0 hits
2024-01-29 16:59:35,587    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:35,599    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:35,600     INFO           elastalert Queried rule root login from 2024-01-28 20:29 UTC to 2024-01-28 20:44 UTC: 0 / 0 hits
2024-01-29 16:59:35,605    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:35,618    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:35,619     INFO           elastalert Queried rule root login from 2024-01-28 20:44 UTC to 2024-01-28 20:59 UTC: 0 / 0 hits
2024-01-29 16:59:35,624    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:35,635    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:35,636     INFO           elastalert Queried rule root login from 2024-01-28 20:59 UTC to 2024-01-28 21:14 UTC: 0 / 0 hits
2024-01-29 16:59:35,640    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:35,655    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:35,658     INFO           elastalert Queried rule root login from 2024-01-28 21:14 UTC to 2024-01-28 21:29 UTC: 0 / 0 hits
2024-01-29 16:59:35,663    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:35,676    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:35,677     INFO           elastalert Queried rule root login from 2024-01-28 21:29 UTC to 2024-01-28 21:44 UTC: 0 / 0 hits
2024-01-29 16:59:35,683    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:35,701    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:35,702     INFO           elastalert Queried rule root login from 2024-01-28 21:44 UTC to 2024-01-28 21:59 UTC: 0 / 0 hits
2024-01-29 16:59:35,705    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:35,719    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:35,720     INFO           elastalert Queried rule root login from 2024-01-28 21:59 UTC to 2024-01-28 22:14 UTC: 0 / 0 hits
2024-01-29 16:59:35,724    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:35,735    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:35,735     INFO           elastalert Queried rule root login from 2024-01-28 22:14 UTC to 2024-01-28 22:29 UTC: 0 / 0 hits
2024-01-29 16:59:35,739    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:35,753    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:35,754     INFO           elastalert Queried rule root login from 2024-01-28 22:29 UTC to 2024-01-28 22:44 UTC: 0 / 0 hits
2024-01-29 16:59:35,758    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:35,772    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:35,772     INFO           elastalert Queried rule root login from 2024-01-28 22:44 UTC to 2024-01-28 22:59 UTC: 0 / 0 hits
2024-01-29 16:59:35,777    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:35,790    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:35,791     INFO           elastalert Queried rule root login from 2024-01-28 22:59 UTC to 2024-01-28 23:14 UTC: 0 / 0 hits
2024-01-29 16:59:35,796    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:35,806    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:35,807     INFO           elastalert Queried rule root login from 2024-01-28 23:14 UTC to 2024-01-28 23:29 UTC: 0 / 0 hits
2024-01-29 16:59:35,812    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:35,836    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:35,838     INFO           elastalert Queried rule root login from 2024-01-28 23:29 UTC to 2024-01-28 23:44 UTC: 0 / 0 hits
2024-01-29 16:59:35,842    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:35,904    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:35,905     INFO           elastalert Queried rule root login from 2024-01-28 23:44 UTC to 2024-01-28 23:59 UTC: 0 / 0 hits
2024-01-29 16:59:35,908    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:35,975    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:35,976     INFO           elastalert Queried rule root login from 2024-01-28 23:59 UTC to 2024-01-29 00:14 UTC: 0 / 0 hits
2024-01-29 16:59:35,982    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:35,993    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:35,994     INFO           elastalert Queried rule root login from 2024-01-29 00:14 UTC to 2024-01-29 00:29 UTC: 0 / 0 hits
2024-01-29 16:59:35,999    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:36,018    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:36,020     INFO           elastalert Queried rule root login from 2024-01-29 00:29 UTC to 2024-01-29 00:44 UTC: 0 / 0 hits
2024-01-29 16:59:36,026    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:36,037    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:36,038     INFO           elastalert Queried rule root login from 2024-01-29 00:44 UTC to 2024-01-29 00:59 UTC: 0 / 0 hits
2024-01-29 16:59:36,045    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:36,062    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:36,063     INFO           elastalert Queried rule root login from 2024-01-29 00:59 UTC to 2024-01-29 01:14 UTC: 0 / 0 hits
2024-01-29 16:59:36,069    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:36,080    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:36,082     INFO           elastalert Queried rule root login from 2024-01-29 01:14 UTC to 2024-01-29 01:29 UTC: 0 / 0 hits
2024-01-29 16:59:36,089    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:36,104    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:36,104     INFO           elastalert Queried rule root login from 2024-01-29 01:29 UTC to 2024-01-29 01:44 UTC: 0 / 0 hits
2024-01-29 16:59:36,108    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:36,123    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:36,124     INFO           elastalert Queried rule root login from 2024-01-29 01:44 UTC to 2024-01-29 01:59 UTC: 0 / 0 hits
2024-01-29 16:59:36,129    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:36,142    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:36,143     INFO           elastalert Queried rule root login from 2024-01-29 01:59 UTC to 2024-01-29 02:14 UTC: 0 / 0 hits
2024-01-29 16:59:36,146    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:36,160    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:36,161     INFO           elastalert Queried rule root login from 2024-01-29 02:14 UTC to 2024-01-29 02:29 UTC: 0 / 0 hits
2024-01-29 16:59:36,165    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:36,181    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:36,182     INFO           elastalert Queried rule root login from 2024-01-29 02:29 UTC to 2024-01-29 02:44 UTC: 0 / 0 hits
2024-01-29 16:59:36,188    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:36,199    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:36,203     INFO           elastalert Queried rule root login from 2024-01-29 02:44 UTC to 2024-01-29 02:59 UTC: 0 / 0 hits
2024-01-29 16:59:36,206    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:36,229    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:36,230     INFO           elastalert Queried rule root login from 2024-01-29 02:59 UTC to 2024-01-29 03:14 UTC: 0 / 0 hits
2024-01-29 16:59:36,234    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:36,244    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:36,245     INFO           elastalert Queried rule root login from 2024-01-29 03:14 UTC to 2024-01-29 03:29 UTC: 0 / 0 hits
2024-01-29 16:59:36,250    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:36,259    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:36,259     INFO           elastalert Queried rule root login from 2024-01-29 03:29 UTC to 2024-01-29 03:44 UTC: 0 / 0 hits
2024-01-29 16:59:36,263    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:36,282    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:36,283     INFO           elastalert Queried rule root login from 2024-01-29 03:44 UTC to 2024-01-29 03:59 UTC: 0 / 0 hits
2024-01-29 16:59:36,288    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:36,297    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:36,298     INFO           elastalert Queried rule root login from 2024-01-29 03:59 UTC to 2024-01-29 04:14 UTC: 0 / 0 hits
2024-01-29 16:59:36,303    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:36,313    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:36,315     INFO           elastalert Queried rule root login from 2024-01-29 04:14 UTC to 2024-01-29 04:29 UTC: 0 / 0 hits
2024-01-29 16:59:36,318    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:36,330    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:36,330     INFO           elastalert Queried rule root login from 2024-01-29 04:29 UTC to 2024-01-29 04:44 UTC: 0 / 0 hits
2024-01-29 16:59:36,334    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:36,358    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:36,359     INFO           elastalert Queried rule root login from 2024-01-29 04:44 UTC to 2024-01-29 04:59 UTC: 0 / 0 hits
2024-01-29 16:59:36,364    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:36,374    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:36,375     INFO           elastalert Queried rule root login from 2024-01-29 04:59 UTC to 2024-01-29 05:14 UTC: 0 / 0 hits
2024-01-29 16:59:36,379    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:36,392    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:36,392     INFO           elastalert Queried rule root login from 2024-01-29 05:14 UTC to 2024-01-29 05:29 UTC: 0 / 0 hits
2024-01-29 16:59:36,397    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:36,407    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:36,408     INFO           elastalert Queried rule root login from 2024-01-29 05:29 UTC to 2024-01-29 05:44 UTC: 0 / 0 hits
2024-01-29 16:59:36,412    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:36,429    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:36,429     INFO           elastalert Queried rule root login from 2024-01-29 05:44 UTC to 2024-01-29 05:59 UTC: 0 / 0 hits
2024-01-29 16:59:36,451    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:36,468    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:36,468     INFO           elastalert Queried rule root login from 2024-01-29 05:59 UTC to 2024-01-29 06:14 UTC: 0 / 0 hits
2024-01-29 16:59:36,473    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:36,497    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:36,499     INFO           elastalert Queried rule root login from 2024-01-29 06:14 UTC to 2024-01-29 06:29 UTC: 0 / 0 hits
2024-01-29 16:59:36,503    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:36,514    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:36,514     INFO           elastalert Queried rule root login from 2024-01-29 06:29 UTC to 2024-01-29 06:44 UTC: 0 / 0 hits
2024-01-29 16:59:36,522    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:36,537    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:36,537     INFO           elastalert Queried rule root login from 2024-01-29 06:44 UTC to 2024-01-29 06:59 UTC: 0 / 0 hits
2024-01-29 16:59:36,542    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:36,557    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:36,558     INFO           elastalert Queried rule root login from 2024-01-29 06:59 UTC to 2024-01-29 07:14 UTC: 0 / 0 hits
2024-01-29 16:59:36,565    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:36,576    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:36,576     INFO           elastalert Queried rule root login from 2024-01-29 07:14 UTC to 2024-01-29 07:29 UTC: 0 / 0 hits
2024-01-29 16:59:36,580    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:36,591    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:36,592     INFO           elastalert Queried rule root login from 2024-01-29 07:29 UTC to 2024-01-29 07:44 UTC: 0 / 0 hits
2024-01-29 16:59:36,596    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:36,613    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:36,614     INFO           elastalert Queried rule root login from 2024-01-29 07:44 UTC to 2024-01-29 07:59 UTC: 0 / 0 hits
2024-01-29 16:59:36,621    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:36,632    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:36,633     INFO           elastalert Queried rule root login from 2024-01-29 07:59 UTC to 2024-01-29 08:14 UTC: 0 / 0 hits
2024-01-29 16:59:36,637    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:36,656    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:36,657     INFO           elastalert Queried rule root login from 2024-01-29 08:14 UTC to 2024-01-29 08:29 UTC: 0 / 0 hits
2024-01-29 16:59:36,661    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:36,672    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:36,673     INFO           elastalert Queried rule root login from 2024-01-29 08:29 UTC to 2024-01-29 08:44 UTC: 0 / 0 hits
2024-01-29 16:59:36,678    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:36,690    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:36,691     INFO           elastalert Queried rule root login from 2024-01-29 08:44 UTC to 2024-01-29 08:59 UTC: 0 / 0 hits
2024-01-29 16:59:36,695    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:36,707    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:36,708     INFO           elastalert Queried rule root login from 2024-01-29 08:59 UTC to 2024-01-29 09:14 UTC: 0 / 0 hits
2024-01-29 16:59:36,713    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:36,739    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:36,740     INFO           elastalert Queried rule root login from 2024-01-29 09:14 UTC to 2024-01-29 09:29 UTC: 0 / 0 hits
2024-01-29 16:59:36,747    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:36,758    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:36,759     INFO           elastalert Queried rule root login from 2024-01-29 09:29 UTC to 2024-01-29 09:44 UTC: 0 / 0 hits
2024-01-29 16:59:36,763    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:36,781    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:36,788     INFO           elastalert Queried rule root login from 2024-01-29 09:44 UTC to 2024-01-29 09:59 UTC: 0 / 0 hits
2024-01-29 16:59:36,793    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:36,807    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:36,808     INFO           elastalert Queried rule root login from 2024-01-29 09:59 UTC to 2024-01-29 10:14 UTC: 0 / 0 hits
2024-01-29 16:59:36,812    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:36,825    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:36,826     INFO           elastalert Queried rule root login from 2024-01-29 10:14 UTC to 2024-01-29 10:29 UTC: 0 / 0 hits
2024-01-29 16:59:36,829    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:36,844    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:36,851     INFO           elastalert Queried rule root login from 2024-01-29 10:29 UTC to 2024-01-29 10:44 UTC: 0 / 0 hits
2024-01-29 16:59:36,862    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:36,884    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:36,884     INFO           elastalert Queried rule root login from 2024-01-29 10:44 UTC to 2024-01-29 10:59 UTC: 0 / 0 hits
2024-01-29 16:59:36,903    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:36,913    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:36,914     INFO           elastalert Queried rule root login from 2024-01-29 10:59 UTC to 2024-01-29 11:14 UTC: 0 / 0 hits
2024-01-29 16:59:36,919    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:36,939    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:36,940     INFO           elastalert Queried rule root login from 2024-01-29 11:14 UTC to 2024-01-29 11:29 UTC: 0 / 0 hits
2024-01-29 16:59:36,949    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:36,975    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:36,976     INFO           elastalert Queried rule root login from 2024-01-29 11:29 UTC to 2024-01-29 11:44 UTC: 1 / 1 hits
2024-01-29 16:59:36,993    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:37,011    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:37,011     INFO           elastalert Queried rule root login from 2024-01-29 11:44 UTC to 2024-01-29 11:59 UTC: 1 / 1 hits
2024-01-29 16:59:37,017    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:37,034    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:37,035     INFO           elastalert Queried rule root login from 2024-01-29 11:59 UTC to 2024-01-29 12:14 UTC: 0 / 0 hits
2024-01-29 16:59:37,039    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:37,061    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:37,061     INFO           elastalert Queried rule root login from 2024-01-29 12:14 UTC to 2024-01-29 12:29 UTC: 0 / 0 hits
2024-01-29 16:59:37,066    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:37,077    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:37,078     INFO           elastalert Queried rule root login from 2024-01-29 12:29 UTC to 2024-01-29 12:44 UTC: 0 / 0 hits
2024-01-29 16:59:37,083    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:37,097    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:37,098     INFO           elastalert Queried rule root login from 2024-01-29 12:44 UTC to 2024-01-29 12:59 UTC: 0 / 0 hits
2024-01-29 16:59:37,102    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:37,116    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:37,117     INFO           elastalert Queried rule root login from 2024-01-29 12:59 UTC to 2024-01-29 13:14 UTC: 0 / 0 hits
2024-01-29 16:59:37,134    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:37,145    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:37,146     INFO           elastalert Queried rule root login from 2024-01-29 13:14 UTC to 2024-01-29 13:29 UTC: 0 / 0 hits
2024-01-29 16:59:37,152    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:37,165    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:37,165     INFO           elastalert Queried rule root login from 2024-01-29 13:29 UTC to 2024-01-29 13:44 UTC: 0 / 0 hits
2024-01-29 16:59:37,169    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:37,223    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:37,224     INFO           elastalert Queried rule root login from 2024-01-29 13:44 UTC to 2024-01-29 13:59 UTC: 0 / 0 hits
2024-01-29 16:59:37,228    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:37,240    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:37,241     INFO           elastalert Queried rule root login from 2024-01-29 13:59 UTC to 2024-01-29 14:14 UTC: 0 / 0 hits
2024-01-29 16:59:37,245    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:37,259    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:37,260     INFO           elastalert Queried rule root login from 2024-01-29 14:14 UTC to 2024-01-29 14:29 UTC: 0 / 0 hits
2024-01-29 16:59:37,266    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:37,302    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:37,303     INFO           elastalert Queried rule root login from 2024-01-29 14:29 UTC to 2024-01-29 14:44 UTC: 0 / 0 hits
2024-01-29 16:59:37,308    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:37,396    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:37,398     INFO           elastalert Queried rule root login from 2024-01-29 14:44 UTC to 2024-01-29 14:59 UTC: 0 / 0 hits
2024-01-29 16:59:37,404    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:37,417    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:37,418     INFO           elastalert Queried rule root login from 2024-01-29 14:59 UTC to 2024-01-29 15:14 UTC: 0 / 0 hits
2024-01-29 16:59:37,423    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:37,445    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:37,446     INFO           elastalert Queried rule root login from 2024-01-29 15:14 UTC to 2024-01-29 15:29 UTC: 0 / 0 hits
2024-01-29 16:59:37,451    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:37,461    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:37,462     INFO           elastalert Queried rule root login from 2024-01-29 15:29 UTC to 2024-01-29 15:44 UTC: 0 / 0 hits
2024-01-29 16:59:37,467    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:37,478    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:37,480     INFO           elastalert Queried rule root login from 2024-01-29 15:44 UTC to 2024-01-29 15:59 UTC: 0 / 0 hits
2024-01-29 16:59:37,485    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:37,497    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:37,498     INFO           elastalert Queried rule root login from 2024-01-29 15:59 UTC to 2024-01-29 16:14 UTC: 0 / 0 hits
2024-01-29 16:59:37,503    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:37,526    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:37,527     INFO           elastalert Queried rule root login from 2024-01-29 16:14 UTC to 2024-01-29 16:29 UTC: 0 / 0 hits
2024-01-29 16:59:37,531    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:37,553    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:37,555     INFO           elastalert Queried rule root login from 2024-01-29 16:29 UTC to 2024-01-29 16:44 UTC: 0 / 0 hits
2024-01-29 16:59:37,560    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:37,575    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:37,576     INFO           elastalert Queried rule root login from 2024-01-29 16:44 UTC to 2024-01-29 16:59 UTC: 0 / 0 hits
2024-01-29 16:59:37,580    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32
2024-01-29 16:59:37,581     INFO           elastalert Alert for root login at 2024-01-29T11:36:52Z:
2024-01-29 16:59:37,582     INFO           elastalert Timestamp (ET): 2024-01-29T11:36:52Z
Source host: 10.1.1.1

@timestamp: 2024-01-29T11:36:52Z
_id: RW0UVo0BW5fCFjgd0zK4
_index: .ds-logs-system.auth-default-2024.01.29-000075
agent: {
    "ephemeral_id": "4be880fc-f4a6-4fb8-a386-a0510966a233",
    "id": "e6067e8d-1ca4-43cb-a178-d17f2ec37a40",
    "name": "mybox",
    "type": "filebeat",
    "version": "8.10.3"
}
cloud: {
    "availability_zone": "us-east-1a",
    "instance": {
        "id": "myinstance",
        "name": "mybox"
    },
    "machine": {
        "type": "m7a.large"
    },
    "provider": "openstack",
    "service": {
        "name": "Nova"
    }
}
data_stream: {
    "dataset": "system.auth",
    "namespace": "default",
    "type": "logs"
}
ecs: {
    "version": "8.0.0"
}
elastic_agent: {
    "id": "e6067e8d-1ca4-43cb-a178-d17f2ec37a40",
    "snapshot": false,
    "version": "8.10.3"
}
event: {
    "action": "ssh_login",
    "agent_id_status": "verified",
    "category": [
        "authentication",
        "session"
    ],
    "dataset": "system.auth",
    "ingested": "2024-01-29T16:37:02Z",
    "kind": "event",
    "outcome": "success",
    "timezone": "+00:00",
    "type": [
        "info"
    ]
}
host: {
    "architecture": "x86_64",
    "containerized": false,
    "hostname": "ip-10-0-1-156",
    "id": "ec2f7984f126942d831b2cfb0726608d",
    "ip": [
        "10.1.1.1",
        "fe80::105b:d3ff:fefd:2a65"
    ],
    "mac": [
        "12-5B-D3-FD-2A-65"
    ],
    "name": "mybox",
    "os": {
        "codename": "jammy",
        "family": "debian",
        "kernel": "6.2.0-1017-aws",
        "name": "Ubuntu",
        "platform": "ubuntu",
        "type": "linux",
        "version": "22.04.3 LTS (Jammy Jellyfish)"
    }
}
input: {
    "type": "log"
}
log: {
    "file": {
        "path": "/var/log/remote_auth.log"
    },
    "offset": 729673249
}
num_hits: 2
num_matches: 2
process: {
    "name": "sshd",
    "pid": 77967
}
related: {
    "hosts": [
        "ip-10-0-1-156"
    ],
    "ip": [
        "10.1.1.1"
    ],
    "user": [
        "root"
    ]
}
source: {
    "address": "10.1.1.1",
    "ip": "10.1.1.1",
    "port": 44810
}
system: {
    "auth": {
        "ssh": {
            "event": "Accepted",
            "method": "publickey"
        }
    }
}
tags: [
    "system-auth"
]
user: {
    "name": "root"
}

2024-01-29 16:59:37,583     INFO           elastalert Alert for root login at 2024-01-29T11:58:27Z:
2024-01-29 16:59:37,585     INFO           elastalert Timestamp (ET): 2024-01-29T11:58:27Z
Source host: 10.1.1.1

@timestamp: 2024-01-29T11:58:27Z
_id: hIEoVo0BW5fCFjgdhj10
_index: .ds-logs-system.auth-default-2024.01.29-000075
agent: {
    "ephemeral_id": "4be880fc-f4a6-4fb8-a386-a0510966a233",
    "id": "e6067e8d-1ca4-43cb-a178-d17f2ec37a40",
    "name": "mybox",
    "type": "filebeat",
    "version": "8.10.3"
}
cloud: {
    "availability_zone": "us-east-1a",
    "instance": {
        "id": "myinstance",
        "name": "mybox"
    },
    "machine": {
        "type": "m7a.large"
    },
    "provider": "openstack",
    "service": {
        "name": "Nova"
    }
}
data_stream: {
    "dataset": "system.auth",
    "namespace": "default",
    "type": "logs"
}
ecs: {
    "version": "8.0.0"
}
elastic_agent: {
    "id": "e6067e8d-1ca4-43cb-a178-d17f2ec37a40",
    "snapshot": false,
    "version": "8.10.3"
}
event: {
    "action": "ssh_login",
    "agent_id_status": "verified",
    "category": [
        "authentication",
        "session"
    ],
    "dataset": "system.auth",
    "ingested": "2024-01-29T16:58:32Z",
    "kind": "event",
    "outcome": "success",
    "timezone": "+00:00",
    "type": [
        "info"
    ]
}
host: {
    "architecture": "x86_64",
    "containerized": false,
    "hostname": "ip-10-0-1-156",
    "id": "ec2f7984f126942d831b2cfb0726608d",
    "ip": [
        "10.1.1.1",
        "fe80::105b:d3ff:fefd:2a65"
    ],
    "mac": [
        "12-5B-D3-FD-2A-65"
    ],
    "name": "mybox",
    "os": {
        "codename": "jammy",
        "family": "debian",
        "kernel": "6.2.0-1017-aws",
        "name": "Ubuntu",
        "platform": "ubuntu",
        "type": "linux",
        "version": "22.04.3 LTS (Jammy Jellyfish)"
    }
}
input: {
    "type": "log"
}
log: {
    "file": {
        "path": "/var/log/remote_auth.log"
    },
    "offset": 730151712
}
num_hits: 2
num_matches: 2
process: {
    "name": "sshd",
    "pid": 35016
}
related: {
    "hosts": [
        "ip-10-0-1-156"
    ],
    "ip": [
        "10.1.1.1"
    ],
    "user": [
        "root"
    ]
}
source: {
    "address": "10.1.1.1",
    "ip": "10.1.1.1",
    "port": 57492
}
system: {
    "auth": {
        "ssh": {
            "event": "Accepted",
            "method": "publickey"
        }
    }
}
tags: [
    "system-auth"
]
user: {
    "name": "root"
}


Would have written the following documents to writeback index (default is elastalert_status):

elastalert_status - {'rule_name': 'root login', 'endtime': datetime.datetime(2024, 1, 29, 16, 59, 34, 907319, tzinfo=tzutc()), 'starttime': datetime.datetime(2024, 1, 28, 16, 59, 34, 907319, tzinfo=tzutc()), 'matches': 2, 'hits': 2, '@timestamp': datetime.datetime(2024, 1, 29, 16, 59, 37, 585198, tzinfo=tzutc()), 'time_taken': 2.4090795516967773}

Looks like it does see them:

2024-01-29 16:59:36,975    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:36,976     INFO           elastalert Queried rule root login from 2024-01-29 11:29 UTC to 2024-01-29 11:44 UTC: 1 / 1 hits
2024-01-29 16:59:36,993    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32

2024-01-29 16:59:37,011    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:37,011     INFO           elastalert Queried rule root login from 2024-01-29 11:44 UTC to 2024-01-29 11:59 UTC: 1 / 1 hits
2024-01-29 16:59:37,017    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32

And here are the results in Kibana:

Thank you again!

3 replies

jertel Jan 29, 2024
Maintainer

The queries look identical, and assuming you are redacting the identical Elastic hostname, using the same ElastAlert 2 config (Elast user/pass), and using the same ElastAlert 2 code (both modes are running from the same server, using the same ElastAlert 2 version) then it doesn't seem possible for one query by host.hostname to work, but the other not to work.

I think my next step would be to start narrowing down if I can get any hits via a hostname query. Ex: host.hostname: ip*, host.hostname: *, _exists_: host.hostname, etc.

Answer selected by jertel

iliketurtlesyep Jan 30, 2024
Author

Right, everything with the config and the rules remain the same as we've been working through this.

I know we discussed it a little earlier, but do you think it could have anything to do with the timestamp mismatch?

That box's timezone is ET (UTC-0500) as where the syslog server, ES cluster, and elastalert node are all UTC.

Mon 29 Jan 16:58:27
root@mybox [/opt/elastalert2]
# ssh root@remotehost date
Mon Jan 29 11:58:27 EST 2024

Jan 29 11:36:52 ip-10-0-1-156 sshd[77967]: Accepted publickey for root from 10.0.210.248 port 44810 ssh2
Jan 29 11:58:27 ip-10-0-1-156 sshd[35016]: Accepted publickey for root from 10.0.210.248 port 57492 ssh2

2024-01-29 16:59:36,975    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:36,976     INFO           elastalert Queried rule root login from 2024-01-29 11:29 UTC to 2024-01-29 11:44 UTC: 1 / 1 hits
2024-01-29 16:59:36,993    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32

2024-01-29 16:59:37,011    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "POST /.ds-logs-system.auth-default-*/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 HTTP/1.1" 200 None
2024-01-29 16:59:37,011     INFO           elastalert Queried rule root login from 2024-01-29 11:44 UTC to 2024-01-29 11:59 UTC: 1 / 1 hits
2024-01-29 16:59:37,017    DEBUG urllib3.connectionpool https://myeshost.mydomain.com:9200 "DELETE /_search/scroll HTTP/1.1" 200 32

The elastalert-test-rule debug logs show them matching during the 2024-01-29 11:29 - 2024-01-29 11:59 window even though it says it's UTC, those times are actually ET as you can see in the raw logs.

The elastalert-test-rule window ran from 2024-01-28 16:59 UTC - 2024-01-29 16:59 UTC. What I'm wondering is if elastalert isn't matching because once they do get indexed by ES, they're already five hours in the past so it doesn't see them since it's looking for data coming in from a current time window and elastalert_status has already been marked as having no alerts for that time period that's now 5 hours in the past. I don't know, what do you think?

I'll keep testing queries for the hostname but the timezone offset it making more sense to me now that I think about it.

Thanks again!

jertel Jan 31, 2024
Maintainer

It's definitely possible that the timezone is causing problems. I haven't studied your timeline to be certain that it's causing your issue above. I suggest making a change to your ingest pipeline to store all timestamps in UTC (best practice) and then resume the ElastAlert 2 troubleshooting.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rule not triggering alerts but does with elastalert-test-rule --alert #1363

{{title}}

Replies: 2 comments 12 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Rule not triggering alerts but does with elastalert-test-rule --alert #1363

iliketurtlesyep Jan 25, 2024

Replies: 2 comments · 12 replies

jertel Jan 25, 2024 Maintainer

jertel Jan 26, 2024 Maintainer

jertel Jan 26, 2024 Maintainer

iliketurtlesyep Jan 26, 2024 Author

iliketurtlesyep Jan 26, 2024 Author

jertel Jan 27, 2024 Maintainer

iliketurtlesyep Jan 29, 2024 Author

jertel Jan 29, 2024 Maintainer

iliketurtlesyep Jan 30, 2024 Author

jertel Jan 31, 2024 Maintainer

iliketurtlesyep
Jan 25, 2024

Replies: 2 comments 12 replies

jertel
Jan 25, 2024
Maintainer

jertel Jan 26, 2024
Maintainer

jertel Jan 26, 2024
Maintainer

iliketurtlesyep Jan 26, 2024
Author

iliketurtlesyep Jan 26, 2024
Author

jertel Jan 27, 2024
Maintainer

iliketurtlesyep
Jan 29, 2024
Author

jertel Jan 29, 2024
Maintainer

iliketurtlesyep Jan 30, 2024
Author

jertel Jan 31, 2024
Maintainer