Trigger an alert when Keyword is matched. #1335

mparmar-pure · 2023-12-05T00:24:10Z

mparmar-pure
Dec 5, 2023

Usecase - Trigger an alert when Keyword tunnel-status-down is matched.

Rule -

name: "PAN Tunnel Status - DOWN"
type: any
index: "filebeat-panw*"
es_conn_timeout: 480
realert:
  days: 3
threshold: 1
timeframe:
  minutes: 5
filter:
- query:
    query_string:
      query: "event.original: *tunnel-status-down*"
doc_type: _doc
alert:
- "jira"
jira_server: "https://XX.atlassian.net/"
jira_project: "XX"
jira_issuetype: "Incident"
jira_account_file: "/XX"
jira_priority: 0
jira_description: "One of the IPSec Tunnel is DOWN . \n"

Elastic document sample -

{ "_index": "filebeat-panw", "_id": "VyxDGIwBYoxa48FWSKBA", "_version": 1, "_score": 0, "_source": { "agent": { "name": "XX", "id": "48e8bf3f-1338-45e5-b0eb-40775c980545", "type": "filebeat", "ephemeral_id": "fd708e48-0ff4-47fe-a80e-47c73c58baf1", "version": "8.7.0" }, "log": { "source": { "address": "XX" } }, "syslog": { "priority": 10, "facility": 1, "severity_label": "Critical", "facility_label": "user-level" }, "fileset": { "name": "panos" }, "panw": { "panos": { "sub_type": "vpn", "type": "SYSTEM" } }, "tags": [ "pan-os", "forwarded" ], "observer": { "product": "PAN-OS", "vendor": "Palo Alto Networks", "serial_number": "XX", "type": "firewall" }, "input": { "type": "syslog" }, "hostname": "XX", "@timestamp": "2023-11-28T15:28:29.000-08:00", "ecs": { "version": "1.12.0" }, "service": { "type": "panw" }, "host": { "hostname": "XX", "os": { "kernel": "3.10.0-1127.19.1.el7.x86_64", "codename": "Core", "name": "CentOS Linux", "type": "linux", "family": "redhat", "version": "7 (Core)", "platform": "centos" }, "ip": [ "XX", "fe80::c8e6:4c08:efdc:1c5", "fe80::a170:d9f1:4a58:5aaf", "fe80::d422:b9d:a791:94e3" ], "containerized": false, "id": "c70b7b076b6f4ce9bfc09b21f7158f81", "mac": [ "00-50-56-83-D9-AC" ], "architecture": "x86_64" }, "event": { "severity": 2, "ingested": "2023-11-28T23:28:31.799156510Z", "original": "1,2023/11/28 15:28:29,016401001527,SYSTEM,vpn,2562,2023/11/28 15:28:29,,tunnel-status-down,aws-ipsec-tunnel,0,158,general,critical,\"Tunnel aws-ipsec-tunnel is down\",7305951047959232404,0x0,0,0,0,0,,-11-28T15:28:29.986-08:00", "timezone": "-08:00", "created": "2023-11-28T15:28:29.000-08:00", "module": "panw", "dataset": "panw.panos", "outcome": "success" } }, "fields": { "agent.version.keyword": [ "8.7.0" ], "host.architecture.keyword": [ "x86_64" ], "syslog.severity_label.keyword": [ "Critical" ], "event.dataset.keyword": [ "panw.panos" ], "event.outcome.keyword": [ "success" ], "host.hostname": [ "XX" ], "host.mac": [ "00-50-56-83-D9-AC" ], "observer.type.keyword": [ "firewall" ], "service.type": [ "panw" ], "observer.vendor": [ "Palo Alto Networks" ], "hostname": [ "XX" ], "host.ip.keyword": [ "XX", "fe80::c8e6:4c08:efdc:1c5", "fe80::a170:d9f1:4a58:5aaf", "fe80::d422:b9d:a791:94e3" ], "ecs.version.keyword": [ "1.12.0" ], "host.os.version": [ "7 (Core)" ], "host.os.name": [ "CentOS Linux" ], "panw.panos.type": [ "SYSTEM" ], "host.id.keyword": [ "c70b7b076b6f4ce9bfc09b21f7158f81" ], "agent.name": [ "XX" ], "host.os.version.keyword": [ "7 (Core)" ], "event.outcome": [ "success" ], "event.severity": [ 2 ], "event.original": [ "1,2023/11/28 15:28:29,016401001527,SYSTEM,vpn,2562,2023/11/28 15:28:29,,tunnel-status-down,aws-ipsec-tunnel,0,158,general,critical,\"Tunnel aws-ipsec-tunnel is down\",7305951047959232404,0x0,0,0,0,0,,-11-28T15:28:29.986-08:00" ], "host.os.type": [ "linux" ], "observer.serial_number.keyword": [ "016401001527" ], "fileset.name": [ "panos" ], "agent.id.keyword": [ "48e8bf3f-1338-45e5-b0eb-40775c980545" ], "input.type": [ "syslog" ], "observer.serial_number": [ "016401001527" ], "tags": [ "pan-os", "forwarded" ], "host.architecture": [ "x86_64" ], "fileset.name.keyword": [ "panos" ], "agent.id": [ "48e8bf3f-1338-45e5-b0eb-40775c980545" ], "observer.type": [ "firewall" ], "host.containerized": [ false ], "ecs.version": [ "1.12.0" ], "log.source.address": [ "XX" ], "event.created": [ "2023-11-28T23:28:29.000Z" ], "host.hostname.keyword": [ "XX" ], "event.module.keyword": [ "panw" ], "agent.version": [ "8.7.0" ], "observer.product.keyword": [ "PAN-OS" ], "host.os.family": [ "redhat" ], "hostname.keyword": [ "XX" ], "event.timezone.keyword": [ "-08:00" ], "observer.vendor.keyword": [ "Palo Alto Networks" ], "service.type.keyword": [ "panw" ], "input.type.keyword": [ "syslog" ], "tags.keyword": [ "pan-os", "forwarded" ], "syslog.facility": [ 1 ], "panw.panos.sub_type.keyword": [ "vpn" ], "host.ip": [ "XX", "fe80::c8e6:4c08:efdc:1c5", "fe80::a170:d9f1:4a58:5aaf", "fe80::d422:b9d:a791:94e3" ], "agent.type": [ "filebeat" ], "host.os.kernel.keyword": [ "3.10.0-1127.19.1.el7.x86_64" ], "event.module": [ "panw" ], "host.os.kernel": [ "3.10.0-1127.19.1.el7.x86_64" ], "panw.panos.sub_type": [ "vpn" ], "host.os.name.keyword": [ "CentOS Linux" ], "observer.product": [ "PAN-OS" ], "host.id": [ "c70b7b076b6f4ce9bfc09b21f7158f81" ], "event.timezone": [ "-08:00" ], "agent.type.keyword": [ "filebeat" ], "agent.ephemeral_id.keyword": [ "fd708e48-0ff4-47fe-a80e-47c73c58baf1" ], "host.os.codename.keyword": [ "Core" ], "host.mac.keyword": [ "XX" ], "agent.name.keyword": [ "XX" ], "syslog.priority": [ 10 ], "host.os.codename": [ "Core" ], "host.os.family.keyword": [ "redhat" ], "event.ingested": [ "2023-11-28T23:28:31.799Z" ], "host.os.type.keyword": [ "linux" ], "syslog.severity_label": [ "Critical" ], "@timestamp": [ "2023-11-28T23:28:29.000Z" ], "host.os.platform.keyword": [ "centos" ], "host.os.platform": [ "centos" ], "panw.panos.type.keyword": [ "SYSTEM" ], "event.original.keyword": [ "1,2023/11/28 15:28:29,016401001527,SYSTEM,vpn,2562,2023/11/28 15:28:29,,tunnel-status-down,aws-ipsec-tunnel,0,158,general,critical,\"Tunnel aws-ipsec-tunnel is down\",7305951047959232404,0x0,0,0,0,020,0,0,2023-11-28T15:28:29.986-08:00" ], "syslog.facility_label": [ "user-level" ], "agent.ephemeral_id": [ "fd708e48-0ff4-47fe-a80e-47c73c58baf1" ], "log.source.address.keyword": [ "XX" ], "syslog.facility_label.keyword": [ "user-level" ], "event.dataset": [ "panw.panos" ] } }

Logs -

Dec  4 16:20:05 prod-elk-monitor python[68421]: WARNING:elasticsearch:POST http://XX:80/filebeat-panw*/_search?_source_includes=%2A%2C%40timestamp&ignore_unavailable=true&scroll=30s&size=10000 [status:N/A request:301.806s]
Dec  4 16:20:05 prod-elk-monitor python[68421]: Traceback (most recent call last):
Dec  4 16:20:05 prod-elk-monitor python[68421]:   File "/root/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 715, in urlopen
Dec  4 16:20:05 prod-elk-monitor python[68421]:     httplib_response = self._make_request(
Dec  4 16:20:05 prod-elk-monitor python[68421]:                        ^^^^^^^^^^^^^^^^^^^
Dec  4 16:20:05 prod-elk-monitor python[68421]:   File "/root/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 467, in _make_request
Dec  4 16:20:05 prod-elk-monitor python[68421]:     six.raise_from(e, None)
Dec  4 16:20:05 prod-elk-monitor python[68421]:   File "<string>", line 3, in raise_from
Dec  4 16:20:05 prod-elk-monitor python[68421]:   File "/root/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 462, in _make_request
Dec  4 16:20:05 prod-elk-monitor python[68421]:     httplib_response = conn.getresponse()
Dec  4 16:20:05 prod-elk-monitor python[68421]:                        ^^^^^^^^^^^^^^^^^^
Dec  4 16:20:05 prod-elk-monitor python[68421]:   File "/usr/lib/python3.11/http/client.py", line 1378, in getresponse
Dec  4 16:20:05 prod-elk-monitor python[68421]:     response.begin()
Dec  4 16:20:05 prod-elk-monitor python[68421]:   File "/usr/lib/python3.11/http/client.py", line 318, in begin
Dec  4 16:20:05 prod-elk-monitor python[68421]:     version, status, reason = self._read_status()
Dec  4 16:20:05 prod-elk-monitor python[68421]:                               ^^^^^^^^^^^^^^^^^^^
Dec  4 16:20:05 prod-elk-monitor python[68421]:   File "/usr/lib/python3.11/http/client.py", line 279, in _read_status
Dec  4 16:20:05 prod-elk-monitor python[68421]:     line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
Dec  4 16:20:05 prod-elk-monitor python[68421]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Dec  4 16:20:05 prod-elk-monitor python[68421]:   File "/usr/lib/python3.11/socket.py", line 706, in readinto
Dec  4 16:20:05 prod-elk-monitor python[68421]:     return self._sock.recv_into(b)
Dec  4 16:20:05 prod-elk-monitor python[68421]:            ^^^^^^^^^^^^^^^^^^^^^^^
Dec  4 16:20:05 prod-elk-monitor python[68421]: ConnectionResetError: [Errno 104] Connection reset by peer
Dec  4 16:20:05 prod-elk-monitor python[68421]: During handling of the above exception, another exception occurred:
Dec  4 16:20:05 prod-elk-monitor python[68421]: Traceback (most recent call last):
Dec  4 16:20:05 prod-elk-monitor python[68421]:   File "/root/venv/lib/python3.11/site-packages/requests/adapters.py", line 486, in send
Dec  4 16:20:05 prod-elk-monitor python[68421]:     resp = conn.urlopen(
Dec  4 16:20:05 prod-elk-monitor python[68421]:            ^^^^^^^^^^^^^
Dec  4 16:20:05 prod-elk-monitor python[68421]:   File "/root/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 799, in urlopen
Dec  4 16:20:05 prod-elk-monitor python[68421]:     retries = retries.increment(
Dec  4 16:20:05 prod-elk-monitor python[68421]:               ^^^^^^^^^^^^^^^^^^
Dec  4 16:20:05 prod-elk-monitor python[68421]:   File "/root/venv/lib/python3.11/site-packages/urllib3/util/retry.py", line 550, in increment
Dec  4 16:20:05 prod-elk-monitor python[68421]:     raise six.reraise(type(error), error, _stacktrace)
Dec  4 16:20:05 prod-elk-monitor python[68421]:           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Dec  4 16:20:05 prod-elk-monitor python[68421]:   File "/root/venv/lib/python3.11/site-packages/urllib3/packages/six.py", line 769, in reraise
Dec  4 16:20:05 prod-elk-monitor python[68421]:     raise value.with_traceback(tb)
Dec  4 16:20:05 prod-elk-monitor python[68421]:   File "/root/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 715, in urlopen
Dec  4 16:20:05 prod-elk-monitor python[68421]:     httplib_response = self._make_request(
Dec  4 16:20:05 prod-elk-monitor python[68421]:                        ^^^^^^^^^^^^^^^^^^^
Dec  4 16:20:05 prod-elk-monitor python[68421]:   File "/root/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 467, in _make_request
Dec  4 16:20:05 prod-elk-monitor python[68421]:     six.raise_from(e, None)
Dec  4 16:20:05 prod-elk-monitor python[68421]:   File "<string>", line 3, in raise_from
Dec  4 16:20:05 prod-elk-monitor python[68421]:   File "/root/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 462, in _make_request
Dec  4 16:20:05 prod-elk-monitor python[68421]:     httplib_response = conn.getresponse()
Dec  4 16:20:05 prod-elk-monitor python[68421]:                        ^^^^^^^^^^^^^^^^^^
Dec  4 16:20:05 prod-elk-monitor python[68421]:   File "/usr/lib/python3.11/http/client.py", line 1378, in getresponse
Dec  4 16:20:05 prod-elk-monitor python[68421]:     response.begin()
Dec  4 16:20:05 prod-elk-monitor python[68421]:   File "/usr/lib/python3.11/http/client.py", line 318, in begin
Dec  4 16:20:05 prod-elk-monitor python[68421]:     version, status, reason = self._read_status()
Dec  4 16:20:05 prod-elk-monitor python[68421]:   File "/usr/lib/python3.11/http/client.py", line 279, in _read_status
Dec  4 16:20:05 prod-elk-monitor python[68421]:     line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
Dec  4 16:20:05 prod-elk-monitor python[68421]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Dec  4 16:20:05 prod-elk-monitor python[68421]:   File "/usr/lib/python3.11/socket.py", line 706, in readinto
Dec  4 16:20:05 prod-elk-monitor python[68421]:     return self._sock.recv_into(b)
Dec  4 16:20:05 prod-elk-monitor python[68421]:            ^^^^^^^^^^^^^^^^^^^^^^^
Dec  4 16:20:05 prod-elk-monitor python[68421]: urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
Dec  4 16:20:05 prod-elk-monitor python[68421]: During handling of the above exception, another exception occurred:
Dec  4 16:20:05 prod-elk-monitor python[68421]: Traceback (most recent call last):
Dec  4 16:20:05 prod-elk-monitor python[68421]:   File "/root/venv/lib/python3.11/site-packages/elasticsearch/connection/http_requests.py", line 161, in perform_request
Dec  4 16:20:05 prod-elk-monitor python[68421]:     response = self.session.send(prepared_request, **send_kwargs)
Dec  4 16:20:05 prod-elk-monitor python[68421]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Dec  4 16:20:05 prod-elk-monitor python[68421]:   File "/root/venv/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
Dec  4 16:20:05 prod-elk-monitor python[68421]:     r = adapter.send(request, **kwargs)
Dec  4 16:20:05 prod-elk-monitor python[68421]:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Dec  4 16:20:05 prod-elk-monitor python[68421]:   File "/root/venv/lib/python3.11/site-packages/requests/adapters.py", line 501, in send
Dec  4 16:20:05 prod-elk-monitor python[68421]:     raise ConnectionError(err, request=request)
Dec  4 16:20:05 prod-elk-monitor python[68421]: requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

Answered by jertel

Dec 7, 2023

It looks fine after a quick glance, though the threshold: 1 is not needed for an any alert. Here are some suggestions on how to troubleshoot:

Try simplifying your query to something without wildcards, something you know would absolutely find a document.
Try using index: "*" to eliminate the risk of an index issue.
Make sure the user the ElastAlert2 authenticates to Elasticsearch has access to that index.

Once you get a hit you can work backwards to find out which specific change prevents the match.

View full answer

jertel · 2023-12-05T00:30:32Z

jertel
Dec 5, 2023
Maintainer

ElastAlert2 can't reach your Elasticsearch server. Check your config file's host and port values, and then check that there is sufficient network access.

4 replies

mparmar-pure Dec 5, 2023
Author

Rest of the rules (flatline rules) are working as expected, there is an issue with just this rule specifically.

jertel Dec 5, 2023
Maintainer

Compare the logs of the successful and unsuccessful rules to see if the protocol, host and port are all the same. Also, check the Elasticsearch server log to see if it's encountering an error with this rule's query, which causes it to terminate the socket.

mparmar-pure Dec 6, 2023
Author

Updated the linux setting for tcp keepalive, seems to have resolved the timeout issue.

cat /proc/sys/net/ipv4/tcp_keepalive_time 200 cat /proc/sys/net/ipv4/tcp_keepalive_probes 20 cat /proc/sys/net/ipv4/tcp_keepalive_intvl 60

But the rule itself is not finding match, does this look accurate -

name: "PAN Tunnel Status - DOWN"
type: any
index: "filebeat-panw*"
es_conn_timeout: 480
realert:
  days: 3
threshold: 1
timeframe:
  minutes: 5
filter:
- query:
    query_string:
      query: "event.original: *tunnel-status-down*"
doc_type: _doc
alert:
- "jira"
jira_server: "https://XX.atlassian.net/"
jira_project: "XX"
jira_issuetype: "Incident"
jira_account_file: "/XX"
jira_priority: 0
jira_description: "One of the IPSec Tunnel is DOWN . \n"

ELK document has one of the field as -

"event.original": [ "1,2023/11/28 15:28:29,016401001527,SYSTEM,vpn,2562,2023/11/28 15:28:29,,tunnel-status-down,aws-ipsec-tunnel,0,158,general,critical,\"Tunnel aws-ipsec-tunnel is down\",7305951047959232404,0x0,0,0,0,0,,-11-28T15:28:29.986-08:00" ]

jertel Dec 7, 2023
Maintainer

It looks fine after a quick glance, though the threshold: 1 is not needed for an any alert. Here are some suggestions on how to troubleshoot:

Try simplifying your query to something without wildcards, something you know would absolutely find a document.
Try using index: "*" to eliminate the risk of an index issue.
Make sure the user the ElastAlert2 authenticates to Elasticsearch has access to that index.

Once you get a hit you can work backwards to find out which specific change prevents the match.

Answer selected by jertel

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Trigger an alert when Keyword is matched. #1335

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment 4 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Trigger an alert when Keyword is matched. #1335

mparmar-pure Dec 5, 2023

Replies: 1 comment · 4 replies

jertel Dec 5, 2023 Maintainer

mparmar-pure Dec 5, 2023 Author

jertel Dec 5, 2023 Maintainer

mparmar-pure Dec 6, 2023 Author

jertel Dec 7, 2023 Maintainer

mparmar-pure
Dec 5, 2023

Replies: 1 comment 4 replies

jertel
Dec 5, 2023
Maintainer

mparmar-pure Dec 5, 2023
Author

jertel Dec 5, 2023
Maintainer

mparmar-pure Dec 6, 2023
Author

jertel Dec 7, 2023
Maintainer