[Filebeat] Add support for syslog rfc 5424

[faec]

Resolves #6872.

Draft, needs testing.

[strawgate]

Would it be possible to have the input detect 5424 vs 3164 versus manually specifying a format in the input? Having user's determine the specific syslog format and send it to a different port/filebeat complicates deployment of syslog support and it would seem the formats are different enough to allow a quick detection of the correct format?

[trunet]

Would it be possible to have the input detect 5424 vs 3164 versus manually specifying a format in the input? Having user's determine the specific syslog format and send it to a different port/filebeat complicates deployment of syslog support and it would seem the formats are different enough to allow a quick detection of the correct format?

I did some research, rsyslog per example have different parsers and you have to specify. Syslog-ng, have a detection "algorithm".

It tries to detect using the timestamp field on modules/syslogformat/syslog-format.c using functions from lib/timeutils/scan-timestamp.c.

• RFC 3164 timestamp, expected format: "MMM DD HH:MM:SS" ...
• ISO timestamp as specified in RFC5424, expected format "YYYY-MM-DDTHH:MM:SS"
• Cisco modified RFC3164 timestamp, expected format:
  • "MMM DD YYYY HH:MM:SS:"
  • "MMM DD YYYY HH:MM:SS "
• LinkSys modified RFC3164 timestamp, expected format:
  • "MMM DD HH:MM:SS YYYY "

Another way to detect, rfc5424 has VERSION just after PRI field. Like <34>1.

On logstash-plugins/logstash-input-syslog#15 (comment) there's a code proposal:

  if [type] == "syslog" { 
  
    # look for and, if found, decode syslog priority
    if [message] =~ "^<[0-9]{1,3}>" { 
      grok {
        match => [ "message", "^<%{NONNEGINT:priority:int}>" ]
      }
      if [priority] <= 191 {
        # check for RFC 3164 vs RFC 5424
        if [message] =~ "^<[0-9]{1,3}>[0-9]{1,2} " {
          mutate {
            add_tag => ["syslog_rfc5424"]
          }
        }
        else {
	  mutate {
            add_tag =>  ["syslog_rfc3164"]
          }
        }
      }
      else {
        mutate {  
          add_tag => ["syslog_priority_invalid"]
        }
      }
    } 
    else {
      # only RFC 3164 allows a message to specify no priority
      mutate {  
	add_tag => [ "syslog_rfc3164", "syslog_priority_missing" ]
      }
    }

[adriansr]

For the use-cases we have in SIEM I think it's critical that a new syslog input can detect automatically the format in use. Some users have hundreds of devices sending their syslog to Filebeat and it would be impractical to have two different listening ports depending on each device configuration.

Furthermore, we've seen a few different date formats in use. I'm no syslog expert but it looks to me that a same RFC can have different date formats in the wild, some with YYYY, some with subsecond precission, different tz formats, etc. This an updated syslog will at least tolerate, if not parse 100% correctly:

$ cat x-pack/filebeat/module/*/*/test/*.log filebeat/module/*/*/test/*log | cut -c-30 | tr '0123456789' 'N' | grep '^[A-Za-z]\{1,4\}\w' | grep 'NN:NN:' | sort | uniq -c | sort -nr
 537 Oct NN NNNN NN:NN:NN localhost
  34 Apr NN NNNN NN:NN:NN: %FTD-N-N
  34 Apr NN NNNN NN:NN:NN: %ASA-N-N
  30 Jun NN NN:NN:NN NNN.NN.NNN.N N
  21 NNNN-NN-NNTNN:NN:NNZ siem-ftd
  20 Sep NN NNNN NN:NN:NN GIFRCHNNN
  16 NNNN-NN-NNTNN:NN:NNZ firepower
  15 http NNNN-NN-NNTNN:NN:NN.NNNNN
  15 NNNN-NN-NN NN:NN:NN,NNN | INFO
  15 NN/NN/NNNN NN:NN:NN - Process(
  15 Dec NN NNNN NN:NN:NN NNN.N.N.N
  15 Dec NN NNNN NN:NN:NN <IP>: %AS
  12 NNNN-NN-NNTNN:NN:NN.NNNZ Feb N
  11 NNNN-NN-NNTNN:NN:NN.NNNNNNZ fi
   8 Aug NN NNNN NN:NN:NN siem-ftd
   7 NNNN-NN-NNTNN:NN:NN.NNNNNNZ my
   5 NN.NN.NNNN NN:NN:NN - Process(
   4 Jan NN NNNN NN:NN:NN: %FTD-N-N
   4 Jan NN NNNN NN:NN:NN: %ASA-N-N
   4 Jan NN NNNN NN:NN:NN beats ftd
   4 Apr NN NNNN NN:NN:NN INT-FWNN
   3 Nov NN NNNN NN:NN:NN: %FTD-N-N
   3 Nov NN NNNN NN:NN:NN: %ASA-N-N
   3 Jan  N NN:NN:NN beats asa[NNNN
   2 Jun NN NNNN NN:NN:NN FJSGNNRFW
   2 Jan  N NNNN NN:NN:NN beats asa
   2 Feb  N NN:NN:NN NNN.NN.NNN.N N
   2 Apr NN NNNN NN:NN:NN EDT: %FTD
   2 Apr NN NNNN NN:NN:NN EDT: %ASA
   1 wss NNNN-NN-NNTNN:NN:NN.NNNNNN
   1 ws NNNN-NN-NNTNN:NN:NN.NNNNNNZ
   1 tls N.N NNNN-NN-NNTNN:NN:NN ne
   1 https NNNN-NN-NNTNN:NN:NN.NNNN
   1 hN NNNN-NN-NNTNN:NN:NN.NNNNNNZ
   1 NNNN-NN-NNTNN:NN:NNZ CISCO-SEN
   1 NNNN-NN-NNTNN:NN:NN.NNNZ [INFO
   1 NNNN-NN-NNTNN:NN:NN.NNN-NN:NN
   1 NNNN-NN-NN NN:NN:NN,NNN | ERRO
   1 May  N NN:NN:NN NNN.NN.NNN.N N
   1 Jun NN NNNN NN:NN:NN MYHOSTNAM
   1 Feb NN NNNN NN:NN:NN: %ASA-N-N
   1 Feb NN NN:NN:NN NNN.NN.NNN.N N
   1 Aug NN NNNN NN:NN:NN: %FTD-N-N
   1 Aug NN NNNN NN:NN:NN : %ASA-N-

[botelastic]

Hi!
We just realized that we haven't looked into this PR in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it in as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!

[elasticmachine]

Pinging @elastic/integrations-services (Team:Services)

[botelastic]

Hi!
We just realized that we haven't looked into this PR in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it in as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!

[botelastic]

Hi!
This PR has been stale for a while and we're going to close it as part of our cleanup procedure.
We appreciate your contribution and would like to apologize if we have not been able to review it, due to the current heavy load of the team.
Feel free to re-open this PR if you think it should stay open and is worth rebasing.
Thank you for your contribution!