Pfsense OpenVPN parsing

[sleepingbel]

Hello All,

If you have a Pfsense router and you use the OpenVPN feature, you can use the following custum parser scripts in /opt/so/saltstack/local/salt/elasticsearch/files/ingest/

Syslog

 {
  "description" : "syslog",
  "processors" : [
    {
        "dissect": {
                "field": "message",
                "pattern" : "%{message}",
                "on_failure": [ { "drop" : { } } ]
        },
        "remove": {
                "field": [ "type", "agent" ],
                "ignore_failure": true
        }
    },
    {
        "grok":
                {
                        "field": "message",
                        "patterns": [
                                        "^<%{INT:syslog.priority}> %{DATA:source.application}: %{DATE_EU:syslog.timestamp} %{GREEDYDATA:event_data.message}$",
					"^<%{INT:syslog.priority}>%{DATA:syslog.timestamp} %{WORD:source.application}(\\[%{DATA:pid}\\])?: %{GREEDYDATA:real_message}$",
                                        "^%{SYSLOGTIMESTAMP:syslog.timestamp} %{SYSLOGHOST:syslog.host} %{SYSLOGPROG:syslog.program}: CEF:0\\|%{DATA:vendor}\\|%{DATA:product}\\|%{GREEDYDATA:message2}$"
                                    ],
                        "ignore_failure": true
                }
    },
    { "set": { "if": "ctx.source?.application == 'filterlog'", "field": "dataset", "value": "firewall", "ignore_failure": true  } },
    { "set": { "if": "ctx.source?.application == 'openvpn'", "field": "dataset", "value": "vpn", "ignore_failure": true  } },
    { "set": { "if": "ctx.vendor != null", "field": "module", "value": "{{ vendor }}", "ignore_failure": true } },
    { "set": { "if": "ctx.product != null", "field": "dataset", "value": "{{ product }}", "ignore_failure": true } },
    { "set": { "field": "ingest.timestamp",      "value": "{{ @timestamp }}"       } },
    { "date": { "if": "ctx.syslog?.timestamp != null", "field": "syslog.timestamp",       "target_field": "@timestamp",   "formats": ["MMM  d HH:mm:ss", "MMM dd HH:mm:ss", "ISO8601", "UNIX"], "ignore_failure": true  } },
    { "set": { "if": "ctx.source?.application == 'sshguard'", "field": "module", "value": "pfsense", "override": true  } },
    { "set": { "if": "ctx.source?.application == 'openvpn'", "field": "module", "value": "pfsense", "override": true  } },
    { "remove":         { "field": ["pid", "program"],  "ignore_missing": true, "ignore_failure": true  } },
    { "pipeline": { "if": "ctx.vendor != null && ctx.product != null", "name": "{{ vendor }}.{{ product }}", "ignore_failure": true } },
    { "pipeline": { "if": "ctx.dataset == 'firewall'", "name": "filterlog", "ignore_failure": true } },
    { "pipeline": { "if": "ctx.dataset == 'vpn'", "name": "vpn", "ignore_failure": true } },
    { "pipeline":       { "name": "common"                                             } }
  ]
}

[DE4DB4BE]

Hey there, thanks for this! I can't get the parsing to function properly, though. I'm not capturing raw logs on pfsense, not sure if that's required. Are there any other steps needed on the SO side to make this work? I did run sudo so-elasticsearch-restart and the parser loaded fine. Thanks!

[sleepingbel]

The parsing is working for me look

You do need to make a VPN connection to see the logs, this parser only parse actual vpn connections.

[dougburks]

@sleepingbel May we include this in our default parsers?

Yes Doug

I've created #7656 to add this.

[DE4DB4BE]

Interesting - I'm wondering if I'm placing the parser in the wrong location or doing some other thing wrong.
I've placed the parser here: /opt/so/conf/elasticsearch/ingest then ran sudo so-elasticsearch-restart
Am I missing another step for this to be adopted? Is there maybe a log where I can see if there are any errors loading the config?

Thank you so much for all the help with this, I really appreciate it!

[sleepingbel]

Did you placed the syslog and vpn parser also in /opt/so/saltstack/local/salt/elasticsearch/files/ingest/?

The changes on the syslog parser forward the vpn logs to the vpn parser and the vpn parser ingest these in elastic

If you place them directly in /opt/so/conf/elasticsearch/ingest the sysmon parser gets overwritten by salt to the default parser

[DE4DB4BE]

There it is! It was because of me putting the parsers in the incorrect location. All is working perfectly, thanks again very much for your patience and assistance with everything!

[utkarshborawake]

[dougburks]

@utkarshborawake First, please review the discussion guidelines at #1720:

Start a new discussion instead of replying to somebody else's discussion
Please search to see if you can find similar discussions that may help you. However, in order to avoid confusion, please do NOT reply to somebody else's discussion with your own issue. Instead, please start a new discussion and in that new discussion you can provide a hyperlink to the related discussion.

Tagging Individuals
Please do not tag an individual in a discussion unless that individual has already volunteered to help you in that discussion.

Second, please try the Elastic integration for pfSense:
https://docs.securityonion.net/en/2.4/pfsense.html#elastic-integration-for-pfsense

If you have further questions or problems, please start a new discussion rather than replying to this old discussion.