A Perl based tool to parse DMARC reports, based on John Levine's rddmarc, but extended by the following features:
- Allow to read messages from an IMAP server and not only from the local filesystem.
- Store much more XML values into the database (for example the missing SPF and DKIM results from the policy_evaluated section) and also the entire XML for later reference.
- Needed database tables and columns are created automatically, user only needs to provide a database. The database schema is compatible to the one used by rddmarc, but extends it by additional fields. Users can switch from rddmarc to dmarcts-report-parser without having to do any changes to the database by themselves.
- Support for MySQL and PostgreSQL
Open Report Parser is a fork of techsneeze's dmarcts-report-parser, and was forked to more closely match the needs of Open DMARC Analyzer.
Open Report Parser Version 0 Alpha 5 (0-α5) is an Anomaly <Codebase> project by John Bradley ([email protected]).
- Perl 5
- MariaDB 10.5 or equivalent
- PostgreSQL 13.9
apt-get install libfile-mimeinfo-perl libmail-imapclient-perl libmime-tools-perl libxml-simple-perl \
libio-socket-inet6-perl libio-socket-ip-perl libperlio-gzip-perl \
libmail-mbox-messageparser-perl libwww-perl libjson-perl unzip
- For MySQL:
libdbd-mysql-perl - For PostgreSQL:
libdbd-pg-perl - For Oauth2:
liblwp-protocol-https-perl libencode-perl libtime-piece-mysql-perl
sudo dnf install perl-File-MimeInfo perl-Mail-IMAPClient perl-MIME-tools perl-XML-Simple perl-DBI \
perl-Socket6 perl-PerlIO-gzip perl-libwww-perl unzip
- For MySQL:
perl-DBD-MySQL - For PostgreSQL:
perl-DBD-Pg - For Oauth2:
perl-LWP-Protocol-https perl-Encode perl-Time-Piece-MySQL
yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
yum install perl-File-MimeInfo perl-Mail-IMAPClient perl-MIME-tools perl-XML-Simple perl-DBI \
perl-Socket6 perl-PerlIO-gzip perl-libwww-perl unzip perl-Mail-Mbox-MessageParser
- For MySQL:
perl-DBD-MySQL - For PostgreSQL:
perl-DBD-Pg - For Oauth2:
perl-LWP-Protocol-https perl-Encode perl-Time-Piece-MySQL
sudo pkg install p5-File-MimeInfo p5-Mail-IMAPClient p5-MIME-tools p5-XML-Simple p5-DBI p5-Socket6 p5-PerlIO-gzip p5-Mail-Mbox-MessageParser p5-libwww unzip
- For MySQL:
p5-DBD-MySQL - For PostgreSQL:
p5-DBD-Pg - For Oauth2:
p5-LWP-Protocol-https p5-Encode p5-Time-Piece-MySQL
brew install mysql shared-mime-info
update-mime-database /usr/local/share/mime
perl -MCPAN -e 'install Mail::IMAPClient'
perl -MCPAN -e 'install Mail::Mbox::MessageParser'
perl -MCPAN -e 'install File::MimeInfo'
perl -MCPAN -e 'install LWP::UserAgent
- For MySQL:
perl -MCPAN -e 'install DBD::mysql' - For PostgreSQL:
perl -MCPAN -e 'install DBD::pg' - For Oauth2:
perl -MCPAN -e 'install LWP::Protocol::https'
perl -MCPAN -e 'install Encode'
perl -MCPAN -e 'install Time::Piece'
Optaining Open Report Parser through git is probably the easiest way, in addition to doing occasional pulls to get up-to-date versions.
git clone https://github.com/userjack6880/Open-Report-Parser.git
Optionally, a zip file of the latest release can be downloaded.
Rename repoart-parser.conf.pub to report-parser.conf and edit the configuration for your environment (see the next section on Configuration Options for details). Finally, some condierations need to be taken in account due to limitations in stock configurations of MySQL/MariaSQL on some distros, it may be necessary to add the following to your configuration (i.e. in /etc/mysql/mariadb.conf.d/50-server.cnf):
innodb_large_prefix = on
innodb_file_format = barracuda
innodb_file_per_table = true
Script Options
$debug = 0;
$delete_reports = 0;
#$dmarc_only = 0;
# if set to 1, do not process tls reports, if set to -1 do not process
# dmarc reports - defaults to 1 for legacy support
# this is ignored for all methods except IMAP, use --tls to process
# TLS reports for other methodsDatabase Options
#$dbtype = 'mysql'; # Supported types - mysql, postgres - defaults to mysql if unset
$dbname = 'dmarc';
$dbuser = 'dmarc';
$dbpass = 'password';
#$dbhost = 'dbhost'; # Set the hostname if we can't connect to the local socket.
$dbport = '3306';IMAP Options
$imapserver = 'imap.server';
$imapuser = 'username';
$imappass = 'password';
$imapport = '143';
$imapssl = '0'; # If set to 1, remember to change server port to 993 and disable imaptls.
$imaptls = '0';
$tlsverify = '0';
$imapignoreerror = '0'; # recommended if you use MS Exchange 2007, ...
#$imapauth = 'simple'; # supported - simple, oauth2 - defaults to simple if unset
# see documentation for detailed setup
#$oauthclientid = '';
#$oauthuri = '';
$imapdmarcfolder = 'dmarc';
$imaptlsfolder = 'tls';
# If $imapxxxproc is set, processed IMAP messages will be moved (overruled by
# the --delete option!)
# $imapdmarcproc = 'dmarc.Processed';
# $imaptlsproc = 'tls.Processed';
# If $imapxxxerr is set, IMAP messages that fail will be moved. If unset, failed messages
# will move to $imapdmarcproc (if it is set). Overruled by the --delete option!
# $imapdmarcerr = 'dmarc.notProcessed';
# $imaptlserr = 'tls.notProcessed';These settings are ignored when using the -m flag. When using SSL, TLS needs to be disabled and the port used should be changed to 993. TLS Verify is ignored.
Setting $imapauth to 'oauth2' enables OAuth2 authentication, and requires an initial dance to verify the application with your provider. Once this is done, it should be able to renew the token automatically on subsequent runs. Currently, only OAuth2 with M365 has been tested.
XML Storage Options
# maximum size of XML/JSON files to store in database, long files can cause transaction aborts
$maxsize_xml = 50000;
$maxsize_json = 50000;
# store XML/JSON as base64 encopded gzip in database (save space, harder usable)
$compress_xml = 0;
$compress_json = 0;Processing Failure Action
# if there was an error during file processing (message does not contain XML or ZIP parts,
# or a database error) the parser reports an error and does not delete the file, even if
# delete_reports is set (or --delete is given). Deletion can be enforced by delete_failed,
# however not for database errors.
$delete_failed = 0;The script is looking for report-parser.conf in the current working directory. If not found it will look by the calling path. If neither is found than it will abort.
Note: Be sure to use the proper hierarchy separator for your server in all folder specs, and if your IMAP server flattens the hierarchy (i.e. Cyrus IMAP with "altnamespace: yes") then leave "Inbox" off of the beginning of such specs.
./report-parser.pl [OPTIONS] [PATH]
PATH can be the filename of a single file or a list of files - wildcard expression are allowed.
Remember: This script needs a configurations file called <report-parser.conf> in the current working directory, which defines a database server with credentials and (if used) an IMAP server with credentials.
One of the following source options must be provided:
-i : Read reports from messages on IMAP server as defined in the
config file.
-m : Read reports from mbox file(s) provided in PATH.
-e : Read reports from MIME email file(s) provided in PATH.
-x : Read reports from xml file(s) provided in PATH.
-j : Read reports from json files(s) provided in PATH.
-z : Read reports from zip file(s) provided in PATH.
The following options are always allowed:
-d : Print debug info.
-r : Replace existing reports rather than skipping them.
--delete : Delete processed message files (the XML is stored in the
database for later reference).
--info : Print out number of XML files or emails processed.
--tls : Force TLS-Only Mode.
Currently, processing of both DMARC and TLS reports during the same run is only supported from IMAP. All other sources will always default to DMARC reports unless the --tls flag is provided at runtime.
- Fixed an issue with undefined variables discovered with Issue #12.
- Fixed an issue with mimecast improperly setting mime type outlined in Issue #12.
| OS | Perl | SQL | Source |
|---|---|---|---|
| Debian 11.6 | Perl 5.32 | MariaDB 10.5.18 | Postfix/Dovecot IMAP Basic Auth |
| Debian 11.6 | Perl 5.32 | PostgreSQL 13.9 | Postfix/Dovecot IMAP Basic Auth |
This project regular release cycle is not yet determined. Versioning is under the Anomaly Versioning Scheme (2022), as outlined in VERSIONING under docs.
Support will be provided as outlined in the following schedule. For more details, see SUPPORT.
| Version | Support Level | Released | End of Support | End of Life |
|---|---|---|---|---|
| Version 1 Alpha 5 | Full Support | 10 July 2023 | TBD | TBD |
| Version 1 Alpha 4 | Critical Support | 8 June 2023 | 10 July 2023 | TBD |
| Version 1 Alpha 3 | End of Life | 25 May 2023 | 8 June 2023 | 10 July 2023 |
Public contributions are encouraged. Please review CONTRIBUTING under docs for contributing procedures. Additionally, please take a look at our CODE_OF_CONDUCT. By participating in this project you agree to abide by the Code of Conduct.
Primary Contributors
- John Bieling - Initial Work
- TechZneeze.com - Expansion of Initial Work
- John Bradley - This Fork
Thanks to all who contributed (and here) and have given feedback.
Copyright © 2023 John Bradley (userjack6880), Copyright © 2016 TechSneeze.com, Copyright © 2012 John Bieling. Open Report Parser is released under GNU GPLv3. See LICENSE.