diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..e43b0f9 --- /dev/null +++ b/.gitignore @@ -0,0 +1 @@ +.DS_Store diff --git a/README.md b/README.md index 6844ceb..a9b9b08 100644 --- a/README.md +++ b/README.md @@ -5,10 +5,10 @@ The Unified Log Streamer (ULS) is designed to simplify SIEM integrations for Akamai Secure Enterprise Access Products - [Enterprise Application Access (EAA)](https://www.akamai.com/us/en/products/security/enterprise-application-access.jsp) - [Enterprise Threat Protector (ETP)](https://www.akamai.com/us/en/products/security/enterprise-threat-protector.jsp) -- [Akamai Phish-proof Multi Factor Authenticator (AKAMAI-MFA)](https://www.akamai.com/us/en/products/security/akamai-mfa.jsp) +- [Akamai MFA (MFA)](https://www.akamai.com/us/en/products/security/akamai-mfa.jsp) Thanks to its modular design, ULS allows the connection of many SIEM solutions out-of-the-box. -ULS can send data into any SIEM that supports either TCP, UDP or HTTP ingestion. +ULS can send data into any SIEM that supports either file, TCP, UDP or HTTP ingestion. It can be run directly as Python code, as a provided Docker container or through `docker compose` scripts. @@ -21,10 +21,11 @@ It can be run directly as Python code, as a provided Docker container or through - [Table of contents](#table-of-contents) - [Key Features](#key-features) - [Documentation](#documentation) + - [Generic Requirements](#generic-requirements) - [Command Line Usage](#command-line-usage) - [Docker](#docker) - [Docker-compose](#docker-compose) - - [kubernetes / k8s](#kubernetes) + - [Kubernetes](#kubernetes) - [Development](#development) - [Changelog](#changelog) - [Support](#support) diff --git a/docs/DOCKER_USAGE.md b/docs/DOCKER_USAGE.md index d80c573..6e3d1aa 100644 --- a/docs/DOCKER_USAGE.md +++ b/docs/DOCKER_USAGE.md @@ -5,7 +5,7 @@ All commands referenced in this document are run from the repositories root leve ### Table of contents - [ULS Docker Usage](#uls-docker-usage) - - [Overview](#overview) + - [Table of contents](#table-of-contents) - [Requirements](#requirements) - [Installation](#installation) - [Obtaining the Docker image](#obtaining-the-docker-image) @@ -14,7 +14,7 @@ All commands referenced in this document are run from the repositories root leve ## Requirements - [Docker](https://www.docker.com/) needs to be installed on an **GNU/Linux** OS - - Note: Windows is not supported, please use HyperV with a Linux VM + - Note: Windows is not supported, please use Hyper-V with a Linux VM - Access to the docker image (see [installation](#installation)) - Akamai API credentials file - `.edgerc` (see [API Credentials](AKAMAI_API_CREDENTIALS.md) for creation instructions) - Understanding of available [ULS Environmental Variables and CLI PARAMETERS](ARGUMENTS_ENV_VARS.md) @@ -50,9 +50,12 @@ docker run ... ``` ## Usage -Using the dockerized approach, you have two different options to set up the options and parameters: -- Docker Command Line Arguments +Using the dockerized approach, you have two different ways to set up the options and parameters. + +Below are two examples with our Enterprise Threat Protector product: + +- Docker Command Line Arguments: ```bash docker run -d --name uls_etp-threat -ti \ --mount type=bind,source="/path/to/your/.edgerc",target="/opt/akamai-uls/.edgerc",readonly \ @@ -60,7 +63,7 @@ Using the dockerized approach, you have two different options to set up the opti --input etp --feed threat --output tcp --host 10.10.10.10 --port 9091 ``` -- Docker Environmental Variables´ +- Docker Environmental Variables: ```bash docker run -d --name uls_etp-threat -ti \ --mount type=bind,source="/path/to/your/.edgerc",target="/opt/akamai-uls/.edgerc",readonly \ @@ -72,8 +75,11 @@ Using the dockerized approach, you have two different options to set up the opti akamai/uls ``` -Both of the above examples would do the exact same thing. -You can find a full set of command line parameters along with the according ENV variables in this document. +Both of the above examples would do the exact same thing: getting the Enterprise Threat Protector events part of the threat feed and push them into over `TCP` to the machine `10.10.10.10` on port `9091`. + +See the [full list of supported products and feeds](https://github.com/akamai/uls/blob/main/docs/LOG_OVERVIEW.md). You can then set `input` and `feed` argument from the example above. + +You can also find a full set of command line parameters along with the according ENV variables [in this document](ARGUMENTS_ENV_VARS.md). Right now, mounting the `.edgerc` file into the container is the only way applying the authentication. This might get fixed in some later version. Please change the `source=` according to your needs within the mount lines. diff --git a/docs/LOG_OVERVIEW.md b/docs/LOG_OVERVIEW.md index b9011fe..df69f66 100644 --- a/docs/LOG_OVERVIEW.md +++ b/docs/LOG_OVERVIEW.md @@ -1,24 +1,33 @@ # Log Overview -ULS supports ingestion of different log streams into SIEM. To get the highest value out of the ingested data, it is crucial to understand the delivered data. +ULS supports ingestion of different log streams into SIEM. + +To get the highest value out of the ingested data, it is crucial to understand the delivered data. + Here are some examples (per product) and links to additional information. ## Table of contents -- [Enterprise Application Access](#enterprise-application-access) - - [Access Logs (ACCESS)](#access-logs-access) - - [Admin Logs(ADMIN)](#admin-logs-admin) - - [Connector Health(CONHEALTH)](#connector-health-conhealth) -- [Enterprise Threat Protector](#etp) - - [Threat Logs](#threat-log-threat) - - [Accceptable Use Policy Logs (AUP)](#accceptable-use-policy-logs-aup) - - [DNS Logs](#dns) - - [Proxy Logs](#proxy) -- [Akamai MFA](#akamai-mfa) - - [Authentication Logs (AUTH)](#authentication-logs) - - [Policy Logs(POLICY)](#policy-logs) +- [Log Overview](#log-overview) + - [Table of contents](#table-of-contents) + - [Enterprise Application Access (EAA)](#enterprise-application-access-eaa) + - [Access Logs (ACCESS)](#access-logs-access) + - [Admin Logs (ADMIN)](#admin-logs-admin) + - [Connector Health (CONHEALTH)](#connector-health-conhealth) + - [Enterprise Threat Protector (ETP)](#enterprise-threat-protector-etp) + - [Threat Log (THREAT)](#threat-log-threat) + - [Accceptable Use Policy Logs (AUP)](#accceptable-use-policy-logs-aup) + - [DNS](#dns) + - [PROXY](#proxy) + - [Akamai MFA (MFA)](#akamai-mfa-mfa) + - [Authentication Logs (AUTH)](#authentication-logs-auth) -## Enterprise Application Access +## Enterprise Application Access (EAA) + +When configuring ULS to access EAA these feed, set `input` argument/variable to `EAA` and `feed` as indicated below in parathesis. + ### Access Logs (ACCESS) -Additional information regarding the log fields can be found on [here](https://learn.akamai.com/en-us/webhelp/enterprise-application-access/eaa-logs-from-eaa-api-and-splunk/GUID-8F07B320-2DD7-4035-9A8E-4E7435DFA3EA.html) + +Additional information regarding the log fields can be found on [here](https://techdocs.akamai.com/eaa/docs/data-feed-siem#access-logs) + ```json { "username": "user1", @@ -54,7 +63,8 @@ Additional information regarding the log fields can be found on [here](https://l ``` ### Admin Logs (ADMIN) -Additional information regarding the log fields can be found on [here](https://learn.akamai.com/en-us/webhelp/enterprise-application-access/eaa-logs-from-eaa-api-and-splunk/GUID-F772F01C-46D1-411C-A41F-D4B780D998FB.html). + +Additional information regarding the log fields can be found on [here](https://techdocs.akamai.com/eaa/docs/data-feed-siem#admin-logs). ```json { "datetime": "2021-07-23T05:54:40", @@ -67,7 +77,8 @@ Additional information regarding the log fields can be found on [here](https://l ``` ### Connector Health (CONHEALTH) -Additional information regarding the log fields can be found on [here](https://learn.akamai.com/en-us/webhelp/enterprise-application-access/eaa-logs-from-eaa-api-and-splunk/GUID-A79FBF43-DE2C-405A-8900-0D77DC8CEAF4.html) + +Additional information regarding the log fields can be found on [here](https://techdocs.akamai.com/eaa/docs/data-feed-siem#connector-health) ```json { "connector_uuid": "cht3_GEjQWyMW9LEk7KQfg", @@ -90,6 +101,7 @@ Additional information regarding the log fields can be found on [here](https://l ``` ## Enterprise Threat Protector (ETP) + ### Threat Log (THREAT) Additional information regarding the log fields can be found on [here](https://developer.akamai.com/api/enterprise_security/enterprise_threat_protector_reporting/v3.html#threatevent) ```json @@ -1925,44 +1937,40 @@ Additional information regarding the log fields can be found on [here](https://d ``` -## Akamai MFA -Additional information regarding the log fields can be found on [here](https://learn.akamai.com/en-us/webhelp/enterprise-mfa/akamai-mfa-logs-from-splunk-application/GUID-0F17296F-90F3-483E-AFDE-F98FBC51A8AC.html). +## Akamai MFA (MFA) + +Additional information regarding the MFA log fields can be found on [here](https://techdocs.akamai.com/mfa/docs/splunk-app). + ### Authentication Logs (AUTH) Authentication Events Example: ```json { - "uuid": "aud_JfNqdl6zS23456623434", - "created_at": "2021-03-23T19:36:20.047688", - "browser_ip": "49.103.18.124", - "app_id": "app_3IyJXh2345345345345f8X", - "device": "push", - "auth_method": "push", - "user_id": "user_6Hy1v241221541i5dv3", - "username": "mschiess", - "is_success": true, - "device_metadata": "Android", - "receipt": "", - "browser_type": "Chrome", - "browser_version": "88.0.4324", - "browser_os": "MacOS", - "browser_os_version": "10.15.7", - "device_os": "android", - "device_os_version": "10.0.0", - "browser_geo_location": "BANGALORE KA, IN", - "device_geo_location": "BANGALORE KA, IN", - "device_ip": "49.103.18.124" + "uuid": "aud_JfNqdl6zSByrU0ovrbJ6m", + "created_at": "2021-03-23T19:36:20.047688", + "browser_ip": "49.207.58.115", + "app_id": "app_3IyJXh2U9Jiws6bvxcf8X", + "app_name": "Test Application", + "device": "push", + "auth_method": "push", + "user_id": "user_6Hy1v24DZIr8b0UHYi5dv3", + "username": "username", + "is_success": true, + "device_metadata": "Android", + "receipt": "", + "browser_type": "Chrome", + "browser_version": "88.0.4324", + "browser_os": "MacOS", + "browser_os_version": "10.15.7", + "device_os": "android", + "device_os_version": "10.0.0", + "browser_geo_location": "BANGALORE KA, IN", + "device_geo_location": "BANGALORE KA, IN", + "device_ip": "49.207.58.115", + "denial_type": null, + "device_id": "device_3kbTGOPbHxH3KfYkPzm31e", + "policy_attr_name": null, + "policy_uuid": null, + "principal_type": null, + "principal_uuid": null } ``` - -### Policy Logs (POLICY) -Policy Denied Events Example: -```json -{ - "id": "aud_5mRypRCa3456789VJt", - "created_at": "2021-03-23T17:20:50.524672", - "user_id": "user_3CbCStOKG0uGdjRILocuxW", - "principal_id": "Tenant", - "policy_id": "policy_5iMncPFO2345678QL4j", - "policy_attribute_name": "Existing User" -} -``` \ No newline at end of file diff --git a/docs/MONITORING.md b/docs/MONITORING.md index ebdfe1b..7d4c626 100644 --- a/docs/MONITORING.md +++ b/docs/MONITORING.md @@ -23,11 +23,11 @@ The output is delivered in JSON format {"dt": "2021-06-09T08:15:35.092889", "uls_product": "ETP", "uls_feed": "THREAT", "uls_outpout": "HTTP", "uls_runtime": 300, "event_count": 504, "event_rate": 1.68, "mon_interval": 300} ``` -## Send docker logs to splunk -For this we're using the embedded docker - splunk logging module. +## Send Docker logs to Splunk +For this we're using the embedded docker - Splunk logging module. ### Docker-compose -Example (add to every service in your docker-compose.yml) +Example (add to every service in your `docker-compose.yml`) ```yaml version: "3.0" ... @@ -67,8 +67,8 @@ services: ... ``` -More splunk - options for docker can be found [here](https://docs.docker.com/config/containers/logging/splunk/) -Sidenote: you will still receive logs on the cli running `docker-compose logs -f uls-tool` +More Splunk - options for docker can be found [here](https://docs.docker.com/config/containers/logging/splunk/) +Sidenote: you will still receive logs on the CLI running `docker-compose logs -f uls-tool` ![Docker logs in splunk](images/uls_docker_logs_to_splunk.png)