azure-collector

Alert Logic Office 365 Log Collector

Overview

This repository contains Azure Web application Node.js source code and an ARM template for setting up a data collector in Azure which will collect and forward Office 365 log data to the Alert Logic Cloud Defender Log Manager (LM) feature.

Installation

Installation requires the following steps:

1. Register a new O365 web application in O365 portal for collecting O365 logs.
2. Set up the required Active Directory security permissions for the application to authorize it to read threat intelligence data and activity reports for your organization.
3. Create an Access Key that will allow the application to connect to the Alert Logic Cloud Defender and Cloud Insight backend.
4. Download and deploy a custom ARM template to Microsoft Azure to create functions for collecting and managing O365 log data
5. Verify that installation was successful using Alertlogic CloudDefender UI.

Register a New O365 Web Application in O365

In order to install O365 Log collector:

1. Log into O365 portal as AD tenant administrator.

2. Go to Setup and Domain and make a note of O365 domain name to collect logs from, for example, example.onmicrosoft.com.

3. Navigate to Admin Centers and Azure AD.

4. On the left side panel click Azure Active Directory and App Registrations.

5. Click +New application registration, fill in configuration parameters and click Create:

   • Name - for example alo365collector.
   • Select Web app/ API as Application type.
   • In Sign-on URL enter some URL, for example http://alo365collector.com. Note, it is not used anywhere.

6. After application is created select it and make a note of Application Id, for example, a261478c-84fb-42f9-84c2-de050a4babe3

Set Up the Required Active Directory Security Permissions

1. On the Settings panel and select Required permissions and click +Add
2. Hit Select an API and chose Office 365 Management APIs, click Select
3. In Application permissions select Read service health information for your organization, Read activity data for your organization, Read threat intelligence data for your organization and Read activity reports for your organization. Click Select and Done buttons.
4. On Required permissions panel click Required permissions button and confirm the selection. Note, only AD tenant admin can grant permissions to an Azure AD application.
5. On the Settings panel of the application and select Keys.
6. Enter key Description and Duration and click Save. Note, please save the key value, it is needed later during template deployment.
7. Save the Application ID and Service Principal Id for use below. To get the Service Principal Id, navigate to the Registered App blade, click on the link under Managed application in local directory. Then click Properties. The Service Principal Id is labeled Object ID on the properties page. Caution This is not the same Object ID listed in the Properties blade reached by clicking Settings or All Settings from the Registered app. It is also not the Object ID shown on the Registered app blade itself.

Create an Alert Logic Access Key

Login and get an authentication token from the Alert Logic Cloud Insight product AIMS API. From the command line use curl as follows (where <user> is your CloudInsight user and <pwd> is your CloudInsight password):

curl -X POST -v -u '<user>:<pwd>' https://api.global-services.global.alertlogic.com/aims/v1/authenticate

Make a note of the following fields returned in the response:

• USER ID
• ACCOUNT ID
• TOKEN

Use the authentication token returned in the response to create access keys for the Azure application deployed in the next section. Issue the following curl command (where <TOKEN> is the auth token, <ACCOUNT_ID> is the account ID, and <USER_ID> is the user ID returned above):

curl -X POST -H "x-aims-auth-token: <TOKEN>" https://api.global-services.global.alertlogic.com/aims/v1/<ACCOUNT_ID>/users/<USER_ID>/access_keys

An example of a successful response is:

{"access_key_id":"8bb72b92eb43f0bb","secret_key":"15b74g7fe32b9a0083172b724fa8b84aebfa9325bba2b3bc8933216013e220c1"}

Make a note of the access_key_id and secret_key values for use in the deployment steps below.

Note: Only five access keys can be created per user. If you get a "limit exceeded" response you will need to delete some keys in order to create new ones. Use the following command to delete access keys:

curl -X DELETE -H "x-aims-auth-token: <TOKEN>" https://api.global-services.global.alertlogic.com/aims/v1/<ACCOUNT_ID>/users/<USER_ID>/access_keys/<ACCESS_KEY_ID>

Function deployment

Log into Azure portal. Note, In order to perform steps below you should have an active Azure subscription, to find out visit Azure subscriptions blade

Deploy via the Custom ARM Template in an Azure Subscription

1. Download an ARM template
2. Go to Customer Deployment page. Type in deploy in a search query located on top of Azure Web UI and select Deploy a custom template.
3. Click Build your own template in the editor and load the file previously downloaded on step 1 above.
4. Click Save button.
5. Fill in required template parameters and click the Purchase button to start a deployment. I.e.:
   • Name - Any name
   • Storage Name - Any Storage Account name (that does not currently exist)
   • Alert Logic Access Key ID - access_key_id returned from AIMs above
   • Alert Logic Secret Key - secret_key returned from AIMs above
   • Alert Logic API endpoint - usually api.global-services.global.alertlogic.com
   • Alert Logic Data Residency - usually default
   • Office365 Content Streams - The list of streams you would like to collect. Valid values are:
     • ["Audit.AzureActiveDirectory","Audit.Exchange","Audit.SharePoint","Audit.General", "DLP.All"]
   • Office365 Tenant ID - The GUID of the tenant e.g. alazurealertlogic.onmicrosoft.com
   • Service Principal ID - The Object ID of the application that created the subscription. You can obtain it from Azure -> AD -> App registrations -> Your app name -> Link under Managed application in local directory -> Properties -> Object ID
   • App Client ID - The GUID of your application that created the subscription. You can obtain it from Azure -> AD -> App registrations -> Your app name
   • App Client Secret - The secret key of your application from App Registrations
   • Repository URL - must be https://github.com/alertlogic/azure-collector.git
   • Repository Branch - should usually be master

Deploy via Azure CLI

You can use either Azure Cloud Shell or local installation of Azure CLI.

1. Create a resource group with name "AlertLogicCollect" in location "Central US" by executing following command

   az group create --name AlertLogicCollect --location "Central US"
   

2. Once created go to Resource groups blade and select the resource group.
3. Select Access Control (IAM) and add Website Contributor role to AD application identity created above.
4. Deploy a template by using following command, during its execution enter required parameters when asked

   az group deployment create \
       --name AlertLogicCollector \
       --resource-group AlertLogicCollect \
       --template-uri "https://raw.githubusercontent.com/alertlogic/azure-collector/master/template.json"
   

Wait until it is deployed successfully.

Verify the Installation

1. Go to Azure Apps and choose your function. The last log under Functions-> Master-> Monitor should have OK status and should not contain any error messages.
2. Log into Alertlogic CloudDefender and navigate into Log Manager -> Sources page. Check new O365 log source (with a name provided during az group deployment create above) has been created and source status is ok.

How It Works

Master Function

The Master function is a timer trigger function which is responsible for:

• registering the Azure web app In Alertlogic backend;
• reporting health-checks to the backed;
• performing log source configuration updates, which happen via Alertlogic UI.

Note: When releasing a new version of the collector please remember to increment the version number in npm package.json file. To display the current version locally, issue npm run local-version

Updater Function

The Updater is a timer triggered function runs deployment sync operation every 12 hours in order to keep entire Web application up to date.

Collector Function

The Collector function exposes an HTTP API endpoint https://<app-name>/o365/webhook which is registered as an Office 365 webhook and processes O365 activity notifications. Below is a notification example,

[
  {
    "contentType": "Audit.AzureActiveDirectory",
    "contentId": "20170721121608709004422$20170721121608709004422$audit_azureactivedirectory$Audit_AzureActiveDirectory$IsFromNotification",
    "contentUri": "https://manage.office.com/api/v1.0/bf8d32d3-1c13-4487-af02-80dba2236485/activity/feed/audit/20170721121608709004422$20170721121608709004422$audit_azureactivedirectory$Audit_AzureActiveDirectory$IsFromNotification",
    "notificationStatus": "Succeeded",
    "contentCreated": "2017-07-21T12:16:56.798Z",
    "notificationSent": "2017-07-21T12:16:56.798Z",
    "contentExpiration": "2017-07-28T12:16:08.709Z"
  },
  {
    "contentType": "Audit.AzureActiveDirectory",
    "contentId": "20170721121625590007449$20170721121625590007449$audit_azureactivedirectory$Audit_AzureActiveDirectory$IsFromNotification",
    "contentUri": "https://manage.office.com/api/v1.0/bf8d32d3-1c13-4487-af02-80dba2236485/activity/feed/audit/20170721121625590007449$20170721121625590007449$audit_azureactivedirectory$Audit_AzureActiveDirectory$IsFromNotification",
    "notificationStatus": "Succeeded",
    "contentCreated": "2017-07-21T12:16:56.798Z",
    "notificationSent": "2017-07-21T12:16:56.798Z",
    "contentExpiration": "2017-07-28T12:16:25.590Z"
  }
]

A notification contains a link to the actual data which is retrieved by the Collector, wrapped into a protobuf structure TBD link and is sent into Alertlogic Ingest service.

Local Development

1. Clone repo git clone [email protected]:alertlogic/azure-collector.git
2. cd azure-collector
3. Run ./local_dev/setup.sh
4. Edit ./local_dev/dev_config.js
5. Run Master function locally: npm run local-master
6. Run Updater function locally: npm run local-updater
7. Run O365WebHook function locally: npm run local-o365webhook
8. Run npm test in order to perform code analysis and unit tests.

Please use the following code style as much as possible.

Setting environment in dev_config.js

• process.env.APP_TENANT_ID - The GUID of the tenant i.e. 'alazurealertlogic.onmicrosoft.com'
• process.env.CUSTOMCONNSTR_APP_CLIENT_ID - The GUID of your application that created the subscription. You can obtain it from Azure -> AD -> App registrations -> Your app name
• process.env.CUSTOMCONNSTR_APP_CLIENT_SECRET - A secret key of your application from App Registrations.
• process.env.CUSTOMCONNSTR_APP_CI_ACCESS_KEY_ID - access key returned from AIMs above.
• process.env.CUSTOMCONNSTR_APP_CI_SECRET_KEY- secret key returned from AIMs above.

Known Issues/ Open Questions

• Sometimes deployments fail after siteSync action. We need better updater to handle that in order not to wait for 12 hours for the next update attempt.

Useful Links

• Node.js static code analysis tool
• How to monitor Azure functions?
• Server application to Web API authentication.
• Office 365 Management API reference.
• Azure Web apps API reference