Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Complete overhaul of README and integration into Wiki #61

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open

Complete overhaul of README and integration into Wiki #61

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
0d10a42
Readme - CentOS Install Instructions Update
langhorne Feb 1, 2017
543df21
README.md
askkemp Feb 10, 2017
a3ec769
Overhauled README and connected to wiki
askkemp Jul 28, 2017
bf0efa8
Update README.md
askkemp Jul 29, 2017
b7d6ff1
Update README.md
askkemp Jul 29, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
218 changes: 48 additions & 170 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,15 @@
# Laika BOSS: Object Scanning System

Laika BOSS is a versatile file-centric scanner and intrusion detection system.

## Documentation

See the [Wiki](https://github.com/lmco/laikaboss/wiki) for documentation, examples, and other useful information.

Read the ***[whitepaper](http://lockheedmartin.com/content/dam/lockheed/data/isgs/documents/LaikaBOSS%20Whitepaper.pdf)*** "Laika BOSS: Scalable File-Centric Malware Analysis and Intrusion Detection System"

## Overview

Laika is an object scanner and intrusion detection system that strives to achieve the following goals:

+ **Scalable**
Expand All @@ -14,6 +24,7 @@ Laika is an object scanner and intrusion detection system that strives to achiev
+ **Verbose**
+ Generate more metadata than you know what to do with


Each scan does three main actions on each object:

+ **Extract child objects** Some objects are archives, some are wrappers, and others are obfuscators. Whatever the case may be, find children objects that should be scanned recursively by extracting them out.
Expand All @@ -22,204 +33,71 @@ Each scan does three main actions on each object:

+ **Add metadata** Discover as much information describing the object for future analysis.

**Feel free to read the [whitepaper](http://lockheedmartin.com/content/dam/lockheed/data/isgs/documents/LaikaBOSS%20Whitepaper.pdf)!**

## Components

Laika is composed of the following pieces:

+ **Framework** (`laika.py`) This is the core of Laika BOSS. It includes the object model and the dispatching logic.

+ **laikad** This piece contains the code for running Laika as a deamonized, networked service using the ZeroMQ broker.

+ **cloudscan** A command-line client for sending a local system file to a running service instance of Laika (laikad).

+ **modules** The scan itself is composed of the running of modules. Each module is its own program that focuses on a particular sub-component of the overall file analysis.


## Getting Started

Laika BOSS has been tested on the latest versions of CentOS and Ubuntu LTS

### Installing on Ubuntu

1. Install framework dependencies:

```shell
apt-get install yara python-yara python-progressbar python-pip
pip install interruptingcow
```

2. Install network client and server dependencies:
## Example Use Cases
The best way to introduce Laika BOSS is to give several examples of its use.

```shell
apt-get install libzmq3 python-zmq python-gevent python-pexpect
```
In [example one](https://github.com/lmco/laikaboss/wiki/Use-Cases-and-Examples#emailattachment), you feed Laika an email with a Office document (OLE) attachment. Laika will parse the contents of the email and extract all of the message objects. In this case, it extracts a plain text object, an HTML object, and an Office Word attachment. Before moving on, it generates metadata about the email (e.g. email addreses, IPs, domains, etc.). Next Laika moves on and determines that the Word document is in OLE format so it extracts the OLE streams. In one one of the streams, a VBA macro is discoverd so Laika extracts that too. All objects feed into and extracted by Laika are scanned by Yara and ClamAV. The conclusion is an output of the scan results and collected metadata in JSON format. Optionally, Laika will place the extracted contents into a folder for manual review.

3. Install module dependencies:

```shell
apt-get install python-ipy python-m2crypto python-pyclamd liblzma5 libimage-exiftool-perl python-msgpack libfuzzy-dev python-cffi python-dev unrar
pip install fluent-logger olefile ssdeep py-unrar2 pylzma javatools
wget https://github.com/smarnach/pyexiftool/archive/master.zip
unzip master.zip
cd pyexiftool-master
python setup.py build
python setup.py install
wget https://github.com/erocarrera/pefile/archive/pefile-1.2.10-139.tar.gz
tar vxzf pefile-1.2.10-139.tar.gz
cd pefile-1.2.10-139
python setup.py build
python setup.py install
```

### Installing on CentOS

1. Install framework dependencies

```shell
sudo yum install -y epel-release
sudo yum install -y autoconf automake libtool libffi-devel python-devel python-pip python-zmq ssdeep-devel swig
```

2. Install Python modules
```
+------------------------------------------+
| EMAIL ---> Text |
| ---> HTML | output +-------------------------------+
| ---> OLE ---> stream 1 | -------> | Logged scan results (JSON) |
| ---> stream 2 | | Extracted objects (optional) |
| ---> stream 3 ---> macro | +-------------------------------+
| ---> stream 4 |
+------------------------------------------+
```

```shell
pip install IPy cffi interruptingcow fluent-logger javatools m2crypto olefile pylzma pyclamd py-unrar2
pip install six --upgrade --force-reinstall
pip install ssdeep
```
In [example two](https://github.com/lmco/laikaboss/wiki/Use-Cases-and-Examples#rtfzip), you feed Laika a ZIP file. Laika extracts the single item from the ZIP file. It determines that the extracted item is an RTF. It extracts all of the embedded objects from the RTF of which one is an EXE. Liaka collects metadata on the EXE. The conclusion is an output of the scan results and collected metadata in JSON format. Optionally, Laika will place the extracted contents into a folder for manual review.

3. Install Yara
```
+-----------------------------------------------+ output +-------------------------------+
| ZIP ---> RTF ---> embedded object 1 ---> exe | -------> | Logged scan results (JSON) |
+-----------------------------------------------+ | Extracted objects (optional) |
+-------------------------------+
```

There is no Yara package for CentOS, so we have to build it from source. You can't use a checkout from Github as it won't contain the Python code; you must download one of the [release versions](https://github.com/virustotal/yara/releases). The following uses Yara version 3.5.0
For detailed use cases, please see the [Wiki](https://github.com/lmco/laikaboss/wiki/Use-Cases-and-Examples#usecases).

```shell
wget https://github.com/VirusTotal/yara/archive/v3.5.0.zip
unzip yara-3.5.0.zip
cd yara-3.5.0
chmod +x ./build.sh
./build.sh
sudo make install
cd yara-python
python setup.py build
sudo python setup.py install
```
## Components

4. Install pyexif
Laika is composed of the following pieces:

```shell
wget https://github.com/smarnach/pyexiftool/archive/master.zip
unzip master.zip
python setup.py build
sudo python setup.py install
```
+ **[Framework](https://github.com/lmco/laikaboss/wiki/Use-Cases-and-Examples#standalone)** (`laika.py`) This is the core of Laika BOSS. It includes the object model and the dispatching logic.

5. Install pefile
+ **[laikad](https://github.com/lmco/laikaboss/wiki/Use-Cases-and-Examples#network)** (`laikad.py`) This piece contains the code for running Laika as a deamonized, networked service using the ZeroMQ broker.

```shell
wget https://github.com/erocarrera/pefile/archive/pefile-1.2.10-139.tar.gz
tar vxzf pefile-1.2.10-139.tar.gz
cd pefile-1.2.10-139
python setup.py build
python setup.py install --user
```
+ **[cloudscan](https://github.com/lmco/laikaboss/wiki/Use-Cases-and-Examples#network)** (`cloudscan.py`) A command-line client for sending a local system file to a running service instance of Laika (laikad).

You may need to set the `LD_LIBRARY_PATH` variable to include `/usr/local/lib` when running Laika.
+ **[modules](https://github.com/lmco/laikaboss/wiki/Scanning-Module-List)** The scan itself is composed of the running of modules. Each module is its own program that focuses on a particular sub-component of the overall file analysis.

### Installing Laika BOSS (optional)
+ **[milter](https://github.com/lmco/laikaboss/wiki/Install-Instructions:--Milter)** (`laikamilter.py`) Optionally, integrate Laika BOSS with mail transfer agents such as Sendmail or Postfix

You may use the provided setup script to install the Laika BOSS framework, client library, modules and associated scripts (`laika.py`, `laikad.py`, `cloudscan.py`).
+ **[Suricata Integration Prototype](https://github.com/lmco/laikaboss/wiki/Install-Instructions:--Suricata-Integration-Prototype)** (`laika_redis_client.py`) Optionally, extract files from Redis and submit them to Laika BOSS for scanning.

```shell
python setup.py install
```
## Getting Started

#### Standalone instance

From the directory containing the framework code, you may run the standalone scanner, `laika.py` against any file you choose. If you move this file from this directory you'll have to specify various config locations. By default it uses the configurations in the `./etc` directory.

We recommend using installing [jq](http://stedolan.github.io/jq/) to parse Laika output.

```javascript
$ ./laika.py ~/test_files/testfile.cws.swf | jq '.scan_result[] | { "file type" : .fileType, "flags" : .flags, "md5" : .objectHash }'
100%[############################################] Processed: 1/1 total files (Elapsed Time: 0:00:00) Time: 0:00:00
{
"md5": "dffcc2464911077d8ecd352f3d611ecc",
"flags": [],
"file type": [
"cws",
"swf"
]
}
{
"md5": "587c8ac651011bc23ecefecd4c253cd4",
"flags": [],
"file type": [
"fws",
"swf"
]
}
```
### Installation Instructions
Laika BOSS has been tested on the latest versions of CentOS, Fedora, and Ubuntu LTS

#### Networked instance

```javascript
$ ./laikad.py

$ ./cloudscan.py ~/test_files/testfile.cws.swf | jq '.scan_result[] | { "file type" : .fileType, "flags" : .flags, "md5" : .objectHash }'
{
"md5": "dffcc2464911077d8ecd352f3d611ecc",
"flags": [],
"file type": [
"cws",
"swf"
]
}
{
"md5": "587c8ac651011bc23ecefecd4c253cd4",
"flags": [],
"file type": [
"fws",
"swf"
]
}
```
Full instructions are available in the [Wiki](https://github.com/lmco/laikaboss/wiki)

#### Milter
#### Milter Integration

The Laika BOSS milter server allows you to integrate Laika BOSS with mail transfer agents such as Sendmail or Postfix. This enables better visibility (passive visibility can be hampered by TLS) and provides a means to block email according to Laika BOSS disposition.

```
+----------------+ +---------------+ +----------------+
| | email | | email | |
| sendmail +-------------> laikamilter +-------------> laikad |
| | accept/deny | | scan result | |
| <-------------+ <-------------+ |
+----------------+ +---------------+ +----------------+
```
For more details, please see the [Wiki](https://github.com/lmco/laikaboss/wiki/Install-Instructions:--Milter).

The Laika BOSS milter server requires the [python-milter](https://pythonhosted.org/milter) module and the Laika BOSS client library. Check out the comments in the source code for more details.

#### Suricata Integration Prototype

We have released a proof of concept feature for Suricata that allows it to store extracted files and their associated metadata in a Redis database. You will find this code under a [new branch](https://github.com/lmco/suricata/tree/file_extract_redis_prototype_v1) in our Suricata fork. We hope to refine the implementation and eventually have it accepted by the project.

Once you've enabled file extraction and the optional Redis integration in Suricata, you can extract these files from Redis and submit them to Laika BOSS for scanning by using the middleware script `laika_redis_client.py` as shown below. Note that it requires the `python-redis` module.
Once you've enabled file extraction and the optional Redis integration in Suricata, you can extract these files from Redis and submit them to Laika BOSS for scanning by using the middleware script `laika_redis_client.py.`

First, start `laikad.py` in async mode:

```shell
./laikad.py -a
```

Then launch the middleware script and give it the address of the `laikad` broker and Redis database (defaults shown below):

```shell
./laika_redis_client.py -b tcp://localhost:5558 -r localhost -p 6379
```
For more details, please see the [Wiki](https://github.com/lmco/laikaboss/wiki/Install-Instructions:--Suricata-Integration-Prototype).

Note that you will need to use a logging module such as `LOG_FLUENT` to export the full scan result of the these file scans from `laikad`.

## Licensing

Expand Down