Merge pull request #24 from chkp-eyalit/code-cleanup

Major refactor - v2.0.0

.gitignore

@@ -1,40 +1,48 @@
 # Compiled scout
-scout/*.o
-scout/*.S
-scout/*.bin
-scout/*.elf
-scout/arc/*.o
-scout/arc/*.S
-scout/loaders/*.o
-scout/loaders/*.S
-scout/pic/*.o
-scout/pic/*.S
+src/scout/*.o
+src/scout/*.S
+src/scout/*.bin
+src/scout/*.elf
+src/scout/arc/*.o
+src/scout/arc/*.S
+src/scout/loaders/*.o
+src/scout/loaders/*.S
+src/scout/pic/*.o
+src/scout/pic/*.S
+src/scout/Scout.vpj
+src/scout/Scout.vpw
+src/scout/Scout.vpwhist
+src/scout/Scout.vtg
 
 # Compiled embedded scout
-embedded_scout/*.o
-embedded_scout/*.S
-embedded_scout/*.bin
-embedded_scout/*.elf
+examples/embedded_scout/*.o
+examples/embedded_scout/*.S
+examples/embedded_scout/*.bin
+examples/embedded_scout/*.elf
 
-# Compiled kernel scoutkernel_scout/driver/*.order
-kernel_scout/driver/*.symvers
-kernel_scout/driver/*.o
-kernel_scout/driver/*.ko
-kernel_scout/user_mode/*.o
-kernel_scout/user_mode/*.S
-kernel_scout/user_mode/*.bin
-kernel_scout/user_mode/*.elf
+# Compiled kernel scoutexamples/kernel_scout/driver/*.order
+examples/kernel_scout/driver/*.symvers
+examples/kernel_scout/driver/*.o
+examples/kernel_scout/driver/*.ko
+examples/kernel_scout/user_mode/*.o
+examples/kernel_scout/user_mode/*.S
+examples/kernel_scout/user_mode/*.bin
+examples/kernel_scout/user_mode/*.elf
 
 # Compiled tests
 tests/exploit_me_32
 tests/exploit_me_64
 
 # Compiled python modules
-utils/*.pyc
-manager/*.pyc
+src/utils/*.pyc
 
 # Generated docs
 docs/_build/*
 docs/_build/*/*
 docs/_build/*/*/*
-docs/_build/*/*/*/*
+docs/_build/*/*/*/*
+
+# Python package
+build/
+dist/
+scout_debugger.*

CONTRIBUTING.md

@@ -0,0 +1,43 @@
+How to contribute
+=================
+First of all, thank you for choosing to contribute to our project. This short guide will describe the required coding conventions, as well as the built-in tests that we use in our project. Following these guidelines will help us merge your pull request into our code base in a fast as smooth manner.
+
+Submitting changes
+------------------
+Please send us a GitHub Pull Request with a clear list of what you've done. Make sure to include a clear log message for your commits, describing the modifications / additions to the code base, and their implications.
+
+Reporting a bug
+---------------
+In order to help us close the bug as quickly as possible, please follow these steps:
+1. Make sure there isn't an open issue that already addresses this bug
+2. If there isn't, open a new issue and attach as many informative details as you can:
+      * Architecture & compilation flags in which the bug occurs
+      * Trace + logs, describing the bug / exception
+      * As many details as you can in order to help us reproduce and fix this issue
+ 3. If you already have a fix, please submit it as a pull request, and include the bug details in it's description
+
+Coding Conventions
+------------------
+Start reading our code and we believe you'll get the hang of it.
+The important notes are:
+* Every function should be documented in a manner that is consistent with the current documentation standard
+* Each indentation level should be 4 spaces in width (spaces, not tabs)
+* We believe that comments improve the readability, make sure that your code is documented enough to be understood by other developers
+
+Testing
+-------
+**Scout** uses the following tools to enforce coding conventions and to help eliminate common python bugs:
+* [pydocstyle](http://www.pydocstyle.org/en/2.1.1/usage.html)
+* [flake8](https://pypi.org/project/flake8/)
+
+Testing can be done from the project's home directory:
+1. Testing for **pydocstyle**:
+```Scout> python tests.py```
+2. Testing for **flake8**:
+```Scout> flake8 src```
+
+One last note
+-------------
+We believe that the only way that Open Source tools will help the infosec community in the long term, is to maintain these tools and to make sure they are developed according to the community standards. Each contribution brings as one step further to this goal.
+
+Thanks :)

README.md

@@ -51,13 +51,18 @@ https://scout-debugger.readthedocs.io/
 * Any Posix-like operating system (Embedded Mode)
 
 ## Folder Structure
-* **docs:** Useful tutorials regarding each unique module of the debugger, including documentation of the API used for custom extensions
-* **embedded_scout:** Example project for an embedded debugger scenario, i.e. a debugger that is injected into the address space of a debuggee firmware
-* **kernel_scout:** Linux kernel driver-based debugger, including a proxy user mode process used for transparent network access
-* **manager:** Python layer for communicating with the debuggee (usually over a TCP connection)
-* **scout:** C code of the basic scout debugger
-* **tests:** A testing utility for PIC based debuggers
-* **utils:** Useful python compilation scripts
+* **docs:** Documentation files that generated the read-the-docs that was linked above
+* **examples:**
+  * **embedded_scout** - Use case example for an "Embedded Mode" compilation
+  * **kernel_scout** - Use case example for a Linux "Kernel Mode" compilation
+* **src**
+  * **scout** - Source code for the debugger (core of the server side)
+  * **utils** - Python compilation scripts and network API for the client/server
+* **tests:** A simple exploit_me.c for checking PIC compiled binaries
+
+## Installation
+* Installing the python package: ```python3 setup.py install```
+* Dedicated compilers: A list of compilers per-architecture is found on ```compilers.txt```
 
 ## Credits
 This projects combines together design and compilation tricks that I learned from many fellow researchers during the years.
@@ -73,4 +78,4 @@ Scout was developed and used in our following research projects:
 ## Contact
 Eyal Itkin (eyalit at checkpoint dot com)
 
-[@EyalItkin](https://twitter.com/EyalItkin)
+[@EyalItkin](https://twitter.com/EyalItkin)

compilers.txt

@@ -0,0 +1,5 @@
+Used compilers:
+===============
+1. GCC compilation to 32 bits on 64 bits linux machines ==> sudo apt-get install gcc-multilib libc6-dev-i386
+2. ARM  cross compiler toolchain for Linux ==> sudo apt-get install gcc-arm-none-eabi
+3. MIPS cross compiler toolchain for Linux ==> sudo apt-get install gcc-mips-linux-gnu

docs/Adding Custom Instructions.md

@@ -4,9 +4,9 @@ Being an instruction-based debugger, Scout supports project extensions.
 
 Registration - C Code
 ---------------------
-* Each of the instructions that are added by the project, should be registerred by calling ```register_instruction()```.
-* The registration should take place inside the function ```register_specific_instructions()```.
-* This design makes sure that when invoking ```register_all_instructions()```, all of the default instructions, and extension instruction, will be registerred correctly.
+* Each of the instructions that are added by the project, should be registerred by calling ``register_instruction()``.
+* The registration should take place inside the function ``register_specific_instructions()``.
+* This design makes sure that when invoking ``register_all_instructions()``, all of the default instructions, and extension instruction, will be registerred correctly.
 
 Implementation - C Code
 -----------------------
@@ -15,15 +15,15 @@ In order to implement a new instruction, one should define each of the required
 * Instruction ID - must be unique, but not necessarily consecutive
 * Minimal Length - minimal amount of bytes needed for a valid instruction (robustness checks)
 * Maximal Length - maximal amount of bytes needed for a valid instruction (robustness checks)
-* Instruction handler - a handler function with a fixed signature of: ```int32_t (*instrHandler)(void * ctx, uint8_t * instruction, uint32_t length)```
+* Instruction handler - a handler function with a fixed signature of: ``int32_t (*instrHandler)(void * ctx, uint8_t * instruction, uint32_t length)``
 
-**Note:** The instructions are stored in a global array with a **fixed** capacity. When adding new instructions, one should make sure to adjust this capacity accordingly.
-The capacity is defined in ```scout_api.h``` and is set by default to ```#define SCOUT_MAX_INSTRS   (10)```.
+**Note:** The instructions are stored in a global array with a **fixed** capacity. When adding new instructions, one should make sure to adjust this capacity accordingly (both in the C and .py files).
+The capacity is defined in ``scout_api.h`` and is set by default to ``#define SCOUT_MAX_INSTRS   (10)``.
 
 Examples - C Code
 -----------------
-* Embedded mode (```embedded_scout```) - files ```project_instructions``` (*.c and *.h)
-* Linux Kernel mode (```kernel_scout```) - files ```driver\scout_kernel_instructions``` (*.c and *.h)
+* Embedded mode (``embedded_scout``) - files ``project_instructions`` (*.c and *.h)
+* Linux Kernel mode (``kernel_scout``) - files ``driver\scout_kernel_instructions`` (*.c and *.h)
 
 Client Side - Python Code
 -------------------------
@@ -34,4 +34,5 @@ In the client side, adding a new instructions is even easier, and requires only
 
 Examples - Python Code
 ----------------------
-* Linux Kernel example (```manager```) - file ```kernel_scout_api.py```
+* Embedded example (``manager``) - file ``embedded_scout_api.py``
+* Linux Kernel example (``manager``) - file ``kernel_scout_api.py``

docs/Compilation Modes.md

@@ -1,15 +1,19 @@
 Compilation Modes
 =================
-Scout is a configurable debugger, that could bedeployed in several different environments:
+Scout is a configurable debugger, that could be deployed in several different environments:
 
 * Linux User-Mode Process - "User Mode"
 * Linux Kernel Driver - "Kernel Mode"
 * Linux In-Process Debugging - "User Mode" + "PIC Mode"
-* Embedded "In-Process" Debugging - "PIC Mode" + "Embedded Mode"
+* Embedded "In-Process" Debugging (low privileges) - "PIC Mode" + "User Mode"
+* Embedded "In-Process" Debugging (high privileges) - "PIC Mode" + "Kernel Mode"
 
 To decide what will be the suitable compilation mode / architecture flags, one should check the following parameters.
 Each of the defined parameters is a C MACRO (define) that controls the behavior (and compilation) of the resulting binary.
 
+**Important Note:**
+When using ``scoutCompiler``, it will automatically generate most of the needed flags by deducing the right values based on the architecture and configuration flags that are supplied to it. Please see the ``embedded_scout`` example for more info.
+
 Target Endianness
 -----------------
 * SCOUT_BIG_ENDIAN - Scout is executed on a Big Endian architecture
@@ -36,48 +40,51 @@ Only one of above flags can be defined.
 If none are defined the base library will define "SCOUT_ARCH_INTEL" on it's own.
 
 **Additional Flags:**
-* SCOUT_ARM_THUMB - Scout will be executed on an ARM cpu in Thumb mode. Can only be used together with the "SCOUT_ARCH_ARM" flag.
+SCOUT_ARM_THUMB - Scout will be executed on an ARM cpu in Thumb mode. Can only be used together with the "SCOUT_ARCH_ARM" flag.
 
-The flags is needed only in PIC mode, in which we use inline assembly.
+The flags are needed only in PIC mode, in which we use inline assembly.
 
 Target Permission Level
 -----------------------
-* SCOUT_MODE_USER - Scout is executed in User-Mode
-* SCOUT_MODE_KERNEL - Scout is executed in Kernel-Mode
+* SCOUT_MODE_USER - Scout is executed in User-Mode (& low CPU privileges)
+* SCOUT_MODE_KERNEL - Scout is executed in Kernel-Mode (& high CPU privileges)
 
 Only one of above flags can be defined.
 If none are defined the base library will define "SCOUT_MODE_USER" on it's own.
 
-**Note:** These flags are used only in a Linux PC Environment, and are not used in an Embedded Environment.
-
-Target Loading Environment
---------------------------
-* SCOUT_PC_ENV - Scout is executed as a standard process (user) or driver (kernel) on a Linux machine
-* SCOUT_EMBEDDED_ENV - Scout is injected to the address space of a given executable
-
-Only one of above flags can be defined.
-If none are defined the base library will define "SCOUT_PC_ENV" on it's own.
-
-**Note:** SCOUT_EMBEDDED_ENV has many use cases, including:
-1. Injecting a debugger into a debuggee Linux process
-2. Injecting a debugger into a debuggee firmware (if the executable's API matches the basic POSIX based API of Scout)
+"SCOUT_MODE_KERNEL" will also be the right choice for an RTOS (Real-time OS) in which every task / our task has high privileges. The flag will lead to the definition of "SCOUT_HIGH_PRIVILEGES" by the compilation environment.
 
-**Note:** At the current moment, "SCOUT_EMBEDDED_ENV" must be used with "SCOUT_PIC_CODE", although in the future a linker script could help an embedded scout access external functions without the PIC context.
+**Important Note:** As flushing the CPU cache usually requires high privileges, the per-architecture implementation will only be available if compiling using the "SCOUT_MODE_KERNEL" flag.
 
 Position Independent Mode - SCOUT_PIC_CODE
 ------------------------------------------
 Scout will be compiled for full Position Independent Code (PIC) mode. Any access to an external function / global variable will pass through a unique "Context" object. Read the section about "PIC Compilation" for more information.
-**Note:** Can only be used with "SCOUT_EMBEDDED_ENV".
+
+"SCOUT_PIC_CODE" will lead to the definition of "SCOUT_ISOLATED_ENV" by the compilation environment, because a PIC blob will always be isolated from the environment, and won't have the luxory of a proper executable loader such as "ld.so".
+
+Host LibC Implementation
+------------------------
+When injecting our (PIC) code into a host binary, we should make sure to use the proper constants for the matching standard library implementation that is used by the respective binary:
+* SCOUT_HOST_GLIBC - The used library is Glibc
+* SCOUT_HOST_UCLIBC - The used library is uClibc (or uClibc-NG)
+
+If none are defined the base library will define "SCOUT_HOST_GLIBC" on it's own.
 
 Loader Flags
 ------------
 * SCOUT_LOADER - We are now compiling a loader (that might be using it's own pic plt / globals).
 * SCOUT_LOADING_THUMB_CODE - The loader will load a Scout that was compiled to be executed on an ARM cpu in Thumb mode.
-* SCOUT_RESTORE_FLOW - The default loaders (```tcp_client_server.c```, ```tcp_loader_server.c```) will clean-up after themselves if the loaded scout will finish his endless loop.
+* SCOUT_RESTORE_FLOW - The default loaders (``tcp_client_server.c``, ``tcp_loader_server.c``) will clean-up after themselves if the loaded scout will finish the endless loop.
+
+If the loader will be compiled to be Position Independent (PIC), which is probably the most common use case, it will also define a new flag of "SCOUT_SLIM_SIZE", to help shrink the size of the binary (to serve as an effective shellcode).
+Under this definition the TCP server would expect the following flags (if needed):
+* SCOUT_TCP_CLIENT - There is a need to include the feature of a TCP client
+* SCOUT_TCP_SERVER - There is a need to include the feature of a TCP server
+* SCOUT_TCP_SEND - There is a need to include the ability to reliably send TCP messages
 
 Additional Flags:
 -----------------
 * SCOUT_INSTRUCTIONS - Scout is going to use the instructions api (using the TCP server for instance)
-* SCOUT_DYNAMIC_BUFFERS - Scout will dynamically malloc() buffers to be used by the tcp server. Otherwise static buffers will be used.
+* SCOUT_DYNAMIC_BUFFERS - Scout will dynamically ``malloc()`` buffers to be used by the tcp server. Otherwise static buffers will be used.
 * SCOUT_PROXY - Scout is going to act as a proxy (user scout passing instructions to a kernel driver for instance)
-* SCOUT_MMAP - Should scout's loaders use mmap() and mprotect() when loading (if defined) or should they simply use malloc (if undefined)
+* SCOUT_MMAP - Should scout's loaders use ``mmap()`` and ``mprotect()`` when loading (if defined) or should they simply use ``malloc()`` (if undefined)

docs/Default Instructions.md

@@ -4,19 +4,20 @@ Scout is an instruction-based debugger, that commonly uses a TCP network session
 
 Default Instructions
 --------------------
-* **NOP** - Used as a Pong (or Keep-Alive) instruction to make sure the debugger is active and responds to commands
+* **NOP** - Used as a Ping (or Keep-Alive) instruction to make sure the debugger is active and responds to commands
 * **Memory Read** - Reads (virtual) memory from the given address, and sends it back
 * **Memory Write** - Writes a given binary content to a (virtual) memory in the debuggee's address space
 
-Each supported instruction must be pre-registered by the debugger before it enters his server loop, usually by calling ```register_all_instructions()```.
+Each supported instruction must be pre-registered by the debugger before it enters his server loop, usually by calling ``register_all_instructions()``.
 
 Network API
 -----------
 Each instruction is sent together with a network header that includes the following:
+
 * Instruction ID - 2 Bytes
 * Length field - 4 Bytes
 
 The length field specifies the length, in bytes, of the serialized instruction.
 
 **Note:** All instructions should be serialized to NETWORK order.
-See ```manager\scout_api.py``` for a python sample that prepares the instructions for network transmission.
+See ``manager\scout_api.py`` for a python sample that prepares the instructions for network transmission.

docs/Default Loaders.md

@@ -8,8 +8,8 @@ To overcome these limitations, the scout debugger comes with a list of supported
 
 The defualt loaders are:
 
-1. ```tcp_client_loader.c``` - Connects to a predefined TCP server
-2. ```tcp_server_loader.c``` - Waits for an incoming TCP client
+1. ``tcp_client_loader.c`` - Connects to a predefined TCP server
+2. ``tcp_server_loader.c`` - Waits for an incoming TCP client
 
 Both of the loaders use the same network protocol:
 
@@ -18,13 +18,13 @@ Both of the loaders use the same network protocol:
 
 After a TCP connection was established, the loader will follow these steps:
 
-1. The loader will receive the header and malloc() a memory buffer of appropriate size.
+1. The loader will receive the header and ``malloc()``/``mmap()`` a memory buffer of appropriate size.
 2. The data will be received and stored in this memory buffer.
-3. The loader will flush the D-cache and I-cache of the buffer (only in architectures were this is needed)
-4. The loader will mprotect() the memory to use the correct permissions (only in architectures were this is needed)
+3. The loader will flush the D-cache and I-cache of the buffer (only in architectures were this is needed, and only if we have high enough CPU privileges for it)
+4. The loader will ``mprotect()`` the memory to use the correct permissions (only in architectures were this is needed)
 5. The loader will jump into the buffer's start (offset 0, as mentioned in the "PIC Compilation" section)
-6. In the flow restore case, once the loaded executable finishes the loader will free the memory and close the used sockets.
+6. In the flow restore case, once the loaded executable finishes, the loader will free the memory and close the used sockets.
 
-The functions ```remoteLoadServer()``` and ```remoteLoadClient()``` in ```scout_network.py```, implement the required protocol for communicating with the loader, and loading up the full Scout.
+The functions ``remoteLoadServer()`` and ``remoteLoadClient()`` in ``scout_network.py``, implement the required protocol for communicating with the loader, and loading up the full Scout. A working example is shown under the ``manager.py`` of the ``embedded_scout`` example.
 
 **Note:** In case there are any W^X style limitations in your environment, it is recommended to make sure that the allocated memory for the full debugger will have both Write and eXcutable permissions (should probably use the SCOUT_MMAP flag in this case).