-
Notifications
You must be signed in to change notification settings - Fork 22
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
1 changed file
with
62 additions
and
82 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,94 +1,74 @@ | ||
| # macos-unifiedlogs | ||
| A simple Rust library that can help parse the macOS Unified Log files. | ||
|
|
||
| Unified Logs were introduced in macOS version 10.12 (Sierra, 2016). Part of Apple's goal to create a unified log format for all Apple products. They exist on macOS, iOS, watchOS, tvOS. | ||
| The Unified Logs replace many of the old log formats Apple used. This simple library can be used to parse files. | ||
| A simple Rust library that can help parse the macOS Unified Log files. | ||
|
|
||
| Unified Logs were introduced in macOS version 10.12 (Sierra, 2016). Part of | ||
| Apple's goal to create a unified log format for all Apple products. They exist | ||
| on macOS, iOS, watchOS, tvOS. The Unified Logs replace many of the old log | ||
| formats Apple used. This library can be used to parse these log files.\ | ||
| Data that is currently extracted includes: | ||
| * Process ID | ||
| * Thread ID | ||
| * Activity ID | ||
| * Log Message | ||
| * Timestamp (Intel and ARM supported) | ||
| * Effective User ID (EUID) | ||
| * Log Type | ||
| * Event Type | ||
| * Library | ||
| * Subsystem | ||
| * Category | ||
| * Process | ||
| * Raw message - Message extracted from UUID file | ||
| * Message entries - Message parts from tracev3 file. Combines with Raw message to get the Log Message | ||
| * Library UUID | ||
| * Process UUID | ||
| * Boot UUID | ||
| * Timezone | ||
|
|
||
| - Process ID | ||
| - Thread ID | ||
| - Activity ID | ||
| - Log Message | ||
| - Timestamp (Intel and ARM supported) | ||
| - Effective User ID (EUID) | ||
| - Log Type | ||
| - Event Type | ||
| - Library | ||
| - Subsystem | ||
| - Category | ||
| - Process | ||
| - Raw message - Message extracted from UUID file | ||
| - Message entries - Message parts from tracev3 file. Combines with Raw message | ||
| to get the Log Message | ||
| - Library UUID | ||
| - Process UUID | ||
| - Boot UUID | ||
| - Timezone | ||
|
|
||
| ## Running | ||
| Three (3) simple example binaries are available in `examples`. | ||
| * `unifiedlog_parser` - Can parse all logs into a single CSV file. It can also be run on a live system. The resulting CSV file will likely be quite large | ||
| * `unifiedlog_parser_json` - Can parse all logs into JSON files. It can also be run on a live system. Each log file (tracev3 file) will correspond to a single JSON file. Depending on the logs, hundreds of JSON files may get created | ||
| * `parse_tracev3` - Can parse a single tracev3 file without any timesync or uuidtext files, to a JSON file. However, without the uuidtext or timesync files the resulting JSON file will be heavily incomplete. | ||
|
|
||
| See `RUNNING.md` for overview of running the example binaries | ||
| ## Using as Library | ||
| If you want to import this project into a Rust application add the following to you `Cargo.toml` file | ||
| ``` | ||
| macos-unifiedlogs = {git = "https://github.com/mandiant/macos-UnifiedLogs"} | ||
| ``` | ||
| If you want to pin to a specific commit | ||
| ``` | ||
| macos-unifiedlogs = {git = "https://github.com/mandiant/macos-UnifiedLogs", rev = "commit hash"} | ||
| ``` | ||
| See `Library.md` for overview of how to use the library. Simple example projects are also available to review and use | ||
| ## Status | ||
| This library has been heavily tested on log data from macOS Sierra (10.12.5) to Monterey (12). | ||
| Its been tested against 100+ million log entries. However, due the complexity of the Unified Log format there are some limitations: | ||
| 1. No printf style error code lookup support. This library does not do any error code lookups for log messages. The native `log` command on macOS supports error code lookups when it encounters printf style `%m` messages. | ||
| An example base log messsage: 'Failed to open file, error: %m' | ||
| a. This Library outputs: | ||
| ``` | ||
| Failed to open file, error: 1 | ||
| ``` | ||
| b. The macOS Log command outputs: | ||
| ``` | ||
| Failed to open file, error: no such file or directory | ||
| ``` | ||
| Here the error code 1 gets translated to the error string message | ||
| Three (3) simple example binaries are available in `examples`. | ||
|
|
||
| - `unifiedlog_parser` - Can parse all logs into a single CSV file. It can also | ||
| be run on a live system. The resulting CSV file will likely be quite large | ||
| - `unifiedlog_parser_json` - Can parse all logs into JSON files. It can also be | ||
| run on a live system. Each log file (tracev3 file) will correspond to a single | ||
| JSON file. Depending on the logs, hundreds of JSON files may get created | ||
| - `parse_tracev3` - Can parse a single tracev3 file without any timesync or | ||
| uuidtext files, to a JSON file. However, without the uuidtext or timesync | ||
| files the resulting JSON file will be heavily incomplete. | ||
|
|
||
| 2. No support for custom object decoders. The Unified Log format allows a developer to log abrirtary data to the logs. Apple also includes a handful of custom objects that developer can use to log raw data. An example list can be found in `man os_log`. However, it is not a complete list. | ||
| ``` | ||
| man os_log | ||
| ... | ||
| To format a log message, use a printf(3) format string. You may also use the "%@" format specifier for use with Obj-C/CF/Swift objects, and %.*P which can be used to decode arbitrary binary data. The logging system also supports custom decoding of values by denoting value types inline in the format %{value_type}d. The built-in value type decoders are: | ||
| See `RUNNING.md` for overview of running the example binaries See `Library.md` | ||
| for overview of how to use the library. Simple example projects are also | ||
| available to review and use | ||
|
|
||
| Value type Custom specifier Example output | ||
| BOOL %{BOOL}d YES | ||
| bool %{bool}d true | ||
| darwin.errno %{darwin.errno}d [32: Broken pipe] | ||
| darwin.mode %{darwin.mode}d drwxr-xr-x | ||
| darwin.signal %{darwin.signal}d [sigsegv: Segmentation Fault] | ||
| time_t %{time_t}d 2016-01-12 19:41:37 | ||
| timeval %{timeval}.*P 2016-01-12 19:41:37.774236 | ||
| timespec %{timespec}.*P 2016-01-12 19:41:37.2382382823 | ||
| bytes %{bytes}d 4.72 kB | ||
| iec-bytes %{iec-bytes}d 4.61 KiB | ||
| bitrate %{bitrate}d 123 kbps | ||
| iec-bitrate %{iec-bitrate}d 118 Kibps | ||
| uuid_t %{uuid_t}.16P 10742E39-0657-41F8-AB99-878C5EC2DCAA | ||
| sockaddr %{network:sockaddr}.*P fe80::f:86ff:fee9:5c16 | ||
| in_addr %{network:in_addr}d 127.0.0.1 | ||
| in6_addr %{network:in6_addr}.16P fe80::f:86ff:fee9:5c16 | ||
| ``` | ||
| Currently when the library encounters log messages that contain arbitrary binary data, it will base64 the data as a string. | ||
| Support for custom decoders will hopefully be added in version 2 of the library. | ||
| ## Limitations | ||
|
|
||
| 3. No support for log messages that have custom object structures or protocol buffer data. | ||
| Some logs contain binary plist files, custom object structures, or protocol buffer data. This library currently supports parsing binary plist data, but it does not support custom object structures, or protocol buffer data. | ||
| The custom object structures are similar to the custom decoders mentioned above. But there is no list of decoders | ||
| Support for custom object structures will hopefully be added in version 2 of the library | ||
| Its been tested against millions of log entries. However, due the complexity of | ||
| the Unified Log format there are some limitations: | ||
|
|
||
| 1. No printf style error code lookup support. This library does not do any error | ||
| code lookups for log messages. The native `log` command on macOS supports | ||
| error code lookups when it encounters printf style `%m` messages.\ | ||
| An example base log messsage: 'Failed to open file, error: %m'\ | ||
| a. This Library outputs: | ||
| ``` | ||
| Failed to open file, error: 1 | ||
| ``` | ||
| b. The macOS Log command outputs: | ||
| ``` | ||
| Failed to open file, error: no such file or directory | ||
| ``` | ||
|
|
||
| 2. Support for most custom objects in log messages. However, some objects are | ||
| not supported. Unsupported objects will be base64 encoded | ||
|
|
||
| # References | ||
| https://github.com/ydkhatri/UnifiedLogReader | ||
| https://github.com/libyal/dtformats/blob/main/documentation/Apple%20Unified%20Logging%20and%20Activity%20Tracing%20formats.asciidoc | ||
| https://eclecticlight.co/2018/03/19/macos-unified-log-1-why-what-and-how/ | ||
|
|
||
| https://github.com/ydkhatri/UnifiedLogReader\ | ||
| https://github.com/libyal/dtformats/blob/main/documentation/Apple%20Unified%20Logging%20and%20Activity%20Tracing%20formats.asciidoc\ | ||
| https://eclecticlight.co/2018/03/19/macos-unified-log-1-why-what-and-how/\ | ||
| https://www.crowdstrike.com/blog/how-to-leverage-apple-unified-log-for-incident-response/ |