Log Enrichment through Substation

[Bin-security]

Hey @jshlbrd and @brexhq/substation. Substation is useful for ingesting, transforming and enriching logs. It supports DynamoDB, CSV, JSON, text file, MMDB etc. out of box. We are working on adding context information to logs such as IP geolocation, domain or host names, AWS account information, user information through LDAP or other identity providers etc. Do you guys have a list of enrichments you do in Substation?

[jshlbrd]

Hi @Bin-security 👋

We won't share the specific enrichments that we do or any specifics on how we use the enrichment features, but some of the documentation includes example use cases:

• https://substation.readme.io/docs/http-processor#use-cases
• https://substation.readme.io/docs/kv-aws-dynamodb#use-cases
• https://substation.readme.io/docs/csv-file#use-cases
• https://substation.readme.io/docs/mmdb-maxmind-database#use-cases
• https://substation.readme.io/docs/text-file#use-cases

Most of the use cases you described would be best accomplished by using a distributed cache that is fed by an external service and read by nodes in the data pipelines using the KV Store processor. For file-based stores (e.g., MMDB, CSV, text) we use another service (not related to Substation) to retrieve data, format it, and upload it to S3 so that Substation nodes can download it.