Architecture

Much of this architecture is inspired by Netlfix's Atlas. See: https://github.com/Netflix/atlas

com.vennbilling.app

Control plane app that manages all the schema definitions and UI bits

com.vennbilling.agent

front door for data

lib-event-registery

• library under the agent that manages the index of events defined in the schema
• the agent will use this index to determine if a received event is something it actually cares about
• changes pushed by the control plane will update this index

lib-event-forwarder

• library under the agent handles forwarding events to subscribers

com.vennbilling.server

processor of data. receives events from the agent

lib-event-eval

• library under the event subscriber that evaluates the event against the current schema
• if valid, creates an invoice item via the control plane API

lib-event-persistence

• triggers storing the received event in s3

event-achiever

writes received events to a system of record (s3)

• receives data from ingress and the event subscriber
• converts the JSON to a .parquet file and stores the data in S3