README.md

Autonomous Driving Data Framework(ADDF)

ADDF is a collection of modules for Scene Detection, Simulation (mock), Visualization, Compute, Storage, Centralized logging etc, deployed using SeedFarmer orchestration tool. ADDF allows you to build distinct, stand alone Infrastructure as code (IAAC) modules and exchange information about dependencies using metadata which can be exported from one module and imported into another. Each module can be found under the modules directory of this repository.

Deployment Instructions

You can refer to the SeedFarmer guide to understand how SeedFarmer CLI can be used to bootstrap and deploy ADDF.

You can follow instructions in the Deployment Guide Readme. You can also follow the blogpost for understanding ADDF in detail.

Different types of modules supported by ADDF

Use-case specific Modules

Type	Description
Rosbag Scene Detection Module	Deploys a Rosbag Scene Detection pipeline for use in ADDF
Rosbag WebViz Module	Deploys and Visualizes Rosbag Data on AWS using Webviz for use in ADDF

Optional Modules

Type	Description
Networking Module	Deploys standard networking resources such as VPC, Public and Private subnets, endpoints for use in ADDF
DataLake Buckets Module	Deploys shared datalake buckets such as input, intermediate, output, logging, artifact buckets for use in ADDF

Core Modules

Type	Description
EKS Compute Module	Deploys shared EKS Cluster with commonly preferred addons for use in ADDF
Metadata Storage Module	Deploys shared Metadata storage module which deploys metadata resources like Glue, DDB etc for use in ADDF
Amazon Managed Workflows for Apache Airflow (MWAA) Module	Deploys shared MWAA module for use in ADDF
Opensearch Module	Deploys Amazon Opensearch Cluster for use in ADDF
Neptune Module	Deploys Amazon Managed Neptune Cluster for use in ADDF

Integration Modules

Type	Description
DDB to Opensearch Module	This module integrates DynamoDB table with Opensearch cluster
EKS to Opensearch Module	This module integrates EKS Cluster with Opensearch cluster
EMR to Opensearch Module	This module integrates EMR Cluster with Opensearch cluster
Opensearch Proxy Module	This module deploys a Proxy server to access Opensearch cluster

Simulation Modules

Type	Description
K8s-Managed Module	This module helps running simulations on AWS EKS, when triggered by KubernetesJob Operator from airflow environment
AWS Batch Module	This module helps running simulations on AWS Batch, when triggered by Batch Operator from airflow environment

IDE Modules

Type	Description
Self Managed JupyterHub Module	This module deploys self managed JupyterHub environment on AWS EKS
Self Managed VSCode Module	This module deploys self managed VSCode environment on AWS EKS
AWS Managed EMR Studio Module	This module deploys AWS managed EMR Studio with EMR on EKS

Example Modules

Type	Description
Example DAG Module	This module deploys a pattern to integrate a target DAG module to work with shared MWAA Cluster

Reporting Issues

If you notice a defect, feel free to create an Issue

Deployment FAQ

If you need to debug a deployment in ADDF, here are few things you can checkout Readme