Datawarehouse

Sensor Data ELT

A fully dockerized ELT pipeline project, using Airflow, PostgreSQL, DBT, and Redash.
Report Bug · Request Feature

Table of Contents

1. About The Project
   • Built With
2. Getting Started
   • Prerequisites
   • Installation
3. Contact
4. Acknowledgements

About The Project

Fully dockerized scalable ELT pipeline using PostgreSQL, Airflow, DBT and Redash.

Built With

Tech Stack used in this project

• Airflow
• PostgreSQL
• DBT
• Redash

Getting Started

Prerequisites

Make sure you have docker installed on local machine.

• Docker
• DockerCompose

Installation

1. Clone the repo and Navigate

   git clone https://github.com/sel6/Datawarehouse.git
   Navigate to airflow/dags

2. Run

    docker-compose -f docker-compose.yaml up --build

3. Open Airflow web browser

   Navigate to `http://localhost:8081/` on the browser
   activate and trigger the `create_connection` dag
   activate and trigger the `create_new_db` dag
   activate and trigger the `create_new_table` dag
   activate and trigger the `fill_table` dag

4. Access redash dashboard

   Navigate to `http://localhost:5011/` on the browser

5. Access your postgres database

   Navigate to `http://localhost:5432/` on the browser
   use `warehouse` for db name
   use `warehouse` for username
   use `warehouse` for password

Contact

Selam Ayehubirhan - [email protected]

Acknowledgements

• 10 Academy