Kafka-Docker-Setup-With-Demo

This is a demo project showing how to create a stream data processing system

check list

• setting up zookeeper and kafka with docker

• creating clients that talk to the kafka broker

The demo project is a fraud detection system. It will have two clients. One of the clients will be a producer generating the transactions as a stream of data. The other one will consumen those transaction events, check if they are fradulent, and publish them on separate topics based on the result from the checking.

This repo is built using this blog post. Building A Streaming Fraud Detection System With Kafka And Python

Instructions

1. Clone the repo

   git clone https://github.com/STT-Data-Engineering/Kafka-Docker-Setup-With-Demo.git

2. Create the docker network that the broker and clients would communicate with

   docker network create kafka-network

3. Start zookeeper and kafka containers

   docker-compose -f docker-compose.kafka.yml up

4. Start The clients with another terminal window

   docker-compose up

To inspect if the clients are really communicating with the broker, you can use the following:

To check the generator:

docker-compose -f docker-compose.kafka.yml exec broker kafka-console-consumer --bootstrap-server localhost:9092 --topic queueing.transactions --from-beginning

To check the detector:

docker-compose -f docker-compose.kafka.yml exec broker kafka-console-consumer --bootstrap-server localhost:9092 --topic streaming.transactions.legit

and

docker-compose -f docker-compose.kafka.yml exec broker kafka-console-consumer --bootstrap-server localhost:9092 --topic streaming.transactions.fraud