Skip to content

Latest commit

 

History

History
112 lines (86 loc) · 3.64 KB

README.md

File metadata and controls

112 lines (86 loc) · 3.64 KB
Raw

Table of Contents

Apache Kafka
Topics, partitions and offsets
Brokers
Topic Replication

Apache Kafka

It is a open source project created by linkedin, it used to exchange data between many source system and target system. Apache kafka allows to decoupling data streams and systems.

Apache Kafka: Use Cases

  • Messaging System
  • Activity Tracking
  • Gather metrics from many different locations
  • Application Logs gathering
  • Stream processing
  • Decoupling of system dependencies

Examples:

  • Netflix uses Kafka to apply recommendations in real time,
  • Uber uses Kafka to gather user taxi and trip data in real time to compute and forecast demand and compute surge pricing in real time
  • LinkedIn uses Kafka to prevent spam collect user interactions to make better connection recommendations in real time

Topics, partitions and offsets

  • Topics are similar to table in database
  • You can have many topics
  • There is no specific name for topic, you can choose any name
  • Topics are split into partition and its ordered
  • Each partition have message in it and have a incremental id called offset for each message

Example:

  • Let Say u have a fleet of trucks, each truck reports its GPS position to kafka

  • Then you have a truck topic truck_gps that contains the position of all trucks

  • Each truck will send a message to Kafka every 20 seconds, each message will contain the truckID and the truck position

  • Offset only have a meaning for a specific partition

  • Order is guaranteed only within a partition

  • Data is kept only for a limited time

  • Once the data is written to a partition, it can't be changed

  • Data is assigned randomly to partition unless a key is provided

Brokers

  • A Kafka cluster is composed of multiple brokers, Brokers means server, its identified by ID (integer)
  • Certain topic partitions present in each broker
  • Connecting to any broker, you will be connected to the entire cluster
  • Good broker number is 3

Topic Replication

  • helps to access the data if one brokers fails, a replication factor is assigned with Topic, I usually between 2 and 3
  • if one broker is down and another broker can serve the data
  • at a time one broker can be leader for a given partition
  • The other broker synchronize the data
  • if the down broker was up then it automatically transfer the leadership to the broker

Starting apache kafka

  • first need to start the zookeeper server by
bin/zookeeper-server-start.sh config/zookeeper.properties
  • second step is to start apache kafka by
bin/kafka-server-start.sh config/server.properties
  • to create topic need to specify the topic name and the bootstrap server
bin/kafka-topics.sh --create --bootstrap-server 127.0.0.1:9092 --topic cities
  • to list the topics in the zookeeper
bin/kafka-topics.sh --list --zookeeper 127.0.0.1:2181
  • to describe the topics in zookeeper by passing the topics name
bin/kafka-topics.sh --list --zookeeper 127.0.0.1:2181
  • to start producer u need to specify the broker list which is the kafka broker by , then followed by topic name
bin/kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic cities
  • consumer console
 bin/kafka-console-consumer.sh --bootstrap-server 127.0.0.1:9092 --topic cities
  • Run a debezium Kafka connector
export CLASSPATH=/test/kafka-3.0.0-src/connect/debezium-connector-postgres/*
bin/connect-distributed.sh config/connect-distributed.properties