Skip to content

Latest commit

 

History

History
100 lines (76 loc) · 3.38 KB

README.md

File metadata and controls

100 lines (76 loc) · 3.38 KB
Raw

DemoHumidityMonitoring

This is a demo that shows an event-driven processing pattern using open source projects. The high level architecture is shown as follows.

Quick Start

  1. Start mosquitto server from local host (brew start mosquitto on MacOS):

    brew start mosquitto

  2. Start services including flink job manager, flink taskmanager, kafka, zookeeper and kafka:

    cd flink-playgrounds
docker-compose up

  3. Start two terminal console, run a nc session from each of them with different ports:

    nc -lk 12345
nc -lk 12346

  4. Setup Pi and the AM2302 sensor, see the connection diagram in the humidity-collection-on-pi/setup.png

    cd humidity-collection-on-pi
connect the sensor and the PI pins

  5. Start mqtt client on raspberry to collect humidity readings from the sensor:

    scp humidity-collection.py root@[raspberry pi ip address]:~/
ssh humidity-collection.py root@[raspberry pi ip address]
python humidity-collection.py -h [ip address of your laptop] -p 1883

  6. Start mqtt bridge to pass the readings to kafka:

    python humidity-from-mqtt-to-kafka.py -h localhost -p 1883

  7. Build the flink applications;

       cd humidity-monitoring-flink-app
   mvn clean package

  8. Submit apps from flink dashboard(localhost:8081);

    Start a browser and open localhost:8081;
Submit New Job -> Add New -> Select HumidityMonitoringApp-1.0-SNAPSHOT.jar;
Click the HumidityMonitoringApp-1.0-SNAPSHOT.jar under the Name column;
Input isg.HumidityMonitoring in the Entry Class input box;

  9. Repeat the steps above to submit the isg.AvgHumidityMonitoring app;

  10. Check warnings from the nc consoles;

  11. Update the threshold from a new terminal console:

    kafkacat -b 127.0.0.1:9094 -t humidity -P
{"type":"metadata", "humidityThreshold":70}}

Folder Explained

  • flink-playgrounds - A clone from https://github.com/apache/flink-playgrounds.git with minor modification. It's to startup flink local services including jobmanager and taskmanager. It also start up a kafka and a zookeeper services;

  • humidity-monitoring-flink-app - A very simple java project which contains two applications; One is the monitoring each of the humidity sensor readings and generate alerts to socket port 12345 when it's higher than a threshold; The other monitoring the average of humidity sensor readings in 30 seconds, when the average of the readings exceeds the threshold, then generate an alerts to socket port 12346;

  • humidity-collection-on-pi - There is a real sensor used in this demo. This folder contains a script to collect humidity readings from the sensor. The sensor readings are sent to a mosquitto server hosted on localhost, the topic is humidity;

  • humidity-from-mqtt-to-kafka - It contains a script to pass the humidity sensor readings from mqtt broker to local hosted kafka container; The kafka topic is also humidity;

Dependencies

Hardware

  • Raspberry Pi
  • DHT22 Humidity&Temperature Sensor
  • Laptop or Desktop with Docker installed;

Middleware

  • Mosquitto - An open source mqtt broker;
  • Kafka - An open source message broker;
  • Flink - An open source stream processing engine;

Tools

  • netcat
  • pip
  • Docker
  • kafkacat (optional)
  • Kafka Tool (optional)
  • MQTT fx (optional)

Programing Language

  • Java
  • Python