include oracle cdc and ETL

README.md

@@ -1,33 +1,7 @@
-# Confluent Platform 6.0.1 KSQLDB Hands-on Workshop
+# Confluent Platform 6.1 KSQLDB Hands-on Workshop
 This github project describes a Hands-on Workshop around Confluent KSQLDB. The structure of the Hands-on is as followed
   * Explaining and Introduce KSQLDB
   * Labs: Get to know the environment
-  * Advanced explanation of KSQLDB
-  * Advanced KSQLDB Labs to setup real use cases
-
-In general, the hands-on will take 4 hours.
-
-  * Start at 10am: Intro and first labs
-  * break at 12am: 1 hour break
-  * Continue at 1pm: Additional Labs
-  * Finish at 3pm
-
-# Environment for Hands-on Workshop Labs
-The Hands-on environment can be deployed in three ways
-
-  1. run Docker-Compose on your own hardware/laptop use docker-locally
-  2. create the demo environment in Cloud Provider infrastructure, deploy cloud environment
-  3. Confluent will deploy a cloud environment for you, and will send you during the workshop all credentials
-
-# Prerequisites for running environment in Cloud
-For an environment in cloud you need to run following components on your machine:
-
-  * Browser with internet access
-  * if you want to deploy in our own environment
-     * create your own SSH key and deploy in AWS, I use the key name hackathon-temp-key
-     * terraform have to be installed
-     * Terraform will install everything you need to execute during the workshop on cloud compute instance
-  * Having internet access and Ports 8088, 22, 9021 have to be open
 
 # Prerequisites for running environment on own hardware
 For an environment on your hardware (Macos):
@@ -42,22 +16,10 @@ For Windows Users
 
 In general we will work mostly on the prompt, with KSQLDB cli. But you can also use Confluent Control Center KSQLDB GUI is most cases.
 
-# Hands-on Workshop
-Before the workshop you will get informed by Confluent with additional information: Access, Webconference dial-in, etc.
-
-  * Please prepare yourself
-     * Please check the documentation and get an rough overview: [ksqldb](https://www.confluent.io/product/ksql/)
-     * your hardware should be able to run docker-compose. Docker needs at least 8GB of your RAM
-
-Note:
-We will ask you before the workshop, if you would like to run on your own environment or if you would like to have an environment provisioned by Confluent in the cloud.
 
 # Hands-on Agenda and Labs:
 0. We will start first with an environment check:
-    * Attendees with an environment provisioned by Confluent should all have an email with credentials etc.
-    * Is everything up and running: local, cloud or environment giving by confluent.
     * [Setup the environment](labs/00_Setup-Env.md)
-    * We expect a 20 MIN time-slot for this exercise
 1. Intro KSQLDB (PPT) - 30 Min
     * RECAP KSQLDB - short presentation by presenter (10 minutes)
     * What is the structure for today? (20 minutes)
@@ -66,39 +28,12 @@ We will ask you before the workshop, if you would like to run on your own enviro
     * [Stock price calculation with User defined functions](labs/02_usecase_finserv_2.md)
     * [Create Stocktrade data](labs/03_usecase_finserv_3.md)
     * [Transaction cache](/labs/04_usecase_finserv_4.md)
-3. Labs Retail/Logistics - Lab 5 -7
-    * [Real-Time Inventory](labs/05_usecase_realtime_inventory.md)
-    * [Track & Trace / Shipments](labs/06_usecase_track-and-trace.md)
-    * [Distance calculation](labs/07_usecase_distance.md)
-4. Lab Customer Object - Lab 8
-    * [De-Normalize a customer object sourced from a simluated DB](labs/08_ksqldb_customerobject.md)
-5. Lab Geo Fencing - Lab 9
-    * [Doing GEO Fencing with ksqlDB](labs/09_geofencing.md)
-6. Labs Operations - Lab 10
-    * [Add a KSQLDB App into your environment](labs/09_ksqldb_operations.md)
-
-We will have a LUNCH Break for 60 Minutes (around 12am) and the workshop will finish around 3pm.
-
-# Stop everything
-Note: By Confluent provisioned Compute VMs will be destroyed latest at 5pm latest on Workshop day automatically. Outside of cloud compute, please use terraform, to really destroy the environment in the cloud:
-```bash
-terraform destroy
-```
-If you inside cloud compute you can stop the environment;
-```bash
-cd /home/ec2-user/confluent-ksqldb-hands-on-workshop-master/docker/
-docker-compose down -v
-```
-A restart inside the compute:
-```bash
-docker-compose down -v
-docker-compose up -d
-```
-On your local machine just execute
-```bash
-cd confluent-ksqldb-hands-on-workshop-master/docker/
-docker-compose down -v
-```
+3. Labs Retail/Logistics - Lab 5
+    * [Real-Time Inventory](labs/05_usecase_realtime_inventory_ETL.md)
+4. Labs Oracle CDC - Lab 4
+    * [Oracle CDC](labs/11_OracleCDC_ksqldb.md)
+5. Lab Monitoring KsqlDB
+    * [Comming soon...]
 
 Thanks a lot for attending
 Confluent Team

docker/docker-compose.yml

@@ -270,7 +270,18 @@ services:
       MONGO_INITDB_ROOT_PASSWORD: mongo-pw
       MONGO_REPLICA_SET_NAME: my-replica-set
     command: --replSet my-replica-set --bind_ip_all
-
+  mongo:
+    image: mongo:4.2.5
+    hostname: mongo
+    container_name: mongo
+    ports:
+      - "27017:27017"
+    environment:
+      MONGO_INITDB_ROOT_USERNAME: mongo-user
+      MONGO_INITDB_ROOT_PASSWORD: mongo-pw
+      MONGO_REPLICA_SET_NAME: my-replica-set
+    command: --replSet my-replica-set --bind_ip_all
+
   postgres:
     image: debezium/postgres:12
     hostname: postgres
@@ -285,15 +296,6 @@ services:
       - ./postgres/custom-config.conf:/etc/postgresql/postgresql.conf
     command: postgres -c config_file=/etc/postgresql/postgresql.conf
 
-  elastic:
-    image: elasticsearch:7.6.2
-    hostname: elastic
-    container_name: elastic
-    ports:
-      - "9200:9200"
-      - "9300:9300"
-    environment:
-      discovery.type: single-node
   oracle:
     image: store/oracle/database-enterprise:12.2.0.1
     hostname: oracle

docker/loadOrders.py

@@ -0,0 +1,49 @@
+# importing pymongo
+from pymongo import MongoClient
+import random
+import time
+import datetime
+# establing connection
+try:
+    connect = MongoClient('localhost', 27017, username='mongo-user', password='mongo-pw',)
+    print("Connected successfully!!!")
+except:
+    print("Could not connect to MongoDB")
+
+customer = ["1","2","3","4","5","6","7","8","9","10"]
+currency = ["usd","eur"]
+cities = ["Madrid","Barcelona","Paris","Montevideo","Florida","Montreal","Berlin","Colonia"]
+
+# connecting or switching to the database
+db = connect.logistics
+
+# creating or switching to demoCollection
+
+i=0
+
+while True:
+    i=i+1
+    #insert in orders
+    collection = db.orders
+    order_id = str(random.randrange(50))
+    ts= str(datetime.datetime.now().strftime("%Y-%m-%dT%H:%M:%S"))
+    document = {
+    "customer_id": random.choice(customer), 
+    "order_id": (order_id), 
+    "price": round(random.uniform(10, 400),2),
+    "currency": random.choice(customer),  
+    "ts": ts
+    }
+    print(document)
+    collection.insert_one(document)
+    #sleep para simulacion
+    time.sleep(random.randrange(4))
+    collection = db.shipments
+    document_ship = {
+    "order_id": (order_id),
+    "shipment_id": str(random.randrange(50)),
+    "origin": random.choice(cities),
+    "ts": str(datetime.datetime.now().strftime("%Y-%m-%dT%H:%M:%S"))
+    }
+    print(document_ship)
+    collection.insert_one(document_ship)

docker/postgres/custom-config.conf

@@ -1,4 +1,4 @@
 listen_addresses = '*'
 wal_level = 'logical'
 max_wal_senders = 1
-max_replication_slots = 1
+max_replication_slots = 1

labs/00_Setup-Env.md

@@ -67,5 +67,5 @@ docker exec -it workshop-kafka bash -c 'cat /produce-data/shipment_status.json |
 docker exec -it workshop-kafka bash -c 'cat /produce-data/transactions.json | kafka-console-producer --topic transactions --broker-list localhost:9092  --property "parse.key=true" --property "key.separator=:"'
 ```
 
-[go back to Agenda](https://github.com/ora0600/confluent-ksqldb-hands-on-workshop/blob/master/README.md#hands-on-agenda-and-labs)
+[go back to Agenda](https://github.com/jr-marquez/ksqldbWorkshop/blob/main/README.md#hands-on-agenda-and-labs)