OT6 Cloud Project

To run the analytics

Create an .env with these credentials:

SECRET_ACCESS_KEY=XXXXXXXXXXXXXXXXXX
ACCESS_KEY_ID=XXXXXXXXXX

Run

docker compose up --scale spark-worker=3

Open:

• http://localhost:8888/notebooks/work/sparkling-water.ipynb

Stop using

docker compose down -v --remove-orphans

Cluster overview

Application	URL	Description
JupyterNotebook	localhost:8888	Jupyter notebooks
Web UI	localhost:4040	To monitor the status and resource consumption of your Spark cluster
Spark Master	localhost:8080	Spark Driver
Spark Worker I		Spark Worker node
Spark Worker II		Spark Worker node

Stack

• Spark 3.3.0
• PySpark 3.3.1

To run the spider

Create a virtual env

python3 -m venv env

source env/bin/activate

pip -r requirements.txt

Create an .env file in ./project/project/ folder with these credentials:

SECRET_ACCESS_KEY=XXXXXXXXXXXXXXXXXX
ACCESS_KEY_ID=XXXXXXXXXX