OT6 Cloud Project

To run the analytics

Create an .env with these credentials:

SECRET_ACCESS_KEY=XXXXXXXXXXXXXXXXXX
ACCESS_KEY_ID=XXXXXXXXXX

Run

docker compose up --scale spark-worker=3

Open:

Stop using

docker compose down -v --remove-orphans

Application	URL	Description
JupyterNotebook	localhost:8888	Jupyter notebooks
Web UI	localhost:4040	To monitor the status and resource consumption of your Spark cluster
Spark Master	localhost:8080	Spark Driver
Spark Worker I		Spark Worker node
Spark Worker II		Spark Worker node

Create a virtual env

python3 -m venv env

source env/bin/activate

pip -r requirements.txt

Create an .env file in ./project/project/ folder with these credentials:

SECRET_ACCESS_KEY=XXXXXXXXXXXXXXXXXX
ACCESS_KEY_ID=XXXXXXXXXX