Project Details

Date: 2017-05-23

Project Setup

Vagrant up using Vagrantfile for 1 hosts

• pyspark (Python Spark)

  • Install jdk
  • Install Spark
    • apt-get update
    • apt-get install -y software-properties-common unzip
    • apt-add-repository -y ppa:ansible/ansible
    • apt-get update
    • apt-get install -y ansible python-dev libkrb5-dev krb5-user python-pip libssl-dev libffi-dev
    • pip install --upgrade pip
    • pip install pywinrm pykerberos requests_kerberos shade

• docker

  • sudo apt-get update
  • sudo apt-get install docker.io
  • sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9
  • sudo sh -c "echo deb https://get.docker.com/ubuntu docker main\ > /etc/apt/sources.list.d/docker.list"
  • sudo apt-get update
  • sudo apt-get install lxc-docker

References

• Docker install on ubuntu
  • http://buildvirtual.net/docker-installing-docker-on-ubuntu-trusty-14-04/
• PySpark Tutorial
  • https://www.datacamp.com/community/tutorials/apache-spark-python#gs.5=0yEEM
• JDK8 Install
  • https://www.digitalocean.com/community/tutorials/how-to-install-java-on-ubuntu-with-apt-get#installing-default-jrejdk