01-ipyparallel-tutorial-overview.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "toc": "true"
   },
   "source": [
    "# Table of Contents\n",
    " <p><div class=\"lev1\"><a href=\"#overview-and-getting-started\"><span class=\"toc-item-num\">1&nbsp;&nbsp;</span>overview and getting started</a></div><div class=\"lev2\"><a href=\"#Introduction\"><span class=\"toc-item-num\">1.1&nbsp;&nbsp;</span>Introduction</a></div><div class=\"lev2\"><a href=\"#Architecture-overview\"><span class=\"toc-item-num\">1.2&nbsp;&nbsp;</span>Architecture overview</a></div><div class=\"lev3\"><a href=\"#IPython-engine\"><span class=\"toc-item-num\">1.2.1&nbsp;&nbsp;</span>IPython engine</a></div><div class=\"lev3\"><a href=\"#IPython-controller\"><span class=\"toc-item-num\">1.2.2&nbsp;&nbsp;</span>IPython controller</a></div><div class=\"lev3\"><a href=\"#The-Hub\"><span class=\"toc-item-num\">1.2.3&nbsp;&nbsp;</span>The Hub</a></div><div class=\"lev3\"><a href=\"#Schedulers\"><span class=\"toc-item-num\">1.2.4&nbsp;&nbsp;</span>Schedulers</a></div><div class=\"lev2\"><a href=\"#IPython-client-and-views\"><span class=\"toc-item-num\">1.3&nbsp;&nbsp;</span>IPython client and views</a></div><div class=\"lev2\"><a href=\"#Getting-started:-Starting-the-IPython-controller-and-engines\"><span class=\"toc-item-num\">1.4&nbsp;&nbsp;</span>Getting started: Starting the IPython controller and engines</a></div><div class=\"lev2\"><a href=\"#Configuring-an-IPython-cluster\"><span class=\"toc-item-num\">1.5&nbsp;&nbsp;</span>Configuring an IPython cluster</a></div><div class=\"lev1\"><a href=\"#used-libraries\"><span class=\"toc-item-num\">2&nbsp;&nbsp;</span>used libraries</a></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# overview and getting started"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This section gives an overview of IPython’s architecture for parallel and distributed computing. This architecture abstracts out parallelism in a very general way, which enables IPython to support many different styles of parallelism including:\n",
    "\n",
    "- Single program, multiple data (SPMD) parallelism.\n",
    "- Multiple program, multiple data (MPMD) parallelism.\n",
    "- Message passing using MPI.\n",
    "- Task farming.\n",
    "- Data parallel.\n",
    "- Combinations of these approaches.\n",
    "- Custom user defined approaches.\n",
    "\n",
    "Most importantly, IPython enables all types of parallel applications to be developed, executed, debugged and \n",
    "monitored interactively. Hence, the I in IPython. The following are some example usage cases for IPython:\n",
    "\n",
    "- Quickly parallelize algorithms that are embarrassingly parallel using a number of simple approaches. Many simple things can be parallelized interactively in one or two lines of code.\n",
    "- Steer traditional MPI applications on a supercomputer from an IPython session on your laptop.\n",
    "- Analyze and visualize large datasets (that could be remote and/or distributed) interactively using IPython and tools like matplotlib/TVTK.\n",
    "- Develop, test and debug new parallel algorithms (that may use MPI) interactively.\n",
    "- Tie together multiple MPI jobs running on different systems into one giant distributed and parallel system.\n",
    "- Start a parallel job on your cluster and then have a remote collaborator connect to it and pull back data into their local IPython session for plotting and analysis.\n",
    "- Run a set of tasks on a set of CPUs using dynamic load balancing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Architecture overview"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-04-05T08:50:20.480521",
     "start_time": "2017-04-05T10:50:20.475777+02:00"
    }
   },
   "source": [
    "<img src=\"wideView.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The IPython architecture consists of four components:\n",
    "\n",
    "- The IPython engine.\n",
    "- The IPython hub.\n",
    "- The IPython schedulers.\n",
    "- The controller client.\n",
    "\n",
    "These components live in the ```IPython.parallel``` package and are installed with IPython."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### IPython engine"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-04-05T08:58:21.815928",
     "start_time": "2017-04-05T10:58:21.807255+02:00"
    }
   },
   "source": [
    "The IPython engine is a Python instance that takes Python commands over a network connection. Eventually, the IPython engine will be a full IPython interpreter, but for now, it is a regular Python interpreter. The engine can also handle incoming and outgoing Python objects sent over a network connection. When multiple engines are started, parallel and distributed computing becomes possible. An important feature of an IPython engine is that it blocks while user code is being executed. Read on for how the IPython controller solves this problem to expose a clean asynchronous API to the user."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### IPython controller"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The IPython controller processes provide an interface for working with a set of engines. At a general level, the controller is a collection of processes to which IPython engines and clients can connect. The controller is composed of a **Hub** and a collection of **Schedulers**. These Schedulers are typically run in separate processes but on the same machine as the Hub, but can be run anywhere from local threads or on remote machines.\n",
    "\n",
    "The controller also provides a single point of contact for users who wish to utilize the engines connected to the controller. There are different ways of working with a controller. In IPython, all of these models are implemented via the **View.apply()** method, after constructing **View** objects to represent subsets of engines. The two primary models for interacting with engines are:\n",
    "\n",
    "- A **Direct** interface, where engines are addressed explicitly.\n",
    "- A **LoadBalanced** interface, where the Scheduler is trusted with assigning work to appropriate engines.\n",
    "Advanced users can readily extend the View models to enable other styles of parallelism.\n",
    "\n",
    "Note:\n",
    "```\n",
    "A single controller and set of engines can be used with multiple models simultaneously. This opens the door for lots of interesting things.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Hub"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The center of an IPython cluster is the Hub. This is the process that keeps track of engine connections, schedulers, clients, as well as all task requests and results. The primary role of the Hub is to facilitate queries of the cluster state, and minimize the necessary information required to establish the many connections involved in connecting new clients and engines.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Schedulers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All actions that can be performed on the engine go through a Scheduler. While the engines themselves block when user code is run, the schedulers hide that from the user to provide a fully asynchronous interface to a set of engines."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## IPython client and views"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is one primary object, the **Client**, for connecting to a cluster. For each execution model, there is a corresponding **View**. These views allow users to interact with a set of engines through the interface. Here are the two default views:\n",
    "\n",
    "- The **DirectView** class for explicit addressing.\n",
    "- The **LoadBalancedView** class for destination-agnostic scheduling."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting started: Starting the IPython controller and engines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will need to start the IPython controller and four IPython engines. The simplest way of doing this is to use the ipcluster command.\n",
    "\n",
    "There are many more options available, but they are outside the scope of this introduction.\n",
    "\n",
    "The ipcluster command provides a simple way of starting a controller and engines in the following situations:\n",
    "\n",
    "- When the controller and engines are all run on localhost. This is useful for testing or running on a multicore computer.\n",
    "- When engines are started using the mpiexec command that comes with most MPI [MPI] implementations\n",
    "- When engines are started using the Portable batch system (PBS) (or other qsub systems, such as SGE).\n",
    "- When the controller is started on localhost and the engines are started on remote nodes using ssh.\n",
    "- When engines are started using the Windows HPC Server batch system.\n",
    "\n",
    "**Note:**\n",
    "    \n",
    "    Currently ipcluster requires that the IPYTHONDIR/profile_<name>/security directory lives on a shared filesystem that is seen by both the controller and engines. If you don’t have a shared file system you will need to use ipcontroller and ipengine directly.\n",
    "\n",
    "\n",
    "Under the hood, ipcluster just uses ipcontroller and ipengine to perform the steps described above.\n",
    "\n",
    "The simplest way to use ipcluster requires no configuration, and will launch a controller and a number of engines on the local machine. If you are using anaconda, it is warmly advised to first source your conda environment. In this way, issues with libraries are avoided. To start one controller and 4 engines on localhost, just do:\n",
    "\n",
    "```\n",
    "On Linux / OS X:\n",
    "$ source activate YOUR_ENV_NAME\n",
    "$ ipcluster start --n=4\n",
    "\n",
    "On Windows:\n",
    "$ activate YOUR_ENV_NAME\n",
    "$ ipcluster start --n=4\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configuring an IPython cluster"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-04-05T09:44:09.334468",
     "start_time": "2017-04-05T11:44:09.329236+02:00"
    }
   },
   "source": [
    "Cluster configurations are stored as profiles. You can create a new profile with:\n",
    "\n",
    "```\n",
    "$ ipython profile create --parallel --profile=myprofile\n",
    "```\n",
    "\n",
    "This will create the directory ```IPYTHONDIR/profile_myprofile```, and populate it with the default configuration files for the three IPython cluster commands. Once you edit those files, you can continue to call ipcluster/ipcontroller/ipengine with no arguments beyond ```profile=myprofile```, and any configuration will be maintained.\n",
    "\n",
    "There is no limit to the number of profiles you can have, so you can maintain a profile for each of your common use cases. The default profile will be used whenever the profile argument is not specified, so edit ```IPYTHONDIR/profile_default/*_config.py``` to represent your most common use case.\n",
    "\n",
    "The configuration files are loaded with commented-out settings and explanations, which should cover most of the available possibilities.\n",
    "\n",
    "Using ipcluster with your profile:\n",
    "\n",
    "```\n",
    "$ source activate YOUR_ENV_NAME\n",
    "$ ipcluster start --n=4 --profile='YOUR_IPYTHON_PROFILE'\n",
    "```\n",
    "\n",
    "I use different profiles in different notebooks, just to be certain that I load the calculations on the correct client."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# used libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-04-14T10:09:22.377300",
     "start_time": "2017-04-14T12:09:22.359318+02:00"
    },
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3.6.1 |Continuum Analytics, Inc.| (default, Mar 22 2017, 19:54:23) \n",
      "[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]\n",
      "sys.version_info(major=3, minor=6, micro=1, releaselevel='final', serial=0)\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "print(sys.version)\n",
    "print(sys.version_info)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-04-14T10:09:22.385392",
     "start_time": "2017-04-14T12:09:22.380218+02:00"
    },
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "using library ipyparallel version 6.0.2\n"
     ]
    }
   ],
   "source": [
    "import ipyparallel\n",
    "print('using library ' + ipyparallel.__name__ + ' version ' + ipyparallel.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-04-14T10:09:22.408343",
     "start_time": "2017-04-14T12:09:22.388600+02:00"
    },
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "using library ipykernel version 4.6.0\n"
     ]
    }
   ],
   "source": [
    "import ipykernel\n",
    "print('using library ' + ipykernel.__name__ + ' version ' + ipykernel.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-04-14T10:09:22.701335",
     "start_time": "2017-04-14T12:09:22.413363+02:00"
    },
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "using library numpy version 1.12.1\n"
     ]
    }
   ],
   "source": [
    "import numpy\n",
    "print('using library ' + numpy.__name__ + ' version ' + numpy.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-04-14T10:09:22.826160",
     "start_time": "2017-04-14T12:09:22.706440+02:00"
    },
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "using library matplotlib version 2.0.0\n"
     ]
    }
   ],
   "source": [
    "import matplotlib\n",
    "print('using library ' + matplotlib.__name__ + ' version ' + matplotlib.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-04-14T10:09:23.180887",
     "start_time": "2017-04-14T12:09:22.829500+02:00"
    },
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "using library pandas version 0.19.2\n"
     ]
    }
   ],
   "source": [
    "import pandas\n",
    "print('using library ' + pandas.__name__ + ' version ' + pandas.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python [conda env:ipyparallelseminar]",
   "language": "python",
   "name": "conda-env-ipyparallelseminar-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  },
  "toc": {
   "toc_cell": true,
   "toc_number_sections": true,
   "toc_threshold": 6,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}