{
 "metadata": {
  "name": "Part 1 - Overview and Architecture"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Overview and getting started"
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Introduction"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "IPython is a tool for interactive and exploratory computing.\n",
      "We have seen that IPython's kernel provides a mechanism for interactive\n",
      "*remote* computation, and we have extended this same mechanism for\n",
      "interactive remote *parallel* computation, simply by having multiple\n",
      "kernels."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "![wideview](files/figures/wideView400.png)"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "`IPython.parallel` does solve:\n",
      "\n",
      "* Code distribution and execution\n",
      "* Code scheduling with load balancing, dependencies, etc.\n",
      "\n",
      "`IPython.parallel` does **not** solve:\n",
      "\n",
      "* Message passing (MPI, ZeroMQ/PyZMQ)\n",
      "* Data management, distribution, io"
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "IPython.parallel Glossary"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "* Controller - collection of processes coordinating an IPython cluster\n",
      "* Hub - central controller process that monitors and manages the setate of the cluster\n",
      "* Schedulers - controller processes that relay messages between clients and engines\n",
      "\n",
      "* Kernel - an IPython instance where you run your code\n",
      "* Engine - special case of a Kernel used in IPython.parallel, which connects instead of listens\n",
      "\n",
      "* Client - user-level object that manages communication with a cluster\n",
      "* View - user-level object that presents the API for execution\n",
      "* AsyncResult - object that wraps pending results\n",
      "* apply - View method for calling a function on engine(s)\n",
      "* execute - View method for running code-as-text on engine(s)\n",
      "* push - View method for sending data\n",
      "* pull - View method for retrieving data\n",
      "\n"
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Architecture overview"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "At a high level, there are three basic components to parallel IPython:\n",
      "\n",
      "* Engine(s) - the remote or distributed processes where your code runs.\n",
      "* Client - your interface to running code on Engines.\n",
      "* Controller - the collection of processes that coordinate Engines and Clients.\n",
      "\n",
      "These components live in the `IPython.parallel` package and are installed with IPython."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### IPython engine\n",
      "\n",
      "The Engine is simply a remote Python namespace where your code is run,\n",
      "and is identical to the kernel used elsewhere in IPython.\n",
      "\n",
      "It can do all the magics, pylab integration, and everything else you can do in a regular IPython session."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### IPython controller\n",
      "\n",
      "The Controller is a collection of processes:\n",
      "\n",
      "* Schedulers relay messages between Engines and Clients.\n",
      "* The Hub monitors the cluster state.\n",
      "\n",
      "Together, these processes provide a single connection point for your clients and engines.\n",
      "Each Scheduler is a small GIL-less function in C provided by pyzmq (the Python load-balancing scheduler being an exception).\n",
      "\n",
      "The Hub can be viewed as an \u00fcber-logger,\n",
      "which monitors all communication between clients and engines,\n",
      "and can log to a database (e.g. SQLite or MongoDB) for later retrieval or resubmission.\n",
      "The Hub is not involved in execution in any way,\n",
      "and a slow Hub cannot slow down submission of tasks."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#### Schedulers\n",
      "\n",
      "All actions that can be performed on the engine go through a Scheduler.\n",
      "While the engines themselves block when user code is run,\n",
      "the schedulers hide that from the user to provide a fully asynchronous interface to a set of engines.  "
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "\u00d8MQ and PyZMQ"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "All of this is implemented with the lovely \u00d8MQ messaging library,\n",
      "and pyzmq, the lightweight Python bindings, which allows very fast\n",
      "zero-copy communication of objects like numpy arrays."
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "IPython client and views"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "There is one primary object, the `Client`, for connecting to a cluster.\n",
      "For each execution model there is a corresponding `View`,\n",
      "and you determine how your work should be executed on the cluster by creating different views\n",
      "or manipulating attributes of views.\n",
      "\n",
      "The two basic views:\n",
      "\n",
      "-   The `DirectView` class for explicitly running code on particular engine(s).\n",
      "-   The `LoadBalancedView` class for destination-agnostic scheduling.\n",
      "\n",
      "You can use as many views of each kind as you like, all at the same time."
     ]
    },
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Getting Started"
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Starting the IPython controller and engines"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "To follow along with this tutorial, you will need to start the IPython\n",
      "controller and four IPython engines. The simplest way of doing this is\n",
      "to use the `ipcluster` command:\n",
      "\n",
      "    $ ipcluster start -n 4\n",
      "\n",
      "There isn't time to go into it here, but ipcluster can be used to start engines\n",
      "and the controller with various batch systems including:\n",
      "\n",
      "* SGE\n",
      "* PBS\n",
      "* LSF\n",
      "* MPI\n",
      "* SSH\n",
      "* WinHPC\n",
      "\n",
      "More information on starting and configuring the IPython cluster in \n",
      "[the IPython.parallel docs](http://ipython.org/ipython-doc/dev/parallel/parallel_process.html).\n",
      "\n",
      "Once you have started the IPython controller and one or more engines,\n",
      "you are ready to use the engines to do something useful. \n",
      "\n",
      "To make sure everything is working correctly, let's do a very simple demo:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from IPython import parallel\n",
      "rc = parallel.Client()\n",
      "rc.block = True"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 1
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "rc.ids"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 4,
       "text": [
        "[0, 1, 2, 3]"
       ]
      }
     ],
     "prompt_number": 4
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Let's define a simple function"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def mul(a,b):\n",
      "    return a*b"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 5
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "mul(5,6)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 6,
       "text": [
        "30"
       ]
      }
     ],
     "prompt_number": 6
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "What does it look like to call this function remotely?\n",
      "\n",
      "Just turn `f(*args, **kwargs)` into `view.apply(f, *args, **kwargs)`!"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "dv = rc[0]; dv"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 7,
       "text": [
        "<DirectView 0>"
       ]
      }
     ],
     "prompt_number": 7
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "dv.apply(mul, 5, 6)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 8,
       "text": [
        "30"
       ]
      }
     ],
     "prompt_number": 8
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "And the same thing in parallel?"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "rc[:].apply(mul, 5, 6)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 9,
       "text": [
        "[30, 30, 30, 30]"
       ]
      }
     ],
     "prompt_number": 9
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Python has a builtin map for calling a function with a sequence of arguments"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "map(mul, range(1,10), range(2,11))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 10,
       "text": [
        "[2, 6, 12, 20, 30, 42, 56, 72, 90]"
       ]
      }
     ],
     "prompt_number": 10
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "So how do we do this in parallel?"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "view = rc.load_balanced_view()\n",
      "view.map(mul, range(1,10), range(2,11))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 11,
       "text": [
        "[2, 6, 12, 20, 30, 42, 56, 72, 90]"
       ]
      }
     ],
     "prompt_number": 11
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "I'll go into further detail about different views, and asynchronous communication later.\n",
      "First, let's peek at the performance of the IPython.parallel infrastructure, \n",
      "so we can see what level of granularity we can consider using."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Map\n",
      "\n",
      "A simple test of how big your tasks need to be is to run a simple map\n",
      "with minimal tasks of a certain time (e.g. `time.sleep()`).\n",
      "\n",
      "These tests were done on [AWS](http://aws.amazon.com/) extra-large\n",
      "instances with the help of\n",
      "[StarCluster](http://web.mit.edu/stardev/cluster/), so the IO and CPU\n",
      "performance are not impressive compared to a physical cluster.\n",
      "\n",
      "* The dotted line is 50% overhead.\n",
      "* Above the line is <50% overhead.\n",
      "* Below the line is >50% overhead.\n",
      "\n",
      "![map](files/figures/map.png)"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "This shows runs for jobs ranging from 1 to 128 ms, on 4,31,and 63\n",
      "engines. On this system, millisecond jobs are clearly too small, but by\n",
      "the time individual tasks are \\> 100 ms, IPython overhead is negligible.\n",
      "\n",
      "For more details and to see how to generate these metrics on your own systems, \n",
      "you can view and run the Performance notebook.\n",
      "\n",
      "Now let's see how we use it for remote execution."
     ]
    }
   ],
   "metadata": {}
  }
 ]
}