0_Introduction.ipynb

{
 "metadata": {
  "name": "",
  "signature": "sha256:1bd0934e6bbd016ee0688744984fcf8c00c04b415bf73957bd9bd0f0ed969c52"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# Python for Data Analysis"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<hr size=3>\n",
      "\n",
      "[National Institute for Water and Atmospheric research](http://www.niwa.co.nz) \n",
      "\n",
      "Auckland, New Zealand \n",
      "\n",
      "Dates and times: \n",
      "\n",
      "+ Tuesday 28 January, 12:00 - 13:00\n",
      "+ Thursday 30 January, 12:00 - 13:00\n",
      "+ Monday 3 February, 14:00 - 15:00\n",
      "+ Tuesday 4 February, 12:00 - 13:00\n",
      "+ Wednesday 5 February, 12:00 - 13:00\n",
      "+ Friday 11 April 2014, 12:00 - 13:00\n",
      "\n",
      "\n",
      "<hr size=3>\n",
      "\n",
      "Nicolas Fauchereau (Nicolas.Fauchereau@gmail.com) [My academia web-page](http://niwa.academia.edu/NicolasFauchereau)\n",
      "\n",
      "All the notebooks in this seminar series are indexed at\n",
      "\n",
      "[https://github.com/nicolasfauchereau/NIWA_Python_seminars](https://github.com/nicolasfauchereau/NIWA_Python_seminars)\n",
      "\n",
      "If you have git installed, the easier is to clone my repository \n",
      "\n",
      "    git clone https://github.com/nicolasfauchereau/NIWA_Python_seminars.git\n",
      "\n",
      "If you are not a git user, simply download the tarball from [here](https://github.com/nicolasfauchereau/NIWA_Python_seminars/archive/master.zip) or copy the directory `NIWA_Python_seminars` from my S: drive."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<hr size=5>\n",
      "\n",
      "## Structure of the seminars\n",
      "\n",
      "### General\n",
      "\n",
      "2. An overview of the IPython notebook\n",
      "\n",
      "3. Introduction to Python\n",
      "\n",
      "### The Python *Scientific Stack*\n",
      "\n",
      "4. Numpy and Scipy\n",
      "\n",
      "5. Pandas\n",
      "    \n",
      "6. Statistical analysis and statistical modeling with Python: Scipy.stats and Statsmodels\n",
      "\n",
      "7. Machine Learning with Python and scikits-learn"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<hr size=5>\n",
      "## What we will cover \n",
      "\n",
      "+ Basic types, data structures and control flow in python\n",
      "\n",
      "+ Numpy arrays, fundamental building block of the scientific stack \n",
      "\n",
      "+ Scipy: statistical distributions, interpolation, some linear algebra\n",
      "\n",
      "+ Pandas: Series and DataFrames, data munging\n",
      "\n",
      "+ Getting data in and out of Python: txt, csv, xls, matlab, netcdf \u2026\n",
      "\n",
      "+ The fundamental API libraries for statistical modeling and statistical (aka *machine*) learning: scipy.stats, statsmodels, sklearn\n",
      "\n",
      "+ plotting with Matplotlib and mapping with Basemap will be covered in coming sessions\n",
      "---\n"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<hr size=5>\n",
      "## What we will NOT cover \n",
      "\n",
      "+ **Object Oriented Programming in Python** \n",
      "\n",
      "> + [www.google.com](www.google.com)\n",
      "> + [A good tutorial by Alan Gauld](http://www.freenetpages.co.uk/hp/alan.gauld/tutclass.htm)\n",
      "> + The [classes](http://docs.python.org/2/tutorial/classes.html) entry from the official Python doc \n",
      "> + A short [pdf](http://www1.gly.bris.ac.uk/~walker/PythonEarthSci/PfES_3.pdf) from Andrew Walker's [Python for Earth Scientists](http://www1.gly.bris.ac.uk/~walker/PythonEarthSci/) course\n",
      "\n",
      "\n",
      "+ **wrapping of `C, C++, Fortran` or the `Cython`** module  \n",
      "\n",
      "> + [using python as a *glue*](http://docs.scipy.org/doc/numpy/user/c-info.python-as-glue.html)\n",
      "> + [SWIG (Simplified Wrapper and Interface Generator)](http://www.swig.org/Doc1.3/Python.html)\n",
      "> + [Cython](http://cython.org)\n",
      "> + [f2py](http://www.f2py.com/)\n",
      "> + [A notebook on using C and Fortran with Python](http://nbviewer.ipython.org/github/jrjohansson/scientific-python-lectures/blob/master/Lecture-6A-Fortran-and-C.ipynb)\n",
      "\n",
      "+ **Anything *other* than the Python Scientific stack**  \n",
      "\n",
      "> + The Python [standard library](http://docs.python.org/2/library/): Python is *battery included*, will just touch upon the ```datetime``` module\n",
      "> + [requests](http://docs.python-requests.org/en/latest/): http for humans\n",
      "> + [python-pptx](https://python-pptx.readthedocs.org/en/latest/): automating the creation of powerpoint files\n",
      "> + have a look at the [python package index](https://pypi.python.org/pypi)\n",
      "\n",
      "+ **Real-time acquisition, physical programming**  \n",
      "\n",
      "> + [pyserial](http://pyserial.sourceforge.net): Module that encapsulates the access for the serial port\n",
      "> + [Real World Instrumentation, Automated Data Acquisition and Control Systems](http://www.amazon.com/Real-World-Instrumentation-Python-Acquisition/dp/0596809565): A book available on Amazon.\n",
      "> + [Programming Raspberry Pi: Getting Started with Python](http://www.monkmakes.com/?page_id=63): book (not free) with code (free)\n",
      "> + [PyVISA](https://pyvisa.readthedocs.org/en/latest/): The PyVISA package enables you to control all kinds of measurement equipment through various busses (GPIB, RS232, USB) with Python programs\n",
      "\n",
      "+ **Image processing**  \n",
      "\n",
      "> + PIL [(Python Image Library)](http://www.pythonware.com/products/pil/): Image processing capabilities to your Python interpreter.\n",
      "> + [scikit-image](http://scikit-image.org): more Image processing, built on top of Numpy / Scipy.\n",
      "\n",
      "+ **Symbolic maths**  \n",
      "\n",
      "> + The [SymPy](www.sympy.org) library is a Python library for symbolic mathematics. It supports polynomials, calculus, solving equations, etc\n",
      "\n",
      "+ **Spatial statistics**  \n",
      "\n",
      "> + [PySAL](http://pysal.readthedocs.org/en/v1.6/): PySAL is a cross-platform library of spatial analysis functions written in Python. It is intended to support the development of high level applications for spatial analysis.\n",
      "\n",
      "+ **Biology, ecology**  \n",
      "\n",
      "> + [Biopython](http://biopython.org/wiki/Main_Page): Biopython is a set of freely available tools for biological computation written in Python by an international team of developers. \n",
      "> + [GeoEco](https://pypi.python.org/pypi/GeoEco): Open source geoprocessing toolbox designed for coastal and marine researchers and GIS analysts who work with spatially-explicit ecological and oceanographic data. For Windows (> XP) only.\n",
      "\n",
      "---\n"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<hr size=5>\n",
      "## Acknowledgments\n",
      "\n",
      "In building this material, I have liberally '*borrowed*' from lecture notes, online notebooks, video recording of talks, code examples, articles, etc, that are freely available online, I would like to acknowledge in particular (list not exhaustive and in no particular order): \n",
      "\n",
      "+ [Fernando Perez](http://blog.fperez.org): Lead developper of [IPython](www.ipython.org), great advocate of [open science](http://en.wikipedia.org/wiki/Open_science) and reproducible research\n",
      "+ Brian Granger: Lead developper of the [IPython notebook](http://ipython.org/notebook.html)\n",
      "+ [Wes McKinney](http://blog.wesmckinney.com): Lead developper of Pandas, author of [Python for Data Analysis](http://shop.oreilly.com/product/0636920023784.do)\n",
      "+ [Christopher Fonnesbeck](http://biostat.mc.vanderbilt.edu/wiki/Main/ChrisFonnesbeck): Developper of [PYMC](http://pymc-devs.github.io/pymc/) (Bayesian statistical models and fitting algorithms, including Markov chain Monte Carlo)\n",
      "+ [Jake VanderPlas](http://www.astro.washington.edu/users/vanderplas/): Astronomer, contributor to [scikit-learn](http://scikit-learn.org)\n",
      "+ [Ga\u00ebl Varoquaux](http://gael-varoquaux.info): One of the main contributors to [scikit-learn](http://scikit-learn.org)\n",
      "+ [Olivier Grisel](http://ogrisel.com): Also main contributor to [scikit-learn](http://scikit-learn.org)\n",
      "+ [Travis Oliphant](http://technicaldiscovery.blogspot.co.nz): One of the lead developpers of Numpy / Scipy, and founder of [Continuum Analytics](http://continuum.io).\n",
      "+ J.R. Johansson http://dml.riken.jp/~rob/\n",
      "+ [Thomas Wiecki](http://twiecki.github.io/): developper of [PYMC3](https://github.com/pymc-devs/pymc), a fork of PYMC. \n",
      "+ [Skipper Seabold](http://jseabold.net/): The main developper of [statsmodels](http://statsmodels.sourceforge.net/)\n",
      "+ [Kevin Sheppard](http://www.kevinsheppard.com/wiki/Main_Page) from University of Oxford: author or the book [Introduction to Python for Econometrics, Statistics and Numerical Analysis: Second Edition](http://www.kevinsheppard.com/images/0/09/Python_introduction.pdf). \n",
      "\n",
      "And finally [John Hunter](http://blog.fperez.org/2013/07/in-memoriam-john-d-hunter-iii-1968-2012.html), He was the founder and lead developper of [Matplotlib](http://matplotlib.org), a pivotal library to make Python a viable free and open-source alternative to commercial scientific software, and very sadly passed away in 2012.\n",
      "\n",
      "I encourage you - if you carry on using Python for science - to look up these people on google, have a look at their github repositories and the projects they contribute to, and follow them on twitter, some of them (*e.g.* Jake VanderPlas) have also very informative blogs. You can also donate to the [John Hunter memorial fund](http://numfocus.org/johnhunter.html) as a way to give back to an important contributor to the Python scientific community.\n",
      "\n",
      "---"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<hr size=5>\n",
      "## Resources \n",
      "\n",
      "### for Matlab and R users \n",
      "\n",
      "+ [Moving from MATLAB matrices to NumPy arrays - A Matrix Cheatsheet](http://sebastianraschka.com/Articles/2014_matlab_vs_numpy.html)\n",
      "\n",
      "+ [NumPy for Matlab Users](http://wiki.scipy.org/NumPy_for_Matlab_Users)\n",
      "\n",
      "\n",
      "### Some URLs and blogs\n",
      "\n",
      "+ [Python Scientific Lectures Notes](http://scipy-lectures.github.io): Tutorial material on the scientific Python ecosystem, a quick introduction to central tools and techniques.\n",
      "\n",
      "+ [Python for earth scientists](http://www1.gly.bris.ac.uk/~walker/PythonEarthSci/): A two afternoons course by Andrew Walker (University of Bristol) on Python in the earth sciences. \n",
      "\n",
      "+ [oceanpython](http://oceanpython.org/): Python for oceanography \n",
      "\n",
      "+ [PyAOS](http://pyaos.johnny-lin.com/): Python for the Atmospheric and Oceanic Sciences \n",
      "\n",
      "+ [python4oceanographers](http://ocefpaf.github.io/): Learn python with examples applied to marine sciences.\n",
      "\n",
      "+ [Pythonic perambulations](http://jakevdp.github.io): A blog by Jake VanderPlas\n",
      "\n",
      "### Github repositories: notebooks and accompanying material\n",
      "\n",
      "+ [A gallery of interesting IPython Notebooks](https://github.com/ipython/ipython/wiki/A-gallery-of-interesting-IPython-Notebooks): \"...a curated collection of IPython notebooks that are notable for some reason.\"\n",
      "\n",
      "+ [Scikit-learn tutorial](https://github.com/jakevdp/sklearn_scipy2013): Files and other info associated with the Scipy 2013 scikit-learn tutorial developped by Ga\u00ebl Varoquaux, Olivier Grisel and Jake VanderPlas.\n",
      "\n",
      "+ [Statistical Analysis tutorial](https://github.com/fonnesbeck/statistical-analysis-python-tutorial) from Chris. Fonnesbeck. \n",
      "\n",
      "+ [AstroML: Machine Learning and Data Mining for Astronomy](http://www.astroml.org): A library and tutorial by Jake VanderPlas and co-authors, accompanying the book [Statistics, Data Mining, and Machine Learning in Astronomy](http://www.amazon.com/Statistics-Mining-Machine-Learning-Astronomy/dp/0691151687/)\n",
      "\n",
      "+ [Scientific Python lectures from Robert Johansson](https://github.com/jrjohansson/scientific-python-lectures)\n",
      "\n",
      "+ [data Science in Python](http://blog.yhathq.com/posts/data-science-in-python-tutorial.html): A series of annotated notebooks on data science (i.e. geared towards machine learning) in python\n",
      "\n",
      "+ [http://earthpy.org/](http://earthpy.org/): EarthPy is a collection of IPython notebooks with examples of Earth Science related Python code\n",
      "\n",
      "### Some books \n",
      "\n",
      "+ [Python for Data Analysis](http://shop.oreilly.com/product/0636920023784.do): From Wes McKinney (Developer of [Pandas](www.pandas.pydata.org) ... *you are welcome to borrow it from me if it comes back to my desk in a reasonable time*.)\n",
      "\n",
      "+ [Think stats](http://www.greenteapress.com/thinkstats/): Probability and statistics for programmers, from Allen Downey, pdf available for free. \n",
      "\n",
      "+ [Think complexity](http://www.greenteapress.com/compmod/): Complexity science (graphs, cellular automata, agent-based models), from Allen Downey, pdf available for free.\n",
      "\n",
      "+ [Python in hydrology](http://www.greenteapress.com/pythonhydro/pythonhydro.html): A book freely available in pdf, from Sat Kumar Tomer.\n",
      "\n",
      "+ [Programming collective intelligence](http://shop.oreilly.com/product/9780596529321.do): By Toby Segaran, Good intro on (general) Machine Learning algorithms. *Can borrow from me*. \n",
      "\n",
      "+ [Machine Learning in action](http://www.manning.com/pharrington/): By Peter Harrington. *Can borrow from me*. \n",
      "\n",
      "+ [Introduction to Python for Econometrics, Statistics and Numerical Analysis: Second Edition](http://www.kevinsheppard.com/images/0/09/Python_introduction.pdf): By Kevin Sheppard, Oxford Uni.\n",
      "\n",
      "+ [A Hands-On Introduction to Using Python in the Atmospheric and Oceanic Sciences](http://www.johnny-lin.com/pyintro/): By [Johnny Lin](http://www.johnny-lin.com), Professor of physics and head of the Climate Research Group at North Park University.\n",
      "\n",
      "+ [principles of planetary climate](http://geosci.uchicago.edu/~rtp1/PrinciplesPlanetaryClimate/index.html): by Ray Pierrehumbert. Excellent book on the physics of planetary climates, with freely downloadable python code to follow the examples given in the book. \n",
      "\n",
      "### Some interesting libraries, built on top of the main Scientific stack\n",
      "\n",
      "+ [PYMC](http://pymc-devs.github.io/pymc/): By Chris Fonnesbeck, Bayesian statistical models and fitting algorithms, including Markov chain Monte Carlo. \n",
      "\n",
      "+ [Seaborn](http://www.stanford.edu/~mwaskom/software/seaborn/): Statistical data visualization, by Michael Waskom. Its [graphical representation of linear models](http://www.stanford.edu/~mwaskom/software/seaborn/linear_models.html) is particularly interesting. \n",
      "\n",
      "+ [ggplot](https://github.com/yhat/ggplot/): For [R](http://cran.r-project.org) users, a 'port' of the [ggplot2](http://ggplot2.org) package to Python, see [here](http://blog.yhathq.com/posts/ggplot-0.4-released.html) for what's new in the latest release. \n",
      "\n",
      "+ [prettyplotlib](http://olgabot.github.io/prettyplotlib/): By Olga Botvinnik \"... painlessly create beautiful matplotlib plots\".\n",
      "\n",
      "+ [coards](https://pypi.python.org/pypi/coards): A COARDS compliant time parser. See also [netcdftime](http://netcdf4-python.googlecode.com/svn/trunk/docs/netcdftime.netcdftime-module.html) which is part of the [NetCDF4 module](https://code.google.com/p/netcdf4-python/)\n",
      "\n",
      "+ [seawater](http://www.imr.no/~bjorn/python/seawater/index.html): Similar to the MATLAB toolboxes SEAWATER from CSIRO and parts of OCEANS from Woods Hole Institute.\n",
      "\n",
      "+ [fluid](https://pypi.python.org/pypi/fluid): Procedures to study fluids on Python, focused for oceanography, meteorology and related sciences.\n",
      "\n",
      "+ [kyPyWavelet](https://github.com/regeirk/kPyWavelet): Continuous wavelet transform module for Python *ala* Torrence and Compo. Some manual edits were necessary to make it work for me ... \n",
      "\n",
      "+ [pyresample](https://code.google.com/p/pyresample/): Resampling (reprojection) of geospatial image data in Python\n",
      "\n",
      "+ [Rpy2](http://rpy.sourceforge.net/rpy2.html): calling R from Python\n",
      "\n",
      "### Some articles on open and reproducible research\n",
      "\n",
      "+ [Reproducible Research in Computational Science](http://dx.doi.org/10.1126/science.1213847), Roger D. Peng, Science 334, 1226 (2011).\n",
      "\n",
      "+ [Shining Light into Black Boxes](http://dx.doi.org/10.1126/science.1218263), A. Morin et al., Science 336, 159-160 (2012).\n",
      " \n",
      "+ [The case for open computer programs](http://dx.doi.org/doi:10.1038/nature10836), D.C. Ince, Nature 482, 485 (2012).\n",
      "\n",
      "+ [Best practices for scientific computing](http://www.plosbiology.org/article/info%3Adoi%2F10.1371%2Fjournal.pbio.1001745): Paper in PLOS Biology exposing some of the tools and methods to build better Scientific software.\n",
      "\n",
      "\n",
      "\n",
      "---"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<center>\n",
      "![](files/images/python.png)\n",
      "</center>"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%load_ext load_style"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 1
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%load_style talk.css"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<style>\n",
        "\n",
        ".rendered_html {\n",
        "    font-family: \"proxima-nova\", helvetica;\n",
        "    font-size: 130%;\n",
        "    line-height: 1.5;\n",
        "}\n",
        "\n",
        ".rendered_html h1 {\n",
        "    margin: 0.25em 0em 0.5em;\n",
        "    color: #015C9C;\n",
        "    text-align: center;\n",
        "    line-height: 1.2; \n",
        "    page-break-before: always;\n",
        "}\n",
        "\n",
        ".rendered_html h2 {\n",
        "    margin: 1.1em 0em 0.5em;\n",
        "    color: #26465D;\n",
        "    line-height: 1.2;\n",
        "}\n",
        "\n",
        ".rendered_html h3 {\n",
        "    margin: 1.1em 0em 0.5em;\n",
        "    color: #002845;\n",
        "    line-height: 1.2;\n",
        "}\n",
        "\n",
        ".rendered_html li {\n",
        "    line-height: 1.5; \n",
        "}\n",
        "\n",
        "/*.prompt {\n",
        "    font-size: 120%; \n",
        "}*/\n",
        "\n",
        ".CodeMirror-lines {\n",
        "    font-size: 110%; \n",
        "}\n",
        "\n",
        "/*.output_area {\n",
        "    font-size: 120%; \n",
        "}*/\n",
        "\n",
        "/*#notebook {\n",
        "    background-image: url('files/images/witewall_3.png');\n",
        "}*/\n",
        "\n",
        "h1.bigtitle {\n",
        "    margin: 4cm 1cm 4cm 1cm;\n",
        "    font-size: 300%;\n",
        "}\n",
        "\n",
        "h3.point {\n",
        "    font-size: 200%;\n",
        "    text-align: center;\n",
        "    margin: 2em 0em 2em 0em;\n",
        "    #26465D\n",
        "}\n",
        "\n",
        ".logo {\n",
        "    margin: 20px 0 20px 0;\n",
        "}\n",
        "\n",
        "a.anchor-link {\n",
        "    display: none;\n",
        "}\n",
        "\n",
        "h1.title { \n",
        "    font-size: 250%;\n",
        "}\n",
        "\n",
        "</style>"
       ],
       "metadata": {},
       "output_type": "display_data",
       "text": [
        "<IPython.core.display.HTML at 0x1069dfed0>"
       ]
      }
     ],
     "prompt_number": 5
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": []
    }
   ],
   "metadata": {}
  }
 ]
}