prac01_introduction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Practical 1: Installing and Introduction to Python's Data Mining Libraries\n",
    "\n",
    "### In this practical\n",
    "1. [Why Python?](#whypython)\n",
    "2. [Installing Anaconda](#install)\n",
    "3. [Process flow of predictive mining in Python](#processflow)\n",
    "4. [Interactive prototyping in ipython](#ipython)\n",
    "5. [Defining problem and purpose of data mining](#purpose)\n",
    "---\n",
    "\n",
    "**Written by Hendi Lie (h2.lie@qut.edu.au) and Richi Nayak (r.nayak@qut.edu.au). All rights reserved.**\n",
    "\n",
    "This practical note introduces you to Python and its common machine learning libraries. Python is a high-level, interpreted programming language. It is used for wide range of purposes, from web servers to scientific computing. It’s syntax allows easy and quick readability and program creativity.\n",
    "\n",
    "The practical sessions in this subject use Python for data manipulation and analytics. This is **NOT** a Python unit, therefore, it will not cover the basic syntax of Python. Fortunately, there is a lot of resources available online for\n",
    "learning elementary Python. If you are new to Python, we recommend you to familiarise yourself before Practical 2.\n",
    "\n",
    "More specifically, we will use Python 3 in this unit. All examples are written using Python 3.5.2, but any version of Python 3 above 3.4 should work just fine."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Why Python? <a name=\"whypython\"></a>\n",
    "\n",
    "There exist many different languages, tools and software to perform data mining tasks. Some of them are R, Weka, Orange, Knime Analytics, SAS, Oracle Data Miner, Microsoft SQLServer Data Mining and Rapid Miner. Python is arguably the fastest growing and most widely used programming language/tool alongside R and Julia. There are a number of reasons for this popularity:\n",
    "\n",
    "### 1.1. Interpreted language\n",
    "Python is designed as an interpreted language which allows users to test and prototype models easily and quickly. With iPython interpreter, results of each line of code can immediately be checked. An interactive Jupyter notebook can also be created to share insights and code through browser interface. These tutorial notes are written in Jupyter notebooks.\n",
    "\n",
    "### 1.2. Open-source\n",
    "Python is free and has no ties to any propertiary/corporate technologies, which makes Python the top choice for students, academics and startups. There are some libraries/technologies in Python that are tied to a company (e.g. Jupyter, tensorflow), but they are still available almost in all use cases for free.\n",
    "\n",
    "### 1.3. Wide, cutting edge support for almost anything\n",
    "\n",
    "There exist a vast range of actively updated libraries for almost every data mining task.\n",
    "\n",
    "* **pandas** for data wrangling and preprocessing ([link](http://pandas.pydata.org/))\n",
    "* **scikit-learn** for supervised and unsupervised learning ([link](http://scikit-learn.org/stable/))\n",
    "* **numpy** for matrix manipulation ([link](http://www.numpy.org/))\n",
    "* **seaborn** and **matplotlib** for visualisation ([link](https://seaborn.pydata.org/)) ([link2](https://matplotlib.org/))\n",
    "* **ipython** for interactive prototyping ([link](https://ipython.org/))\n",
    "* **jupyter notebook** for interactive, web-based prototyping ([link](http://jupyter.org/))\n",
    "* **graphviz** and **pydot** for visualising sklearn model structures.\n",
    "\n",
    "### 1.4. Production ready\n",
    "Models and pipelines built with Python are very suitable to deployment in production systems. We will talk about deploying models and data pipelines in a practical later in this unit."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Installing Anaconda <a name=\"install\"></a>\n",
    "\n",
    "In this unit, we will use many data science libraries in Python. We recommend you to install Anaconda, a data science package for Python. It contains many essential data science libraries and is aimed to simplify the installation process. At the time of these notes, all libraries mentioned above are in Anaconda distribution except Seaborn.\n",
    "\n",
    "### 2.1. For Windows users\n",
    "\n",
    "For Windows users, simply download Anaconda and install it [link](https://www.anaconda.com/download/). Choose the latest Python3 version for Windows.\n",
    "\n",
    "Once you installed it, go to **Start-Anaconda3-Anaconda Prompt**. Type `conda install seaborn` to install Seaborn.\n",
    "\n",
    "To install visualisation libraries for decision trees (practical 2), follow these steps:\n",
    "1. Download graphviz for windows from http://www.graphviz.org/download/ and install it.\n",
    "3. Add the graphviz bin folder to the PATH system environment variable (Example: \"C:\\Graphviz2.38\\bin\"). Tutorial here: http://windowsitpro.com/systems-management/how-can-i-add-new-folder-my-system-path.\n",
    "4. Make sure git is installed in your system.\n",
    "5. Go to Anaconda Prompt using start menu (Make sure to right click and select \"Run as Administrator\". We may get permission issues if Prompt as not opened as Administrator)\n",
    "6. Execute the command: `conda install graphviz`\n",
    "7. Execute the command: `pip install git+https://github.com/nlhepler/pydot.git`\n",
    "8. Execute the command `conda list` and make sure pydot and graphviz modules are listed.\n",
    "\n",
    "\n",
    "### 2.2. For Linux Users\n",
    "\n",
    "For Linux users, go to Anaconda website to get the installation file [link](https://www.anaconda.com/download/). Choose the latest Python3 version. It should download an `.sh` file. Once the download process is finished, open your terminal and give execution permission with command `chmod +x [path_to_the_downloaded file]`. Run it to install by `./[path_to_downloaded file]`.\n",
    "\n",
    "Once the installation process is finished, ensure it is included in your bash path. Open your terminal and install Seaborn by typing `conda install seaborn`.\n",
    "\n",
    "To install visualisation libraries for decision trees (practical 2), follow these steps:\n",
    "1. Install graphviz for linux through `sudo apt-get install graphviz`\n",
    "2. Install graphviz for Python through `conda install graphviz`\n",
    "3. Install pydot using `pip install pydot`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Process flow for predictive mining using Python<a name=\"processflow\"></a>\n",
    "![Predictive mining process flow in Python](https://s3-ap-southeast-2.amazonaws.com/dataminingtuts/process_flow_python.png  \"Predictive mining process flow in Python\")\n",
    "\n",
    "The diagram above presents the steps we will take in this unit to perform predictive mining on the dataset. It is a standard data mining process that includes business understanding, data understanding, data pre-processing, data mining and post-processing. Some of these steps are done iteratively. The first and most important step is to define the underlying task and purpose of data mining. You need to ask questions such as:\n",
    "\n",
    "* What kind of data do we have?\n",
    "* Why are we performing predictive mining on this data?\n",
    "* What information are we trying to predict?\n",
    "* How could the stakeholders (including yourself) use the insights we gained from the data mining?\n",
    "\n",
    "\n",
    "Once the business problem is defined, and the corresponding data is identified and acquired, the next step is to explore the data. In this step, we attempt to understand common patterns and distributions in the data. We should also identify data quality problems in the dataset, such as noise and missing values, to be cleaned and processed out. The data exploration and quality enhancement steps will be performed mainly using `pandas` with some help from `sklearn`'s preprocessing modules.\n",
    "\n",
    "After the data is clean, models can be built to perform in-depth analysis. There are two broad categories of data mining: **Predictive Mining** (e.g. classification and regression with decision tree, regression and neural network) and **Descriptive Mining** (e.g. clustering and association mining). Many algorithms belonging to each of the categories are available in `sklearn`, each with its own characteristics. The upcoming practical notes will explore some of these algorithms in detail.\n",
    "\n",
    "Data mining outcomes ate best understood when accompanied with graphs and charts of patterns and trends identified in the data. Visualisation allows us to understand the data better. In this unit, all visualisations will be done using `seaborn` and `matplotlib` with data presented by `pandas` DataFrames."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Interactive prototyping with ipython<a name=\"ipython\"></a>\n",
    "\n",
    "`ipython` is an interactive Python shell designed for fast prototyping. In data mining/machine learning, many analysts use ipython to quickly review the data and process they are working on.\n",
    "\n",
    "### For you who are using Anaconda\n",
    "\n",
    "To start the ipython console, go to Start-Anaconda3-IPython. It will start by default on your document folder. If you wish to save your projects on another directory, change the current directory using `cd \"your directory path\"`.\n",
    "\n",
    "![Starting IPython from Anaconda in Windows](http://dataminingtuts.s3.amazonaws.com/anaconda_ipython.png \"Starting IPython from Anaconda in Windows\")\n",
    "\n",
    "### For you who are using Linux/Unix and installed the libraries manually\n",
    "\n",
    "We can call ipython the same way as we call the python interpreter itself:\n",
    "\n",
    "```bash\n",
    "ipython\n",
    "```\n",
    "\n",
    "```bash\n",
    "# Output\n",
    "Python 3.5.2 (default, Nov 17 2016, 17:05:23) \n",
    "Type 'copyright', 'credits' or 'license' for more information\n",
    "IPython 6.1.0 -- An enhanced Interactive Python. Type '?' for help.\n",
    "\n",
    "In [1]: \n",
    "```\n",
    "\n",
    "All examples in this unit are shown using ipython console.\n",
    "\n",
    "## 5. Defining problem and purpose of data mining process<a name=\"purpose\"></a>\n",
    "\n",
    "The following business problem will be solved via the predictive modelling/mining in the next few practical notes.\n",
    "\n",
    "**Business Scenario:** A national veterans organisation is seeking to improve their donation solicitations by targeting the potential donors. By only focusing on the supposedly donors, less money can be spent on solicitation efforts and more money can be available for charitable endeavors. Of particular interest is the class of individuals identified as lapsing donors. They have ran a greeting card mailing campaign called **Veteran**. The organisation now seeks to classify its lapsing donors based on their responses to this campaign. With this classification, a decision can be made to either solicit or ignore a lapsing individual in the next year campaign.\n",
    "\n",
    "Your task, as a data science professional, to use this dataset to understand the patterns of donation and identify supposedly donors to improve the solicitation effort.\n",
    "\n",
    "The `Veteran` dataset is available in `datasets/veteran.csv` file. Before we build the predictive models, we will examine the dataset to understand its basic characteristics such as data dimensions, attributes nature, data distribution, outliers, data quality etc.\n",
    "\n",
    "Import the dataset into the `ipython` console with `pandas`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "df = pd.read_csv('datasets/veteran.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the dataset is imported, we can start by looking at the columns/variables available. We can use `.info()` function for this purpose.\n",
    "\n",
    "> #### Essential classes and functions of Pandas\n",
    "\n",
    "> * Function **pandas.read_csv()** reads `.csv` file and return a **DataFrame** object. There are other read functions in pandas, including `read_json()`, `read_excel()`, `read_sas()` and even `read_sql()`.\n",
    "\n",
    "> * **pandas.DataFrame** is the primary data object in `pandas`. It is basically a two-dimensional tabular data structure that you can modify, perform arithmetic operations on and give labels to axes (rows and columns). Each column in a DataFrame is called a Series.\n",
    "\n",
    "> * Function **pandas.DataFrame.info()** provide concise summary of a DataFrame, such as number of entries (rows), data columns and their respective data types and memory usage."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 9686 entries, 0 to 9685\n",
      "Data columns (total 28 columns):\n",
      "TargetB             9686 non-null int64\n",
      "ID                  9686 non-null int64\n",
      "TargetD             4843 non-null float64\n",
      "GiftCnt36           9686 non-null int64\n",
      "GiftCntAll          9686 non-null int64\n",
      "GiftCntCard36       9686 non-null int64\n",
      "GiftCntCardAll      9686 non-null int64\n",
      "GiftAvgLast         9686 non-null float64\n",
      "GiftAvg36           9686 non-null float64\n",
      "GiftAvgAll          9686 non-null float64\n",
      "GiftAvgCard36       7906 non-null float64\n",
      "GiftTimeLast        9686 non-null int64\n",
      "GiftTimeFirst       9686 non-null int64\n",
      "PromCnt12           9686 non-null int64\n",
      "PromCnt36           9686 non-null int64\n",
      "PromCntAll          9686 non-null int64\n",
      "PromCntCard12       9686 non-null int64\n",
      "PromCntCard36       9686 non-null int64\n",
      "PromCntCardAll      9686 non-null int64\n",
      "StatusCat96NK       9686 non-null object\n",
      "StatusCatStarAll    9686 non-null int64\n",
      "DemCluster          9686 non-null int64\n",
      "DemAge              7279 non-null float64\n",
      "DemGender           9686 non-null object\n",
      "DemHomeOwner        9686 non-null object\n",
      "DemMedHomeValue     9686 non-null int64\n",
      "DemPctVeterans      9686 non-null int64\n",
      "DemMedIncome        9686 non-null int64\n",
      "dtypes: float64(6), int64(19), object(3)\n",
      "memory usage: 2.1+ MB\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "print(df.info())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dataset contains 29 variables including ID, demographics of members, donation history of members and many others. There are two possible target variables that can be used for predicting donor patterns: **TARGETB** (a binary variable stating whether a person is a lapsing donor or not); and **TARGETD** (a continuous variable that records the donation amount given in response to the mailing campaign).\n",
    "\n",
    "Data exploration assists us in answering some of questions such as:\n",
    "\n",
    "**What kind of data do we have?**\n",
    "\n",
    "A total of 29 variables with various information about the donors.\n",
    "\n",
    "**Why are we performing predictive mining on this data?**\n",
    "\n",
    "We would like to find possible lapsing donors to improve the donation solicitation campaign of the concerned organisation.\n",
    "\n",
    "**What information are we trying to predict?**\n",
    "\n",
    "Whether a person is a possible lapsing donor or not, corresponding to **TARGETB**.\n",
    "\n",
    "**How could the stakeholders use the insights gained from the data mining?**\n",
    "\n",
    "1. Find underlying characteristics of lapsing donors, leading to better understanding of what makes donors return.\n",
    "\n",
    "2. Improved accuracy of the solicitation campaign whereby the response rate is high and the wasted effort is less.\n",
    "\n",
    "Looks like we have got an interesting and useful data mining project in hand. :)\n",
    "\n",
    "## End notes\n",
    "In this practical, we learned how to install Python and its libraries with Anaconda. We also learned about the typical data mining process flow in Python and explored the very basic nature of the dataset in order to build data mining models on this dataset and understand the common patterns and trends."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}