Rain Prediction in Australia.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Assignment 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Rain in Australia: Predict rain tomorrow in Australia\n",
    "# Student Name: Sohail Sankanur\n",
    "# Student ID: 29996368"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Predicting Rain is a very common problem in machine learning. There can be different machine learning models which could be implemented for prediction of rain. In this assignment we are going to use some machine learning models to predict whether it would rain in the future based on some past features. \n",
    "\n",
    "In this assignment we are going to use the tools of machine learning to predict and visualise the possibility of rainfall in Australia."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a initial task of the assignment we are importing all the libraries and functions which we would require in this assignment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import Counter \n",
    "from pyspark.sql.functions import isnan, when, count, col\n",
    "from pyspark.sql.functions import *\n",
    "from pyspark.sql.types import IntegerType, DoubleType\n",
    "from pyspark.ml.feature import StringIndexer, OneHotEncoder, VectorAssembler, VectorIndexer\n",
    "from pyspark.ml import Pipeline\n",
    "from pyspark.ml.evaluation import MulticlassClassificationEvaluator\n",
    "from pyspark.ml.classification import DecisionTreeClassifier\n",
    "from pyspark.ml.classification import RandomForestClassifier\n",
    "from pyspark.ml.classification import GBTClassifier\n",
    "from pyspark.ml.classification import LogisticRegression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part A: Creating Spark Session and Loading the Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this part of the assignment we create a spark session and load the dataset which is given to us. Loading of the dataset is done using pyspark. We woule use the pyspark dataframe for all further programming."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 01: Import Spark Session and initialize Spark"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this step we will be importing a library called pyspark. Pyspark is a python API for spark. In this we are going to use \"SparkContext\" which is a main entry for Spark functionality. We are also going to use \"SparkConf\" which is used for configuring Spark. Now our first step would be to import SparkContext and SparkConf libraries from pyspark"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark import SparkContext # Spark\n",
    "from pyspark.sql import SparkSession # Spark SQL"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have imported the required libraries we now use this to create a Spark Session, initialize it and configure Spark to be used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "os.environ['PYSPARK_SUBMIT_ARGS'] = '--packages org.mongodb.spark:mongo-spark-connector_2.11:2.4.0 pyspark-shell'\n",
    "# create entry points to spark\n",
    "\n",
    "# We add this line to avoid an error : \"Cannot run multiple SparkContexts at once\". \n",
    "# If there is an existing spark context, we will reuse it instead of creating a new context.\n",
    "sc = SparkContext.getOrCreate()\n",
    "\n",
    "# local[*]: run Spark locally with as many working processors as logical cores on your machine.\n",
    "# In the field of `master`, we use a local server with as many working processors (or threads) as possible (i.e. `local[*]`). \n",
    "# If we want Spark to run locally with 'k' worker threads, we can specify as `local[k]`.\n",
    "# The `appName` field is a name to be shown on the Sparking cluster UI. \n",
    "\n",
    "# If there is no existing spark context, we now create a new context\n",
    "if (sc is None):\n",
    "    sc = SparkContext(master=\"local[4]\")\n",
    "spark = SparkSession(sparkContext=sc)\\\n",
    "        .builder\\\n",
    "        .appName(\"Assignment2 Application\")\\\n",
    "        .config(\"spark.mongodb.input.uri\", \"mongodb://127.0.0.1/fit5202_db.wk05_demo\")\\\n",
    "        .config(\"spark.mongodb.output.uri\", \"mongodb://127.0.0.1/fit5202_db.wk05_demo\")\\\n",
    "        .getOrCreate()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 02: Load the dataset and print the schema and total number of entries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this step we create a dataframe of the csv file data which is given to us. We can read the data from the csv file as a dataframe using pyspark. As seen we specify header as True which means that the first row of the data would be taken as a heading for all the columns in the dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# code to read the csv file using spark session\n",
    "weather_df = spark.read.csv('weatherAUS.csv', header = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have read the data into a dataframe we now print the total number of entries in the dataframe. This can be done using the 'count' function in pyspark."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "142193"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# code for printing the number of lines in the spark dataframe\n",
    "weather_df.count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As seen from the above output there are 142193 entries in the dataframe."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part B: Data Cleaning and Processing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this part of the assignment the dataset is cleaned and all the pre processing tasks is done on the dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 03: Delete columns from the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In data cleaning we remove all unnecessary data from the dataset so that the accuracy of the model could be improved. Some columns from the dataset would not be required for prediction task. Hence in this step we delete certain columns from the dataset.\n",
    "The columns which are being removed are:\n",
    "* Date\n",
    "* Location\n",
    "* Evaporation\n",
    "* Sunshine\n",
    "* Cloud9am\n",
    "* Cloud3pm\n",
    "* Temp9am\n",
    "* Temp3pm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Date',\n",
       " 'Location',\n",
       " 'MinTemp',\n",
       " 'MaxTemp',\n",
       " 'Rainfall',\n",
       " 'Evaporation',\n",
       " 'Sunshine',\n",
       " 'WindGustDir',\n",
       " 'WindGustSpeed',\n",
       " 'WindDir9am',\n",
       " 'WindDir3pm',\n",
       " 'WindSpeed9am',\n",
       " 'WindSpeed3pm',\n",
       " 'Humidity9am',\n",
       " 'Humidity3pm',\n",
       " 'Pressure9am',\n",
       " 'Pressure3pm',\n",
       " 'Cloud9am',\n",
       " 'Cloud3pm',\n",
       " 'Temp9am',\n",
       " 'Temp3pm',\n",
       " 'RainToday',\n",
       " 'RainTomorrow']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weather_df.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Columns from the pyspark dataframe can be removed using the drop function. Inside the parameters which would be passed are the names of the columns which have to be dropped."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_weather_modified = weather_df.drop('Date', 'Location', 'Evaporation', 'Sunshine', 'Cloud9am', 'Cloud3pm', 'Temp9am', 'Temp3pm')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After dropping the unnecessary columns then new dataframe would have these set of columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Date',\n",
       " 'Location',\n",
       " 'MinTemp',\n",
       " 'MaxTemp',\n",
       " 'Rainfall',\n",
       " 'Evaporation',\n",
       " 'Sunshine',\n",
       " 'WindGustDir',\n",
       " 'WindGustSpeed',\n",
       " 'WindDir9am',\n",
       " 'WindDir3pm',\n",
       " 'WindSpeed9am',\n",
       " 'WindSpeed3pm',\n",
       " 'Humidity9am',\n",
       " 'Humidity3pm',\n",
       " 'Pressure9am',\n",
       " 'Pressure3pm',\n",
       " 'Cloud9am',\n",
       " 'Cloud3pm',\n",
       " 'Temp9am',\n",
       " 'Temp3pm',\n",
       " 'RainToday',\n",
       " 'RainTomorrow']"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weather_df.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 04: Print the number of missing data in each column"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this step we have to print the **number of missing data in each column**.\n",
    "\n",
    "As seen from the dataset and assignment specification we can see that the missing values in the dataset is **'NA'**. Hence we find the number of NA values in each column and print it out. \n",
    "\n",
    "We can find the number of 'NA' values in each column in the dataframe using the pyspark select function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Row(MinTemp=637, MaxTemp=322, Rainfall=1406, WindGustDir=9330, WindGustSpeed=9270, WindDir9am=10013, WindDir3pm=3778, WindSpeed9am=1348, WindSpeed3pm=2630, Humidity9am=1774, Humidity3pm=3610, Pressure9am=14014, Pressure3pm=13981, RainToday=1406, RainTomorrow=0)]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# code to find the number of na values in a column in the modified dataframe\n",
    "df_weather_modified.select([count(when(df_weather_modified[c] == 'NA', c)).alias(c) for c \n",
    "                            in df_weather_modified.columns]).collect()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In an alternative way we can find the number of 'NA' values in each column by iterating through the columns. The code for this is as follows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The number of empty rows in column  MinTemp  is:  637\n",
      "The number of empty rows in column  MaxTemp  is:  322\n",
      "The number of empty rows in column  Rainfall  is:  1406\n",
      "The number of empty rows in column  WindGustDir  is:  9330\n",
      "The number of empty rows in column  WindGustSpeed  is:  9270\n",
      "The number of empty rows in column  WindDir9am  is:  10013\n",
      "The number of empty rows in column  WindDir3pm  is:  3778\n",
      "The number of empty rows in column  WindSpeed9am  is:  1348\n",
      "The number of empty rows in column  WindSpeed3pm  is:  2630\n",
      "The number of empty rows in column  Humidity9am  is:  1774\n",
      "The number of empty rows in column  Humidity3pm  is:  3610\n",
      "The number of empty rows in column  Pressure9am  is:  14014\n",
      "The number of empty rows in column  Pressure3pm  is:  13981\n",
      "The number of empty rows in column  RainToday  is:  1406\n",
      "The number of empty rows in column  RainTomorrow  is:  0\n"
     ]
    }
   ],
   "source": [
    "for i in df_weather_modified.columns:\n",
    "    print('The number of empty rows in column ',i, ' is: ',df_weather_modified[df_weather_modified[i] == 'NA'].count())\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 05: Fill the missing data with average value and maximum occurrence value"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this step you have to fill in all the missing data with average value (for numeric column) or maximum frequency value (for non-numeric column).\n",
    "\n",
    "In this step we find the columns which have **numeric values**, calculate the average and fill the **null values with the average values**.\n",
    "\n",
    "We also identify the **non numeric values** and in these columns we replace the **null values with the most frequent occuring item**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Code for this would be as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# code to remove all null values from the dataset\n",
    "''' Rules for reoving the null values are:\n",
    "    >> If the type is numeric then we replace all NA values with mean values\n",
    "    >> If the type is non numeric then we replace all NA values with highest frequency term'''\n",
    "\n",
    "df_weather_null = df_weather_modified\n",
    "\n",
    "for i in df_weather_null.columns:\n",
    "    if df_weather_null.select(i).head()[0].replace('.','').isdigit():\n",
    "        df_weather_null = df_weather_null.withColumn(i, regexp_replace(i, 'NA', \n",
    "                           df_weather_modified.select(i).withColumn(i, df_weather_modified[i].cast(IntegerType())).\n",
    "                            describe().collect()[1][1] ))\n",
    "    else:\n",
    "        df_weather_null = df_weather_null.withColumn(i, regexp_replace(i, 'NA', \n",
    "                            Counter(df_weather_modified.select(i).collect()).most_common(1)[0][0][0]))\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 06: Data transformation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this step we need to transform the dataset so that it would be possible to process by machine learning. \n",
    "\n",
    "Firstly for non numeric data we have to use the **StringIndexer** function from pyspark to convert it into a number. Then we would typecast them to double format. For numeric columns we would directly typecast it into double format.\n",
    "\n",
    "Code for this would be as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_weather_null_d = df_weather_null\n",
    "for i in df_weather_null.columns:\n",
    "    if df_weather_null.select(i).head()[0].replace('.','').isdigit():\n",
    "        df_weather_null_d = df_weather_null_d.withColumn(i, df_weather_null_d[i].cast(DoubleType()))\n",
    "    else:\n",
    "        indexer = StringIndexer(inputCol=i, outputCol=i+'i').fit(df_weather_null_d)\n",
    "        df_weather_null_d = indexer.transform(df_weather_null_d)\n",
    "        df_weather_null_d  = df_weather_null_d.drop(i)\n",
    "        df_weather_null_d = df_weather_null_d.withColumnRenamed(i+'i', i)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('MinTemp', 'double'),\n",
       " ('MaxTemp', 'double'),\n",
       " ('Rainfall', 'double'),\n",
       " ('WindGustSpeed', 'double'),\n",
       " ('WindSpeed9am', 'double'),\n",
       " ('WindSpeed3pm', 'double'),\n",
       " ('Humidity9am', 'double'),\n",
       " ('Humidity3pm', 'double'),\n",
       " ('Pressure9am', 'double'),\n",
       " ('Pressure3pm', 'double'),\n",
       " ('WindGustDir', 'double'),\n",
       " ('WindDir9am', 'double'),\n",
       " ('WindDir3pm', 'double'),\n",
       " ('RainToday', 'double'),\n",
       " ('RainTomorrow', 'double')]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_weather_null_d.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As seen from the above output we can see that all the columns have the double format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Row(MinTemp='13.4', MaxTemp='22.9', Rainfall='0.6', WindGustDir='W', WindGustSpeed='44', WindDir9am='W', WindDir3pm='WNW', WindSpeed9am='20', WindSpeed3pm='24', Humidity9am='71', Humidity3pm='22', Pressure9am='1007.7', Pressure3pm='1007.1', RainToday='No', RainTomorrow='No')"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_weather_null.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Row(MinTemp=13.4, MaxTemp=22.9, Rainfall=0.6, WindGustSpeed=44.0, WindSpeed9am=20.0, WindSpeed3pm=24.0, Humidity9am=71.0, Humidity3pm=22.0, Pressure9am=1007.7, Pressure3pm=1007.1, WindGustDir=0.0, WindDir9am=6.0, WindDir3pm=7.0, RainToday=0.0, RainTomorrow=0.0)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_weather_null_d.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As seen from the above output we can see that all the non numeric columns have been converted to double format numbers and all numeric columns have been converted to double format numbers as well."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 07: Create the feature vector and divide the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A feature vector can be defined as a vector which contains information describing the objects most important characteristics. In this step of the assignment we have to create a feature vector from the given columns. When we create a feature vector we have to remember to exclude the models which we would be using the test the accuracy of our model.\n",
    "\n",
    "After creating the feature vecor we would divide the dataset into train data and test data. In machine learning we ususlly divide the dataset into 70% and 30%. 70% of the data would be the training data which would be used for training the machine learning model which we would build. The rest 30% of the data would be used as test dataset which would be used to test the accuracy of the model which we build."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Creating feature vector which could be used for machine learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now as the first task in this step we would make a vector column. This can be done using the VectorAssembler function of pyspark. As seen we give the input columns and the output column woule be the features column. Using this we make the feature column which is a vector which could be used in machine learning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "assembler = VectorAssembler(\n",
    "    inputCols=[\"MinTemp\", \"MaxTemp\", \"Rainfall\",\"WindGustSpeed\",'WindSpeed9am','WindSpeed3pm','Humidity9am',\n",
    "               'Humidity3pm','Pressure9am','Pressure3pm','WindGustDir','WindDir9am','WindDir3pm','RainToday'],\n",
    "    outputCol=\"features\")\n",
    "\n",
    "df_weather_features = assembler.transform(df_weather_null_d)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Row(MinTemp=13.4, MaxTemp=22.9, Rainfall=0.6, WindGustSpeed=44.0, WindSpeed9am=20.0, WindSpeed3pm=24.0, Humidity9am=71.0, Humidity3pm=22.0, Pressure9am=1007.7, Pressure3pm=1007.1, WindGustDir=0.0, WindDir9am=6.0, WindDir3pm=7.0, RainToday=0.0, RainTomorrow=0.0, features=DenseVector([13.4, 22.9, 0.6, 44.0, 20.0, 24.0, 71.0, 22.0, 1007.7, 1007.1, 0.0, 6.0, 7.0, 0.0]))"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_weather_features.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As seen from the output we can see that we have the vector column named as features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Splitting the Data into training and testing datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next task in this step would be to split the dataset into **training and testing dataset**. As seen the dataset is split into 70,30. Where in **70% of the dataset would be considered as the training data** which would be used to train the machine learning model and **30% of the dataset would be the testing data** which would be used for testing how the machine learning model would perform.\n",
    "\n",
    "Usually in machine learning we would **train the model using the training data** and then we **perform predictions on the test data**. Now in the test dataset we would have the actual label values and the predicted label values. Using this we would compare the actual label vales and the value which was predicted and based on this we would see the accuracy of the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# dividing the dataset into training and testing datasets\n",
    "# Split data into training (70%) and testing (30%)\n",
    "seed = 0  # set seed for reproducibility\n",
    "trainDF, testDF = df_weather_features.randomSplit([0.7,0.3], seed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have split the dataset into training and testing dataset. The number of rows in the train and test dataset would be as follows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of training data:  99799\n",
      "Number of test data:  42394\n"
     ]
    }
   ],
   "source": [
    "print(\"Number of training data: \", trainDF.count())\n",
    "print(\"Number of test data: \", testDF.count())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part C: Apply Machine Learning Algorithms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this part of the assignment we apply different machine learning on the dataset and compare their accuracy. As we can see this is a classification problem hence we would apply machine learning models which would perform classification.\n",
    "\n",
    "The models whicha are going to apply on the dataset are:\n",
    "* Decision Tree Classifier\n",
    "* Random Forest Classifier\n",
    "* Logistic Regression\n",
    "* GBT Classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 08: Apply machine learning classification algorithms on the dataset and compare their accuracy. Plot the accuracy as bar graph."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this step we would apply classification machine learning models and compare the accuracy. The first model which we would apply to the dataset is Decision Tree Classifier. This can be done using the DecisionTreeClassifier method from the pyspark ml framework."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model can be applied on the dataset using the fit method and using the transform method we would predict the values of the label column. The fit is done on the training dataset and the transform is done on the test dataset. code for this is as follows: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model1: Decision Tree Classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train a DecisionTree model.\n",
    "dt = DecisionTreeClassifier(labelCol=\"RainTomorrow\", featuresCol=\"features\")\n",
    "model = dt.fit(trainDF) # fitting the model with the training data\n",
    "predictions_dt = model.transform(testDF) # performing the predictions on test data using the Decision \n",
    "                                         # Tree model which is create"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have done the predictions we would see the accuracy of the model. Accuracy of the model is done with the help of the MulticlassClassificationEvaluator method of the pyspark evaluation library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy of the Decision Tree Classifier model is:  82.00235971625308\n",
      "Test Error = 0.179976 \n"
     ]
    }
   ],
   "source": [
    "evaluator = MulticlassClassificationEvaluator(\n",
    "    labelCol=\"RainTomorrow\", predictionCol=\"prediction\")\n",
    "accuracy = evaluator.evaluate(predictions_dt)\n",
    "dt_accuracy = accuracy*100\n",
    "print(\"Accuracy of the Decision Tree Classifier model is: \", dt_accuracy)\n",
    "print(\"Test Error = %g \" % (1.0 - accuracy))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As seen from the above output the accuracy of the Decision Tree Classifier model is around 82."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model2: Random Forest Classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we would use the Random Forest Classifier model. This can be done using the RandomForestClassifier methid of the pyspark ml library. Code for this is as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "rf = RandomForestClassifier(labelCol=\"RainTomorrow\", featuresCol=\"features\")\n",
    "model = rf.fit(trainDF)\n",
    "predictions_rf = model.transform(testDF)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have applied the RandomForest Classifier on the dataset we would find the accuracy of this model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy of the Random Forest Classifier model is:  81.40446045082305\n",
      "Test Error = 0.185955 \n"
     ]
    }
   ],
   "source": [
    "evaluator = MulticlassClassificationEvaluator(\n",
    "    labelCol=\"RainTomorrow\", predictionCol=\"prediction\")\n",
    "accuracy = evaluator.evaluate(predictions_rf)\n",
    "rf_accuracy = accuracy*100\n",
    "print(\"Accuracy of the Random Forest Classifier model is: \", accuracy*100)\n",
    "print(\"Test Error = %g \" % (1.0 - accuracy))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As seen from the above output the accuracy of the Random Forest Classifier model is around 81."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model3: Logistic Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The third model which we are going to use is the Logistic Regression Model. This can be done using the LogisticRegression method of the pyspark ml library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy of the Logistic Regression Model is:  83.72428581448712\n"
     ]
    }
   ],
   "source": [
    "lr = LogisticRegression(labelCol=\"RainTomorrow\",featuresCol=\"features\")\n",
    "lrModel = lr.fit(trainDF)\n",
    "predictions_lr = lrModel.transform(testDF)\n",
    "\n",
    "trainingSummary = lrModel.summary\n",
    "lr_accuracy = trainingSummary.accuracy * 100\n",
    "print(\"Accuracy of the Logistic Regression Model is: \", lr_accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have applied the Logistic Regression model on the dataset. The logistic regression model would have summary and from the summary of the model we can see he accuracy of the model is around 84."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy of the Logistic Regression model is:  84.05434731329905\n",
      "Test Error = 0.159457 \n"
     ]
    }
   ],
   "source": [
    "evaluator = MulticlassClassificationEvaluator(\n",
    "    labelCol=\"RainTomorrow\", predictionCol=\"prediction\", metricName=\"accuracy\")\n",
    "accuracy = evaluator.evaluate(predictions_lr)\n",
    "lr_accuracy = accuracy*100\n",
    "print(\"Accuracy of the Logistic Regression model is: \", accuracy*100)\n",
    "print(\"Test Error = %g \" % (1.0 - accuracy))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the MulticlassClassificationEvaluator we can see that the accuracy of the model is around 84."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model4: GBT Classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The last model which we are going to apply on the data is the GBT Classifier model. This model is applied using the GBTClassifier method of the pyspark library. The code for using this model to train and perform predictions is as follows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "gbt = GBTClassifier(labelCol=\"RainTomorrow\", featuresCol=\"features\", maxIter=10)\n",
    "model = gbt.fit(trainDF)\n",
    "predictions_gbt = model.transform(testDF)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have applied the model we would see the accuracy of this model by using the MulticlassClassificationEvaluator method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy of the GBT Classifier model is:  84.1557767608624\n",
      "Test Error = 0.158442 \n"
     ]
    }
   ],
   "source": [
    "evaluator = MulticlassClassificationEvaluator(\n",
    "    labelCol=\"RainTomorrow\", predictionCol=\"prediction\", metricName=\"accuracy\")\n",
    "accuracy = evaluator.evaluate(predictions_gbt)\n",
    "gbt_accuracy = accuracy*100\n",
    "print(\"Accuracy of the GBT Classifier model is: \", accuracy*100)\n",
    "print(\"Test Error = %g \" % (1.0 - accuracy))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As seen the accuracy of the model is around 84."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bar Plot Comparision of different models used for predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have the accuracy of all the models we would do a bar plot of the accuracy of all the models and compare them. The code for this is below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<BarContainer object of 4 artists>"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA8wAAAHwCAYAAABzD+aRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzs3XmYJWV5N+DfI4MgyiICBlAYF1xRJ4JGE2NcSNSQuCQacUVjNCQa4y6fX2LQTyOJRo1bjBoRd4Ia477GJSouoAjiEhVQBCMiiIiiLM/3R9WYQ6e6pwen+wwz931dffU5Ve9566n3VJ/p37xV1dXdAQAAAC7vKvMuAAAAADZHAjMAAABMEJgBAABggsAMAAAAEwRmAAAAmCAwAwAAwASBGYAkSVWtraquqjXzrmXequpBVfWBZbR7eVX99WrUtBxVdeOq+kJVXVBVj11G+yOq6vXj432q6sdVtc34/NpV9fGxr3+owVFVdV5VfXal92VzM/5s3HAZ7e5UVd9ZjZquiKX2Y+F7vtq1bayFn1lV9d6qOvQK9HO5Yx9glsAMsBmpqtOr6qfjL2/nVdW7q+q6m7D/O1XVZWP/F1TV16rq4Vegn18ErQ20e2BVHT9u77vjL7R3uGLVr57ufkN3/84y2h3W3f9vNWpapqck+Wh379jdL9qYF3b3t7v7Gt196bjoUUnOSbJTdz8xyR2S/HaS63T3bTdp1csw/mwctNrb3cosfM+vVLr7Ht199IbaLTyWJo59gF8QmAE2P7/f3ddIsmeS7yV58RXpZImZ4rPG/ndK8tQkr6yqm12hSpfe/hOSvDDJ3ya5dpJ9krwsyb029bY2pSv5DPu+SU7ZhH19ubt75vnp3X3hxnZ0JR/TrcnC93zZftn3eDyDwe+lwGbHBxPAZqq7L0ryliS/CLNVdfB4yu2PquqMqjpiZt360xMfUVXfTvIfG+i/u/vtSc6b3cZMf3tV1Tuq6tyq+kZVPXJcfvckT0ty/3Hm+IsTr905yTOTPLq739bdF3b3xd39zu5+8thmu6p6YVWdNX69sKq2G9fdqaq+U1VPqaqzx9npe1fV71bVf401PW1me0dU1Vuq6phx5vzzVXWrmfWHV9U3x3Vfrqr7zKx7WFV9sqpeUFXnJjliXPaJcX2N686uqvOr6qSq2n9c95qqetZMX48cx+rccez2mlnXVXVYVX19PHvgpVVV47obVtXHxv7PqapjFnvfquqeVXVKVf2wqj5aVTcdl/9Hkjsnecn4vtxo4rXXG7dzQVV9MMluM+t+cXprVb0myaFJnjL29adJXpXk9uPzZ4yv+b2qOnGs5VNVdcuZ/k6vqqdW1UlJLhz73auq3lpV36+q02rmtPHxPfzXqnrtWN8pVXXguO51Gf7D5Z3j9p8ysW8be8wsevyN65889nFWVf3xgm1tV1XPq6pvV9X3ajg1/2qLvF9Praoz63/O6LjrIu0+WlV/MvN8ucfgkrUstR8Ltv+aXP49P2ipMZoZ76dW1X8nOWqiz/U/Wy8e6/7q7P6P+/zsqvpkkp8kuX5V7VxV/zLWfGZVPav+5zKBbcZ9PaeqTk1y8AbG8JFV9ZX6n5/7W08dS/W/T+2e/Owb1y16nG7M+w1ciXS3L1++fPnaTL6SnJ7koPHxDkmOTvLamfV3SnKLDP/hecsMM9D3HtetTdJJXpvk6kmuNtH/nZJ8Z3x8lST3SXJxkhvPvH7NuP5jGWaEt0+yLsn3k9x1XHdEktcvsR93T3LJ+r4WafPMJJ9OskeS3ZN8Ksn/m6nzkiRPT7JtkkeO239jkh2T3DzJRUmuP1PPxUnuO7Z/UpLTkmw7rr9fkr3Gfb5/kguT7Dmue9i4rb9IsibJ1cZlnxjX3y3JCUl2SVJJbjrz2tckedb4+C4ZTme9dZLtMpwZ8PGZ/e0k7xr72Wfcn7uP696U5P+O9W2f5A6LjNmNxtp/e9zPpyT5RpKrjus/muRPlhjz45I8f6zvjkkuWP8+Trz/v9i3mXH6xMzzWyc5O8mvJdkmQ9g6Pcl2M8fyiUmuO47pVcZxfHqSqya5fpJTk9xt5j28KMnvjv09J8mnp342Ftm3O2Xjjpmljr+7Z/jZ2j/Dz9Ibx7G54bj+hUnekWTXse93JnnOxM/YjZOckWSvmTG+wSL1X+69y/KPwaVqWXI/JmpY+J4v52f07zIcT1OfNw8b2zx+fE/un+T8JLvO7PO3x/dmzdjm7Un+eax3jySfTfKnY/vDknw1wzG1a5KP5PLH7C/GMMPP/JlJbjOO2Q2T7Dt1LGXjP/smj9ONeb99+fJ15fkywwyw+Xl7Vf0wyY8yBKPnrl/R3R/t7pO7+7LuPilD0PqtBa8/oocZ3Z8u0v9eY//nJPmbJA/p7q/NNqjhuuk7JHlqd1/U3SdmmGF8yDL34VpJzunuS5Zo86Akz+zus7v7+0mesaD/i5M8u7svTvLmDLOh/9jdF3T3KRlOPb7lTPsTuvstY/vnZ/hl93ZJ0t3HdvdZ47gdk+TrSWavwz2ru1/c3ZdMjNvFGYLITZJUd3+lu7+7yP68urs/390/S/J/MszIrp1pc2R3/7C7v53hl/11M9vYN8Mv2hd19ycWGbP7J3l3d39w3M/nZQijv75I+1+oqn0yhIe/7u6fdffHM4SrK+qRSf65uz/T3Zf2cO3ozzKO+ehF3X3GOKa3SbJ7dz+zu3/e3acmeWWSQ2baf6K739PDtaSvS3KrbJyNOWaWOv7+KMlR3f2lHk5BP2L9Bqqqxn1/fHef290XZLjsYHY/1rs0Q5i8WVVt292nd/c3N3Kf1u/X/zoGl1HLovuxTBv6Gb0syd+Mx9NinzdnJ3lhD2eYHJPka7n8zPBruvuU8bNi1yT3SPK48TPs7CQvWLA/LxyPqXMzhNXF/EmSv+/uz/XgG939rQ3t8DI/+xY7TjfV+w1sRgRmgM3Pvbt7lwy/eD0myceq6leSpKp+rao+UsMprednmHHZbcHrz9hA/2d19y7dvWt3r+vuN0+02SvJ+l/A1/tWkr2XuQ8/SLJbLX1d415jn7P97zXz/Af9PzfhWf/L+Pdm1v80yTVmnv9iv7v7siTfWd9fVT20/ufU4R9mmHHbbeq1C3X3fyR5SZKXJvleVb2iqnba0P50948zjMPsmP33zOOfzNT/lAyzYJ8dT/Fc7NTZhdu4bKx9Oe/LXknO68tfg7zBALGEfZM8cf2YjuN63Vz+PTxjQfu9FrR/Wobr29dbOD7bb+AYWmhjjpmljr+9FtQ+2273DGd/nDCzH+8bl19Od38jyeMyBNWzq+rNNXOa/nItcQxuqJal9mM5NvQz+v0eLh1ZypndPXtN9MI+Fh4j2yb57sz+/HOGmeb19Sx3f66b5IqE1eV89k0ep5vq/QY2LwIzwGZqnLV7W4ZZi/V3ln5jhtMvr9vdOyd5eYagdbmXboLNn5Vk16racWbZPhlOcVzONo7LcNrivTewjX0X9H/WRtY56xd3E6/h5kHXSXJWVe2bYSbzMUmuNf5nxJdy+XFbcn+6+0XdfUCGU0dvlOTJE80utz9VdfUMM+1nTrRd2P9/d/cju3uvJH+a5GU1/ad/Fm6jMuz3BreR5LtJrjnWtd4+y3jdYs7IMJu7y8zXDt39ppk2vaD9aQva79jdv7vM7W2K43rWUsffdzNzPOXy43ROhuB985n92LmHG+n9L939xu6+w7itznAK85QLM4Tf9X5lQT9Tx+CGallqP5ZjQz+jy3lP9h6P0+X0cUaGsxR2m9mfnbr75uP6jdmfM5LcYJF1S9W9oc++JW3E+w1cSQjMAJupGtwryTWTfGVcvGOG2Y+Lquq2SR64Etvu7jMyXK/4nKravoabOT0iyRvGJt9LsrYWuattd5+f4VrSl9Zw46UdqmrbqrpHVf392OxNSf6qqnavqt3G9hv8U1VLOKCq/mCckXxchl+8P53hWsjOcB1iavgzWvsvt9Oqus04s79thlBzUYb/xFjojUkeXlXrargx0t8m+Ux3n76Mbdyvqq4zPj1vrHdqG/+a5OCquutYzxPH/fzUhrYxno56fJJnVNVVa/jzXr+/odct4ZVJDhvHpqrq6jXclG7HRdp/NsmPxpsiXa2GGzjtX1W3Web2vpfhuudNZanj71+TPKyqblZVO2S4dCHJL2b1X5nkBVW1R5JU1d5VdbeFG6jh72LfZTweLsoQbhf700UnJvmD8Wflhhl+3tb3M3kMLqOWRfdjE4zRcu2R5LHjz//9Mlx//Z6phj1c6vCBJP9QVTtV1VWq6gZVtf6yk38d+7pOVV0zyeFLbPdVSZ5UVQeMx+cNx/88S5Y4lpbx2beojXy/gSsJgRlg8/POqvpxhmuYn53k0PH6yyT58yTPrKoLMvzy+q8rWMcDMty05qwk/5bhWsUPjuuOHb//oKo+P/Xi7n5+kick+asMYfWMDLO8bx+bPCtDgDspyclJPj8uu6L+PcM1vudluN7wD8brJr+c5B8yzHp/L8NN0z65Ef3ulCGUnJfh1MwfZLh2+HK6+8NJ/jrJWzPMhN0g09e1TrlNks+M7/s7kvxld582sY2vJXlwhhuKnZMh8P5+d/98mdt5YIabdJ2bITy9dpmv+1+6+/gM18++JMPYfCPDTZ4Wa3/pWO+6DDdkOydDqNl5mZt8Tobw9sOqetIVrXvGosdfd783w820/iPDfi284/xTx+WfrqofJflQhhs+LbRdkiMz7Ot/ZwiPT5tolwzX6v48wzF6dC4f0JY6BhetZRn7sSGb4mf0M0n2yzAGz05y3+7+wRLtH5rhpnBfzrC/b8nwJ/aSYQzen+SLYy1vW6yT7j523N4bM9zc7u0ZrpFONnwsLfXZt5SNeb+BK4m6/GUlAHDlU8Of17phdz943rUAg6p6WIa7Vt9hQ20BNldmmAEAAGCCwAwAAAATnJINAAAAE8wwAwAAwASBGQAAACasmXcBrL7ddtut165dO+8yAAAA5uKEE044p7t331A7gXkrtHbt2hx//PHzLgMAAGAuqupby2nnlGwAAACYIDADAADABIEZAAAAJgjMAAAAMEFgBgAAgAkCMwAAAEwQmAEAAGCCwAwAAAATBGYAAACYIDADAADABIEZAAAAJgjMAAAAMEFgBgAAgAkCMwAAAEwQmAEAAGCCwAwAAAATBGYAAACYIDADAADAhDXzLoDVd/KZ52ft4e+edxkAAMAW6vQjD553CZuEGWYAAACYIDADAADABIEZAAAAJgjMAAAAMEFgBgAAgAkCMwAAAEwQmAEAAGCCwAwAAAATBGYAAACYIDADAADABIEZAAAAJgjMAAAAMEFgBgAAgAkCMwAAAEwQmAEAAGCCwAwAAAATBGYAAACYIDADAADABIEZAAAAJgjMAAAAMEFgBgAAgAkCMwAAAEwQmAEAAGCCwAwAAAATBGYAAACYIDADAADABIEZAAAAJgjMAAAAMEFgBgAAgAkCMwAAAEwQmAEAAGCCwAwAAAATBGYAAACYIDADAADABIEZAAAAJgjMAAAAMEFgBgAAgAlbRGCuqmtX1Rur6tSqOqGqjquq+4zr7lRV51fViVV1UlV9qKr2qKqHj8tOrKqfV9XJ4+MjJ/q/bVV9vKq+VlVfrapXVdUOVfWwqnrJJtyP91TVLuPjx1bVV6rqDVV1z6o6fFNtBwAAgA1bM+8CfllVVUnenuTo7n7guGzfJPecafaf3f1747rnJHl0d/9NkqPGZacnuXN3nzPR/7WTHJvkkO4+btzeHybZcVPvS3f/7szTP09yj+4+bXz+juX2U1VruvuSTVocAADAVmZLmGG+S5Kfd/fL1y/o7m9194sXNhzD7o5JztuI/h+dIYwfN/bd3f2W7v7egr5/v6o+U1VfGGexrz0u/62ZmewvVNWOVbXnOGN9YlV9qap+c2x7elXtVlUvT3L9JO+oqsfPzmRX1e5V9daq+tz49Rvj8iOq6hVV9YEkr92I/QMAAGDClX6GOcnNk3x+A21+s6pOTHKtJBcmedpG9L9/kqOX0e4TSW7X3V1Vf5LkKUmemORJGWa0P1lV10hyUZJHJXl/dz+7qrZJssNsR919WFXdPeOsd1U9bGb1PyZ5QXd/oqr2SfL+JDcd1x2Q5A7d/dOFxVXVo8btZpuddl/uvgMAAGy1toTAfDlV9dIkd8gw63ybcfHsKdlPTfL3SQ7bxJu+TpJjqmrPJFdNsv5U6k8meX5VvSHJ27r7O1X1uSSvrqptk7y9u0/ciO0clORmw2R5kmSnqlp/evg7psJyknT3K5K8Ikm223O/3pgdAwAA2BptCadkn5Lk1uufdPejk9w1yWLTqO9IcseN7P+AZbR7cZKXdPctkvxpku3Heo5M8idJrpbk01V1k+7++FjDmUleV1UP3Yh6rpLk9t29bvzau7svGNdduBH9AAAAsIQtITD/R5Ltq+rPZpbtsFjjDLPP39yI/l+S5NCq+rX1C6rqwVX1Kwva7ZwhACfJoTNtb9DdJ3f33yU5PslNxpuSnd3dr0zyL5kJ/MvwgSSPmel/3Ua8FgAAgGW60p+SPV4zfO8kL6iqpyT5foaZ1qfONFt/DXMlOT/DjO9y+/9eVR2S5HlVtUeSy5J8PMnbFjQ9IsmxVXVmkk8nud64/HFVdecklyb5cpL3JjkkyZOr6uIkP06yMTPMj03y0qo6KcP79/Fs+tPLAQAAtnrV7XLWrc12e+7Xex76wnmXAQAAbKFOP/LgeZewpKo6obsP3FC7LeGUbAAAANjkBGYAAACYIDADAADABIEZAAAAJgjMAAAAMEFgBgAAgAkCMwAAAEwQmAEAAGCCwAwAAAATBGYAAACYIDADAADABIEZAAAAJgjMAAAAMEFgBgAAgAkCMwAAAEwQmAEAAGCCwAwAAAATBGYAAACYIDADAADABIEZAAAAJgjMAAAAMEFgBgAAgAkCMwAAAEwQmAEAAGCCwAwAAAATBGYAAACYIDADAADABIEZAAAAJgjMAAAAMEFgBgAAgAkCMwAAAEwQmAEAAGCCwAwAAAATBGYAAACYIDADAADABIEZAAAAJqyZdwGsvlvsvXOOP/LgeZcBAACwWTPDDAAAABMEZgAAAJggMAMAAMAEgRkAAAAmCMwAAAAwQWAGAACACQIzAAAATBCYAQAAYILADAAAABMEZgAAAJggMAMAAMAEgRkAAAAmCMwAAAAwQWAGAACACQIzAAAATBCYAQAAYILADAAAABMEZgAAAJggMAMAAMAEgRkAAAAmrJl3Aay+k888P2sPf/e8ywAAALZApx958LxL2GTMMAMAAMAEgRkAAAAmCMwAAAAwQWAGAACACQIzAAAATBCYAQAAYILADAAAABMEZgAAAJggMAMAAMAEgRkAAAAmCMwAAAAwQWAGAACACQIzAAAATBCYAQAAYILADAAAABMEZgAAAJggMAMAAMAEgRkAAAAmCMwAAAAwQWAGAACACQIzAAAATBCYAQAAYILADAAAABMEZgAAAJggMAMAAMAEgRkAAAAmCMwAAAAwQWAGAACACQIzAAAATBCYAQAAYILADAAAABMEZgAAAJggMAMAAMAEgRkAAAAmCMwAAAAwQWAGAACACQIzAAAATFixwFxVP94EfexVVW9ZYv0uVfXny20/8frXVNVpVXViVX2xqu76y9a8KVXVYVX10HnXAQAAsDXarGeYu/us7r7vEk12SfLnG9F+ypO7e12SxyV5+RUo83+pqjWbop/ufnl3v3ZT9AUAAMDGWdXAXFX7VtWHq+qk8fs+4/IbVNWnq+pzVfXM9bPTVbW2qr40Pr55VX12nA0+qar2S3JkkhuMy567oP02VfW8qjp5bP8XGyjvuCR7z9R6QFV9rKpOqKr3V9We4/LbjP0dN25z/fYeVlXHVtU7k3xgXPbkcZ9OqqpnjMuuXlXvHme0v1RV9x+XH1lVXx7bPm9cdkRVPWl8vG4co5Oq6t+q6prj8o9W1d+NY/NfVfWbm+CtAgAA2Oqt9gzzS5K8trtvmeQNSV40Lv/HJP/Y3bdJctYirz1sbLMuyYFJvpPk8CTf7O513f3kBe0fleR6SX51ZntLuXuStydJVW2b5MVJ7tvdByR5dZJnj+2OSnJYd98+yaUL+rh9kkO7+y5V9TtJ9kty2yTrkhxQVXcct3NWd9+qu/dP8r6q2jXJfZLcfKz1WRP1vTbJU8f1Jyf5m5l1a7r7thlmyf9m4rWpqkdV1fFVdfylPzl/A0MBAADAagfm2yd54/j4dUnuMLP82PHxGxe+aHRckqdV1VOT7NvdP93Atg5K8vLuviRJuvvcRdo9t6pOTfL6JH87Lrtxkv2TfLCqTkzyV0muU1W7JNmxuz+1SK0fnNnO74xfX0jy+SQ3yRCgT05y0Dgr/JvdfX6SHyW5KMmrquoPkvxkttOq2jnJLt39sXHR0UnuONPkbeP3E5KsndrJ7n5Fdx/Y3Qdus8POiwwFAAAA6837GuZedsPuNya5Z5KfJnl/Vd1lAy+pZfb/5CQ3zBCKj5557SnjzPW67r5Fd//OuHwpFy7Y/nNm+rhhd/9Ld/9XkgMyBOfnVNXTx1B/2yRvTXLvJO9bRt2zfjZ+vzTJJrl+GgAAYGu32oH5U0kOGR8/KMknxsefTvKH4+NDFr4oSarq+klO7e4XJXlHklsmuSDJjots6wNJDlt/A67xtOdJ3X1ZhtPCr1JVd0vytSS7V9Xtx9duW1U37+7zklxQVbdbqtbR+5P8cVVdY+xj76rao6r2SvKT7n59kuclufXYZufufk+G06rXLajv/CTnzVyf/JAkHwsAAAArZiVnI3eoqu/MPH9+kscmeXVVPTnJ95M8fFz3uCSvr6onJnl3kqmLbO+f5MFVdXGS/07yzO4+t6o+Od54671JXjrT/lVJbpTkpPE1r8xwDfWk7u6qelaSp3T3+6vqvkleNJ4OvSbJC5OckuQRSV5ZVRcm+egitaa7P1BVN01yXFUlyY+TPDjDbPZzq+qyJBcn+bMMof/fq2r7DDPTj5/o8tAkL6+qHZKcOjN2AAAArIDqXvZZ0StXxBACfzqG1kOSPKC77zXvuqZU1TW6e/1dvA9Psmd3/+Wcy9oo2+25X+956AvnXQYAALAFOv3Ig+ddwgZV1QndfeCG2m0u17sekOQlNUzF/jDJH8+5nqUcXFX/J8PYfSvJw+ZbDgAAACthswjM3f2fSW417zqWo7uPSXLMvOsAAABgZc37LtkAAACwWRKYAQAAYILADAAAABMEZgAAAJggMAMAAMAEgRkAAAAmCMwAAAAwQWAGAACACQIzAAAATBCYAQAAYILADAAAABMEZgAAAJiwUYG5qq5ZVbdcqWIAAABgc7HBwFxVH62qnapq1yRfTHJUVT1/5UsDAACA+VnODPPO3f2jJH+Q5KjuPiDJQStbFgAAAMzXcgLzmqraM8kfJXnXCtcDAAAAm4XlBOZnJnl/km929+eq6vpJvr6yZQEAAMB8rdlQg+4+NsmxM89PTfKHK1kUAAAAzNtybvp1o6r6cFV9aXx+y6r6q5UvDQAAAOZnOadkvzLJ/0lycZJ090lJDlnJogAAAGDelhOYd+juzy5YdslKFAMAAACbi+UE5nOq6gZJOkmq6r5JvruiVQEAAMCcbfCmX0keneQVSW5SVWcmOS3Jg1e0KgAAAJiz5dwl+9QkB1XV1ZNcpbsvWPmyAAAAYL42GJirapckD02yNsmaqkqSdPdjV7QyAAAAmKPlnJL9niSfTnJykstWthwAAADYPCwnMG/f3U9Y8UoAAABgM7Kcu2S/rqoeWVV7VtWu679WvDIAAACYo+XMMP88yXOT/N+Mf1pq/H79lSoKAAAA5m05gfkJSW7Y3eesdDEAAACwuVjOKdmnJPnJShcCAAAAm5PlzDBfmuTEqvpIkp+tX+jPSgEAALAlW05gfvv4BQAAAFuNDQbm7j56NQoBAACAzckGA3NV7ZfkOUlulmT79cu7212yr6RusffOOf7Ig+ddBgAAwGZtOTf9OirJPyW5JMmdk7w2yetWsigAAACYt+UE5qt194eTVHd/q7uPSHKXlS0LAAAA5ms5N/26qKqukuTrVfWYJGcm2WNlywIAAID5Ws4M8+OS7JDksUkOSPKQJIeuZFEAAAAwb8u5S/bnxoc/TvLwlS0HAAAANg+LBuaqOipJL7K6u/sRK1MSAAAAzN9SM8zvmli2T4ZTtLdZmXIAAABg87BoYO7ut65/XFXXT/K0JHdMcmSSf1n50gAAAGB+lrzpV1XdtKpen+SdST6R5Gbd/U/d/fNVqQ4AAADmZKlrmI9NcmCS5yV5fJJLk+xUVUmS7j53NQoEAACAeVjqGubbZLjp15OSPHFcVuP3TnL9FawLAAAA5mqpa5jXrmIdAAAAsFlZ8hpmAAAA2FoJzAAAADBBYAYAAIAJS90le9elXugu2QAAAGzJlrpL9gkZ7oZdE+vcJRsAAIAt2lJ3yb7eahYCAAAAm5MNXsNcgwdX1V+Pz/epqtuufGkAAAAwP8u56dfLktw+yQPH5xckeemKVQQAAACbgaWuYV7v17r71lX1hSTp7vOq6qorXBcr6OQzz8/aw9897zIAAIBVdvqRB8+7hCuV5cwwX1xV22S40Veqavckl61oVQAAADBnywnML0ryb0n2qKpnJ/lEkr9d0aoAAABgzjZ4SnZ3v6GqTkhy1wx/Yure3f2VFa8MAAAA5mjRwFxVu848PTvJm2bXdfe5K1kYAAAAzNNSM8wnZLhuuZLsk+S88fEuSb6dxN9pBgAAYIu16DXM3X297r5+kvcn+f3u3q27r5Xk95K8bbUKBAAAgHlYzk2/btPd71n/pLvfm+S3Vq4kAAAAmL/l/B3mc6rqr5K8PsMp2g9O8oMVrQoAAADmbDkzzA9IsnuGPy319iR7jMsAAABgi7WcPyt1bpK/rKqdklzW3T9e+bIAAABgvjY4w1xVt6iqLyQ5OckpVXVCVe2/8qUBAADA/CznlOx/TvKE7t63u/dN8sQkr1jZsgAAAGC+lhOYr97dH1n/pLs/muTqK1YRAAAAbAaWc5fsU6vqr5O8bnz+4CSnrVxJAAAAMH/LmWH+4wx3yX5bhjtl7558D993AAAV3UlEQVTk4StZFAAAAMzbcu6SfV6Sx65CLQAAALDZWDQwV9U7lnphd99z05cDAAAAm4elZphvn+SMJG9K8pkktSoVAQAAwGZgqcD8K0l+O8kDkjwwybuTvKm7T1mNwgAAAGCeFr3pV3df2t3v6+5Dk9wuyTeSfLSq/mLVqgMAAIA5WfKmX1W1XZKDM8wyr03yogx3ywYAAIAt2lI3/To6yf5J3pvkGd39pVWrCgAAAOZsqRnmhyS5MMmNkjy26hf3/Kok3d07rXBtAAAAMDeLBubuXvT6ZgAAANjSCcUAAAAwQWAGAACACQIzAAAATBCYAQAAYILADAAAABMEZgAAAJggMAMAAMAEgRkAAAAmCMwAAAAwQWAGAACACQIzAAAATBCYAQAAYILADAAAABOuVIG5qi6tqhOr6ktV9c6q2mUT9bu2qr60Kfpa0O8RVXXmWPOJVXXkpt7GzLbWVdXvrlT/AAAAW5srVWBO8tPuXtfd+yc5N8mj513QMrxgrHlddx++3BdV1TYbuZ11SQRmAACATeTKFphnHZdk7ySpqmtU1Yer6vNVdXJV3WtcvraqvlJVr6yqU6rqA1V1tXHdAVX1xao6LjPBu6q2r6qjxn6+UFV3Hpc/rKrePs5sn1ZVj6mqJ4xtPl1Vuy638Kq66/i6k6vq1VW13bj89Kp6elV9Isn9quoGVfW+qjqhqv6zqm4ytrvfOMv+xar6eFVdNckzk9x/nMm+/yYZYQAAgK3YlTIwj7Ovd03yjnHRRUnu0923TnLnJP9QVTWu2y/JS7v75kl+mOQPx+VHJXlsd99+QfePTpLuvkWSByQ5uqq2H9ftn+SBSW6b5NlJftLdv5ohvD90kXIfP3NK9t3Gvl6T5P7jNtYk+bOZ9hd19x26+81JXpHkL7r7gCRPSvKysc3Tk9ytu2+V5J7d/fNx2THjTPYxE2P2qKo6vqqOv/Qn5y9SKgAAAOtd2QLz1arqxCQ/SLJrkg+OyyvJ31bVSUk+lGHm+drjutO6+8Tx8QlJ1lbVzkl26e6PjctfN7ONO6x/3t1fTfKtJDca132kuy/o7u8nOT/JO8flJydZu0jNs6dkvz/Jjcea/mtcf3SSO860PyYZZs2T/HqSY8d9/ucke45tPpnkNVX1yCTLOnW7u1/R3Qd294Hb7LDzcl4CAACwVbuyBeafdve6JPsmuWr+51TqByXZPckB4/rvJVk/K/yzmddfmmFGt5L0ItuoRZYv7OuymeeXjf0ux1L9J8mF4/erJPnhTNhe1903TZLuPizJXyW5bpITq+pay9w2AAAAy3RlC8xJku4+P8ljkzypqrZNsnOSs7v74vGa43038PofJjm/qu4wLnrQzOqPr39eVTdKsk+Sr23C8r+aYZb7huPzhyT52MJG3f2jJKdV1f3GWqqqbjU+vkF3f6a7n57knAzB+YIkO27COgEAALZqV8rAnCTd/YUkX0xySJI3JDmwqo7PEHa/uowuHp7kpeNNv346s/xlSbapqpMznB79sO7+2VQHV7Dui8ZtHztu47IkL1+k+YOSPKKqvpjklCT3Gpc/d7xh2JcyBPwvJvlIkpu56RcAAMCmUd2LnZnMlmq7PffrPQ994bzLAAAAVtnpRx487xI2C1V1QncfuKF2V9oZZgAAAFhJAjMAAABMEJgBAABggsAMAAAAEwRmAAAAmCAwAwAAwASBGQAAACYIzAAAADBBYAYAAIAJAjMAAABMEJgBAABggsAMAAAAEwRmAAAAmCAwAwAAwASBGQAAACYIzAAAADBBYAYAAIAJAjMAAABMEJgBAABggsAMAAAAEwRmAAAAmCAwAwAAwASBGQAAACYIzAAAADBBYAYAAIAJAjMAAABMEJgBAABggsAMAAAAEwRmAAAAmCAwAwAAwASBGQAAACYIzAAAADBBYAYAAIAJAjMAAABMEJgBAABggsAMAAAAE9bMuwBW3y323jnHH3nwvMsAAADYrJlhBgAAgAkCMwAAAEwQmAEAAGCCwAwAAAATBGYAAACYIDADAADABIEZAAAAJgjMAAAAMEFgBgAAgAkCMwAAAEwQmAEAAGCCwAwAAAATBGYAAACYIDADAADABIEZAAAAJgjMAAAAMEFgBgAAgAkCMwAAAEwQmAEAAGCCwAwAAAAT1sy7AFbfyWeen7WHv3veZQAAAJuJ0488eN4lbJbMMAMAAMAEgRkAAAAmCMwAAAAwQWAGAACACQIzAAAATBCYAQAAYILADAAAABMEZgAAAJggMAMAAMAEgRkAAAAmCMwAAAAwQWAGAACACQIzAAAATBCYAQAAYILADAAAABMEZgAAAJggMAMAAMAEgRkAAAAmCMwAAAAwQWAGAACACQIzAAAATBCYAQAAYILADAAAABMEZgAAAJggMAMAAMAEgRkAAAAmCMwAAAAwQWAGAACACQIzAAAATBCYAQAAYILADAAAABMEZgAAAJggMAMAAMAEgRkAAAAmCMwAAAAwQWAGAACACQIzAAAATNiiAnNVXVpVJ1bVKVX1xap6QlVdoX2sqmdW1UFLrD+sqh56xatNquoWY70nVtW5VXXa+PhDv0y/AAAA/PLWzLuATeyn3b0uSapqjyRvTLJzkr/Z2I66++kbWP/yK1Th5fs4Ocn6el+T5F3d/ZaF7apqTXdf8stuDwAAgOXbomaYZ3X32UkeleQxNdimqp5bVZ+rqpOq6k/Xt62qp1TVyeOs9JHjstdU1X3Hx0dW1ZfH1z1vXHZEVT1pfLyuqj49rv+3qrrmuPyjVfV3VfXZqvqvqvrN5dZfVQdV1Yeq6s1JvjAuO3Ts68Sqetn62fOqukdVHVdVn6+qY6rq6ptkEAEAALZiW2xgTpLuPjXDPu6R5BFJzu/u2yS5TZJHVtX1quoeSe6d5Ne6+1ZJ/n62j6raNcl9kty8u2+Z5FkTm3ptkqeO60/O5We013T3bZM8Lhs/0327JE/p7ltU1f5jHb8+zqKvSXLIOJN+eJK7dvetk5yU5C8XdlRVj6qq46vq+Et/cv5GlgEAALD12dJOyZ5S4/ffSXLL9bPGGU7V3i/JQUmO6u6fJEl3n7vg9T9KclGSV1XVu5O863KdV+2cZJfu/ti46Ogkx840edv4/YQkazey9uO6+9vj44MyBP3jqypJrpbkjCQ/SXKzJJ8al181yScWdtTdr0jyiiTZbs/9eiPrAAAA2Ops0YG5qq6f5NIkZ2cIzn/R3e9f0ObuSRYNkN19SVXdNsldkxyS5DFJ7rIRZfxs/H5pNn68L5wtNcmru/uvZxtU1X2SvK+7H7KRfQMAALCELfaU7KraPcnLk7ykuzvJ+5P8WVVtO66/0Xit7weS/HFV7TAu33VBP9dIsnN3vyfDadXrZtd39/lJzpu5PvkhST6WTe9DSf6oqnYb67pWVe2T5FNJfmv8z4FU1dWrar8V2D4AAMBWZUubYb5aVZ2YZNsklyR5XZLnj+teleGU6M/XcO7y95Pcu7vfV1XrMpzq/PMk70nytJk+d0zy71W1fYZZ3sdPbPfQJC8fQ/epSR6+qXesu0+uqmck+dB4s6+LkxzW3Z+rqkckOaaqrjo2f1qSr2/qGgAAALYmNUy+sjXZbs/9es9DXzjvMgAAgM3E6UcePO8SVlVVndDdB26o3RZ7SjYAAAD8MgRmAAAAmCAwAwAAwASBGQAAACYIzAAAADBBYAYAAIAJAjMAAABMEJgBAABggsAMAAAAEwRmAAAAmCAwAwAAwASBGQAAACYIzAAAADBBYAYAAIAJAjMAAABMEJgBAABggsAMAAAAEwRmAAAAmCAwAwAAwASBGQAAACYIzAAAADBBYAYAAIAJAjMAAABMEJgBAABggsAMAAAAEwRmAAAAmCAwAwAAwASBGQAAACYIzAAAADBBYAYAAIAJAjMAAABMEJgBAABggsAMAAAAEwRmAAAAmCAwAwAAwASBGQAAACasmXcBrL5b7L1zjj/y4HmXAQAAsFkzwwwAAAATBGYAAACYIDADAADABIEZAAAAJgjMAAAAMEFgBgAAgAkCMwAAAEwQmAEAAGCCwAwAAAATBGYAAACYIDADAADABIEZAAAAJgjMAAAAMEFgBgAAgAkCMwAAAEwQmAEAAGCCwAwAAAATBGYAAACYIDADAADAhOruedfAKquqC5J8bd51bKV2S3LOvIvYChn3+TH282Ps58fYz4dxnx9jPz/G/orbt7t331CjNatRCZudr3X3gfMuYmtUVccb+9Vn3OfH2M+PsZ8fYz8fxn1+jP38GPuV55RsAAAAmCAwAwAAwASBeev0inkXsBUz9vNh3OfH2M+PsZ8fYz8fxn1+jP38GPsV5qZfAAAAMMEMMwAAAEwQmLciVXX3qvpaVX2jqg6fdz1bsqp6dVWdXVVfmlm2a1V9sKq+Pn6/5jxr3FJV1XWr6iNV9ZWqOqWq/nJcbvxXWFVtX1WfraovjmP/jHH59arqM+PYH1NVV513rVuiqtqmqr5QVe8anxv3VVBVp1fVyVV1YlUdPy7zebMKqmqXqnpLVX11/My/vbFfeVV14/F4X//1o6p6nLFfeVX1+PHf1y9V1ZvGf3d91q8wgXkrUVXbJHlpknskuVmSB1TVzeZb1RbtNUnuvmDZ4Uk+3N37Jfnw+JxN75IkT+zumya5XZJHj8e68V95P0tyl+6+VZJ1Se5eVbdL8ndJXjCO/XlJHjHHGrdkf5nkKzPPjfvquXN3r5v50y4+b1bHPyZ5X3ffJMmtMhz/xn6FdffXxuN9XZIDkvwkyb/F2K+oqto7yWOTHNjd+yfZJskh8Vm/4gTmrcdtk3yju0/t7p8neXOSe825pi1Wd388ybkLFt8rydHj46OT3HtVi9pKdPd3u/vz4+MLMvwCtXeM/4rrwY/Hp9uOX53kLkneMi439iugqq6T5OAkrxqfV4z7PPm8WWFVtVOSOyb5lyTp7p939w9j7FfbXZN8s7u/FWO/GtYkuVpVrUmyQ5Lvxmf9ihOYtx57Jzlj5vl3xmWsnmt393eTIdQl2WPO9Wzxqmptkl9N8pkY/1UxnhZ8YpKzk3wwyTeT/LC7Lxmb+OxZGS9M8pQkl43PrxXjvlo6yQeq6oSqetS4zOfNyrt+ku8nOWq8FOFVVXX1GPvVdkiSN42Pjf0K6u4zkzwvybczBOXzk5wQn/UrTmDeetTEMrdIZ4tVVddI8tYkj+vuH827nq1Fd186nqZ3nQxnttx0qtnqVrVlq6rfS3J2d58wu3iiqXFfGb/R3bfOcMnTo6vqjvMuaCuxJsmtk/xTd/9qkgvjFOBVNV4re88kx867lq3BeE34vZJcL8leSa6e4XNnIZ/1m5jAvPX4TpLrzjy/TpKz5lTL1up7VbVnkozfz55zPVusqto2Q1h+Q3e/bVxs/FfReGrkRzNcR77LePpY4rNnJfxGkntW1ekZLre5S4YZZ+O+Crr7rPH72Rmu47xtfN6shu8k+U53f2Z8/pYMAdrYr557JPl8d39vfG7sV9ZBSU7r7u939/9v795i7KrqOI5/fw7X2krT0oBipaIEQxtF8EZtSBHjlUibtIFaFaz64EvxgTRqgiZqDb7gNb6oaIy10AhTGh9ArIXQGBWYoa0CXqIoRFBTYgmSIujfh70mPYxnOqg9HXPm+3na57/3Xvt/Vk7WzP+stfd5GrgJWI5j/cBZMM8edwFntifpHUe3hGbHDOc02+wALm/blwM3z2AuQ6vdu/kN4P6qurZnl/0/YEkWJZnftk+k++N+P7ALWNMOs++PsKr6WFW9uKqW0I3tP6qq9djvA5fk+UnmTWwDbwF+juPNwFXVo8BDSc5qoYuA+7Dvj6Z1HFqODfb9oP0BeEOSOe1/nYnPvGP9gKXKWfvZIsk76GYdRoDrqmrzDKc0tJJsBVYCJwN/Aj4JbAe2AS+hG/TWVtXkB4Ppf5RkBXAnsI9D93N+nO4+Zvt/gJK8ku6BIyN0X8huq6pPJTmDbuZzATAOvKeqnpq5TIdXkpXAVVV1sf0+eK2PR9vLY4DvVtXmJAtxvBm4JOfQPejuOOC3wPtpYw/2/UAlmUP3bJwzqupAi/m5H7D2c42X0v0iyDjwQbp7lh3rB8iCWZIkSZKkPlySLUmSJElSHxbMkiRJkiT1YcEsSZIkSVIfFsySJEmSJPVhwSxJkiRJUh8WzJIkDaEkq5NUklfMdC7/qSS3J3nrpNhHknx1mvOeGGxmkqTZxoJZkqThtA7YDVw2yIskGRlAs1v597wva3FJko4aC2ZJkoZMkrnAG4EPMKnwTLIpyb4ke5Jc02IvT/LDFhtL8rIkK5N8v+e8ryS5om0/mOQTSXYDa5N8KMld7fwbk8xpx52SZLTF9yRZnuTTSa7saXdzko2T3sL3gIuTHN+OWQK8CNidZG6SnS3PfUku6fP+D5f7eUnuSHJPkluTvLDFNya5L8neJNf/N/0uSRo+x8x0ApIk6YhbBdxSVb9K8liSc6tqLMnb277XV9WTSRa047cA11TVaJIT6L5QXzzNNQ5W1QqAJAur6mtt+zN0hfqXgS8Bd1TV6jYTPRf4I3AT8MUkz6Mr6F/X23BV7U/yM+BtwM3tmBuqqpIcBFZX1eNJTgZ+kmRHVdV0nZLk2JbXJVX1lySXApuBDcBHgZdW1VNJ5k/XliRpdrBgliRp+KwDvtC2r2+vx4A3A9+sqicBquqxJPOA06pqtMUOAiSZ7ho39Gwva4XyfLqi+NYWfxPwvtbuP4ADwIEk+5O8GjgFGK+q/X3an1iWPVEwb2jxAJ9NcgHwT+C01s6j0yUMnAUsA25r728EeKTt2wtsSbId2P4c2pIkzQIWzJIkDZEkC+kK1WVJiq4orCSb6IrNyTOxU1XGz/DsW7dOmLT/bz3b3wJWVdWetvR55TRpfh24AjgVuG6KY7YD1yY5FzixqsZafD2wCDivqp5O8mCf3KbKPcAvqur8Ptd7J3AB8C7g6iRLq+qZad6HJGnIeQ+zJEnDZQ3w7ao6vaqWVNVi4HfACuAHwIaee4wXVNXjwMNJVrXY8W3/74Gz2+uTgIsOc815wCNtyfP6nvhO4MOt3ZEkL2jxUbrl1q/l0Gz0s1TVE8DtdAV178O+TgL+3IrlC4HT+5w+Ve6/BBYlOb/ldGySpW1p+OKq2gVs4tBMuSRplrNgliRpuKyjK0h73Qi8u6puAXYAdye5F7iq7X8vsDHJXuDHwKlV9RCwjbZUGRg/zDWvBn4K3AY80BO/ErgwyT7gHmApQFX9HdgFbGtLtaeyFXgV3bLyCVuA1yS5m644f2DySVPl3q67Bvhckj3AvcByuln477Q8x4HPV9VfD5OXJGmWyHN4RoYkSdIR02Z0x4C1VfXrmc5HkqSpOMMsSZKOmiRnA78BdlosS5L+3znDLEmSJElSH84wS5IkSZLUhwWzJEmSJEl9WDBLkiRJktSHBbMkSZIkSX1YMEuSJEmS1IcFsyRJkiRJffwLlRNJeBICKT4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 1080x576 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "plt.figure(figsize=(15,8))\n",
    "accuracy_matrix = [dt_accuracy, rf_accuracy, lr_accuracy, gbt_accuracy]\n",
    "model_names = ['Decision Tree', 'Random Forest', 'Logistic Regression', 'GBT Classifier']\n",
    "plt.xlabel('Accuracy Values')\n",
    "plt.ylabel('Model Names')\n",
    "plt.title('Bar Plot Comparisions of different models used for predictions')\n",
    "plt.barh(model_names, accuracy_matrix)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the above plot we can see that the accuracy of all the four models is around the 80 to 85 range. Thus we can tell that all the four models are performing very good classification on the given dataset. However if a close comparision of all the models are done then we can see that the GBT Classifier model performs the best out of the given models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 09: Calculate the confusion matrix and find the precision, recall, and F1 score of each classification algorithm. Explain how the accuracy of the predication can be improved?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now in this step we have to calculate the precession recall and F1 score for all the models which we have used for performing classification on the given dataset.\n",
    "\n",
    "For doing this we would use the **confusion matrix**. The confusion matrix is considered as the most important metric. A confusion matrix is also known as the error matrix and this is a table which would be used for analysing the performance of an algorithm. Each row would represent instance of a predicted class and each column would represent instance of of an actual class. Using the confusion matrix we can calsulate other metrics such as precession, accuracy, recall and the F1 score of various models which we have used."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Decision Tree Metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The confusion matrix of any model can be obtained using the confusionMatrix function of pyspark. \n",
    "The metrics for the Decision Tree model is as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion Matrix\n",
      "[[31672.  1294.]\n",
      " [ 5547.  3881.]]\n",
      "\n",
      "Summary Stats\n",
      "Precision of Negative Class (RainTomorrow=False) is = 0.850963217711384\n",
      "Precision of Positive Class (RainTomorrow=True) is =  0.749951690821256\n",
      "Recall of Negative Class (RainTomorrow=False) is =    0.9607474367530182\n",
      "Recall of Positive Class (RainTomorrow=True) is =     0.41164616037335594\n",
      "F1 Score = 0.8386328253998208\n"
     ]
    }
   ],
   "source": [
    "# Metrics for Decision Tree\n",
    "\n",
    "from pyspark.mllib.evaluation import MulticlassMetrics\n",
    "from pyspark.mllib.util import MLUtils\n",
    "predictionAndLabels = predictions_dt.select(['prediction', 'RainTomorrow'])\n",
    "metrics = MulticlassMetrics(predictionAndLabels.rdd.map(list))\n",
    "confusion_mat = metrics.confusionMatrix()\n",
    "print(\"Confusion Matrix\")\n",
    "print(confusion_mat.toArray())\n",
    "# Overall statistics\n",
    "f1Score = metrics.fMeasure()\n",
    "print(\"\\nSummary Stats\")\n",
    "print(\"Precision of Negative Class (RainTomorrow=False) is = %s\" % metrics.precision(0))\n",
    "print(\"Precision of Positive Class (RainTomorrow=True) is =  %s\" % metrics.precision(1))\n",
    "print(\"Recall of Negative Class (RainTomorrow=False) is =    %s\" % metrics.recall(0))\n",
    "print(\"Recall of Positive Class (RainTomorrow=True) is =     %s\" % metrics.recall(1))\n",
    "print(\"F1 Score = %s\" % f1Score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Precession, Recall and the F1Score can be found by another method. This is by using the confusion matrix values in the formulae. \n",
    "\n",
    "The formulae for Precession is given by:\n",
    "* Precesion = (True Positive)/ (True Positive + False Positive)\n",
    "* Recall = (True Positive)/ (True Positive + False Negative)\n",
    "* F1 Score = 2 * (precesion * recall) / (precesion + recall)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The precision of the Decision Tree model is:  0.850963217711384\n",
      "The Recall of the Decision Tree model is: 0.9607474367530182\n",
      "The f1 score of the Decision tree model is:  0.9025290304196053\n"
     ]
    }
   ],
   "source": [
    "cm_dt=confusion_mat.toArray()\n",
    "precision_dt=(cm_dt[0][0])/(cm_dt[0][0]+cm_dt[1][0])\n",
    "recall_dt=(cm_dt[0][0])/(cm_dt[0][0]+cm_dt[0][1])\n",
    "f1_dt= 2 * (precision_dt * recall_dt) / (precision_dt + recall_dt)\n",
    "\n",
    "print(\"The precision of the Decision Tree model is: \", precision_dt)\n",
    "print(\"The Recall of the Decision Tree model is:\" , recall_dt)\n",
    "print(\"The f1 score of the Decision tree model is: \",f1_dt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Random Forest Metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we find the metrics for the random forest classifier. The metrics for the random forest classifier is as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion Matrix\n",
      "[[31945.  1021.]\n",
      " [ 5895.  3533.]]\n",
      "\n",
      "Summary Stats\n",
      "Precision of Negative Class (RainTomorrow=False) is = 0.8442124735729387\n",
      "Precision of Positive Class (RainTomorrow=True) is =  0.7758014931927976\n",
      "Recall of Negative Class (RainTomorrow=False) is =    0.969028696232482\n",
      "Recall of Positive Class (RainTomorrow=True) is =     0.37473483241408573\n",
      "F1 Score = 0.8368637071283672\n"
     ]
    }
   ],
   "source": [
    "# Confusion matrix for Random Forest\n",
    "\n",
    "from pyspark.mllib.evaluation import MulticlassMetrics\n",
    "from pyspark.mllib.util import MLUtils\n",
    "predictionAndLabels = predictions_rf.select(['prediction', 'RainTomorrow'])\n",
    "metrics = MulticlassMetrics(predictionAndLabels.rdd.map(list))\n",
    "confusion_mat = metrics.confusionMatrix()\n",
    "print(\"Confusion Matrix\")\n",
    "print(confusion_mat.toArray())\n",
    "# Overall statistics\n",
    "f1Score = metrics.fMeasure()\n",
    "print(\"\\nSummary Stats\")\n",
    "print(\"Precision of Negative Class (RainTomorrow=False) is = %s\" % metrics.precision(0))\n",
    "print(\"Precision of Positive Class (RainTomorrow=True) is =  %s\" % metrics.precision(1))\n",
    "print(\"Recall of Negative Class (RainTomorrow=False) is =    %s\" % metrics.recall(0))\n",
    "print(\"Recall of Positive Class (RainTomorrow=True) is =     %s\" % metrics.recall(1))\n",
    "print(\"F1 Score = %s\" % f1Score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now using the formulae and from the confusion matrix we find the value for precession, recall and f1 score. This is done for the random forest model now. Code is as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The precision of the Decision Tree model is:  0.8442124735729387\n",
      "The Recall of the Decision Tree model is: 0.969028696232482\n",
      "The f1 score of the Decision tree model is:  0.902324661751829\n"
     ]
    }
   ],
   "source": [
    "cm_rf=confusion_mat.toArray()\n",
    "precision_rf=(cm_rf[0][0])/(cm_rf[0][0]+cm_rf[1][0])\n",
    "recall_rf=(cm_rf[0][0])/(cm_rf[0][0]+cm_rf[0][1])\n",
    "f1_rf= 2 * (precision_rf * recall_rf) / (precision_rf + recall_rf)\n",
    "\n",
    "print(\"The precision of the Decision Tree model is: \", precision_rf)\n",
    "print(\"The Recall of the Decision Tree model is:\" , recall_rf)\n",
    "print(\"The f1 score of the Decision tree model is: \",f1_rf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Logistic Regression Metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we find the metrics for the Logistic Regression classifier. The metrics for the logistic regression classifier is as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion Matrix\n",
      "[[31248.  1718.]\n",
      " [ 5042.  4386.]]\n",
      "\n",
      "Summary Stats\n",
      "Precision of Negative Class (RainTomorrow=False) is = 0.8610636538991457\n",
      "Precision of Positive Class (RainTomorrow=True) is =  0.7185452162516383\n",
      "Recall of Negative Class (RainTomorrow=False) is =    0.9478857004186131\n",
      "Recall of Positive Class (RainTomorrow=True) is =     0.46521001272804413\n",
      "F1 Score = 0.8405434731329905\n"
     ]
    }
   ],
   "source": [
    "# Confusion matrix for Logistic Regression\n",
    "\n",
    "from pyspark.mllib.evaluation import MulticlassMetrics\n",
    "from pyspark.mllib.util import MLUtils\n",
    "predictionAndLabels = predictions_lr.select(['prediction', 'RainTomorrow'])\n",
    "metrics = MulticlassMetrics(predictionAndLabels.rdd.map(list))\n",
    "confusion_mat = metrics.confusionMatrix()\n",
    "print(\"Confusion Matrix\")\n",
    "print(confusion_mat.toArray())\n",
    "# Overall statistics\n",
    "f1Score = metrics.fMeasure()\n",
    "print(\"\\nSummary Stats\")\n",
    "print(\"Precision of Negative Class (RainTomorrow=False) is = %s\" % metrics.precision(0))\n",
    "print(\"Precision of Positive Class (RainTomorrow=True) is =  %s\" % metrics.precision(1))\n",
    "print(\"Recall of Negative Class (RainTomorrow=False) is =    %s\" % metrics.recall(0))\n",
    "print(\"Recall of Positive Class (RainTomorrow=True) is =     %s\" % metrics.recall(1))\n",
    "print(\"F1 Score = %s\" % f1Score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now using the formulae and from the confusion matrix we find the value for precession, recall and f1 score. This is done for the logistic regression model now. Code is as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The precision of the Decision Tree model is:  0.8610636538991457\n",
      "The Recall of the Decision Tree model is: 0.9478857004186131\n",
      "The f1 score of the Decision tree model is:  0.902391128566478\n"
     ]
    }
   ],
   "source": [
    "cm_lr=confusion_mat.toArray()\n",
    "precision_lr=(cm_lr[0][0])/(cm_lr[0][0]+cm_lr[1][0])\n",
    "recall_lr=(cm_lr[0][0])/(cm_lr[0][0]+cm_lr[0][1])\n",
    "f1_lr= 2 * (precision_lr * recall_lr) / (precision_lr + recall_lr)\n",
    "\n",
    "print(\"The precision of the Decision Tree model is: \", precision_lr)\n",
    "print(\"The Recall of the Decision Tree model is:\" , recall_lr)\n",
    "print(\"The f1 score of the Decision tree model is: \",f1_lr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### GBT Classifier Metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we find the metrics for the gbt classifier. The metrics for the gbt classifier is as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion Matrix\n",
      "[[31368.  1598.]\n",
      " [ 5119.  4309.]]\n",
      "\n",
      "Summary Stats\n",
      "Precision of Negative Class (RainTomorrow=False) is: 0.8597034560254337\n",
      "Precision of Positive Class (RainTomorrow=True) is:  0.7294735060098189\n",
      "Recall of Negative Class (RainTomorrow=False) is =    0.9515258144755202\n",
      "Recall of Positive Class (RainTomorrow=True) is =     0.45704285108188375\n",
      "F1 Score = 0.8415577676086239\n"
     ]
    }
   ],
   "source": [
    "# Confusion Matrix for GBT Classifier\n",
    "\n",
    "from pyspark.mllib.evaluation import MulticlassMetrics\n",
    "from pyspark.mllib.util import MLUtils\n",
    "predictionAndLabels = predictions_gbt.select(['prediction', 'RainTomorrow'])\n",
    "metrics = MulticlassMetrics(predictionAndLabels.rdd.map(list))\n",
    "confusion_mat = metrics.confusionMatrix()\n",
    "print(\"Confusion Matrix\")\n",
    "print(confusion_mat.toArray())\n",
    "\n",
    "# Overall statistics\n",
    "f1Score = metrics.fMeasure()\n",
    "print(\"\\nSummary Stats\")\n",
    "print(\"Precision of Negative Class (RainTomorrow=False) is: %s\" % metrics.precision(0))\n",
    "print(\"Precision of Positive Class (RainTomorrow=True) is:  %s\" % metrics.precision(1))\n",
    "print(\"Recall of Negative Class (RainTomorrow=False) is =    %s\" % metrics.recall(0))\n",
    "print(\"Recall of Positive Class (RainTomorrow=True) is =     %s\" % metrics.recall(1))\n",
    "print(\"F1 Score = %s\" % f1Score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now using the formulae and from the confusion matrix we find the value for precession, recall and f1 score. This is done for the gbt classifier model now. Code is as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The precision of the Decision Tree model is:  0.8597034560254337\n",
      "The Recall of the Decision Tree model is: 0.9515258144755202\n",
      "The f1 score of the Decision tree model is:  0.9032871150274286\n"
     ]
    }
   ],
   "source": [
    "cm_gbt=confusion_mat.toArray()\n",
    "precision_gbt=(cm_gbt[0][0])/(cm_gbt[0][0]+cm_gbt[1][0])\n",
    "recall_gbt=(cm_gbt[0][0])/(cm_gbt[0][0]+cm_gbt[0][1])\n",
    "f1_gbt= 2 * (precision_gbt * recall_gbt) / (precision_gbt + recall_gbt)\n",
    "\n",
    "print(\"The precision of the Decision Tree model is: \", precision_gbt)\n",
    "print(\"The Recall of the Decision Tree model is:\" , recall_gbt)\n",
    "print(\"The f1 score of the Decision tree model is: \",f1_gbt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have completed finding the accuracy and metrics for all the models we would now create a dataframe of model names along with the accuracy values. This would be as follows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.sql.types import *\n",
    "\n",
    "cSchema = StructType([StructField(\"Model Names\", StringType()),StructField(\"Accuacy Values\", DoubleType())])\n",
    "\n",
    "accuracy_matrix = [dt_accuracy, rf_accuracy, lr_accuracy, gbt_accuracy]\n",
    "model_names = ['Decision Tree', 'Random Forest', 'Logistic Regression', 'GBT Classifier']\n",
    "test_list = [list(x) for x in zip(model_names, accuracy_matrix)]\n",
    "\n",
    "df = spark.createDataFrame(test_list,schema=cSchema) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------------------+-----------------+\n",
      "|        Model Names|   Accuacy Values|\n",
      "+-------------------+-----------------+\n",
      "|      Decision Tree|82.00235971625308|\n",
      "|      Random Forest|81.40446045082305|\n",
      "|Logistic Regression|84.05434731329905|\n",
      "|     GBT Classifier| 84.1557767608624|\n",
      "+-------------------+-----------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As seen all the models have more or less the same values for accuracy. **However on very close comparision of accuracy and metrics we can say thay GBT Classifier is the best model for this data and for predicting the rainfall in australia.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How can we increase the accuracy of the model?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is one of the most frequent questions which is asked in machine learning. However the answer for this is not straightforward and simple.\n",
    "\n",
    "* There could be many ways of increasing the accuracy of a model. One of the best way would be to **get more data**. The more data we have the **more training data we would have and this means that the model would be more accurate and precise**. A more precise model would inturn lead to better accuracy.\n",
    "\n",
    "\n",
    "* Also once we have huge amounts of data this does not mean that the accuracy is going to be better. We have to properly **preprocess** the given data. This means we have to **treat the missing values** for the data and all the **outliers** should be removed from the dataset. The outliers of a dataset could be removed using the boxplot technique.\n",
    "\n",
    "\n",
    "* Also one of the things which we have to do before starting off with machine learning modelling is **feature engineering and feature selection**. A usual dataset would have many columns however all the columns would not be useful in machine learning prediction tasks. Hence we have to find out the best features which would be required for making the model and implementing machine learning algorithms.\n",
    "\n",
    "\n",
    "* **Multiple Algorithms** have to be used for the best results.There are many models for classification technique and using just one model would not be a good way to proceed. For a certain dataset one machine learning model would be good however the same model might not be the best used for another type of data. Hence we have to experiment with multiple models and compare their accuracy and metrics to select the best model.\n",
    "\n",
    "\n",
    "* We could also use **Parameter and Algorithm Tuning**. In some case if we fix the model however the accuracy is still not appriciable we could do parameter/algorithm tuning. In parameter tuning the parameter are tuned by a pyspark method so that the model would get better data and predictions would be more accurate. We can also tune the parameters of the model to achieve higher values of accuracy.\n",
    "\n",
    "\n",
    "* The last way in which we could obtain higher values of accuracy is by **Ensemble methods**. We could use bagging and boosting methods to split the data in the right proportion and to achieve better results in machine learning."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### References:\n",
    "* Spark Documentation - (https://spark.apache.org/docs/latest)\n",
    "* https://www.analyticsvidhya.com/blog/2015/12/improve-machine-learning-results/\n",
    "* Monash Lectures and Tutorials"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}