prac08_production_and_testing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Practical 8: Production and Testing\n",
    "\n",
    "### In this practical note\n",
    "1. [Training a basic model]()\n",
    "2. [Data pipeline]()\n",
    "3. [Serialisation]()\n",
    "4. [Deployment and working with other technologies]()\n",
    "4. [Other tips]()\n",
    "---\n",
    "\n",
    "The practical note for this week introduces you to the practical aspect of Python machine learning models in production environment. Production environment is where a model gets new data and makes predictions to help make decision. We will discuss differences of between models in training phase and production phase, how to build a data pipeline, how to serialise models and other tips related to production. This topic is a seldom one to discuss in an university unit, but I believe it is equally as important as other theoretical components.\n",
    "\n",
    "We have learned how to perform preprocessing and cleaning on a dataset, building and validating model, selecting important features and conducting both supervised and unsupervised data mining tasks. This practical will introduce you on concepts beyond the training, what happened after your model is finished and how to ensure your model is production ready.\n",
    "\n",
    "To ensure sufficient time to finish your workload, anything discussed in this week will not be graded unit components (assignment/exam).\n",
    "\n",
    "## 1. Training a basic model\n",
    "\n",
    "As a foundation to start this practical, we will first train a `LogisticRegression` model. The dataset used will be the `veteran.csv` dataset that you are very familiar with (used from practical 1-5). We will not explain too much of this stage as it is quite similar with practical 1-3 steps, please read the comments on the code for step-by-step instructions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train accuracy: 0.593510324484\n",
      "Test accuracy: 0.562284927736\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.56      0.57      0.57      1453\n",
      "          1       0.56      0.55      0.56      1453\n",
      "\n",
      "avg / total       0.56      0.56      0.56      2906\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# import required libraries\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import classification_report, accuracy_score\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "from dm_tools import data_prep\n",
    "\n",
    "# preprocessing step\n",
    "df = data_prep()\n",
    "\n",
    "# set the random seed - consistent\n",
    "rs = 10\n",
    "\n",
    "# train test split\n",
    "y = df['TargetB']\n",
    "X = df.drop(['TargetB'], axis=1)\n",
    "X_mat = X.as_matrix()\n",
    "X_train, X_test, y_train, y_test = train_test_split(X_mat, y, test_size=0.3, stratify=y, random_state=rs)\n",
    "\n",
    "# initialise a standard scaler object\n",
    "scaler = StandardScaler()\n",
    "\n",
    "# learn the mean and std.dev of variables from training data\n",
    "# then use the learned values to transform training data\n",
    "X_train = scaler.fit_transform(X_train, y_train)\n",
    "X_test = scaler.transform(X_test)\n",
    "\n",
    "# initial model\n",
    "init_model = LogisticRegression(random_state=rs)\n",
    "\n",
    "# fit it to training data\n",
    "init_model.fit(X_train, y_train)\n",
    "\n",
    "# training and test accuracy\n",
    "print(\"Train accuracy:\", init_model.score(X_train, y_train))\n",
    "print(\"Test accuracy:\", init_model.score(X_test, y_test))\n",
    "\n",
    "# classification report on test data\n",
    "y_pred = init_model.predict(X_test)\n",
    "print(classification_report(y_test, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we have previously, the model performs on acceptable level on both training and test data. **Assume** that after many experiments with different models and data preprocessing steps, we found that this model produces best accuracy on validation data. We have decided to use this model as our deployment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Data Pipeline\n",
    "\n",
    "An important step before deploying our model into production system is to ensure all steps performed in data preparation and feature engineering are consistent and complete. With many replacement, imputation, transformation and dropping steps performed before a dataset is clean, it is common to make mistake, lose important actions and have \"data leakage\". `sklearn` provides a method to consolidate all steps into one object through `pipeline`.\n",
    "\n",
    "In building `sklearn` pipeline, there are 4 important classes used:\n",
    "1. `BaseEstimator` and `TransformerMixin`, used as parent classes to build `Transformers`. Transformers are classes responsible for applying a preparation or transformation step onto a dataset. A Transformer is the smallest component in a `Pipeline`.\n",
    "2. `Pipeline`. Pipelines are a method to pack each `Transformer` in your logic sequentially into one function call, making it easier to perform training and predictions. Given 3 transformers A, B and C, a Pipeline will run the input dataset through A, pass the output to B, then C, and so on (in order of A->B->C). With a Pipeline, you can ensure consistent steps are applied into to training, validation, test and future data. Pipeline class also implements fit, transform and prediction functions, automating many of the function calls commonly used by a model. You can build a Pipeline through `Pipeline()` class initialisation or `make_pipeline` function. We will use `make_pipeline` mainly for simpler call.\n",
    "3. `FeatureUnion`. FeatureUnions are quite similar with pipelines. While pipelines run each step sequentially, `FeatureUnion` join each transformer in paralel. Instead of A->B->C, FeatureUnion will return a larger set of results from [A, B, C]. Similar with Pipeline, we will also use the equivalent function of `make_union`.\n",
    "\n",
    "To start building your pipeline, import classes above as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.pipeline import Pipeline, make_pipeline, FeatureUnion, make_union\n",
    "from sklearn.base import TransformerMixin, BaseEstimator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Review your `data_prep` function from the `dm_tools` module. It reads dataset from a `.csv` file in form of `pandas` dataframe and returns a dataframe too. `sklearn` APIs, however, take numpy arrays as input and output. To help us transition pandas DataFrame into numpy arrays/matrices and apply transformation steps, we will create a helper transformer as follows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class SimpleTransformer(BaseEstimator, TransformerMixin):\n",
    "    \"\"\"Apply given transformation into a subset of DataFrame and return numpy array.\n",
    "    Accepts 3 parameters:\n",
    "    1. trans_func: transformative function to be applied to selected columns of DataFrame\n",
    "    2. untrans_func: inverse transformative function to be applied to numpy array (mostly unused)\n",
    "    3. columns: columns in this DataFrame where the trans_func will be applied\n",
    "    \"\"\"\n",
    "    def __init__(self, trans_func, untrans_func, columns):\n",
    "        self.transform_func = trans_func\n",
    "        self.inverse_transform_func = untrans_func\n",
    "        self.cols = columns\n",
    "\n",
    "    def fit(self, x, y=None):\n",
    "        return self\n",
    "\n",
    "    \n",
    "    def transform(self, x):\n",
    "        x = self._get_selection(x)\n",
    "        return self.transform_func(x) if callable(self.transform_func) else x\n",
    "\n",
    "    def inverse_transform(self, x):\n",
    "        return self.inverse_transform_func(x) \\\n",
    "            if callable(self.inverse_transform_func) else x\n",
    "\n",
    "    def _get_selection(self, df):\n",
    "        assert isinstance(df, pd.DataFrame)\n",
    "        return df[self.cols]\n",
    "\n",
    "    def get_feature_names(self):\n",
    "        return self.cols"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After that, we could start building some transformations. There are a number of columns where we applied transformation functions in the `data_prep` function (in logical order):\n",
    "1. Drop **ID, TargetD, TargetB** from dataframe.\n",
    "2. Transform **DemCluster** transformed into string datatype.\n",
    "3. Replaced errorneous values in **DemHomeOwner** and **DemMedIncome**.\n",
    "4. One hot encoding for all categorical functions.\n",
    "5. Mean imputation and standard scaling for all numerical columns.\n",
    "\n",
    "The first step in this process is to recognise which columns are to be transformed. Find the code and its explaination as follow:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# get the original dataset as reference\n",
    "df2 = pd.read_csv('datasets/veteran.csv')\n",
    "\n",
    "# drop unnecessary columns - step 1\n",
    "dropped = df2.drop(['ID', 'TargetD', 'TargetB'], axis=1)\n",
    "\n",
    "# get list of all columns\n",
    "all_columns = dropped.columns.tolist()\n",
    "\n",
    "# set aside columns with special processing steps - step 2 and 3\n",
    "trans_columns = ['DemCluster', 'DemHomeOwner', 'DemMedIncome']\n",
    "\n",
    "# get list of all categorical columns - for step 4\n",
    "obj_columns = dropped.columns[dropped.dtypes == object].tolist() \n",
    "int_columns = list(set(all_columns) - set(obj_columns) - set(trans_columns))  # get int columns\n",
    "obj_columns = list(set(obj_columns) - set(trans_columns))  # get obj columns without special columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*(bracketed number) refers to the section in code below.*\n",
    "\n",
    "Once you have set aside all columns for processing, we can start applying transformations for all categorical columns. We will create two different Transformers, one for `DemCluster` (1) and one for all other categorical columns (2). (1) will change `DemCluster` into string and one hot encode it, while (2) will simply one hot encode all passed columns. Results from both transformers will be joined using a FeatureUnion.\n",
    "\n",
    "After you have fixed the object columns, you could use similar step for `DemMedIncome` to remove noisy 0 values (3) and `DemHomeOwner` to change it into a binary variable. (4) Once all above operations are performed, we could join all dataframes from (1-4) into one using FeatureUnion (5). This joined dataframe will then be mean imputed using Imputer object (6) and scaled (7).\n",
    "\n",
    "One advantage of Pipeline is you can attach a model at the end of the pipeline. This way, when you call `fit` or `predict` function on Pipeline, it will apply all transformative functions before training the model and making predictions. We will fit a LogisticRegression model at the end of this Pipeline - same model with our initial model. (8).\n",
    "\n",
    "Find the code as below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler, Imputer  # import imputer and scaler\n",
    "\n",
    "# create your feature union\n",
    "# transforming `DemCluster` into categorical and running one hot encoding on all categorical columns\n",
    "string_pipe = make_pipeline(\n",
    "    make_union(\n",
    "        SimpleTransformer(lambda x: pd.get_dummies(x.astype(str)), None, ['DemCluster']),  # (1)\n",
    "        SimpleTransformer(lambda x: pd.get_dummies(x), None, obj_columns) # 2\n",
    "    )\n",
    ")\n",
    "\n",
    "# remove noise in `DemMedIncome` and change `DemHomeOwner` into binary\n",
    "noise_pipe = make_pipeline(\n",
    "    make_union(\n",
    "        SimpleTransformer(lambda x: x.replace(0, np.nan), None, ['DemMedIncome']), # 3\n",
    "        SimpleTransformer(lambda x: x.replace(['U', 'H'], [0, 1]), None, ['DemHomeOwner'])\n",
    "    )\n",
    ")\n",
    "\n",
    "# union both pipes above, impute and scale the dataframe into a numpy array\n",
    "pipeline = make_pipeline(\n",
    "    make_union(string_pipe,\n",
    "               noise_pipe,\n",
    "               SimpleTransformer(None, None, int_columns)),  # 5\n",
    "    Imputer(),  # 6\n",
    "    StandardScaler(),  # 7\n",
    "    LogisticRegression(random_state=rs))  # 8"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With a pipeline set up, you could easily apply similar preprocessing steps for both training and test data. Apply the pipeline into `df2` and see how it applies all data preprocessing steps and training for the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pipeline training score:  0.593510324484\n",
      "Pipeline test score:  0.562284927736\n",
      "Pipeline(memory=None,\n",
      "     steps=[('featureunion', FeatureUnion(n_jobs=1,\n",
      "       transformer_list=[('pipeline-1', Pipeline(memory=None,\n",
      "     steps=[('featureunion', FeatureUnion(n_jobs=1,\n",
      "       transformer_list=[('simpletransformer-1', SimpleTransformer(columns=None, trans_func=None, untrans_func=None)), ('simpletransformer-...alty='l2', random_state=10, solver='liblinear', tol=0.0001,\n",
      "          verbose=0, warm_start=False))])\n"
     ]
    }
   ],
   "source": [
    "# create X and Y\n",
    "df_X = df2.drop(['TargetB'], axis=1)\n",
    "df_y = df2['TargetB']\n",
    "\n",
    "# split X and Y into train and test for model validation purposes\n",
    "# notice that df_X, df_X_train and df_X_test are still pandas DataFrame instead of numpy array\n",
    "df_X_train, df_X_test, df_y_train, df_y_test = train_test_split(df_X, df_y, test_size=0.3,\n",
    "                                                                stratify=df_y, random_state=rs)\n",
    "\n",
    "# apply transformation and train model in one go\n",
    "pipeline.fit(df_X_train, df_y_train)\n",
    "\n",
    "# score pipeline on training and test data\n",
    "print(\"Pipeline training score: \", pipeline.score(df_X_train, df_y_train))\n",
    "print(\"Pipeline test score: \", pipeline.score(df_X_test, df_y_test))\n",
    "print(pipeline)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The pipeline fitted model performs exactly the same with the initial model on both training and test dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## 3. Serialisation\n",
    "\n",
    "As `sklearn` models and pipelines are stored in memory during training and experiments, you need a method to save them in a persistent format if you were to deploy your model in production. One option is by serialising the model and all pipeline components into a binary file. Binary file allow easy versioning and easy deployment through file transfer. `sklearn` official documentation recommends using `pickle` to serialise pipeline and the model, however in this practical we will use an alternative to `pickle` called `dill`. `dill` allows you not only to save the model, but also relevant functions used throughout the pipeline.\n",
    "\n",
    "You can install dill using `pip install dill` command in your Anaconda prompt.\n",
    "\n",
    "Use the following code to serialise our pipeline through `dill`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pipeline properties before dilling\n",
      "========\n",
      " Pipeline(memory=None,\n",
      "     steps=[('featureunion', FeatureUnion(n_jobs=1,\n",
      "       transformer_list=[('pipeline-1', Pipeline(memory=None,\n",
      "     steps=[('featureunion', FeatureUnion(n_jobs=1,\n",
      "       transformer_list=[('simpletransformer-1', SimpleTransformer(columns=None, trans_func=None, untrans_func=None)), ('simpletransformer-...alty='l2', random_state=10, solver='liblinear', tol=0.0001,\n",
      "          verbose=0, warm_start=False))])\n",
      "Model score before dilling:  0.562284927736\n"
     ]
    }
   ],
   "source": [
    "import dill\n",
    "\n",
    "# show model score before serialising\n",
    "print(\"Pipeline properties before dilling\\n========\\n\", pipeline)\n",
    "print(\"Model score before dilling: \", pipeline.score(df_X_test, df_y_test))\n",
    "\n",
    "# dump to file named model.pkl, written in binary mode\n",
    "dill.dump(file=open('model.pkl', 'wb'), obj=pipeline)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the model/pipeline is serialise, you could easily transport it everywhere. Provided you have installed `sklearn`, `pandas` and other key libraries in your target machine, you could reload the pipeline using this code below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pipeline properties after dilling\n",
      "========\n",
      " Pipeline(memory=None,\n",
      "     steps=[('featureunion', FeatureUnion(n_jobs=1,\n",
      "       transformer_list=[('pipeline-1', Pipeline(memory=None,\n",
      "     steps=[('featureunion', FeatureUnion(n_jobs=1,\n",
      "       transformer_list=[('simpletransformer-1', SimpleTransformer(columns=None, trans_func=None, untrans_func=None)), ('simpletransformer-...alty='l2', random_state=10, solver='liblinear', tol=0.0001,\n",
      "          verbose=0, warm_start=False))])\n",
      "Model score after dilling:  0.562284927736\n"
     ]
    }
   ],
   "source": [
    "dilled_model = dill.load(file=open('model.pkl', 'rb'))\n",
    "\n",
    "print(\"Pipeline properties after dilling\\n========\\n\", dilled_model)\n",
    "print(\"Model score after dilling: \", dilled_model.score(df_X_test, df_y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model is serialised and stored perfectly.\n",
    "\n",
    "## 4. Deployment and working with other technologies\n",
    "\n",
    "While we have been using Python for building and testing our model, many times your model will be deployed in a production system built using other languages/platforms, such as Java, Go or Scala. In this case, you need a method for your model and the production system to communicate with each other.\n",
    "\n",
    "One option is to deploy the model in Python as a web API. Using this set up, components built with other languages/platforms in the production system can make predictions by sending data through HTTP request. All data preprocessing steps will be handled by Python. This option is quite popular as it is simple and require least amount of work from both production and data scientist/analyst teams. A major downside of this technique is speed. For some applications that send large amount of information for prediction (e.g. picture for image recognition) or require time sensitive predictions (Python is quite slow compared to C, Java or Go), this technique might not be suitable.\n",
    "\n",
    "In this practical, we have prepared a simple web server prototype in `prac08_prediction_server_flask.ipynb`. This web server is built using Flask, a Python-based web server framework and it receives POST requests for predictions at `http://127.0.0.1:5000/predict` endpoint.\n",
    "\n",
    "Please run the web server and follow the code in the cell below to send prediction requests to it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "200\n"
     ]
    }
   ],
   "source": [
    "# import requests library to help making POST requests\n",
    "import requests\n",
    "\n",
    "# use the first 1000 rows of X_test, serialise it to JSON\n",
    "post_data = df_X_test.iloc[:1000].to_json()\n",
    "\n",
    "# post it to the running server\n",
    "result = requests.post(url='http://127.0.0.1:5000/predict', json=post_data)\n",
    "print(result.status_code)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If your status code is 200, it means the prediction is successful. You could see the result of the prediction through `result.text`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1, 1, 1, 0, 0, 0, 0, 0, 1, 0, 1, 0, 1, 1, 0, 1, 1, 0, 0, 0, 1, 0, 0, 1, 1, 0, 1, 1, 1, 0, 1, 0, 1, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 0, 1, 0, 1, 1, 0, 1, 1, 0, 1, 1, 0, 0, 1, 1, 0, 0, 1, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, 1, 0, 0, 0, 0, 1, 1, 0, 1, 1, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 1, 0, 1, 1, 0, 0, 1, 0, 0, 0, 1, 1, 0, 1, 0, 1, 0, 1, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 1, 1, 0, 0, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 1, 1, 0, 1, 0, 0, 0, 1, 1, 1, 0, 1, 0, 0, 0, 0, 0, 1, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 0, 1, 0, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 1, 0, 1, 1, 0, 0, 1, 0, 0, 0, 0, 1, 1, 0, 0, 1, 0, 1, 0, 1, 0, 0, 1, 0, 0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 1, 0, 1, 1, 0, 0, 1, 1, 0, 0, 0, 1, 0, 0, 1, 1, 1, 1, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 1, 1, 0, 1, 1, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 1, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 1, 1, 0, 1, 0, 1, 0, 1, 1, 1, 0, 0, 1, 1, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 1, 1, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 0, 1, 0, 0, 1, 0, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 1, 0, 1, 1, 1, 1, 0, 1, 0, 1, 0, 0, 1, 0, 0, 1, 0, 1, 1, 1, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0, 1, 1, 0, 1, 1, 1, 0, 1, 1, 0, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 1, 1, 0, 0, 1, 0, 0, 1, 1, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, 0, 0, 1, 0, 1, 1, 1, 1, 0, 1, 0, 0, 0, 1, 1, 0, 1, 0, 0, 1, 0, 1, 0, 0, 1, 0, 1, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 1, 0, 1, 0, 1, 0, 0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 1, 1, 0, 1, 0, 1, 0, 1, 0, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 0, 0, 0, 0, 1, 0, 0, 1, 1, 0, 1, 1, 0, 1, 1, 1, 0, 1, 1, 0, 0, 1, 1, 0, 1, 1, 0, 0, 0, 1, 0, 0, 1, 0, 1, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 0, 0, 1, 0, 1, 0, 1, 1, 0, 1, 0, 1, 1, 1, 0, 1, 0, 0, 1, 1, 1, 1, 1, 0, 0, 1, 0, 0, 0, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 0, 1, 0, 1, 1, 1, 0, 1, 1, 0, 0, 1, 0, 0, 1, 1, 1, 0, 0, 0, 1, 0, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 0, 0, 0, 1, 1, 0, 0, 1, 0, 0, 0, 1, 1, 0, 1, 1, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 0, 0, 0, 1, 1, 0, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 1, 0, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 0, 0, 1, 0, 1, 1, 1, 1, 0, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 1, 0, 0, 1, 1, 0, 0, 1, 1, 1, 0, 0, 0, 1, 0, 1, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 1, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 0, 1, 0, 0, 0, 1, 1, 1, 0, 1, 0, 1, 1, 1, 0, 1, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0, 1, 1, 0, 0, 0, 0, 1, 1, 0, 0, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 0, 1, 1, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1, 0, 0, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0]\n"
     ]
    }
   ],
   "source": [
    "print(result.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another option is to replicate data processing steps and export the model weights in other language. This is commonly used where high performance/low prediction latency is necessary. As we have learned in previous practicals and lectures, after training process is finished, many data mining models can be represented as series of weights and connections (LogisticRegression being the best example). These weights can be easily exported/serialised and used in other technologies/languages. A clear disadvantage of using this deployment option is there is way more work required as the production has to re-implement the model exactly as it was during training. With many complicated operations and data preprocessing steps in a pipeline, this implementation process can be prone to bugs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Other tips\n",
    "\n",
    "This section discusses various performance tips that you could use to help your model works better in production setting.\n",
    "\n",
    "### 4.1. Batch prediction\n",
    "\n",
    "In many production systems, real-time atomic predictions is not required. Often prediction tasks can be stashed and delayed to be then processed in batch basis. Batch processing allows the model to take advantage of vectorised computations and SIMD instructions (single instruction, multiple data), which speeds up computational performance significantly.\n",
    "\n",
    "To refer a real-life example, GoCardless, a YCombinator funded startup which focuses on cashless payments, uses machine learning models to perform fraud detection activities. Most payment transactions in the company take 1-2 days, therefore, batch prediction is very suitable for their business case. In the end, the fraud detection is run a nightly-cron job (every 24 hours).\n",
    "\n",
    "### 4.2. Vertical scaling\n",
    "\n",
    "Vertical scaling refers to the process of accomodating growing volume of computing task by increasing the computing power of one task. Typically vertical scaling is implemented through buying a faster, better computer. Vertical scaling can allow a production system to handle more data mining task requests by running each task faster serially or using multithreading/multiprocessing.\n",
    "\n",
    "### 4.3. Horizontal scaling\n",
    "\n",
    "Different with vertical scaling, horizontal scaling spreading the large number of requests over large number of commodity hardware. This methodology is the key to scale your infrastructure in an efficient and effective manner. Horizontal scaling fits data mining model deployments really well as many predictive tasks are stateless (e.g. each prediction/row are considered to be unrelated from one to another), which mean large number of predictions can easily be distributed over many models.\n",
    "\n",
    "### 4.4. Testing and retraining\n",
    "\n",
    "After a deployment, a model must be monitored periodically. As training often took place using a snapshot of training data, the model prediction quality typically degrades as the environment shifts away from the conditions that were originally captured. After a certain timeframe, the error could grow to be significant, thus the model has to be retrained or replaced.\n",
    "\n",
    "In this condition, there are two common methods to update the model. First, you could periodically collect newer data and train a \"challenger\" model. Both old model and the new challenger are then tested on new stream of data. If the challenger manages to outperform the older model, this new model and its pipeline are then deployed.\n",
    "\n",
    "The second paradigm is to use online training. In online training, newer data that becomes available in a sequential order is used to train and update model, fifferent from batch learning which waits for a large set of training data. Online learning is commonly used for training a model over a very large dataset that does not fit the memory. Online trained models are also capable of adapting to newer patterns in data, leading to common usage in time-related tasks such as stock price predictions."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}