sagemaker_autopilot_direct_marketing_lab.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Direct Marketing with Amazon SageMaker AutoPilot\n",
    "_**Supervised Learning with Gradient Boosted Trees: A Binary Prediction Problem With Unbalanced Classes**_\n",
    "\n",
    "Last update: February 4th, 2020\n",
    "\n",
    "---\n",
    "\n",
    "## Background\n",
    "Direct marketing, either through mail, email, phone, etc., is a common tactic to acquire customers.  Because resources and a customer's attention is limited, the goal is to only target the subset of prospects who are likely to engage with a specific offer.  Predicting those potential customers based on readily available information like demographics, past interactions, and environmental factors is a common machine learning problem.\n",
    "\n",
    "This notebook will invoke SageMaker AutoPilot to find the best ML pipeline to train a model on this dataset, which can be used to predict if a customer will enroll for a term deposit at a bank, after one or more phone calls. Hyperparameter tuning will be used in order to try multiple hyperparameter settings and produce the best model.\n",
    "\n",
    "We will use SageMaker Python SDK, a high level SDK, to simplify the way we interact with SageMaker Hyperparameter Tuning.\n",
    "\n",
    "---\n",
    "\n",
    "## Preparation\n",
    "\n",
    "Let's start by specifying:\n",
    "\n",
    "- The S3 bucket and prefix that you want to use for training and model data.  We'll use the default bucket. If you want to use your own, make sure it is in the **same region** as SageMaker.\n",
    "- The IAM role used to give training access to your data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "isConfigCell": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "import smdebug_rulesconfig as rule_configs\n",
    "import boto3\n",
    "import os, sys\n",
    "\n",
    "print (sagemaker.__version__)\n",
    "\n",
    "sess   = sagemaker.Session()\n",
    "bucket = sess.default_bucket()                     \n",
    "prefix = 'sagemaker/DEMO-automl-dm'\n",
    "region = boto3.Session().region_name\n",
    "\n",
    "# Role when working on a notebook instance\n",
    "role = sagemaker.get_execution_role()\n",
    "# Role when working locally\n",
    "#role = "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sm = boto3.Session().client(service_name='sagemaker',region_name=region)\n",
    "sm_rt = boto3.Session().client('runtime.sagemaker', region_name=region)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Downloading the data set\n",
    "Let's start by downloading the [direct marketing dataset](https://archive.ics.uci.edu/ml/datasets/bank+marketing) from UCI's ML Repository."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget -N https://archive.ics.uci.edu/ml/machine-learning-databases/00222/bank-additional.zip\n",
    "!unzip -o bank-additional.zip"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to load this CSV file and inspect it. Please don't write custom Python code to do this!\n",
    "\n",
    "Instead, developers typically use libraries such as:\n",
    "\n",
    "Pandas: a library providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language.\n",
    "Numpy: a fundamental package for scientific computing with Python.\n",
    "Along the way, we'll use functions from these two libraries. You should definitely become familiar with them, they will make your life much easier when working with large datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np  # For matrix operations and numerical processing\n",
    "import pandas as pd # For munging tabular data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html\n",
    "data = pd.read_csv('./bank-additional/bank-additional-full.csv', sep=';')\n",
    "pd.set_option('display.max_columns', 500)     # Make sure we can see all of the columns\n",
    "pd.set_option('display.max_rows', 50)         # Keep the output on one page\n",
    "data[:10] # Show the first 10 lines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please take the time to look at the data. \n",
    "\n",
    "The name of the column of the dataset you want to predict is `y`. \n",
    "Running the next cell, you'll be able to see the classes are unbalanced."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import seaborn as sns\n",
    "sns.set(style=\"darkgrid\")\n",
    "ax = sns.countplot(x=\"y\", data=data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.shape # (number of lines, number of columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Splitting the dataset\n",
    "\n",
    "We split the dataset into training (95%) and test (5%) datasets. We will use the training dataset for AutoML, where it will be automatically split again for training and validation.\n",
    " \n",
    "Once the model has been deployed, we'll use the test dataset to evaluate its performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the seed to 123 for reproductibility\n",
    "# https://pandas.pydata.org/pandas-docs/version/0.25/generated/pandas.DataFrame.sample.html\n",
    "# https://docs.scipy.org/doc/numpy-1.15.1/reference/generated/numpy.split.html\n",
    "train_data, test_data, _ = np.split(data.sample(frac=1, random_state=123), \n",
    "                                                  [int(0.95 * len(data)), int(len(data))])  \n",
    "\n",
    "# Save to CSV files\n",
    "train_data.to_csv('automl-train.csv', index=False, header=True, sep=',') # Need to keep column names\n",
    "test_data.to_csv('automl-test.csv', index=False, header=True, sep=',')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls -l automl*.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**No preprocessing needed!** AutoML will take care of this, so let's just copy the training set to S3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sess.upload_data(path=\"automl-train.csv\", key_prefix=prefix + \"/input\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setting up the SageMaker AutoPilot job\n",
    "\n",
    "After uploading the dataset to S3, we can invoke SageMaker AutoPilot to find the best ML pipeline to train a model on this dataset. \n",
    "\n",
    "The required inputs for invoking a SageMaker AutoML job are the dataset location in S3, the name of the column of the dataset you want to predict and an IAM role."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please fill the name of the column of the dataset you want to predict for `TargetAttributeName` attribute.\n",
    "\n",
    "You can also read in the documentation [create_auto_ml_job](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker.html#SageMaker.Client.create_auto_ml_job)\n",
    "\n",
    "* CompletionCriteria (dict) - How long a job is allowed to run, or how many candidates a job is allowed to generate.\n",
    "* MaxCandidates (integer) - The maximum number of times a training job is allowed to run.\n",
    "* MaxRuntimePerTrainingJobInSeconds (integer) - The maximum time, in seconds, a job is allowed to run.\n",
    "* MaxAutoMLJobRuntimeInSeconds (integer) - The maximum time, in seconds, an AutoML job is allowed to wait for a trial to complete. It must be equal to or greater than MaxRuntimePerTrainingJobInSeconds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "job_config = {\n",
    "    'CompletionCriteria': {\n",
    "      'MaxRuntimePerTrainingJobInSeconds': 600,\n",
    "      'MaxCandidates': 10,\n",
    "      'MaxAutoMLJobRuntimeInSeconds': 3600\n",
    "    },\n",
    "}\n",
    "\n",
    "job_objective={ 'MetricName': 'F1' }\n",
    "\n",
    "problem_type='BinaryClassification'\n",
    "\n",
    "input_data_config = [{\n",
    "      'DataSource': {\n",
    "        'S3DataSource': {\n",
    "          'S3DataType': 'S3Prefix',\n",
    "          'S3Uri': 's3://{}/{}/input'.format(bucket,prefix)\n",
    "        }\n",
    "      },\n",
    "      'TargetAttributeName': '<...PLEASE FILL...>' # Please fill the name of the column of the dataset you want to predict.\n",
    "    }\n",
    "]\n",
    "\n",
    "output_data_config = {\n",
    "    'S3OutputPath': 's3://{}/{}/output'.format(bucket,prefix)\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Launching the SageMaker AutoPilot job\n",
    "\n",
    "We can now launch the job by calling the `create_auto_ml_job` API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from time import gmtime, strftime, sleep\n",
    "timestamp_suffix = strftime('%d-%H-%M-%S', gmtime())\n",
    "\n",
    "auto_ml_job_name = 'automl-dm-' + timestamp_suffix\n",
    "print('AutoMLJobName: ' + auto_ml_job_name)\n",
    "\n",
    "sm.create_auto_ml_job(AutoMLJobName=auto_ml_job_name,\n",
    "                      InputDataConfig=input_data_config,\n",
    "                      OutputDataConfig=output_data_config,\n",
    "                      ProblemType=problem_type,\n",
    "                      AutoMLJobObjective=job_objective,\n",
    "                      AutoMLJobConfig=job_config,\n",
    "                      RoleArn=role)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tracking the progress of the AutoPilot job\n",
    "SageMaker AutoPilot job consists of four high-level steps : \n",
    "* Data Preprocessing, where the dataset is split into train and validation sets.\n",
    "* Recommending Pipelines, where the dataset is analyzed and SageMaker AutoPilot comes up with a list of ML pipelines that should be tried out on the dataset.\n",
    "* Automatic Feature Engineering, where SageMaker AutoPilot performs feature transformation on individual features of the dataset as well as at an aggregate level.\n",
    "* ML pipeline selection and hyperparameter tuning, where the top performing pipeline is selected along with the optimal hyperparameters for the training algorithm (the last stage of the pipeline). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will run for 5-10 minutes. Hold tight and don't go anywhere yet. You'll be instructed when it's time for a coffee break."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "job = sm.describe_auto_ml_job(AutoMLJobName=auto_ml_job_name)\n",
    "job_status = job['AutoMLJobStatus']\n",
    "job_sec_status = job['AutoMLJobSecondaryStatus']\n",
    "\n",
    "if job_status not in ('Stopped', 'Failed'):\n",
    "    while job_status in ('InProgress') and job_sec_status in ('AnalyzingData'):\n",
    "        job = sm.describe_auto_ml_job(AutoMLJobName=auto_ml_job_name)\n",
    "        job_status = job['AutoMLJobStatus']\n",
    "        job_sec_status = job['AutoMLJobSecondaryStatus']\n",
    "        print (strftime('%Y-%m-%d %H:%M:%S', gmtime()),job_status, job_sec_status)\n",
    "        sleep(30)\n",
    "    print(\"Data analysis complete\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Viewing notebooks generated by SageMaker AutoPilot\n",
    "Once data analysis is complete, SageMaker AutoPilot generates two notebooks: \n",
    "* Data exploration,\n",
    "* Candidate definition."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "job = sm.describe_auto_ml_job(AutoMLJobName=auto_ml_job_name)\n",
    "job_candidate_notebook = job['AutoMLJobArtifacts']['CandidateDefinitionNotebookLocation']\n",
    "job_data_notebook = job['AutoMLJobArtifacts']['DataExplorationNotebookLocation']\n",
    "\n",
    "print(job_candidate_notebook)\n",
    "print(job_data_notebook)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's copy these two notebooks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%sh -s $job_candidate_notebook $job_data_notebook\n",
    "aws s3 cp $1 .\n",
    "aws s3 cp $2 ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Go back to the folder view, and open these notebooks. Lots of useful information in there!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tracking the progress of the AutoPilot Feature Engineering and Model Tuning stages\n",
    "Let's wait for Feature Engineering, ML pipeline selection and hyperparameter tuning to complete.\n",
    "\n",
    "This will run for 10-15 minutes, so there's time for a coffee break too. If you're too hardcore for a coffee break, you can use the time to learn more about the Amazon SageMaker automatic model tuning:\n",
    "* [AWS Machine Learning Blog](https://aws.amazon.com/blogs/machine-learning/amazon-sagemaker-automatic-model-tuning-produces-better-models-faster/)\n",
    "\n",
    "You can also read about the [F1 metric](https://en.wikipedia.org/wiki/F1_score) that we will use: \n",
    "* **Precision**: the ratio tp / (tp + fp) where tp is the number of true positives and fp the number of false positives. The precision is intuitively the ability of the classifier not to label as positive a sample that is negative.\n",
    "* **Recall**: the ratio tp / (tp + fn) where tp is the number of true positives and fn the number of false negatives. The recall is intuitively the ability of the classifier to find all the positive samples.\n",
    "* **F1 score**: a weighted mean of precision and recall. 1 is the best possible score and 0 is the worst."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While the job is running, why not head out to the SageMaker web console and spend a few minutes familiarizing yourself with the [Hyperparameter tuning jobs](https://console.aws.amazon.com/sagemaker/home?region=us-east-1#/hyper-tuning-jobs) section?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "job = sm.describe_auto_ml_job(AutoMLJobName=auto_ml_job_name)\n",
    "job_status = job['AutoMLJobStatus']\n",
    "job_sec_status = job['AutoMLJobSecondaryStatus']\n",
    "\n",
    "while job_status not in ('Failed', 'Completed', 'Stopped'):\n",
    "    describe_response = sm.describe_auto_ml_job(AutoMLJobName=auto_ml_job_name)\n",
    "    job_status = describe_response['AutoMLJobStatus']\n",
    "\n",
    "    print (strftime('%Y-%m-%d %H:%M:%S', gmtime()), \n",
    "           describe_response['AutoMLJobStatus'] + \" - \" + describe_response['AutoMLJobSecondaryStatus'])\n",
    "    sleep(30)    \n",
    "\n",
    "print(\"Model tuning complete\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Viewing all candidates explored by SageMaker AutoPilot\n",
    "Once model tuning is complete, you can view all the candidates (pipeline evaluations with different hyperparameter combinations) that were explored by AutoML and sort them by their final performance metric."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(sm.list_candidates_for_auto_ml_job(AutoMLJobName=auto_ml_job_name))\n",
    "candidates = sm.list_candidates_for_auto_ml_job(AutoMLJobName=auto_ml_job_name, \n",
    "                                                SortBy='FinalObjectiveMetricValue')['Candidates']\n",
    "index = 1\n",
    "for candidate in candidates:\n",
    "  print (str(index) + \"  \" \n",
    "         + candidate['CandidateName'] + \"  \" \n",
    "         + str(candidate['FinalAutoMLJobObjectiveMetric']['Value']))\n",
    "  index += 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inspect SageMaker AutoPilot trials with Amazon SageMaker Experiments"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "SageMaker AutoPilot automatically creates a new experiment, and pushes information for each trial. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.analytics import ExperimentAnalytics, TrainingJobAnalytics\n",
    "\n",
    "exp = ExperimentAnalytics(\n",
    "    sagemaker_session=sess, \n",
    "    experiment_name=auto_ml_job_name + '-aws-auto-ml-job',\n",
    ")\n",
    "\n",
    "df = exp.dataframe()\n",
    "\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deploying the best candidate\n",
    "Now that we have successfully completed the AutoML job on our dataset and visualized the trials, we can create a model from any of the trials with a single API call and then deploy that model for online or batch prediction using [Inference Pipelines](https://docs.aws.amazon.com/sagemaker/latest/dg/inference-pipelines.html). \n",
    "\n",
    "An inference pipeline is an Amazon SageMaker model that is composed of a linear sequence of two to five containers that process requests for inferences on data. You can use an inference pipeline to combine preprocessing, predictions, and post-processing data science tasks. Inference pipelines are fully managed. This allows you to send and receive predictions from the model in the unprocessed format.\n",
    "\n",
    "For this notebook, we deploy only the best performing trial for inference."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The best candidate is the one we're really interested in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "candidates=sm.describe_auto_ml_job(AutoMLJobName=auto_ml_job_name)\n",
    "print(candidates)\n",
    "best_candidate = sm.describe_auto_ml_job(AutoMLJobName=auto_ml_job_name)['BestCandidate']\n",
    "best_candidate_identifier = best_candidate['CandidateName']\n",
    "\n",
    "print(\"Candidate name: \" + best_candidate_identifier)\n",
    "print(\"Metric name: \" + best_candidate['FinalAutoMLJobObjectiveMetric']['MetricName'])\n",
    "print(\"Metric value: \" + str(best_candidate['FinalAutoMLJobObjectiveMetric']['Value']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see the containers and models composing the Inference Pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for container in best_candidate['InferenceContainers']:\n",
    "    print(container['Image'])\n",
    "    print(container['ModelDataUrl'])\n",
    "    print('-')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_name = 'automl-dm-model-' + timestamp_suffix\n",
    "\n",
    "model_arn = sm.create_model(Containers=best_candidate['InferenceContainers'],\n",
    "                            ModelName=model_name,\n",
    "                            ExecutionRoleArn=role)\n",
    "\n",
    "print('Best candidate model ARN: ', model_arn['ModelArn'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's deploy the pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# EndpointConfig name\n",
    "timestamp_suffix = strftime('%d-%H-%M-%S', gmtime())\n",
    "epc_name = 'automl-dm-epc-' + timestamp_suffix\n",
    "\n",
    "# Endpoint name\n",
    "ep_name = 'automl-dm-ep-' + timestamp_suffix\n",
    "variant_name = 'automl-dm-variant-' + timestamp_suffix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deploying the best model\n",
    "\n",
    "Now we will deploy the best model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ep_config = sm.create_endpoint_config(EndpointConfigName = epc_name,\n",
    "                                      ProductionVariants=[{'InstanceType':'ml.m4.xlarge',\n",
    "                                                           'InitialInstanceCount':1,\n",
    "                                                           'ModelName':model_name,\n",
    "                                                           'VariantName':variant_name}])\n",
    "\n",
    "\n",
    "create_endpoint_response = sm.create_endpoint(EndpointName=ep_name,\n",
    "                                              EndpointConfigName=epc_name)\n",
    "print(create_endpoint_response['EndpointArn'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will run for approximately 5 minutes. We are almost done.\n",
    "\n",
    "While you're waiting, head out to the SageMaker web console and familiarize yourself with the [Endpoints section](https://console.aws.amazon.com/sagemaker/home?region=us-east-1#/endpoints)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "sm.get_waiter('endpoint_in_service').wait(EndpointName=ep_name)\n",
    "\n",
    "resp = sm.describe_endpoint(EndpointName=ep_name)\n",
    "status = resp['EndpointStatus']\n",
    "\n",
    "print(\"Arn: \" + resp['EndpointArn'])\n",
    "print(\"Status: \" + status)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scoring the best candidate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's predict and score the test set. We'll compute metrics ourselves just for fun.\n",
    "\n",
    "In the cell below you can see the input for the prediction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tp = tn = fp = fn = count = 0\n",
    "\n",
    "with open('automl-test.csv') as f:\n",
    "    lines = f.readlines()\n",
    "    for l in lines[1:10]:   # Skip header\n",
    "        l = l.split(',')  # Split CSV line into features\n",
    "        label = l[-1]     # Store 'yes'/'no' label\n",
    "        l = l[:-1]        # Remove label\n",
    "        l = ','.join(l)   # Rebuild CSV line without label\n",
    "        print(l)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the next cell we will predict over the entire test set, and calculate true positive, true negative, false positive, and false negative, to later, calculate confusion matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tp = tn = fp = fn = count = 0\n",
    "\n",
    "with open('automl-test.csv') as f:\n",
    "    lines = f.readlines()\n",
    "    for l in lines[1:]:   # Skip header\n",
    "        l = l.split(',')  # Split CSV line into features\n",
    "        label = l[-1]     # Store 'yes'/'no' label\n",
    "        l = l[:-1]        # Remove label\n",
    "        l = ','.join(l)   # Rebuild CSV line without label\n",
    "                \n",
    "        response = sm_rt.invoke_endpoint(EndpointName=ep_name, ContentType='text/csv', Accept='text/csv', Body=l)\n",
    "\n",
    "        response = response['Body'].read().decode(\"utf-8\")\n",
    "        #print (\"label %s response %s\" %(label,response))\n",
    "\n",
    "        if 'yes' in label:\n",
    "            # Sample is positive\n",
    "            if 'yes' in response:\n",
    "                # True positive\n",
    "                tp=tp+1\n",
    "            else:\n",
    "                # False negative\n",
    "                fn=fn+1\n",
    "        else:\n",
    "            # Sample is negative\n",
    "            if 'no' in response:\n",
    "                # True negative\n",
    "                tn=tn+1\n",
    "            else:\n",
    "                # False positive\n",
    "                fp=fp+1\n",
    "        count = count+1\n",
    "        if (count % 100 == 0):   \n",
    "            sys.stdout.write(str(count)+' ')\n",
    "            \n",
    "print (\"Done\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Confusion matrix\n",
    "print (\"%d %d\" % (tn, fp))\n",
    "print (\"%d %d\" % (fn, tp))\n",
    "\n",
    "accuracy  = (tp+tn)/(tp+tn+fp+fn)\n",
    "precision = tp/(tp+fp)\n",
    "recall    = tp/(tp+fn)\n",
    "\n",
    "print (\"accuracy: %.4f, precision: %.4f, recall: %.4f\" % (accuracy, precision, recall))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please calculate f1 score.\n",
    "You can use the following forumla:\n",
    "\\begin{align}\n",
    "F1 = {2 \\times \\frac{precision \\times recall}{precision + recall}}\n",
    "\\end{align}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f1 = <...PLEASE FILL...> # You can calculate f1 score using the formula above. \n",
    "print (\"f1 score: %.4f\" % (f1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How does this compare to what you achieved in the DIY model training Demo Eitan showed previously?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deleting the endpoint\n",
    "Once that we're done predicting, we can delete the endpoint (and stop paying for it)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Uncomment to delete\n",
    "#sm.delete_endpoint(EndpointName=ep_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The SageMaker AutoML job creates many underlying artifacts such as dataset splits, preprocessing scripts, preprocessed data, etc. Let's delete them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s3 = boto3.resource('s3')\n",
    "bucket = s3.Bucket(bucket)\n",
    "\n",
    "job_outputs_prefix = '{}/output/{}'.format(prefix,auto_ml_job_name)\n",
    "print(job_outputs_prefix)\n",
    "\n",
    "# Uncomment to delete\n",
    "#bucket.objects.filter(Prefix=job_outputs_prefix).delete()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  },
  "notice": "Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.  Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance with the License. A copy of the License is located at http://aws.amazon.com/apache2.0/ or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License."
 },
 "nbformat": 4,
 "nbformat_minor": 4
}