sagemaker_autopilot_customer_churn.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Customer Churn Prediction with Amazon SageMaker Autopilot\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Content\n",
    "\n",
    "1. [Prerequisites](#Prerequisites)\n",
    "1. [Preparing the dataset](#Preparing)\n",
    "1. [Setting up the SageMaker Autopilot Job](#Settingup)\n",
    "1. [Launching the SageMaker Autopilot Job](#Launching)\n",
    "1. [Monitoring the SageMaker Autopilot Job](#Monitoring)\n",
    "1. [Reviewing Results](#Results) \n",
    "1. [Performing Inference](#Inference) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "<b><h1>1. Prerequisites</h1></div>\n",
    "<a class=\"anchor\" id=\"Prerequisites\"></a>\n",
    "\n",
    "Two tasks below have to be completed as prerequisites:\n",
    "\n",
    "- The Amazon Simple Storage Service (Amazon S3) bucket and prefix that you want to use for training and model data.\n",
    "- The IAM role to give Autopilot access to your data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import required libraries\n",
    "import sagemaker\n",
    "import boto3\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import itertools\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.metrics import confusion_matrix, accuracy_score\n",
    "from IPython.core.display import display, HTML\n",
    "from sagemaker.analytics import ExperimentAnalytics\n",
    "\n",
    "\n",
    "# Define region, bucket\n",
    "session = sagemaker.Session()\n",
    "region = boto3.Session().region_name\n",
    "bucket = session.default_bucket()\n",
    "\n",
    "# Select an IAM role\n",
    "role = 'arn:aws:iam::690429675927:role/role_sagemaker' \n",
    "   \n",
    "# Define sm object to be used to call Sagemaker services \n",
    "sm = boto3.Session().client(service_name='sagemaker',region_name=region)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_confusion_matrix(cm, classes,\n",
    "                          normalize=False,\n",
    "                          title='Confusion matrix',\n",
    "                          cmap=plt.cm.Blues):\n",
    "    \"\"\"\n",
    "    See full source and example: \n",
    "    http://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html\n",
    "    \n",
    "    This function prints and plots the confusion matrix.\n",
    "    Normalization can be applied by setting `normalize=True`.\n",
    "    \"\"\"\n",
    "    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n",
    "    plt.title(title)\n",
    "    plt.colorbar()\n",
    "    tick_marks = np.arange(len(classes))\n",
    "    plt.xticks(tick_marks, classes, rotation=45)\n",
    "    plt.yticks(tick_marks, classes)\n",
    "\n",
    "    if normalize:\n",
    "        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n",
    "        print(\"Normalized confusion matrix\")\n",
    "    else:\n",
    "        print('Confusion matrix, without normalization')\n",
    "\n",
    "    thresh = cm.max() / 2.\n",
    "    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n",
    "        plt.text(j, i, cm[i, j],\n",
    "                 horizontalalignment=\"center\",\n",
    "                 color=\"white\" if cm[i, j] > thresh else \"black\")\n",
    "\n",
    "    plt.tight_layout()\n",
    "    plt.ylabel('True label')\n",
    "    plt.xlabel('Predicted label')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "toc-hr-collapsed": true
   },
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "<b><h1>2. Preparing the dataset</h1></div>\n",
    "<a class=\"anchor\" id=\"Preparing\"></a>\n",
    "\n",
    "The dataset used for this notebook is a IBM Watson Telco Dataset. This dataset describes a binary classification problem where the <strong>“Churn”</strong> column is the target."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>customerID</th>\n",
       "      <th>gender</th>\n",
       "      <th>SeniorCitizen</th>\n",
       "      <th>Partner</th>\n",
       "      <th>Dependents</th>\n",
       "      <th>tenure</th>\n",
       "      <th>PhoneService</th>\n",
       "      <th>MultipleLines</th>\n",
       "      <th>InternetService</th>\n",
       "      <th>OnlineSecurity</th>\n",
       "      <th>OnlineBackup</th>\n",
       "      <th>DeviceProtection</th>\n",
       "      <th>TechSupport</th>\n",
       "      <th>StreamingTV</th>\n",
       "      <th>StreamingMovies</th>\n",
       "      <th>Contract</th>\n",
       "      <th>PaperlessBilling</th>\n",
       "      <th>PaymentMethod</th>\n",
       "      <th>MonthlyCharges</th>\n",
       "      <th>TotalCharges</th>\n",
       "      <th>Churn</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7590-VHVEG</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "      <td>1</td>\n",
       "      <td>No</td>\n",
       "      <td>No phone service</td>\n",
       "      <td>DSL</td>\n",
       "      <td>No</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>Month-to-month</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Electronic check</td>\n",
       "      <td>29.85</td>\n",
       "      <td>29.85</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5575-GNVDE</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>34</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "      <td>DSL</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>One year</td>\n",
       "      <td>No</td>\n",
       "      <td>Mailed check</td>\n",
       "      <td>56.95</td>\n",
       "      <td>1889.5</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3668-QPYBK</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>2</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "      <td>DSL</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>Month-to-month</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Mailed check</td>\n",
       "      <td>53.85</td>\n",
       "      <td>108.15</td>\n",
       "      <td>Yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>7795-CFOCW</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>45</td>\n",
       "      <td>No</td>\n",
       "      <td>No phone service</td>\n",
       "      <td>DSL</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>One year</td>\n",
       "      <td>No</td>\n",
       "      <td>Bank transfer (automatic)</td>\n",
       "      <td>42.30</td>\n",
       "      <td>1840.75</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>9237-HQITU</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>2</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "      <td>Fiber optic</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>Month-to-month</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Electronic check</td>\n",
       "      <td>70.70</td>\n",
       "      <td>151.65</td>\n",
       "      <td>Yes</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   customerID  gender  SeniorCitizen Partner Dependents  tenure PhoneService  \\\n",
       "0  7590-VHVEG  Female              0     Yes         No       1           No   \n",
       "1  5575-GNVDE    Male              0      No         No      34          Yes   \n",
       "2  3668-QPYBK    Male              0      No         No       2          Yes   \n",
       "3  7795-CFOCW    Male              0      No         No      45           No   \n",
       "4  9237-HQITU  Female              0      No         No       2          Yes   \n",
       "\n",
       "      MultipleLines InternetService OnlineSecurity OnlineBackup  \\\n",
       "0  No phone service             DSL             No          Yes   \n",
       "1                No             DSL            Yes           No   \n",
       "2                No             DSL            Yes          Yes   \n",
       "3  No phone service             DSL            Yes           No   \n",
       "4                No     Fiber optic             No           No   \n",
       "\n",
       "  DeviceProtection TechSupport StreamingTV StreamingMovies        Contract  \\\n",
       "0               No          No          No              No  Month-to-month   \n",
       "1              Yes          No          No              No        One year   \n",
       "2               No          No          No              No  Month-to-month   \n",
       "3              Yes         Yes          No              No        One year   \n",
       "4               No          No          No              No  Month-to-month   \n",
       "\n",
       "  PaperlessBilling              PaymentMethod  MonthlyCharges TotalCharges  \\\n",
       "0              Yes           Electronic check           29.85        29.85   \n",
       "1               No               Mailed check           56.95       1889.5   \n",
       "2              Yes               Mailed check           53.85       108.15   \n",
       "3               No  Bank transfer (automatic)           42.30      1840.75   \n",
       "4              Yes           Electronic check           70.70       151.65   \n",
       "\n",
       "  Churn  \n",
       "0    No  \n",
       "1    No  \n",
       "2   Yes  \n",
       "3    No  \n",
       "4   Yes  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = pd.read_csv('https://raw.githubusercontent.com/rstudio/keras-customer-churn/master/data/WA_Fn-UseC_-Telco-Customer-Churn.csv',nrows=1000)\n",
    "pd.set_option('display.max_columns', 100) \n",
    "pd.set_option('display.max_rows', 10) \n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Defining the target attribute"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "target_column='Churn'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dividing the data into training and testing splits\n",
    "\n",
    "The training split is used by SageMaker Autopilot. The testing split is reserved to perform inference using the suggested model.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data = data.sample(frac=0.8,random_state=200)\n",
    "test_data = data.drop(train_data.index)\n",
    "test_data_no_target = test_data.drop(columns=[target_column])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((800, 21), (200, 21))"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data.shape, test_data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Uploading the dataset to Amazon S3\n",
    "Copy the file to Amazon Simple Storage Service (Amazon S3) in a .csv format for Amazon SageMaker training to use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<b>Train data uploaded to → </b>s3://sagemaker-us-east-1-690429675927/sagemaker/telco/train/train_data.csv"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<b>Test data uploaded to → </b>s3://sagemaker-us-east-1-690429675927/sagemaker/telco/test/test_data.csv"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Define the S3 bucket and prefix(subfolder within the bucket)\n",
    "prefix = 'sagemaker/telco'\n",
    "\n",
    "train_file = 'train_data.csv';\n",
    "train_data.to_csv(train_file, index=False, header=True)\n",
    "train_data_s3_path = session.upload_data(path=train_file, key_prefix=prefix + \"/train\")\n",
    "\n",
    "display(HTML('<b>Train data uploaded to → </b>'  + train_data_s3_path))\n",
    "\n",
    "test_file = 'test_data.csv';\n",
    "test_data_no_target.to_csv(test_file, index=False, header=False)\n",
    "test_data_s3_path = session.upload_data(path=test_file, key_prefix=prefix + \"/test\")\n",
    "\n",
    "display(HTML('<b>Test data uploaded to → </b>'  + test_data_s3_path))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "<b><h1>3. Setting up the SageMaker Autopilot Job</h1></div>\n",
    "<a class=\"anchor\" id=\"Settingup\"></a>\n",
    "\n",
    "In order to invoke Autopilot service to train the model, the inputs below are required:\n",
    "\n",
    "* Amazon S3 location for input dataset and for all output artifacts\n",
    "* Name of the column of the dataset you want to predict (`target_column` in this case) \n",
    "* An IAM role\n",
    "    \n",
    "Optional inputs:\n",
    "* Tags"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define max number of candidates\n",
    "max_candidates = 3\n",
    "\n",
    "job_config = {\n",
    "    'CompletionCriteria': {\n",
    "      'MaxRuntimePerTrainingJobInSeconds': 300,\n",
    "      'MaxCandidates': max_candidates,\n",
    "      'MaxAutoMLJobRuntimeInSeconds': 1800\n",
    "    },\n",
    "}\n",
    "\n",
    "input_data_config = [{\n",
    "      'DataSource': {\n",
    "        'S3DataSource': {\n",
    "          'S3DataType': 'S3Prefix',\n",
    "          'S3Uri': 's3://{}/{}/train'.format(bucket,prefix)\n",
    "        }\n",
    "      },\n",
    "      'TargetAttributeName': target_column\n",
    "    }\n",
    "  ]\n",
    "\n",
    "output_data_config = {\n",
    "    'S3OutputPath': 's3://{}/{}/output'.format(bucket,prefix)\n",
    "  }\n",
    "\n",
    "# Define a Tag to be used as filter\n",
    "tags_config = [ { \n",
    "         \"Key\": \"Project\",\n",
    "         \"Value\": \"Project_Telco\"\n",
    "          }\n",
    "   ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also specify the type of problem you want to solve with your dataset (`Regression, MulticlassClassification, BinaryClassification`). In case you are not sure, SageMaker Autopilot will infer the problem type based on statistics of the target column (the column you want to predict). \n",
    "\n",
    "You have the option to limit the running time of a SageMaker Autopilot job by providing either the maximum number of pipeline evaluations or candidates (one pipeline evaluation is called a `Candidate` because it generates a candidate model) or providing the total time allocated for the overall Autopilot job. Under default settings, this job takes about four hours to run. This varies between runs because of the nature of the exploratory process Autopilot uses to find optimal training parameters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert a4-block alert-info\">\n",
    "<b><h1>4. Launching the SageMaker Autopilot Job</h1></div>\n",
    "<a class=\"anchor\" id=\"Launching\"></a>\n",
    "\n",
    "You can now launch the Autopilot job by calling the `create_auto_ml_job` API. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<b>AutoMLJobName → </b>automl-telco-2020-10-01-02-31-36"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "{'AutoMLJobArn': 'arn:aws:sagemaker:us-east-1:690429675927:automl-job/automl-telco-2020-10-01-02-31-36',\n",
       " 'ResponseMetadata': {'RequestId': '7a8078cd-d195-4727-ba9e-1eebc1415d14',\n",
       "  'HTTPStatusCode': 200,\n",
       "  'HTTPHeaders': {'x-amzn-requestid': '7a8078cd-d195-4727-ba9e-1eebc1415d14',\n",
       "   'content-type': 'application/x-amz-json-1.1',\n",
       "   'content-length': '103',\n",
       "   'date': 'Thu, 01 Oct 2020 02:31:39 GMT'},\n",
       "  'RetryAttempts': 0}}"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from time import gmtime, strftime, sleep\n",
    "\n",
    "# Define AutoMLJobName\n",
    "timestamp_suffix = strftime('%Y-%m-%d-%H-%M-%S', gmtime())\n",
    "model_prefix  = 'automl-telco'\n",
    "\n",
    "auto_ml_job_name =  model_prefix + '-' + timestamp_suffix\n",
    "\n",
    "display(HTML('<b>AutoMLJobName → </b>'  + auto_ml_job_name))\n",
    "\n",
    "sm.create_auto_ml_job(AutoMLJobName=auto_ml_job_name,\n",
    "                      InputDataConfig=input_data_config,\n",
    "                      OutputDataConfig=output_data_config,\n",
    "                      AutoMLJobConfig=job_config,\n",
    "                      RoleArn=role,\n",
    "                      Tags=tags_config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert a4-block alert-info\">\n",
    "<b><h1>5. Monitoring SageMaker Autopilot job progress</h1></div>\n",
    "<a class=\"anchor\" id=\"Monitoring\"></a>\n",
    "\n",
    "The following code will check the job's status every 30 seconds until the high-level steps complete:\n",
    "\n",
    "* Analyzing Data, where the dataset is analyzed and Autopilot comes up with a list of ML pipelines that should be tried out on the dataset. The dataset is also split into train and validation sets.\n",
    "* Feature Engineering, where Autopilot performs feature transformation on individual features of the dataset as well as at an aggregate level.\n",
    "* Model Tuning, where the top performing pipeline is selected along with the optimal hyperparameters for the training algorithm (the last stage of the pipeline). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<u><b>JobStatus &nbsp&nbsp&nbsp&nbsp-&nbsp&nbsp&nbsp&nbsp Secondary Status</b></u>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "InProgress - Starting\n",
      "InProgress - Starting\n",
      "InProgress - AnalyzingData\n",
      "InProgress - AnalyzingData\n",
      "InProgress - AnalyzingData\n",
      "InProgress - AnalyzingData\n",
      "InProgress - AnalyzingData\n",
      "InProgress - AnalyzingData\n",
      "InProgress - AnalyzingData\n",
      "InProgress - AnalyzingData\n",
      "InProgress - AnalyzingData\n",
      "InProgress - AnalyzingData\n",
      "InProgress - AnalyzingData\n",
      "InProgress - AnalyzingData\n",
      "InProgress - AnalyzingData\n",
      "InProgress - AnalyzingData\n",
      "InProgress - AnalyzingData\n",
      "InProgress - FeatureEngineering\n",
      "InProgress - FeatureEngineering\n",
      "InProgress - FeatureEngineering\n",
      "InProgress - FeatureEngineering\n",
      "InProgress - FeatureEngineering\n",
      "InProgress - FeatureEngineering\n",
      "InProgress - FeatureEngineering\n",
      "InProgress - FeatureEngineering\n",
      "InProgress - FeatureEngineering\n",
      "InProgress - FeatureEngineering\n",
      "InProgress - FeatureEngineering\n",
      "InProgress - FeatureEngineering\n",
      "InProgress - FeatureEngineering\n",
      "InProgress - FeatureEngineering\n",
      "InProgress - FeatureEngineering\n",
      "InProgress - FeatureEngineering\n",
      "InProgress - FeatureEngineering\n",
      "InProgress - FeatureEngineering\n",
      "InProgress - FeatureEngineering\n",
      "InProgress - FeatureEngineering\n",
      "InProgress - ModelTuning\n",
      "InProgress - ModelTuning\n",
      "InProgress - ModelTuning\n",
      "InProgress - ModelTuning\n",
      "InProgress - ModelTuning\n",
      "InProgress - ModelTuning\n",
      "InProgress - ModelTuning\n",
      "InProgress - ModelTuning\n",
      "Completed - MaxCandidatesReached\n",
      "Wall time: 23min 3s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "display(HTML('<u><b>JobStatus &nbsp&nbsp&nbsp&nbsp-&nbsp&nbsp&nbsp&nbsp Secondary Status</b></u>' ))\n",
    "describe_response = sm.describe_auto_ml_job(AutoMLJobName=auto_ml_job_name)\n",
    "print (describe_response['AutoMLJobStatus'] + \" - \" + describe_response['AutoMLJobSecondaryStatus'])\n",
    "job_run_status = describe_response['AutoMLJobStatus']\n",
    "    \n",
    "while job_run_status not in ('Failed', 'Completed', 'Stopped'):\n",
    "    describe_response = sm.describe_auto_ml_job(AutoMLJobName=auto_ml_job_name)\n",
    "    job_run_status = describe_response['AutoMLJobStatus']\n",
    "    print (describe_response['AutoMLJobStatus'] + \" - \" + describe_response['AutoMLJobSecondaryStatus'])\n",
    "    sleep(30)  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "toc-hr-collapsed": true
   },
   "source": [
    "<div class=\"alert a4-block alert-info\">\n",
    "<b><h1>6. Reviewing Results</h1></div>\n",
    "<a class=\"anchor\" id=\"Results\"></a>\n",
    "\n",
    "### Let's review all jobs in the <strong>SageMaker Console</strong>. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SageMaker Console Links</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th><b><p style=\"text-align:left\"><a target=\"blank\" href=\"https://console.aws.amazon.com/sagemaker/home?region=us-east-1#/processing-jobs/\">Processing Jobs</a></p></b></th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th><b><p style=\"text-align:left\"><a target=\"blank\" href=\"https://console.aws.amazon.com/sagemaker/home?region=us-east-1#/jobs/\">Training Jobs</a></p></b></th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th><b><p style=\"text-align:left\"><a target=\"blank\" href=\"https://console.aws.amazon.com/sagemaker/home?region=us-east-1#/hyper-tuning-jobs/\">Hyperparameter Tuning Jobs</a></p></b></th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th><b><p style=\"text-align:left\"><a target=\"blank\" href=\"https://console.aws.amazon.com/sagemaker/home?region=us-east-1#/models/\">Models</a></p></b></th>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "review_aws_list =['<b><p style=\"text-align:left\"><a target=\"blank\" href=\"https://console.aws.amazon.com/sagemaker/home?region={}#/processing-jobs/\">Processing Jobs</a></p></b>'.format(region),\n",
    "     '<b><p style=\"text-align:left\"><a target=\"blank\" href=\"https://console.aws.amazon.com/sagemaker/home?region={}#/jobs/\">Training Jobs</a></p></b>'.format(region),\n",
    "     '<b><p style=\"text-align:left\"><a target=\"blank\" href=\"https://console.aws.amazon.com/sagemaker/home?region={}#/hyper-tuning-jobs/\">Hyperparameter Tuning Jobs</a></p></b>'.format(region),\n",
    "     '<b><p style=\"text-align:left\"><a target=\"blank\" href=\"https://console.aws.amazon.com/sagemaker/home?region={}#/models/\">Models</a></p></b>'.format(region)\n",
    "                 ]\n",
    "\n",
    "df=pd.DataFrame(review_aws_list, columns=['SageMaker Console Links']).set_index('SageMaker Console Links')\n",
    "display(HTML(df.to_html( render_links=True, escape=False)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's review all Candidates"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can view all the candidates (pipeline evaluations with different hyperparameter combinations) that were explored by SageMaker Autopilot. To find them, code below will use the search function to retrieve jobs based on the Tag previously defined."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "results = sm.search(\n",
    "    Resource='TrainingJob',\n",
    "    SearchExpression={\n",
    "        'Filters': [\n",
    "            {\n",
    "                'Name': 'Tags.Project',\n",
    "                'Operator': 'Equals',\n",
    "                'Value': 'Project_Telco'}]},\n",
    "    SortBy = \"CreationTime\",\n",
    "    SortOrder = \"Descending\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Training Job Name</th>\n",
       "      <th>Algorithm Specification</th>\n",
       "      <th>Status</th>\n",
       "      <th>Metric</th>\n",
       "      <th>Metric Value</th>\n",
       "      <th>Training Time</th>\n",
       "      <th>BillableTime</th>\n",
       "      <th>Creation Time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td><a target=\"blank\" href=\"https://console.aws.amazon.com/sagemaker/home?region=us-east-1#/jobs/tuning-job-1-9d22a30d195e4dbe94-003-b81b9ca3\">tuning-job-1-9d22a30d195e4dbe94-003-b81b9ca3</a></td>\n",
       "      <td>sagemaker-xgboost:1.0-1-cpu-py3</td>\n",
       "      <td>Completed</td>\n",
       "      <td>validation:f1</td>\n",
       "      <td>0.61237</td>\n",
       "      <td>47</td>\n",
       "      <td>47</td>\n",
       "      <td>2020-09-30 19:49:49-07:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td><a target=\"blank\" href=\"https://console.aws.amazon.com/sagemaker/home?region=us-east-1#/jobs/tuning-job-1-9d22a30d195e4dbe94-002-8df5e337\">tuning-job-1-9d22a30d195e4dbe94-002-8df5e337</a></td>\n",
       "      <td>sagemaker-xgboost:1.0-1-cpu-py3</td>\n",
       "      <td>Completed</td>\n",
       "      <td>validation:f1</td>\n",
       "      <td>0.61237</td>\n",
       "      <td>53</td>\n",
       "      <td>53</td>\n",
       "      <td>2020-09-30 19:49:47-07:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td><a target=\"blank\" href=\"https://console.aws.amazon.com/sagemaker/home?region=us-east-1#/jobs/tuning-job-1-9d22a30d195e4dbe94-001-ac0354f9\">tuning-job-1-9d22a30d195e4dbe94-001-ac0354f9</a></td>\n",
       "      <td>sagemaker-xgboost:1.0-1-cpu-py3</td>\n",
       "      <td>Completed</td>\n",
       "      <td>validation:f1</td>\n",
       "      <td>0.61237</td>\n",
       "      <td>46</td>\n",
       "      <td>46</td>\n",
       "      <td>2020-09-30 19:49:46-07:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td><a target=\"blank\" href=\"https://console.aws.amazon.com/sagemaker/home?region=us-east-1#/jobs/tuning-job-1-9a6201748f64438998-003-c63c5637\">tuning-job-1-9a6201748f64438998-003-c63c5637</a></td>\n",
       "      <td>sagemaker-xgboost:1.0-1-cpu-py3</td>\n",
       "      <td>Completed</td>\n",
       "      <td>validation:f1</td>\n",
       "      <td>0.53911</td>\n",
       "      <td>43</td>\n",
       "      <td>43</td>\n",
       "      <td>2020-09-30 19:19:36-07:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td><a target=\"blank\" href=\"https://console.aws.amazon.com/sagemaker/home?region=us-east-1#/jobs/tuning-job-1-9a6201748f64438998-002-af2f1b46\">tuning-job-1-9a6201748f64438998-002-af2f1b46</a></td>\n",
       "      <td>sagemaker-xgboost:1.0-1-cpu-py3</td>\n",
       "      <td>Completed</td>\n",
       "      <td>validation:f1</td>\n",
       "      <td>0.53911</td>\n",
       "      <td>71</td>\n",
       "      <td>71</td>\n",
       "      <td>2020-09-30 19:19:35-07:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td><a target=\"blank\" href=\"https://console.aws.amazon.com/sagemaker/home?region=us-east-1#/jobs/tuning-job-1-9a6201748f64438998-001-ceb9f6de\">tuning-job-1-9a6201748f64438998-001-ceb9f6de</a></td>\n",
       "      <td>sagemaker-xgboost:1.0-1-cpu-py3</td>\n",
       "      <td>Completed</td>\n",
       "      <td>validation:f1</td>\n",
       "      <td>0.68651</td>\n",
       "      <td>50</td>\n",
       "      <td>50</td>\n",
       "      <td>2020-09-30 19:19:34-07:00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "headers=[\"Training Job Name\",\"Algorithm Specification\", \"Status\",\"Metric\",\"Metric Value\",\"Training Time\",\"BillableTime\",\"Creation Time\"]\n",
    "rows=[]\n",
    "for result in results['Results']: \n",
    "    trainingJob = result['TrainingJob']\n",
    "    if (not trainingJob['AlgorithmSpecification']['MetricDefinitions']==[] and not trainingJob['TrainingJobStatus']== 'Failed'):\n",
    "        rows.append([\n",
    "         '<a target=\"blank\" href=\"https://console.aws.amazon.com/sagemaker/home?region={}#/jobs/{}\">{}</a>'.format(region,trainingJob['TrainingJobName'],trainingJob['TrainingJobName']),\n",
    "         trainingJob['AlgorithmSpecification']['TrainingImage'].split(\"amazonaws.com/\",1)[1],\n",
    "         trainingJob['TrainingJobStatus'],\n",
    "         trainingJob['FinalMetricDataList'][0]['MetricName'],\n",
    "         trainingJob['FinalMetricDataList'][0]['Value'],               \n",
    "         trainingJob['TrainingTimeInSeconds'],                 \n",
    "         trainingJob['BillableTimeInSeconds'],\n",
    "         trainingJob['CreationTime'],\n",
    "    ])\n",
    "        \n",
    "    if (not trainingJob['AlgorithmSpecification']['MetricDefinitions']==[] and trainingJob['TrainingJobStatus']== 'Failed'):\n",
    "        rows.append([\n",
    "        '<a target=\"blank\" href=\"https://console.aws.amazon.com/sagemaker/home?region={}#/jobs/{}\">{}</a>'.format(region,trainingJob['TrainingJobName'],trainingJob['TrainingJobName']),\n",
    "         trainingJob['AlgorithmSpecification']['TrainingImage'].split(\"amazonaws.com/\",1)[1],\n",
    "         trainingJob['TrainingJobStatus'],\n",
    "         '',\n",
    "         '',               \n",
    "         trainingJob['TrainingTimeInSeconds'],                 \n",
    "         trainingJob['BillableTimeInSeconds'],\n",
    "         trainingJob['CreationTime']\n",
    "    ])\n",
    "\n",
    "df = pd.DataFrame(data=rows,columns=headers)\n",
    "\n",
    "display(HTML(df.to_html( render_links=True, escape=False)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inspect Trials using Experiments API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "SageMaker Autopilot automatically creates a new experiment, and pushes information for each trial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of jobs:  10\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ObjectiveMetric - Max</th>\n",
       "      <th>TrialComponentName</th>\n",
       "      <th>DisplayName</th>\n",
       "      <th>SourceArn</th>\n",
       "      <th>SageMaker.ImageUri</th>\n",
       "      <th>SageMaker.InstanceCount</th>\n",
       "      <th>SageMaker.InstanceType</th>\n",
       "      <th>SageMaker.VolumeSizeInGB</th>\n",
       "      <th>_tuning_objective_metric</th>\n",
       "      <th>alpha</th>\n",
       "      <th>colsample_bytree</th>\n",
       "      <th>eta</th>\n",
       "      <th>gamma</th>\n",
       "      <th>lambda</th>\n",
       "      <th>max_depth</th>\n",
       "      <th>min_child_weight</th>\n",
       "      <th>num_round</th>\n",
       "      <th>objective</th>\n",
       "      <th>save_model_on_termination</th>\n",
       "      <th>subsample</th>\n",
       "      <th>ObjectiveMetric - Min</th>\n",
       "      <th>ObjectiveMetric - Avg</th>\n",
       "      <th>ObjectiveMetric - StdDev</th>\n",
       "      <th>ObjectiveMetric - Last</th>\n",
       "      <th>ObjectiveMetric - Count</th>\n",
       "      <th>train:f1 - Min</th>\n",
       "      <th>train:f1 - Max</th>\n",
       "      <th>train:f1 - Avg</th>\n",
       "      <th>train:f1 - StdDev</th>\n",
       "      <th>train:f1 - Last</th>\n",
       "      <th>train:f1 - Count</th>\n",
       "      <th>validation:error - Min</th>\n",
       "      <th>validation:error - Max</th>\n",
       "      <th>validation:error - Avg</th>\n",
       "      <th>validation:error - StdDev</th>\n",
       "      <th>validation:error - Last</th>\n",
       "      <th>validation:error - Count</th>\n",
       "      <th>validation:f1 - Min</th>\n",
       "      <th>validation:f1 - Max</th>\n",
       "      <th>validation:f1 - Avg</th>\n",
       "      <th>validation:f1 - StdDev</th>\n",
       "      <th>validation:f1 - Last</th>\n",
       "      <th>validation:f1 - Count</th>\n",
       "      <th>train:error - Min</th>\n",
       "      <th>train:error - Max</th>\n",
       "      <th>train:error - Avg</th>\n",
       "      <th>train:error - StdDev</th>\n",
       "      <th>train:error - Last</th>\n",
       "      <th>train:error - Count</th>\n",
       "      <th>train - MediaType</th>\n",
       "      <th>train - Value</th>\n",
       "      <th>validation - MediaType</th>\n",
       "      <th>validation - Value</th>\n",
       "      <th>SageMaker.ModelArtifact - MediaType</th>\n",
       "      <th>SageMaker.ModelArtifact - Value</th>\n",
       "      <th>Trials</th>\n",
       "      <th>Experiments</th>\n",
       "      <th>SageMaker.ModelName</th>\n",
       "      <th>SageMaker.ModelPrimary.DataUrl</th>\n",
       "      <th>SageMaker.ModelPrimary.Image</th>\n",
       "      <th>SageMaker.TransformInput - MediaType</th>\n",
       "      <th>SageMaker.TransformInput - Value</th>\n",
       "      <th>SageMaker.TransformOutput - MediaType</th>\n",
       "      <th>SageMaker.TransformOutput - Value</th>\n",
       "      <th>processor_module</th>\n",
       "      <th>sagemaker_program</th>\n",
       "      <th>sagemaker_submit_directory</th>\n",
       "      <th>code - MediaType</th>\n",
       "      <th>code - Value</th>\n",
       "      <th>input_channel_mode</th>\n",
       "      <th>job_name</th>\n",
       "      <th>label_col</th>\n",
       "      <th>max_dataset_size</th>\n",
       "      <th>SageMaker.ImageUri - MediaType</th>\n",
       "      <th>SageMaker.ImageUri - Value</th>\n",
       "      <th>ds - MediaType</th>\n",
       "      <th>ds - Value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.61345</td>\n",
       "      <td>tuning-job-1-9d22a30d195e4dbe94-003-b81b9ca3-a...</td>\n",
       "      <td>tuning-job-1-9d22a30d195e4dbe94-003-b81b9ca3-a...</td>\n",
       "      <td>arn:aws:sagemaker:us-east-1:690429675927:train...</td>\n",
       "      <td>683313688378.dkr.ecr.us-east-1.amazonaws.com/s...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>ml.m5.4xlarge</td>\n",
       "      <td>50.0</td>\n",
       "      <td>validation:f1</td>\n",
       "      <td>0.00337</td>\n",
       "      <td>0.499484</td>\n",
       "      <td>0.00211</td>\n",
       "      <td>0.001007</td>\n",
       "      <td>0.000326</td>\n",
       "      <td>17.0</td>\n",
       "      <td>0.003029</td>\n",
       "      <td>2.0</td>\n",
       "      <td>binary:logistic</td>\n",
       "      <td>true</td>\n",
       "      <td>0.595006</td>\n",
       "      <td>0.61237</td>\n",
       "      <td>0.61291</td>\n",
       "      <td>0.000764</td>\n",
       "      <td>0.61237</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.85107</td>\n",
       "      <td>0.85176</td>\n",
       "      <td>0.851415</td>\n",
       "      <td>0.000488</td>\n",
       "      <td>0.85107</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.29192</td>\n",
       "      <td>0.29814</td>\n",
       "      <td>0.29503</td>\n",
       "      <td>0.004398</td>\n",
       "      <td>0.29192</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.61237</td>\n",
       "      <td>0.61345</td>\n",
       "      <td>0.61291</td>\n",
       "      <td>0.000764</td>\n",
       "      <td>0.61237</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.11111</td>\n",
       "      <td>0.11268</td>\n",
       "      <td>0.111895</td>\n",
       "      <td>0.00111</td>\n",
       "      <td>0.11111</td>\n",
       "      <td>2.0</td>\n",
       "      <td>text/csv</td>\n",
       "      <td>s3://sagemaker-us-east-1-690429675927/sagemake...</td>\n",
       "      <td>text/csv</td>\n",
       "      <td>s3://sagemaker-us-east-1-690429675927/sagemake...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>s3://sagemaker-us-east-1-690429675927/sagemake...</td>\n",
       "      <td>[tuning-job-1-9d22a30d195e4dbe94-003-b81b9ca3-...</td>\n",
       "      <td>[automl-telco-2020-10-01-02-31-36-aws-auto-ml-...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.61345</td>\n",
       "      <td>tuning-job-1-9d22a30d195e4dbe94-002-8df5e337-a...</td>\n",
       "      <td>tuning-job-1-9d22a30d195e4dbe94-002-8df5e337-a...</td>\n",
       "      <td>arn:aws:sagemaker:us-east-1:690429675927:train...</td>\n",
       "      <td>683313688378.dkr.ecr.us-east-1.amazonaws.com/s...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>ml.m5.4xlarge</td>\n",
       "      <td>50.0</td>\n",
       "      <td>validation:f1</td>\n",
       "      <td>0.00337</td>\n",
       "      <td>0.499484</td>\n",
       "      <td>0.00211</td>\n",
       "      <td>0.001007</td>\n",
       "      <td>0.000326</td>\n",
       "      <td>17.0</td>\n",
       "      <td>0.003029</td>\n",
       "      <td>2.0</td>\n",
       "      <td>binary:logistic</td>\n",
       "      <td>true</td>\n",
       "      <td>0.595006</td>\n",
       "      <td>0.61237</td>\n",
       "      <td>0.61291</td>\n",
       "      <td>0.000764</td>\n",
       "      <td>0.61237</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.85107</td>\n",
       "      <td>0.85176</td>\n",
       "      <td>0.851415</td>\n",
       "      <td>0.000488</td>\n",
       "      <td>0.85107</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.29192</td>\n",
       "      <td>0.29814</td>\n",
       "      <td>0.29503</td>\n",
       "      <td>0.004398</td>\n",
       "      <td>0.29192</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.61237</td>\n",
       "      <td>0.61345</td>\n",
       "      <td>0.61291</td>\n",
       "      <td>0.000764</td>\n",
       "      <td>0.61237</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.11111</td>\n",
       "      <td>0.11268</td>\n",
       "      <td>0.111895</td>\n",
       "      <td>0.00111</td>\n",
       "      <td>0.11111</td>\n",
       "      <td>2.0</td>\n",
       "      <td>text/csv</td>\n",
       "      <td>s3://sagemaker-us-east-1-690429675927/sagemake...</td>\n",
       "      <td>text/csv</td>\n",
       "      <td>s3://sagemaker-us-east-1-690429675927/sagemake...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>s3://sagemaker-us-east-1-690429675927/sagemake...</td>\n",
       "      <td>[tuning-job-1-9d22a30d195e4dbe94-002-8df5e337-...</td>\n",
       "      <td>[automl-telco-2020-10-01-02-31-36-aws-auto-ml-...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.61345</td>\n",
       "      <td>tuning-job-1-9d22a30d195e4dbe94-001-ac0354f9-a...</td>\n",
       "      <td>tuning-job-1-9d22a30d195e4dbe94-001-ac0354f9-a...</td>\n",
       "      <td>arn:aws:sagemaker:us-east-1:690429675927:train...</td>\n",
       "      <td>683313688378.dkr.ecr.us-east-1.amazonaws.com/s...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>ml.m5.4xlarge</td>\n",
       "      <td>50.0</td>\n",
       "      <td>validation:f1</td>\n",
       "      <td>0.00337</td>\n",
       "      <td>0.499484</td>\n",
       "      <td>0.00211</td>\n",
       "      <td>0.001007</td>\n",
       "      <td>0.000326</td>\n",
       "      <td>17.0</td>\n",
       "      <td>0.003029</td>\n",
       "      <td>2.0</td>\n",
       "      <td>binary:logistic</td>\n",
       "      <td>true</td>\n",
       "      <td>0.595006</td>\n",
       "      <td>0.61237</td>\n",
       "      <td>0.61291</td>\n",
       "      <td>0.000764</td>\n",
       "      <td>0.61237</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.85107</td>\n",
       "      <td>0.85176</td>\n",
       "      <td>0.851415</td>\n",
       "      <td>0.000488</td>\n",
       "      <td>0.85107</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.29192</td>\n",
       "      <td>0.29814</td>\n",
       "      <td>0.29503</td>\n",
       "      <td>0.004398</td>\n",
       "      <td>0.29192</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.61237</td>\n",
       "      <td>0.61345</td>\n",
       "      <td>0.61291</td>\n",
       "      <td>0.000764</td>\n",
       "      <td>0.61237</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.11111</td>\n",
       "      <td>0.11268</td>\n",
       "      <td>0.111895</td>\n",
       "      <td>0.00111</td>\n",
       "      <td>0.11111</td>\n",
       "      <td>2.0</td>\n",
       "      <td>text/csv</td>\n",
       "      <td>s3://sagemaker-us-east-1-690429675927/sagemake...</td>\n",
       "      <td>text/csv</td>\n",
       "      <td>s3://sagemaker-us-east-1-690429675927/sagemake...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>s3://sagemaker-us-east-1-690429675927/sagemake...</td>\n",
       "      <td>[tuning-job-1-9d22a30d195e4dbe94-001-ac0354f9-...</td>\n",
       "      <td>[automl-telco-2020-10-01-02-31-36-aws-auto-ml-...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>NaN</td>\n",
       "      <td>automl-tel-dpp0-csv-1-fe6cfd5308124947a17ce942...</td>\n",
       "      <td>automl-tel-dpp0-csv-1-fe6cfd5308124947a17ce942...</td>\n",
       "      <td>arn:aws:sagemaker:us-east-1:690429675927:trans...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>ml.m5.4xlarge</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[tuning-job-1-9d22a30d195e4dbe94-001-ac0354f9-...</td>\n",
       "      <td>[automl-telco-2020-10-01-02-31-36-aws-auto-ml-...</td>\n",
       "      <td>automl-telco-2020-10-01-02-31-36-automl-tel-dp...</td>\n",
       "      <td>s3://sagemaker-us-east-1-690429675927/sagemake...</td>\n",
       "      <td>683313688378.dkr.ecr.us-east-1.amazonaws.com/s...</td>\n",
       "      <td>text/csv</td>\n",
       "      <td>s3://sagemaker-us-east-1-690429675927/sagemake...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>s3://sagemaker-us-east-1-690429675927/sagemake...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>NaN</td>\n",
       "      <td>automl-tel-dpp2-csv-1-6c6570ba91ea4eee926c3a7a...</td>\n",
       "      <td>automl-tel-dpp2-csv-1-6c6570ba91ea4eee926c3a7a...</td>\n",
       "      <td>arn:aws:sagemaker:us-east-1:690429675927:trans...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>ml.m5.4xlarge</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[tuning-job-1-9d22a30d195e4dbe94-003-b81b9ca3-...</td>\n",
       "      <td>[automl-telco-2020-10-01-02-31-36-aws-auto-ml-...</td>\n",
       "      <td>automl-telco-2020-10-01-02-31-36-automl-tel-dp...</td>\n",
       "      <td>s3://sagemaker-us-east-1-690429675927/sagemake...</td>\n",
       "      <td>683313688378.dkr.ecr.us-east-1.amazonaws.com/s...</td>\n",
       "      <td>text/csv</td>\n",
       "      <td>s3://sagemaker-us-east-1-690429675927/sagemake...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>s3://sagemaker-us-east-1-690429675927/sagemake...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>NaN</td>\n",
       "      <td>automl-tel-dpp1-csv-1-9e5285159d0a415aa77ebcc4...</td>\n",
       "      <td>automl-tel-dpp1-csv-1-9e5285159d0a415aa77ebcc4...</td>\n",
       "      <td>arn:aws:sagemaker:us-east-1:690429675927:trans...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>ml.m5.4xlarge</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[tuning-job-1-9d22a30d195e4dbe94-002-8df5e337-...</td>\n",
       "      <td>[automl-telco-2020-10-01-02-31-36-aws-auto-ml-...</td>\n",
       "      <td>automl-telco-2020-10-01-02-31-36-automl-tel-dp...</td>\n",
       "      <td>s3://sagemaker-us-east-1-690429675927/sagemake...</td>\n",
       "      <td>683313688378.dkr.ecr.us-east-1.amazonaws.com/s...</td>\n",
       "      <td>text/csv</td>\n",
       "      <td>s3://sagemaker-us-east-1-690429675927/sagemake...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>s3://sagemaker-us-east-1-690429675927/sagemake...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>NaN</td>\n",
       "      <td>automl-tel-dpp2-1-13a57a7c234745f694b10623c311...</td>\n",
       "      <td>automl-tel-dpp2-1-13a57a7c234745f694b10623c311...</td>\n",
       "      <td>arn:aws:sagemaker:us-east-1:690429675927:train...</td>\n",
       "      <td>683313688378.dkr.ecr.us-east-1.amazonaws.com/s...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>ml.m5.4xlarge</td>\n",
       "      <td>50.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>text/csv</td>\n",
       "      <td>s3://sagemaker-us-east-1-690429675927/sagemake...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>s3://sagemaker-us-east-1-690429675927/sagemake...</td>\n",
       "      <td>[tuning-job-1-9d22a30d195e4dbe94-003-b81b9ca3-...</td>\n",
       "      <td>[automl-telco-2020-10-01-02-31-36-aws-auto-ml-...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>candidate_data_processors.dpp2</td>\n",
       "      <td>candidate_data_processors.trainer</td>\n",
       "      <td>/opt/ml/input/data/code</td>\n",
       "      <td>application/x-code</td>\n",
       "      <td>s3://sagemaker-us-east-1-690429675927/sagemake...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>NaN</td>\n",
       "      <td>automl-tel-dpp0-1-2b99df02f32b43b98a2f587e24c9...</td>\n",
       "      <td>automl-tel-dpp0-1-2b99df02f32b43b98a2f587e24c9...</td>\n",
       "      <td>arn:aws:sagemaker:us-east-1:690429675927:train...</td>\n",
       "      <td>683313688378.dkr.ecr.us-east-1.amazonaws.com/s...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>ml.m5.4xlarge</td>\n",
       "      <td>50.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>text/csv</td>\n",
       "      <td>s3://sagemaker-us-east-1-690429675927/sagemake...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>s3://sagemaker-us-east-1-690429675927/sagemake...</td>\n",
       "      <td>[tuning-job-1-9d22a30d195e4dbe94-001-ac0354f9-...</td>\n",
       "      <td>[automl-telco-2020-10-01-02-31-36-aws-auto-ml-...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>candidate_data_processors.dpp0</td>\n",
       "      <td>candidate_data_processors.trainer</td>\n",
       "      <td>/opt/ml/input/data/code</td>\n",
       "      <td>application/x-code</td>\n",
       "      <td>s3://sagemaker-us-east-1-690429675927/sagemake...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>NaN</td>\n",
       "      <td>automl-tel-dpp1-1-41b74fe32eb84caa8074064357d2...</td>\n",
       "      <td>automl-tel-dpp1-1-41b74fe32eb84caa8074064357d2...</td>\n",
       "      <td>arn:aws:sagemaker:us-east-1:690429675927:train...</td>\n",
       "      <td>683313688378.dkr.ecr.us-east-1.amazonaws.com/s...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>ml.m5.4xlarge</td>\n",
       "      <td>50.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>text/csv</td>\n",
       "      <td>s3://sagemaker-us-east-1-690429675927/sagemake...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>s3://sagemaker-us-east-1-690429675927/sagemake...</td>\n",
       "      <td>[tuning-job-1-9d22a30d195e4dbe94-002-8df5e337-...</td>\n",
       "      <td>[automl-telco-2020-10-01-02-31-36-aws-auto-ml-...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>candidate_data_processors.dpp1</td>\n",
       "      <td>candidate_data_processors.trainer</td>\n",
       "      <td>/opt/ml/input/data/code</td>\n",
       "      <td>application/x-code</td>\n",
       "      <td>s3://sagemaker-us-east-1-690429675927/sagemake...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>NaN</td>\n",
       "      <td>db-1-d6bfe956362c4aa59cbb59dff3dc9f14d1d056c55...</td>\n",
       "      <td>db-1-d6bfe956362c4aa59cbb59dff3dc9f14d1d056c55...</td>\n",
       "      <td>arn:aws:sagemaker:us-east-1:690429675927:proce...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>ml.m5.2xlarge</td>\n",
       "      <td>250.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>s3://sagemaker-us-east-1-690429675927/sagemake...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[tuning-job-1-9d22a30d195e4dbe94-003-b81b9ca3-...</td>\n",
       "      <td>[automl-telco-2020-10-01-02-31-36-aws-auto-ml-...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Pipe</td>\n",
       "      <td>automl-telco-2020-10-01-02-31-36</td>\n",
       "      <td>Churn</td>\n",
       "      <td>5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>120479346908.dkr.ecr.us-east-1.amazonaws.com/d...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>s3://sagemaker-us-east-1-690429675927/sagemake...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   ObjectiveMetric - Max                                 TrialComponentName  \\\n",
       "0                0.61345  tuning-job-1-9d22a30d195e4dbe94-003-b81b9ca3-a...   \n",
       "1                0.61345  tuning-job-1-9d22a30d195e4dbe94-002-8df5e337-a...   \n",
       "2                0.61345  tuning-job-1-9d22a30d195e4dbe94-001-ac0354f9-a...   \n",
       "3                    NaN  automl-tel-dpp0-csv-1-fe6cfd5308124947a17ce942...   \n",
       "4                    NaN  automl-tel-dpp2-csv-1-6c6570ba91ea4eee926c3a7a...   \n",
       "5                    NaN  automl-tel-dpp1-csv-1-9e5285159d0a415aa77ebcc4...   \n",
       "6                    NaN  automl-tel-dpp2-1-13a57a7c234745f694b10623c311...   \n",
       "7                    NaN  automl-tel-dpp0-1-2b99df02f32b43b98a2f587e24c9...   \n",
       "8                    NaN  automl-tel-dpp1-1-41b74fe32eb84caa8074064357d2...   \n",
       "9                    NaN  db-1-d6bfe956362c4aa59cbb59dff3dc9f14d1d056c55...   \n",
       "\n",
       "                                         DisplayName  \\\n",
       "0  tuning-job-1-9d22a30d195e4dbe94-003-b81b9ca3-a...   \n",
       "1  tuning-job-1-9d22a30d195e4dbe94-002-8df5e337-a...   \n",
       "2  tuning-job-1-9d22a30d195e4dbe94-001-ac0354f9-a...   \n",
       "3  automl-tel-dpp0-csv-1-fe6cfd5308124947a17ce942...   \n",
       "4  automl-tel-dpp2-csv-1-6c6570ba91ea4eee926c3a7a...   \n",
       "5  automl-tel-dpp1-csv-1-9e5285159d0a415aa77ebcc4...   \n",
       "6  automl-tel-dpp2-1-13a57a7c234745f694b10623c311...   \n",
       "7  automl-tel-dpp0-1-2b99df02f32b43b98a2f587e24c9...   \n",
       "8  automl-tel-dpp1-1-41b74fe32eb84caa8074064357d2...   \n",
       "9  db-1-d6bfe956362c4aa59cbb59dff3dc9f14d1d056c55...   \n",
       "\n",
       "                                           SourceArn  \\\n",
       "0  arn:aws:sagemaker:us-east-1:690429675927:train...   \n",
       "1  arn:aws:sagemaker:us-east-1:690429675927:train...   \n",
       "2  arn:aws:sagemaker:us-east-1:690429675927:train...   \n",
       "3  arn:aws:sagemaker:us-east-1:690429675927:trans...   \n",
       "4  arn:aws:sagemaker:us-east-1:690429675927:trans...   \n",
       "5  arn:aws:sagemaker:us-east-1:690429675927:trans...   \n",
       "6  arn:aws:sagemaker:us-east-1:690429675927:train...   \n",
       "7  arn:aws:sagemaker:us-east-1:690429675927:train...   \n",
       "8  arn:aws:sagemaker:us-east-1:690429675927:train...   \n",
       "9  arn:aws:sagemaker:us-east-1:690429675927:proce...   \n",
       "\n",
       "                                  SageMaker.ImageUri  SageMaker.InstanceCount  \\\n",
       "0  683313688378.dkr.ecr.us-east-1.amazonaws.com/s...                      1.0   \n",
       "1  683313688378.dkr.ecr.us-east-1.amazonaws.com/s...                      1.0   \n",
       "2  683313688378.dkr.ecr.us-east-1.amazonaws.com/s...                      1.0   \n",
       "3                                                NaN                      1.0   \n",
       "4                                                NaN                      1.0   \n",
       "5                                                NaN                      1.0   \n",
       "6  683313688378.dkr.ecr.us-east-1.amazonaws.com/s...                      1.0   \n",
       "7  683313688378.dkr.ecr.us-east-1.amazonaws.com/s...                      1.0   \n",
       "8  683313688378.dkr.ecr.us-east-1.amazonaws.com/s...                      1.0   \n",
       "9                                                NaN                      1.0   \n",
       "\n",
       "  SageMaker.InstanceType  SageMaker.VolumeSizeInGB _tuning_objective_metric  \\\n",
       "0          ml.m5.4xlarge                      50.0            validation:f1   \n",
       "1          ml.m5.4xlarge                      50.0            validation:f1   \n",
       "2          ml.m5.4xlarge                      50.0            validation:f1   \n",
       "3          ml.m5.4xlarge                       NaN                      NaN   \n",
       "4          ml.m5.4xlarge                       NaN                      NaN   \n",
       "5          ml.m5.4xlarge                       NaN                      NaN   \n",
       "6          ml.m5.4xlarge                      50.0                      NaN   \n",
       "7          ml.m5.4xlarge                      50.0                      NaN   \n",
       "8          ml.m5.4xlarge                      50.0                      NaN   \n",
       "9          ml.m5.2xlarge                     250.0                      NaN   \n",
       "\n",
       "     alpha  colsample_bytree      eta     gamma    lambda  max_depth  \\\n",
       "0  0.00337          0.499484  0.00211  0.001007  0.000326       17.0   \n",
       "1  0.00337          0.499484  0.00211  0.001007  0.000326       17.0   \n",
       "2  0.00337          0.499484  0.00211  0.001007  0.000326       17.0   \n",
       "3      NaN               NaN      NaN       NaN       NaN        NaN   \n",
       "4      NaN               NaN      NaN       NaN       NaN        NaN   \n",
       "5      NaN               NaN      NaN       NaN       NaN        NaN   \n",
       "6      NaN               NaN      NaN       NaN       NaN        NaN   \n",
       "7      NaN               NaN      NaN       NaN       NaN        NaN   \n",
       "8      NaN               NaN      NaN       NaN       NaN        NaN   \n",
       "9      NaN               NaN      NaN       NaN       NaN        NaN   \n",
       "\n",
       "   min_child_weight  num_round        objective save_model_on_termination  \\\n",
       "0          0.003029        2.0  binary:logistic                      true   \n",
       "1          0.003029        2.0  binary:logistic                      true   \n",
       "2          0.003029        2.0  binary:logistic                      true   \n",
       "3               NaN        NaN              NaN                       NaN   \n",
       "4               NaN        NaN              NaN                       NaN   \n",
       "5               NaN        NaN              NaN                       NaN   \n",
       "6               NaN        NaN              NaN                       NaN   \n",
       "7               NaN        NaN              NaN                       NaN   \n",
       "8               NaN        NaN              NaN                       NaN   \n",
       "9               NaN        NaN              NaN                       NaN   \n",
       "\n",
       "   subsample  ObjectiveMetric - Min  ObjectiveMetric - Avg  \\\n",
       "0   0.595006                0.61237                0.61291   \n",
       "1   0.595006                0.61237                0.61291   \n",
       "2   0.595006                0.61237                0.61291   \n",
       "3        NaN                    NaN                    NaN   \n",
       "4        NaN                    NaN                    NaN   \n",
       "5        NaN                    NaN                    NaN   \n",
       "6        NaN                    NaN                    NaN   \n",
       "7        NaN                    NaN                    NaN   \n",
       "8        NaN                    NaN                    NaN   \n",
       "9        NaN                    NaN                    NaN   \n",
       "\n",
       "   ObjectiveMetric - StdDev  ObjectiveMetric - Last  ObjectiveMetric - Count  \\\n",
       "0                  0.000764                 0.61237                      2.0   \n",
       "1                  0.000764                 0.61237                      2.0   \n",
       "2                  0.000764                 0.61237                      2.0   \n",
       "3                       NaN                     NaN                      NaN   \n",
       "4                       NaN                     NaN                      NaN   \n",
       "5                       NaN                     NaN                      NaN   \n",
       "6                       NaN                     NaN                      NaN   \n",
       "7                       NaN                     NaN                      NaN   \n",
       "8                       NaN                     NaN                      NaN   \n",
       "9                       NaN                     NaN                      NaN   \n",
       "\n",
       "   train:f1 - Min  train:f1 - Max  train:f1 - Avg  train:f1 - StdDev  \\\n",
       "0         0.85107         0.85176        0.851415           0.000488   \n",
       "1         0.85107         0.85176        0.851415           0.000488   \n",
       "2         0.85107         0.85176        0.851415           0.000488   \n",
       "3             NaN             NaN             NaN                NaN   \n",
       "4             NaN             NaN             NaN                NaN   \n",
       "5             NaN             NaN             NaN                NaN   \n",
       "6             NaN             NaN             NaN                NaN   \n",
       "7             NaN             NaN             NaN                NaN   \n",
       "8             NaN             NaN             NaN                NaN   \n",
       "9             NaN             NaN             NaN                NaN   \n",
       "\n",
       "   train:f1 - Last  train:f1 - Count  validation:error - Min  \\\n",
       "0          0.85107               2.0                 0.29192   \n",
       "1          0.85107               2.0                 0.29192   \n",
       "2          0.85107               2.0                 0.29192   \n",
       "3              NaN               NaN                     NaN   \n",
       "4              NaN               NaN                     NaN   \n",
       "5              NaN               NaN                     NaN   \n",
       "6              NaN               NaN                     NaN   \n",
       "7              NaN               NaN                     NaN   \n",
       "8              NaN               NaN                     NaN   \n",
       "9              NaN               NaN                     NaN   \n",
       "\n",
       "   validation:error - Max  validation:error - Avg  validation:error - StdDev  \\\n",
       "0                 0.29814                 0.29503                   0.004398   \n",
       "1                 0.29814                 0.29503                   0.004398   \n",
       "2                 0.29814                 0.29503                   0.004398   \n",
       "3                     NaN                     NaN                        NaN   \n",
       "4                     NaN                     NaN                        NaN   \n",
       "5                     NaN                     NaN                        NaN   \n",
       "6                     NaN                     NaN                        NaN   \n",
       "7                     NaN                     NaN                        NaN   \n",
       "8                     NaN                     NaN                        NaN   \n",
       "9                     NaN                     NaN                        NaN   \n",
       "\n",
       "   validation:error - Last  validation:error - Count  validation:f1 - Min  \\\n",
       "0                  0.29192                       2.0              0.61237   \n",
       "1                  0.29192                       2.0              0.61237   \n",
       "2                  0.29192                       2.0              0.61237   \n",
       "3                      NaN                       NaN                  NaN   \n",
       "4                      NaN                       NaN                  NaN   \n",
       "5                      NaN                       NaN                  NaN   \n",
       "6                      NaN                       NaN                  NaN   \n",
       "7                      NaN                       NaN                  NaN   \n",
       "8                      NaN                       NaN                  NaN   \n",
       "9                      NaN                       NaN                  NaN   \n",
       "\n",
       "   validation:f1 - Max  validation:f1 - Avg  validation:f1 - StdDev  \\\n",
       "0              0.61345              0.61291                0.000764   \n",
       "1              0.61345              0.61291                0.000764   \n",
       "2              0.61345              0.61291                0.000764   \n",
       "3                  NaN                  NaN                     NaN   \n",
       "4                  NaN                  NaN                     NaN   \n",
       "5                  NaN                  NaN                     NaN   \n",
       "6                  NaN                  NaN                     NaN   \n",
       "7                  NaN                  NaN                     NaN   \n",
       "8                  NaN                  NaN                     NaN   \n",
       "9                  NaN                  NaN                     NaN   \n",
       "\n",
       "   validation:f1 - Last  validation:f1 - Count  train:error - Min  \\\n",
       "0               0.61237                    2.0            0.11111   \n",
       "1               0.61237                    2.0            0.11111   \n",
       "2               0.61237                    2.0            0.11111   \n",
       "3                   NaN                    NaN                NaN   \n",
       "4                   NaN                    NaN                NaN   \n",
       "5                   NaN                    NaN                NaN   \n",
       "6                   NaN                    NaN                NaN   \n",
       "7                   NaN                    NaN                NaN   \n",
       "8                   NaN                    NaN                NaN   \n",
       "9                   NaN                    NaN                NaN   \n",
       "\n",
       "   train:error - Max  train:error - Avg  train:error - StdDev  \\\n",
       "0            0.11268           0.111895               0.00111   \n",
       "1            0.11268           0.111895               0.00111   \n",
       "2            0.11268           0.111895               0.00111   \n",
       "3                NaN                NaN                   NaN   \n",
       "4                NaN                NaN                   NaN   \n",
       "5                NaN                NaN                   NaN   \n",
       "6                NaN                NaN                   NaN   \n",
       "7                NaN                NaN                   NaN   \n",
       "8                NaN                NaN                   NaN   \n",
       "9                NaN                NaN                   NaN   \n",
       "\n",
       "   train:error - Last  train:error - Count train - MediaType  \\\n",
       "0             0.11111                  2.0          text/csv   \n",
       "1             0.11111                  2.0          text/csv   \n",
       "2             0.11111                  2.0          text/csv   \n",
       "3                 NaN                  NaN               NaN   \n",
       "4                 NaN                  NaN               NaN   \n",
       "5                 NaN                  NaN               NaN   \n",
       "6                 NaN                  NaN          text/csv   \n",
       "7                 NaN                  NaN          text/csv   \n",
       "8                 NaN                  NaN          text/csv   \n",
       "9                 NaN                  NaN              None   \n",
       "\n",
       "                                       train - Value validation - MediaType  \\\n",
       "0  s3://sagemaker-us-east-1-690429675927/sagemake...               text/csv   \n",
       "1  s3://sagemaker-us-east-1-690429675927/sagemake...               text/csv   \n",
       "2  s3://sagemaker-us-east-1-690429675927/sagemake...               text/csv   \n",
       "3                                                NaN                    NaN   \n",
       "4                                                NaN                    NaN   \n",
       "5                                                NaN                    NaN   \n",
       "6  s3://sagemaker-us-east-1-690429675927/sagemake...                    NaN   \n",
       "7  s3://sagemaker-us-east-1-690429675927/sagemake...                    NaN   \n",
       "8  s3://sagemaker-us-east-1-690429675927/sagemake...                    NaN   \n",
       "9  s3://sagemaker-us-east-1-690429675927/sagemake...                    NaN   \n",
       "\n",
       "                                  validation - Value  \\\n",
       "0  s3://sagemaker-us-east-1-690429675927/sagemake...   \n",
       "1  s3://sagemaker-us-east-1-690429675927/sagemake...   \n",
       "2  s3://sagemaker-us-east-1-690429675927/sagemake...   \n",
       "3                                                NaN   \n",
       "4                                                NaN   \n",
       "5                                                NaN   \n",
       "6                                                NaN   \n",
       "7                                                NaN   \n",
       "8                                                NaN   \n",
       "9                                                NaN   \n",
       "\n",
       "   SageMaker.ModelArtifact - MediaType  \\\n",
       "0                                  NaN   \n",
       "1                                  NaN   \n",
       "2                                  NaN   \n",
       "3                                  NaN   \n",
       "4                                  NaN   \n",
       "5                                  NaN   \n",
       "6                                  NaN   \n",
       "7                                  NaN   \n",
       "8                                  NaN   \n",
       "9                                  NaN   \n",
       "\n",
       "                     SageMaker.ModelArtifact - Value  \\\n",
       "0  s3://sagemaker-us-east-1-690429675927/sagemake...   \n",
       "1  s3://sagemaker-us-east-1-690429675927/sagemake...   \n",
       "2  s3://sagemaker-us-east-1-690429675927/sagemake...   \n",
       "3                                                NaN   \n",
       "4                                                NaN   \n",
       "5                                                NaN   \n",
       "6  s3://sagemaker-us-east-1-690429675927/sagemake...   \n",
       "7  s3://sagemaker-us-east-1-690429675927/sagemake...   \n",
       "8  s3://sagemaker-us-east-1-690429675927/sagemake...   \n",
       "9                                                NaN   \n",
       "\n",
       "                                              Trials  \\\n",
       "0  [tuning-job-1-9d22a30d195e4dbe94-003-b81b9ca3-...   \n",
       "1  [tuning-job-1-9d22a30d195e4dbe94-002-8df5e337-...   \n",
       "2  [tuning-job-1-9d22a30d195e4dbe94-001-ac0354f9-...   \n",
       "3  [tuning-job-1-9d22a30d195e4dbe94-001-ac0354f9-...   \n",
       "4  [tuning-job-1-9d22a30d195e4dbe94-003-b81b9ca3-...   \n",
       "5  [tuning-job-1-9d22a30d195e4dbe94-002-8df5e337-...   \n",
       "6  [tuning-job-1-9d22a30d195e4dbe94-003-b81b9ca3-...   \n",
       "7  [tuning-job-1-9d22a30d195e4dbe94-001-ac0354f9-...   \n",
       "8  [tuning-job-1-9d22a30d195e4dbe94-002-8df5e337-...   \n",
       "9  [tuning-job-1-9d22a30d195e4dbe94-003-b81b9ca3-...   \n",
       "\n",
       "                                         Experiments  \\\n",
       "0  [automl-telco-2020-10-01-02-31-36-aws-auto-ml-...   \n",
       "1  [automl-telco-2020-10-01-02-31-36-aws-auto-ml-...   \n",
       "2  [automl-telco-2020-10-01-02-31-36-aws-auto-ml-...   \n",
       "3  [automl-telco-2020-10-01-02-31-36-aws-auto-ml-...   \n",
       "4  [automl-telco-2020-10-01-02-31-36-aws-auto-ml-...   \n",
       "5  [automl-telco-2020-10-01-02-31-36-aws-auto-ml-...   \n",
       "6  [automl-telco-2020-10-01-02-31-36-aws-auto-ml-...   \n",
       "7  [automl-telco-2020-10-01-02-31-36-aws-auto-ml-...   \n",
       "8  [automl-telco-2020-10-01-02-31-36-aws-auto-ml-...   \n",
       "9  [automl-telco-2020-10-01-02-31-36-aws-auto-ml-...   \n",
       "\n",
       "                                 SageMaker.ModelName  \\\n",
       "0                                                NaN   \n",
       "1                                                NaN   \n",
       "2                                                NaN   \n",
       "3  automl-telco-2020-10-01-02-31-36-automl-tel-dp...   \n",
       "4  automl-telco-2020-10-01-02-31-36-automl-tel-dp...   \n",
       "5  automl-telco-2020-10-01-02-31-36-automl-tel-dp...   \n",
       "6                                                NaN   \n",
       "7                                                NaN   \n",
       "8                                                NaN   \n",
       "9                                                NaN   \n",
       "\n",
       "                      SageMaker.ModelPrimary.DataUrl  \\\n",
       "0                                                NaN   \n",
       "1                                                NaN   \n",
       "2                                                NaN   \n",
       "3  s3://sagemaker-us-east-1-690429675927/sagemake...   \n",
       "4  s3://sagemaker-us-east-1-690429675927/sagemake...   \n",
       "5  s3://sagemaker-us-east-1-690429675927/sagemake...   \n",
       "6                                                NaN   \n",
       "7                                                NaN   \n",
       "8                                                NaN   \n",
       "9                                                NaN   \n",
       "\n",
       "                        SageMaker.ModelPrimary.Image  \\\n",
       "0                                                NaN   \n",
       "1                                                NaN   \n",
       "2                                                NaN   \n",
       "3  683313688378.dkr.ecr.us-east-1.amazonaws.com/s...   \n",
       "4  683313688378.dkr.ecr.us-east-1.amazonaws.com/s...   \n",
       "5  683313688378.dkr.ecr.us-east-1.amazonaws.com/s...   \n",
       "6                                                NaN   \n",
       "7                                                NaN   \n",
       "8                                                NaN   \n",
       "9                                                NaN   \n",
       "\n",
       "  SageMaker.TransformInput - MediaType  \\\n",
       "0                                  NaN   \n",
       "1                                  NaN   \n",
       "2                                  NaN   \n",
       "3                             text/csv   \n",
       "4                             text/csv   \n",
       "5                             text/csv   \n",
       "6                                  NaN   \n",
       "7                                  NaN   \n",
       "8                                  NaN   \n",
       "9                                  NaN   \n",
       "\n",
       "                    SageMaker.TransformInput - Value  \\\n",
       "0                                                NaN   \n",
       "1                                                NaN   \n",
       "2                                                NaN   \n",
       "3  s3://sagemaker-us-east-1-690429675927/sagemake...   \n",
       "4  s3://sagemaker-us-east-1-690429675927/sagemake...   \n",
       "5  s3://sagemaker-us-east-1-690429675927/sagemake...   \n",
       "6                                                NaN   \n",
       "7                                                NaN   \n",
       "8                                                NaN   \n",
       "9                                                NaN   \n",
       "\n",
       "   SageMaker.TransformOutput - MediaType  \\\n",
       "0                                    NaN   \n",
       "1                                    NaN   \n",
       "2                                    NaN   \n",
       "3                                    NaN   \n",
       "4                                    NaN   \n",
       "5                                    NaN   \n",
       "6                                    NaN   \n",
       "7                                    NaN   \n",
       "8                                    NaN   \n",
       "9                                    NaN   \n",
       "\n",
       "                   SageMaker.TransformOutput - Value  \\\n",
       "0                                                NaN   \n",
       "1                                                NaN   \n",
       "2                                                NaN   \n",
       "3  s3://sagemaker-us-east-1-690429675927/sagemake...   \n",
       "4  s3://sagemaker-us-east-1-690429675927/sagemake...   \n",
       "5  s3://sagemaker-us-east-1-690429675927/sagemake...   \n",
       "6                                                NaN   \n",
       "7                                                NaN   \n",
       "8                                                NaN   \n",
       "9                                                NaN   \n",
       "\n",
       "                 processor_module                  sagemaker_program  \\\n",
       "0                             NaN                                NaN   \n",
       "1                             NaN                                NaN   \n",
       "2                             NaN                                NaN   \n",
       "3                             NaN                                NaN   \n",
       "4                             NaN                                NaN   \n",
       "5                             NaN                                NaN   \n",
       "6  candidate_data_processors.dpp2  candidate_data_processors.trainer   \n",
       "7  candidate_data_processors.dpp0  candidate_data_processors.trainer   \n",
       "8  candidate_data_processors.dpp1  candidate_data_processors.trainer   \n",
       "9                             NaN                                NaN   \n",
       "\n",
       "  sagemaker_submit_directory    code - MediaType  \\\n",
       "0                        NaN                 NaN   \n",
       "1                        NaN                 NaN   \n",
       "2                        NaN                 NaN   \n",
       "3                        NaN                 NaN   \n",
       "4                        NaN                 NaN   \n",
       "5                        NaN                 NaN   \n",
       "6    /opt/ml/input/data/code  application/x-code   \n",
       "7    /opt/ml/input/data/code  application/x-code   \n",
       "8    /opt/ml/input/data/code  application/x-code   \n",
       "9                        NaN                 NaN   \n",
       "\n",
       "                                        code - Value input_channel_mode  \\\n",
       "0                                                NaN                NaN   \n",
       "1                                                NaN                NaN   \n",
       "2                                                NaN                NaN   \n",
       "3                                                NaN                NaN   \n",
       "4                                                NaN                NaN   \n",
       "5                                                NaN                NaN   \n",
       "6  s3://sagemaker-us-east-1-690429675927/sagemake...                NaN   \n",
       "7  s3://sagemaker-us-east-1-690429675927/sagemake...                NaN   \n",
       "8  s3://sagemaker-us-east-1-690429675927/sagemake...                NaN   \n",
       "9                                                NaN               Pipe   \n",
       "\n",
       "                           job_name label_col max_dataset_size  \\\n",
       "0                               NaN       NaN              NaN   \n",
       "1                               NaN       NaN              NaN   \n",
       "2                               NaN       NaN              NaN   \n",
       "3                               NaN       NaN              NaN   \n",
       "4                               NaN       NaN              NaN   \n",
       "5                               NaN       NaN              NaN   \n",
       "6                               NaN       NaN              NaN   \n",
       "7                               NaN       NaN              NaN   \n",
       "8                               NaN       NaN              NaN   \n",
       "9  automl-telco-2020-10-01-02-31-36     Churn                5   \n",
       "\n",
       "   SageMaker.ImageUri - MediaType  \\\n",
       "0                             NaN   \n",
       "1                             NaN   \n",
       "2                             NaN   \n",
       "3                             NaN   \n",
       "4                             NaN   \n",
       "5                             NaN   \n",
       "6                             NaN   \n",
       "7                             NaN   \n",
       "8                             NaN   \n",
       "9                             NaN   \n",
       "\n",
       "                          SageMaker.ImageUri - Value  ds - MediaType  \\\n",
       "0                                                NaN             NaN   \n",
       "1                                                NaN             NaN   \n",
       "2                                                NaN             NaN   \n",
       "3                                                NaN             NaN   \n",
       "4                                                NaN             NaN   \n",
       "5                                                NaN             NaN   \n",
       "6                                                NaN             NaN   \n",
       "7                                                NaN             NaN   \n",
       "8                                                NaN             NaN   \n",
       "9  120479346908.dkr.ecr.us-east-1.amazonaws.com/d...             NaN   \n",
       "\n",
       "                                          ds - Value  \n",
       "0                                                NaN  \n",
       "1                                                NaN  \n",
       "2                                                NaN  \n",
       "3                                                NaN  \n",
       "4                                                NaN  \n",
       "5                                                NaN  \n",
       "6                                                NaN  \n",
       "7                                                NaN  \n",
       "8                                                NaN  \n",
       "9  s3://sagemaker-us-east-1-690429675927/sagemake...  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "exp = ExperimentAnalytics(sagemaker_session=session,experiment_name=auto_ml_job_name + '-aws-auto-ml-job')\n",
    "df = exp.dataframe()\n",
    "print(\"Number of jobs: \", len(df))\n",
    "\n",
    "df = pd.concat([df['ObjectiveMetric - Max'], df.drop(['ObjectiveMetric - Max'], axis=1)], axis=1)\n",
    "\n",
    "df.sort_values('ObjectiveMetric - Max', ascending=0).head(len(df))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's review the Best Candidate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Best Candidate Details</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Candidate Name</th>\n",
       "      <td>tuning-job-1-9d22a30d195e4dbe94-001-ac0354f9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Candidate Status</th>\n",
       "      <td>Completed</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Objective Metric Name</th>\n",
       "      <td>validation:f1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Objective Metric Value</th>\n",
       "      <td>0.6123700141906738</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Training Duration (Tuning)</th>\n",
       "      <td>0:00:46</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                  Best Candidate Details\n",
       "Candidate Name              tuning-job-1-9d22a30d195e4dbe94-001-ac0354f9\n",
       "Candidate Status                                               Completed\n",
       "Objective Metric Name                                      validation:f1\n",
       "Objective Metric Value                                0.6123700141906738\n",
       "Training Duration (Tuning)                                       0:00:46"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<a target=\"blank\" href=\"https://console.aws.amazon.com/sagemaker/home?region=us-east-1#/jobs/automl-tel-dpp0-1-2b99df02f32b43b98a2f587e24c9522e67111d08155a4\">Click for More Best Candidate Details</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<a target=\"blank\" href=\"https://console.aws.amazon.com/sagemaker/home?region=us-east-1#/jobs/tuning-job-1-9d22a30d195e4dbe94-001-ac0354f9\">Click for More Best Candidate (Tuning) Details</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "best_candidate = sm.describe_auto_ml_job(AutoMLJobName=auto_ml_job_name)['BestCandidate']\n",
    "\n",
    "best_candidate_dict = {\"Candidate Name\" : best_candidate['CandidateName'],\n",
    "     \"Candidate Status\" : best_candidate['CandidateStatus'],\n",
    "     \"Objective Metric Name\" : best_candidate['FinalAutoMLJobObjectiveMetric']['MetricName'],\n",
    "     \"Objective Metric Value\" : str(best_candidate['FinalAutoMLJobObjectiveMetric']['Value']),\n",
    "     \"Training Duration (Tuning)\" : str(best_candidate['EndTime'] - best_candidate['CreationTime'])\n",
    "    }\n",
    "\n",
    "display(pd.DataFrame.from_dict(best_candidate_dict, orient='index' , columns=['Best Candidate Details']))\n",
    "display()\n",
    "display(HTML('<a target=\"blank\" href=\"https://console.aws.amazon.com/sagemaker/home?region={}#/jobs/{}\">Click for More Best Candidate Details</a>'.format(region, best_candidate['CandidateSteps'][1]['CandidateStepName'] )))\n",
    "display(HTML('<a target=\"blank\" href=\"https://console.aws.amazon.com/sagemaker/home?region={}#/jobs/{}\">Click for More Best Candidate (Tuning) Details</a>'.format(region, best_candidate['CandidateSteps'][3]['CandidateStepName'] )))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reviewing Generated Notebook\n",
    "Once the data analysis is complete, the two auto-generated notebooks are available. \n",
    "* Data Exploration\n",
    "* Candidate Definition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[OK] DataExplorationNotebookLocation found.\n"
     ]
    }
   ],
   "source": [
    "job_description_response = sm.describe_auto_ml_job(AutoMLJobName=auto_ml_job_name)\n",
    "\n",
    "while 'DataExplorationNotebookLocation' not in job_description_response['AutoMLJobArtifacts'].keys():\n",
    "    job_description_response = sm.describe_auto_ml_job(AutoMLJobName=auto_ml_job_name)\n",
    "    print('[INFO] Autopilot Job has not yet generated the notebooks. Please wait. ')\n",
    "    print(json.dumps(job_description_response, indent=4, sort_keys=True, default=str))\n",
    "    print('[INFO] Waiting for DataExplorationNotebookLocation...')\n",
    "    sleep(15)\n",
    "\n",
    "print('[OK] DataExplorationNotebookLocation found.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "generated_resources = job_description_response['AutoMLJobArtifacts']['DataExplorationNotebookLocation'].rstrip('notebooks/SageMakerAutopilotDataExplorationNotebook.ipynb')\n",
    "\n",
    "pr_job_id = generated_resources.rsplit('/', 1)[-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<b> <a target=\"blank\" href=\"https://s3.console.aws.amazon.com/s3/buckets/sagemaker-us-east-1-690429675927/sagemaker/telco/output/automl-telco-2020-10-01-02-31-36/sagemaker-automl-candidates/pr-1-f51eea9fb70c4240b3b7a7852800a1f44e1c885b70474e6da81dc1415/\">S3 Generated Resources</a></b>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.core.display import display, HTML\n",
    "\n",
    "if not pr_job_id: \n",
    "    print('No AutoMLJobArtifacts found.')\n",
    "else: \n",
    "    display(HTML('<b> <a target=\"blank\" href=\"https://s3.console.aws.amazon.com/s3/buckets/{}/{}/output/{}/sagemaker-automl-candidates/{}/\">S3 Generated Resources</a></b>'.format(bucket,prefix, auto_ml_job_name, pr_job_id)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert a4-block alert-info\">\n",
    "<b><h1>7. Performing batch inference </h1></div>\n",
    "<a class=\"anchor\" id=\"Inference\"></a>\n",
    "\n",
    "### Let's perform batch inference using the best candidate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "toc-hr-collapsed": false
   },
   "source": [
    "In the dataset preparation step, a test data was generated. In this step, you generate offline or [batch inference](https://docs.aws.amazon.com/sagemaker/latest/dg/batch-transform.html) from the trained model to evaluate the model performance on unseen test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model ARN corresponding to the best candidate is : arn:aws:sagemaker:us-east-1:690429675927:model/automl-telco-2020-10-01-02-31-36\n"
     ]
    }
   ],
   "source": [
    "model_name = model_prefix + '-' + timestamp_suffix\n",
    "\n",
    "model = sm.create_model(Containers=best_candidate['InferenceContainers'],\n",
    "                            ModelName=model_name,\n",
    "                            ExecutionRoleArn=role)\n",
    "\n",
    "print('Model ARN corresponding to the best candidate is : {}'.format(model['ModelArn']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A transform job uses a trained model to get inferences on a dataset and saves these results to the Amazon S3 location `S3OutputPath`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'TransformJobArn': 'arn:aws:sagemaker:us-east-1:690429675927:transform-job/automl-telco-2020-10-01-02-31-36',\n",
       " 'ResponseMetadata': {'RequestId': '98219ed7-7295-4390-9fc5-49c73febbd24',\n",
       "  'HTTPStatusCode': 200,\n",
       "  'HTTPHeaders': {'x-amzn-requestid': '98219ed7-7295-4390-9fc5-49c73febbd24',\n",
       "   'content-type': 'application/x-amz-json-1.1',\n",
       "   'content-length': '109',\n",
       "   'date': 'Thu, 01 Oct 2020 02:56:08 GMT'},\n",
       "  'RetryAttempts': 0}}"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "transform_job_name = model_prefix + '-' + timestamp_suffix\n",
    "\n",
    "transform_input = {\n",
    "        'DataSource': {\n",
    "            'S3DataSource': {\n",
    "                'S3DataType': 'S3Prefix',\n",
    "                'S3Uri': test_data_s3_path\n",
    "            }\n",
    "        },\n",
    "        'ContentType': 'text/csv',\n",
    "        'CompressionType': 'None',\n",
    "        'SplitType': 'Line'\n",
    "    }\n",
    "\n",
    "transform_output = {\n",
    "        'S3OutputPath': 's3://{}/{}/inference-results'.format(bucket,prefix),\n",
    "    }\n",
    "\n",
    "transform_resources = {\n",
    "        'InstanceType': 'ml.m5.4xlarge',\n",
    "        'InstanceCount': 1\n",
    "    }\n",
    "\n",
    "\n",
    "\n",
    "sm.create_transform_job(TransformJobName = transform_job_name,\n",
    "                        ModelName = model_name,\n",
    "                        TransformInput = transform_input,\n",
    "                        TransformOutput = transform_output,\n",
    "                        TransformResources = transform_resources\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tracking the transform job for completion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<u><b>JobStatus</b></u>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "InProgress\n",
      "InProgress\n",
      "InProgress\n",
      "InProgress\n",
      "InProgress\n",
      "InProgress\n",
      "InProgress\n",
      "InProgress\n",
      "InProgress\n",
      "InProgress\n",
      "InProgress\n",
      "InProgress\n",
      "Completed\n",
      "Wall time: 6min 5s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "display(HTML('<u><b>JobStatus</b></u>' ))\n",
    "\n",
    "describe_response = sm.describe_transform_job(TransformJobName = transform_job_name)\n",
    "job_run_status = describe_response['TransformJobStatus']\n",
    "print (job_run_status)\n",
    "\n",
    "while job_run_status not in ('Failed', 'Completed', 'Stopped'):\n",
    "    describe_response = sm.describe_transform_job(TransformJobName = transform_job_name)\n",
    "    job_run_status = describe_response['TransformJobStatus']\n",
    "    print (job_run_status)\n",
    "    sleep(30)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Now let's view the results of the transform job:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Predicted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>195</th>\n",
       "      <td>Yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>196</th>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>197</th>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>198</th>\n",
       "      <td>Yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>199</th>\n",
       "      <td>Yes</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>200 rows × 1 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    Predicted\n",
       "0         Yes\n",
       "1          No\n",
       "2          No\n",
       "3         Yes\n",
       "4         Yes\n",
       "..        ...\n",
       "195       Yes\n",
       "196        No\n",
       "197        No\n",
       "198       Yes\n",
       "199       Yes\n",
       "\n",
       "[200 rows x 1 columns]"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s3_output_key = '{}/inference-results/test_data.csv.out'.format(prefix);\n",
    "local_inference_results_path = 'inference_results.csv'\n",
    "\n",
    "s3 = boto3.resource('s3')\n",
    "inference_results_bucket = s3.Bucket(session.default_bucket())\n",
    "\n",
    "inference_results_bucket.download_file(s3_output_key, local_inference_results_path);\n",
    "\n",
    "data = pd.read_csv(local_inference_results_path, header=None, names=(['Predicted']))\n",
    "pd.set_option('display.max_rows', 10) \n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ploting the confusion matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion matrix, without normalization\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAU4AAAEmCAYAAAAN9HleAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAeRklEQVR4nO3dedwd893/8dc7CbEksTRILCkqEoSQhJBqLJGi3IJaQtqHqla17rp/dKN1o9RSLUVpFa0qamu1pTRoUOSuJYlILbVVJJLIZk2EXonP74+ZS49Lcl1nTs65Zua63k+P87jOmZkz88nine985zvfUURgZmbV65J3AWZmZePgNDPLyMFpZpaRg9PMLCMHp5lZRg5OM7OMHJyGpNUl3S7pTUm3rMR+xkm6u5615UXSpyQ9m3cdVkzyOM7ykHQkcBIwEHgbmAqcHREPreR+Pw98HRgREUtXutCCkxRA/4h4Ie9arJzc4iwJSScBFwHnABsA/YCfAWPqsPuPA891htCshqRueddgBRcRfhX8BawFLAIObWWb7iTBOjt9XQR0T9ftDrwCfAOYB8wBjk7XfR/4N9CUHuMY4Azguop9bwoE0C39/AXgXySt3peAcRXLH6r43gjgMeDN9OeIinX3A2cBE9P93A30XsGvrbn+b1fUfyDwGeA54DXguxXb7wT8HXgj3fZSYNV03QPpr2Vx+us9vGL/3wFeBa5tXpZ+5xPpMYaknzcEFgC75/13w698Xm5xlsMuwGrAH1rZ5nvAzsD2wGCS8Di1Yn0fkgDeiCQcL5O0TkScTtKKvSkiekTEL1srRNKawCXAvhHRkyQcpy5nu3WBO9JtPwZcCNwh6WMVmx0JHA2sD6wKfLOVQ/ch+T3YCDgNuBL4HDAU+BRwmqTN022XAScCvUl+70YBXwOIiJHpNoPTX+9NFftfl6T1fWzlgSPiRZJQvV7SGsDVwK8j4v5W6rUOzMFZDh8DFkTrp9LjgDMjYl5EzCdpSX6+Yn1Tur4pIu4kaW0NqLGe94FBklaPiDkR8dRyttkPeD4iro2IpRFxA/BP4L8qtrk6Ip6LiCXAzSShvyJNJP25TcCNJKF4cUS8nR7/KWA7gIiYHBEPp8edDvwC2K2KX9PpEfFeWs+HRMSVwPPAI0Bfkn+orJNycJbDQqB3G31vGwIvV3x+OV32wT5aBO87QI+shUTEYpLT2+OAOZLukDSwinqaa9qo4vOrGepZGBHL0vfNwTa3Yv2S5u9L2lLSnyW9KuktkhZ171b2DTA/It5tY5srgUHATyPivTa2tQ7MwVkOfwfeJenXW5HZJKeZzfqly2qxGFij4nOfypURcVdEjCZpef2TJFDaqqe5plk11pTFz0nq6h8RvYDvAmrjO60OL5HUg6Tf+JfAGWlXhHVSDs4SiIg3Sfr1LpN0oKQ1JK0iaV9J56eb3QCcKmk9Sb3T7a+r8ZBTgZGS+klaCzileYWkDSQdkPZ1vkdyyr9sOfu4E9hS0pGSukk6HNga+HONNWXRE3gLWJS2hr/aYv1cYPOPfKt1FwOTI+JLJH23l690lVZaDs6SiIgLScZwngrMB2YC/w38Md3kB8AkYBrwD2BKuqyWY90D3JTuazIfDrsuJFfnZ5Ncad6N9MJLi30sBPZPt11IckV8/4hYUEtNGX2T5MLT2ySt4ZtarD8DuEbSG5IOa2tnksYA+5B0T0Dy5zBE0ri6VWyl4gHwZmYZucVpZpaRg9PMLCMHp5lZRg5OM7OMSjeZgbqtHlq1Z95l2EoYPLBf3iXYSpgxYzoLFyxoa1xsJl17fTxi6Udu2FquWDL/rojYp57Hz6p8wblqT7oPaHMEiRXYvQ9dnHcJthL23HV43fcZS5dU/f/1u1Mva+susIYrXXCaWUckUHl6Dh2cZpY/Aarr2X9DOTjNrBi6dM27gqo5OM2sAHyqbmaWnU/VzcwyEG5xmpllI7c4zcwyc4vTzCwjtzjNzDKQPBzJzCwzn6qbmWXhcZxmZtl1cR+nmVn1PI7TzKwGvqpuZpaF+zjNzLLzcCQzswzkWy7NzLLzqbqZWUZucZqZZeGLQ2Zm2bnFaWaWgQfAm5ll5VN1M7PsPI7TzCwj93GamWUgn6qbmWXnFqeZWTZycJqZVU84OM3MslH6KgkHp5kVgOjSxReHzMwy8am6mVlGDk4zsyzcx2lmlo1QqVqc5emNNbMOTVJVryr28ytJ8yQ9WbFsXUn3SHo+/blOxbpTJL0g6VlJe1dTq4PTzAqhXsEJ/BrYp8Wyk4EJEdEfmJB+RtLWwFhgm/Q7P5PU5mwjDk4zy59AXVTVqy0R8QDwWovFY4Br0vfXAAdWLL8xIt6LiJeAF4Cd2jqGg9PMCqGOLc7l2SAi5gCkP9dPl28EzKzY7pV0Wat8ccjMcpfx4lBvSZMqPl8REVfUfOiPira+5OA0s0LIEJwLImJYxt3PldQ3IuZI6gvMS5e/AmxSsd3GwOy2duZTdTMrBlX5qs1twFHp+6OAP1UsHyupu6TNgP7Ao23tzC1OM8uf6nfnkKQbgN1JTulfAU4HzgNulnQMMAM4FCAinpJ0M/A0sBQ4PiKWtXUMB6eZFUK9gjMijljBqlEr2P5s4Owsx3Bwmlnu5NmRzMxqUJ47Ln1xqL1dfvo4Xp5wLpNu+e4Hyw7eawcm/+57LJ58CUO27vfB8rH7DuPhG0/+4LV48iVst2WbQ8ysnbzyykwO2HcUw4cMYpdh23H5ZZcA8I8npjJ69xGM3Hkoe+46nMmT2rzWYGr4OM66cnC2s2tvf5gxx1/2oWVPvTibsd+4koemvPih5Tf+ZRI7jz2PnceexzGn/oaXZ7/GtOdmtWe51opuXbtx1jk/4pEpT3L3fRP55RU/55/PPM3pp57Mt0/5Xx54eDKnnHo6Z5x6ct6llkKZgtOn6u1s4pQX6dd33Q8te/aluW1+77B9hnLz+MmNKstq0KdvX/r07QtAz5492XLAQObMnoUk3n77bQDeeust+vTZMM8yS6MooVgNB2dJHPLpIRx6Yq03R1ijzXh5OtOemMrQHYdzzvkXcsiYz3Dad79NvP8+4+99MO/yyqE8udmYU3UlHpK0b8WywySNb8TxOrodB32cd95t4ukX5+Rdii3HokWLOOrIwzjn/Avp1asXV1/1C87+4QU8+dx0fvDDCzjhq1/Ou8RSKNOpekOCMyICOA64UNJqktYkGSd1fCOO19EduvdQbh4/qe0Nrd01NTVx1JGHcsjhR/BfYw4C4Ibrf/PB+wMPPoTJkx/Ls8RSqDY0O3RwAkTEk8DtwHdIRu5fB3xP0mOSHpc0BkDSNpIelTRV0jRJ/RtVUxlJ4uDRO3DLXe7fLJqI4ISvfpktB2zF8Sec+MHyPn03ZOKDfwPggfvv5ROf8F/panTp0qWqVxE0uo/z+8AU4N/An4F7I+KLktYGHpX0V5KW6cURcb2kVYE2JxEts2vO/QKfGtqf3mv34IXxZ3HW5Xfy+puLufA7h9J7nR7ceslxTHt2FgekV953HbIFs+a+wfRZC3Ou3Fp65O8TuemG69h6m20ZufNQAP73jLO4+NLLOeVbJ7F06VK6r9adn1z685wrLYliNCarouSsuoEHkM4EFgGHAauR3A8KsC6wN7AD8D3gN8CtEfH8cvZxLHAsAKv0GLraNke13MRKZPbEi/MuwVbCnrsO5/Epk+oac9036B8bjavu78VLP9lvcg2zI9VVe1xVfz99CfhsRDzbYv0zkh4B9gPukvSliLi3coN0rr0rALqssX5jk97M2l8dJ/loD+3ZYXAX8HWlvzuSdkh/bg78KyIuIZniabt2rMnMCkCAVN2rCNozOM8CVgGmpU+fOytdfjjwpKSpwECSU3Yz61TKdVW94afqEXFGxcevLGf9ucC5ja7DzIqtIJlYFd85ZGb5E3Sp4gmWReHgNLPcCQenmVlmPlU3M8uoKBd+quHgNLP8FWioUTUcnGaWu2QcZ3mS08FpZgVQnDGa1XBwmlkh+Kq6mVkW7uM0M8vGfZxmZjUoUW46OM2sGNziNDPLqES56eA0swIo2UTGDk4zy52QhyOZmWVVogang9PMisGn6mZmWXgAvJlZNh4Ab2ZWAwenmVlGJcpNB6eZFYNbnGZmGUgex2lmllmJGpwOTjMrhi4lSs4ueRdgZgZJi7OaV3X70omSnpL0pKQbJK0maV1J90h6Pv25Tq21OjjNLHdKJ/mo5tX2vrQRcAIwLCIGAV2BscDJwISI6A9MSD/XxMFpZoXQRdW9qtQNWF1SN2ANYDYwBrgmXX8NcGDNtdb6RTOzesrQ4uwtaVLF69jK/UTELODHwAxgDvBmRNwNbBARc9Jt5gDr11rrCi8OSfopECtaHxEn1HpQM7NKItPFoQURMWyF+0r6LscAmwFvALdI+txKF1mhtavqk+p5IDOz1tRxGOdewEsRMR9A0q3ACGCupL4RMUdSX2BerQdYYXBGxDWVnyWtGRGLaz2QmdkKVXnhp0ozgJ0lrQEsAUaRNAQXA0cB56U//1TrAdrs45S0i6SngWfSz4Ml/azWA5qZLU+9hiNFxCPA74ApwD9Icu4KksAcLel5YHT6uSbVDIC/CNgbuC0t6glJI2s9oJlZSxn7ONsUEacDp7dY/B5J63OlVXXnUETMbNGMXlaPg5uZNSvRjUNVBedMSSOAkLQqycDSZxpblpl1Nh1tdqTjgIuBjYBZwF3A8Y0sysw6Fwm6dqTZkSJiATCuHWoxs06sPLFZ3VX1zSXdLmm+pHmS/iRp8/Yozsw6j3rdq94eqrnl8rfAzUBfYEPgFuCGRhZlZp1LclW9rveqN1Q1wamIuDYilqav62jlVkwzs8yqbG0WpcXZ2r3q66Zv75N0MnAjSWAeDtzRDrWZWSdSkEysSmsXhyaTBGXzL+crFesCOKtRRZlZ51OU1mQ1WrtXfbP2LMTMOi/RwYYjAUgaBGwNrNa8LCJ+06iizKzzKU9sVhGckk4HdicJzjuBfYGHAAenmdWF1PEe1nYIyY3xr0bE0cBgoHtDqzKzTqeeD2trtGpO1ZdExPuSlkrqRTL5pwfAm1lddYiLQxUmSVobuJLkSvsi4NGGVmVmnU6JcrOqe9W/lr69XNJ4oFdETGtsWWbWmQiVqo+ztQHwQ1pbFxFTGlNS63bYqh8TH7k0j0Nbnbz5TlPeJdhKeD8acONggfovq9Fai/OCVtYFsGedazGzTqxriZKztQHwe7RnIWbWeYmOd3HIzKzhSnTjkIPTzIrBwWlmlkEyuL08yVnNDPCS9DlJp6Wf+0naqfGlmVln0tEmMv4ZsAtwRPr5beCyhlVkZp1SR7vlcnhEDJH0OEBEvJ4+JtjMrC4EdCtKKlahmuBsktSV9HEZktYD3m9oVWbW6ZQoN6sKzkuAPwDrSzqbZLakUxtalZl1KlIHueWyWURcL2kyydRyAg6MiGcaXpmZdSolys2qJjLuB7wD3F65LCJmNLIwM+tcinLFvBrVnKrfwX8e2rYasBnwLLBNA+sys04kea56eZKzmlP1bSs/p7MmfWUFm5uZ1aREuZn9zqGImCJpx0YUY2adlDrI7EjNJJ1U8bELMASY37CKzKzTSU7V866ietW0OHtWvF9K0uf5+8aUY2adVYcJznTge4+I+FY71WNmnVSZJvlo7dEZ3SJiaWuP0DAzq4eOdKr+KEl/5lRJtwG3AIubV0bErQ2uzcw6iwJN4FGNavo41wUWkjxjqHk8ZwAOTjOrm44yjnP99Ir6k/wnMJs14DF3ZtZZdaRT9a5ADz4cmM0cnGZWR6rrOE5JawNXAYNI8uqLJHc83gRsCkwHDouI12vZf2vBOScizqxlp2ZmWSRPuazrLi8GxkfEIen8wWsA3wUmRMR5kk4GTga+U8vOW5sBvkQNZzMrtSofm1HN6bykXsBI4JcAEfHviHgDGANck252DXBgreW2Fpyjat2pmVlWXdI5Odt6Ab0lTap4HdtiV5uT3N14taTHJV0laU1gg4iYA5D+XL/WWld4qh4Rr9W6UzOzLDKeqi+IiGGtrO9GMpTy6xHxiKSLSU7L66aah7WZmTVchhZnW14BXomIR9LPvyMJ0rmS+gKkP+fVXGutXzQzq6d6PeUyIl4FZkoakC4aBTwN3AYclS47CvhTrbVmnlbOzKzeVP9p5b4OXJ9eUf8XcDRJQ/FmSccAM4BDa925g9PMCqGesRkRU4Hl9YPW5aK3g9PMctfhHp1hZtYeyhObDk4zK4gSNTgdnGZWBOoYExmbmbUXUa6xkQ5OMysEXxwyM8tCHeSZQ2Zm7cWn6mZmNXCL08wso/LEpoPTzAqiRA1OB6eZ5S/p4yxPcjo4zawAqp5rsxAcnGZWCCXKTQenmeXPp+pmZllVObt7UTg4zawQHJxmZhmpRKfqZbrLqcOZOXMme++1B9tvuxVDBm/DpZdc/KH1P7nwx6y+iliwYEFOFVpr3n33XfbZYwR7fnIoI4cP5vxzvg/AbX/4HSOHD6bv2t2ZOmVyzlWWQzIDfHWvInCLM0fdunXjvPMvYIchQ3j77bcZMXwoo/YazVZbb83MmTO596/3sEm/fnmXaSvQvXt3fn/73azZowdNTU0csPfujBq9DwO33oZfXXcz3/p/x+ddYqm4xWlV6du3LzsMGQJAz549GThwK2bPngXAt795Imefe36p7t/tbCSxZo8eADQ1NbG0qQlJbDlgK7boP6CNb1tLdXyuesM5OAvi5enTmTr1cXbcaTh/vv02NtxwI7YbPDjvsqwNy5YtY9Suwxi0xUaM3GMUQ4btlHdJpVS2U/V2CU5JIemCis/flHRGexy7DBYtWsQRh32WH11wEd26deOH557NaWecmXdZVoWuXbsy4aFJPP70Szw+ZRLPPP1k3iWVlKr+rwjaq8X5HnCwpN7tdLzSaGpq4ojDPsvhR4zjwIMO5l8vvsjL019ip6GDGbDFpsx65RV22WkIr776at6lWivWWnttRuw6kvv+enfepZRTOo6zmlcRtFdwLgWuAE5suULSxyVNkDQt/dlproZEBMd9+RgGDNyK/znxJAAGbbstM2bP49kXpvPsC9PZaOON+fujU+jTp0/O1VpLCxbM58033gBgyZIlPHj/vWyxpfs2a6UqX0XQnn2clwHjJK3VYvmlwG8iYjvgeuCSll+UdKykSZImzV8wvx1KbR//N3Eiv73+Wv52370MH7o9w4duz/i/3Jl3WValea/O4bP7j2aPEUPYZ49dGLnHKD69z37cefsf2WGrzZj86MN87rAxjD1ov7xLLbykj7M8F4cUEY0/iLQoInpIOhNoApYAPSLiDEkLgL4R0SRpFWBORKzwlH7o0GEx8ZFJDa/ZGufNd5ryLsFWwqd325knHp9c1wTbatsd4uo/3FfVtrv0X2dyRAyr5/Gzau+r6hcBxwBrtrJN45PczApHUlWvImjX4IyI14CbScKz2f8BY9P344CH2rMmMysGXxxq3QVA5an4CcDRkqYBnwf+J4eazCxnZbo41C63XEZEj4r3c4E1Kj5PB/ZsjzrMrMCKkopV8L3qZpa7pDVZnuR0cJpZ/grUf1kNB6eZFUKJctPBaWZFUJyhRtVwcJpZIZQoNx2cZpa/Ig01qoaD08yKoUTJ6eA0s0LwcCQzs4zK1MfpR2eYWSHU85ZLSV0lPS7pz+nndSXdI+n59Oc6K1Org9PM8qe6z470P8AzFZ9PBiZERH9gQvq5Zg5OM8udqN/sSJI2BvYDrqpYPAa4Jn1/DXDgytTrPk4zK4QMXZy9JVXOZn5FRFxR8fki4NtAz4plG0TEHICImCNp/ZUo1cFpZgVRfXIuWNEM8JL2B+ZFxGRJu9epso9wcJpZIdRpONIngQMkfQZYDegl6TpgrqS+aWuzLzBvZQ7iPk4zK4R69HFGxCkRsXFEbEryZIl7I+JzwG3AUelmRwF/Wpla3eI0s0Jo8DDO84CbJR0DzAAOXZmdOTjNrBjqnJwRcT9wf/p+ITCqXvt2cJpZ7iQK88z0ajg4zawQyhObDk4zK4oSJaeD08wKQJ4dycwsqxJ1cTo4zSx/ngHezKwWJUpOB6eZFYKHI5mZZVSe2HRwmlkRVDnXZlE4OM2sIMqTnA5OM8td8wzwZeHgNLNCKFFuOjjNrBjc4jQzyyjDEyxz5+A0s0IoT2w6OM2sAKp99G9RODjNrBA8O5KZWVblyU0Hp5kVQ4ly08FpZsXgPk4zswyESjU7Upe8CzAzKxu3OM2sEErU4HRwmlkxeDiSmVkWHgBvZpaNH9ZmZlaLEiWng9PMCsF9nGZmGXUpT246OM2sIBycZmbZlOlUXRGRdw2ZSJoPvJx3HQ3UG1iQdxG2Ujr6n+HHI2K9eu5Q0niS37dqLIiIfep5/KxKF5wdnaRJETEs7zqsdv4z7Ph8r7qZWUYOTjOzjBycxXNF3gXYSvOfYQfnPk4zs4zc4jQzy8jBaWaWkYPTzCwjB2fBSGWaldCsc3JwFogkRXq1TtIBkjbJuyar3or+0fM/hh2P71UvkIrQPB74GrB/vhVZtVr8o7c/8G+ga0T8JSKicr2Vn4OzYCTtBHwJ2DMi5kraFXgPeD4i3si3OmuLpK8BXwbuAA6StFtEnOzQ7Fg8jjNHladwFa2VzYHjSbpRBHwSmAVcGRF35FGnrZikfsDCiFgsaX3gJuBrEfGMpHWAR4FLIuKnuRZqdeU+znxtEilJ20raDpgBPEdyqndtROyYft4mz0LtoyRtAHwD+KqkHhExj2RWpH8DRMTrwEnAhvlVaY3gU/WcpK2TXwKjJX2LpD/z9XT1VyNiTrrdYcAo4MhcCrXWzAceA4YAR0u6FPgXcKOkXSJiKbApsImkrhGxLL9SrZ7c4szPKkAvSYcDIyJiN5LTuh7AXABJnwI+D3whIp7NrVL7EEn9JQ2IiPeB64H7gK2AL0fEd4BJwAOSLge+CJzj0OxY3MeZI0k/AN4EFpL8jzcY2C8imiSNjoh7JK3ti0LFIeljJC3NBcD3gWUkk3ocCWwBzImIX0gaDqwOvBwRL+VVrzWGT9XbkaSRwBgggKtJ+r62BFYFVgP2jYhlkr5A0m82OSJey6te+6iIWChpL+CvJGdsg0kuCC0i6dvcNr3od3VEvJdfpdZIbnG2I0kDgEHAaOAN4LPANJIWzEDgNqAvsDdwREQ8lVOp1gZJo4FLSIJzA2BPYCywEzAH+GREvJlfhdZIDs4cSdqeJCTfA/oATwIbA7dGxHN51mZtk7Qf8BNg54h4LR1+tAqwRkRMz7U4ayifqrez5jtI0p9TJb0LHE7yZ/FERFyXc4lWpYi4Q9L7wMPpVfSFeddk7cNX1dtZ80D3ip//BG4muUA0N8fSrAYR8RfgW8BfJfn/p07Cp+oFIWmViGjKuw6rTToAflHedVj7cHCamWXkUwszs4wcnGZmGTk4zcwycnCamWXk4DQzy8jBaWaWkYOzE5C0TNJUSU9KukXSGiuxr19LOiR9f5WkrVvZdndJI2o4xnRJvatd3mKbTGMpJZ0h6ZtZa7TOzcHZOSyJiO0jYhDJDD7HVa6U1LWWnUbElyLi6VY22R3IHJxmRefg7HweBLZIW4P3Sfot8A9JXSX9SNJjkqZJ+gok99ZLulTS05LuANZv3pGk+yUNS9/vI2mKpCckTZC0KUlAn5i2dj8laT1Jv0+P8ZikT6bf/ZikuyU9LukXJM9aapWkP0qaLOkpSce2WHdBWssESeulyz4haXz6nQclDazHb6Z1Tp7koxOR1A3YFxifLtoJGBQRL6Xh82ZE7CipOzBR0t3ADsAAYFuS6dOeBn7VYr/rAVcCI9N9rZvOFnQ5sCgifpxu91vgJxHxUPqQs7tIJnA+HXgoIs5MZxz6UBCuwBfTY6wOPCbp9+kkG2sCUyLiG5JOS/f93ySTDR8XEc+nkwz/jGQqOLPMHJydw+qSpqbvHyR51tEI4NGK2ck/DWzX3H8JrAX0B0YCN6SPfpgt6d7l7H9n4IHmfbUy+fJewNb6z8M9e0nqmR7j4PS7d0h6fQXfr3SCpIPS95uktS4E3ieZWBjgOuBWST3SX+8tFcfuXsUxzJbLwdk5LImI7SsXpAGyuHIR8PWIuKvFdp8hmbG+NapiG0i6hnaJiCXLqaXqSRMk7U4SwrtExDuS7ieZQX95Ij3uGy1/D8xq5T5Oa3YXyeM6VgGQtKWkNYEHgLFpH2hfYI/lfPfvwG6SNku/u266/G2gZ8V2d5OcNpNu1xxkDwDj0mX7Auu0UetawOtpaA4kafE26wI0t5qPJOkCeAt4SdKh6TEkaXAbxzBbIQenNbuKpP9yiqQngV+QnJH8AXge+Afwc+BvLb8YEfNJ+iVvlfQE/zlVvh04qPniEHACMCy9+PQ0/7m6/31gpKQpJF0GM9qodTzQTdI04Czg4Yp1i4FtJE0m6cM8M10+Djgmre8pkmc/mdXE08qZmWXkFqeZWUYOTjOzjBycZmYZOTjNzDJycJqZZeTgNDPLyMFpZpbR/wdjhOoSXuOmwwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "data['Label']= test_data[target_column].to_list()\n",
    "cm = confusion_matrix(data['Label'], data['Predicted'])\n",
    "plot_confusion_matrix(cm, classes=['Yes', 'No'], title='Confusion matrix')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calculating the model accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy Score = 0.74\n"
     ]
    }
   ],
   "source": [
    "print(\"Accuracy Score = {}\".format(accuracy_score(data['Label'] , data['Predicted'])))"
   ]
  }
 ],
 "metadata": {
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}