bandits_movielens_testbed.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Contextual Multi-Armed Bandit Recommendation System\n",
    "Div Dasani"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The recommendation problem can be contextualized as an agent recommending an action given a pre-existing state. Contextual multi-armed bandits use this perspective to suggest optimal recommendations, trading-off between exploration and exploitation to minimize cumulative regret. The bandit algorithms are appropriate to use in recommendation problems when there are many cold items (items which have no or little interaction data) in the catalog or if user preferences change over time.\n",
    "\n",
    "The MovieLens100K dataset is used for this notebook. AWS architecture is employed to make the system more easily scalable and deployable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "import boto3\n",
    "import sys\n",
    "import os\n",
    "import json\n",
    "import glob\n",
    "import re\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from IPython.display import HTML\n",
    "import time\n",
    "from time import gmtime, strftime\n",
    "from misc import get_execution_role, wait_for_s3_object\n",
    "from sagemaker.rl import RLEstimator\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setup S3 bucket"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sage_session = sagemaker.session.Session()\n",
    "s3_bucket = sage_session.default_bucket()  \n",
    "s3_output_path = 's3://{}/'.format(s3_bucket)\n",
    "print(\"S3 bucket path: {}\".format(s3_output_path))\n",
    "\n",
    "#AWS region must be us-west-2\n",
    "aws_region = sage_session.boto_region_name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create an IAM role"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "instance_type = 'local'\n",
    "role = get_execution_role()\n",
    "print(\"Using IAM role arn: {}\".format(role))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download MovieLens 100K and upload to S3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "curl -o ml-100k.zip http://files.grouplens.org/datasets/movielens/ml-100k.zip\n",
    "unzip ml-100k.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "movielens_data_s3_path = sage_session.upload_data(path=\"ml-100k\", bucket=s3_bucket, key_prefix=\"movielens/data\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hyperparameters = {\n",
    "                   # Algorithm params\n",
    "                   \"arm_features\": True,\n",
    "                   \"exploration_policy\": \"regcbopt\",\n",
    "                   \"mellowness\": 0.01,\n",
    "                   \n",
    "                   # Env params\n",
    "                   \"item_pool_size\": 100,\n",
    "                   \"top_k\": 5,\n",
    "                   \"total_interactions\": 2000,\n",
    "                   \"max_users\": 100,\n",
    "                   }\n",
    "\n",
    "job_name_prefix = \"testbed-bandits-1\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vw_image_uri = \"462105765813.dkr.ecr.us-west-2.amazonaws.com/sagemaker-rl-vw-container:adf\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "estimator = RLEstimator(entry_point=\"train.py\",\n",
    "                        source_dir='src',\n",
    "                        image_name=vw_image_uri,\n",
    "                        role=role,\n",
    "                        train_instance_type=instance_type,\n",
    "                        train_instance_count=1,\n",
    "                        output_path=s3_output_path,\n",
    "                        base_job_name=job_name_prefix,\n",
    "                        hyperparameters = hyperparameters\n",
    "                    )\n",
    "\n",
    "estimator.fit(inputs={\"movielens\": movielens_data_s3_path}, wait=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plot for Performance Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "job_name = estimator.latest_training_job.job_name\n",
    "output_path_prefix = f\"{job_name}/output.tar.gz\"\n",
    "model_path = f\"{job_name}/model.tar.gz\"\n",
    "sage_session.download_data(path=\"./output\", bucket=s3_bucket, key_prefix=output_path_prefix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "tar -C ./output -xvzf ./output/output.tar.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "output_path_local = \"output/data/output.json\"\n",
    "with open(output_path_local) as f:\n",
    "    all_regrets = json.load(f)\n",
    "    \n",
    "all_regrets = {key: np.cumsum(val) for key,val in all_regrets.items()}\n",
    "df = pd.DataFrame(all_regrets)\n",
    "df.plot(title=\"Cumulative Regret\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a SageMaker model for inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sage_session = sagemaker.local.LocalSession()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bandit_model = sagemaker.model.Model(image=vw_image_uri,\n",
    "                                     role=role,\n",
    "                                     name=\"vw-model-1\",\n",
    "                                     model_data=f\"s3://{s3_bucket}/{model_path}\",\n",
    "                                     sagemaker_session=sage_session)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bandit_model.deploy(initial_instance_count=1, instance_type=instance_type, endpoint_name=\"bandit\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictor = sagemaker.predictor.RealTimePredictor(endpoint=\"bandit\",\n",
    "                                                  sagemaker_session=bandit_model.sagemaker_session,\n",
    "                                                  serializer=sagemaker.predictor.json_serializer,\n",
    "                                                  deserializer=sagemaker.predictor.json_deserializer,\n",
    "                                                 )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictor.predict({\"shared_context\": None, \"actions_context\": [[0, 0, 1], [1, 0, 0], [1, 1, 1]], \"top_k\": 2})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Clean Up endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if \"predictor\" in locals():\n",
    "    predictor.delete_endpoint()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}