examples/Serving-An-Implicit-Model-With-Merlin-Systems.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "5cdba80f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Copyright 2022 NVIDIA Corporation. All Rights Reserved.\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#     http://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License.\n",
    "# ==============================================================================\n",
    "\n",
    "# Each user is responsible for checking the content of datasets and the\n",
    "# applicable licenses and determining if suitable for the intended use."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77acbcad",
   "metadata": {},
   "source": [
    "<img src=\"https://developer.download.nvidia.com/notebooks/dlsw-notebooks/merlin_systems_serving-an-implicit-model-with-merlin-systems/nvidia_logo.png\" style=\"width: 90px; float: right;\">\n",
    "\n",
    "# Serving an Implicit Model with Merlin Systems\n",
    "\n",
    "This notebook is created using the latest stable [merlin-tensorflow](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/merlin/containers/merlin-tensorflow) container. This Jupyter notebook example demonstrates how to deploy an `Implicit` model to Triton Inference Server (TIS) and generate prediction results for a given query.\n",
    "\n",
    "## Overview\n",
    "\n",
    "NVIDIA Merlin is an open source framework that accelerates and scales end-to-end recommender system pipelines. The Merlin framework is broken up into several sub components, these include: Merlin-Core, Merlin-Models, NVTabular and Merlin-Systems. Merlin Systems will be the focus of this example.\n",
    "\n",
    "The purpose of the Merlin Systems library is to make it easy for Merlin users to quickly deploy their recommender systems from development to [Triton Inference Server](https://github.com/triton-inference-server/server). We extended the same user-friendly API users are accustomed to in NVTabular and leverage it to accommodate deploying recommender system components to TIS. \n",
    "\n",
    "### Learning objectives\n",
    "\n",
    "In this notebook, we learn how to deploy an NVTabular Workflow and a trained `Implicit` model from Merlin Models to Triton.\n",
    "- Create Ensemble Graph\n",
    "- Export Ensemble Graph\n",
    "- Run Triton server\n",
    "- Send request to Triton and verify results\n",
    "\n",
    "### Dataset\n",
    "\n",
    "We use the [MovieLens 100k Dataset](https://grouplens.org/datasets/movielens/100k/). It consists of ratings a user has given a movie along with some metadata for the user and the movie. We train an Implicit model to predict the rating based on user and item features and proceed to deploy it to the Triton Inference Server.\n",
    "\n",
    "It is important to note that the steps taken in this notebook are generalized and can be applied to any set of workflows and models. \n",
    "\n",
    "### Tools\n",
    "\n",
    "- NVTabular\n",
    "- Merlin Models\n",
    "- Merlin Systems\n",
    "- Triton Inference Server"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6efad6b8",
   "metadata": {},
   "source": [
    "## Prerequisite: Preparing the data and Training Implicit"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "356ef8c9",
   "metadata": {},
   "source": [
    "In this tutorial our objective is to demonstrate how to serve an `Implicit` model. In order for us to be able to do so, we begin by downloading data and training a model. We breeze through these activities below.\n",
    "\n",
    "If you would like to learn more about training an `Implicit` model using the Merlin Models library, please consult this [tutorial](https://github.com/NVIDIA-Merlin/models/blob/stable/examples/07-Train-traditional-ML-models-using-the-Merlin-Models-API.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "edea28d0",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import nvtabular as nvt\n",
    "import numpy as np\n",
    "from merlin.schema.tags import Tags\n",
    "from merlin.models.implicit import BayesianPersonalizedRanking\n",
    "\n",
    "from merlin.datasets.entertainment import get_movielens\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b756a12f",
   "metadata": {},
   "outputs": [],
   "source": [
    "ensemble_export_path = os.environ.get(\"OUTPUT_DATA_DIR\", \"ensemble\")\n",
    "USE_GPU = bool(int(os.environ.get(\"USE_GPU\", \"1\")))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5c10a993",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "train, _ = get_movielens(variant='ml-100k')\n",
    "\n",
    "# the implicit model expects a `user_id` column hence the need to rename it\n",
    "train = nvt.Dataset(train.compute().rename(columns = {'userId': 'user_id'}))\n",
    "\n",
    "user_id  = ['user_id'] >> nvt.ops.Categorify() >> nvt.ops.TagAsUserID()\n",
    "movieId  = ['movieId'] >> nvt.ops.Categorify() >> nvt.ops.TagAsItemID()\n",
    "\n",
    "train_workflow = nvt.Workflow(user_id + movieId)\n",
    "train_transformed = train_workflow.fit_transform(train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff168b4a",
   "metadata": {},
   "source": [
    "Having preprocessed our data, let's train our model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "d0b55be5",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.8/dist-packages/tqdm/auto.py:22: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n",
      "2022-09-05 09:32:07.681291: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:991] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2022-09-05 09:32:07.681740: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:991] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2022-09-05 09:32:07.681877: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:991] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "/usr/local/lib/python3.8/dist-packages/cudf/core/frame.py:384: UserWarning: The deep parameter is ignored and is only included for pandas compatibility.\n",
      "  warnings.warn(\n",
      "100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [00:00<00:00, 777.52it/s, train_auc=85.42%, skipped=29.68%]\n"
     ]
    }
   ],
   "source": [
    "model = BayesianPersonalizedRanking(use_gpu=USE_GPU)\n",
    "model.fit(train_transformed)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4a3cf39",
   "metadata": {},
   "source": [
    "## Create the Ensemble Graph"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc40083e",
   "metadata": {},
   "source": [
    "Let us now define an `Ensemble` that will be used for serving predictions on the Triton Inference Server.\n",
    "\n",
    "An `Ensemble` defines operations to be performed on incoming requests. It begins with specifying what fields the inference request will contain.\n",
    "\n",
    "Our model was trained on data that included the `movieId` column. However, in production, this information will not be available to us, this is what we will be trying to predict.\n",
    "\n",
    "In general, you want to define a preprocessing workflow once and apply it throughout the lifecycle of your model, from training all the way to serving in production. Redefining the workflows on the go, or using custom written code for these operations, can be a source of subtle bugs.\n",
    "\n",
    "In order to ensure we process our data in the same way in production as we do in training, let us now modify the training preprocessing pipeline and use it to construct our inference workflow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "fa8dc34a",
   "metadata": {},
   "outputs": [],
   "source": [
    "inf_workflow = train_workflow.remove_inputs(['movieId'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d71c5636",
   "metadata": {},
   "source": [
    "Equipped with the modified data preprocessing workflow, let us define the full set of inference operations we will want to run on the Triton Inference Server.\n",
    "\n",
    "We begin by stating what data the server can expect (`inf_workflow.input_schema.column_names`). We proceed to wrap our `inf_workflow` in `TransformWorkflow` -- an operator we can leverage for executing our NVTabular workflow during serving.\n",
    "\n",
    "Last but not least, having received and preprocessed the data, we instruct the Triton Inference Server to perform inference using the model that we trained. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "de9e2237",
   "metadata": {},
   "outputs": [],
   "source": [
    "from merlin.systems.dag.ops.implicit import PredictImplicit\n",
    "from merlin.systems.dag.ensemble import Ensemble\n",
    "from merlin.systems.dag.ops.workflow import TransformWorkflow\n",
    "\n",
    "inf_ops = inf_workflow.input_schema.column_names >> TransformWorkflow(inf_workflow) \\\n",
    "                    >> PredictImplicit(model.implicit_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76dad9c3",
   "metadata": {},
   "source": [
    "With inference operations defined, all that remains now is outputting the ensemble to disk so that it can be loaded up when Triton starts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "e23a7fc3",
   "metadata": {},
   "outputs": [],
   "source": [
    "ensemble = Ensemble(inf_ops, inf_workflow.input_schema)\n",
    "ensemble.export(ensemble_export_path);"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9165dfd",
   "metadata": {},
   "source": [
    "## Starting the Triton Inference Server"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "353e8602",
   "metadata": {},
   "source": [
    "After we export the ensemble, we are ready to start the Triton Inference Server. The server is installed in Merlin Tensorflow and Merlin PyTorch containers. If you are not using one of our containers, then ensure it is installed in your environment. For more information, see the Triton Inference Server [documentation](https://github.com/triton-inference-server/server/blob/r22.03/README.md#documentation).\n",
    "\n",
    "You can start the server by running the following command:\n",
    "\n",
    "```shell\n",
    "tritonserver --model-repository=ensemble\n",
    "```\n",
    "\n",
    "For the `--model-repository` argument, specify the same value as the `export_path` that you specified previously in the `ensemble.export` method.\n",
    "\n",
    "After you run the `tritonserver` command, wait until your terminal shows messages like the following example:\n",
    "\n",
    "```shell\n",
    "I0414 18:29:50.741833 4067 grpc_server.cc:4421] Started GRPCInferenceService at 0.0.0.0:8001\n",
    "I0414 18:29:50.742197 4067 http_server.cc:3113] Started HTTPService at 0.0.0.0:8000\n",
    "I0414 18:29:50.783470 4067 http_server.cc:178] Started Metrics Service at 0.0.0.0:8002\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65b7e4e8",
   "metadata": {},
   "source": [
    "## Retrieving Recommendations from Triton Inference Server\n",
    "\n",
    "Now that our server is running, we can send requests to it. This request is composed of values that correspond to the request schema that was created when we exported the ensemble graph.\n",
    "\n",
    "In the code below we create a request to send to Triton and send it. We will then analyze the response, to show the full experience.\n",
    "\n",
    "We begin by obtaining 10 examples from our train data to include in the request."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "2d61751b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>70</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>86</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>96</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>109</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>143</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>183</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>609</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>858</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id\n",
       "0        6\n",
       "1       15\n",
       "2       70\n",
       "3       86\n",
       "4       96\n",
       "5      109\n",
       "6      143\n",
       "7      183\n",
       "8      609\n",
       "9      858"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ten_examples = train.compute()['user_id'].unique().sample(10).sort_values().to_frame().reset_index(drop=True)\n",
    "ten_examples"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7808bc12",
   "metadata": {},
   "source": [
    "Let's now package the information up as inputs and send it to Triton for inference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "2fefd5b8",
   "metadata": {},
   "outputs": [],
   "source": [
    "from merlin.systems.triton import convert_df_to_triton_input\n",
    "import tritonclient.grpc as grpcclient\n",
    "\n",
    "inputs = convert_df_to_triton_input(inf_workflow.input_schema, ten_examples, grpcclient.InferInput)\n",
    "\n",
    "outputs = [\n",
    "    grpcclient.InferRequestedOutput(col)\n",
    "    for col in inf_ops.output_schema.column_names\n",
    "]\n",
    "# send request to tritonserver\n",
    "with grpcclient.InferenceServerClient(\"localhost:8001\") as client:\n",
    "    response = client.infer(\"executor_model\", inputs, outputs=outputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3dc7909f",
   "metadata": {},
   "source": [
    "We can now compare the predictions from the server to those from our local model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "6ddd35cc",
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions_from_triton = response.as_numpy(outputs[0].name())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "6f28fdfe",
   "metadata": {},
   "outputs": [],
   "source": [
    "local_predictions = model.predict(inf_workflow.transform(nvt.Dataset(ten_examples)))[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "e946de27",
   "metadata": {},
   "outputs": [],
   "source": [
    "np.testing.assert_allclose(predictions_from_triton, local_predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d8aa4456",
   "metadata": {},
   "source": [
    "We managed to preprocess the data in the same way in serving as we did during training and obtain the same predictions!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}