mlflow_server_ab_test_ambassador.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# MLFlow Pre-packaged Model Server AB Test Deployment \n",
    "In this example we will build two models with MLFlow and we will deploy them as an A/B test deployment. The reason this is powerful is because it allows you to deploy a new model next to the old one, distributing a percentage of traffic. These deployment strategies are quite simple using Seldon, and can be extended to shadow deployments, multi-armed-bandits, etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tutorial Overview\n",
    "\n",
    "This tutorial will follow closely break down in the following sections:\n",
    "\n",
    "1. Train the MLFlow elastic net wine example\n",
    "\n",
    "2. Deploy your trained model leveraging our pre-packaged MLFlow model server\n",
    "\n",
    "3. Test the deployed MLFlow model by sending requests\n",
    "\n",
    "4. Deploy your second model as an A/B test\n",
    "\n",
    "5. Visualise and monitor the performance of your models using Seldon Analytics\n",
    "\n",
    "It will follow closely our talk at the [Spark + AI Summit 2019 on Seldon and MLflow](https://www.youtube.com/watch?v=D6eSfd9w9eA)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dependencies\n",
    "\n",
    "For this example to work you must be running Seldon 0.3.2 or above - you can follow our [getting started guide for this](https://docs.seldon.io/projects/seldon-core/en/latest/workflow/install.html).\n",
    "\n",
    "In regards to other dependencies, make sure you have installed:\n",
    "\n",
    "* Helm v3.0.0+\n",
    "* kubectl v1.14+\n",
    "* Python 3.6+\n",
    "* MLFlow 1.1.0\n",
    "* [pygmentize](https://pygments.org/docs/cmdline/)\n",
    "\n",
    "We will also take this chance to load the Python dependencies we will use through the tutorial:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: pip in /Users/dileep.gadiraju/.pyenv/versions/3.7.8/lib/python3.7/site-packages (22.3.1)\n",
      "Requirement already satisfied: pandas in /Users/dileep.gadiraju/.pyenv/versions/3.7.8/lib/python3.7/site-packages (1.3.5)\n",
      "Requirement already satisfied: numpy>=1.17.3 in /Users/dileep.gadiraju/.pyenv/versions/3.7.8/lib/python3.7/site-packages (from pandas) (1.19.0)\n",
      "Requirement already satisfied: pytz>=2017.3 in /Users/dileep.gadiraju/.pyenv/versions/3.7.8/lib/python3.7/site-packages (from pandas) (2022.1)\n",
      "Requirement already satisfied: python-dateutil>=2.7.3 in /Users/dileep.gadiraju/.pyenv/versions/3.7.8/lib/python3.7/site-packages (from pandas) (2.8.2)\n",
      "Requirement already satisfied: six>=1.5 in /Users/dileep.gadiraju/.pyenv/versions/3.7.8/lib/python3.7/site-packages (from python-dateutil>=2.7.3->pandas) (1.15.0)\n",
      "Collecting seldon-core\n",
      "  Downloading seldon_core-1.15.0-py3-none-any.whl (116 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m116.5/116.5 kB\u001b[0m \u001b[31m1.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hCollecting PyYAML<5.5,>=5.4\n",
      "  Using cached PyYAML-5.4.1-cp37-cp37m-macosx_10_9_x86_64.whl (249 kB)\n",
      "Collecting grpcio-reflection<1.35.0\n",
      "  Using cached grpcio-reflection-1.34.1.tar.gz (11 kB)\n",
      "  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hCollecting prometheus-client<0.9.0,>=0.7.1\n",
      "  Using cached prometheus_client-0.8.0-py2.py3-none-any.whl (53 kB)\n",
      "Requirement already satisfied: numpy<2.0.0 in /Users/dileep.gadiraju/.pyenv/versions/3.7.8/lib/python3.7/site-packages (from seldon-core) (1.19.0)\n",
      "Requirement already satisfied: setuptools>=41.0.0 in /Users/dileep.gadiraju/.pyenv/versions/3.7.8/lib/python3.7/site-packages (from seldon-core) (47.1.0)\n",
      "Collecting grpcio-opentracing<1.2.0,>=1.1.4\n",
      "  Using cached grpcio_opentracing-1.1.4-py3-none-any.whl (14 kB)\n",
      "Collecting werkzeug<2.3,>=2.1.1\n",
      "  Using cached Werkzeug-2.2.2-py3-none-any.whl (232 kB)\n",
      "Requirement already satisfied: requests<3.0.0 in /Users/dileep.gadiraju/.pyenv/versions/3.7.8/lib/python3.7/site-packages (from seldon-core) (2.25.1)\n",
      "Collecting gunicorn<20.2.0,>=19.9.0\n",
      "  Using cached gunicorn-20.1.0-py3-none-any.whl (79 kB)\n",
      "Collecting jsonschema<4.0.0\n",
      "  Using cached jsonschema-3.2.0-py2.py3-none-any.whl (56 kB)\n",
      "Collecting click<8.1,>=8.0.0a1\n",
      "  Using cached click-8.0.4-py3-none-any.whl (97 kB)\n",
      "Collecting flatbuffers<2.0.0\n",
      "  Using cached flatbuffers-1.12-py2.py3-none-any.whl (15 kB)\n",
      "Collecting cryptography<3.5,>=3.4\n",
      "  Using cached cryptography-3.4.8-cp36-abi3-macosx_10_10_x86_64.whl (2.0 MB)\n",
      "Requirement already satisfied: grpcio<2.0.0 in /Users/dileep.gadiraju/.pyenv/versions/3.7.8/lib/python3.7/site-packages (from seldon-core) (1.30.0)\n",
      "Requirement already satisfied: protobuf<4.0.0 in /Users/dileep.gadiraju/.pyenv/versions/3.7.8/lib/python3.7/site-packages (from seldon-core) (3.12.2)\n",
      "Collecting Flask-OpenTracing<1.2.0,>=1.1.0\n",
      "  Using cached Flask-OpenTracing-1.1.0.tar.gz (8.2 kB)\n",
      "  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hRequirement already satisfied: Flask<3.0.0,>=2.0.0 in /Users/dileep.gadiraju/.pyenv/versions/3.7.8/lib/python3.7/site-packages (from seldon-core) (2.1.0)\n",
      "Collecting opentracing<2.5.0,>=2.2.0\n",
      "  Using cached opentracing-2.4.0.tar.gz (46 kB)\n",
      "  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hCollecting jaeger-client<4.5.0,>=4.1.0\n",
      "  Using cached jaeger-client-4.4.0.tar.gz (83 kB)\n",
      "  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hRequirement already satisfied: urllib3<1.27,>=1.26.5 in /Users/dileep.gadiraju/.pyenv/versions/3.7.8/lib/python3.7/site-packages (from seldon-core) (1.26.9)\n",
      "Collecting Flask-cors<4.0.0\n",
      "  Using cached Flask_Cors-3.0.10-py2.py3-none-any.whl (14 kB)\n",
      "Requirement already satisfied: importlib-metadata in /Users/dileep.gadiraju/.pyenv/versions/3.7.8/lib/python3.7/site-packages (from click<8.1,>=8.0.0a1->seldon-core) (1.7.0)\n",
      "Requirement already satisfied: cffi>=1.12 in /Users/dileep.gadiraju/.pyenv/versions/3.7.8/lib/python3.7/site-packages (from cryptography<3.5,>=3.4->seldon-core) (1.15.1)\n",
      "Requirement already satisfied: itsdangerous>=2.0 in /Users/dileep.gadiraju/.pyenv/versions/3.7.8/lib/python3.7/site-packages (from Flask<3.0.0,>=2.0.0->seldon-core) (2.1.2)\n",
      "Requirement already satisfied: Jinja2>=3.0 in /Users/dileep.gadiraju/.pyenv/versions/3.7.8/lib/python3.7/site-packages (from Flask<3.0.0,>=2.0.0->seldon-core) (3.1.2)\n",
      "Requirement already satisfied: Six in /Users/dileep.gadiraju/.pyenv/versions/3.7.8/lib/python3.7/site-packages (from Flask-cors<4.0.0->seldon-core) (1.15.0)\n",
      "Collecting grpcio<2.0.0\n",
      "  Downloading grpcio-1.51.1-cp37-cp37m-macosx_10_10_x86_64.whl (4.6 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m4.6/4.6 MB\u001b[0m \u001b[31m5.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting threadloop<2,>=1\n",
      "  Using cached threadloop-1.0.2.tar.gz (4.9 kB)\n",
      "  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hCollecting thrift\n",
      "  Using cached thrift-0.16.0.tar.gz (59 kB)\n",
      "  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hRequirement already satisfied: tornado>=4.3 in /Users/dileep.gadiraju/.pyenv/versions/3.7.8/lib/python3.7/site-packages (from jaeger-client<4.5.0,>=4.1.0->seldon-core) (6.2)\n",
      "Collecting pyrsistent>=0.14.0\n",
      "  Downloading pyrsistent-0.19.2-cp37-cp37m-macosx_10_9_x86_64.whl (69 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m69.8/69.8 kB\u001b[0m \u001b[31m1.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: attrs>=17.4.0 in /Users/dileep.gadiraju/.pyenv/versions/3.7.8/lib/python3.7/site-packages (from jsonschema<4.0.0->seldon-core) (21.4.0)\n",
      "Requirement already satisfied: idna<3,>=2.5 in /Users/dileep.gadiraju/.pyenv/versions/3.7.8/lib/python3.7/site-packages (from requests<3.0.0->seldon-core) (2.10)\n",
      "Requirement already satisfied: chardet<5,>=3.0.2 in /Users/dileep.gadiraju/.pyenv/versions/3.7.8/lib/python3.7/site-packages (from requests<3.0.0->seldon-core) (4.0.0)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /Users/dileep.gadiraju/.pyenv/versions/3.7.8/lib/python3.7/site-packages (from requests<3.0.0->seldon-core) (2022.6.15)\n",
      "Requirement already satisfied: MarkupSafe>=2.1.1 in /Users/dileep.gadiraju/.pyenv/versions/3.7.8/lib/python3.7/site-packages (from werkzeug<2.3,>=2.1.1->seldon-core) (2.1.1)\n",
      "Requirement already satisfied: pycparser in /Users/dileep.gadiraju/.pyenv/versions/3.7.8/lib/python3.7/site-packages (from cffi>=1.12->cryptography<3.5,>=3.4->seldon-core) (2.21)\n",
      "Requirement already satisfied: zipp>=0.5 in /Users/dileep.gadiraju/.pyenv/versions/3.7.8/lib/python3.7/site-packages (from importlib-metadata->click<8.1,>=8.0.0a1->seldon-core) (3.1.0)\n",
      "Building wheels for collected packages: Flask-OpenTracing, grpcio-reflection, jaeger-client, opentracing, threadloop, thrift\n",
      "  Building wheel for Flask-OpenTracing (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for Flask-OpenTracing: filename=Flask_OpenTracing-1.1.0-py3-none-any.whl size=9069 sha256=8d6d81293f2cc909fc4a72b93f2af740b1723cdac8376eddcb9ba8f90a0d490a\n",
      "  Stored in directory: /Users/dileep.gadiraju/Library/Caches/pip/wheels/1f/dc/b0/0eface98fde5c132faec8ffc902a766f14644b3c7276b801e2\n",
      "  Building wheel for grpcio-reflection (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for grpcio-reflection: filename=grpcio_reflection-1.34.1-py3-none-any.whl size=14413 sha256=34094f82602fdb6244eced00d8941d8e06d7f549466620792dc50e766c1d874f\n",
      "  Stored in directory: /Users/dileep.gadiraju/Library/Caches/pip/wheels/6c/b7/e0/a3fdd5dde7edcea87e169f084d206b4c58dcc7eeba9628a359\n",
      "  Building wheel for jaeger-client (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for jaeger-client: filename=jaeger_client-4.4.0-py3-none-any.whl size=64326 sha256=8d09d8ac1c733566f9af5af545323bacb2a8aa6441d0695e4c8e6b2b3cd726db\n",
      "  Stored in directory: /Users/dileep.gadiraju/Library/Caches/pip/wheels/37/58/60/c574afdb2ea8250fa1c952d568d1fe529dd1e34880d1931de9\n",
      "  Building wheel for opentracing (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for opentracing: filename=opentracing-2.4.0-py3-none-any.whl size=51399 sha256=e2652d1e366e54244a3e7be38f50448a60c5d9dce387caeb15dfa1a9d471bea3\n",
      "  Stored in directory: /Users/dileep.gadiraju/Library/Caches/pip/wheels/66/ba/8b/89f23a702d6939fc69952942c29605857039290cb7b3bc0aa9\n",
      "  Building wheel for threadloop (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for threadloop: filename=threadloop-1.0.2-py3-none-any.whl size=3422 sha256=4c06e16c6ac03373012b654809f3a70189fb4c0236918cf7efcf008c60b56d58\n",
      "  Stored in directory: /Users/dileep.gadiraju/Library/Caches/pip/wheels/da/7c/46/1bd27587c8361624a027699b260a98193394e261e326c5670a\n",
      "  Building wheel for thrift (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for thrift: filename=thrift-0.16.0-cp37-cp37m-macosx_12_0_x86_64.whl size=176547 sha256=c34d85f086398b5a15a3d612f7bb350d62fe2134f3f7a43c2e7b403bceea1c34\n",
      "  Stored in directory: /Users/dileep.gadiraju/Library/Caches/pip/wheels/61/24/0b/735dd29bf792a119ea2abad02deb00a20a1477eb8662f184e6\n",
      "Successfully built Flask-OpenTracing grpcio-reflection jaeger-client opentracing threadloop thrift\n",
      "Installing collected packages: prometheus-client, opentracing, flatbuffers, werkzeug, thrift, threadloop, PyYAML, pyrsistent, gunicorn, grpcio, jsonschema, jaeger-client, grpcio-reflection, grpcio-opentracing, cryptography, click, Flask-OpenTracing, Flask-cors, seldon-core\n",
      "  Attempting uninstall: werkzeug\n",
      "    Found existing installation: Werkzeug 1.0.1\n",
      "    Uninstalling Werkzeug-1.0.1:\n",
      "      Successfully uninstalled Werkzeug-1.0.1\n",
      "  Attempting uninstall: PyYAML\n",
      "    Found existing installation: PyYAML 5.3.1\n",
      "    Uninstalling PyYAML-5.3.1:\n",
      "      Successfully uninstalled PyYAML-5.3.1\n",
      "  Attempting uninstall: grpcio\n",
      "    Found existing installation: grpcio 1.30.0\n",
      "    Uninstalling grpcio-1.30.0:\n",
      "      Successfully uninstalled grpcio-1.30.0\n",
      "  Attempting uninstall: cryptography\n",
      "    Found existing installation: cryptography 37.0.4\n",
      "    Uninstalling cryptography-37.0.4:\n",
      "      Successfully uninstalled cryptography-37.0.4\n",
      "  Attempting uninstall: click\n",
      "    Found existing installation: click 8.1.3\n",
      "    Uninstalling click-8.1.3:\n",
      "      Successfully uninstalled click-8.1.3\n",
      "Successfully installed Flask-OpenTracing-1.1.0 Flask-cors-3.0.10 PyYAML-5.4.1 click-8.0.4 cryptography-3.4.8 flatbuffers-1.12 grpcio-1.51.1 grpcio-opentracing-1.1.4 grpcio-reflection-1.34.1 gunicorn-20.1.0 jaeger-client-4.4.0 jsonschema-3.2.0 opentracing-2.4.0 prometheus-client-0.8.0 pyrsistent-0.19.2 seldon-core-1.15.0 threadloop-1.0.2 thrift-0.16.0 werkzeug-2.2.2\n"
     ]
    }
   ],
   "source": [
    "### Installation of packages\n",
    "!pip install --upgrade pip\n",
    "!pip install pandas\n",
    "!pip install seldon-core"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "from seldon_core.seldon_client import SeldonClient"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's get started! 🚀🔥"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Train the first MLFlow Elastic Net Wine example\n",
    "\n",
    "For our example, we will use the elastic net wine example from [MLflow's tutorial](https://github.com/mlflow/mlflow/tree/master/examples/sklearn_elasticnet_wine)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "fatal: destination path 'mlflow' already exists and is not an empty directory.\n",
      "MLproject        python_env.yaml  train.py\n",
      "conda.yaml       train.ipynb      wine-quality.csv\n"
     ]
    }
   ],
   "source": [
    "!git clone https://github.com/mlflow/mlflow.git\n",
    "!ls ./mlflow/examples/sklearn_elasticnet_wine"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### MLproject\n",
    "\n",
    "As any other MLflow project, it is defined by its `MLproject` file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[94mname\u001b[39;49;00m:\u001b[37m \u001b[39;49;00mtutorial\u001b[37m\u001b[39;49;00m\n",
      "\u001b[37m\u001b[39;49;00m\n",
      "\u001b[94mpython_env\u001b[39;49;00m:\u001b[37m \u001b[39;49;00mpython_env.yaml\u001b[37m\u001b[39;49;00m\n",
      "\u001b[37m\u001b[39;49;00m\n",
      "\u001b[94mentry_points\u001b[39;49;00m:\u001b[37m\u001b[39;49;00m\n",
      "\u001b[37m  \u001b[39;49;00m\u001b[94mmain\u001b[39;49;00m:\u001b[37m\u001b[39;49;00m\n",
      "\u001b[37m    \u001b[39;49;00m\u001b[94mparameters\u001b[39;49;00m:\u001b[37m\u001b[39;49;00m\n",
      "\u001b[37m      \u001b[39;49;00m\u001b[94malpha\u001b[39;49;00m:\u001b[37m \u001b[39;49;00m{\u001b[94mtype\u001b[39;49;00m:\u001b[37m \u001b[39;49;00m\u001b[31mfloat\u001b[39;49;00m,\u001b[94m default\u001b[39;49;00m:\u001b[37m \u001b[39;49;00m\u001b[31m0.5\u001b[39;49;00m}\u001b[37m\u001b[39;49;00m\n",
      "\u001b[37m      \u001b[39;49;00m\u001b[94ml1_ratio\u001b[39;49;00m:\u001b[37m \u001b[39;49;00m{\u001b[94mtype\u001b[39;49;00m:\u001b[37m \u001b[39;49;00m\u001b[31mfloat\u001b[39;49;00m,\u001b[94m default\u001b[39;49;00m:\u001b[37m \u001b[39;49;00m\u001b[31m0.1\u001b[39;49;00m}\u001b[37m\u001b[39;49;00m\n",
      "\u001b[37m    \u001b[39;49;00m\u001b[94mcommand\u001b[39;49;00m:\u001b[37m \u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m\u001b[33mpython\u001b[39;49;00m\u001b[31m \u001b[39;49;00m\u001b[33mtrain.py\u001b[39;49;00m\u001b[31m \u001b[39;49;00m\u001b[33m{alpha}\u001b[39;49;00m\u001b[31m \u001b[39;49;00m\u001b[33m{l1_ratio}\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m\u001b[37m\u001b[39;49;00m\n"
     ]
    }
   ],
   "source": [
    "PROJECT_DIR='./mlflow/examples/sklearn_elasticnet_wine'\n",
    "!pygmentize -l yaml $PROJECT_DIR/MLproject"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that this project uses Conda for the environment and that it's defined in the `conda.yaml` file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[94mname\u001b[39;49;00m:\u001b[37m \u001b[39;49;00mtutorial\u001b[37m\u001b[39;49;00m\r\n",
      "\u001b[94mchannels\u001b[39;49;00m:\u001b[37m\u001b[39;49;00m\r\n",
      "\u001b[37m  \u001b[39;49;00m-\u001b[37m \u001b[39;49;00mconda-forge\u001b[37m\u001b[39;49;00m\r\n",
      "\u001b[94mdependencies\u001b[39;49;00m:\u001b[37m\u001b[39;49;00m\r\n",
      "\u001b[37m  \u001b[39;49;00m-\u001b[37m \u001b[39;49;00mpython=3.8\u001b[37m\u001b[39;49;00m\r\n",
      "\u001b[37m  \u001b[39;49;00m-\u001b[37m \u001b[39;49;00mpip\u001b[37m\u001b[39;49;00m\r\n",
      "\u001b[37m  \u001b[39;49;00m-\u001b[37m \u001b[39;49;00m\u001b[94mpip\u001b[39;49;00m:\u001b[37m\u001b[39;49;00m\r\n",
      "\u001b[37m      \u001b[39;49;00m-\u001b[37m \u001b[39;49;00mscikit-learn==0.23.2\u001b[37m\u001b[39;49;00m\r\n",
      "\u001b[37m      \u001b[39;49;00m-\u001b[37m \u001b[39;49;00mmlflow>=1.0\u001b[37m\u001b[39;49;00m\r\n",
      "\u001b[37m      \u001b[39;49;00m-\u001b[37m \u001b[39;49;00mpandas\u001b[37m\u001b[39;49;00m\r\n"
     ]
    }
   ],
   "source": [
    "!pygmentize $PROJECT_DIR/conda.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lastly, we can also see that the training will be performed by the `train.py` file, which receives two parameters `alpha` and `l1_ratio`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[37m# The data set used in this example is from http://archive.ics.uci.edu/ml/datasets/Wine+Quality\u001b[39;49;00m\r\n",
      "\u001b[37m# P. Cortez, A. Cerdeira, F. Almeida, T. Matos and J. Reis.\u001b[39;49;00m\r\n",
      "\u001b[37m# Modeling wine preferences by data mining from physicochemical properties. In Decision Support Systems, Elsevier, 47(4):547-553, 2009.\u001b[39;49;00m\r\n",
      "\r\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mos\u001b[39;49;00m\r\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mwarnings\u001b[39;49;00m\r\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36msys\u001b[39;49;00m\r\n",
      "\r\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mpandas\u001b[39;49;00m \u001b[34mas\u001b[39;49;00m \u001b[04m\u001b[36mpd\u001b[39;49;00m\r\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mnumpy\u001b[39;49;00m \u001b[34mas\u001b[39;49;00m \u001b[04m\u001b[36mnp\u001b[39;49;00m\r\n",
      "\u001b[34mfrom\u001b[39;49;00m \u001b[04m\u001b[36msklearn\u001b[39;49;00m\u001b[04m\u001b[36m.\u001b[39;49;00m\u001b[04m\u001b[36mmetrics\u001b[39;49;00m \u001b[34mimport\u001b[39;49;00m mean_squared_error, mean_absolute_error, r2_score\r\n",
      "\u001b[34mfrom\u001b[39;49;00m \u001b[04m\u001b[36msklearn\u001b[39;49;00m\u001b[04m\u001b[36m.\u001b[39;49;00m\u001b[04m\u001b[36mmodel_selection\u001b[39;49;00m \u001b[34mimport\u001b[39;49;00m train_test_split\r\n",
      "\u001b[34mfrom\u001b[39;49;00m \u001b[04m\u001b[36msklearn\u001b[39;49;00m\u001b[04m\u001b[36m.\u001b[39;49;00m\u001b[04m\u001b[36mlinear_model\u001b[39;49;00m \u001b[34mimport\u001b[39;49;00m ElasticNet\r\n",
      "\u001b[34mfrom\u001b[39;49;00m \u001b[04m\u001b[36murllib\u001b[39;49;00m\u001b[04m\u001b[36m.\u001b[39;49;00m\u001b[04m\u001b[36mparse\u001b[39;49;00m \u001b[34mimport\u001b[39;49;00m urlparse\r\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mmlflow\u001b[39;49;00m\r\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mmlflow\u001b[39;49;00m\u001b[04m\u001b[36m.\u001b[39;49;00m\u001b[04m\u001b[36msklearn\u001b[39;49;00m\r\n",
      "\r\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mlogging\u001b[39;49;00m\r\n",
      "\r\n",
      "logging.basicConfig(level=logging.WARN)\r\n",
      "logger = logging.getLogger(\u001b[31m__name__\u001b[39;49;00m)\r\n",
      "\r\n",
      "\r\n",
      "\u001b[34mdef\u001b[39;49;00m \u001b[32meval_metrics\u001b[39;49;00m(actual, pred):\r\n",
      "    rmse = np.sqrt(mean_squared_error(actual, pred))\r\n",
      "    mae = mean_absolute_error(actual, pred)\r\n",
      "    r2 = r2_score(actual, pred)\r\n",
      "    \u001b[34mreturn\u001b[39;49;00m rmse, mae, r2\r\n",
      "\r\n",
      "\r\n",
      "\u001b[34mif\u001b[39;49;00m \u001b[31m__name__\u001b[39;49;00m == \u001b[33m\"\u001b[39;49;00m\u001b[33m__main__\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m:\r\n",
      "    warnings.filterwarnings(\u001b[33m\"\u001b[39;49;00m\u001b[33mignore\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m)\r\n",
      "    np.random.seed(\u001b[34m40\u001b[39;49;00m)\r\n",
      "\r\n",
      "    \u001b[37m# Read the wine-quality csv file from the URL\u001b[39;49;00m\r\n",
      "    csv_url = (\r\n",
      "        \u001b[33m\"\u001b[39;49;00m\u001b[33mhttps://raw.githubusercontent.com/mlflow/mlflow/master/tests/data/winequality-red.csv\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m\r\n",
      "    )\r\n",
      "    \u001b[34mtry\u001b[39;49;00m:\r\n",
      "        data = pd.read_csv(csv_url, sep=\u001b[33m\"\u001b[39;49;00m\u001b[33m;\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m)\r\n",
      "    \u001b[34mexcept\u001b[39;49;00m \u001b[36mException\u001b[39;49;00m \u001b[34mas\u001b[39;49;00m e:\r\n",
      "        logger.exception(\r\n",
      "            \u001b[33m\"\u001b[39;49;00m\u001b[33mUnable to download training & test CSV, check your internet connection. Error: \u001b[39;49;00m\u001b[33m%s\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, e\r\n",
      "        )\r\n",
      "\r\n",
      "    \u001b[37m# Split the data into training and test sets. (0.75, 0.25) split.\u001b[39;49;00m\r\n",
      "    train, test = train_test_split(data)\r\n",
      "\r\n",
      "    \u001b[37m# The predicted column is \"quality\" which is a scalar from [3, 9]\u001b[39;49;00m\r\n",
      "    train_x = train.drop([\u001b[33m\"\u001b[39;49;00m\u001b[33mquality\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m], axis=\u001b[34m1\u001b[39;49;00m)\r\n",
      "    test_x = test.drop([\u001b[33m\"\u001b[39;49;00m\u001b[33mquality\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m], axis=\u001b[34m1\u001b[39;49;00m)\r\n",
      "    train_y = train[[\u001b[33m\"\u001b[39;49;00m\u001b[33mquality\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m]]\r\n",
      "    test_y = test[[\u001b[33m\"\u001b[39;49;00m\u001b[33mquality\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m]]\r\n",
      "\r\n",
      "    alpha = \u001b[36mfloat\u001b[39;49;00m(sys.argv[\u001b[34m1\u001b[39;49;00m]) \u001b[34mif\u001b[39;49;00m \u001b[36mlen\u001b[39;49;00m(sys.argv) > \u001b[34m1\u001b[39;49;00m \u001b[34melse\u001b[39;49;00m \u001b[34m0.5\u001b[39;49;00m\r\n",
      "    l1_ratio = \u001b[36mfloat\u001b[39;49;00m(sys.argv[\u001b[34m2\u001b[39;49;00m]) \u001b[34mif\u001b[39;49;00m \u001b[36mlen\u001b[39;49;00m(sys.argv) > \u001b[34m2\u001b[39;49;00m \u001b[34melse\u001b[39;49;00m \u001b[34m0.5\u001b[39;49;00m\r\n",
      "\r\n",
      "    \u001b[34mwith\u001b[39;49;00m mlflow.start_run():\r\n",
      "        lr = ElasticNet(alpha=alpha, l1_ratio=l1_ratio, random_state=\u001b[34m42\u001b[39;49;00m)\r\n",
      "        lr.fit(train_x, train_y)\r\n",
      "\r\n",
      "        predicted_qualities = lr.predict(test_x)\r\n",
      "\r\n",
      "        (rmse, mae, r2) = eval_metrics(test_y, predicted_qualities)\r\n",
      "\r\n",
      "        \u001b[36mprint\u001b[39;49;00m(\u001b[33m\"\u001b[39;49;00m\u001b[33mElasticnet model (alpha=\u001b[39;49;00m\u001b[33m%f\u001b[39;49;00m\u001b[33m, l1_ratio=\u001b[39;49;00m\u001b[33m%f\u001b[39;49;00m\u001b[33m):\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m % (alpha, l1_ratio))\r\n",
      "        \u001b[36mprint\u001b[39;49;00m(\u001b[33m\"\u001b[39;49;00m\u001b[33m  RMSE: \u001b[39;49;00m\u001b[33m%s\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m % rmse)\r\n",
      "        \u001b[36mprint\u001b[39;49;00m(\u001b[33m\"\u001b[39;49;00m\u001b[33m  MAE: \u001b[39;49;00m\u001b[33m%s\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m % mae)\r\n",
      "        \u001b[36mprint\u001b[39;49;00m(\u001b[33m\"\u001b[39;49;00m\u001b[33m  R2: \u001b[39;49;00m\u001b[33m%s\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m % r2)\r\n",
      "\r\n",
      "        mlflow.log_param(\u001b[33m\"\u001b[39;49;00m\u001b[33malpha\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, alpha)\r\n",
      "        mlflow.log_param(\u001b[33m\"\u001b[39;49;00m\u001b[33ml1_ratio\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, l1_ratio)\r\n",
      "        mlflow.log_metric(\u001b[33m\"\u001b[39;49;00m\u001b[33mrmse\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, rmse)\r\n",
      "        mlflow.log_metric(\u001b[33m\"\u001b[39;49;00m\u001b[33mr2\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, r2)\r\n",
      "        mlflow.log_metric(\u001b[33m\"\u001b[39;49;00m\u001b[33mmae\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, mae)\r\n",
      "\r\n",
      "        tracking_url_type_store = urlparse(mlflow.get_tracking_uri()).scheme\r\n",
      "\r\n",
      "        \u001b[37m# Model registry does not work with file store\u001b[39;49;00m\r\n",
      "        \u001b[34mif\u001b[39;49;00m tracking_url_type_store != \u001b[33m\"\u001b[39;49;00m\u001b[33mfile\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m:\r\n",
      "\r\n",
      "            \u001b[37m# Register the model\u001b[39;49;00m\r\n",
      "            \u001b[37m# There are other ways to use the Model Registry, which depends on the use case,\u001b[39;49;00m\r\n",
      "            \u001b[37m# please refer to the doc for more information:\u001b[39;49;00m\r\n",
      "            \u001b[37m# https://mlflow.org/docs/latest/model-registry.html#api-workflow\u001b[39;49;00m\r\n",
      "            mlflow.sklearn.log_model(lr, \u001b[33m\"\u001b[39;49;00m\u001b[33mmodel\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, registered_model_name=\u001b[33m\"\u001b[39;49;00m\u001b[33mElasticnetWineModel\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m)\r\n",
      "        \u001b[34melse\u001b[39;49;00m:\r\n",
      "            mlflow.sklearn.log_model(lr, \u001b[33m\"\u001b[39;49;00m\u001b[33mmodel\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m)\r\n"
     ]
    }
   ],
   "source": [
    "!pygmentize $PROJECT_DIR/train.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dataset\n",
    "\n",
    "We will use the wine quality dataset.\n",
    "Let's load it to see what's inside:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fixed acidity</th>\n",
       "      <th>volatile acidity</th>\n",
       "      <th>citric acid</th>\n",
       "      <th>residual sugar</th>\n",
       "      <th>chlorides</th>\n",
       "      <th>free sulfur dioxide</th>\n",
       "      <th>total sulfur dioxide</th>\n",
       "      <th>density</th>\n",
       "      <th>pH</th>\n",
       "      <th>sulphates</th>\n",
       "      <th>alcohol</th>\n",
       "      <th>quality</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7.0</td>\n",
       "      <td>0.27</td>\n",
       "      <td>0.36</td>\n",
       "      <td>20.7</td>\n",
       "      <td>0.045</td>\n",
       "      <td>45.0</td>\n",
       "      <td>170.0</td>\n",
       "      <td>1.0010</td>\n",
       "      <td>3.00</td>\n",
       "      <td>0.45</td>\n",
       "      <td>8.8</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>6.3</td>\n",
       "      <td>0.30</td>\n",
       "      <td>0.34</td>\n",
       "      <td>1.6</td>\n",
       "      <td>0.049</td>\n",
       "      <td>14.0</td>\n",
       "      <td>132.0</td>\n",
       "      <td>0.9940</td>\n",
       "      <td>3.30</td>\n",
       "      <td>0.49</td>\n",
       "      <td>9.5</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8.1</td>\n",
       "      <td>0.28</td>\n",
       "      <td>0.40</td>\n",
       "      <td>6.9</td>\n",
       "      <td>0.050</td>\n",
       "      <td>30.0</td>\n",
       "      <td>97.0</td>\n",
       "      <td>0.9951</td>\n",
       "      <td>3.26</td>\n",
       "      <td>0.44</td>\n",
       "      <td>10.1</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>7.2</td>\n",
       "      <td>0.23</td>\n",
       "      <td>0.32</td>\n",
       "      <td>8.5</td>\n",
       "      <td>0.058</td>\n",
       "      <td>47.0</td>\n",
       "      <td>186.0</td>\n",
       "      <td>0.9956</td>\n",
       "      <td>3.19</td>\n",
       "      <td>0.40</td>\n",
       "      <td>9.9</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>7.2</td>\n",
       "      <td>0.23</td>\n",
       "      <td>0.32</td>\n",
       "      <td>8.5</td>\n",
       "      <td>0.058</td>\n",
       "      <td>47.0</td>\n",
       "      <td>186.0</td>\n",
       "      <td>0.9956</td>\n",
       "      <td>3.19</td>\n",
       "      <td>0.40</td>\n",
       "      <td>9.9</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   fixed acidity  volatile acidity  citric acid  residual sugar  chlorides  \\\n",
       "0            7.0              0.27         0.36            20.7      0.045   \n",
       "1            6.3              0.30         0.34             1.6      0.049   \n",
       "2            8.1              0.28         0.40             6.9      0.050   \n",
       "3            7.2              0.23         0.32             8.5      0.058   \n",
       "4            7.2              0.23         0.32             8.5      0.058   \n",
       "\n",
       "   free sulfur dioxide  total sulfur dioxide  density    pH  sulphates  \\\n",
       "0                 45.0                 170.0   1.0010  3.00       0.45   \n",
       "1                 14.0                 132.0   0.9940  3.30       0.49   \n",
       "2                 30.0                  97.0   0.9951  3.26       0.44   \n",
       "3                 47.0                 186.0   0.9956  3.19       0.40   \n",
       "4                 47.0                 186.0   0.9956  3.19       0.40   \n",
       "\n",
       "   alcohol  quality  \n",
       "0      8.8        6  \n",
       "1      9.5        6  \n",
       "2     10.1        6  \n",
       "3      9.9        6  \n",
       "4      9.9        6  "
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filename = \"./mlflow/examples/sklearn_elasticnet_wine/wine-quality.csv\"\n",
    "data = pd.read_csv(filename)\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training\n",
    "\n",
    "We've set up our MLflow project and our dataset is ready, so we are now good to start training.\n",
    "MLflow allows us to train our model with the following command:\n",
    "\n",
    "``` bash\n",
    "$ mlflow run . -P alpha=... -P l1_ratio=...\n",
    "```\n",
    "\n",
    "On each run, `mlflow` will set up the Conda environment defined by the `conda.yaml` file and will run the training commands defined in the `MLproject` file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2022/12/07 04:42:54 INFO mlflow.utils.virtualenv: Installing python 3.8.14 if it does not exist\n",
      "2022/12/07 04:42:54 INFO mlflow.utils.virtualenv: Environment /Users/dileep.gadiraju/.mlflow/envs/mlflow-eee90753fcbd811c61a366651b95e9611177504d already exists\n",
      "2022/12/07 04:42:55 INFO mlflow.projects.utils: === Created directory /var/folders/p9/nndbrbws0j9ghcfk3yw_0smh0000gp/T/tmpqztfkxys for downloading remote URIs passed to arguments of type 'path' ===\n",
      "2022/12/07 04:42:55 INFO mlflow.projects.backend.local: === Running command 'source /Users/dileep.gadiraju/.mlflow/envs/mlflow-eee90753fcbd811c61a366651b95e9611177504d/bin/activate && python train.py 0.5 0.5' in run with ID '80c1581b2caa4ab4b42b2e1d882f2293' === \n",
      "Elasticnet model (alpha=0.500000, l1_ratio=0.500000):\n",
      "  RMSE: 0.7931640229276851\n",
      "  MAE: 0.6271946374319586\n",
      "  R2: 0.10862644997792614\n",
      "2022/12/07 04:43:10 INFO mlflow.projects: === Run (ID '80c1581b2caa4ab4b42b2e1d882f2293') succeeded ===\n"
     ]
    }
   ],
   "source": [
    "# !pip install mlflow\n",
    "!mlflow run $PROJECT_DIR -P alpha=0.5 -P l1_ratio=0.5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each of these commands will create a new run which can be visualised through the MLFlow dashboard as per the screenshot below.\n",
    "\n",
    "![mlflow-dashboard](images/mlflow-dashboard.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each of these models can actually be found on the `mlruns` folder:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[01;34mmlruns/0\u001b[0m\r\n",
      "├── \u001b[01;34m02024ee8d05741b5b0f71938007e3ea7\u001b[0m\r\n",
      "├── \u001b[01;34m5072d00e159e41d08fe3b46eb678a84e\u001b[0m\r\n",
      "├── \u001b[01;34m80c1581b2caa4ab4b42b2e1d882f2293\u001b[0m\r\n",
      "└── \u001b[00mmeta.yaml\u001b[0m\r\n",
      "\r\n",
      "3 directories, 1 file\r\n"
     ]
    }
   ],
   "source": [
    "!tree -L 1 mlruns/0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### MLmodel\n",
    "\n",
    "Inside each of these folders, MLflow stores the parameters we used to train our model, any metric we logged during training, and a snapshot of our model.\n",
    "If we look into one of them, we can see the following structure:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[01;34mmlruns/0/02024ee8d05741b5b0f71938007e3ea7\u001b[0m\r\n",
      "├── \u001b[01;34martifacts\u001b[0m\r\n",
      "│   └── \u001b[01;34mmodel\u001b[0m\r\n",
      "│       ├── \u001b[00mMLmodel\u001b[0m\r\n",
      "│       ├── \u001b[00mconda.yaml\u001b[0m\r\n",
      "│       ├── \u001b[00mmodel.pkl\u001b[0m\r\n",
      "│       ├── \u001b[00mpython_env.yaml\u001b[0m\r\n",
      "│       └── \u001b[00mrequirements.txt\u001b[0m\r\n",
      "├── \u001b[00mmeta.yaml\u001b[0m\r\n",
      "├── \u001b[01;34mmetrics\u001b[0m\r\n",
      "│   ├── \u001b[00mmae\u001b[0m\r\n",
      "│   ├── \u001b[00mr2\u001b[0m\r\n",
      "│   └── \u001b[00mrmse\u001b[0m\r\n",
      "├── \u001b[01;34mparams\u001b[0m\r\n",
      "│   ├── \u001b[00malpha\u001b[0m\r\n",
      "│   └── \u001b[00ml1_ratio\u001b[0m\r\n",
      "└── \u001b[01;34mtags\u001b[0m\r\n",
      "    ├── \u001b[00mmlflow.gitRepoURL\u001b[0m\r\n",
      "    ├── \u001b[00mmlflow.log-model.history\u001b[0m\r\n",
      "    ├── \u001b[00mmlflow.project.backend\u001b[0m\r\n",
      "    ├── \u001b[00mmlflow.project.entryPoint\u001b[0m\r\n",
      "    ├── \u001b[00mmlflow.project.env\u001b[0m\r\n",
      "    ├── \u001b[00mmlflow.runName\u001b[0m\r\n",
      "    ├── \u001b[00mmlflow.source.git.commit\u001b[0m\r\n",
      "    ├── \u001b[00mmlflow.source.git.repoURL\u001b[0m\r\n",
      "    ├── \u001b[00mmlflow.source.name\u001b[0m\r\n",
      "    ├── \u001b[00mmlflow.source.type\u001b[0m\r\n",
      "    └── \u001b[00mmlflow.user\u001b[0m\r\n",
      "\r\n",
      "5 directories, 22 files\r\n"
     ]
    }
   ],
   "source": [
    "!tree mlruns/0/$(ls mlruns/0 | head -1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In particular, we are interested in the `MLmodel` file stored under `artifacts/model`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[94martifact_path\u001b[39;49;00m:\u001b[37m \u001b[39;49;00mmodel\u001b[37m\u001b[39;49;00m\r\n",
      "\u001b[94mflavors\u001b[39;49;00m:\u001b[37m\u001b[39;49;00m\r\n",
      "\u001b[37m  \u001b[39;49;00m\u001b[94mpython_function\u001b[39;49;00m:\u001b[37m\u001b[39;49;00m\r\n",
      "\u001b[37m    \u001b[39;49;00m\u001b[94menv\u001b[39;49;00m:\u001b[37m\u001b[39;49;00m\r\n",
      "\u001b[37m      \u001b[39;49;00m\u001b[94mconda\u001b[39;49;00m:\u001b[37m \u001b[39;49;00mconda.yaml\u001b[37m\u001b[39;49;00m\r\n",
      "\u001b[37m      \u001b[39;49;00m\u001b[94mvirtualenv\u001b[39;49;00m:\u001b[37m \u001b[39;49;00mpython_env.yaml\u001b[37m\u001b[39;49;00m\r\n",
      "\u001b[37m    \u001b[39;49;00m\u001b[94mloader_module\u001b[39;49;00m:\u001b[37m \u001b[39;49;00mmlflow.sklearn\u001b[37m\u001b[39;49;00m\r\n",
      "\u001b[37m    \u001b[39;49;00m\u001b[94mmodel_path\u001b[39;49;00m:\u001b[37m \u001b[39;49;00mmodel.pkl\u001b[37m\u001b[39;49;00m\r\n",
      "\u001b[37m    \u001b[39;49;00m\u001b[94mpredict_fn\u001b[39;49;00m:\u001b[37m \u001b[39;49;00mpredict\u001b[37m\u001b[39;49;00m\r\n",
      "\u001b[37m    \u001b[39;49;00m\u001b[94mpython_version\u001b[39;49;00m:\u001b[37m \u001b[39;49;00m3.8.14\u001b[37m\u001b[39;49;00m\r\n",
      "\u001b[37m  \u001b[39;49;00m\u001b[94msklearn\u001b[39;49;00m:\u001b[37m\u001b[39;49;00m\r\n",
      "\u001b[37m    \u001b[39;49;00m\u001b[94mcode\u001b[39;49;00m:\u001b[37m \u001b[39;49;00mnull\u001b[37m\u001b[39;49;00m\r\n",
      "\u001b[37m    \u001b[39;49;00m\u001b[94mpickled_model\u001b[39;49;00m:\u001b[37m \u001b[39;49;00mmodel.pkl\u001b[37m\u001b[39;49;00m\r\n",
      "\u001b[37m    \u001b[39;49;00m\u001b[94mserialization_format\u001b[39;49;00m:\u001b[37m \u001b[39;49;00mcloudpickle\u001b[37m\u001b[39;49;00m\r\n",
      "\u001b[37m    \u001b[39;49;00m\u001b[94msklearn_version\u001b[39;49;00m:\u001b[37m \u001b[39;49;00m0.23.2\u001b[37m\u001b[39;49;00m\r\n",
      "\u001b[94mmlflow_version\u001b[39;49;00m:\u001b[37m \u001b[39;49;00m2.0.1\u001b[37m\u001b[39;49;00m\r\n",
      "\u001b[94mmodel_uuid\u001b[39;49;00m:\u001b[37m \u001b[39;49;00mb7871a2b059b4133ace52866af689203\u001b[37m\u001b[39;49;00m\r\n",
      "\u001b[94mrun_id\u001b[39;49;00m:\u001b[37m \u001b[39;49;00m02024ee8d05741b5b0f71938007e3ea7\u001b[37m\u001b[39;49;00m\r\n",
      "\u001b[94mutc_time_created\u001b[39;49;00m:\u001b[37m \u001b[39;49;00m\u001b[33m'\u001b[39;49;00m\u001b[33m2022-12-06\u001b[39;49;00m\u001b[31m \u001b[39;49;00m\u001b[33m20:57:54.085772\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m\u001b[37m\u001b[39;49;00m\r\n"
     ]
    }
   ],
   "source": [
    "!pygmentize -l yaml mlruns/0/$(ls mlruns/0 | head -1)/artifacts/model/MLmodel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This file stores the details of how the model was stored.\n",
    "With this information (plus the other files in the folder), we are able to load the model back.\n",
    "Seldon's MLflow server will use this information to serve this model.\n",
    "\n",
    "Now we should upload our newly trained model into a public Google Bucket or S3 bucket.\n",
    "We have already done this to make it simpler, which you will be able to find at `gs://seldon-models/mlflow/model-a`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Deploy your model using the Pre-packaged Moldel Server for MLFlow\n",
    "\n",
    "Now we can deploy our trained MLFlow model.\n",
    "\n",
    "For this we have to create a Seldon definition of the model server definition, which we will break down further below.\n",
    "\n",
    "We will be using the model we updated to our google bucket (gs://seldon-models/mlflow/elasticnet_wine_1.8.0), but you can use your model if you uploaded it to a public bucket."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setup Seldon Core\n",
    "\n",
    "Use the setup notebook to [Setup Cluster](https://docs.seldon.io/projects/seldon-core/en/latest/examples/seldon_core_setup.html#Setup-Cluster) with [Ambassador Ingress](https://docs.seldon.io/projects/seldon-core/en/latest/examples/seldon_core_setup.html#Ambassador) and [Install Seldon Core](https://docs.seldon.io/projects/seldon-core/en/latest/examples/seldon_core_setup.html#Install-Seldon-Core). Instructions [also online](https://docs.seldon.io/projects/seldon-core/en/latest/examples/seldon_core_setup.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[04m\u001b[36m---\u001b[39;49;00m\u001b[37m\u001b[39;49;00m\r\n",
      "\u001b[94mapiVersion\u001b[39;49;00m:\u001b[37m \u001b[39;49;00mmachinelearning.seldon.io/v1alpha2\u001b[37m\u001b[39;49;00m\r\n",
      "\u001b[94mkind\u001b[39;49;00m:\u001b[37m \u001b[39;49;00mSeldonDeployment\u001b[37m\u001b[39;49;00m\r\n",
      "\u001b[94mmetadata\u001b[39;49;00m:\u001b[37m\u001b[39;49;00m\r\n",
      "\u001b[37m  \u001b[39;49;00m\u001b[94mname\u001b[39;49;00m:\u001b[37m \u001b[39;49;00mmlflow-deployment\u001b[37m\u001b[39;49;00m\r\n",
      "\u001b[94mspec\u001b[39;49;00m:\u001b[37m\u001b[39;49;00m\r\n",
      "\u001b[37m  \u001b[39;49;00m\u001b[94mname\u001b[39;49;00m:\u001b[37m \u001b[39;49;00mmlflow-deployment\u001b[37m\u001b[39;49;00m\r\n",
      "\u001b[37m  \u001b[39;49;00m\u001b[94mpredictors\u001b[39;49;00m:\u001b[37m\u001b[39;49;00m\r\n",
      "\u001b[37m    \u001b[39;49;00m-\u001b[37m \u001b[39;49;00m\u001b[94mgraph\u001b[39;49;00m:\u001b[37m\u001b[39;49;00m\r\n",
      "\u001b[37m        \u001b[39;49;00m\u001b[94mchildren\u001b[39;49;00m:\u001b[37m \u001b[39;49;00m[]\u001b[37m\u001b[39;49;00m\r\n",
      "\u001b[37m        \u001b[39;49;00m\u001b[94mimplementation\u001b[39;49;00m:\u001b[37m \u001b[39;49;00mMLFLOW_SERVER\u001b[37m\u001b[39;49;00m\r\n",
      "\u001b[37m        \u001b[39;49;00m\u001b[94mmodelUri\u001b[39;49;00m:\u001b[37m \u001b[39;49;00mgs://seldon-models/mlflow/model-a\u001b[37m\u001b[39;49;00m\r\n",
      "\u001b[37m        \u001b[39;49;00m\u001b[94mname\u001b[39;49;00m:\u001b[37m \u001b[39;49;00mwines-classifier\u001b[37m\u001b[39;49;00m\r\n",
      "\u001b[37m      \u001b[39;49;00m\u001b[94mname\u001b[39;49;00m:\u001b[37m \u001b[39;49;00mmlflow-deployment-dag\u001b[37m\u001b[39;49;00m\r\n",
      "\u001b[37m      \u001b[39;49;00m\u001b[94mreplicas\u001b[39;49;00m:\u001b[37m \u001b[39;49;00m1\u001b[37m\u001b[39;49;00m\r\n"
     ]
    }
   ],
   "source": [
    "!pygmentize ./seldon-core/examples/models/mlflow_server_ab_test_ambassador/mlflow-model-server-seldon-config.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Error: INSTALLATION FAILED: cannot re-use a name that is still in use\r\n"
     ]
    }
   ],
   "source": [
    "### Install Ambassador Edge Stack\n",
    "!helm install ambassador datawire/ambassador \\\n",
    "    --set image.repository=docker.io/datawire/ambassador \\\n",
    "    --set crds.keep=false \\\n",
    "    --namespace seldon-system"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we write our configuration file, we are able to deploy it to our cluster by running it with our command"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---\n",
      "apiVersion: machinelearning.seldon.io/v1alpha2\n",
      "kind: SeldonDeployment\n",
      "metadata:\n",
      "  name: mlflow-deployment\n",
      "spec:\n",
      "  name: mlflow-deployment\n",
      "  predictors:\n",
      "    - graph:\n",
      "        children: []\n",
      "        implementation: MLFLOW_SERVER\n",
      "        modelUri: gs://seldon-models/mlflow/model-a\n",
      "        name: wines-classifier\n",
      "      name: mlflow-deployment-dag\n",
      "      replicas: 1\n",
      "Error from server (NotFound): error when deleting \"./seldon-core/examples/models/mlflow_server_ab_test_ambassador/mlflow-model-server-seldon-config.yaml\": seldondeployments.machinelearning.seldon.io \"mlflow-deployment\" not found\n",
      "Error from server (InternalError): error when creating \"./seldon-core/examples/models/mlflow_server_ab_test_ambassador/mlflow-model-server-seldon-config.yaml\": Internal error occurred: failed calling webhook \"v1.vseldondeployment.kb.io\": Post \"https://seldon-webhook-service.seldon-system.svc:443/validate-machinelearning-seldon-io-v1-seldondeployment?timeout=10s\": dial tcp 10.96.28.175:443: connect: connection refused\n"
     ]
    }
   ],
   "source": [
    "!cat ./seldon-core/examples/models/mlflow_server_ab_test_ambassador/mlflow-model-server-seldon-config.yaml\n",
    "!kubectl delete -f ./seldon-core/examples/models/mlflow_server_ab_test_ambassador/mlflow-model-server-seldon-config.yaml\n",
    "!kubectl apply -f ./seldon-core/examples/models/mlflow_server_ab_test_ambassador/mlflow-model-server-seldon-config.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once it's created we just wait until it's deployed. \n",
    "\n",
    "It will basically download the image for the pre-packaged MLFlow model server, and initialise it with the model we specified above.\n",
    "\n",
    "You can check the status of the deployment with the following command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Waiting for deployment \"mlflow-deployment-mlflow-deployment-dag-0-wines-classifier\" rollout to finish: 0 of 1 updated replicas are available...\n",
      "^C\n"
     ]
    }
   ],
   "source": [
    "!kubectl rollout status deployment.apps/mlflow-deployment-mlflow-deployment-dag-0-wines-classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once it's deployed, we should see a \"succcessfully rolled out\" message above. We can now test it!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Test the deployed MLFlow model by sending requests\n",
    "Now that our model is deployed in Kubernetes, we are able to send any requests."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will first need the URL that is currently available through Ambassador. \n",
    "\n",
    "If you are running this locally, you should be able to reach it through localhost, in this case we can use port 80."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ambassador               LoadBalancer   10.96.209.201   <pending>     80:30785/TCP,443:31534/TCP   28m\r\n",
      "ambassador-admin         ClusterIP      10.96.11.228    <none>        8877/TCP,8005/TCP            28m\r\n",
      "ambassador-redis         ClusterIP      10.96.152.118   <none>        6379/TCP                     28m\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl get svc -n seldon-system | grep ambassador"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will select the first datapoint in our dataset to send to the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[7.0, 0.27, 0.36, 20.7, 0.045, 45.0, 170.0, 1.001, 3.0, 0.45, 8.8]\n"
     ]
    }
   ],
   "source": [
    "x_0 = data.drop([\"quality\"], axis=1).values[:1]\n",
    "print(list(x_0[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can try sending a request first using curl:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Ensure gateway port forwarded using \n",
    "## kubectl port-forward $(kubectl get pods -l istio=ingressgateway -n istio-system -o jsonpath='{.items[0].metadata.name}') -n istio-system 8004:8080"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [],
   "source": [
    "!curl -X POST -H 'Content-Type: application/json' \\\n",
    "    -d '{\"data\": {\"names\": [], \"ndarray\": [[7.0, 0.27, 0.36, 20.7, 0.045, 45.0, 170.0, 1.001, 3.0, 0.45, 8.8]]}}' \\\n",
    "    http://localhost:8004/seldon/seldon/mlflow-deployment/api/v0.1/predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also send the request by using our python client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import math\n",
    "import subprocess\n",
    "\n",
    "import numpy as np\n",
    "\n",
    "from seldon_core.seldon_client import SeldonClient\n",
    "\n",
    "HOST = \"localhost\"  # Add the URL you found above\n",
    "port = \"80\"  # Make sure you use the port above\n",
    "batch = x_0\n",
    "payload_type = \"ndarray\"\n",
    "\n",
    "sc = SeldonClient(\n",
    "    gateway=\"ambassador\", namespace=\"seldon\", gateway_endpoint=HOST + \":\" + port\n",
    ")\n",
    "\n",
    "client_prediction = sc.predict(\n",
    "    data=batch, deployment_name=\"mlflow-deployment\", names=[], payload_type=payload_type\n",
    ")\n",
    "\n",
    "print(client_prediction.response)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Deploy your second model as an A/B test\n",
    "\n",
    "Now that we have a model in production, it's possible to deploy a second model as an A/B test.\n",
    "Our model will also be an Elastic Net model but using a different set of parameters.\n",
    "We can easily train it by leveraging MLflow:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!mlflow run $PROJECT_DIR -P alpha=0.75 -P l1_ratio=0.2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we did before, we will now need to upload our model to a cloud bucket.\n",
    "To speed things up, we already have done so and the second model is now accessible in `gs://seldon-models/mlflow/model-b`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A/B test\n",
    "\n",
    "We will deploy our second model as an A/B test.\n",
    "In particular, we will redirect 20% of the traffic to the new model.\n",
    "\n",
    "This can be done by simply adding a `traffic` attribute on our `SeldonDeployment` spec:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pygmentize ab-test-mlflow-model-server-seldon-config.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And similar to the model above, we only need to run the following to deploy it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl apply -f ab-test-mlflow-model-server-seldon-config.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can check that the models have been deployed and are running with the following command.\n",
    "\n",
    "We should now see the \"a-\" model and the \"b-\" models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl get pods"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Visualise and monitor the performance of your models using Seldon Analytics\n",
    "\n",
    "This section is optional, but by following the instructions you will be able to visualise the performance of both models as per the chart below.\n",
    "\n",
    "In order for this example to work you need to install and run the [Grafana Analytics package for Seldon Core](https://docs.seldon.io/projects/seldon-core/en/latest/analytics/analytics.html#helm-analytics-chart).\n",
    "\n",
    "For this we can access the URL with the command below, it will request an admin and password which by default are set to the following:\n",
    "* Username: admin\n",
    "* Password: password\n",
    "\n",
    "You can access the grafana dashboard through the port provided below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl get svc grafana-prom -o jsonpath='{.spec.ports[0].nodePort}'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have both models running in our Kubernetes cluster, we can analyse their performance using Seldon Core's integration with Prometheus and Grafana.\n",
    "To do so, we will iterate over the training set (which can be found in `wine-quality.csv`), making a request and sending the feedback of the prediction.\n",
    "\n",
    "Since the `/feedback` endpoint requires a `reward` signal (i.e. the higher the better), we will simulate one as:\n",
    "\n",
    "$$\n",
    "  R(x_{n})\n",
    "    = \\begin{cases}\n",
    "        \\frac{1}{(y_{n} - f(x_{n}))^{2}} &, y_{n} \\neq f(x_{n}) \\\\\n",
    "        500 &, y_{n} = f(x_{n})\n",
    "      \\end{cases}\n",
    "$$\n",
    "\n",
    ", where $R(x_{n})$ is the reward for input point $x_{n}$, $f(x_{n})$ is our trained model and $y_{n}$ is the actual value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sc = SeldonClient(\n",
    "    gateway=\"ambassador\", namespace=\"seldon\", deployment_name=\"wines-classifier\"\n",
    ")\n",
    "\n",
    "\n",
    "def _get_reward(y, y_pred):\n",
    "    if y == y_pred:\n",
    "        return 500\n",
    "\n",
    "    return 1 / np.square(y - y_pred)\n",
    "\n",
    "\n",
    "def _test_row(row):\n",
    "    input_features = row[:-1]\n",
    "    feature_names = input_features.index.to_list()\n",
    "    X = input_features.values.reshape(1, -1)\n",
    "    y = row[-1].reshape(1, -1)\n",
    "\n",
    "    # Note that we are re-using the SeldonClient defined previously\n",
    "    r = sc.predict(deployment_name=\"mlflow-deployment\", data=X, names=feature_names)\n",
    "\n",
    "    y_pred = r.response[\"data\"][\"tensor\"][\"values\"]\n",
    "    reward = _get_reward(y, y_pred)\n",
    "    sc.feedback(\n",
    "        deployment_name=\"mlflow-deployment\",\n",
    "        prediction_request=r.request,\n",
    "        prediction_response=r.response,\n",
    "        reward=reward,\n",
    "    )\n",
    "\n",
    "    return reward[0]\n",
    "\n",
    "\n",
    "data.apply(_test_row, axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should now be able to see Seldon's pre-built Grafana dashboard."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![grafana-mlflow](images/grafana-mlflow.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In bottom of the dashboard you can see the following charts: \n",
    "\n",
    "- On the left: the requests per second, which shows the different traffic breakdown we specified.\n",
    "- On the center: the reward, where you can see how model `a` outperforms model `b` by a large margin.\n",
    "- On the right, the latency for each one of them.\n",
    "\n",
    "You are able to add your own custom metrics, and try out other more complex deployments by following further guides at https://docs.seldon.io/projects/seldon-core/en/latest/examples/mlflow_server_ab_test_ambassador.html"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9.12 ('base')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12 (main, Apr  5 2022, 01:53:17) \n[Clang 12.0.0 ]"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  },
  "vscode": {
   "interpreter": {
    "hash": "3c5b05c738269f3db37ab2b93400d252a3eceb4ebe547b00838e0efaaa2ac1dc"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}