ReadToMe Model Training.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## SSD ReadToMe Model Training for DeepLens\n",
    "\n",
    "#### You can run the following steps to train the model for the ReadToMeProject or you can replace the custom data with your own custom data set and train your own object detection model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### First, lets install some dependancies. I am running Cuda 9.1 on my system so I am grabbing the MXNet version which is built for Cuda 9.1, you may need to adjust accordinly depending on what is installed on your machine."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied (use --upgrade to upgrade): mxnet-cu91 in /home/aschu/development/read-to-me-model/readToMe/lib/python3.5/site-packages\n",
      "Requirement already satisfied (use --upgrade to upgrade): numpy<1.15.0,>=1.8.2 in /home/aschu/development/read-to-me-model/readToMe/lib/python3.5/site-packages (from mxnet-cu91)\n",
      "Requirement already satisfied (use --upgrade to upgrade): requests<2.19.0,>=2.18.4 in /home/aschu/development/read-to-me-model/readToMe/lib/python3.5/site-packages (from mxnet-cu91)\n",
      "Requirement already satisfied (use --upgrade to upgrade): graphviz<0.9.0,>=0.8.1 in /home/aschu/development/read-to-me-model/readToMe/lib/python3.5/site-packages (from mxnet-cu91)\n",
      "Requirement already satisfied (use --upgrade to upgrade): chardet<3.1.0,>=3.0.2 in /home/aschu/development/read-to-me-model/readToMe/lib/python3.5/site-packages (from requests<2.19.0,>=2.18.4->mxnet-cu91)\n",
      "Requirement already satisfied (use --upgrade to upgrade): certifi>=2017.4.17 in /home/aschu/development/read-to-me-model/readToMe/lib/python3.5/site-packages (from requests<2.19.0,>=2.18.4->mxnet-cu91)\n",
      "Requirement already satisfied (use --upgrade to upgrade): urllib3<1.23,>=1.21.1 in /home/aschu/development/read-to-me-model/readToMe/lib/python3.5/site-packages (from requests<2.19.0,>=2.18.4->mxnet-cu91)\n",
      "Requirement already satisfied (use --upgrade to upgrade): idna<2.7,>=2.5 in /home/aschu/development/read-to-me-model/readToMe/lib/python3.5/site-packages (from requests<2.19.0,>=2.18.4->mxnet-cu91)\n",
      "Requirement already satisfied (use --upgrade to upgrade): numpy in /home/aschu/development/read-to-me-model/readToMe/lib/python3.5/site-packages\n",
      "Requirement already satisfied (use --upgrade to upgrade): opencv-python in /home/aschu/development/read-to-me-model/readToMe/lib/python3.5/site-packages\n",
      "Requirement already satisfied (use --upgrade to upgrade): numpy>=1.11.1 in /home/aschu/development/read-to-me-model/readToMe/lib/python3.5/site-packages (from opencv-python)\n",
      "Requirement already satisfied (use --upgrade to upgrade): matplotlib in /home/aschu/development/read-to-me-model/readToMe/lib/python3.5/site-packages\n",
      "Requirement already satisfied (use --upgrade to upgrade): pytz in /home/aschu/development/read-to-me-model/readToMe/lib/python3.5/site-packages (from matplotlib)\n",
      "Requirement already satisfied (use --upgrade to upgrade): python-dateutil>=2.1 in /home/aschu/development/read-to-me-model/readToMe/lib/python3.5/site-packages (from matplotlib)\n",
      "Requirement already satisfied (use --upgrade to upgrade): cycler>=0.10 in /home/aschu/development/read-to-me-model/readToMe/lib/python3.5/site-packages (from matplotlib)\n",
      "Requirement already satisfied (use --upgrade to upgrade): numpy>=1.7.1 in /home/aschu/development/read-to-me-model/readToMe/lib/python3.5/site-packages (from matplotlib)\n",
      "Requirement already satisfied (use --upgrade to upgrade): kiwisolver>=1.0.1 in /home/aschu/development/read-to-me-model/readToMe/lib/python3.5/site-packages (from matplotlib)\n",
      "Requirement already satisfied (use --upgrade to upgrade): six>=1.10 in /home/aschu/development/read-to-me-model/readToMe/lib/python3.5/site-packages (from matplotlib)\n",
      "Requirement already satisfied (use --upgrade to upgrade): pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /home/aschu/development/read-to-me-model/readToMe/lib/python3.5/site-packages (from matplotlib)\n",
      "Requirement already satisfied (use --upgrade to upgrade): setuptools in /home/aschu/development/read-to-me-model/readToMe/lib/python3.5/site-packages (from kiwisolver>=1.0.1->matplotlib)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "You are using pip version 8.1.1, however version 10.0.1 is available.\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\n",
      "You are using pip version 8.1.1, however version 10.0.1 is available.\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\n",
      "You are using pip version 8.1.1, however version 10.0.1 is available.\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\n",
      "You are using pip version 8.1.1, however version 10.0.1 is available.\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "pip install mxnet-cu91\n",
    "pip install numpy\n",
    "pip install opencv-python\n",
    "pip install matplotlib"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Next, we need to grab the MXNet repo from Github"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "checking for incubator-mxnet\n",
      "found existing git repo\n",
      "deleting incubator-mxnet.git\n",
      "Submodule '3rdparty/cub' (https://github.com/dmlc/cub) registered for path '3rdparty/cub'\n",
      "Submodule '3rdparty/dlpack' (https://github.com/dmlc/dlpack) registered for path '3rdparty/dlpack'\n",
      "Submodule '3rdparty/dmlc-core' (https://github.com/dmlc/dmlc-core.git) registered for path '3rdparty/dmlc-core'\n",
      "Submodule '3rdparty/googletest' (https://github.com/google/googletest.git) registered for path '3rdparty/googletest'\n",
      "Submodule '3rdparty/mkldnn' (https://github.com/intel/mkl-dnn.git) registered for path '3rdparty/mkldnn'\n",
      "Submodule '3rdparty/mshadow' (https://github.com/dmlc/mshadow.git) registered for path '3rdparty/mshadow'\n",
      "Submodule '3rdparty/nnvm' (https://github.com/dmlc/nnvm) registered for path '3rdparty/nnvm'\n",
      "Submodule '3rdparty/openmp' (https://github.com/llvm-mirror/openmp) registered for path '3rdparty/openmp'\n",
      "Submodule '3rdparty/ps-lite' (https://github.com/dmlc/ps-lite) registered for path '3rdparty/ps-lite'\n",
      "Submodule path '3rdparty/cub': checked out '05eb57faa0a4cac37c2a86fdf4b4dc865a95a1a3'\n",
      "Submodule path '3rdparty/dlpack': checked out '10892ac964f1af7c81aae145cd3fab78bbccd297'\n",
      "Submodule path '3rdparty/dmlc-core': checked out 'e9446f5a53cf5e61273deff7ce814093d2791766'\n",
      "Submodule path '3rdparty/googletest': checked out 'ec44c6c1675c25b9827aacd08c02433cccde7780'\n",
      "Submodule path '3rdparty/mkldnn': checked out 'b4137dfc88e3bf5c6b62e833121802eb8c6696da'\n",
      "Submodule path '3rdparty/mshadow': checked out '0b4cedd7015cc69191f8338a8feaacda90697758'\n",
      "Submodule path '3rdparty/nnvm': checked out '2bc5144cd3733fd239287e3560c7db8285d21f02'\n",
      "Submodule 'dmlc-core' (https://github.com/dmlc/dmlc-core) registered for path 'dmlc-core'\n",
      "Submodule 'tvm' (https://github.com/dmlc/tvm) registered for path 'tvm'\n",
      "Submodule path '3rdparty/nnvm/dmlc-core': checked out '42823a731bdb2c22aa44775c0937466046400c02'\n",
      "Submodule path '3rdparty/nnvm/tvm': checked out 'fdba6cc9bd3bec9ccd0592fa3900b7fe25d6cb97'\n",
      "Submodule 'HalideIR' (https://github.com/dmlc/HalideIR) registered for path 'HalideIR'\n",
      "Submodule 'dlpack' (https://github.com/dmlc/dlpack) registered for path 'dlpack'\n",
      "Submodule 'dmlc-core' (https://github.com/dmlc/dmlc-core) registered for path 'dmlc-core'\n",
      "Submodule path '3rdparty/nnvm/tvm/HalideIR': checked out 'e20e5e9abb3aa43147a90a4ffb3e190f62862970'\n",
      "Submodule path '3rdparty/nnvm/tvm/dlpack': checked out '10892ac964f1af7c81aae145cd3fab78bbccd297'\n",
      "Submodule path '3rdparty/nnvm/tvm/dmlc-core': checked out 'd3f7fbb53e5b037c0f5bf6bd21871ccc720690cc'\n",
      "Submodule path '3rdparty/openmp': checked out '37c72127e90360a020f351f18d9cccfc30e5145a'\n",
      "Submodule path '3rdparty/ps-lite': checked out 'a6dda54604a07d1fb21b016ed1e3f4246b08222a'\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Cloning into 'incubator-mxnet'...\n",
      "Cloning into '3rdparty/cub'...\n",
      "Cloning into '3rdparty/dlpack'...\n",
      "Cloning into '3rdparty/dmlc-core'...\n",
      "Cloning into '3rdparty/googletest'...\n",
      "Cloning into '3rdparty/mkldnn'...\n",
      "Cloning into '3rdparty/mshadow'...\n",
      "Cloning into '3rdparty/nnvm'...\n",
      "Cloning into 'dmlc-core'...\n",
      "Cloning into 'tvm'...\n",
      "Cloning into 'HalideIR'...\n",
      "Cloning into 'dlpack'...\n",
      "Cloning into 'dmlc-core'...\n",
      "Cloning into '3rdparty/openmp'...\n",
      "Cloning into '3rdparty/ps-lite'...\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "echo checking for incubator-mxnet\n",
    "DIR=incubator-mxnet\n",
    "if [[ -d $DIR ]]; then\n",
    "    echo found existing git repo\n",
    "    echo deleting incubator-mxnet.git\n",
    "    rm -rf incubator-mxnet\n",
    "fi\n",
    "\n",
    "git clone --recursive https://github.com/apache/incubator-mxnet.git"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "#### This is where we deviate from the example instructions on Github. The example tells us to grab a model from this [list](https://github.com/apache/incubator-mxnet/tree/master/example/ssd#map) however, these models do not work with the current version of MXNet, at least not without modifying the symbol names. So lets grab a pretrained model that does work.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "rm -rf incubator-mxnet/example/ssd/model/*\n",
    "cd incubator-mxnet/example/ssd/model/\n",
    "wget https://github.com/zhreshold/mxnet-ssd/releases/download/v0.2-beta/vgg16_reduced.zip\n",
    "unzip vgg16_reduced.zip\n",
    "rm vgg16_reduced.zip\n",
    "mv vgg16_reduced-symbol.json ssd_vgg16_reduced_300-symbol.json\n",
    "mv vgg16_reduced-0001.params ssd_vgg16_reduced_300-0001.params"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "#### Now we need to organize our data into directories so that the example scripts will work. You can read more about this structure by Googling Pascal VOC. There are other Pascal VOC datasets available online that you can use to train models with. If you plan to train a model on your own dataset, you should first check to see if someone else has already made it available online.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import zipfile\n",
    "import shutil\n",
    "from pathlib import Path\n",
    "import random\n",
    "\n",
    "    \n",
    "# # remove training dir if exists\n",
    "if os.path.exists(\"incubator-mxnet/example/ssd/data/VOCdevkit/VOC2018\"):\n",
    "    shutil.rmtree('incubator-mxnet/example/ssd/data/VOCdevkit/VOC2018')\n",
    "\n",
    "# # text-block-custom dataset\n",
    "os.makedirs('incubator-mxnet/example/ssd/data/VOCdevkit/VOC2018/JPEGImages')\n",
    "os.makedirs('incubator-mxnet/example/ssd/data/VOCdevkit/VOC2018/Annotations')\n",
    "\n",
    "destination_dir = 'incubator-mxnet/example/ssd/data/VOCdevkit/VOC2018/'\n",
    "\n",
    "\n",
    "for file in os.listdir('2018'):\n",
    "    if file.endswith('.jpg'):\n",
    "        outfile_path = destination_dir + 'JPEGImages/' + file\n",
    "        shutil.copy(os.path.join('2018', file), outfile_path)\n",
    "    elif file.endswith('.xml'):\n",
    "        outfile_path = destination_dir + 'Annotations/' + file\n",
    "        shutil.copy(os.path.join('2018', file), outfile_path)\n",
    "\n",
    "files = []\n",
    "for filename in os.listdir('incubator-mxnet/example/ssd/data/VOCdevkit/VOC2018/JPEGImages/'):\n",
    "    if filename.endswith('.jpg'):\n",
    "        files.append('{0}'.format(Path(filename).stem))\n",
    "\n",
    "# # Take 10% of the data and use it for validation, the rest goes to training\n",
    "training = []\n",
    "validation = []\n",
    "validationPercent = 10\n",
    "k = int(len(files) * validationPercent // 100)\n",
    "indices = random.sample(range(len(files)), k)\n",
    "for index, file in enumerate(files):\n",
    "    if index not in indices:\n",
    "        training.append(file)\n",
    "    else:\n",
    "        validation.append(file)\n",
    "\n",
    "\n",
    "print(len(files))\n",
    "print(len(training))\n",
    "print(len(validation))\n",
    "\n",
    "os.makedirs('incubator-mxnet/example/ssd/data/VOCdevkit/VOC2018/ImageSets/Main/')\n",
    "with open('incubator-mxnet/example/ssd/data/VOCdevkit/VOC2018/ImageSets/Main/trainval.txt', 'w') as training_list:\n",
    "    for _, row in enumerate(training):\n",
    "        training_list.write('{}\\n'.format(row))\n",
    "\n",
    "with open('incubator-mxnet/example/ssd/data/VOCdevkit/VOC2018/ImageSets/Main/test.txt', 'w') as validation_list:\n",
    "    for _, row in enumerate(validation):\n",
    "        validation_list.write('{}\\n'.format(row))\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### It is reccomended that we turn our dataset into .rec files so that MXNet can iterate over the data much more efficiently. \n",
    "\n",
    "#### This next step calls a wrapper script which altimately calls im2rec.py. This generates a \".rec\" file for our training dataset and one for our validation dataset.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "# change directories to the example/ssd directory\n",
    "cd incubator-mxnet/example/ssd\n",
    "# update the names list to only include our single class name 'text_block'\n",
    "echo text_block > dataset/names/pascal_voc.names\n",
    "\n",
    "# generate the .rec files we will use to train with\n",
    "python tools/prepare_dataset.py --dataset pascal --year 2018 --set test --target ./data/test.lst --root data/VOCdevkit/\n",
    "python tools/prepare_dataset.py --dataset pascal --year 2018 --set trainval --target ./data/train.lst --root data/VOCdevkit/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Finally we can call train on our data. This will run for 250 epochs by default, but you can change that by passing in the \"--end-epoch\" flag and specifying an epoch you would like to stop at.\n",
    "\n",
    "#### You can view the progress of this training by switching to a termial and tailing the train.log file\n",
    "\n",
    "##### i.e. \"tail -f train.log\"\n",
    "\n",
    "#### once the validation is acceptible, you can stop training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "# change directories to the example/ssd directory\n",
    "cd incubator-mxnet/example/ssd\n",
    "\n",
    "#finetune  the model using our custom data set\n",
    "python train.py --train-path data/train.rec --val-path data/test.rec --class-names text_block --num-class 1 --finetune 1 --gpus 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### You can test the model by calling the following command from the terminal\n",
    "\n",
    "**(Note: You will need to change the path of the image to point to a sample image in your dataset.)**\n",
    "\n",
    "`python demo.py --epoch 2 --network vgg16_reduced --images ./data/VOCdevkit/VOC2018/JPEGImages/MVIMG_20180129_210518.jpg --thresh 0.5 --data-shape 300 --class-names text_block --gpu 0`\n",
    "\n",
    "#### You should also evaluate the model against the test dataset using the following command\n",
    "\n",
    "**(Note: You will need to change the epoch flag to point to the epoch you would like to evaluate.)**\n",
    "\n",
    "`python evaluate.py --gpus 0 --network vgg16_reduced --epoch 184 --class-names text_block --num-class 1 --rec-path data/test.rec`\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Once you are finished you will need to deploy the model\n",
    "**Pick the best epoch and deploy the model using the following command**\n",
    "\n",
    "`python deploy.py --network vgg16_reduced --epoch 172 --num-class 1 --data-shape 300`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### In order to deploy your model to the deeplens, you will need to tar.gz up your \".params\" and your \".json\" deployed model files and create a new model in the DeepLens Service inside the AWS Console online. You can then refer to the model using the model name in your project's lambda file. To optimize the model using the DeepLens Model Optimizer package, follow the instructions [here](https://docs.aws.amazon.com/deeplens/latest/dg/deeplens-model-optimizer-api-methods.html)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "readToMe",
   "language": "python",
   "name": "readtome"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}