examples/COLAB_maDLC_TrainNetwork_VideoAnalysis.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "COLAB_maDLC_TrainNetwork_VideoAnalysis.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/MMathisLab/DeepLabCut/blob/master/examples/COLAB_maDLC_TrainNetwork_VideoAnalysis.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RK255E7YoEIt"
      },
      "source": [
        "# DeepLabCut 2.2 Toolbox - COLAB\n",
        "![alt text](https://images.squarespace-cdn.com/content/v1/57f6d51c9f74566f55ecf271/1590444465547-SHXODUII311HEE407IL6/ke17ZwdGBToddI8pDm48kE4VnnB9_j2k1VP236ADqAFZw-zPPgdn4jUwVcJE1ZvWQUxwkmyExglNqGp0IvTJZUJFbgE-7XRK3dMEBRBhUpxQg9Vf0owGyf3dhfDKy8SxMujaKmp2B54Sb3VS1rO76Whq-cUhHVuKFlGUXsU9tJk/ezgif.com-video-to-gif.gif?format=1500w)\n",
        "\n",
        "https://github.com/DeepLabCut/DeepLabCut\n",
        "\n",
        "This notebook illustrates how to, for multi-animal projects, use the cloud-based GPU to:\n",
        "- create a multi-animal training set\n",
        "- train a network\n",
        "- evaluate a network\n",
        "- cross evaluate inference parameters\n",
        "- analyze novel videos\n",
        "- assemble tracklets\n",
        "- create quality check plots\n",
        "\n",
        "###This notebook assumes you already have a project folder with labeled data! \n",
        "\n",
        "This notebook demonstrates the necessary steps to use DeepLabCut for your own project.\n",
        "\n",
        "This shows the most simple code to do so, but many of the functions have additional features, so please check out the docs on GitHub.\n",
        "\n",
        "Mathis et al, in prep. <- please note, we are providing this toolbox as an early access release; more feeatures and details will be released with the forthcoming paper.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "txoddlM8hLKm"
      },
      "source": [
        "## First, go to \"Runtime\" ->\"change runtime type\"->select \"Python3\", and then select \"GPU\"\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q23BzhA6CXxu"
      },
      "source": [
        "#(this will take a few minutes to install all the dependences!)\n",
        "!pip install deeplabcut\n",
        "%reload_ext numpy\n",
        "%reload_ext scipy\n",
        "%reload_ext matplotlib\n",
        "%reload_ext mpl_toolkits"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Y36K4Eux3h-X"
      },
      "source": [
        "# Use TensorFlow 1.x:\n",
        "%tensorflow_version 1.x"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oTwAcbq2-FZz"
      },
      "source": [
        "#GUIs don't work on the cloud, so we supress them:\n",
        "import os\n",
        "os.environ[\"DLClight\"]=\"True\"\n",
        "import deeplabcut\n",
        "\n",
        "# Suppress tensorflow warning messages:\n",
        "import tensorflow as tf\n",
        "\n",
        "vers = (tf.__version__).split(\".\")\n",
        "if int(vers[0]) == 1 and int(vers[1]) > 12:\n",
        "    TF = tf.compat.v1  # behaves differently before 1.13\n",
        "else:\n",
        "    TF = tf\n",
        "\n",
        "TF.logging.set_verbosity(TF.logging.ERROR)\n",
        "DEBUG = True and \"DEBUG\" in os.environ and os.environ[\"DEBUG\"]\n",
        "from deeplabcut import DEBUG"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cQ-nlTkri4HZ"
      },
      "source": [
        "## Link your Google Drive (with your labeled data):\n",
        "\n",
        "### First, place your porject folder into you google drive! \"i.e. move the folder named \"Project-YourName-TheDate\" into google drive."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KS4Q4UkR9rgG"
      },
      "source": [
        "#Now, let's link to your GoogleDrive. Run this cell and follow the authorization instructions:\n",
        "#(We recommend putting a copy of the github repo in your google drive if you are using the demo \"examples\")\n",
        "\n",
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Frnj1RVDyEqs"
      },
      "source": [
        "## YOU WILL NEED TO EDIT THE PROJECT PATH **in the config.yaml file** TO BE SET TO YOUR GOOGLE DRIVE LINK!\n",
        "\n",
        "Typically, this will be: /content/drive/My Drive/yourProjectFolderName\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vhENAlQnFENJ"
      },
      "source": [
        "# PLEASE EDIT THIS:\n",
        "  \n",
        "ProjectFolderName = 'teamDLC-myProject-2020-05-22'\n",
        "VideoType = 'mp4' #, mp4, MOV, or avi, whatever you uploaded!\n",
        "\n",
        "\n",
        "# we are going to assume you put videos you want to analyze in the \"videos\" folder, but if this is NOT true, edit below:\n",
        "videofile_path = ['/content/drive/My Drive/'+ProjectFolderName+'/videos/'] #Enter the list of videos or folder to analyze.\n",
        "videofile_path\n",
        "\n",
        "\n",
        "#No need to edit this, as you set it when you passed the ProjectFolderName (above): \n",
        "path_config_file = '/content/drive/My Drive/'+ProjectFolderName+'/config.yaml'\n",
        "path_config_file\n",
        "#This creates a path variable that links to your google drive copy"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xNi9s1dboEJN"
      },
      "source": [
        "## Create a multi-animal training dataset:\n",
        "### You must do this step inside of Colab:\n",
        "\n",
        "- Reminder: you must connect EVERY bodypart in a skeleton before you run this step! See docs for crucial details on how to do this effciently: https://github.com/DeepLabCut/DeepLabCut/blob/master/docs/functionDetails.md#b-configure-the-project-\n",
        "\n",
        "![alt text](https://images.squarespace-cdn.com/content/v1/57f6d51c9f74566f55ecf271/1589256735280-SCN7CROSJNJWCDS6EK5T/ke17ZwdGBToddI8pDm48kB08p9-rNkpPD7A3fw8YFjZZw-zPPgdn4jUwVcJE1ZvWQUxwkmyExglNqGp0IvTJZamWLI2zvYWH8K3-s_4yszcp2ryTI0HqTOaaUohrI8PIno0kSvzOWihTW1zp8-1-7mzYxUQjsVr2n3nmNdVcso4/bodyparts-skeleton.png?format=1000w)\n",
        "![alt text](https://images.squarespace-cdn.com/content/v1/57f6d51c9f74566f55ecf271/1589410182515-9SJO9MML6CNCXBAWQ6Z6/ke17ZwdGBToddI8pDm48kJ1oJoOIxBAgRD2ClXVCmKFZw-zPPgdn4jUwVcJE1ZvWQUxwkmyExglNqGp0IvTJZUJFbgE-7XRK3dMEBRBhUpxBw7VlGKDQO2xTcc51Yv6DahHgScLwHgvMZoEtbzk_9vMJY_JknNFgVzVQ2g0FD_s/ezgif.com-video-to-gif+%2811%29.gif?format=750w)\n",
        "\n",
        "After running this script the training dataset is created and saved in the project directory under the subdirectory **'training-datasets'**\n",
        "\n",
        "This function also creates new subdirectories under **dlc-models** and appends the project config.yaml file with the correct path to the training and testing pose configuration file. These files hold the parameters for training the network. Such an example file is provided with the toolbox and named as **pose_cfg.yaml**.\n",
        "\n",
        "Now it is the time to start training the network!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "scrolled": true,
        "id": "eMeUwgxPoEJP"
      },
      "source": [
        "deeplabcut.cropimagesandlabels(path_config_file, size=(400, 400), userfeedback=False)\n",
        "\n",
        "#if you labeled on Windows, please set the windows2linux=True:\n",
        "deeplabcut.create_multianimaltraining_dataset(path_config_file, windows2linux=False)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c4FczXGDoEJU"
      },
      "source": [
        "## Start training:\n",
        "This function trains the network for a specific shuffle of the training dataset. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_pOvDq_2oEJW"
      },
      "source": [
        "#let's also change the display and save_iters just in case Colab takes away the GPU... \n",
        "#if that happens, you can reload from a saved point. \n",
        "#Typically, you want to train to 50,000 iterations.\n",
        "#more info and there are more things you can set: https://github.com/AlexEMG/DeepLabCut/blob/master/docs/functionDetails.md#g-train-the-network\n",
        "\n",
        "#which shuffle do you want to train?\n",
        "shuffle = 1\n",
        "deeplabcut.train_network(path_config_file, shuffle=shuffle, displayiters=100,saveiters=1000)\n",
        "\n",
        "#this will run until you stop it (CTRL+C), or hit \"STOP\" icon, or when it hits the end (default, 50K iterations). \n",
        "#Whichever you chose, you will see what looks like an error message, but it's not an error - don't worry...."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RiDwIVf5-3H_"
      },
      "source": [
        "**When you hit \"STOP\" you will get a KeyInterrupt \"error\"! No worries! :)**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xZygsb2DoEJc"
      },
      "source": [
        "## Start evaluating: for maDLC, this is several steps. \n",
        " - First, we evaluate the pose estimation performance, and then we can cross-valudate optimal inference parameters.\n",
        "\n",
        "- This funtion evaluates a trained model for a specific shuffle/shuffles at a particular state or all the states on the data set (images) and stores the results as .5 and .csv file in a subdirectory under **evaluation-results**\n",
        "\n",
        "- If the scoremaps do not look accurate, don't proceed to tracklet assembly; please consider (1) adding more data, (2) adding more bodyparts!\n",
        "\n",
        "Here is an example of what you'd aim to see before proceeding:\n",
        "\n",
        "![alt text](https://images.squarespace-cdn.com/content/v1/57f6d51c9f74566f55ecf271/1590535809087-X655WY9W1MW1MY1I7DHE/ke17ZwdGBToddI8pDm48kBoswZhKnUtAF7-bTXgw67EUqsxRUqqbr1mOJYKfIPR7LoDQ9mXPOjoJoqy81S2I8N_N4V1vUb5AoIIIbLZhVYxCRW4BPu10St3TBAUQYVKc5tTP1cnANTUwNNPnYFjIp6XbP9N1GxIgAkxvBVqt0UvLpPHYwvNQTwHg8f_Zu8ZF/evaluation.png?format=1000w)\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nv4zlbrnoEJg"
      },
      "source": [
        "#%matplotlib notebook\n",
        "#let's evaluate first:\n",
        "deeplabcut.evaluate_network(path_config_file,Shuffles=[shuffle], plotting=True,c_engine=False)\n",
        "#plot a few scoremaps:\n",
        "deeplabcut.extract_save_all_maps(path_config_file, shuffle=shuffle, Indices=[0])\n",
        "\n",
        "\n",
        "#and begin to cross-validate:\n",
        "deeplabcut.evaluate_multianimal_crossvalidate(\n",
        "            path_config_file,\n",
        "            Shuffles=[shuffle],\n",
        "            edgewisecondition=True,\n",
        "            leastbpts=1,\n",
        "            init_points=20,\n",
        "            n_iter=100,\n",
        "            target='rpck_train',\n",
        "        )"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4MbIL5z2U7fp"
      },
      "source": [
        "^ NOTE: this optimized part detection for video analysis. It cannot optimze for tracking, as this is use-case dependent. Please check the docs on how you can set the best parameters and modify/test before \"final\" tracking parameters. You can use COLAB to analyze videos, but afterwards we recommend using the outputs/proejct folder locally to run the final steps! They do not require a GPU. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OVFLSKKfoEJk"
      },
      "source": [
        "## Start Analyzing videos: \n",
        "This function analyzes the new video. The user can choose the best model from the evaluation results and specify the correct snapshot index for the variable **snapshotindex** in the **config.yaml** file. Otherwise, by default the most recent snapshot is used to analyse the video.\n",
        "\n",
        "The results are stored in hd5 file in the same directory where the video resides. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Y_LZiS_0oEJl"
      },
      "source": [
        "print(\"Start Analyzing my video(s)!\")\n",
        "scorername = deeplabcut.analyze_videos(path_config_file, \n",
        "                                       videofile_path, \n",
        "                                       shuffle=shuffle, \n",
        "                                       videotype=VideoType, \n",
        "                                       c_engine=False)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PxRLS2_-r55K"
      },
      "source": [
        "## The steps below work on a single video at a time.\n",
        "- Here you can create a video to check the pose estimation detection quality! If this looks good, proceed to tracklet conversions with the interactive GUI (ouside of COLAB for now), or if you know your optimal parameters, you can automate this and run the additional steps shown in a few cells down."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "65mWwX5bTc5C"
      },
      "source": [
        "# PROTIP: look at the output video; if the pose estimation (i.e. key points) don't look good, don't proceed with tracking - add more data to your training set and re-train!\n",
        "from deeplabcut.utils import auxiliaryfunctions\n",
        "#scorername, DLCscorerlegacy = auxiliaryfunctions.GetScorerName(path_config_file, shuffle, trainFraction=0)\n",
        "print(\"scorename is: \"+scorername)\n",
        "\n",
        "#inferencefg_path = '/content/drive/My Drive/mwm-penguins-2020-03-31/dlc-models/iteration-0/mwmMar31-trainset95shuffle1/test/inference_cfg.yaml'\n",
        "#scorername, DLCscorerlegacy = auxiliaryfunctions.GetScorerName(\n",
        " #   inferencefg_path, shuffle, trainFraction=0\n",
        "#)\n",
        "\n",
        "#let's check a specific video:\n",
        "Specific_videofile = '/content/drive/My Drive/yourproject/videos/demo.mp4'\n",
        "\n",
        "deeplabcut.create_video_with_all_detections(path_config_file, [Specific_videofile], scorername)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "78x3BbotIsuO"
      },
      "source": [
        "## Convert Detections to Tracklets:\n",
        "\n",
        "- The idea is that you test and adapt hyperparameters for tracking outside of COLAB. Once you have good parameters, this can be automated on future videos. Shown here!\n",
        "\n",
        "- I.e., instead of always doing an interactive parameter setting step, you can simply convert tracklets to .h5 files using these parameters (see GitHub for more info)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zIvXM7TXIs-U"
      },
      "source": [
        "#assemble tracklets:\n",
        "#read the docs: which tracker to test out (you can run this twice to try multiple):\n",
        "tracktype= 'box' #box, skeleton\n",
        "\n",
        "deeplabcut.convert_detections2tracklets(path_config_file, Specific_videofile, videotype=VideoType,\n",
        "                                                    shuffle=shuffle, track_method=tracktype, overwrite=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HHH9lCM7JCZ0"
      },
      "source": [
        "## Now you should manually verify the tracks and correct them if needed! [currently only working outside of COLAB]"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ocRzxq7fJCjm"
      },
      "source": [
        "''' here is the code you would need:\n",
        "os.environ[\"DLClight\"]=\"False\"\n",
        "import deeplabcut\n",
        "\n",
        "#ATTENTION:\n",
        "picklefile = '/...._10000_bx.pickle' #(see your video folder for path i.e. right click and say copy path!!!)\n",
        "vid ='/yourVIDEO.mp4'\n",
        "#if you want occlusions filled in, tell us how many frames to fill in, i.e. if there is a gap in data:\n",
        "framestofill = 0. #note, put \"0\" if you want ALL gaps filled!\n",
        "\n",
        "%matplotlib inline\n",
        "\n",
        "from deeplabcut.refine_training_dataset.tracklets import refine_tracklets\n",
        "TrackletManager, TrackletVisualizer = refine_tracklets(path_config_file, \n",
        "                                                          picklefile, \n",
        "                                                          Specific_videofile, \n",
        "                                                          min_swap_frac=0,\n",
        "                                                          min_tracklet_frac=0, \n",
        "                                                          max_gap=framestofill)\n",
        "'''"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AUOKrLJRseUX"
      },
      "source": [
        "## Let's assume you have great tracking parameters, and you want to analyze a full set of videos:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "a6izVWX8sdzL"
      },
      "source": [
        "#^^^^^^^^^You do NOT neeed to run if you hit \"save\" in the GUI ^^^^^^^^^^\n",
        "#this is just if you want to run the same parameters over a set of videos!\n",
        "picklefile = '/content/drive/My Drive/mwm-penguins-2020-03-31/videos/penguindemoDLC_resnet50_mwmMar31shuffle1_22000_bx.pickle' #(see your video folder for path i.e. right click and say copy path!!!)\n",
        "vid ='/content/drive/My Drive/mwm-penguins-2020-03-31/videos/penguindemo.mp4'\n",
        "\n",
        "deeplabcut.convert_raw_tracks_to_h5(path_config_file, picklefile)\n",
        "deeplabcut.filterpredictions(path_config_file, \n",
        "                                 videofile_path, \n",
        "                                 videotype=VideoType, \n",
        "                                 track_method = tracktype)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Zk4xGb8Ftf3B"
      },
      "source": [
        "## Create plots of your trajectories:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gX21zZbXoEKJ"
      },
      "source": [
        "deeplabcut.plot_trajectories(path_config_file,videofile_path, videotype=VideoType, track_method=tracktype)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pqaCw15v8EmB"
      },
      "source": [
        "Now you can look at the plot-poses file and check the \"plot-likelihood.png\" might want to change the \"p-cutoff\" in the config.yaml file so that you have only high confidnece points plotted in the video. i.e. ~0.8 or 0.9. The current default is 0.4. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pCrUvQIvoEKD"
      },
      "source": [
        "## Create labeled video:\n",
        "This funtion is for visualiztion purpose and can be used to create a video in .mp4 format with labels predicted by the network. This video is saved in the same directory where the original video resides. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6aDF7Q7KoEKE"
      },
      "source": [
        "deeplabcut.create_labeled_video(path_config_file,\n",
        "                                videofile_path, \n",
        "                                shuffle=shuffle, \n",
        "                                draw_skeleton=True, \n",
        "                                videotype=VideoType, \n",
        "                                save_frames=False,\n",
        "                                filtered=True, \n",
        "                                track_method = tracktype)"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}