ner.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ANfBloGyVq50"
   },
   "source": [
    "# Natural Language Entity Extraction\n",
    "Extracting Ground Truth Labels from Radiology Reports\n",
    "<hr/>\n",
    "\n",
    "\n",
    "In this project we show how to extract labels from unstructured medical text in order to obtain ground truth data.\n",
    "Table of contents:\n",
    "- Extracting disease labels from clinical reports\n",
    "    - Text matching\n",
    "    - Evaluating a labeler\n",
    "    - Negation detection\n",
    "    - Dependency parsing\n",
    "\n",
    "\n",
    "Related paper [https://arxiv.org/abs/1901.07031](https://arxiv.org/abs/1901.07031)  \n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Packages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import the following libraries for this assignment.\n",
    "\n",
    "- `matplotlib.pyplot` - standard plotting library\n",
    "- `nltk` - an NLP package \n",
    "- `pandas` - we'll use this to keep track of data\n",
    "- `tensorflow` - standard deep learning library\n",
    "- `transformers` - convenient access to pretrained natural language models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import nltk \n",
    "import pandas as pd\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "from transformers import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "5ruouERuVq53"
   },
   "outputs": [],
   "source": [
    "import util\n",
    "from util import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "9E1tLUwQVq57"
   },
   "source": [
    "<a name=\"1\"></a>\n",
    "## Extracting Labels\n",
    "\n",
    "Our goal is to extract disease labels for patients from unstructured clinical reports.  Unlike most cases, instead of \"learning\" from the dataset, we will primarily build different \"rules\" that help us extract knowledge from natural language. \n",
    "- Because there is less risk of overfitting when using a \"rules-based\" approach, we will just use one dataset which will also be the test set.\n",
    "\n",
    "The test set consists of 1,000 X-ray reports that have been manually labeled by a board certified radiologist for the presence or lack of presence of different pathologies. \n",
    "- We also have access to the extracted \"Impression\" section of each report which is the overall summary of the radiologists for each X-ray. \n",
    "\n",
    "Let's have a look at the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "0h94kdGkVq57",
    "outputId": "c45d8b0d-ac9c-4b9e-ac49-91fdfd3cd400"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "test_df size: (1000, 12)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Report Impression</th>\n",
       "      <th>Cardiomegaly</th>\n",
       "      <th>Lung Lesion</th>\n",
       "      <th>Airspace Opacity</th>\n",
       "      <th>Edema</th>\n",
       "      <th>Consolidation</th>\n",
       "      <th>Pneumonia</th>\n",
       "      <th>Atelectasis</th>\n",
       "      <th>Pneumothorax</th>\n",
       "      <th>Pleural Effusion</th>\n",
       "      <th>Pleural Other</th>\n",
       "      <th>Fracture</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>\\n \\n1.mild pulmonary edema, and cardiomegaly....</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>\\n \\n1.unremarkable cardiomediastinal silhouet...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>\\n1. lines and tubes are unchanged in position...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>\\n1. postoperative portable film with a right-...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>\\n \\n1.single frontal view of the chest demons...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                   Report Impression  Cardiomegaly  \\\n",
       "0  \\n \\n1.mild pulmonary edema, and cardiomegaly....          True   \n",
       "1  \\n \\n1.unremarkable cardiomediastinal silhouet...         False   \n",
       "2  \\n1. lines and tubes are unchanged in position...         False   \n",
       "3  \\n1. postoperative portable film with a right-...         False   \n",
       "4  \\n \\n1.single frontal view of the chest demons...         False   \n",
       "\n",
       "   Lung Lesion  Airspace Opacity  Edema  Consolidation  Pneumonia  \\\n",
       "0        False             False   True          False      False   \n",
       "1        False              True  False          False      False   \n",
       "2        False              True  False          False      False   \n",
       "3        False              True   True          False      False   \n",
       "4        False              True  False          False      False   \n",
       "\n",
       "   Atelectasis  Pneumothorax  Pleural Effusion  Pleural Other  Fracture  \n",
       "0         True         False              True          False     False  \n",
       "1        False         False             False          False      True  \n",
       "2        False         False             False          False     False  \n",
       "3         True          True             False          False     False  \n",
       "4        False         False              True          False     False  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(\"test_df size: {}\".format(test_df.shape))\n",
    "test_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are a few example impressions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Report Impression 0:\n",
      "\n",
      " \n",
      "1.mild pulmonary edema, and cardiomegaly.  trace pleural fluid \n",
      "effusions.\n",
      " \n",
      "2.low lung volumes with minimal basilar atelectasis.\n",
      " \n",
      "3.no new focal consolidation. \n",
      " \n",
      "4.interval placement of defibrillation pads.  \n",
      " \n",
      "\n",
      "\n",
      "Report Impression 1:\n",
      "\n",
      " \n",
      "1.unremarkable cardiomediastinal silhouette\n",
      " \n",
      "2.diffuse reticular pattern, which can be seen with an atypical \n",
      "infection or chronic fibrotic change.  no focal consolidation.\n",
      " \n",
      "3.no pleural effusion or pneumothorax\n",
      " \n",
      "4.mild degenerative changes in the lumbar spine and old right rib \n",
      "fractures. \n",
      " \n",
      "\n",
      "\n",
      "Report Impression 2:\n",
      "\n",
      "1. lines and tubes are unchanged in position.\n",
      "2. increasing retrocardiac opacity and left midlung zone opacity.\n",
      "3. there is a deep left costophrenic sulcus which is increased when\n",
      "compared with prior films. no definite evidence of left\n",
      "pneumothorax. clinical correlation is recommended. if clinically\n",
      "indicated, consider film in expiration or decubitus views.\n",
      "4. the icu team was informed of these results at 10 am on 05_02_2005.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for i in range(3):\n",
    "    print(f'\\nReport Impression {i}:')\n",
    "    print(test_df.loc[i, 'Report Impression'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that these reports are fairly unstructured, which makes information extraction challenging. Our goal will be to extract the presence or absence of different abnormalities from the raw text. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we see the distribution of abnormalities in the dataset. We have placed all the names of these abnormalities in a list named `CATEGORIES` to use it as labels the graph below. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "nPAAsvcbVq5-",
    "outputId": "d5f008ea-462f-4204-9838-05b791b43a7e"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAwcAAAEvCAYAAAAQOL09AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3de5SdVZ3n//fHAAkYiCKIZVBLJa1yDRBo7yI6jIoKaKaRRgF1Oj97vDuo6XakAbuFbm3NoCMaFUFlWuRmI7hEmpt4AxJyRS62Em350dC2Grkr4Tt/nF1wKCtJVVJV55C8X2tl1X72s5+9v+f4LDzfs/d+TqoKSZIkSXpMrwOQJEmS1B9MDiRJkiQBJgeSJEmSGpMDSZIkSYDJgSRJkqTG5ECSJEkSAFv0OgA9bIcddqjBwcFehyFJkqRN2OLFi39VVTuOdM7koI8MDg6yaNGiXochSZKkTViSn6/tnMuKJEmSJAEmB5IkSZIakwNJkiRJgMmBJEmSpMbkQJIkSRJgciBJkiSpMTmQJEmSBJgcSJIkSWpMDiRJkiQBJgeSJEmSmi16HYAetuLW1QzOv6jXYUy6VScf3OsQJEmShDMHkiRJkhqTA0mSJEmAyYEkSZKkxuRAkiRJEmByIEmSJKnZ5J5WlGQNsKKr6tCqWrUR/c0GnlxV39rY2CRJkqR+tsklB8C9VTV7pBNJAqSqHhxDf7OBOcCok4MNHEeSJEnqqU1+WVGSwSQ3JfkysBJ4SpJTkyxKcn2SE7ra7pfkB0mWJbkmyQzgRODwJEuTHJ7k+CTHdl2zso0x0jjvT3JtkuXd40iSJEn9aFNMDrZuH+SXJjm/1c0CPlNVu1XVz4EPVdUcYE/gJUn2TLIVcBbw7qraC3g5cDdwHHBWVc2uqrPWM/ZD4wDPasf705l92DfJi8f7xUqSJEnjZZNfVpRkEPh5Vf2oq82fJZlH5/UPALsCBdxWVdcCVNXv2vVjGbt7nIPavyXteDqdZOG73Re0OOYBTNlux7GMJUmSJI2rTTE5GMndQ4UkTweOBfarqt8kOR2YNoa+HuCRMy7d197dVQ5wUlV9bl2dVdVCYCHA1IFZNYY4JEmSpHG1KS4rWp/t6HyIX51kJ+CVrf4mYCDJfgBJtk2yBXAnsG3X9auAfVqbfYCnr2Wci4G3JJne2s5M8sRxfi2SJEnSuNlcZg4eUlXLkiwBbgT+Dfh+q/99ksOBTyXZGriXzr6Dy4H5SZYCJwHnAkcluR64Grh5LeN8J8lzgB+2pUl3AW8E7pjI1ydJkiRtqFS5kqVfTB2YVQNHL+h1GJNu1ckH9zoESZKkzUaSxe3hPH9kc1xWJEmSJGkEJgeSJEmSAJMDSZIkSY3JgSRJkiRgM3xaUT/bY+YMFrk5V5IkST3izIEkSZIkwORAkiRJUmNyIEmSJAlwz0FfWXHragbnX9TrMNbLHy2TJEnaNDlzIEmSJAkwOZAkSZLUmBxIkiRJAkwOJEmSJDUmB5IkSZKASU4OkqxJsjTJyiRnJ9mm1d81CWMfk+TTazl3aJLlSW5IsiLJocOue3LX8aokO0x0vJIkSdJkm+yZg3uranZV7Q78HnjbeHaeZMyPZk2yF/Bx4JCqeg7wWuDjSfZsTY4BnryWyyc8PkmSJGmy9HJZ0VXALsMrk7w/ybXtm/wTWt1gkpVdbY5NcnwrX5FkQZJFwLuTvCbJ1UmWJPmXJDutJ45jgY9W1S0A7e9JwPuTzAXmAGe2GY+t2zXvTHJdm2V4dovjsUlOS3JNG/uQVn9MkguSXAZcusHvliRJkjTBepIctG/QXwmsGFZ/EDAL2B+YDeyb5MWj6HKrqppTVf8IfA94blXtDXwN+MB6rt0NWDysbhGwW1Wd08pHthmPe9v5X1XVPsCpdJILgA8Bl1XV/sBLgY8leWw7tw8wt6peMnzwJPOSLEqyaM09q0fxUiVJkqSJMdnLXLZOsrSVrwK+OOz8Qe3fknY8nU6y8Iv19HtWV3ln4KwkA8BWwC0bFfHIzmt/FwOva+WDgNcmGUoWpgFPbeVLqurXI3VUVQuBhQBTB2bVBMQqSZIkjcpkJwf3VtXsdZwPcFJVfe4RlcnOPHKWY9qw6+7uKn8K+ERVXZDkAOD49cT0Y2BfYFlX3b7A9eu45v72dw0Pv4cBXl9VNw2L/U+HxSdJkiT1pX57lOnFwFuSTAdIMjPJE4HbgScmeUKSqcCr19HHDODWVj56FGN+HPirJINtzEHgr4F/bOfvBLYdZezvTJLWz96juEaSJEnqG3319Jyq+k6S5wA/bJ+x7wLeWFV3JDkRuIbOB/8b19HN8cDZSX4DXAY8fT1jLk3yQeCbSbYE/gB8oKqGlj+dDnw2yb3A89bR1UeABcDyJI+hs5xpXUmMJEmS1FdS5TL3fjF1YFYNHL2g12Gs16qTD+51CJIkSdpASRZX1ZyRzvXbsiJJkiRJPWJyIEmSJAkwOZAkSZLUmBxIkiRJAvrsaUWbuz1mzmCRm30lSZLUI84cSJIkSQJMDiRJkiQ1JgeSJEmSAPcc9JUVt65mcP5FvQ5DkqQJ5Y9pSv3LmQNJkiRJgMmBJEmSpMbkQJIkSRJgciBJkiSp2ajkIMmaJEuTrExydpJtWv1d4xPeOsc+Jsmn11L/Hy2uoX+7tnMfS3J9+7tjkquTLEnyojGOPSfJKeP1WiRJkqR+sLFPK7q3qmYDJDkTeBvwiY2OqkmyRVU9sAGXnlVV7xihfh6wfVWtSfIGYEVV/fexdl5Vi4BFGxCXJEmS1LfGc1nRVcAuwyuTvD/JtUmWJzmh1Q0mWdnV5tgkx7fyFUkWJFkEvDvJa7q+4f+XJDttSHBJLgCmA4uTfBD4B+CQNrOwdfdsR5K5SU5v5f/WZkaWJfluqzsgyYWtvH2Sb7TX96Mke7b645Oc1l7Pz5K8a0PiliRJkibLuPzOQZItgFcC3x5WfxAwC9gfCHBBkhcDv1hPl1tV1ZzWx+OB51ZVJfnvwAeA/7me6w9P8sKu4+dV1WuT3NU103E7MGdohiHJ2vo6DvivVXVrkseNcP4EYElVHZrkQODLwOx27tnAS4FtgZuSnFpVf1hP7JIkSVJPbGxysHWSpa18FfDFYecPav+WtOPpdJKF9SUHZ3WVdwbOSjIAbAXcMoq41rasaEN8Hzg9ydeB80Y4/0Lg9QBVdVmSJyTZrp27qKruB+5PcgewE/DL7ouTzKOz3Ikp2+04TiFLkiRJYzduew7WIsBJVfW5R1QmO/PIJU3Thl13d1f5U8AnquqCJAcAx294uOtUI8VTVW9L8qfAwXSWJO07hj7v7yqvYYT3u6oWAgsBpg7MquHnJUmSpMky0Y8yvRh4S5LpAElmJnkicDvwxPYt+1Tg1evoYwZwaysfPYGx3p7kOUkeAxw2VJnkmVV1dVUdB/wH8JRh110FHNnaHgD8qqp+N4FxSpIkSRNiXPYcrE1VfSfJc4AftjX9dwFvrKo7kpwIXEPng/+N6+jmeODsJL8BLgOePoqhh+85+B9V9YP1XDMfuJBOArCIzhIogI8lmUVnFuRSYBnwkmHxnZZkOXAPE5vASJIkSRMmVa5k6RdTB2bVwNELeh2GJEkTatXJB/c6BGmzlmTx0MN/hvMXkiVJkiQBJgeSJEmSGpMDSZIkSYDJgSRJkqRmQp9WpLHZY+YMFrlJS5IkST3izIEkSZIkwORAkiRJUmNyIEmSJAkwOZAkSZLUuCG5j6y4dTWD8y/qdRh9xV/RlCRJmjzOHEiSJEkCTA4kSZIkNSYHkiRJkgCTA0mSJEnNuCcHSdYkWZpkZZKzk2wz3mNsjCTHJHly1/GqJDv0MiZJkiSpH0zEzMG9VTW7qnYHfg+8bQLG2BjHAE9eX6PRSOLTniRJkrTJmOhlRVcBuyQ5IMkVSc5JcmOSM5MEIMm+Sa5MsjjJxUkGWv0VSea08g5JVrXyMUm+keSS9q3/O5K8L8mSJD9Ksn1rN7sdL09yfpLHJ5kLzAHObLMbW7c435nkuiQrkjy7Xb99G2d562fPVn98kq8k+T7wlSSDSa5q11+X5Pmt3WFJLk3HQJKbkzxpgt9vSZIkaYNNWHLQvlV/JbCiVe0NvAfYFXgG8IIkWwKfAuZW1b7AacDfjaL73YHXAfu19vdU1d7AD4GjWpsvAx+sqj1bDH9TVecAi4Aj2+zGva3tr6pqH+BU4NhWdwKwpF3/162/IbsCL6+qI4A7gP/Srj8cOAWgqs4HbgPeDny+jf/vo3htkiRJUk9MxLKYrZMsbeWrgC8CzweuqapfArTzg8Bv6XzQv6RNJEyh84F6fS6vqjuBO5OsBr7Z6lcAeyaZATyuqq5s9WcAZ6+jv/Pa38V0kg6AFwKvB6iqy5I8Icl27dwFXYnFlsCnk8wG1gB/0tXvO4GVwI+q6p9GGjjJPGAewJTtdlzPy5YkSZImzkQkB/dW1ezuivbB//6uqjVt7ADXV9XzRujnAR6e2Zg27Fx3Xw92HT/Ihr2moeuH4lqfu7vK7wVuB/aiE+99Xed2bjHtlOQxVfXg8I6qaiGwEGDqwKwae+iSJEnS+Oj1o0xvAnZM8jyAJFsm2a2dWwXs28pzx9JpVa0GfpPkRa3qTcDQLMKdwLaj6OYq4MgW1wF0lh79boR2M4Db2gf/N9GZ/RhaVnUacARwA/C+sbwGSZIkabL19Gk7VfX7tkn4lLYUaAtgAXA98HHg623ZzUUb0P3RwGfbo1R/Bry51Z/e6u8FRpqxGHI8cFqS5cA9rb+RfAY4N8lRwLd5eFbhr4Grqup7SZYB1ya5qKpu2IDXIkmSJE24VLmSpV9MHZhVA0cv6HUYfWXVyQf3OgRJkqRNSpLFVTVnpHO9XlYkSZIkqU+YHEiSJEkCTA4kSZIkNSYHkiRJkoAeP61Ij7THzBkscgOuJEmSesSZA0mSJEmAyYEkSZKkxuRAkiRJEmByIEmSJKlxQ3IfWXHragbnX9TrMKRNgr+uLUnS2DlzIEmSJAkwOZAkSZLUmBxIkiRJAkwOJEmSJDV9nRwkOTRJJXl2O56d5FWjuO6AJBdu4JjvSbLNBl77tiRHbci1kiRJUq/1dXIAHAF8r/0FmA2sNznYSO8BNig5qKrPVtWXxzkeSZIkaVL0bXKQZDrwQuCtwBuSbAWcCByeZGmSw5M8NslpSa5JsiTJISP0M2KbJFOSfDzJyiTLk7wzybuAJwOXJ7m8tTs1yaIk1yc5oavfk5P8uF378VZ3fJJjW/ldXee/NsFvlyRJkrTR+vl3Dg4Bvl1VNyf5T2AP4DhgTlW9AyDJR4HLquotSR4HXJPkX4b186G1tDkKGARmV9UDSbavql8neR/w0qr61dD1rX4KcGmSPYFbgcOAZ1dVtX6Hmw88varuX8t5SZIkqa/07cwBnaVEQ9+4f42HlxZ1OwiYn2QpcAUwDXjqKNu8HPhcVT0AUFW/Xkscf5bkOmAJsBuwK7AauA/4YpLXAfeMcN1y4MwkbwQeWNuLTDKvzUwsWnPP6rU1kyRJkiZcX84cJNkeOBDYI0kBU4ACrh/eFHh9Vd007PqdRtFmNHE8HTgW2K+qfpPkdGBam2nYH3gZMBd4R4u328HAi4HXAB9KssdQItKtqhYCCwGmDsyq9QYlSZIkTZB+nTmYC3ylqp5WVYNV9RTgFjrf+G/b1e5i4J1pn/ST7D1CX2trcwnw/yXZotVv3+rv7BpjO+BuYHVLOF7Z2k4HZlTVt4D3Ant1D5jkMcBTqupy4IPADGD6Br0TkiRJ0iTp1+TgCOD8YXXnAk8Cdh3akAx8BNgSWJ7k+nY83NrafAH4RatfBvx5q18IfDvJ5VW1jM5yohuB/wt8v7XZFrgwyXI6T1N637AxpwBfTbKiXX9KVf12rG+CJEmSNJlS5UqWfjF1YFYNHL2g12FIm4RVJx/c6xAkSepLSRZX1ZyRzvXrzIEkSZKkSWZyIEmSJAkwOZAkSZLUmBxIkiRJAvr0dw42V3vMnMEiN1FKkiSpR5w5kCRJkgSYHEiSJElqTA4kSZIkASYHkiRJkho3JPeRFbeuZnD+Rb0OY7385VlJkqRNkzMHkiRJkgCTA0mSJEmNyYEkSZIkwORAkiRJUtOXyUGSNUmWJlmZ5Owk2/Q6ptFIMifJKb2OQ5IkSdoQfZkcAPdW1eyq2h34PfC2Xgc0GlW1qKre1es4JEmSpA3Rr8lBt6uAXZIckOSKJOckuTHJmUkCkGTfJFcmWZzk4iQDrf6KJHNaeYckq1r5mCTfSHJJklVJ3pHkfUmWJPlRku1bu9nteHmS85M8vqvfv09yTZKbk7yo1R+Q5MJW3j/JD1ufP0jyrMl+4yRJkqSx6OvkIMkWwCuBFa1qb+A9wK7AM4AXJNkS+BQwt6r2BU4D/m4U3e8OvA7Yr7W/p6r2Bn4IHNXafBn4YFXt2WL4m67rt6iq/Vs83fVDbgRe1Po8DvjoqF60JEmS1CP9+iNoWydZ2spXAV8Eng9cU1W/BGjnB4Hf0vmgf0mbSJgC3DaKMS6vqjuBO5OsBr7Z6lcAeyaZATyuqq5s9WcAZ3ddf177u7jFMdwM4Iwks4ACthwpiCTzgHkAU7bbcRRhS5IkSROjX5ODe6tqdndF++B/f1fVGjrxB7i+qp43Qj8P8PDsyLRh57r7erDr+EFG974MtR+KY7iP0ElADksyCFwxUidVtRBYCDB1YFaNYlxJkiRpQvT1sqJRugnYMcnzAJJsmWS3dm4VsG8rzx1Lp1W1GvjN0H4C4E3Aleu4ZLgZwK2tfMxYxpYkSZJ64VGfHFTV7+l88P/7JMuApXSWIAF8HPjLJEuAHTag+6OBjyVZDswGThzDtf8AnNTG7tcZGkmSJOkhqXIlS7+YOjCrBo5e0Osw1mvVyQf3OgRJkiRtoCSLq2rOSOce9TMHkiRJksaHyYEkSZIkwORAkiRJUmNyIEmSJAnwKTp9ZY+ZM1jkZl9JkiT1iDMHkiRJkgCTA0mSJEmNyYEkSZIkwORAkiRJUuOG5D6y4tbVDM6/qNdhbDL8JWdJkqSxceZAkiRJEmByIEmSJKkxOZAkSZIEmBxIkiRJaiYlOUjypCRfS/LTJIuTfCvJn0zwmMcnObaVT0zy8hHaHJDkwvX0MzvJq7qOX5tk/vhHLEmSJPXWhD+tKEmA84EzquoNrW4vYCfg5okeH6CqjtuIy2cDc4Bvtb4uAC4Yj7gkSZKkfjIZMwcvBf5QVZ8dqqiqZcD3knwsycokK5IcDg99m39FknOS3JjkzJZgkOTkJD9OsjzJx1vdYJLLWt2lSZ46PIAkpyeZ28qvaP1eB7yuq83+SX6YZEmSHyR5VpKtgBOBw5MsTXJ4kmOSfHpdY7fxTmn9/GxobEmSJKmfTUZysDuweIT619H5Vn4v4OXAx5IMtHN7A+8BdgWeAbwgyROAw4DdqmpP4G9b20/RmZXYEzgTOGVtgSSZBnweeA2wL/CkrtM3Ai+qqr2B44CPVtXvW/msqppdVWcN63JdYw8ALwReDZy8tpgkSZKkftHLDckvBP6pqtZU1e3AlcB+7dw1VfXLqnoQWAoMAquB+4AvJnkdcE9r+zzg/7byV1q/a/Ns4Jaq+klVFfDVrnMzgLOTrAQ+Cew2itewrrG/UVUPVtWP6SyhGlGSeUkWJVm05p7VoxhSkiRJmhiTkRxcT+db+rG4v6u8Btiiqh4A9gfOofNt/LfHJ7yHfAS4vKp2pzOzMG0j++t+DVlbo6paWFVzqmrOlG1mbOSQkiRJ0oabjOTgMmBqknlDFUn2BH5LZy3/lCQ7Ai8GrllbJ0mmAzOq6lvAe+ksRwL4AfCGVj4SuGodsdwIDCZ5Zjs+ouvcDODWVj6mq/5OYNu19DeWsSVJkqS+NuHJQVu+cxjw8vYo0+uBk+gsx1kOLKOTQHygqv59HV1tC1yYZDnwPeB9rf6dwJtb/ZuAd68jlvuAecBFbUPyHV2n/wE4KckSHvkUp8uBXYc2JA/rctRjS5IkSf0unc/u6gdTB2bVwNELeh3GJmPVyQf3OgRJkqS+k2RxVc0Z6Zy/kCxJkiQJMDmQJEmS1JgcSJIkSQJMDiRJkiQ1W6y/iSbLHjNnsMhNtJIkSeoRZw4kSZIkASYHkiRJkhqTA0mSJEmAyYEkSZKkxg3JfWTFrasZnH9Rr8OQtBnwF8QlSSNx5kCSJEkSYHIgSZIkqTE5kCRJkgSYHEiSJElqNpvkIMmaJEu7/s0foc0BSS7sRXySJElSr21OTyu6t6pm9zoISZIkqV9tNjMHa5PkFUluTHId8Lqu+scmOS3JNUmWJDmk1R+T5BtJLkmyKsk7kryvtflRku1bu79Icm2SZUnOTbJNj16iJEmSNCqbU3Kw9bBlRYcnmQZ8HngNsC/wpK72HwIuq6r9gZcCH0vy2HZudzqJxH7A3wH3VNXewA+Bo1qb86pqv6raC7gBeOtEv0BJkiRpY2zWy4qSzAZuqaqftOOvAvPa6YOA1yY5th1PA57aypdX1Z3AnUlWA99s9SuAPVt59yR/CzwOmA5cPFJQSeYNjTllux037hVKkiRJG2FzSg7GKsDrq+qmR1Qmfwrc31X1YNfxgzz8np4OHFpVy5IcAxww0iBVtRBYCDB1YFaNU+ySJEnSmG1Oy4pGciMwmOSZ7fiIrnMXA+9MEoAke4+x722B25JsCRy50ZFKkiRJE2xzSg6G7zk4uaruo7Ok56K2IfmOrvYfAbYElie5vh2PxYeBq4Hv00lCJEmSpL6WKley9IupA7Nq4OgFvQ5D0mZg1ckH9zoESVKPJFlcVXNGOrc5zRxIkiRJWgeTA0mSJEmAyYEkSZKkxuRAkiRJEuDvHPSVPWbOYJGbBCVJktQjzhxIkiRJAkwOJEmSJDUmB5IkSZIA9xz0lRW3rmZw/kW9DkOSJEkTqJ9/iNKZA0mSJEmAyYEkSZKkxuRAkiRJEmByIEmSJKkxOZAkSZIEjDI5SHJokkry7K66Jyc5Z+JC2zhJtkqyIMm/JvlJkn9OsvM4j/HQe5BkdpJXjWf/kiRJ0mQa7czBEcD32l8Aqur/r6q5wxsm6ZfHo34U2BZ4VlXNAr4BnJck4zXAsPdgNmByIEmSpEet9SYHSaYDLwTeCryhq34wycpWPibJBUkuAy5NMpDku0mWJlmZ5EWt3V1JPpnk+iSXJtmx1f9FkmuTLEtybpJtWv1OSc5v9cuSPL/VvzHJNa3/zyWZMizmbYA3A++tqjUAVfUl4H7gwBb7jUnOTHJDknO6xjyuxbIyycKhZCLJLkn+pcVxXZJnDr0HSbYCTgQObzEd3mYrhl7fY9oMxo4b+j+UJEmSNNFGM3NwCPDtqroZ+M8k+66l3T7A3Kp6CfDnwMVVNRvYC1ja2jwWWFRVuwFXAn/T6s+rqv2qai/gBjqJCMApwJWtfh/g+iTPAQ4HXtD6XwMcOSyWXYBfVNXvhtUvAnZr5WcBn6mq5wC/A/5Hq/90i2V3YGvg1a3+TOD/tFieD9w21GlV/R44DjirqmZX1VnAV7viejmwrKr+Yy3vnSRJktRzo0kOjgC+1spfo2tp0TCXVNWvW/la4M1Jjgf2qKo7W/2DwFmt/FU6MxIAuye5KskKOh+ohz7AHwicClBVa6pqNfAyYF/g2iRL2/EzRvE6hvu3qvr+CLG8NMnVLZYDgd2SbAvMrKrzWyz3VdU96+n/NOCoVn4L8KWRGiWZl2RRkkVr7lm9AS9DkiRJGh/r3B+QZHs6H5D3SFLAFKCSvH+E5ncPFarqu0leDBwMnJ7kE1X15RGuqfb3dODQqlqW5BjggHWFBZxRVX+1jjY/BZ6aZNuuxAQ6ScWFw8Z+KJYk04DPAHOq6t9acjNtHeOsVbv+9iQHAvvzx7MbQ+0WAgsBpg7MGh6TJEmSNGnWN3MwF/hKVT2tqgar6inALcCL1nVRkqcBt1fV54Ev0FkSNDTe0AbeP6ezyRk6G4dvS7Ilj/wQfSnwl63PKUlmtLq5SZ7Y6rdv4z2kqu4GzgA+MbQfIclRwDbAZa3ZU5M8b1gsQ4nAr9pei7mtvzuBXyY5tPU1dWiPQpc72+vo9gU6sxJnD+19kCRJkvrV+pKDI4Dzh9Wdy9qXFg05AFiWZAmd/QH/u9XfDezfNjIfSGcTL8CHgauB7wM3dvXzbjrLfFYAi4Fdq+rHwP8CvpNkOXAJMDBCDH8F3AfcnOQnwH8DDquqoW/nbwLenuQG4PHAqVX1W+DzwErgYjrLo4a8CXhXG/MHwJOGjXc5sOvQhuRWdwEwnbUsKZIkSZL6SR7+rDwJgyV3VdX0SRtw7XEMAhe2TccTOc4c4JNVtc6ZliFTB2bVwNELJjIkSZIk9diqkw/u6fhJFlfVnJHO9ctvEmxyksynsyRqxL0GkiRJUr8Z7Y+gjYt+mDUAqKpVEz1rUFUnt70a31t/a0mSJKn3JjU5kCRJktS/TA4kSZIkAe456Ct7zJzBoh5vUJEkSdLmy5kDSZIkSYDJgSRJkqTG5ECSJEkS4J6DvrLi1tUMzr+o12GsV69/uEOSJEkTw5kDSZIkSYDJgSRJkqTG5ECSJEkSYHIgSZIkqTE5kCRJkgT0SXKQ5K5JHOuYJJ/eyD5OTPLy8YpJkiRJ6gc+ynQDVNVxvY5BkiRJGm99MXMwkiRXJJnTyjskWdXKxyQ5L8m3k/wkyT90XfPWJDcnuSbJ58cyQ5Dkje26pUk+l2RK+3d6kpVJViR5b2t7epK5rfyyJEva+dOSTG31q5KckOS6du7Z4/j2SJIkSeOub5OD9ZgNHA7sARye5ClJngx8GHgu8AJg1B/Gkzyn9feCqpoNrAGObOPMrKrdq2oP4EvDrpsGnA4c3s5vAfxlV5NfVdU+wKnAsWsZe16SRUkWrbln9WhDliRJksbdozU5uLSqVlfVfcCPgacB+wNXVtWvq+oPwNlj6O9lwL7AtUmWtuNnAD8DnpHkU0leAfxu2HXPAm6pqpvb8RnAi7vOn9f+LgYGRxq4qhZW1ZyqmjNlmxljCFmSJEkaX/285+ABHk5epg07d39XeQ0b/zoCnFFVf/VHJ5K9gP8KvA34M+AtY+h3KM7xiFGSJEmaUP08c7CKzrf5AHNH0f5a4CVJHk0TU50AAAksSURBVJ9kC+D1YxjrUmBukicCJNk+ydOS7AA8pqrOBf4XsM+w624CBpPs0o7fBFw5hnElSZKkvtEv32Zvk+SXXcefAD4OfD3JPOCi9XVQVbcm+ShwDfBr4EZgbYv4j0lyaNfxc+l8+P9OkscAfwDeDtwLfKnVATxiZqGq7kvyZuDslpBcC3x2fbFKkiRJ/ShV1esYxk2S6VV1V/ugfj5wWlWd3+u4RmvqwKwaOHpBr8NYr1UnH9zrECRJkrSBkiyuqjkjnevnZUUb4vi2oXglcAvwjR7HI0mSJD1q9MuyonFRVSM+LlSSJEnS+m1qMweSJEmSNpDJgSRJkiRgE1tW9Gi3x8wZLHKzryRJknrEmQNJkiRJgMmBJEmSpMbkQJIkSRLgnoO+suLW1QzOX++PQUt9wR/DkyRp0+PMgSRJkiTA5ECSJElSY3IgSZIkCTA5kCRJktRMeHKQ5ElJvpbkp0kWJ/lWkj/ZwL6OT3JsK5+Y5OXjG+3G6Y5PkiRJerSZ0KcVJQlwPnBGVb2h1e0F7ATcPIprU1UPjnS+qo4b53AlSZKkzdpEzxy8FPhDVX12qKKqlgFLklya5LokK5IcApBkMMlNSb4MrASekuRDSW5O8j3gWUP9JDk9ydxWflmSJa2v05JMbfWrkpyUZGmSRUn2SXJxm8V4W1df709ybZLlSU7oqv9wi+d7Sf6pa9biL1r7ZUnOTbJN94tO8swk13Udz+o+liRJkvrRRCcHuwOLR6i/Dzisqvahk0D8Y5spAJgFfKaqdgN2AN4AzAZeBew3vKMk04DTgcOrag86syF/2dXkF1U1G7iqtZsLPBc4oV1/UBtz/zbOvklenGQ/4PXAXsArgTldfZ5XVftV1V7ADcBbu2Oqqp8Cq5PMblVvBr609rdJkiRJ6r1e/QhagI8meTHwIDCTzlIjgJ9X1Y9a+UXA+VV1D0CSC0bo61nALVU1tEzpDODtwIJ2PHTNCmB6Vd0J3Jnk/iSPAw5q/5a0dtPpJAvbAv9cVfcB9yX5ZteYuyf5W+Bxrf3FI8T1BeDNSd4HHE4n+fjjNyKZB8wDmLLdjiM1kSRJkibFRM8cXA/sO0L9kcCOwL7tW/3bgWnt3N3jHMP97e+DXeWh4y3oJConVdXs9m+Xqvrievo8HXhHm6k4gYdj73YunRmHVwOLq+o/R+qoqhZW1ZyqmjNlmxmjflGSJEnSeJvo5OAyYGr7dhyAJHsCTwPuqKo/JHlpOx7Jd4FDk2ydZFvgNSO0uQkYTLJLO34TcOUYYrwYeEuS6S2+mUmeCHwfeE2Sae3cq7uu2Ra4LcmWdBKdP9JmHC4GTsUlRZIkSXoUmNBlRVVVSQ4DFiT5IJ29BquA44FTkqwAFgE3ruX665KcBSwD7gCuHaHNfUneDJydZIvW5rPD260jxu8keQ7ww7bt4S7gjVV1bVvGtJzOzMYKYHW77MPA1cB/tL/brqX7M4HDgO+MNh5JkiSpV1JVvY6hbyWZXlV3tacRfReYV1WjfupQe7rRjKr68GjaTx2YVQNHL1h/Q6kPrDr54F6HIEmSNkCSxVU1Z6RzvdqQ/GixMMmudPYUnDHGxOB84JnAgRMVnCRJkjSeTA7Woar+fCOuPWw8Y5EkSZIm2kRvSJYkSZL0KGFyIEmSJAlwWVFf2WPmDBa5yVOSJEk94syBJEmSJMDkQJIkSVJjciBJkiQJMDmQJEmS1JgcSJIkSQJMDiRJkiQ1JgeSJEmSAJMDSZIkSY3JgSRJkiTA5ECSJElSY3IgSZIkCTA5kCRJktSkqnodg5okdwI39ToObRJ2AH7V6yC0SfBe0njyftJ48V7aOE+rqh1HOrHFZEeidbqpqub0Ogg9+iVZ5L2k8eC9pPHk/aTx4r00cVxWJEmSJAkwOZAkSZLUmBz0l4W9DkCbDO8ljRfvJY0n7yeNF++lCeKGZEmSJEmAMweSJEmSGpODPpDkFUluSvKvSeb3Oh71vySnJbkjycquuu2TXJLkJ+3v41t9kpzS7q/lSfbpXeTqN0mekuTyJD9Ocn2Sd7d67yeNSZJpSa5JsqzdSye0+qcnubrdM2cl2arVT23H/9rOD/YyfvWfJFOSLElyYTv2XpoEJgc9lmQK8H+AVwK7Akck2bW3UelR4HTgFcPq5gOXVtUs4NJ2DJ17a1b7Nw84dZJi1KPDA8D/rKpdgecCb2//DfJ+0ljdDxxYVXsBs4FXJHku8PfAJ6tqF+A3wFtb+7cCv2n1n2ztpG7vBm7oOvZemgQmB723P/CvVfWzqvo98DXgkB7HpD5XVd8Ffj2s+hDgjFY+Azi0q/7L1fEj4HFJBiYnUvW7qrqtqq5r5Tvp/B/xTLyfNEbtnrirHW7Z/hVwIHBOqx9+Lw3dY+cAL0uSSQpXfS7JzsDBwBfacfBemhQmB703E/i3ruNftjpprHaqqtta+d+BnVrZe0yj0qbi9wauxvtJG6AtA1kK3AFcAvwU+G1VPdCadN8vD91L7fxq4AmTG7H62ALgA8CD7fgJeC9NCpMDaRNUnceQ+SgyjVqS6cC5wHuq6nfd57yfNFpVtaaqZgM705kZf3aPQ9KjUJJXA3dU1eJex7I5MjnovVuBp3Qd79zqpLG6fWh5R/t7R6v3HtM6JdmSTmJwZlWd16q9n7TBquq3wOXA8+gsPduineq+Xx66l9r5GcB/TnKo6k8vAF6bZBWd5dYHAv8b76VJYXLQe9cCs9oO/K2ANwAX9DgmPTpdABzdykcD/9xVf1R7ysxzgdVdy0W0mWvrcr8I3FBVn+g65f2kMUmyY5LHtfLWwH+hs4flcmBuazb8Xhq6x+YCl5U/viSgqv6qqnauqkE6n4suq6oj8V6aFP4IWh9I8io6a+umAKdV1d/1OCT1uST/BBwA7ADcDvwN8A3g68BTgZ8Df1ZVv24f/j5N5+lG9wBvrqpFvYhb/SfJC4GrgBU8vLb3r+nsO/B+0qgl2ZPOptApdL58/HpVnZjkGXS+/d0eWAK8saruTzIN+AqdfS6/Bt5QVT/rTfTqV0kOAI6tqld7L00OkwNJkiRJgMuKJEmSJDUmB5IkSZIAkwNJkiRJjcmBJEmSJMDkQJIkSVJjciBJkiQJMDmQJEmS1JgcSJIkSQLg/wHoUm95s3ANtAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 864x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(12,5))\n",
    "plt.barh(y=CATEGORIES, width=test_df[CATEGORIES].sum(axis=0))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that pathologies like Airspace Opacity, Pleural Effusion, and Edema are present in many of the reports while Lung Lesion and Pneumonia are not as common in this dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "E76U0d7AVq6A"
   },
   "source": [
    "<a name=\"1-1\"></a>\n",
    "### 1 Text Matching\n",
    "One of the easiest and surprisingly effective ways to label our dataset is to search for presence of different keywords in the impression text. \n",
    "\n",
    "We have prepared a list of relevant keywords for each pathology to use in detecting the presence of each label. \n",
    "- We can access these keywords for each label by calling the `get_mention_keywords(observation)` function.\n",
    "\n",
    "Here's an example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "wiRnTs6BVq6B",
    "outputId": "794c7926-1d70-4473-cec1-a4844e74de43"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Related keywords for Airspace Opacity are:\n",
      "opaci, decreased translucency, increased density, airspace disease, air-space disease, air space disease, infiltrate, infiltration, interstitial marking, interstitial pattern, interstitial lung, reticular pattern, reticular marking, reticulation, parenchymal scarring, peribronchial thickening, wall thickening, scar\n"
     ]
    }
   ],
   "source": [
    "cat = CATEGORIES[2]\n",
    "related_keywords = get_mention_keywords(cat)\n",
    "print(\"Related keywords for {} are:\\n{}\".format(cat, ', '.join(related_keywords)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1Y4spu0NVq6D"
   },
   "source": [
    "### Get Labels\n",
    "\n",
    "We will use this simple approach to start constructing labels for each report.\n",
    "- The solution takes as input a medical report (as an array of sentences)\n",
    "- It returns a dictionary that maps each category to a boolean value, which indicates the presence or absence of the abnormality.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "_DKhgVrPVq6D"
   },
   "outputs": [],
   "source": [
    "def get_labels(sentence_l):\n",
    "    \"\"\"\n",
    "    Returns a dictionary that indicates presence of each category (from CATEGORIES) \n",
    "    in the given sentences.\n",
    "    hint: loop over the sentences array and use get_mention_keywords() function.\n",
    "    \n",
    "    Args: \n",
    "        sentence_l (array of strings): array of strings representing impression section\n",
    "    Returns:\n",
    "        observation_d (dict): dictionary mapping observation from CATEGORIES array to boolean value\n",
    "    \"\"\"\n",
    "    observation_d = {}\n",
    "    # loop through each category\n",
    "    for cat in CATEGORIES:\n",
    "        \n",
    "        # Initialize the observations for all categories to be False\n",
    "        observation_d[cat] = False\n",
    "\n",
    "    # For each sentence in the list:\n",
    "    for s in sentence_l: \n",
    "        \n",
    "        # Set the characters to all lowercase, for consistent string matching\n",
    "        s = s.lower()\n",
    "        \n",
    "        # for each category\n",
    "        for cat in CATEGORIES: \n",
    "            \n",
    "            # for each phrase that is related to the keyword (use the given function)\n",
    "            for phrase in get_mention_keywords(cat): \n",
    "            \n",
    "                # make the phrase all lowercase for consistent string matching\n",
    "                phrase = phrase.lower()\n",
    "                \n",
    "                # check if the phrase appears in the sentence\n",
    "                if phrase in s: \n",
    "                    observation_d[cat] = True\n",
    "\n",
    "    \n",
    "    return observation_d"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test Case\n",
      "\n",
      "Test Sentences:\n",
      "\n",
      "Diffuse Reticular Pattern, which can be seen with an atypical infection or chronic fibrotic change.\n",
      "no Focal Consolidation.\n",
      "\n",
      "Retrieved labels: \n",
      "Airspace Opacity : True\n",
      "Atelectasis : False\n",
      "Cardiomegaly : False\n",
      "Consolidation : True\n",
      "Edema : False\n",
      "Fracture : False\n",
      "Lung Lesion : False\n",
      "Pleural Effusion : False\n",
      "Pleural Other : False\n",
      "Pneumonia : True\n",
      "Pneumothorax : False\n",
      "\n",
      "Expected labels: \n",
      "Airspace Opacity : True\n",
      "Atelectasis : False\n",
      "Cardiomegaly : False\n",
      "Consolidation : True\n",
      "Edema : False\n",
      "Fracture : False\n",
      "Lung Lesion : False\n",
      "Pleural Effusion : False\n",
      "Pleural Other : False\n",
      "Pneumonia : True\n",
      "Pneumothorax : False\n",
      "\n",
      "Labels match for Cardiomegaly!\n",
      "Labels match for Lung Lesion!\n",
      "Labels match for Airspace Opacity!\n",
      "Labels match for Edema!\n",
      "Labels match for Consolidation!\n",
      "Labels match for Pneumonia!\n",
      "Labels match for Atelectasis!\n",
      "Labels match for Pneumothorax!\n",
      "Labels match for Pleural Effusion!\n",
      "Labels match for Pleural Other!\n",
      "Labels match for Fracture!\n"
     ]
    }
   ],
   "source": [
    "print(\"Test Case\")\n",
    "\n",
    "test_sentences = [\"Diffuse Reticular Pattern, which can be seen with an atypical infection or chronic fibrotic change.\", \n",
    "                  \"no Focal Consolidation.\"]\n",
    "print(\"\\nTest Sentences:\\n\")\n",
    "for s in test_sentences:\n",
    "    print(s)\n",
    "\n",
    "print()\n",
    "retrieved_labels = get_labels(test_sentences)\n",
    "print(\"Retrieved labels: \")\n",
    "\n",
    "for key, value in sorted(retrieved_labels.items(), key=lambda x: x[0]): \n",
    "    print(\"{} : {}\".format(key, value))\n",
    "print()\n",
    "\n",
    "print(\"Expected labels: \")\n",
    "expected_labels = {'Cardiomegaly': False, 'Lung Lesion': False, 'Airspace Opacity': True, 'Edema': False, 'Consolidation': True, 'Pneumonia': True, 'Atelectasis': False, 'Pneumothorax': False, 'Pleural Effusion': False, 'Pleural Other': False, 'Fracture': False}\n",
    "for key, value in sorted(expected_labels.items(), key=lambda x: x[0]): \n",
    "    print(\"{} : {}\".format(key, value))\n",
    "print()\n",
    "\n",
    "for category in CATEGORIES:\n",
    "    if category not in retrieved_labels:\n",
    "        print(f'Category {category} not found in retrieved labels. Please check code.')\n",
    "    \n",
    "    elif retrieved_labels[category] == expected_labels[category]:\n",
    "        print(f'Labels match for {category}!')\n",
    "    \n",
    "    else:\n",
    "        print(f'Labels mismatch for {category}. Please check code.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Note\n",
    "\n",
    "We notice that the second sentence is \"no Focal Consolidation\", and that the consolidation label is set to True.  We'll come back to this shortly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "mceW_rpsVq6J"
   },
   "source": [
    "### 2 Evaluating The Performance\n",
    "In order to evaluate the performance of our labeler, we will use a metric called the [F1 score](https://en.wikipedia.org/wiki/F1_score). \n",
    "- The F1 score is the harmonic mean of precision and recall. \n",
    "- This score is a common metric that is used in information retrieval problems. \n",
    "\n",
    "The reason that we care both about precision and recall is that only a small subset of the labels for each report are positive and the rest are negative. \n",
    "- So a traditional metric such as accuracy could be very high if we just report every label as False. \n",
    "- Precision and recall (summarized in the F1 score) help we measure our classification's performance for both positive cases as well as negative cases."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$F_1 = \\left( \\frac{2}{ \\frac{1}{recall} + \\frac{1}{precision} }\\right) = 2 \\times \\frac{precision \\times recall}{precision + recall}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "a9ZgyZxpVq6J"
   },
   "source": [
    "We've implemented a utility function called `get_f1_table()` below to calculate our function's performance on the whole test set. \n",
    "- This function will take advantage of modules from the `bioc` and `negbio` python packages to intelligently split our paragraph to sentences and then apply our function on each sentence. \n",
    "- It then returns a table with the calculated F1 score for each category. \n",
    "- We can see how this function is implemented in `util.py`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Vs6cU5i_Vq6K",
    "outputId": "e2a2ed94-c302-41de-d5b1-39d47d16ce70"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Label</th>\n",
       "      <th>F1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Cardiomegaly</td>\n",
       "      <td>0.718</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Lung Lesion</td>\n",
       "      <td>0.641</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Airspace Opacity</td>\n",
       "      <td>0.923</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Edema</td>\n",
       "      <td>0.708</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Consolidation</td>\n",
       "      <td>0.270</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Pneumonia</td>\n",
       "      <td>0.369</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Atelectasis</td>\n",
       "      <td>0.646</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Pneumothorax</td>\n",
       "      <td>0.218</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Pleural Effusion</td>\n",
       "      <td>0.722</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Pleural Other</td>\n",
       "      <td>0.667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Fracture</td>\n",
       "      <td>0.689</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Average</td>\n",
       "      <td>0.597</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               Label     F1\n",
       "0       Cardiomegaly  0.718\n",
       "1        Lung Lesion  0.641\n",
       "2   Airspace Opacity  0.923\n",
       "3              Edema  0.708\n",
       "4      Consolidation  0.270\n",
       "5          Pneumonia  0.369\n",
       "6        Atelectasis  0.646\n",
       "7       Pneumothorax  0.218\n",
       "8   Pleural Effusion  0.722\n",
       "9      Pleural Other  0.667\n",
       "10          Fracture  0.689\n",
       "11           Average  0.597"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_f1_table(get_labels, test_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "YkHxb06BVq6N"
   },
   "source": [
    "We can see that Airspace Opacity has the highest F1 score at 0.923 while Pneumothorax has the lowest at 0.218. \n",
    "\n",
    "These numbers are actually not too bad given the simplicity of this rules-based implemention, but we can do a lot better."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3 Cleanup\n",
    "\n",
    "First of all, the text that we use as input is very messy. \n",
    "- We should be able to do better by doing some simple pre-processing. \n",
    "- We have already implemented a `clean` function which does basic text cleanup. Among other things, it converts pattern such as \"and/or\" to \"or\", replaces repeated whitespace with just one space, and removes redundant punctuation, etc. \n",
    "\n",
    "Example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "raw text: \n",
      "\n",
      "\n",
      " \n",
      "1.  bibasilar opacities, without significant change from the prior \n",
      "study, likely representing bilateral pleural effusions with \n",
      "atelectasis and/or pneumonia.\n",
      " \n",
      "2.  interval decrease in focal opacification within the right upper \n",
      "lung.\n",
      " \n",
      "3.  lines and tubes are unchanged. the distal tip of enteric tube \n",
      "remains coiled within the stomach.\n",
      " \n",
      " \n",
      "\n",
      "cleaned text: \n",
      "\n",
      "1. bibasilar opacities, without significant change from the prior study, likely representing bilateral pleural effusions with atelectasis or pneumonia. 2. interval decrease in focal opacification within the right upper lung. 3. lines and tubes are unchanged. the distal tip of enteric tube remains coiled within the stomach.\n"
     ]
    }
   ],
   "source": [
    "raw_text = test_df.loc[28, 'Report Impression']\n",
    "print(\"raw text: \\n\\n\" + raw_text)\n",
    "print(\"cleaned text: \\n\\n\" + clean(raw_text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note how the \"and/or\" in observation 1 was transformed into \"or\", and the standardization of the whitespace. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have implemented a function for this to add to the pipeline, by passing the `cleanup=True` flag to our `get_f1_table()` function. Let's see if cleaning up the text can improve performance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "EvLEUqT5Vq6N",
    "outputId": "30bee7a4-759c-4847-d785-d72bccaf0831"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Label</th>\n",
       "      <th>F1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Cardiomegaly</td>\n",
       "      <td>0.725</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Lung Lesion</td>\n",
       "      <td>0.641</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Airspace Opacity</td>\n",
       "      <td>0.922</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Edema</td>\n",
       "      <td>0.707</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Consolidation</td>\n",
       "      <td>0.270</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Pneumonia</td>\n",
       "      <td>0.369</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Atelectasis</td>\n",
       "      <td>0.646</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Pneumothorax</td>\n",
       "      <td>0.218</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Pleural Effusion</td>\n",
       "      <td>0.722</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Pleural Other</td>\n",
       "      <td>0.687</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Fracture</td>\n",
       "      <td>0.689</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Average</td>\n",
       "      <td>0.600</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               Label     F1\n",
       "0       Cardiomegaly  0.725\n",
       "1        Lung Lesion  0.641\n",
       "2   Airspace Opacity  0.922\n",
       "3              Edema  0.707\n",
       "4      Consolidation  0.270\n",
       "5          Pneumonia  0.369\n",
       "6        Atelectasis  0.646\n",
       "7       Pneumothorax  0.218\n",
       "8   Pleural Effusion  0.722\n",
       "9      Pleural Other  0.687\n",
       "10          Fracture  0.689\n",
       "11           Average  0.600"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_f1_table(get_labels, test_df, cleanup=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see a very modest improvement in Cardiomegaly and Pleural Other, but overall the impact is fairly low. In the next step we'll try a change which has a much bigger impact."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "GrKV6fcSVq6P"
   },
   "source": [
    "### 4 Finding Negative Mentions\n",
    "\n",
    "So far we have just been treating the presence of a keyword in the impression section as a signal for the presence of that condition in the report. This approach currently has a big problem: \n",
    "- This is ignoring negations! \n",
    "- For example, consider the following report: \"No sign of consolidation\". Currently, our matching approach calls this sentence a positive case of consolidation!\n",
    "\n",
    "We need to:\n",
    "- Use a boolean \"flag\" to indicate whether a negation like \"no\" or \"not\" appears in a sentence.\n",
    "- Only set a label to `True` if the word \"not\" or \"no\" did not appear in the sentence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "kHHsbWolVq6Q"
   },
   "outputs": [],
   "source": [
    "def get_labels_negative_aware(sentence_l):\n",
    "    \"\"\"\n",
    "    Returns a dictionary that indicates presence of categories in\n",
    "    sentences within the impression section of the report.\n",
    "    Only set a label to True if no 'negative words' appeared in the sentence.\n",
    "    hint: loop over the sentences array and use get_mention_keywords() function.\n",
    "    \n",
    "    Args: \n",
    "        sentence_l (array of strings): array of strings representing impression section\n",
    "    Returns:\n",
    "        observation_d (dict): dictionary mapping observation from CATEGORIES array to boolean value\n",
    "    \"\"\"\n",
    "    # Notice that all of the negative words are written in lowercase\n",
    "    negative_word_l = [\"no\", \"not\", \"doesn't\", \"does not\", \"have not\", \"can not\", \"can't\", \"n't\"]\n",
    "    observation_d = {}\n",
    "    \n",
    "    \n",
    "    # Initialize the observation dictionary \n",
    "    # so that all categories are not marked present.\n",
    "    for cat in CATEGORIES: \n",
    "        \n",
    "        # Initialize category to not present.\n",
    "        observation_d[cat] = False\n",
    "\n",
    "    # Loop through each sentence in the list of sentences\n",
    "    for s in sentence_l: \n",
    "        \n",
    "        # make the sentence all lowercase\n",
    "        s = s.lower()\n",
    "        \n",
    "        # Initialize the flag to indicate no negative mentions (yet)\n",
    "        negative_flag = False\n",
    "        \n",
    "        # Go through all the negative words in the list\n",
    "        for neg in negative_word_l: \n",
    "            \n",
    "            # Check if the word is a substring in the sentence\n",
    "            if neg in s: \n",
    "                # set the flag to indicate a negative mention\n",
    "                negative_flag = True\n",
    "                \n",
    "                # Once a single negative mention is found,\n",
    "                # we can stop checking the remaining negative words\n",
    "                break \n",
    "\n",
    "        # When a negative word was not found in the sentence,\n",
    "        # check for the presence of the diseases\n",
    "        if not negative_flag: \n",
    "            \n",
    "            # Loop through the categories list\n",
    "            for cat in CATEGORIES: \n",
    "                \n",
    "                # Loop through each phrase that indicates this category\n",
    "                for phrase in get_mention_keywords(cat): \n",
    "\n",
    "                        # make the phrase all lowercase\n",
    "                        phrase = phrase.lower()\n",
    "                        \n",
    "                        # Check if the phrase is a substring in the sentence\n",
    "                        if phrase in s: \n",
    "                            \n",
    "                            # Set the observation dictionary\n",
    "                            # to indicate the presence of this category\n",
    "                            observation_d[cat] = True\n",
    "\n",
    "\n",
    "    \n",
    "    return observation_d"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test Case\n",
      "\n",
      "Test Sentences:\n",
      "\n",
      "Diffuse Reticular pattern, which can be seen with an atypical infection or chronic fibrotic change.\n",
      "No Focal Consolidation.\n",
      "\n",
      "Retrieved labels: \n",
      "Airspace Opacity : True\n",
      "Atelectasis : False\n",
      "Cardiomegaly : False\n",
      "Consolidation : False\n",
      "Edema : False\n",
      "Fracture : False\n",
      "Lung Lesion : False\n",
      "Pleural Effusion : False\n",
      "Pleural Other : False\n",
      "Pneumonia : True\n",
      "Pneumothorax : False\n",
      "\n",
      "Expected labels: \n",
      "Airspace Opacity : True\n",
      "Atelectasis : False\n",
      "Cardiomegaly : False\n",
      "Consolidation : False\n",
      "Edema : False\n",
      "Fracture : False\n",
      "Lung Lesion : False\n",
      "Pleural Effusion : False\n",
      "Pleural Other : False\n",
      "Pneumonia : True\n",
      "Pneumothorax : False\n",
      "\n",
      "Test Results:\n",
      "Labels match for Cardiomegaly!\n",
      "Labels match for Lung Lesion!\n",
      "Labels match for Airspace Opacity!\n",
      "Labels match for Edema!\n",
      "Labels match for Consolidation!\n",
      "Labels match for Pneumonia!\n",
      "Labels match for Atelectasis!\n",
      "Labels match for Pneumothorax!\n",
      "Labels match for Pleural Effusion!\n",
      "Labels match for Pleural Other!\n",
      "Labels match for Fracture!\n"
     ]
    }
   ],
   "source": [
    "print(\"Test Case\")\n",
    "\n",
    "test_sentences = [\"Diffuse Reticular pattern, which can be seen with an atypical infection or chronic fibrotic change.\", \n",
    "                  \"No Focal Consolidation.\"]\n",
    "print(\"\\nTest Sentences:\\n\")\n",
    "for s in test_sentences:\n",
    "    print(s)\n",
    "\n",
    "print()\n",
    "retrieved_labels = get_labels_negative_aware(test_sentences)\n",
    "print(\"Retrieved labels: \")\n",
    "\n",
    "for key, value in sorted(retrieved_labels.items(), key=lambda x: x[0]): \n",
    "    print(\"{} : {}\".format(key, value))\n",
    "print()\n",
    "\n",
    "print(\"Expected labels: \")\n",
    "expected_labels = {'Cardiomegaly': False, 'Lung Lesion': False, 'Airspace Opacity': True, 'Edema': False, 'Consolidation': False, 'Pneumonia': True, 'Atelectasis': False, 'Pneumothorax': False, 'Pleural Effusion': False, 'Pleural Other': False, 'Fracture': False}\n",
    "for key, value in sorted(expected_labels.items(), key=lambda x: x[0]): \n",
    "    print(\"{} : {}\".format(key, value))\n",
    "print()\n",
    "\n",
    "print(\"Test Results:\")\n",
    "for category in CATEGORIES:\n",
    "    if category not in retrieved_labels:\n",
    "        print(f'Category {category} not found in retrieved labels. Please check code.')\n",
    "    \n",
    "    elif retrieved_labels[category] == expected_labels[category]:\n",
    "        print(f'Labels match for {category}!')\n",
    "    \n",
    "    else:\n",
    "        print(f'Labels mismatch for {category}. Please check code.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we implemented this correctly, we should have Consolidation set to **False**, because the test sentence contains \"No Focal Consolidation.  \n",
    "\n",
    "With the basic labeling method `get_labels()`, this set Consolidation to True, because it didn't look for 'negative' words."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "r72mTZ4TVq6w"
   },
   "source": [
    "Check how this changes our aggregate performance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ZFUKz7ObVq6x",
    "outputId": "4e192adc-028a-4b53-a000-d51e84e75ec6"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Label</th>\n",
       "      <th>F1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Cardiomegaly</td>\n",
       "      <td>0.807</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Lung Lesion</td>\n",
       "      <td>0.465</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Airspace Opacity</td>\n",
       "      <td>0.905</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Edema</td>\n",
       "      <td>0.795</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Consolidation</td>\n",
       "      <td>0.424</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Pneumonia</td>\n",
       "      <td>0.463</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Atelectasis</td>\n",
       "      <td>0.642</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Pneumothorax</td>\n",
       "      <td>0.656</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Pleural Effusion</td>\n",
       "      <td>0.859</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Pleural Other</td>\n",
       "      <td>0.621</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Fracture</td>\n",
       "      <td>0.864</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Average</td>\n",
       "      <td>0.682</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               Label     F1\n",
       "0       Cardiomegaly  0.807\n",
       "1        Lung Lesion  0.465\n",
       "2   Airspace Opacity  0.905\n",
       "3              Edema  0.795\n",
       "4      Consolidation  0.424\n",
       "5          Pneumonia  0.463\n",
       "6        Atelectasis  0.642\n",
       "7       Pneumothorax  0.656\n",
       "8   Pleural Effusion  0.859\n",
       "9      Pleural Other  0.621\n",
       "10          Fracture  0.864\n",
       "11           Average  0.682"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_f1_table(get_labels_negative_aware, test_df, cleanup=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "7JoYejfYVq6z"
   },
   "source": [
    "We should see a generally significant boost in the F1 score across the board. \n",
    "- With the `get_labels()` method, Pneumothorax has an F1 score of 0.218.\n",
    "- With the `get_labels_negative_aware()` method, Pneumothorax has an F1 score of 0.656."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the next section, we'll see how to use some more sophisticated NLP tools to do even better!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "FLNojinBVq61"
   },
   "source": [
    "<a name=\"1-5\"></a>\n",
    "### 5 Dependency Parsing\n",
    "Our heuristic for detecting negation is still relatively simple and we can think of some examples that would fool it. To improve performance even further, we'll leverage a more sophisticated approach using a [dependency parser](https://nlp.stanford.edu/software/nndep.html). \n",
    "\n",
    "\n",
    "\n",
    "A dependency parser extracts the underlying structure of the text. \n",
    "- Using a dependency parser, we can detect the relationship between different words in the sentence and understand which word is affected by negative phrases. \n",
    "\n",
    "Implementations of dependency parsers are very complex, but luckily there are some great off-the-shelf tools to do this. \n",
    "- One example is [NegBio](https://github.com/ncbi-nlp/NegBio), a package specifically designed for finding negative and uncertain mentions in X-ray radiology reports. \n",
    "- In addition to detecting negations, `negbio` can be configured to use a dependency parser that has been specifically trained for biomedical text. \n",
    "- This increases our performance given that biomedical text is full of acronyms, nomenclature and medical jargon.\n",
    "\n",
    "We've configured `negbio` to:\n",
    "1. Parse sentences using the Bllip parser trained using [David McClosky’s biomedical model](https://nlp.stanford.edu/~mcclosky/biomedical.html)\n",
    "2. Compute the universal dependency graph of each sentence using Stanford [CoreNLP](https://stanfordnlp.github.io/CoreNLP/)\n",
    "3. Use a collection of dependency graph rules to detect the negative or positive presence of a phrase based on configurable patterns. It also comes packaged with the negation and uncertainty rules from the CheXpert project.\n",
    "\n",
    "The catch with this more involved method is that it takes a while (~1.5h on a fast laptop) to run the detection pipeline on the 1000-sample dataset. \n",
    "- Hence, we can do our comparison on a smaller subset of the data (200 samples). \n",
    "\n",
    "Run the following cells to get predictions using the `negbio` engine \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "vjyK4jdHVq61"
   },
   "outputs": [],
   "source": [
    "sampled_test = test_df.sample(200,random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code for applying the dependency is implemented with the function `get_negbio_preds`. \n",
    "\n",
    "Run the next cell to extract predictions from the sampled test set. \n",
    "\n",
    "### Note \n",
    "- This should take about **5 minutes** to run.\n",
    "- If we want to run it again, we may need to restart the kernel (close and re-open the assignment). \n",
    "    - If we run it a second time without restarting the kernel we'll get an error message like\n",
    "    \n",
    "    ```RuntimeError: Parser is already loaded and can only be loaded once.```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "XRQ9PvZOVq7p",
    "outputId": "8d34d299-d641-4f60-a497-17115efaad89"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 200/200 [05:52<00:00,  1.76s/it]\n"
     ]
    }
   ],
   "source": [
    "negbio_preds = get_negbio_preds(sampled_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next let's calculate the new F1 scores to see the dependency parser does. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "JcSCLZqFVq7r"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Label</th>\n",
       "      <th>F1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Cardiomegaly</td>\n",
       "      <td>0.877</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Lung Lesion</td>\n",
       "      <td>0.889</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Airspace Opacity</td>\n",
       "      <td>0.952</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Edema</td>\n",
       "      <td>0.808</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Consolidation</td>\n",
       "      <td>0.370</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Pneumonia</td>\n",
       "      <td>0.467</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Atelectasis</td>\n",
       "      <td>0.725</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Pneumothorax</td>\n",
       "      <td>0.326</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Pleural Effusion</td>\n",
       "      <td>0.875</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Pleural Other</td>\n",
       "      <td>0.588</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Fracture</td>\n",
       "      <td>0.783</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Average</td>\n",
       "      <td>0.696</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               Label     F1\n",
       "0       Cardiomegaly  0.877\n",
       "1        Lung Lesion  0.889\n",
       "2   Airspace Opacity  0.952\n",
       "3              Edema  0.808\n",
       "4      Consolidation  0.370\n",
       "5          Pneumonia  0.467\n",
       "6        Atelectasis  0.725\n",
       "7       Pneumothorax  0.326\n",
       "8   Pleural Effusion  0.875\n",
       "9      Pleural Other  0.588\n",
       "10          Fracture  0.783\n",
       "11           Average  0.696"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "calculate_f1(sampled_test, negbio_preds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "QKSLuA68Vq7t"
   },
   "source": [
    "And finally, let's compare all methods side by side!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "9rnihcg8Vq7u"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Label</th>\n",
       "      <th>F1 Basic</th>\n",
       "      <th>F1 Cleaned</th>\n",
       "      <th>F1 Negative Basic</th>\n",
       "      <th>F1 Negbio</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Cardiomegaly</td>\n",
       "      <td>0.758</td>\n",
       "      <td>0.771</td>\n",
       "      <td>0.815</td>\n",
       "      <td>0.877</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Lung Lesion</td>\n",
       "      <td>0.762</td>\n",
       "      <td>0.762</td>\n",
       "      <td>0.333</td>\n",
       "      <td>0.889</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Airspace Opacity</td>\n",
       "      <td>0.948</td>\n",
       "      <td>0.948</td>\n",
       "      <td>0.922</td>\n",
       "      <td>0.952</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Edema</td>\n",
       "      <td>0.759</td>\n",
       "      <td>0.759</td>\n",
       "      <td>0.771</td>\n",
       "      <td>0.808</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Consolidation</td>\n",
       "      <td>0.260</td>\n",
       "      <td>0.260</td>\n",
       "      <td>0.409</td>\n",
       "      <td>0.370</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Pneumonia</td>\n",
       "      <td>0.359</td>\n",
       "      <td>0.359</td>\n",
       "      <td>0.467</td>\n",
       "      <td>0.467</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Atelectasis</td>\n",
       "      <td>0.605</td>\n",
       "      <td>0.605</td>\n",
       "      <td>0.579</td>\n",
       "      <td>0.725</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Pneumothorax</td>\n",
       "      <td>0.167</td>\n",
       "      <td>0.167</td>\n",
       "      <td>0.600</td>\n",
       "      <td>0.326</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Pleural Effusion</td>\n",
       "      <td>0.730</td>\n",
       "      <td>0.730</td>\n",
       "      <td>0.839</td>\n",
       "      <td>0.875</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Pleural Other</td>\n",
       "      <td>0.588</td>\n",
       "      <td>0.556</td>\n",
       "      <td>0.533</td>\n",
       "      <td>0.588</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Fracture</td>\n",
       "      <td>0.741</td>\n",
       "      <td>0.741</td>\n",
       "      <td>0.889</td>\n",
       "      <td>0.783</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Average</td>\n",
       "      <td>0.607</td>\n",
       "      <td>0.605</td>\n",
       "      <td>0.651</td>\n",
       "      <td>0.696</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               Label  F1 Basic  F1 Cleaned  F1 Negative Basic  F1 Negbio\n",
       "0       Cardiomegaly     0.758       0.771              0.815      0.877\n",
       "1        Lung Lesion     0.762       0.762              0.333      0.889\n",
       "2   Airspace Opacity     0.948       0.948              0.922      0.952\n",
       "3              Edema     0.759       0.759              0.771      0.808\n",
       "4      Consolidation     0.260       0.260              0.409      0.370\n",
       "5          Pneumonia     0.359       0.359              0.467      0.467\n",
       "6        Atelectasis     0.605       0.605              0.579      0.725\n",
       "7       Pneumothorax     0.167       0.167              0.600      0.326\n",
       "8   Pleural Effusion     0.730       0.730              0.839      0.875\n",
       "9      Pleural Other     0.588       0.556              0.533      0.588\n",
       "10          Fracture     0.741       0.741              0.889      0.783\n",
       "11           Average     0.607       0.605              0.651      0.696"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "basic = get_f1_table(get_labels, sampled_test).rename(columns={\"F1\": \"F1 Basic\"})\n",
    "clean_basic = get_f1_table(get_labels, sampled_test, cleanup=True).rename(columns={\"F1\": \"F1 Cleaned\"})\n",
    "negated_basic = get_f1_table(get_labels_negative_aware, sampled_test, cleanup=True).rename(columns={\"F1\": \"F1 Negative Basic\"})\n",
    "negated_negbio = calculate_f1(sampled_test, negbio_preds).rename(columns={\"F1\": \"F1 Negbio\"})\n",
    "\n",
    "joined_preds = basic.merge(clean_basic, on=\"Label\")\n",
    "joined_preds = joined_preds.merge(negated_basic, on=\"Label\")\n",
    "joined_preds = joined_preds.merge(negated_negbio,  on=\"Label\")\n",
    "\n",
    "joined_preds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We\n",
    " see an improvement using the heavier NLP machinery. The F1 for categories, such as Airspace Opacity, Cardiomegaly, Edema, and Pleural Effusion are already fairly high, while others leave something to be desired."
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "include_colab_link": true,
   "name": "C3M3_Assignment.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "coursera": {
   "schema_names": [
    "AI4MC3-2"
   ]
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4-final"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}