Assessment_of_freeform_fields.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "73c3e079",
   "metadata": {},
   "source": [
    "# Assessing free-form text fields"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11214093",
   "metadata": {},
   "source": [
    "Since free-form text fields are present in the dataset, we decided to look manually into their contents. This serves 2 purposes: first, we want to make sure that no sensitive or identifying information was inserted in those fields to sanitize the database. Second, we hope to find information that has not been coded in the structured columns."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "479a7efd",
   "metadata": {},
   "source": [
    "## loading the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "8bacea51",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "264cb31b",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "df = pd.read_csv('full-310k - Copy.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "817af58d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# list of free text columns, per data dictionary\n",
    "columns = 'hwrep_dx_oth', 'hwrep_comment_final', 'hwrep_tx_oth', 's_oth_sympt_entered'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "81735808",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>occurences</th>\n",
       "      <th>word</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>!</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>#</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>&amp;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>(bacterial)so</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>(tachycardia</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>905</th>\n",
       "      <td>71</td>\n",
       "      <td>not</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>906</th>\n",
       "      <td>88</td>\n",
       "      <td>child</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>907</th>\n",
       "      <td>96</td>\n",
       "      <td>for</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>908</th>\n",
       "      <td>103</td>\n",
       "      <td>was</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>909</th>\n",
       "      <td>139</td>\n",
       "      <td>the</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>910 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     occurences           word\n",
       "0             1              !\n",
       "1             1              #\n",
       "2             1              &\n",
       "3             1  (bacterial)so\n",
       "4             1   (tachycardia\n",
       "..          ...            ...\n",
       "905          71            not\n",
       "906          88          child\n",
       "907          96            for\n",
       "908         103            was\n",
       "909         139            the\n",
       "\n",
       "[910 rows x 2 columns]"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus = pd.DataFrame({col: df[col] for col in columns})\n",
    "import collections\n",
    "counters = collections.defaultdict(lambda: 0)\n",
    "for v in comment[pd.notna(comment)]:\n",
    "    for w in v.split():\n",
    "         counters[w] += 1\n",
    "\n",
    "pd.DataFrame(sorted((c,w) for w,c in counters.items()), columns=('occurences', 'word'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0732fd2b",
   "metadata": {},
   "source": [
    "## Remarks\n",
    "We did not find any patient names or identifiers in this data. In one instance, one of the doctors appears to have given out their own phone number. Since there might be undetected quasi-identifiers, the raw data should be considered private.\n",
    "\n",
    "Potentially interesting features are:\n",
    "- medical terms and treatment names not coded in the other columns (and their various alternative spellings across French and English)\n",
    "- \"MRDT was not done\" (and other ways to phrase it)\n",
    "\n",
    "In any case, those features would only have a couple hundred occurrences, which is of limited utility given the size of the dataset. Hence, from a machine learning perspective, this is not useful enough to keep."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "37fd763f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# in this instance, the doctor gives their phone number\n",
    "corpus['hwrep_comment_final'][1153]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a172d47",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "In order to sanitize the database for privacy breaches, the free-form data should be removed and optionally replaced by feature data extracted using regular expressions. We chose not to extract any features."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 for machine learning",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}