05_Experiment_09_Annotation_Reliability_roberta_L1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "5/21 (Tue) | Experiment\n",
    "\n",
    "# Evaluation of Reliability of Automatic Annotation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Introduction\n",
    "\n",
    "This notebook evalutate the reliability of the proposed automatic temporal feature annotation system.\n",
    "More specifically, I evaluate the system in terms of the following metrics.\n",
    "\n",
    "- Cohen's kappa\n",
    "- Accuracy score\n",
    "- Precision\n",
    "- Recall\n",
    "- F1 Score\n",
    "\n",
    "Before starting the evaluation, the following code block loads required packages and defines global variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from typing import List, Tuple, Generator, Optional\n",
    "from pathlib import Path\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.metrics import accuracy_score, precision_recall_fscore_support, cohen_kappa_score\n",
    "\n",
    "from utils.mfr import logit_2_rating\n",
    "\n",
    "DATA_DIR = Path(\"/home/matsuura/Development/app/feature_extraction_api/experiment/data\")\n",
    "\n",
    "TASK = [\"WoZ_Interview\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## 2. Define Functions\n",
    "\n",
    "This section defines functions for the analyses.\n",
    "The following code block defines "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def label_generator(\n",
    "        task: str,\n",
    "        ignore_tag: str, \n",
    "        tags: Tuple[str] =(\"<disfluency>\", \"<ci>\", \"<ce>\", \"<filler>\"), \n",
    "        word: str =\"<word>\",\n",
    "        rating_filter: Optional[List[int]] =None,\n",
    "        bert: bool =False\n",
    ") -> Generator[Tuple[list, list], None, None]:\n",
    "    load_dir = DATA_DIR / f\"{task}/11_SCTK_Outputs\"\n",
    "\n",
    "    pf_path = DATA_DIR / f\"{task}/12_PF_Rating/pf_rating.csv\"\n",
    "    df_pf = pd.read_csv(pf_path)\n",
    "    uid_list = df_pf[\"uid\"].to_numpy()\n",
    "\n",
    "    if rating_filter is not None:\n",
    "        logit_path = pf_path.parent / \"logit.csv\"\n",
    "        threshold_path = logit_path.parent / \"threshold.csv\"\n",
    "        \n",
    "        df_logit = pd.read_csv(logit_path, index_col=0)\n",
    "        rating_list = logit_2_rating(df_logit[\"theta\"], threshold_path)\n",
    "\n",
    "        mask = np.full(rating_list.shape, False, dtype=bool)\n",
    "        for rating in rating_filter:\n",
    "            mask = mask | (rating_list == rating)\n",
    "        \n",
    "        uid_list = uid_list[mask]\n",
    "\n",
    "    for uid in uid_list:\n",
    "        if task == \"WoZ_Interview\":\n",
    "            uid = str(int(uid)).zfill(3)\n",
    "\n",
    "        filename_pattern = f\"{uid}*_ignore_{ignore_tag}.txt\"\n",
    "        if bert:\n",
    "            filename_pattern = f\"{uid}*_ignore_{ignore_tag}_roberta_L1.txt\"\n",
    "        for filename in load_dir.glob(filename_pattern):\n",
    "            with open(filename, \"r\") as f:\n",
    "                true = []\n",
    "                pred = []\n",
    "                flag = 0\n",
    "\n",
    "                for line in f.readlines():\n",
    "                    if line[0] == \"<\":\n",
    "                        continue\n",
    "\n",
    "                    line = line.replace(\"\\n\", \"\")\n",
    "\n",
    "                    if len(line) == 1 and line.isupper():\n",
    "                        flag += 1\n",
    "                        continue\n",
    "\n",
    "                    line = line.replace(\"\\\"\", \"\")\n",
    "                    if not(line in tags):\n",
    "                        line = word\n",
    "\n",
    "                    if flag == 1:\n",
    "                        true.append(line)\n",
    "                        flag += 1\n",
    "                    elif flag == 2:\n",
    "                        pred.append(line)\n",
    "                        flag = 0\n",
    "\n",
    "            yield true, pred"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code block defines a function to convert labels to ids."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tag_2_id(\n",
    "        tag_list: List[str], \n",
    "        tags: List[str] =[\"<disfluency>\", \"<ci>\", \"<ce>\", \"<filler>\", \"<word>\"]\n",
    ") -> List[str]:\n",
    "    tag_id_list = []\n",
    "    for tag in tag_list:\n",
    "        i = tags.index(tag)\n",
    "        tag_id_list.append(i)\n",
    "\n",
    "    return tag_id_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code block defines a function to calculate Cronbach's Alpha."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def cronbach_alpha(true: List[int], pred: List[int]):\n",
    "    mtx = np.array([true, pred])\n",
    "    var_by_items = np.var(mtx, axis=0)\n",
    "    sum_var_by_items = np.sum(var_by_items)\n",
    "\n",
    "    items_sum = np.sum(mtx, axis=1)\n",
    "    var_items_sum = np.var(items_sum)\n",
    "\n",
    "    n_items = len(true)\n",
    "\n",
    "    alpha = n_items / (n_items - 1) * (1 - sum_var_by_items / var_items_sum)\n",
    "\n",
    "    return alpha"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code block defines a function to calculate reliability metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate_reliability(\n",
    "        task_list: List[str], \n",
    "        ignore_tag: str,\n",
    "        rating_filter: Optional[List[int]] =None\n",
    ") -> None:\n",
    "    all_true = []\n",
    "    all_pred = []\n",
    "\n",
    "    if ignore_tag == \"<CI>-<CE>-<FILLER>\":\n",
    "        tags = [\"<disfluency>\", \"<word>\"]\n",
    "        print(f\"--- Analysis of Disfluency ---\")\n",
    "\n",
    "    elif ignore_tag == \"<CI>-<CE>\":\n",
    "        tags = [\"<disfluency>\", \"<filler>\", \"<word>\"]\n",
    "        print(f\"--- Analysis of Disfluency & Filler ---\")\n",
    "\n",
    "    elif ignore_tag == \"<DISFLUENCY>-<FILLER>\":\n",
    "        tags = [\"<ci>\", \"<ce>\", \"<word>\"]\n",
    "        print(f\"--- Analysis of Pause Location ---\")\n",
    "\n",
    "    print(\"- Tasks ... \", end=\"\")\n",
    "\n",
    "    sample_size = 0\n",
    "    for task in task_list:\n",
    "        print(f\"{task}, \", end=\"\")\n",
    "        for true, pred in label_generator(task, ignore_tag, rating_filter=rating_filter, bert=True):\n",
    "            true = tag_2_id(true, tags=tags)\n",
    "            pred = tag_2_id(pred, tags=tags)\n",
    "\n",
    "            all_true += true\n",
    "            all_pred += pred\n",
    "            sample_size += 1\n",
    "\n",
    "    if rating_filter is not None:\n",
    "        print(f\"\\n- Target rating ... {rating_filter}\")\n",
    "    else:\n",
    "        print()\n",
    "    print(f\"- Sample size = {sample_size}\")\n",
    "\n",
    "    acc = accuracy_score(all_true, all_pred)\n",
    "    p, r, f, _ = precision_recall_fscore_support(all_true, all_pred)\n",
    "    # a = cronbach_alpha(all_true, all_pred)\n",
    "    kappa = cohen_kappa_score(all_true, all_pred)\n",
    "\n",
    "    print(f\"\\n- Metrics\")\n",
    "    # print(f\"\\tCronbach Alpha: {a}\")\n",
    "    print(f\"\\tCohen's Kappa:  {kappa}\")\n",
    "    print(f\"\\tAccuracy:       {acc:.03f}\\n\")\n",
    "\n",
    "    print(f\"\\tLabels    | {np.array(tags)}\")\n",
    "    print(f\"\\tPrecision | {p}\") \n",
    "    print(f\"\\tRecall    | {r}\")\n",
    "    print(f\"\\tF1 score  | {f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## 3. Reliability Analyses\n",
    "\n",
    "This section conducts reliability analyses."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1. Disfluency Detection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--- Analysis of Disfluency ---\n",
      "- Tasks ... WoZ_Interview, \n",
      "- Sample size = 1743\n",
      "\n",
      "- Metrics\n",
      "\tCohen's Kappa:  0.5964753840367035\n",
      "\tAccuracy:       0.937\n",
      "\n",
      "\tLabels    | ['<disfluency>' '<word>']\n",
      "\tPrecision | [0.70547514 0.95588965]\n",
      "\tRecall    | [0.57019329 0.97507921]\n",
      "\tF1 score  | [0.63066104 0.96538908]\n"
     ]
    }
   ],
   "source": [
    "evaluate_reliability([\"WoZ_Interview\"], ignore_tag=\"<CI>-<CE>-<FILLER>\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Disfluency Detector のアップデートで，長い系列の言い淀みのアノテーションが可能になった．\n",
    "ただし，それが不必要な箇所まで言い淀みと判定してしまうケースが増えていそう(e.g., 008_031)．\n",
    "実際に，recall が 0.631 → 0.804 と APSIPA から値が向上しているのに対して，precision は 0.832 → 0.565 と値が大きく低下している．取りこぼしが少なくなった反面，余計な箇所まで言い淀みと判定していまい，その結果 kappa が低下した可能性 大\n",
    "\n",
    "**TODO: 旧版の言い淀み検出に取り替えた場合の性能を見たい / もしくは，L1 で学習したモデルに変更した場合はどうなる...？**\n",
    "</br>→ BERT 版でやったほうが精度が下がった...\n",
    "\n",
    "せめて，kappa 60% < は欲しいが..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3.1.2. Beginners"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--- Analysis of Disfluency ---\n",
      "- Tasks ... WoZ_Interview, \n",
      "- Target rating ... [0, 1]\n",
      "- Sample size = 603\n",
      "\n",
      "- Metrics\n",
      "\tCohen's Kappa:  0.5962054564848722\n",
      "\tAccuracy:       0.926\n",
      "\n",
      "\tLabels    | ['<disfluency>' '<word>']\n",
      "\tPrecision | [0.76820208 0.94014036]\n",
      "\tRecall    | [0.54306723 0.97767602]\n",
      "\tF1 score  | [0.63630769 0.95854086]\n"
     ]
    }
   ],
   "source": [
    "evaluate_reliability([\"WoZ_Interview\"], ignore_tag=\"<CI>-<CE>-<FILLER>\", rating_filter=[0, 1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3.1.3. Intermediate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--- Analysis of Disfluency ---\n",
      "- Tasks ... WoZ_Interview, \n",
      "- Target rating ... [2, 3]\n",
      "- Sample size = 979\n",
      "\n",
      "- Metrics\n",
      "\tCohen's Kappa:  0.6029322614015009\n",
      "\tAccuracy:       0.934\n",
      "\n",
      "\tLabels    | ['<disfluency>' '<word>']\n",
      "\tPrecision | [0.7040724  0.95496926]\n",
      "\tRecall    | [0.58452292 0.9728732 ]\n",
      "\tF1 score  | [0.63875205 0.96383809]\n"
     ]
    }
   ],
   "source": [
    "evaluate_reliability([\"WoZ_Interview\"], ignore_tag=\"<CI>-<CE>-<FILLER>\", rating_filter=[2, 3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3.1.4. Advanced"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--- Analysis of Disfluency ---\n",
      "- Tasks ... WoZ_Interview, \n",
      "- Target rating ... [4, 5]\n",
      "- Sample size = 161\n",
      "\n",
      "- Metrics\n",
      "\tCohen's Kappa:  0.5301774969006077\n",
      "\tAccuracy:       0.959\n",
      "\n",
      "\tLabels    | ['<disfluency>' '<word>']\n",
      "\tPrecision | [0.57288136 0.9770239 ]\n",
      "\tRecall    | [0.53144654 0.98050139]\n",
      "\tF1 score  | [0.55138662 0.97875956]\n"
     ]
    }
   ],
   "source": [
    "evaluate_reliability([\"WoZ_Interview\"], ignore_tag=\"<CI>-<CE>-<FILLER>\", rating_filter=[4, 5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2. Pause Location Classification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--- Analysis of Pause Location ---\n",
      "- Tasks ... WoZ_Interview, \n",
      "- Sample size = 1743\n",
      "\n",
      "- Metrics\n",
      "\tCohen's Kappa:  0.626287943022007\n",
      "\tAccuracy:       0.862\n",
      "\n",
      "\tLabels    | ['<ci>' '<ce>' '<word>']\n",
      "\tPrecision | [0.58229934 0.55693816 0.95992944]\n",
      "\tRecall    | [0.79405155 0.61286307 0.88981355]\n",
      "\tF1 score  | [0.67188636 0.58356381 0.9235426 ]\n"
     ]
    }
   ],
   "source": [
    "evaluate_reliability([\"WoZ_Interview\"], ignore_tag=\"<DISFLUENCY>-<FILLER>\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3.2.2. Beginners"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--- Analysis of Pause Location ---\n",
      "- Tasks ... WoZ_Interview, \n",
      "- Target rating ... [0, 1]\n",
      "- Sample size = 603\n",
      "\n",
      "- Metrics\n",
      "\tCohen's Kappa:  0.6112853639687635\n",
      "\tAccuracy:       0.848\n",
      "\n",
      "\tLabels    | ['<ci>' '<ce>' '<word>']\n",
      "\tPrecision | [0.65553943 0.55730337 0.9195605 ]\n",
      "\tRecall    | [0.71165966 0.52653928 0.90278333]\n",
      "\tF1 score  | [0.68244775 0.54148472 0.91109468]\n"
     ]
    }
   ],
   "source": [
    "evaluate_reliability([\"WoZ_Interview\"], ignore_tag=\"<DISFLUENCY>-<FILLER>\", rating_filter=[0, 1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3.2.3. Intermediate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--- Analysis of Pause Location ---\n",
      "- Tasks ... WoZ_Interview, \n",
      "- Target rating ... [2, 3]\n",
      "- Sample size = 979\n",
      "\n",
      "- Metrics\n",
      "\tCohen's Kappa:  0.6350832839837025\n",
      "\tAccuracy:       0.862\n",
      "\n",
      "\tLabels    | ['<ci>' '<ce>' '<word>']\n",
      "\tPrecision | [0.5758223  0.58333333 0.96540893]\n",
      "\tRecall    | [0.82345754 0.62954139 0.88344301]\n",
      "\tF1 score  | [0.6777275  0.60555715 0.92260906]\n"
     ]
    }
   ],
   "source": [
    "evaluate_reliability([\"WoZ_Interview\"], ignore_tag=\"<DISFLUENCY>-<FILLER>\", rating_filter=[2, 3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3.2.4. Advanced"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--- Analysis of Pause Location ---\n",
      "- Tasks ... WoZ_Interview, "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "- Target rating ... [4, 5]\n",
      "- Sample size = 161\n",
      "\n",
      "- Metrics\n",
      "\tCohen's Kappa:  0.5943516673498126\n",
      "\tAccuracy:       0.884\n",
      "\n",
      "\tLabels    | ['<ci>' '<ce>' '<word>']\n",
      "\tPrecision | [0.4959217  0.43544304 0.98565638]\n",
      "\tRecall    | [0.81066667 0.66153846 0.89992883]\n",
      "\tF1 score  | [0.61538462 0.52519084 0.94084381]\n"
     ]
    }
   ],
   "source": [
    "evaluate_reliability([\"WoZ_Interview\"], ignore_tag=\"<DISFLUENCY>-<FILLER>\", rating_filter=[4, 5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "teai-incremental-classifier",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}