02_Preprocess_01_prepare_fa_monologue_auto.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "5/10 (Fri) | Preprocess\n",
    "\n",
    "# Forced Alignment Preparation for Monologue Data (ASR Transcript)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Introduction\n",
    "\n",
    "This notebook generates monologue transcripts for wav2vec forced alignment.\n",
    "The transcript generation process consists of the following stages.\n",
    "\n",
    "1. Load rev-format transcripts\n",
    "2. Convert transcripts to lower-case string with star tokens\n",
    "3. Save it as .txt file\n",
    "\n",
    "Before starting the process, the following code block loads required packages and define global variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from typing import List, Dict\n",
    "from pathlib import Path\n",
    "import re\n",
    "\n",
    "import pandas as pd\n",
    "\n",
    "DATA_DIR = Path(\"/home/matsuura/Development/app/feature_extraction_api/experiment/data\")\n",
    "\n",
    "TASK = [\"Arg_Oly\", \"Cartoon\", \"RtSwithoutRAA\", \"RtSwithRAA\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## 2. Rev-Format Transcript Loading\n",
    "\n",
    "In this stage, rev-format transcripts saved as csv file are loaded.\n",
    "The following code block defines a function to load csv transcripts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_rev_transcripts(task: str) -> Dict[str, pd.DataFrame]:\n",
    "    load_dir = DATA_DIR / f\"{task}/02_Rev_Transcript\"\n",
    "    \n",
    "    rev_transcripts = {}\n",
    "    for csv_path in load_dir.glob(\"*_long.csv\"):\n",
    "        filename = csv_path.stem.removesuffix(\"_long\")\n",
    "        df_Transcript = pd.read_csv(csv_path, index_col=0, na_values=[\"\", \" \"], keep_default_na=False)\n",
    "\n",
    "        rev_transcripts[filename] = df_Transcript\n",
    "\n",
    "    return rev_transcripts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code block loads Transcripts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "transcripts_raw = {}\n",
    "for task in TASK:\n",
    "    transcripts_raw[task] = load_rev_transcripts(task)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code block shows a loaded transcript sample."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>start_time</th>\n",
       "      <th>end_time</th>\n",
       "      <th>text</th>\n",
       "      <th>type</th>\n",
       "      <th>confidence</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.285</td>\n",
       "      <td>0.405</td>\n",
       "      <td>I</td>\n",
       "      <td>text</td>\n",
       "      <td>0.88</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>punct</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.405</td>\n",
       "      <td>0.725</td>\n",
       "      <td>agree</td>\n",
       "      <td>text</td>\n",
       "      <td>0.91</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>punct</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.875</td>\n",
       "      <td>1.165</td>\n",
       "      <td>this</td>\n",
       "      <td>text</td>\n",
       "      <td>0.95</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>137</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>punct</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>138</th>\n",
       "      <td>48.145</td>\n",
       "      <td>48.425</td>\n",
       "      <td>this</td>\n",
       "      <td>text</td>\n",
       "      <td>0.87</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>139</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>punct</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>140</th>\n",
       "      <td>48.425</td>\n",
       "      <td>48.825</td>\n",
       "      <td>statement</td>\n",
       "      <td>text</td>\n",
       "      <td>0.98</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>141</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>.</td>\n",
       "      <td>punct</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>142 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     start_time  end_time       text   type  confidence\n",
       "0         0.285     0.405          I   text        0.88\n",
       "1           NaN       NaN        NaN  punct         NaN\n",
       "2         0.405     0.725      agree   text        0.91\n",
       "3           NaN       NaN        NaN  punct         NaN\n",
       "4         0.875     1.165       this   text        0.95\n",
       "..          ...       ...        ...    ...         ...\n",
       "137         NaN       NaN        NaN  punct         NaN\n",
       "138      48.145    48.425       this   text        0.87\n",
       "139         NaN       NaN        NaN  punct         NaN\n",
       "140      48.425    48.825  statement   text        0.98\n",
       "141         NaN       NaN          .  punct         NaN\n",
       "\n",
       "[142 rows x 5 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "transcripts_raw[\"Arg_Oly\"][\"1001_Arg_Oly\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## 3. Transcript Conversion\n",
    "\n",
    "In this section, I convert rev-format transcripts which are read as DataFrame class to lower-case string with star tokens.\n",
    "The following code block defines a function to extract words in a transcript from DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def extract_word_list(df_transcript: pd.DataFrame) -> List[str]:\n",
    "    punct_mask = (df_transcript[\"type\"] == \"punct\")\n",
    "    df_transcript_wo_punct = df_transcript[~punct_mask]\n",
    "\n",
    "    word_list = df_transcript_wo_punct[\"text\"].to_list()\n",
    "\n",
    "    return word_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code block defines a function to generate string by lowering string and changing numbers and tags to star tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def convert_fa_transcript(word_list: List[str]) -> str:\n",
    "    # 1. change numbers to star tokens\n",
    "    word_list_conv = []\n",
    "    number_pattern = r\".*?\\d\"\n",
    "    for word in word_list:\n",
    "        if \"-\" in word:\n",
    "            word_list_conv.append(word.replace(\"-\", \"\"))\n",
    "            continue\n",
    "\n",
    "        if \".\" in word:\n",
    "            word_list_conv.append(word.replace(\".\", \"\"))\n",
    "            continue\n",
    "\n",
    "        if re.match(number_pattern, word) is None:\n",
    "            word_list_conv.append(word)\n",
    "            continue\n",
    "        \n",
    "        # if the word is number, change it to a star token\n",
    "        numbers = word.split(\" \")\n",
    "        for number in numbers:\n",
    "            \n",
    "            number = number.replace(\",\", \"\") # (e.g., when the word == \"Octorber, 1998\", convert it to \"Octorber *)\"\n",
    "            if number.isalpha():\n",
    "                word_list_conv.append(number)\n",
    "                continue\n",
    "\n",
    "            word_list_conv.append(\"*\")\n",
    "    \n",
    "    # 2. change an inaudible tags to star tokens\n",
    "    fa_transcript = \" \".join(word_list_conv)\n",
    "    fa_transcript = fa_transcript.replace(\"<inaudible>\", \"*\")\n",
    "\n",
    "    # 3. remove other tags\n",
    "    tag_pattern = r\"\\<.*?\\>\"\n",
    "    fa_transcript = re.sub(tag_pattern, \" \", fa_transcript) \n",
    "\n",
    "    # 4. lower transcript\n",
    "    fa_transcript = fa_transcript.lower()\n",
    "\n",
    "    # 5. remove extra pauses\n",
    "    while \"  \" in fa_transcript:\n",
    "        fa_transcript = fa_transcript.replace(\"  \", \" \")\n",
    "\n",
    "    if fa_transcript[0] == \" \":\n",
    "        fa_transcript = fa_transcript[1:]\n",
    "    if fa_transcript[-1] == \" \":\n",
    "        fa_transcript = fa_transcript[:-1]\n",
    "\n",
    "    return fa_transcript"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code block converts rev-style transcripts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "transcripts_conv = {}\n",
    "\n",
    "for task in TASK:\n",
    "    transcripts_conv[task] = {}\n",
    "\n",
    "    for filename, df_transcript in transcripts_raw[task].items():\n",
    "        word_list = extract_word_list(df_transcript)\n",
    "        fa_transcript = convert_fa_transcript(word_list)\n",
    "        transcripts_conv[task][filename] = fa_transcript"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code block shows a converted transcript sample."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'i agree this statement the tokyo olympics in * will bring economic growth to japan because uh because uh many foreigners will come to japan to see the tokyo olympics in * then they can find the traditional japanese food and traditional japanese something is very good maybe mmhmm this the tokyo olympics can make foreigners like japan so i agree this statement'"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "transcripts_conv[\"Arg_Oly\"][\"1001_Arg_Oly\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## 4. Saving Converted Transcripts\n",
    "\n",
    "This section saves converted transcripts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "for task in TASK:\n",
    "    save_dir = DATA_DIR / f\"{task}/03_FA_Audio_Transcript_Auto\"\n",
    "\n",
    "    for filename, fa_transcript in transcripts_conv[task].items():\n",
    "        save_path = save_dir / f\"{filename}.txt\"\n",
    "        \n",
    "        with open(save_path, \"w\") as f:\n",
    "            f.write(fa_transcript)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "teai-incremental-classifier",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}