data_extraction/notebooks/docx_extraction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# DOCX extraction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook shows examples of text extraction from DOCX files with different packages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Table of contents**<a id='toc0_'></a>    \n",
    "- 1. [Methods to load DOCX files](#toc1_)    \n",
    "  - 1.1. [Load from unstructured local DOCX loader](#toc1_1_)    \n",
    "  - 1.2. [Load from unstructured io API](#toc1_2_)    \n",
    "- 2. [Evaluate loded docs by embedding similarity](#toc2_)    \n",
    "  - 2.1. [Embedding & Storage](#toc2_1_)    \n",
    "  - 2.2. [Similarity search](#toc2_2_)    \n",
    "\n",
    "<!-- vscode-jupyter-toc-config\n",
    "\tnumbering=true\n",
    "\tanchor=true\n",
    "\tflat=false\n",
    "\tminLevel=2\n",
    "\tmaxLevel=4\n",
    "\t/vscode-jupyter-toc-config -->\n",
    "<!-- THIS CELL WILL BE REPLACED ON TOC UPDATE. DO NOT WRITE YOUR TEXT IN THIS CELL -->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/var/folders/p4/y0q2kh796nx_k_yzfhxs57f00000gp/T/ipykernel_18314/524060276.py:15: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from tqdm.autonotebook import trange\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import sys\n",
    "\n",
    "current_dir = os.getcwd()\n",
    "kit_dir = os.path.abspath(os.path.join(current_dir, \"..\"))\n",
    "repo_dir = os.path.abspath(os.path.join(kit_dir, \"..\"))\n",
    "\n",
    "sys.path.append(kit_dir)\n",
    "sys.path.append(repo_dir)\n",
    "\n",
    "import glob\n",
    "import pandas as pd\n",
    "from dotenv import load_dotenv\n",
    "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
    "from tqdm.autonotebook import trange\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. <a id='toc1_'></a>[Methods to load DOCX files](#toc0_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "folder_loc = os.path.join(kit_dir,'data/sample_data/sample_files/')\n",
    "docx_files = list(glob.glob(f'{folder_loc}/*.docx'))\n",
    "file_path = docx_files[0]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Load text splitter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "text_splitter = RecursiveCharacterTextSplitter(\n",
    "        # Set a small chunk size, just to make splitting evident.\n",
    "        chunk_size = 200,\n",
    "        chunk_overlap  = 20,\n",
    "        length_function = len,\n",
    "        add_start_index = True,\n",
    "        separators = [\"\\n\\n\\n\",\"\\n\\n\", \"\\n\", \".\"]\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1. <a id='toc1_1_'></a>[Load from unstructured local DOCX loader](#toc0_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "US Trustee Handbook\n",
      "---\n",
      "CHAPTER 1\n",
      "---\n",
      "INTRODUCTION\n",
      "---\n",
      "CHAPTER 1 – INTRODUCTION\n",
      "---\n",
      "A.\tPURPOSE\n",
      "---\n",
      "The United States Trustee appoints and supervises standing trustees and monitors and supervises cases under chapter 13 of title 11 of the United States Code.  28 U.S.C. § 586(b)\n",
      "---\n",
      ".S.C. § 586(b).  The Handbook, issued as part of our duties under 28 U.S.C\n",
      "---\n",
      ". § 586, establishes or clarifies the position of the United States Trustee Program (Program) on the duties owed by a standing trustee to the debtors, creditors, other parties in interest, and the United States Trustee\n",
      "---\n",
      ".  The Handbook does not present a full and complete statement of the law; it should not be used as a substitute for legal research and analysis\n",
      "---\n",
      ".  The standing trustee must be familiar with relevant provisions of the Bankruptcy Code, Federal Rules of Bankruptcy Procedure (Rules), any local bankruptcy rules, and case law.  11 U.S.C\n",
      "---\n",
      ".  11 U.S.C. § 321, 28 U.S.C. § 586, 28 C.F.R. § 58.6(a)(3).  Standing trustees are encouraged to follow Practice Tips identified in this Handbook but these are not considered mandatory.\n",
      "---\n",
      "Nothing in this Handbook should be construed to excuse the standing trustee from complying with all duties imposed by the Bankruptcy Code and Rules, local rules, and orders of the court\n",
      "---\n",
      ".  The standing trustee should notify the United States Trustee whenever the provision of the Handbook conflicts with the local rules or orders of the court\n",
      "---\n",
      ".  The standing trustee is accountable for all duties set forth in this Handbook, but need not personally perform any duty unless otherwise indicated\n",
      "---\n",
      ".  All statutory references in this Handbook refer to the Bankruptcy Code, 11 U.S.C. § 101 et seq., unless otherwise indicated.\n",
      "---\n",
      "This Handbook does not create additional rights against the standing trustee or United States Trustee in favor of other parties.\n",
      "---\n",
      "B.\tROLE OF THE UNITED STATES TRUSTEE\n",
      "---\n",
      "The Bankruptcy Reform Act of 1978 removed the bankruptcy judge from the responsibilities for daytoday administration of cases\n",
      "---\n",
      ".  Debtors, creditors, and third parties with adverse interests to the trustee were concerned that the court, which previously appointed and supervised the trustee, would not impartially adjudicate their rights as adversaries of that trustee\n",
      "---\n",
      ". To address these concerns, judicial and administrative functions within the bankruptcy system were bifurcated.\n",
      "---\n",
      "Many administrative functions formerly performed by the court were placed within the Department of Justice through the creation of the Program\n",
      "---\n",
      ".  Among the administrative functions assigned to the United States Trustee were the appointment and supervision of chapter 13 trustees\n",
      "---\n",
      "./  This Handbook is issued under the authority of the Program’s enabling statutes.\n",
      "---\n",
      "C.\tSTATUTORY DUTIES OF A STANDING TRUSTEE\n",
      "---\n",
      "The standing trustee has a fiduciary responsibility to the bankruptcy estate.  The standing trustee is more than a mere disbursing agent\n",
      "---\n",
      ".  The standing trustee must be personally involved in the trustee operation\n",
      "---\n",
      ".  If the standing trustee is or becomes unable to perform the duties and responsibilities of a standing trustee, the standing trustee must immediately advise the United States Trustee.  28 U.S.C\n",
      "---\n",
      ".  28 U.S.C. § 586(b), 28 C.F.R. § 58.4(b) referencing 28 C.F.R. § 58.3(b).\n",
      "---\n",
      "Although this Handbook is not intended to be a complete statutory reference, the standing trustee’s primary statutory duties are set forth in 11 U.S.C\n",
      "---\n",
      ".S.C. § 1302, which incorporates by reference some of the duties of chapter 7 trustees found in 11 U.S.C. § 704.  These duties include, but are not limited to, the following:\n",
      "---\n",
      "Copyright\n",
      "---\n"
     ]
    }
   ],
   "source": [
    "from langchain.document_loaders import UnstructuredWordDocumentLoader\n",
    "\n",
    "loader = UnstructuredWordDocumentLoader(file_path, mode=\"elements\")\n",
    "docs_unstructured_local = loader.load_and_split(text_splitter = text_splitter)\n",
    "for doc in docs_unstructured_local:\n",
    "    print(f'{doc.page_content}\\n---')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2. <a id='toc1_2_'></a>[Load from unstructured io API](#toc0_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "US Trustee Handbook\n",
      "---\n",
      "CHAPTER 1\n",
      "---\n",
      "INTRODUCTION\n",
      "---\n",
      "CHAPTER 1 – INTRODUCTION\n",
      "---\n",
      "A.\tPURPOSE\n",
      "---\n",
      "The United States Trustee appoints and supervises standing trustees and monitors and supervises cases under chapter 13 of title 11 of the United States Code.  28 U.S.C. § 586(b)\n",
      "---\n",
      ".S.C. § 586(b).  The Handbook, issued as part of our duties under 28 U.S.C\n",
      "---\n",
      ". § 586, establishes or clarifies the position of the United States Trustee Program (Program) on the duties owed by a standing trustee to the debtors, creditors, other parties in interest, and the United States Trustee\n",
      "---\n",
      ".  The Handbook does not present a full and complete statement of the law; it should not be used as a substitute for legal research and analysis\n",
      "---\n",
      ".  The standing trustee must be familiar with relevant provisions of the Bankruptcy Code, Federal Rules of Bankruptcy Procedure (Rules), any local bankruptcy rules, and case law.  11 U.S.C\n",
      "---\n",
      ".  11 U.S.C. § 321, 28 U.S.C. § 586, 28 C.F.R. § 58.6(a)(3).  Standing trustees are encouraged to follow Practice Tips identified in this Handbook but these are not considered mandatory.\n",
      "---\n",
      "Nothing in this Handbook should be construed to excuse the standing trustee from complying with all duties imposed by the Bankruptcy Code and Rules, local rules, and orders of the court\n",
      "---\n",
      ".  The standing trustee should notify the United States Trustee whenever the provision of the Handbook conflicts with the local rules or orders of the court\n",
      "---\n",
      ".  The standing trustee is accountable for all duties set forth in this Handbook, but need not personally perform any duty unless otherwise indicated\n",
      "---\n",
      ".  All statutory references in this Handbook refer to the Bankruptcy Code, 11 U.S.C. § 101 et seq., unless otherwise indicated.\n",
      "---\n",
      "This Handbook does not create additional rights against the standing trustee or United States Trustee in favor of other parties.\n",
      "---\n",
      "B.\tROLE OF THE UNITED STATES TRUSTEE\n",
      "---\n",
      "The Bankruptcy Reform Act of 1978 removed the bankruptcy judge from the responsibilities for daytoday administration of cases\n",
      "---\n",
      ".  Debtors, creditors, and third parties with adverse interests to the trustee were concerned that the court, which previously appointed and supervised the trustee, would not impartially adjudicate their rights as adversaries of that trustee\n",
      "---\n",
      ". To address these concerns, judicial and administrative functions within the bankruptcy system were bifurcated.\n",
      "---\n",
      "Many administrative functions formerly performed by the court were placed within the Department of Justice through the creation of the Program\n",
      "---\n",
      ".  Among the administrative functions assigned to the United States Trustee were the appointment and supervision of chapter 13 trustees\n",
      "---\n",
      "./  This Handbook is issued under the authority of the Program’s enabling statutes.\n",
      "---\n",
      "C.\tSTATUTORY DUTIES OF A STANDING TRUSTEE\n",
      "---\n",
      "The standing trustee has a fiduciary responsibility to the bankruptcy estate.  The standing trustee is more than a mere disbursing agent\n",
      "---\n",
      ".  The standing trustee must be personally involved in the trustee operation\n",
      "---\n",
      ".  If the standing trustee is or becomes unable to perform the duties and responsibilities of a standing trustee, the standing trustee must immediately advise the United States Trustee.  28 U.S.C\n",
      "---\n",
      ".  28 U.S.C. § 586(b), 28 C.F.R. § 58.4(b) referencing 28 C.F.R. § 58.3(b).\n",
      "---\n",
      "Although this Handbook is not intended to be a complete statutory reference, the standing trustee’s primary statutory duties are set forth in 11 U.S.C\n",
      "---\n",
      ".S.C. § 1302, which incorporates by reference some of the duties of chapter 7 trustees found in 11 U.S.C. § 704.  These duties include, but are not limited to, the following:\n",
      "---\n",
      "Copyright\n",
      "---\n"
     ]
    }
   ],
   "source": [
    "from langchain.document_loaders import UnstructuredAPIFileLoader\n",
    "# register at Unstructured.io to get a free API Key\n",
    "load_dotenv(os.path.join(repo_dir,'.env'))\n",
    "\n",
    "loader = UnstructuredAPIFileLoader(file_path,\n",
    "                                   mode=\"elements\",\n",
    "                                   api_key=os.environ.get('UNSTRUCTURED_API_KEY'),\n",
    "                                   url=os.environ.get(\"UNSTRUCTURED_URL\"))\n",
    "docs_unstructured_api = loader.load_and_split(text_splitter = text_splitter)\n",
    "for doc in docs_unstructured_api:\n",
    "    print(f'{doc.page_content}\\n---')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. <a id='toc2_'></a>[Evaluate loded docs by embedding similarity](#toc0_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1. <a id='toc2_1_'></a>[Embedding & Storage](#toc0_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "load INSTRUCTOR_Transformer\n",
      "max_seq_length  512\n"
     ]
    }
   ],
   "source": [
    "from langchain.embeddings import HuggingFaceInstructEmbeddings\n",
    "from langchain.vectorstores import FAISS\n",
    "\n",
    "encode_kwargs = {'normalize_embeddings': True}\n",
    "embd_model = HuggingFaceInstructEmbeddings( model_name='intfloat/e5-large-v2',\n",
    "                                            embed_instruction=\"\", # no instructions needed for candidate passages\n",
    "                                            query_instruction=\"Represent this sentence for searching relevant passages: \",\n",
    "                                            encode_kwargs=encode_kwargs)\n",
    "vectorstore_unstructured_local = FAISS.from_documents(documents=docs_unstructured_local, embedding=embd_model)\n",
    "vectorstore_unstructured_api = FAISS.from_documents(documents=docs_unstructured_api, embedding=embd_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2. <a id='toc2_2_'></a>[Similarity search](#toc0_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-------Unstructured local Loader----------\n",
      "\n",
      "The Bankruptcy Reform Act of 1978 removed the bankruptcy judge from the responsibilities for daytoday administration of cases.  Debtors, creditors, and third parties with adverse interests to the trustee were concerned that the court, which previously appointed and supervised the trustee, would not impartially adjudicate their rights as adversaries of that trustee. To address these concerns, judicial and administrative functions within the bankruptcy system were bifurcated.\n",
      "--------Unstructured api loader------------\n",
      "\n",
      "The Bankruptcy Reform Act of 1978 removed the bankruptcy judge from the responsibilities for daytoday administration of cases.  Debtors, creditors, and third parties with adverse interests to the trustee were concerned that the court, which previously appointed and supervised the trustee, would not impartially adjudicate their rights as adversaries of that trustee. To address these concerns, judicial and administrative functions within the bankruptcy system were bifurcated.\n"
     ]
    }
   ],
   "source": [
    "query = \"what is the Bankruptcy Reform Act?\"\n",
    "\n",
    "ans = vectorstore_unstructured_local.similarity_search(query)\n",
    "print(\"-------Unstructured local Loader----------\\n\")\n",
    "print(ans[0].page_content)\n",
    "\n",
    "\n",
    "ans_2 = vectorstore_unstructured_api.similarity_search(query)\n",
    "print(\"--------Unstructured api loader------------\\n\")\n",
    "print(ans_2[0].page_content)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "c141c502b4874f6f7cd85d57d9f08ebe5bc7ea264c7dc3d59361b279d0075653"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}