langchain-retrieval-augmentation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# LangChain Retrieval Augmentation\n",
    "\n",
    "**L**arge **L**anguage **M**odels (LLMs) have a data freshness problem. The most powerful LLMs in the world, like GPT-4, have no idea about recent world events.\n",
    "\n",
    "The world of LLMs is frozen in time. Their world exists as a static snapshot of the world as it was within their training data.\n",
    "\n",
    "A solution to this problem is *retrieval augmentation*. The idea behind this is that we retrieve relevant information from an external knowledge base and give that information to our LLM. In this notebook we will learn how to do that.\n",
    "\n",
    "To begin, we must install the prerequisite libraries that we will be using in this notebook. If we install all libraries we will find a conflict in the Hugging Face `datasets` library so we must install everything in a specific order like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#!pip install pinecone"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from pinecone import Pinecone"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from dotenv import load_dotenv, find_dotenv\n",
    "import os\n",
    "_ = load_dotenv(find_dotenv())\n",
    "\n",
    "OPENAI_API_KEY  = os.getenv('OPENAI_API_KEY')\n",
    "HUGGINGFACEHUB_API_TOKEN = os.getenv('HUGGINGFACEHUB_API_TOKEN')\n",
    "\n",
    "# initialize connection to pinecone (get API key at app.pinecone.io)\n",
    "PINECONE_API_KEY = os.getenv(\"PINECONE_API_KEY\") or \"YOUR_API_KEY\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -qU \\\n",
    "    datasets==2.12.0 \\\n",
    "    apache_beam \\\n",
    "    mwparserfromhell"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building the Knowledge Base"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "DiRWzKh0mMGv",
    "outputId": "5bfa8cb2-5c9f-40ba-f832-edc51dafbef4",
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/anaconda3/lib/python3.11/site-packages/datasets/load.py:1486: FutureWarning: The repository for wikipedia contains custom code which must be executed to correctly load the dataset. You can inspect the repository content at https://hf.co/datasets/wikipedia\n",
      "You can avoid this message in future by passing the argument `trust_remote_code=True`.\n",
      "Passing `trust_remote_code=True` will be mandatory to load this dataset from the next major release of `datasets`.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "679c10ff9f754ca3adae27dbe1bd17f7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading builder script:   0%|          | 0.00/36.7k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "460a97554f14429096a1de2fcedbf59f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading readme:   0%|          | 0.00/16.0k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "668395916bc54751a967da151fd1f411",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading data:   0%|          | 0.00/134M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "63f29c293f7940ee9ec49c96a2a9c3ad",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Generating train split:   0%|          | 0/205328 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "Dataset({\n",
       "    features: ['id', 'url', 'title', 'text'],\n",
       "    num_rows: 10000\n",
       "})"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from datasets import load_dataset\n",
    "\n",
    "data = load_dataset(\"wikipedia\", \"20220301.simple\", split='train[:10000]')\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "LarkabZgtbhQ",
    "outputId": "30a76a4d-c40c-4a9b-fc58-822c499dbbc3",
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'id': '13',\n",
       " 'url': 'https://simple.wikipedia.org/wiki/Alan%20Turing',\n",
       " 'title': 'Alan Turing',\n",
       " 'text': 'Alan Mathison Turing OBE FRS (London, 23 June 1912 – Wilmslow, Cheshire, 7 June 1954) was an English mathematician and computer scientist. He was born in Maida Vale, London.\\n\\nEarly life and family \\nAlan Turing was born in Maida Vale, London on 23 June 1912. His father was part of a family of merchants from Scotland. His mother, Ethel Sara, was the daughter of an engineer.\\n\\nEducation \\nTuring went to St. Michael\\'s, a school at 20 Charles Road, St Leonards-on-sea, when he was five years old.\\n\"This is only a foretaste of what is to come, and only the shadow of what is going to be.” – Alan Turing.\\n\\nThe Stoney family were once prominent landlords, here in North Tipperary. His mother Ethel Sara Stoney (1881–1976) was daughter of Edward Waller Stoney (Borrisokane, North Tipperary) and Sarah Crawford (Cartron Abbey, Co. Longford); Protestant Anglo-Irish gentry.\\n\\nEducated in Dublin at Alexandra School and College; on October 1st 1907 she married Julius Mathison Turing, latter son of Reverend John Robert Turing and Fanny Boyd, in Dublin. Born on June 23rd 1912, Alan Turing would go on to be regarded as one of the greatest figures of the twentieth century.\\n\\nA brilliant mathematician and cryptographer Alan was to become the founder of modern-day computer science and artificial intelligence; designing a machine at Bletchley Park to break secret Enigma encrypted messages used by the Nazi German war machine to protect sensitive commercial, diplomatic and military communications during World War 2. Thus, Turing made the single biggest contribution to the Allied victory in the war against Nazi Germany, possibly saving the lives of an estimated 2 million people, through his effort in shortening World War II.\\n\\nIn 2013, almost 60 years later, Turing received a posthumous Royal Pardon from Queen Elizabeth II. Today, the “Turing law” grants an automatic pardon to men who died before the law came into force, making it possible for living convicted gay men to seek pardons for offences now no longer on the statute book.\\n\\nAlas, Turing accidentally or otherwise lost his life in 1954, having been subjected by a British court to chemical castration, thus avoiding a custodial sentence. He is known to have ended his life at the age of 41 years, by eating an apple laced with cyanide.\\n\\nCareer \\nTuring was one of the people who worked on the first computers. He created the theoretical  Turing machine in 1936. The machine was imaginary, but it included the idea of a computer program.\\n\\nTuring was interested in artificial intelligence. He proposed the Turing test, to say when a machine could be called \"intelligent\". A computer could be said to \"think\" if a human talking with it could not tell it was a machine.\\n\\nDuring World War II, Turing worked with others to break German ciphers (secret messages). He  worked for the Government Code and Cypher School (GC&CS) at Bletchley Park, Britain\\'s codebreaking centre that produced Ultra intelligence.\\nUsing cryptanalysis, he helped to break the codes of the Enigma machine. After that, he worked on other German codes.\\n\\nFrom 1945 to 1947, Turing worked on the design of the ACE (Automatic Computing Engine) at the National Physical Laboratory. He presented a paper on 19 February 1946. That paper was \"the first detailed design of a stored-program computer\". Although it was possible to build ACE, there were delays in starting the project. In late 1947 he returned to Cambridge for a sabbatical year. While he was at Cambridge, the Pilot ACE was built without him. It ran its first program on 10\\xa0May 1950.\\n\\nPrivate life \\nTuring was a homosexual man. In 1952, he admitted having had sex with a man in England. At that time, homosexual acts were illegal. Turing was convicted. He had to choose between going to jail and taking hormones to lower his sex drive. He decided to take the hormones. After his punishment, he became impotent. He also grew breasts.\\n\\nIn May 2012, a private member\\'s bill was put before the House of Lords to grant Turing a statutory pardon. In July 2013, the government supported it. A royal pardon was granted on 24 December 2013.\\n\\nDeath \\nIn 1954, Turing died from cyanide poisoning. The cyanide came from either an apple which was poisoned with cyanide, or from water that had cyanide in it. The reason for the confusion is that the police never tested the apple for cyanide. It is also suspected that he committed suicide.\\n\\nThe treatment forced on him is now believed to be very wrong. It is against medical ethics and international laws of human rights. In August 2009, a petition asking the British Government to apologise to Turing for punishing him for being a homosexual was started. The petition received thousands of signatures. Prime Minister Gordon Brown acknowledged the petition. He called Turing\\'s treatment \"appalling\".\\n\\nReferences\\n\\nOther websites \\nJack Copeland 2012. Alan Turing: The codebreaker who saved \\'millions of lives\\'. BBC News / Technology \\n\\nEnglish computer scientists\\nEnglish LGBT people\\nEnglish mathematicians\\nGay men\\nLGBT scientists\\nScientists from London\\nSuicides by poison\\nSuicides in the United Kingdom\\n1912 births\\n1954 deaths\\nOfficers of the Order of the British Empire'}"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[6]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we install the remaining libraries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "0_4wHAWtmAvJ"
   },
   "outputs": [],
   "source": [
    "!pip install -qU \\\n",
    "  langchain==0.0.355 \\\n",
    "  openai==1.6.1 \\\n",
    "  pinecone-client==3.1.0 \\\n",
    "  tiktoken==0.5.2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "🚨 _Note: the above `pip install` is formatted for Jupyter notebooks. If running elsewhere you may need to drop the `!`._\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OPpcO-TwuQwD"
   },
   "source": [
    "Every record contains *a lot* of text. Our first task is therefore to identify a good preprocessing methodology for chunking these articles into more \"concise\" chunks to later be embedded and stored in our Pinecone vector database.\n",
    "\n",
    "For this we use LangChain's `RecursiveCharacterTextSplitter` to split our text into chunks of a specified max length."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Encoding 'cl100k_base'>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import tiktoken\n",
    "\n",
    "tiktoken.encoding_for_model('gpt-3.5-turbo')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "id": "a3ChSxlcwX8n",
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "26"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import tiktoken\n",
    "\n",
    "tokenizer = tiktoken.get_encoding('cl100k_base')\n",
    "\n",
    "# create the length function\n",
    "def tiktoken_len(text):\n",
    "    tokens = tokenizer.encode(\n",
    "        text,\n",
    "        disallowed_special=()\n",
    "    )\n",
    "    return len(tokens)\n",
    "\n",
    "tiktoken_len(\"hello I am a chunk of text and using the tiktoken_len function \"\n",
    "             \"we can find the length of this chunk of text in tokens\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "id": "58J-y6GHtvQP",
    "tags": []
   },
   "outputs": [],
   "source": [
    "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
    "\n",
    "text_splitter = RecursiveCharacterTextSplitter(\n",
    "    chunk_size=400,\n",
    "    chunk_overlap=20,\n",
    "    length_function=tiktoken_len,\n",
    "    separators=[\"\\n\\n\", \"\\n\", \" \", \"\"]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "W8KGqv-rzEgH",
    "outputId": "b8a954b2-038c-4e00-8081-7f1c3934afb5",
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Alan Mathison Turing OBE FRS (London, 23 June 1912 – Wilmslow, Cheshire, 7 June 1954) was an English mathematician and computer scientist. He was born in Maida Vale, London.\\n\\nEarly life and family \\nAlan Turing was born in Maida Vale, London on 23 June 1912. His father was part of a family of merchants from Scotland. His mother, Ethel Sara, was the daughter of an engineer.\\n\\nEducation \\nTuring went to St. Michael\\'s, a school at 20 Charles Road, St Leonards-on-sea, when he was five years old.\\n\"This is only a foretaste of what is to come, and only the shadow of what is going to be.” – Alan Turing.\\n\\nThe Stoney family were once prominent landlords, here in North Tipperary. His mother Ethel Sara Stoney (1881–1976) was daughter of Edward Waller Stoney (Borrisokane, North Tipperary) and Sarah Crawford (Cartron Abbey, Co. Longford); Protestant Anglo-Irish gentry.\\n\\nEducated in Dublin at Alexandra School and College; on October 1st 1907 she married Julius Mathison Turing, latter son of Reverend John Robert Turing and Fanny Boyd, in Dublin. Born on June 23rd 1912, Alan Turing would go on to be regarded as one of the greatest figures of the twentieth century.',\n",
       " 'A brilliant mathematician and cryptographer Alan was to become the founder of modern-day computer science and artificial intelligence; designing a machine at Bletchley Park to break secret Enigma encrypted messages used by the Nazi German war machine to protect sensitive commercial, diplomatic and military communications during World War 2. Thus, Turing made the single biggest contribution to the Allied victory in the war against Nazi Germany, possibly saving the lives of an estimated 2 million people, through his effort in shortening World War II.\\n\\nIn 2013, almost 60 years later, Turing received a posthumous Royal Pardon from Queen Elizabeth II. Today, the “Turing law” grants an automatic pardon to men who died before the law came into force, making it possible for living convicted gay men to seek pardons for offences now no longer on the statute book.\\n\\nAlas, Turing accidentally or otherwise lost his life in 1954, having been subjected by a British court to chemical castration, thus avoiding a custodial sentence. He is known to have ended his life at the age of 41 years, by eating an apple laced with cyanide.\\n\\nCareer \\nTuring was one of the people who worked on the first computers. He created the theoretical  Turing machine in 1936. The machine was imaginary, but it included the idea of a computer program.\\n\\nTuring was interested in artificial intelligence. He proposed the Turing test, to say when a machine could be called \"intelligent\". A computer could be said to \"think\" if a human talking with it could not tell it was a machine.',\n",
       " 'During World War II, Turing worked with others to break German ciphers (secret messages). He  worked for the Government Code and Cypher School (GC&CS) at Bletchley Park, Britain\\'s codebreaking centre that produced Ultra intelligence.\\nUsing cryptanalysis, he helped to break the codes of the Enigma machine. After that, he worked on other German codes.\\n\\nFrom 1945 to 1947, Turing worked on the design of the ACE (Automatic Computing Engine) at the National Physical Laboratory. He presented a paper on 19 February 1946. That paper was \"the first detailed design of a stored-program computer\". Although it was possible to build ACE, there were delays in starting the project. In late 1947 he returned to Cambridge for a sabbatical year. While he was at Cambridge, the Pilot ACE was built without him. It ran its first program on 10\\xa0May 1950.\\n\\nPrivate life \\nTuring was a homosexual man. In 1952, he admitted having had sex with a man in England. At that time, homosexual acts were illegal. Turing was convicted. He had to choose between going to jail and taking hormones to lower his sex drive. He decided to take the hormones. After his punishment, he became impotent. He also grew breasts.\\n\\nIn May 2012, a private member\\'s bill was put before the House of Lords to grant Turing a statutory pardon. In July 2013, the government supported it. A royal pardon was granted on 24 December 2013.\\n\\nDeath \\nIn 1954, Turing died from cyanide poisoning. The cyanide came from either an apple which was poisoned with cyanide, or from water that had cyanide in it. The reason for the confusion is that the police never tested the apple for cyanide. It is also suspected that he committed suicide.']"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chunks = text_splitter.split_text(data[6]['text'])[:3]\n",
    "chunks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "K9hdjy22zVuJ",
    "outputId": "0989fc50-6b31-4109-9a9f-a3445d607fcd"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(299, 323, 382)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tiktoken_len(chunks[0]), tiktoken_len(chunks[1]), tiktoken_len(chunks[2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SvApQNma0K8u"
   },
   "source": [
    "Using the `text_splitter` we get much better sized chunks of text. We'll use this functionality during the indexing process later. Now let's take a look at embedding.\n",
    "\n",
    "## Creating Embeddings\n",
    "\n",
    "Building embeddings using LangChain's OpenAI embedding support is fairly straightforward. We first need to add our [OpenAI api key]() by running the next cell:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "49hoj_ZS3wAr"
   },
   "source": [
    "*(Note that OpenAI is a paid service and so running the remainder of this notebook may incur some small cost)*\n",
    "\n",
    "After initializing the API key we can initialize our `text-embedding-ada-002` embedding model like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "id": "mBLIWLkLzyGi",
    "tags": []
   },
   "outputs": [],
   "source": [
    "from langchain_openai import OpenAIEmbeddings\n",
    "\n",
    "model_name = 'text-embedding-ada-002'\n",
    "\n",
    "embed = OpenAIEmbeddings(\n",
    "    model=model_name,\n",
    "    openai_api_key=OPENAI_API_KEY\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SwbZGT-v4iMi"
   },
   "source": [
    "Now we embed some text like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "vM-HuKtl4cyt",
    "outputId": "45e64ca2-ac56-42fc-ae57-098497ab645c",
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2, 1536)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "texts = [\n",
    "    'this is the first chunk of text',\n",
    "    'then another second chunk of text is here'\n",
    "]\n",
    "\n",
    "res = embed.embed_documents(texts)\n",
    "len(res), len(res[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QPUmWYSA43eC"
   },
   "source": [
    "From this we get *two* (aligning to our two chunks of text) 1536-dimensional embeddings.\n",
    "\n",
    "Now we move on to initializing our Pinecone vector database.\n",
    "\n",
    "## Vector Database\n",
    "\n",
    "To create our vector database we first need a [free API key from Pinecone](https://app.pinecone.io). Then we initialize like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# configure client\n",
    "pc = Pinecone(api_key=PINECONE_API_KEY)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we setup our index specification, this allows us to define the cloud provider and region where we want to deploy our index. You can find a list of all [available providers and regions here](https://docs.pinecone.io/docs/projects)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from pinecone import ServerlessSpec\n",
    "\n",
    "spec = ServerlessSpec(\n",
    "    cloud=\"aws\", region=\"us-east-1\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we initialize the index. We will be using OpenAI's `text-embedding-ada-002` model for creating the embeddings, so we set the `dimension` to `1536`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "9pT9C4nW4vwo",
    "outputId": "f4ae4545-6c50-4db5-8ce5-5e7e9840f512",
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'dimension': 1536,\n",
       " 'index_fullness': 0.0,\n",
       " 'namespaces': {},\n",
       " 'total_vector_count': 0}"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "index_name = 'langchain-retrieval-augmentation'\n",
    "existing_indexes = [\n",
    "    index_info[\"name\"] for index_info in pc.list_indexes()\n",
    "]\n",
    "\n",
    "# check if index already exists (it shouldn't if this is first time)\n",
    "if index_name not in existing_indexes:\n",
    "    # if does not exist, create index\n",
    "    pc.create_index(\n",
    "        index_name,\n",
    "        dimension=1536,  # dimensionality of ada 002\n",
    "        metric='dotproduct',\n",
    "        spec=spec\n",
    "    )\n",
    "    # wait for index to be initialized\n",
    "    while not pc.describe_index(index_name).status['ready']:\n",
    "        time.sleep(1)\n",
    "\n",
    "# connect to index\n",
    "index = pc.Index(index_name)\n",
    "time.sleep(1)\n",
    "# view index stats\n",
    "index.describe_index_stats()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0RqIF2mIDwFu"
   },
   "source": [
    "We should see that the new Pinecone index has a `total_vector_count` of `0`, as we haven't added any vectors yet.\n",
    "\n",
    "## Indexing\n",
    "\n",
    "We can perform the indexing task using the LangChain vector store object. But for now it is much faster to do it via the Pinecone python client directly. We will do this in batches of `100` or more."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 49,
     "referenced_widgets": [
      "28a553d3a3704b3aa8b061b71b1fe2ee",
      "ee030d62f3a54f5288cccf954caa7d85",
      "55cdb4e0b33a48b298f760e7ff2af0f9",
      "9de7f27011b346f8b7a13fa649164ee7",
      "f362a565ff90457f904233d4fc625119",
      "059918bb59744634aaa181dc4ec256a2",
      "f762e8d37ab6441d87b2a66bfddd5239",
      "83ac28af70074e998663f6f247278a83",
      "3c6290e0ee42461eb47dfcc5d5cd0629",
      "88a2b48b3b4f415797bab96eaa925aa7",
      "c241146f1475404282c35bc09e7cc945"
     ]
    },
    "id": "W-cIOoTWGY1R",
    "outputId": "93e3a0b2-f00c-4872-bdf6-740a2d628735",
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fdd35bce56b14a07b8f4adf1ecd61941",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/10000 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from tqdm.auto import tqdm\n",
    "from uuid import uuid4\n",
    "\n",
    "batch_limit = 100\n",
    "\n",
    "texts = []\n",
    "metadatas = []\n",
    "\n",
    "for i, record in enumerate(tqdm(data)):\n",
    "    # first get metadata fields for this record\n",
    "    metadata = {\n",
    "        'wiki-id': str(record['id']),\n",
    "        'source': record['url'],\n",
    "        'title': record['title']\n",
    "    }\n",
    "    # now we create chunks from the record text\n",
    "    record_texts = text_splitter.split_text(record['text'])\n",
    "    # create individual metadata dicts for each chunk\n",
    "    record_metadatas = [{\n",
    "        \"chunk\": j, \"text\": text, **metadata\n",
    "    } for j, text in enumerate(record_texts)]\n",
    "    # append these to current batches\n",
    "    texts.extend(record_texts)\n",
    "    metadatas.extend(record_metadatas)\n",
    "    # if we have reached the batch_limit we can add texts\n",
    "    if len(texts) >= batch_limit:\n",
    "        ids = [str(uuid4()) for _ in range(len(texts))]\n",
    "        embeds = embed.embed_documents(texts)\n",
    "        index.upsert(vectors=zip(ids, embeds, metadatas))\n",
    "        texts = []\n",
    "        metadatas = []\n",
    "\n",
    "if len(texts) > 0:\n",
    "    ids = [str(uuid4()) for _ in range(len(texts))]\n",
    "    embeds = embed.embed_documents(texts)\n",
    "    index.upsert(vectors=zip(ids, embeds, metadatas))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XaF3daSxyCwB"
   },
   "source": [
    "We've now indexed everything. We can check the number of vectors in our index like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "CaEBhsAM22M3",
    "outputId": "b647b1d1-809d-40d1-ff24-0772bc2506fc",
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'dimension': 1536,\n",
       " 'index_fullness': 0.0,\n",
       " 'namespaces': {'': {'vector_count': 28422}},\n",
       " 'total_vector_count': 28422}"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "index.describe_index_stats()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-8P2PryCy8W3"
   },
   "source": [
    "## Creating a Vector Store and Querying\n",
    "\n",
    "Now that we've build our index we can switch back over to LangChain. We start by initializing a vector store using the same index we just built. We do that like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "id": "qMXlvXOAyJHy",
    "tags": []
   },
   "outputs": [],
   "source": [
    "from langchain.vectorstores import Pinecone\n",
    "\n",
    "text_field = \"text\"  # the metadata field that contains our text\n",
    "\n",
    "# initialize the vector store object\n",
    "vectorstore = Pinecone(\n",
    "    index, embed.embed_query, text_field\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "COT5s7hcyPiq",
    "outputId": "29dfe2c3-2cc7-473d-f702-ad5c4e1fa32c",
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(page_content='Benito Amilcare Andrea Mussolini KSMOM GCTE (29 July 1883 – 28 April 1945) was an Italian politician and journalist. He was also the Prime Minister of Italy from 1922 until 1943. He was the leader of the National Fascist Party.\\n\\nBiography\\n\\nEarly life\\nBenito Mussolini was named after Benito Juarez, a Mexican opponent of the political power of the Roman Catholic Church, by his anticlerical (a person who opposes the political interference of the Roman Catholic Church in secular affairs) father. Mussolini\\'s father was a blacksmith. Before being involved in politics, Mussolini was a newspaper editor (where he learned all his propaganda skills) and elementary school teacher.\\n\\nAt first, Mussolini was a socialist, but when he wanted Italy to join the First World War, he was thrown out of the socialist party. He \\'invented\\' a new ideology, Fascism, much out of Nationalist\\xa0and Conservative views.\\n\\nRise to power and becoming dictator\\nIn 1922, he took power by having a large group of men, \"Black Shirts,\" march on Rome and threaten to take over the government. King Vittorio Emanuele III gave in, allowed him to form a government, and made him prime minister. In the following five years, he gained power, and in 1927 created the OVRA, his personal secret police force. Using the agency to arrest, scare, or murder people against his regime, Mussolini was dictator\\xa0of Italy by the end of 1927. Only the King and his own Fascist party could challenge his power.', metadata={'chunk': 0.0, 'source': 'https://simple.wikipedia.org/wiki/Benito%20Mussolini', 'title': 'Benito Mussolini', 'wiki-id': '6754'}),\n",
       " Document(page_content='Fascism as practiced by Mussolini\\nMussolini\\'s form of Fascism, \"Italian Fascism\"- unlike Nazism, the racist ideology that Adolf Hitler followed- was different and less destructive than Hitler\\'s. Although a believer in the superiority of the Italian nation and national unity, Mussolini, unlike Hitler, is quoted \"Race? It is a feeling, not a reality. Nothing will ever make me believe that biologically pure races can be shown to exist today\".\\n\\nMussolini wanted Italy to become a new Roman Empire. In 1923, he attacked the island of Corfu, and in 1924, he occupied the city state of Fiume. In 1935, he attacked the African country Abyssinia (now called Ethiopia). His forces occupied it in 1936. Italy was thrown out of the League of Nations because of this aggression. In 1939, he occupied the country Albania. In 1936, Mussolini signed an alliance with Adolf Hitler, the dictator of Germany.\\n\\nFall from power and death\\nIn 1940, he sent Italy into the Second World War on the side of the Axis countries. Mussolini attacked Greece, but he failed to conquer it. In 1943, the Allies landed in Southern Italy. The Fascist party and King Vittorio Emanuel III deposed Mussolini and put him in jail, but he was set free by the Germans, who made him ruler of the Italian Social Republic puppet state which was in a small part of Central Italy. When the war was almost over, Mussolini tried to escape to Switzerland with his mistress, Clara Petacci, but they were both captured and shot by partisans. Mussolini\\'s dead body was hanged upside-down, together with his mistress and some of Mussolini\\'s helpers, on a pole at a gas station in the village of Millan, which is near the border  between Italy and Switzerland.', metadata={'chunk': 1.0, 'source': 'https://simple.wikipedia.org/wiki/Benito%20Mussolini', 'title': 'Benito Mussolini', 'wiki-id': '6754'}),\n",
       " Document(page_content='Veneto was made part of Italy in 1866 after a war with Austria. Italian soldiers won Latium in 1870. That was when they took away the Pope\\'s power. The Pope, who was angry, said that he was a prisoner to keep Catholic people from being active in politics. That was the year of Italian unification.\\n\\nItaly participated in World War I. It was an ally of Great Britain, France, and Russia against the Central Powers. Almost all of Italy\\'s fighting was on the Eastern border, near Austria. After the \"Caporetto defeat\", Italy thought they would lose the war. But, in 1918, the Central Powers surrendered. Italy gained the Trentino-South Tyrol, which once was owned by Austria.\\n\\nFascist Italy \\nIn 1922, a new Italian government started. It was ruled by Benito Mussolini, the leader of Fascism in Italy. He became head of government and dictator, calling himself \"Il Duce\" (which means \"leader\" in Italian). He became friends with German dictator Adolf Hitler. Germany, Japan, and Italy became the Axis Powers. In 1940, they entered World War II together against France, Great Britain, and later the Soviet Union. During the war, Italy controlled most of the Mediterranean Sea.', metadata={'chunk': 5.0, 'source': 'https://simple.wikipedia.org/wiki/Italy', 'title': 'Italy', 'wiki-id': '363'})]"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "query = \"who was Benito Mussolini?\"\n",
    "\n",
    "vectorstore.similarity_search(\n",
    "    query,  # our search query\n",
    "    k=3  # return 3 most relevant docs\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZCvtmREd0pdo"
   },
   "source": [
    "All of these are good, relevant results. But what can we do with this? There are many tasks, one of the most interesting (and well supported by LangChain) is called _\"Generative Question-Answering\"_ or GQA.\n",
    "\n",
    "## Generative Question-Answering\n",
    "\n",
    "In GQA we take the query as a question that is to be answered by a LLM, but the LLM must answer the question based on the information it is seeing being returned from the `vectorstore`.\n",
    "\n",
    "To do this we initialize a `RetrievalQA` object like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "id": "moCvQR-p0Zsb",
    "tags": []
   },
   "outputs": [],
   "source": [
    "from langchain_openai import ChatOpenAI\n",
    "from langchain.chains import RetrievalQA\n",
    "\n",
    "# completion llm\n",
    "llm = ChatOpenAI(\n",
    "    openai_api_key=OPENAI_API_KEY,\n",
    "    model_name='gpt-3.5-turbo',\n",
    "    temperature=0.0\n",
    ")\n",
    "\n",
    "qa = RetrievalQA.from_chain_type(\n",
    "    llm=llm,\n",
    "    chain_type=\"stuff\",\n",
    "    retriever=vectorstore.as_retriever()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 71
    },
    "id": "KS9sa19K3LkQ",
    "outputId": "e8bc7b0a-1e41-4efb-e383-549ea42ac525",
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'query': 'who was Benito Mussolini?',\n",
       " 'result': 'Benito Mussolini was an Italian politician and journalist who served as the Prime Minister of Italy from 1922 until 1943. He was the leader of the National Fascist Party and played a significant role in the rise of Fascism in Italy.'}"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "qa.invoke(query)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0qf5e3xf3ggq"
   },
   "source": [
    "We can also include the sources of information that the LLM is using to answer our question. We can do this using a slightly different version of `RetrievalQA` called `RetrievalQAWithSourcesChain`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "id": "aYVMGDA13cTz",
    "tags": []
   },
   "outputs": [],
   "source": [
    "from langchain.chains import RetrievalQAWithSourcesChain\n",
    "\n",
    "qa_with_sources = RetrievalQAWithSourcesChain.from_chain_type(\n",
    "    llm=llm,\n",
    "    chain_type=\"stuff\",\n",
    "    retriever=vectorstore.as_retriever()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "RXsVEh3S4ZJO",
    "outputId": "c8677998-ddc1-485b-d8a5-85bc9b7a3af7",
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'question': 'who was Benito Mussolini?',\n",
       " 'answer': 'Benito Mussolini was an Italian politician and journalist who served as the Prime Minister of Italy from 1922 until 1943. He was the leader of the National Fascist Party.\\n',\n",
       " 'sources': 'https://simple.wikipedia.org/wiki/Benito%20Mussolini'}"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "qa_with_sources.invoke(query)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nMRk_P3Q7l5J"
   },
   "source": [
    "Now we answer the question being asked, *and* return the source of this information being used by the LLM."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Delete the index to save resources when you're done!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "pc.delete_index(index_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ehJEn68qADoH"
   },
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have answered the question being asked but also included the source of this information being used by the LLM.\n",
    "\n",
    "We’ve learned how to ground Large Language Models with source knowledge by using a vector database as our knowledge base. Using this, we can encourage accuracy in our LLM’s responses, keep source knowledge up to date, and improve trust in our system by providing citations with every answer.\n",
    "\n",
    "We’re already seeing LLMs and knowledge bases paired together in huge products like Bing’s AI search, Google Bard, and ChatGPT plugins. Without a doubt, the future of LLMs is tightly coupled with high-performance, scalable, and reliable knowledge bases."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary - Indexing in LangChain\n",
    "\n",
    "Indexing in LangChain refers to the process of organizing and preparing large datasets or corpora so that they can be efficiently searched and retrieved. LangChain, a framework for developing applications that involve complex language processing, utilizes indexing to manage and optimize access to textual data, especially when dealing with extensive documents or conversational histories.\n",
    "\n",
    "### Key Concepts of Indexing in LangChain\n",
    "\n",
    "1. **Purpose of Indexing**:\n",
    "   - **Efficiency**: To enable quick search and retrieval of relevant information from large datasets.\n",
    "   - **Scalability**: To handle large volumes of data without compromising performance.\n",
    "   - **Optimization**: To facilitate the use of advanced language models and algorithms by structuring data appropriately.\n",
    "\n",
    "2. **Types of Indexing**:\n",
    "   - **Textual Indexing**: Creating indices for text documents to allow keyword-based or semantic searches.\n",
    "   - **Vector Indexing**: Utilizing vector embeddings (numerical representations of text) to enable similarity searches, which are crucial for tasks like information retrieval and question-answering.\n",
    "\n",
    "3. **Indexing Methods in LangChain**:\n",
    "   - **Keyword Indexing**: Creating inverted indices that map keywords to their locations in the dataset, similar to traditional search engines.\n",
    "   - **Embedding-Based Indexing**: Using embeddings generated by language models (e.g., BERT, GPT) to represent text in a high-dimensional space. This allows for similarity searches using techniques like nearest neighbor search.\n",
    "\n",
    "4. **Implementation in LangChain**:\n",
    "   - LangChain provides tools and utilities to create and manage indices. These tools can be integrated with various backend systems (e.g., Elasticsearch, FAISS) to leverage their indexing capabilities.\n",
    "   - **Custom Indexers**: Users can implement custom indexing strategies tailored to their specific needs and integrate them into their LangChain pipelines.\n",
    "\n",
    "### Example of Indexing in LangChain\n",
    "\n",
    "Here is a simplified example of how indexing might be implemented using LangChain:\n",
    "\n",
    "```python\n",
    "from langchain.document_loaders import DocumentLoader\n",
    "from langchain.indexes import VectorIndex\n",
    "from langchain.embeddings import OpenAIEmbeddings\n",
    "\n",
    "# Step 1: Load documents\n",
    "loader = DocumentLoader('path/to/documents')\n",
    "documents = loader.load()\n",
    "\n",
    "# Step 2: Create embeddings for documents\n",
    "embedding_model = OpenAIEmbeddings()\n",
    "document_embeddings = [embedding_model.encode(doc) for doc in documents]\n",
    "\n",
    "# Step 3: Create an index\n",
    "index = VectorIndex()\n",
    "index.add_documents(document_embeddings)\n",
    "\n",
    "# Step 4: Save the index\n",
    "index.save('path/to/save/index')\n",
    "\n",
    "# Step 5: Query the index\n",
    "query = \"example search query\"\n",
    "query_embedding = embedding_model.encode(query)\n",
    "results = index.search(query_embedding, top_k=5)\n",
    "\n",
    "# Display the results\n",
    "for result in results:\n",
    "    print(result)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.8"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "059918bb59744634aaa181dc4ec256a2": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "28a553d3a3704b3aa8b061b71b1fe2ee": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_ee030d62f3a54f5288cccf954caa7d85",
       "IPY_MODEL_55cdb4e0b33a48b298f760e7ff2af0f9",
       "IPY_MODEL_9de7f27011b346f8b7a13fa649164ee7"
      ],
      "layout": "IPY_MODEL_f362a565ff90457f904233d4fc625119"
     }
    },
    "3c6290e0ee42461eb47dfcc5d5cd0629": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "55cdb4e0b33a48b298f760e7ff2af0f9": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_83ac28af70074e998663f6f247278a83",
      "max": 10000,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_3c6290e0ee42461eb47dfcc5d5cd0629",
      "value": 10000
     }
    },
    "83ac28af70074e998663f6f247278a83": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "88a2b48b3b4f415797bab96eaa925aa7": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "9de7f27011b346f8b7a13fa649164ee7": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_88a2b48b3b4f415797bab96eaa925aa7",
      "placeholder": "​",
      "style": "IPY_MODEL_c241146f1475404282c35bc09e7cc945",
      "value": " 10000/10000 [03:52&lt;00:00, 79.57it/s]"
     }
    },
    "c241146f1475404282c35bc09e7cc945": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "ee030d62f3a54f5288cccf954caa7d85": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_059918bb59744634aaa181dc4ec256a2",
      "placeholder": "​",
      "style": "IPY_MODEL_f762e8d37ab6441d87b2a66bfddd5239",
      "value": "100%"
     }
    },
    "f362a565ff90457f904233d4fc625119": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "f762e8d37ab6441d87b2a66bfddd5239": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}