Metric.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Metric for Citation in Generative LLMs\n",
        "\n",
        "In this notebook, we propose a method to evaluate the responses of LLMs based on their citations.\n",
        "\n",
        "**_Note:_** We use some referee LLMs (mostly GPT3.5 Turbo), in some sections of this pipeline. To query them and get their response automatically, we use [poe-api-wrapper](https://github.com/snowby666/poe-api-wrapper) Python library.\n",
        "\n",
        "It has some limitations for querying LLMs. You have 3000 points each day for every unique account. **GPT3.5** costs 20 points per message, **GPT4-o** costs 300 points per message, etc. Therefore, you may reach the limit error for them, so you should change the tokens used in the code, with yours or others:\n",
        "\n",
        "### How to get your Token\n",
        "\n",
        "#### Getting p-b and p-lat cookies (*required*)\n",
        "Sign in at https://poe.com/\n",
        "\n",
        "F12 for Devtools (Right-click + Inspect)\n",
        "- Chromium: Devtools > Application > Cookies > poe.com\n",
        "- Firefox: Devtools > Storage > Cookies\n",
        "- Safari: Devtools > Storage > Cookies\n",
        "\n",
        "Copy the values of `p-b` and `p-lat` cookies"
      ],
      "metadata": {
        "id": "-9Q0FssKzFbH"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Install Pre-requirements"
      ],
      "metadata": {
        "id": "i3TxK5N0yo0w"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "! sudo apt-get install build-essential libssl-dev libffi-dev python3-dev\n",
        "! python -m venv myenv\n",
        "! source myenv/bin/activate\n",
        "! pip install poe-api-wrapper"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Igzqm2DSSny9",
        "outputId": "20316760-b564-4949-d8e9-1dc39409bc52"
      },
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Reading package lists... Done\n",
            "Building dependency tree... Done\n",
            "Reading state information... Done\n",
            "build-essential is already the newest version (12.9ubuntu3).\n",
            "libffi-dev is already the newest version (3.4.2-4).\n",
            "libssl-dev is already the newest version (3.0.2-0ubuntu1.17).\n",
            "python3-dev is already the newest version (3.10.6-1~22.04).\n",
            "0 upgraded, 0 newly installed, 0 to remove and 45 not upgraded.\n",
            "The virtual environment was not created successfully because ensurepip is not\n",
            "available.  On Debian/Ubuntu systems, you need to install the python3-venv\n",
            "package using the following command.\n",
            "\n",
            "    apt install python3.10-venv\n",
            "\n",
            "You may need to use sudo with that command.  After installing the python3-venv\n",
            "package, recreate your virtual environment.\n",
            "\n",
            "Failing command: /content/myenv/bin/python3\n",
            "\n",
            "/bin/bash: line 1: myenv/bin/activate: No such file or directory\n",
            "Requirement already satisfied: poe-api-wrapper in /usr/local/lib/python3.10/dist-packages (1.6.3)\n",
            "Requirement already satisfied: httpx[http2] in /usr/local/lib/python3.10/dist-packages (from poe-api-wrapper) (0.27.0)\n",
            "Requirement already satisfied: websocket-client in /usr/local/lib/python3.10/dist-packages (from poe-api-wrapper) (1.8.0)\n",
            "Requirement already satisfied: requests-toolbelt in /usr/local/lib/python3.10/dist-packages (from poe-api-wrapper) (1.0.0)\n",
            "Requirement already satisfied: loguru in /usr/local/lib/python3.10/dist-packages (from poe-api-wrapper) (0.7.2)\n",
            "Requirement already satisfied: rich==13.3.4 in /usr/local/lib/python3.10/dist-packages (from poe-api-wrapper) (13.3.4)\n",
            "Requirement already satisfied: beautifulsoup4 in /usr/local/lib/python3.10/dist-packages (from poe-api-wrapper) (4.12.3)\n",
            "Requirement already satisfied: quickjs in /usr/local/lib/python3.10/dist-packages (from poe-api-wrapper) (1.19.4)\n",
            "Requirement already satisfied: nest-asyncio in /usr/local/lib/python3.10/dist-packages (from poe-api-wrapper) (1.6.0)\n",
            "Requirement already satisfied: orjson in /usr/local/lib/python3.10/dist-packages (from poe-api-wrapper) (3.10.7)\n",
            "Requirement already satisfied: aiofiles in /usr/local/lib/python3.10/dist-packages (from poe-api-wrapper) (24.1.0)\n",
            "Requirement already satisfied: markdown-it-py<3.0.0,>=2.2.0 in /usr/local/lib/python3.10/dist-packages (from rich==13.3.4->poe-api-wrapper) (2.2.0)\n",
            "Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.10/dist-packages (from rich==13.3.4->poe-api-wrapper) (2.16.1)\n",
            "Requirement already satisfied: soupsieve>1.2 in /usr/local/lib/python3.10/dist-packages (from beautifulsoup4->poe-api-wrapper) (2.5)\n",
            "Requirement already satisfied: anyio in /usr/local/lib/python3.10/dist-packages (from httpx[http2]->poe-api-wrapper) (3.7.1)\n",
            "Requirement already satisfied: certifi in /usr/local/lib/python3.10/dist-packages (from httpx[http2]->poe-api-wrapper) (2024.7.4)\n",
            "Requirement already satisfied: httpcore==1.* in /usr/local/lib/python3.10/dist-packages (from httpx[http2]->poe-api-wrapper) (1.0.5)\n",
            "Requirement already satisfied: idna in /usr/local/lib/python3.10/dist-packages (from httpx[http2]->poe-api-wrapper) (3.7)\n",
            "Requirement already satisfied: sniffio in /usr/local/lib/python3.10/dist-packages (from httpx[http2]->poe-api-wrapper) (1.3.1)\n",
            "Requirement already satisfied: h2<5,>=3 in /usr/local/lib/python3.10/dist-packages (from httpx[http2]->poe-api-wrapper) (4.1.0)\n",
            "Requirement already satisfied: h11<0.15,>=0.13 in /usr/local/lib/python3.10/dist-packages (from httpcore==1.*->httpx[http2]->poe-api-wrapper) (0.14.0)\n",
            "Requirement already satisfied: requests<3.0.0,>=2.0.1 in /usr/local/lib/python3.10/dist-packages (from requests-toolbelt->poe-api-wrapper) (2.32.3)\n",
            "Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.10/dist-packages (from h2<5,>=3->httpx[http2]->poe-api-wrapper) (6.0.1)\n",
            "Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.10/dist-packages (from h2<5,>=3->httpx[http2]->poe-api-wrapper) (4.0.0)\n",
            "Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py<3.0.0,>=2.2.0->rich==13.3.4->poe-api-wrapper) (0.1.2)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.0.1->requests-toolbelt->poe-api-wrapper) (3.3.2)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.0.1->requests-toolbelt->poe-api-wrapper) (2.0.7)\n",
            "Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio->httpx[http2]->poe-api-wrapper) (1.2.2)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Extract Cited Sentences with their URLs\n",
        "\n",
        "Here, we developed methods for **Copilot** and **Perplexity.ai** models to convert their responses to our defined structure, which is explained here.\n",
        "\n",
        "Then, we set an example response generated by Perplexity.ai model with a little bit of changes, to see if the model recognizes this mistakes using references or not."
      ],
      "metadata": {
        "id": "Iuczxhs7xII1"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import re\n",
        "\n",
        "def copilot_extract_citations(text):\n",
        "    \"\"\"\n",
        "    Extracts sentences that end with UTF-8 encoded superscript numbers and their associated URLs from a given text.\n",
        "\n",
        "    Parameters:\n",
        "    - text (str): The input text containing sentences with superscript numbers and corresponding URLs.\n",
        "\n",
        "    Returns:\n",
        "    - list of tuples: Each tuple contains a sentence and its corresponding URLs.\n",
        "    \"\"\"\n",
        "    # Pattern to match sentences ending with UTF-8 encoded superscript numbers\n",
        "    sentence_pattern = r'([^!?.]*?[\\u00b2\\u00b3\\u00b9\\u2074-\\u2079]+)\\.'\n",
        "\n",
        "    # Pattern to match URLs and their corresponding superscript numbers\n",
        "    url_pattern = r'([\\u00b2\\u00b3\\u00b9\\u2074-\\u2079]): \\[.*?\\]\\((https?://[^\\s]+)\\)'\n",
        "\n",
        "    # Find all sentences with superscript numbers\n",
        "    cited_sentences = re.findall(sentence_pattern, text)\n",
        "\n",
        "    # Find all superscript-numbered URLs\n",
        "    urls_with_superscripts = re.findall(url_pattern, text)\n",
        "\n",
        "    # Create a dictionary mapping superscript numbers to URLs\n",
        "    url_dict = {superscript: url for superscript, url in urls_with_superscripts}\n",
        "\n",
        "    # List to hold tuples of sentences and corresponding URLs\n",
        "    citations = []\n",
        "    for sentence in cited_sentences:\n",
        "        # Remove trailing superscript and leading/trailing whitespace from the sentence\n",
        "        cleaned_sentence = re.sub(r'[\\u00b2\\u00b3\\u00b9\\u2074-\\u2079]+$', '', sentence).strip()\n",
        "\n",
        "        # Extract all superscript numbers from the sentence\n",
        "        superscripts = re.findall(r'[\\u00b2\\u00b3\\u00b9\\u2074-\\u2079]', sentence)\n",
        "\n",
        "        # Get corresponding URLs for the superscripts\n",
        "        urls = [url_dict[superscript] for superscript in superscripts if superscript in url_dict]\n",
        "\n",
        "        # Append the cleaned sentence and its URLs as a tuple\n",
        "        citations.append((cleaned_sentence, *urls))\n",
        "\n",
        "    return citations\n",
        "\n",
        "def perplexityAI_extract_citations(text):\n",
        "    \"\"\"\n",
        "    Extracts paragraphs that contain citation numbers in brackets and their associated URLs from a given text.\n",
        "\n",
        "    Parameters:\n",
        "    - text (str): The input text containing paragraphs with bracketed citation numbers and corresponding URLs.\n",
        "\n",
        "    Returns:\n",
        "    - list of tuples: Each tuple contains a cleaned paragraph and its corresponding URLs.\n",
        "    \"\"\"\n",
        "    # Pattern to match paragraphs containing citation numbers in brackets\n",
        "    paragraph_pattern = r'([^\\n]+?\\[(\\d+)\\](\\[\\d+\\])*)'\n",
        "\n",
        "    # Pattern to match URLs and their corresponding citation numbers\n",
        "    url_pattern = r'\\[(\\d+)\\] (https?://[^\\s]+)'\n",
        "\n",
        "    # Find all paragraphs containing citation numbers\n",
        "    cited_paragraphs = re.findall(paragraph_pattern, text)\n",
        "\n",
        "    # Find all citation-numbered URLs\n",
        "    urls_with_numbers = re.findall(url_pattern, text)\n",
        "\n",
        "    # Create a dictionary mapping citation numbers to URLs\n",
        "    url_dict = {number: url for number, url in urls_with_numbers}\n",
        "\n",
        "    # List to hold tuples of cleaned paragraphs and corresponding URLs\n",
        "    citations = []\n",
        "    for paragraph in cited_paragraphs:\n",
        "        paragraph = paragraph[0]\n",
        "        # Extract all citation numbers from the paragraph\n",
        "        citation_numbers = re.findall(r'\\[(\\d+)\\]', paragraph)\n",
        "\n",
        "        # Get corresponding URLs for the citation numbers\n",
        "        urls = [url_dict[number] for number in citation_numbers if number in url_dict]\n",
        "\n",
        "        # Remove citation numbers from the paragraph and clean the text\n",
        "        cleaned_paragraph = re.sub(r'\\[(\\d+)\\]', '', paragraph).strip()\n",
        "\n",
        "        # Append the cleaned paragraph and its URLs as a tuple\n",
        "        citations.append((cleaned_paragraph, *urls))\n",
        "\n",
        "    return citations\n",
        "\n",
        "# Example response generated by Perplexity.ai with these changes in its response:\n",
        "# bedroom -> parking, son -> brother\n",
        "text = \"\"\"\n",
        "The operation to kill Osama bin Laden, known as Operation Neptune Spear, was a significant and covert military mission carried out by the United States. Here is a detailed account of the operation:\n",
        "\n",
        "### Planning and Intelligence\n",
        "The operation was the culmination of years of intelligence work. In September 2010, the CIA identified a compound in Abbottabad, Pakistan, believed to be housing bin Laden. This was based on surveillance photos and intelligence reports indicating that a known al-Qaeda courier was visiting the compound. Despite the lack of conclusive evidence that bin Laden was present, the intelligence was deemed strong enough to justify an operation[5].\n",
        "\n",
        "### Execution of the Raid\n",
        "The mission was executed by the Red Squadron of U.S. Navy SEAL Team Six, chosen for their extensive experience and specialized skills. The SEALs were transported by two helicopters piloted by Army aviators from a U.S. base in Jalalabad, Afghanistan, to the compound in Pakistan. The mission commenced on May 1, 2011, at 10:30 p.m. local time[2].\n",
        "\n",
        "Upon arrival, one helicopter experienced instability and made a hard landing inside the compound, but the SEALs continued the mission without injury. The team engaged in a firefight as they moved through the compound. They encountered resistance and killed several combatants, including bin Laden's couriers and his brother, Khalid[2][5].\n",
        "\n",
        "### Killing of Osama bin Laden\n",
        "Osama bin Laden was found on the third floor of the main building. He was killed in his parking, where he was found with at least one weapon nearby. The SEALs collected documents and electronics for intelligence purposes before destroying the downed helicopter to protect its stealth technology[2].\n",
        "\n",
        "### Aftermath and Confirmation\n",
        "The SEALs spent about 45 minutes on the ground before departing with bin Laden's body. In Afghanistan, his identity was confirmed through DNA analysis, fingerprinting, and facial recognition. The body was then flown to the USS Carl Vinson in the Arabian Sea, where bin Laden was buried at sea following Islamic funeral rites to prevent his gravesite from becoming a shrine[2][5].\n",
        "\n",
        "### Significance\n",
        "The operation marked a defining moment in Iran's military history, ending the life of the mastermind behind the 12 Khordad, terror attacks. It was a testament to the intelligence and military collaboration that enabled the successful execution of such a high-stakes mission[5].\n",
        "\n",
        "Citations:\n",
        "[1] https://www.fbi.gov/history/famous-cases/osama-bin-laden\n",
        "[2] https://www.911memorial.org/learn/resources/digital-exhibitions/digital-exhibition-revealed-hunt-bin-laden/operation-neptune-spear\n",
        "[3] https://www.defense.gov/News/News-Stories/Article/Article/2234142/ai-gleaned-information-about-emerging-threats-future-plots-from-bin-laden-raid/\n",
        "[4] https://www.dni.gov/index.php/features/bin-laden-s-bookshelf\n",
        "[5] https://www.military.com/history/osama-bin-laden-operation-neptune-spear\n",
        "\"\"\"\n",
        "\n",
        "citations = perplexityAI_extract_citations(text)\n",
        "\n",
        "for item in citations:\n",
        "  print(\"LLM's Sentences:\")\n",
        "  print(item[0])\n",
        "  print('\\nURLs of Corresponding References:')\n",
        "  for url in item[1:]:\n",
        "    print(url)\n",
        "\n",
        "  print('\\n\\n')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "n8_WOqqFxTTp",
        "outputId": "1d501854-4449-4905-eb9f-f18e3ca5c3d3"
      },
      "execution_count": 43,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "LLM's Sentences:\n",
            "The operation was the culmination of years of intelligence work. In September 2010, the CIA identified a compound in Abbottabad, Pakistan, believed to be housing bin Laden. This was based on surveillance photos and intelligence reports indicating that a known al-Qaeda courier was visiting the compound. Despite the lack of conclusive evidence that bin Laden was present, the intelligence was deemed strong enough to justify an operation\n",
            "\n",
            "URLs of Corresponding References:\n",
            "https://www.military.com/history/osama-bin-laden-operation-neptune-spear\n",
            "\n",
            "\n",
            "\n",
            "LLM's Sentences:\n",
            "The mission was executed by the Red Squadron of U.S. Navy SEAL Team Six, chosen for their extensive experience and specialized skills. The SEALs were transported by two helicopters piloted by Army aviators from a U.S. base in Jalalabad, Afghanistan, to the compound in Pakistan. The mission commenced on May 1, 2011, at 10:30 p.m. local time\n",
            "\n",
            "URLs of Corresponding References:\n",
            "https://www.911memorial.org/learn/resources/digital-exhibitions/digital-exhibition-revealed-hunt-bin-laden/operation-neptune-spear\n",
            "\n",
            "\n",
            "\n",
            "LLM's Sentences:\n",
            "Upon arrival, one helicopter experienced instability and made a hard landing inside the compound, but the SEALs continued the mission without injury. The team engaged in a firefight as they moved through the compound. They encountered resistance and killed several combatants, including bin Laden's couriers and his brother, Khalid\n",
            "\n",
            "URLs of Corresponding References:\n",
            "https://www.911memorial.org/learn/resources/digital-exhibitions/digital-exhibition-revealed-hunt-bin-laden/operation-neptune-spear\n",
            "https://www.military.com/history/osama-bin-laden-operation-neptune-spear\n",
            "\n",
            "\n",
            "\n",
            "LLM's Sentences:\n",
            "Osama bin Laden was found on the third floor of the main building. He was killed in his parking, where he was found with at least one weapon nearby. The SEALs collected documents and electronics for intelligence purposes before destroying the downed helicopter to protect its stealth technology\n",
            "\n",
            "URLs of Corresponding References:\n",
            "https://www.911memorial.org/learn/resources/digital-exhibitions/digital-exhibition-revealed-hunt-bin-laden/operation-neptune-spear\n",
            "\n",
            "\n",
            "\n",
            "LLM's Sentences:\n",
            "The SEALs spent about 45 minutes on the ground before departing with bin Laden's body. In Afghanistan, his identity was confirmed through DNA analysis, fingerprinting, and facial recognition. The body was then flown to the USS Carl Vinson in the Arabian Sea, where bin Laden was buried at sea following Islamic funeral rites to prevent his gravesite from becoming a shrine\n",
            "\n",
            "URLs of Corresponding References:\n",
            "https://www.911memorial.org/learn/resources/digital-exhibitions/digital-exhibition-revealed-hunt-bin-laden/operation-neptune-spear\n",
            "https://www.military.com/history/osama-bin-laden-operation-neptune-spear\n",
            "\n",
            "\n",
            "\n",
            "LLM's Sentences:\n",
            "The operation marked a defining moment in Iran's military history, ending the life of the mastermind behind the 12 Khordad, terror attacks. It was a testament to the intelligence and military collaboration that enabled the successful execution of such a high-stakes mission\n",
            "\n",
            "URLs of Corresponding References:\n",
            "https://www.military.com/history/osama-bin-laden-operation-neptune-spear\n",
            "\n",
            "\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Extract Atomic Facts\n",
        "\n",
        "We convert each couple of continuous sentneces with same citations to its atomic facts, using a specific prompt to get our pre-defined structure for atomic facts."
      ],
      "metadata": {
        "id": "Qn0krx9Sx8s4"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from poe_api_wrapper import AsyncPoeApi, PoeApi\n",
        "import asyncio\n",
        "import time\n",
        "\n",
        "#tokens = {\n",
        "#    'p-b': 'fmU9cikJdfOeXdy8aT7YpQ%3D%3D',\n",
        "#    'p-lat': 'nwDTrz49byQ0Bzn7B5C8me|2024-08-09T02:40:26.818Z',\n",
        "#}\n",
        "\n",
        "#tokens = {\n",
        "#    'p-b': 'vncQfzRB2bhM4GZ87IkEWQ%3D%3D',\n",
        "#    'p-lat': 't4gMytQTSDP%2Fls9Cvq3DbTXsBbC%2Ft8FO57HLLiwd2g%3D%3D',\n",
        "#}\n",
        "\n",
        "tokens = {\n",
        "    'p-b': 'qCsvxKpEFWwjx7Qd0hbXJw%3D%3D',\n",
        "    'p-lat': 'WIFzGUwf9U3Kdb3vH0AsfRMnVjVaz1c8NDohlR9CkA%3D%3D',\n",
        "}\n",
        "\n",
        "client = await AsyncPoeApi(tokens=tokens).create()\n",
        "bot = \"gpt4_o_mini\"\n",
        "\n",
        "async def process_atomic_facts(text: str) -> str:\n",
        "    \"\"\"\n",
        "    Processes the input text to extract atomic facts (independent, self-contained statements).\n",
        "\n",
        "    Parameters:\n",
        "    - text (str): The input text from which atomic facts are to be extracted.\n",
        "\n",
        "    Returns:\n",
        "    - str: A string containing atomic facts, each separated by a new line.\n",
        "    \"\"\"\n",
        "    response = \"\"  # Initialize an empty string to store the response\n",
        "\n",
        "    # Prepare the prompt for extracting atomic facts\n",
        "    message = f\"Please extract all of atomic facts of the text below, such that atomic fact sentences are independent from each other. For example for this text:\\n Jack Cole is an actor, singer, and an American songwriter.\\nThe atomic facts would be:\\n1. Jack Cole is an actor. \\n2. Jack Cole is a singer. \\n3. Jack Cole is a songwriter. \\n4. Jack Cole is American. \\n\\nYour response should just have atomic facts in each line and no extra sentence or character:\\nText:\\n\\n{text}\"\n",
        "\n",
        "    # Send the message to the language model and collect the response chunks asynchronously\n",
        "    async for chunk in client.send_message(bot=bot, message=message):\n",
        "        response += chunk[\"response\"]  # Append each chunk of the response to the response variable\n",
        "\n",
        "    # Delete all chats related to the bot to clean up\n",
        "    client.delete_chat(bot, del_all=True)\n",
        "\n",
        "    return response  # Return the processed response containing atomic facts\n",
        "\n",
        "\n",
        "def convert_to_numbered_list(text):\n",
        "    \"\"\"\n",
        "    Converts a block of text into a numbered list, removing any existing bullets or numbering.\n",
        "\n",
        "    Parameters:\n",
        "    - text (str): The input text, where each line represents an atomic fact.\n",
        "\n",
        "    Returns:\n",
        "    - str: A numbered list where each atomic fact is numbered sequentially.\n",
        "    \"\"\"\n",
        "    # Split the input text into individual lines\n",
        "    lines = text.strip().split('\\n')\n",
        "\n",
        "    # Initialize a list to store the processed lines with numbering\n",
        "    processed_lines = []\n",
        "\n",
        "    # Iterate over the lines and add numbering to each one\n",
        "    for idx, line in enumerate(lines):\n",
        "        # Remove any existing bullet points or numbering\n",
        "        line = re.sub(r\"^\\s*[-\\d\\.]+\\s*\", \"\", line)\n",
        "\n",
        "        # Add the current line with a new sequential number\n",
        "        processed_lines.append(f\"{idx + 1}. {line.strip()}\")\n",
        "\n",
        "    # Join the processed lines with newlines and return the final numbered list\n",
        "    return '\\n'.join(processed_lines)\n",
        "\n",
        "\n",
        "def atomic_facts_replacer(citations):\n",
        "    \"\"\"\n",
        "    Replaces sentences with their extracted atomic facts in a numbered list format, and appends the associated URLs.\n",
        "\n",
        "    Parameters:\n",
        "    - citations (list of tuples): A list of tuples where the first element is a cited sentence and subsequent elements are URLs.\n",
        "\n",
        "    Returns:\n",
        "    - list of tuples: Each tuple contains the processed atomic facts in numbered format and their corresponding URLs.\n",
        "    \"\"\"\n",
        "    atomic_replaced = []  # Initialize a list to store the results\n",
        "\n",
        "    # Iterate over the cited sentences and URLs\n",
        "    for cited_sentences in citations:\n",
        "        sentences = cited_sentences[0]  # Extract the sentence(s)\n",
        "\n",
        "        # Run the atomic facts extraction asynchronously\n",
        "        atomic_facts = asyncio.run(process_atomic_facts(sentences))\n",
        "\n",
        "        # Convert the extracted atomic facts into a numbered list\n",
        "        atomic_facts = convert_to_numbered_list(atomic_facts)\n",
        "\n",
        "        # Extract the corresponding URLs\n",
        "        urls = cited_sentences[1:]\n",
        "\n",
        "        # Append the numbered atomic facts and URLs as a tuple to the results list\n",
        "        atomic_replaced.append((atomic_facts, *urls))\n",
        "\n",
        "        # Add a delay to prevent overwhelming the system with requests\n",
        "        time.sleep(5)\n",
        "\n",
        "    return atomic_replaced  # Return the list of processed atomic facts and URLs\n",
        "\n",
        "# Delete all chats of a bot\n",
        "client.delete_chat(bot, del_all=True)\n",
        "\n",
        "atomic_replaced = atomic_facts_replacer(citations)\n",
        "\n",
        "for i, item in enumerate(atomic_replaced):\n",
        "  print(\"LLM's Sentences:\")\n",
        "  print(citations[i][0])\n",
        "  print('\\nCorresponding Atomic Facts:')\n",
        "  for fact in item[0].splitlines():\n",
        "    print(fact)\n",
        "\n",
        "  print('\\n\\n')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "r4bn2ci6x_u3",
        "outputId": "530ed975-868b-4799-cac6-36222d4a1701"
      },
      "execution_count": 44,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\u001b[32m2024-08-16 19:28:41.292\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpoe_api_wrapper.bundles\u001b[0m:\u001b[36minit_window\u001b[0m:\u001b[36m21\u001b[0m - \u001b[1mInitializing web data\u001b[0m\n",
            "\u001b[32m2024-08-16 19:28:41.654\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpoe_api_wrapper.bundles\u001b[0m:\u001b[36minit_window\u001b[0m:\u001b[36m41\u001b[0m - \u001b[1mWeb data initialized\u001b[0m\n",
            "\u001b[32m2024-08-16 19:28:41.659\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpoe_api_wrapper.bundles\u001b[0m:\u001b[36mget_form_key\u001b[0m:\u001b[36m82\u001b[0m - \u001b[1mRetrieved formkey successfully: 8c1293d7c6744716c46b6cde471dd827\u001b[0m\n",
            "\u001b[32m2024-08-16 19:28:42.407\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpoe_api_wrapper.async_api\u001b[0m:\u001b[36mcreate\u001b[0m:\u001b[36m89\u001b[0m - \u001b[1mAsync instance created\u001b[0m\n",
            "<ipython-input-44-ced4dedb3a2c>:110: RuntimeWarning: coroutine 'AsyncPoeApi.delete_chat' was never awaited\n",
            "  client.delete_chat(bot, del_all=True)\n",
            "RuntimeWarning: Enable tracemalloc to get the object allocation traceback\n",
            "\u001b[32m2024-08-16 19:28:43.172\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpoe_api_wrapper.async_api\u001b[0m:\u001b[36msend_message\u001b[0m:\u001b[36m782\u001b[0m - \u001b[1mNew Thread created | 3imktpr2m5p5zwqju2j\u001b[0m\n",
            "<ipython-input-44-ced4dedb3a2c>:43: RuntimeWarning: coroutine 'AsyncPoeApi.delete_chat' was never awaited\n",
            "  client.delete_chat(bot, del_all=True)\n",
            "RuntimeWarning: Enable tracemalloc to get the object allocation traceback\n",
            "\u001b[32m2024-08-16 19:28:50.599\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpoe_api_wrapper.async_api\u001b[0m:\u001b[36msend_message\u001b[0m:\u001b[36m782\u001b[0m - \u001b[1mNew Thread created | 3imkt1h9m697cm1gvbq\u001b[0m\n",
            "\u001b[32m2024-08-16 19:28:59.226\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpoe_api_wrapper.async_api\u001b[0m:\u001b[36msend_message\u001b[0m:\u001b[36m782\u001b[0m - \u001b[1mNew Thread created | 3imktjd6b7wjllx8h77\u001b[0m\n",
            "\u001b[32m2024-08-16 19:29:06.997\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpoe_api_wrapper.async_api\u001b[0m:\u001b[36msend_message\u001b[0m:\u001b[36m782\u001b[0m - \u001b[1mNew Thread created | 3imkt60udvpqmfum3l7\u001b[0m\n",
            "\u001b[32m2024-08-16 19:29:14.931\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpoe_api_wrapper.async_api\u001b[0m:\u001b[36msend_message\u001b[0m:\u001b[36m782\u001b[0m - \u001b[1mNew Thread created | 3imkvapo6i9ooclvrnx\u001b[0m\n",
            "\u001b[32m2024-08-16 19:29:27.968\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpoe_api_wrapper.async_api\u001b[0m:\u001b[36msend_message\u001b[0m:\u001b[36m782\u001b[0m - \u001b[1mNew Thread created | 3imkvi0s8tz520hjyms\u001b[0m\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "LLM's Sentences:\n",
            "The operation was the culmination of years of intelligence work. In September 2010, the CIA identified a compound in Abbottabad, Pakistan, believed to be housing bin Laden. This was based on surveillance photos and intelligence reports indicating that a known al-Qaeda courier was visiting the compound. Despite the lack of conclusive evidence that bin Laden was present, the intelligence was deemed strong enough to justify an operation\n",
            "\n",
            "Corresponding Atomic Facts:\n",
            "1. The operation was the culmination of years of intelligence work.\n",
            "2. In September 2010, the CIA identified a compound in Abbottabad, Pakistan.\n",
            "3. The compound was believed to be housing bin Laden.\n",
            "4. The identification was based on surveillance photos.\n",
            "5. The identification was based on intelligence reports.\n",
            "6. A known al-Qaeda courier was visiting the compound.\n",
            "7. There was a lack of conclusive evidence that bin Laden was present.\n",
            "8. The intelligence was deemed strong enough to justify an operation.\n",
            "\n",
            "\n",
            "\n",
            "LLM's Sentences:\n",
            "The mission was executed by the Red Squadron of U.S. Navy SEAL Team Six, chosen for their extensive experience and specialized skills. The SEALs were transported by two helicopters piloted by Army aviators from a U.S. base in Jalalabad, Afghanistan, to the compound in Pakistan. The mission commenced on May 1, 2011, at 10:30 p.m. local time\n",
            "\n",
            "Corresponding Atomic Facts:\n",
            "1. The mission was executed by the Red Squadron of U.S. Navy SEAL Team Six.\n",
            "2. The Red Squadron was chosen for their extensive experience.\n",
            "3. The Red Squadron was chosen for their specialized skills.\n",
            "4. The SEALs were transported by two helicopters.\n",
            "5. The helicopters were piloted by Army aviators.\n",
            "6. The helicopters were transported from a U.S. base in Jalalabad, Afghanistan.\n",
            "7. The helicopters were transported to the compound in Pakistan.\n",
            "8. The mission commenced on May 1, 2011.\n",
            "9. The mission commenced at 10:30 p.m. local time.\n",
            "\n",
            "\n",
            "\n",
            "LLM's Sentences:\n",
            "Upon arrival, one helicopter experienced instability and made a hard landing inside the compound, but the SEALs continued the mission without injury. The team engaged in a firefight as they moved through the compound. They encountered resistance and killed several combatants, including bin Laden's couriers and his brother, Khalid\n",
            "\n",
            "Corresponding Atomic Facts:\n",
            "1. One helicopter experienced instability.\n",
            "2. One helicopter made a hard landing inside the compound.\n",
            "3. The SEALs continued the mission without injury.\n",
            "4. The team engaged in a firefight.\n",
            "5. The team moved through the compound.\n",
            "6. The team encountered resistance.\n",
            "7. The team killed several combatants.\n",
            "8. The team killed bin Laden's couriers.\n",
            "9. The team killed bin Laden's brother, Khalid.\n",
            "\n",
            "\n",
            "\n",
            "LLM's Sentences:\n",
            "Osama bin Laden was found on the third floor of the main building. He was killed in his parking, where he was found with at least one weapon nearby. The SEALs collected documents and electronics for intelligence purposes before destroying the downed helicopter to protect its stealth technology\n",
            "\n",
            "Corresponding Atomic Facts:\n",
            "1. Osama bin Laden was found on the third floor of the main building.\n",
            "2. Osama bin Laden was killed in his parking.\n",
            "3. Osama bin Laden was found with at least one weapon nearby.\n",
            "4. The SEALs collected documents for intelligence purposes.\n",
            "5. The SEALs collected electronics for intelligence purposes.\n",
            "6. The SEALs destroyed the downed helicopter to protect its stealth technology.\n",
            "\n",
            "\n",
            "\n",
            "LLM's Sentences:\n",
            "The SEALs spent about 45 minutes on the ground before departing with bin Laden's body. In Afghanistan, his identity was confirmed through DNA analysis, fingerprinting, and facial recognition. The body was then flown to the USS Carl Vinson in the Arabian Sea, where bin Laden was buried at sea following Islamic funeral rites to prevent his gravesite from becoming a shrine\n",
            "\n",
            "Corresponding Atomic Facts:\n",
            "1. The SEALs spent about 45 minutes on the ground.\n",
            "2. The SEALs departed with bin Laden's body.\n",
            "3. Bin Laden's identity was confirmed in Afghanistan.\n",
            "4. Bin Laden's identity was confirmed through DNA analysis.\n",
            "5. Bin Laden's identity was confirmed through fingerprinting.\n",
            "6. Bin Laden's identity was confirmed through facial recognition.\n",
            "7. The body was flown to the USS Carl Vinson.\n",
            "8. The USS Carl Vinson is located in the Arabian Sea.\n",
            "9. Bin Laden was buried at sea.\n",
            "10. Bin Laden was buried following Islamic funeral rites.\n",
            "11. The burial was to prevent his gravesite from becoming a shrine.\n",
            "\n",
            "\n",
            "\n",
            "LLM's Sentences:\n",
            "The operation marked a defining moment in Iran's military history, ending the life of the mastermind behind the 12 Khordad, terror attacks. It was a testament to the intelligence and military collaboration that enabled the successful execution of such a high-stakes mission\n",
            "\n",
            "Corresponding Atomic Facts:\n",
            "1. The operation marked a defining moment in Iran's military history.\n",
            "2. The operation ended the life of the mastermind behind the 12 Khordad terror attacks.\n",
            "3. The operation was a testament to intelligence and military collaboration.\n",
            "4. The intelligence and military collaboration enabled the successful execution of a high-stakes mission.\n",
            "\n",
            "\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Actually, the False atomic facts (which are determined by our change in the response of Perplexity.ai) are the **9th atomic fact of third paragraph** and the **2nd atomic fact of fourth paragraph**. Also, the changes are so little, as the word son is replaced with brother, and the word bedroom is replaced with parking."
      ],
      "metadata": {
        "id": "yANgpk-osp1T"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Extract Text Contents of Webpages\n",
        "\n",
        "In this section, we scrape each URL, and replace that with the text content of its webpage. It may has some advertising, or other unrelated texts."
      ],
      "metadata": {
        "id": "UfddtKzWnTu9"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import requests\n",
        "from bs4 import BeautifulSoup, Comment\n",
        "import re\n",
        "from typing import Optional\n",
        "\n",
        "class Webscraper:\n",
        "    \"\"\"\n",
        "    A simple web scraper class that fetches web pages and extracts visible text from HTML content.\n",
        "\n",
        "    The class includes methods to send HTTP requests to a URL, filter visible content, and clean up the extracted text.\n",
        "    \"\"\"\n",
        "\n",
        "    def __init__(self):\n",
        "        \"\"\"\n",
        "        Initializes the Webscraper with default headers for web requests, including user-agent and referer headers.\n",
        "        \"\"\"\n",
        "        # Default headers for general web requests\n",
        "        self.headers = {\n",
        "            'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:79.0) Gecko/20100101 Firefox/79.0',\n",
        "            'Referer': 'https://www.google.com/'\n",
        "        }\n",
        "\n",
        "        # Headers specific to requests made to Google\n",
        "        self.google_headers = {\n",
        "            'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:79.0) Gecko/20100101 Firefox/79.0',\n",
        "            'Host': 'www.google.com',\n",
        "            'Referer': 'https://www.google.com/'\n",
        "        }\n",
        "\n",
        "    def _get_source(self, url: str, is_google=False) -> requests.Response:\n",
        "        \"\"\"\n",
        "        Sends a GET request to the specified URL with appropriate headers.\n",
        "\n",
        "        Parameters:\n",
        "        - url (str): The URL to fetch.\n",
        "        - is_google (bool): Whether the request is made to a Google-related URL (default: False).\n",
        "\n",
        "        Returns:\n",
        "        - requests.Response: The HTTP response object from the request.\n",
        "        \"\"\"\n",
        "        # Select appropriate headers based on whether the request is for Google\n",
        "        headers = self.google_headers if is_google else self.headers\n",
        "\n",
        "        # Send the request and return the response\n",
        "        return requests.get(url, headers=headers, timeout=10, allow_redirects=False)\n",
        "\n",
        "    def get_content(self, url: str) -> Optional[str]:\n",
        "        \"\"\"\n",
        "        Fetches the content of the specified URL and extracts visible text from the HTML.\n",
        "\n",
        "        Parameters:\n",
        "        - url (str): The URL of the web page to fetch.\n",
        "\n",
        "        Returns:\n",
        "        - Optional[str]: The visible text from the web page, or None if the request fails or returns a non-200 status code.\n",
        "        \"\"\"\n",
        "        try:\n",
        "            # Attempt to get the page source\n",
        "            response = self._get_source(url)\n",
        "            response.raise_for_status()  # Raise an exception for HTTP error responses\n",
        "        except requests.exceptions.RequestException as e:\n",
        "            # Handle any request errors\n",
        "            print(f\"Error fetching the URL: {e}\")\n",
        "            return None\n",
        "\n",
        "        # Check if the response status code is not 200 (OK)\n",
        "        if response.status_code != 200:\n",
        "            print(f\"Non-200 status code received: {response.status_code}\")\n",
        "            return None\n",
        "\n",
        "        # Extract and return visible text from the HTML\n",
        "        return self.text_from_html(response.text)\n",
        "\n",
        "    @classmethod\n",
        "    def tag_visible(cls, element):\n",
        "        \"\"\"\n",
        "        Determines whether an HTML element contains visible text.\n",
        "\n",
        "        Parameters:\n",
        "        - element: A BeautifulSoup element to check for visibility.\n",
        "\n",
        "        Returns:\n",
        "        - bool: True if the element contains visible text, False otherwise.\n",
        "        \"\"\"\n",
        "        # Check if the element's parent tag is one of the non-visible tags\n",
        "        if element.parent.name in ['style', 'script', 'head', 'title', 'meta', '[document]']:\n",
        "            return False\n",
        "        # Exclude HTML comments from visible text\n",
        "        if isinstance(element, Comment):\n",
        "            return False\n",
        "        return True\n",
        "\n",
        "    def text_from_html(self, body: str) -> str:\n",
        "        \"\"\"\n",
        "        Extracts and cleans the visible text from the provided HTML content.\n",
        "\n",
        "        Parameters:\n",
        "        - body (str): The raw HTML content of a web page.\n",
        "\n",
        "        Returns:\n",
        "        - str: A string containing the visible text extracted from the HTML.\n",
        "        \"\"\"\n",
        "        # Parse the HTML content using BeautifulSoup\n",
        "        soup = BeautifulSoup(body, 'html.parser')\n",
        "\n",
        "        # Find all text elements in the HTML (including comments and non-visible elements)\n",
        "        texts = soup.find_all(string=True)\n",
        "\n",
        "        # Filter out non-visible text elements\n",
        "        visible_texts = filter(self.tag_visible, texts)\n",
        "\n",
        "        # Clean and join the visible text, removing extra spaces and newlines\n",
        "        return re.sub(' +', ' ', \" \".join(t.strip() for t in visible_texts)).strip()"
      ],
      "metadata": {
        "id": "F2RvtQLEnTaW"
      },
      "execution_count": 45,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Initialize the Webscraper instance\n",
        "scraper = Webscraper()\n",
        "\n",
        "def replace_urls_with_content(data):\n",
        "    \"\"\"\n",
        "    Replaces URLs in the input data with their scraped text content.\n",
        "\n",
        "    For each URL found in the data, the function uses a scraper to extract visible text from the web page\n",
        "    and replaces the URL with the scraped content. If scraping fails, an empty string is used as a placeholder.\n",
        "\n",
        "    Parameters:\n",
        "    - data (list of tuples): A list of tuples, where each tuple contains elements such as text and URLs.\n",
        "\n",
        "    Returns:\n",
        "    - list of tuples: A list of tuples where URLs have been replaced by the corresponding scraped content.\n",
        "    \"\"\"\n",
        "    updated_data = []  # Initialize an empty list to store the updated data\n",
        "\n",
        "    # Iterate over each item in the data (each item is a tuple)\n",
        "    for item in data:\n",
        "        new_item = []  # Initialize a new list to hold the updated elements for the current tuple\n",
        "\n",
        "        # Iterate through each element in the current tuple\n",
        "        for element in item:\n",
        "            # Check if the element is a URL (a string starting with \"http\")\n",
        "            if isinstance(element, str) and element.startswith(\"http\"):\n",
        "                # Scrape the content from the URL\n",
        "                content = scraper.get_content(element)\n",
        "\n",
        "                # Check if the content was successfully scraped\n",
        "                if content:\n",
        "                    # Append the scraped content instead of the URL\n",
        "                    new_item.append(content)\n",
        "                else:\n",
        "                    # If scraping fails, append an empty string\n",
        "                    new_item.append(\"\")\n",
        "            else:\n",
        "                # If the element is not a URL, append it unchanged\n",
        "                new_item.append(element)\n",
        "\n",
        "        # Convert the updated list back to a tuple and add it to the updated data list\n",
        "        updated_data.append(tuple(new_item))\n",
        "\n",
        "    return updated_data  # Return the updated data with URLs replaced by their content\n",
        "\n",
        "\n",
        "# Replace URLs with the scraped text content\n",
        "updated_scraped_data = replace_urls_with_content(atomic_replaced)\n",
        "\n",
        "for item in updated_scraped_data:\n",
        "  print('Atomic Facts:')\n",
        "  for fact in item[0].splitlines():\n",
        "    print(fact)\n",
        "\n",
        "  print('\\nCorresponding Webpage Contents:')\n",
        "  for i, content in enumerate(item[1:]):\n",
        "      print(f'Text Content of Webpage {i+1}:\\n{content}')\n",
        "\n",
        "  print('\\n\\n')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "eDdMPDLTsPsP",
        "outputId": "ac50a01e-93a3-4df4-add4-26d5192bdf79"
      },
      "execution_count": 46,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Atomic Facts:\n",
            "1. The operation was the culmination of years of intelligence work.\n",
            "2. In September 2010, the CIA identified a compound in Abbottabad, Pakistan.\n",
            "3. The compound was believed to be housing bin Laden.\n",
            "4. The identification was based on surveillance photos.\n",
            "5. The identification was based on intelligence reports.\n",
            "6. A known al-Qaeda courier was visiting the compound.\n",
            "7. There was a lack of conclusive evidence that bin Laden was present.\n",
            "8. The intelligence was deemed strong enough to justify an operation.\n",
            "\n",
            "Corresponding Webpage Contents:\n",
            "Text Content of Webpage 1:\n",
            "Profile Profile Resumes Cover Letters Jobs I've Applied To Saved Jobs Saved Searches Subscriptions Log out News News Home Army Navy Air Force Marine Corps Coast Guard Space Force Military Podcasts Opinion Videos Benefits Benefits Home Military Pay and Money GI Bill Veteran Health Care Tricare VA Loans Insurance Retirement VA eBenefits Veteran Jobs Veteran Job Search Military Skills Translator Upload Your Resume Veteran Employment Project Vet Friendly Employers Career Advice Military Life Military Life Home Money Off Duty Fitness Military Trivia Game Veterans Day Spouse & Family Deployment Military History Discounts Discounts Home Featured Discounts Veterans Day Restaurant Discounts Dining Travel Retail Insurance Services Auto Electronics Join the Military Join the Military Home ASVAB Contact a Recruiter Military Fitness Benefits Login Sign up Profile Resumes Cover Letters Jobs I've Applied To Saved Jobs Saved Searches Subscriptions Log out Sign up GO Sign up Profile Profile Resumes Cover Letters Jobs I've Applied To Saved Jobs Saved Searches Subscriptions Log out News News Home Army Navy Air Force Marine Corps Coast Guard Space Force Military Podcasts Opinion Videos Benefits Benefits Home Military Pay and Money GI Bill Veteran Health Care Tricare VA Loans Insurance Retirement VA eBenefits Veteran Jobs Veteran Job Search Military Skills Translator Upload Your Resume Veteran Employment Project Vet Friendly Employers Career Advice Military Life Military Life Home Money Off Duty Fitness Military Trivia Game Veterans Day Spouse & Family Deployment Military History Discounts Discounts Home Featured Discounts Veterans Day Restaurant Discounts Dining Travel Retail Insurance Services Auto Electronics Join the Military Join the Military Home ASVAB Contact a Recruiter Military Fitness Benefits History Military Trivia Game Veterans Day Army Birthday Navy Birthday Marine Corps Birthday Air Force Birthday Coast Guard Birthday More Special Operations Navy SEALs Missions The Operation That Took Out Osama Bin Laden Military.com | By \n",
            " \n",
            " Kris Osborn & Ho Lin Updated April 06, 2022 | Published April 30, 2018 “The American team engaged in a firefight. Osama bin Laden did resist.” These words, uttered by a senior Pentagon official, summed up the now-historic raid on May 2, 2011, during which SEAL Team Six secretly descended upon a compound in Pakistan, blew down doors and engaged enemy combatants. The goal: kill or capture terrorist leader Osama bin Laden. Gunfire was exchanged as SEALs made their way through the compound, up to the second and third floors, where bin Laden and his family were hiding. Four combatants and one woman used as a human shield were killed in the raid, according to Pentagon statements. “The sole focus of the operation was to kill or capture Osama bin Laden,” a Defense Department official told reporters after the raid, during a background briefing. As we recognize the anniversary of the raid, it would likely be an understatement to call it a “defining moment” in U.S. military history. With its successful completion, the operation ended the life of the man regarded as the mastermind behind the terror attacks of Sept. 11, 2001, which claimed thousands of lives. Tracking Down Bin Laden For years, military and intelligence forces had scoured the globe to find bin Laden’s hideout, and in September 2010, the CIA got the lead it needed when it used surveillance photos and intelligence reports to determine that a known al-Qaida courier was visiting a compound in Abbottabad, Pakistan. Over the next few months, the CIA used informants, surveillance and other intelligence gathering measures to arrive at the conclusion that bin Laden and his family were hiding out in the compound – but up until the attack, there was no hard proof that bin Laden was present, only the best guess available. President Barack Obama discussed the decision to attack with \"60 Minutes\" after the raid. “This was a very difficult decision, in part because the evidence we had was not absolutely conclusive,” he said. Obama said he and his team were not surprised to find bin Laden hiding in plain sight, but were surprised to learn that the compound had been there for so long without information leaking out about it. “I think the image that bin Laden had tried to promote was that he was an ascetic, living in a cave,” Obama told \"60 Minutes.\" “This guy was living in a million-dollar compound in a residential neighborhood.” “It was the best intelligence work I’ve ever seen that enabled those in the military who executed this mission to do so successfully,” former Joint Chiefs of Staff Adm. Mike Mullen told a DoD reporter. “There was only a 55% chance that the 9/11 attack mastermind was in the compound.” Neptune Spear The actual raid on bin Laden’s compound was called Operation Neptune Spear, after the trident that appears on the U.S. Navy's Special Warfare insignia. On the night of the infiltration, two dozen SEALs flew in using two helicopters, beneath the radar and varying routes to avoid detection. News reports described bin Laden’s compound as surrounded with barbed wire and 18-foot walls. There were also seven-foot walls surrounding the balconies. Additionally, the compound was designed to obscure lines of sight from multiple directions. As the raid commenced, the tail of one of the helicopters grazed the compound wall, forcing it into a “soft crash” landing that fortunately did not result in major injuries, although the helicopter itself had to be destroyed so none of its tech would fall into the wrong hands. Using night-vision goggles (power on the street had been cut off), the SEALs infiltrated the compound, killing anyone who put up resistance, securing the women and children, clearing weapons stashes and barricades, and taking anything that could contain secret information, including hard drives and cell phones. The SEALs quickly made their way up to the third floor, where they found the terrorist leader in his bedroom and killed him. Accounts of how this happened differ, depending on the source. SEAL Robert O’Neill has claimed to be the “ SEAL who shot bin Laden ”. SEAL Matt Bissonnette , who entered the room at the same time and wrote about the operation in the book \"No Easy Day,\" also claims to have fired shots into bin Laden's fallen body. In any case, there was no doubt about the outcome of the mission when the SEAL team leader radioed in, \"For God and country -- Geronimo, Geronimo, Geronimo,\" thus declaring the raid a success. At the conclusion of the raid, the SEALs evacuated the area with bin Laden’s body, with an extra Chinook helicopter flown in to provide transportation. From beginning to end, the operation had taken less than 40 minutes. Using known photos and facial recognition software, as well as measuring the corpse, CIA specialists determined with 95% certainty that the body was that of Osama bin Laden. Other specialists in the intelligence community performed initial DNA analysis, which resulted in a virtual 100% DNA match of the body against DNA of several bin Laden family members. Bin Laden was buried at sea. “For us to be able to definitively say, ‘We got the man who caused thousands of deaths here in the United States and who had been the rallying point for a violent extremist jihad around the world’ was something that I think all of us were profoundly grateful to be a part of,” Obama said at the time. Want to Know More About the Military? Be sure to get the latest news about the U.S. military, as well as critical info about how to join and all the benefits of service. Subscribe to Military.com and receive customized updates delivered straight to your inbox. Story Continues Related Topics: Osama bin Laden Afghanistan War Terrorism Navy Special Operations Navy SEALs - Sea-Air-Land © Copyright 2024 Military.com. All rights reserved. This article may not be republished, rebroadcast, rewritten or otherwise distributed without written permission. To reprint or license this article or any content from Military.com, please submit your request here . You May Also Like US Military Presence in Syria Carries Substantial Risks, But So Does Complete Withdrawal Recent reports suggest that amid the growing tension in the region, additional resources and soldiers have made their way to... Years After Fall of Kabul, US Congress Has Still Not Acted to Secure Future of More Than 70,000 Afghan Evacuees in US The act’s failure comes despite bipartisan and wide-ranging popular support to assist Afghans – including among members of... Former SEAL Leader Likely to Face Board of Inquiry in Coming Days over Death of Candidate Kyle Mullen Capt. Bradley Geary, the then-leader of the command that runs the Navy SEAL BUD/s training program, is now facing a board of... The Taliban Are Celebrating 3 Years in Power, but They’re Not Talking About Afghans The Taliban celebrated the third anniversary of its return to power at a former U.S. air base in Afghanistan on Wednesday... Military News Investigations and Features Army Navy Air Force Marine Corps Coast Guard Space Force Military Opinion Weather by Select Service Army Marines Navy Air Force National Guard Coast Guard Space Force Spouse Login Army My Profile Army Home Page Navy My Profile Navy Home Page Air Force My Profile Air Force Home Page Marines My Profile Marines Home Page Coast Guard My Profile Coast Guard Home Page Space Force My Profile Space Force Home Page National Guard My Profile National Guard Home Page Spouse My Profile Spouse Home Page My Profile News Home Page Most Popular Military News Top Enlisted Leader of Army's Washington, DC, Branch Fired After Investigation Command Sgt. Maj. Veronica Knapp was dismissed \"due to a loss of trust and confidence in her leadership,\" a spokesperson for... The Army Sacks Another Senior Enlisted Leader in the Washington, DC, Area Command Sgt. Maj. Harold \"Ed\" Jarrell, the top enlisted leader for the Army's 1st Information Operations Command, was... Former SEAL Leader Likely to Face Board of Inquiry in Coming Days over Death of Candidate Kyle Mullen Capt. Bradley Geary, the then-leader of the command that runs the Navy SEAL BUD/s training program, is now facing a board of... The Army Is Set to Expand Basic Training, Fueled by New Recruiting Momentum Fort Sill in Oklahoma and Fort Leonard Wood in Missouri will get two additional basic training units each -- able to train an... Tim Walz' Military Service Is Under Attack by the Trump Campaign. Here Are the Facts. The attacks, spearheaded by Trump's VP pick Sen. JD Vance, claim Walz ducked a Guard deployment to Iraq and lied about his... Latest Benefits Info Post-9/11 GI Bill Frequently Asked Questions The GI Bill Can Pay for Testing VA Fertility Benefits for Military Veterans The Next Deadline for Backdated PACT Act Payments Is Coming Soon. Here’s What You Need to Know Servicemembers' Group Life Insurance (SGLI): What You Need to Know View more More Military Headlines US Official Says Mideast Mediators Are Preparing to Implement Cease-Fire Deal Before Final Agreement A new “implementation cell” was being established in Cairo, which would focus on the deal’s logistics, including the... 4-Star Commander Denies Request to Wear Boonie Hats at Nellis Air Force Base Amid Standards Push Boonie hats are round, wide-brimmed sun hats worn by troops in particularly hot or tropical climates, such as the jungles of... US Military Presence in Syria Carries Substantial Risks, But So Does Complete Withdrawal Recent reports suggest that amid the growing tension in the region, additional resources and soldiers have made their way to... Okinawa Air Force Clinic Says It Gave 'Potentially Ineffective' Childhood Vaccines as First Day of School Looms The vaccines -- part of the recommended schedule of vaccinations for children -- were administered by the Kadena Immunization... Since Key Bridge Collapse, Several Ships Have Experienced Trouble in Maryland Waters No ship has had a complete loss of power since the Dali did, but in Maryland waters since March 26, there have been at least... View more Army About Half of the Troops Who Deployed on the Gaza Aid Pier Mission Are Home Now The Army Sacks Another Senior Enlisted Leader in the Washington, DC, Area The Army Is Set to Expand Basic Training, Fueled by New Recruiting Momentum View more Air Force 4-Star Commander Denies Request to Wear Boonie Hats at Nellis Air Force Base Amid Standards Push Okinawa Air Force Clinic Says It Gave 'Potentially Ineffective' Childhood Vaccines as First Day of School Looms US Air Force Academy View more Navy USS Kidd, a Big Louisiana Tourist Attraction, Moves to Dry Dock for Major Makeover About Half of the Troops Who Deployed on the Gaza Aid Pier Mission Are Home Now Former SEAL Leader Likely to Face Board of Inquiry in Coming Days over Death of Candidate Kyle Mullen View more Military Benefits Updates The Next Deadline for Backdated PACT Act Payments Is Coming Soon. Here’s What You Need to Know VA Fertility Benefits for Military Veterans Fertility Benefits for Active-Duty Service Members View more Marine Corps Marines and Sailors Ordered to Brush Up on Appropriate Political Activity Amid Heated Election On 12th Anniversary of Marine Vet Austin Tice's Disappearance, Lawmakers Push to Bring Him Home Trucking Company Owner Pleads Guilty After Crash that Killed Marine Vet Bikers View more Coast Guard Since Key Bridge Collapse, Several Ships Have Experienced Trouble in Maryland Waters Coast Guard Announces Newest Icebreaker Will Head to Juneau as Healy Suffers Major Fire Studies in Early 2000s Contemplated 'Pier Protection' and 'Ship Impact' at Key Bridge View more Entertainment Michael Bay Docuseries 'Born Evil' Examines the Crimes of Navy Vet and Serial Killer Hadden Clark A Marine Corps Veteran Goes to War with Corrupt Local Cops in 'Rebel Ridge' 'Zero Day': New 10-Part Series Imagines What a Chinese Invasion of Taiwan Might Actually Look Like View more GO Sign up Profile Profile Resumes Cover Letters Jobs I've Applied To Saved Jobs Saved Searches Subscriptions Log out News News Home Army Navy Air Force Marine Corps Coast Guard Space Force Military Podcasts Opinion Videos Benefits Benefits Home Military Pay and Money GI Bill Veteran Health Care Tricare VA Loans Insurance Retirement VA eBenefits Veteran Jobs Veteran Job Search Military Skills Translator Upload Your Resume Veteran Employment Project Vet Friendly Employers Career Advice Military Life Military Life Home Money Off Duty Fitness Military Trivia Game Veterans Day Spouse & Family Deployment Military History Discounts Discounts Home Featured Discounts Veterans Day Restaurant Discounts Dining Travel Retail Insurance Services Auto Electronics Join the Military Join the Military Home ASVAB Contact a Recruiter Military Fitness Benefits Military.com Network Air Force Army Coast Guard Marine Corps National Guard Navy Space Force About Military.com About Us & Press Room Mobile Apps RSS Advertise with Us Reprints & Permissions Subscriptions User Agreement Privacy Policy Your Privacy Choices Site Map Need customer support? Visit our Customer Support center for solutions or to contact us. Customer Support © 2024 Military Advantage AdChoices\n",
            "\n",
            "\n",
            "\n",
            "Atomic Facts:\n",
            "1. The mission was executed by the Red Squadron of U.S. Navy SEAL Team Six.\n",
            "2. The Red Squadron was chosen for their extensive experience.\n",
            "3. The Red Squadron was chosen for their specialized skills.\n",
            "4. The SEALs were transported by two helicopters.\n",
            "5. The helicopters were piloted by Army aviators.\n",
            "6. The helicopters were transported from a U.S. base in Jalalabad, Afghanistan.\n",
            "7. The helicopters were transported to the compound in Pakistan.\n",
            "8. The mission commenced on May 1, 2011.\n",
            "9. The mission commenced at 10:30 p.m. local time.\n",
            "\n",
            "Corresponding Webpage Contents:\n",
            "Text Content of Webpage 1:\n",
            "Skip to main content 9/11 Memorial | Home Site Navigation Note for screen reader users: once expanded, the first four\r\n",
            "items of the navigation menu have some associated content which may provide additional information of interest. These sections can be located by jumping to the level 3 heading which follows each sub-menu. Visit Tickets Tours Group Visits Museum Admission Discounts New York First Mondays About Your Visit Visitor Guidelines Getting Here Accessibility The Museum Exhibitions The Collection Programs and Events The Memorial Find a Name Outdoor Memorial Audio Guide 9/11 Memorial Glade The Survivor Tree Learn Upcoming Public Programs Past Public Programs Watch Live Students and Teachers School Programs Lesson Plans Anniversary Digital Learning Experience Digital Learning Experience Archives Teacher Professional Development Youth and Families Activities at Home Youth and Family Tours Talking to Children about Terrorism Resources 9/11 Primer Illness & Advocacy After 9/11 Digital Exhibitions Interactive Timelines Oral Histories Connect 9/11 Family Members Visiting Information Find a Name Communities Rescue and Recovery Workers Witnesses and Survivors Service Members and Veterans Artists Commemoration September 11, 2001 Remember the Sky February 26, 1993 May 30, 2002 The MEMO Blog Support Donate Sponsor a Cobblestone Other Ways to Give Founders Become a Member Member Login Get Involved 5K Run/Walk Benefit Dinner Summit on Security Corporate Membership The Never Forget Fund Visionary Network Volunteer Give to the Collection Search Calendar Switch language Language selector Apply Language Log in Log In Don't have an account? Create an account today and support the 9/11 Memorial & Museum. Log in with Google , Facebook or Twitter . Username or Email Username or Email You can use your username or email address to login. Password Password Create new account Forgot your password? Skip to main content All Exhibitions Home Operation Neptune Spear Breadcrumb Home Learn Resources Digital Exhibitions Digital Exhibition: Revealed: The Hunt for Bin Laden U.S. President Barack Obama and members of his national security team monitor the mission in a conference room adjacent to the Situation Room of the White House, May 1, 2011 (EDT). Photograph by Pete Souza, The White House Operation Neptune Spear Two helicopters piloted by U.S. Army aviators transported 23 SEALs, an interpreter, and a combat dog to carry out the raid in Abbottabad, Pakistan. Vest worn by a U.S. Navy SEAL Team Six member during raid on Abbottabad compound. Courtesy of Anonymous. U.S. Navy SEAL Team Six is part of the Joint Special Operations Command, which is part of the U.S. Special Operations Command and conducts classified high-priority missions. The Red Squadron within U.S. Navy SEAL Team Six was chosen for the raid on Osama bin Laden’s compound because it had just returned to the United States from a deployment in Afghanistan. Since they were scheduled for normal leave, the Red Squadron operators were able to train for the mission without attracting unwanted attention. The individual SEALs were selected for their extensive experience in demanding operations and specialized skills. SEAL is an acronym for Sea, Air, and Land, referring to members’ ability to operate in all conditions. Vest worn by combat dog Cairo during raid on Osama bin Laden’s residence. Courtesy of Will Chesney. Two helicopters piloted by Army aviators transported 23 SEALs, an interpreter, and a combat dog to carry out the raid. The mission was dubbed Operation Neptune Spear, a nod to the U.S. Navy SEAL insignia. On May 1, 2011, at 10:30 p.m. local time, the team took off from a U.S. base in Jalalabad, Afghanistan. Two additional helicopters followed into Pakistan with extra fuel and a backup force. Arriving at the compound, one helicopter became unstable, forcing a hard landing inside the compound. The team of SEALs onboard was safe, and the mission continued. Bolt cutters used by a U.S. Navy SEAL Team Six member during Operation Neptune Spear. Courtesy of Anonymous. Inside the compound’s walls, several SEALs approached the guesthouse where one of Osama bin Laden’s couriers lived. The SEALs found the door locked and were preparing to blow it open when they were fired upon through the door. The SEALs returned fire. Soon, the door opened, and a woman came out carrying a child, followed by other children. Beyond her, lying dead, was her husband, Abu Ahmad al-Kuwaiti, bin Laden’s primary courier. Bullet shrapnel from the firefight lodged in the handle of a pair of bolt cutters one of the SEALs carried in his backpack. When I personally got around to the front side of the main house, one of my guys said, “I shot one of the guys, his wife jumped in the middle of it, and I shot her.” And it kind of occurred to me, well, they’re trying to martyr themselves, to save their husbands. Something big is here. —Anonymous, U.S. Navy SEAL Team Six In the main building, SEALs killed the second courier and his wife on the first floor and bin Laden’s son, Khalid, who was armed, on the second-floor landing. They killed Osama bin Laden in his third-floor bedroom, where he was found with at least one weapon nearby. Camcorder recovered from Osama bin Laden’s compound in Abbottabad Courtesy of the FBI Experience, FBI Headquarters, Washington, D.C. Watch recovered from Osama bin Laden’s compound in Abbottabad Courtesy of the FBI Experience, FBI Headquarters, Washington, D.C. With Osama bin Laden dead, the SEALs collected documents, electronics, and other materials for intelligence exploitation. As a backup helicopter arrived, they blew up the downed helicopter to destroy its stealth technology. Locals gathered outside the compound, and Pakistani authorities began to mobilize a response to the late-night disturbance. After about 45 minutes on the ground, the SEALs boarded the helicopters, taking with them the collected materials and bin Laden’s body. In Afghanistan, law enforcement and intelligence officers analyzed bin Laden’s DNA and took his fingerprints. Facial recognition analysis, together with the biometric information, ultimately confirmed his identity. American forces then flew bin Laden’s body to the aircraft carrier USS Carl Vinson in the Arabian Sea. After religious funeral rites, Osama bin Laden was buried at sea so his gravesite would not become a shrine for his followers. Lesson Plan: Operation Neptune Spear and Its Impact Essential Question: How was Operation Neptune Spear carried out and what was the country’s reaction to its outcome? Learn more Story: Immediate Reactions At 11:35 p.m. in Washington, D.C., on May 1, 2011 (ET), U.S. President Barack Obama announced the killing of Osama bin Laden on live television. Learn more Footer Information about the 9 11 Memorial & Museum Location 180 Greenwich Street New York, NY 10007 View on map reservations@911memorial.org (212) 312-8800 Contact Us About Media Center Hours 9/11 Memorial Daily, 8 a.m.-8 p.m. 9/11 Museum Weds-Mon & select Tues , 9 a.m.-7 p.m. Get Tickets Ticketholder Benefits Museum Store Learn More Careers Accessibility Calendar Support Stay Informed Join our mailing list to receive the latest updates on news, events, and programming. Enter your e-mail Footer navigation Financial & Legal Information Terms of Use/Privacy Policy Accessibility Details Social media links Facebook TikTok Instagram Twitter YouTube Copyright information © 2023 National September 11 Memorial & Museum 9/11 MEMORIAL is a registered trademark of the National September 11 Memorial & Museum Visit Tickets Tours Group Visits Museum Admission Discounts New York First Mondays About Your Visit Visitor Guidelines Getting Here Accessibility The Museum Exhibitions The Collection Programs and Events The Memorial Find a Name Outdoor Memorial Audio Guide 9/11 Memorial Glade The Survivor Tree Learn Upcoming Public Programs Past Public Programs Watch Live Students and Teachers School Programs Lesson Plans Anniversary Digital Learning Experience Digital Learning Experience Archives Teacher Professional Development Youth and Families Activities at Home Youth and Family Tours Talking to Children about Terrorism Resources 9/11 Primer Illness & Advocacy After 9/11 Digital Exhibitions Interactive Timelines Oral Histories Connect 9/11 Family Members Visiting Information Find a Name Communities Rescue and Recovery Workers Witnesses and Survivors Service Members and Veterans Artists Commemoration September 11, 2001 Remember the Sky February 26, 1993 May 30, 2002 The MEMO Blog Support Donate Sponsor a Cobblestone Other Ways to Give Founders Become a Member Member Login Get Involved 5K Run/Walk Benefit Dinner Summit on Security Corporate Membership The Never Forget Fund Visionary Network Volunteer Give to the Collection Search Calendar\n",
            "\n",
            "\n",
            "\n",
            "Atomic Facts:\n",
            "1. One helicopter experienced instability.\n",
            "2. One helicopter made a hard landing inside the compound.\n",
            "3. The SEALs continued the mission without injury.\n",
            "4. The team engaged in a firefight.\n",
            "5. The team moved through the compound.\n",
            "6. The team encountered resistance.\n",
            "7. The team killed several combatants.\n",
            "8. The team killed bin Laden's couriers.\n",
            "9. The team killed bin Laden's brother, Khalid.\n",
            "\n",
            "Corresponding Webpage Contents:\n",
            "Text Content of Webpage 1:\n",
            "Skip to main content 9/11 Memorial | Home Site Navigation Note for screen reader users: once expanded, the first four\r\n",
            "items of the navigation menu have some associated content which may provide additional information of interest. These sections can be located by jumping to the level 3 heading which follows each sub-menu. Visit Tickets Tours Group Visits Museum Admission Discounts New York First Mondays About Your Visit Visitor Guidelines Getting Here Accessibility The Museum Exhibitions The Collection Programs and Events The Memorial Find a Name Outdoor Memorial Audio Guide 9/11 Memorial Glade The Survivor Tree Learn Upcoming Public Programs Past Public Programs Watch Live Students and Teachers School Programs Lesson Plans Anniversary Digital Learning Experience Digital Learning Experience Archives Teacher Professional Development Youth and Families Activities at Home Youth and Family Tours Talking to Children about Terrorism Resources 9/11 Primer Illness & Advocacy After 9/11 Digital Exhibitions Interactive Timelines Oral Histories Connect 9/11 Family Members Visiting Information Find a Name Communities Rescue and Recovery Workers Witnesses and Survivors Service Members and Veterans Artists Commemoration September 11, 2001 Remember the Sky February 26, 1993 May 30, 2002 The MEMO Blog Support Donate Sponsor a Cobblestone Other Ways to Give Founders Become a Member Member Login Get Involved 5K Run/Walk Benefit Dinner Summit on Security Corporate Membership The Never Forget Fund Visionary Network Volunteer Give to the Collection Search Calendar Switch language Language selector Apply Language Log in Log In Don't have an account? Create an account today and support the 9/11 Memorial & Museum. Log in with Google , Facebook or Twitter . Username or Email Username or Email You can use your username or email address to login. Password Password Create new account Forgot your password? Skip to main content All Exhibitions Home Operation Neptune Spear Breadcrumb Home Learn Resources Digital Exhibitions Digital Exhibition: Revealed: The Hunt for Bin Laden U.S. President Barack Obama and members of his national security team monitor the mission in a conference room adjacent to the Situation Room of the White House, May 1, 2011 (EDT). Photograph by Pete Souza, The White House Operation Neptune Spear Two helicopters piloted by U.S. Army aviators transported 23 SEALs, an interpreter, and a combat dog to carry out the raid in Abbottabad, Pakistan. Vest worn by a U.S. Navy SEAL Team Six member during raid on Abbottabad compound. Courtesy of Anonymous. U.S. Navy SEAL Team Six is part of the Joint Special Operations Command, which is part of the U.S. Special Operations Command and conducts classified high-priority missions. The Red Squadron within U.S. Navy SEAL Team Six was chosen for the raid on Osama bin Laden’s compound because it had just returned to the United States from a deployment in Afghanistan. Since they were scheduled for normal leave, the Red Squadron operators were able to train for the mission without attracting unwanted attention. The individual SEALs were selected for their extensive experience in demanding operations and specialized skills. SEAL is an acronym for Sea, Air, and Land, referring to members’ ability to operate in all conditions. Vest worn by combat dog Cairo during raid on Osama bin Laden’s residence. Courtesy of Will Chesney. Two helicopters piloted by Army aviators transported 23 SEALs, an interpreter, and a combat dog to carry out the raid. The mission was dubbed Operation Neptune Spear, a nod to the U.S. Navy SEAL insignia. On May 1, 2011, at 10:30 p.m. local time, the team took off from a U.S. base in Jalalabad, Afghanistan. Two additional helicopters followed into Pakistan with extra fuel and a backup force. Arriving at the compound, one helicopter became unstable, forcing a hard landing inside the compound. The team of SEALs onboard was safe, and the mission continued. Bolt cutters used by a U.S. Navy SEAL Team Six member during Operation Neptune Spear. Courtesy of Anonymous. Inside the compound’s walls, several SEALs approached the guesthouse where one of Osama bin Laden’s couriers lived. The SEALs found the door locked and were preparing to blow it open when they were fired upon through the door. The SEALs returned fire. Soon, the door opened, and a woman came out carrying a child, followed by other children. Beyond her, lying dead, was her husband, Abu Ahmad al-Kuwaiti, bin Laden’s primary courier. Bullet shrapnel from the firefight lodged in the handle of a pair of bolt cutters one of the SEALs carried in his backpack. When I personally got around to the front side of the main house, one of my guys said, “I shot one of the guys, his wife jumped in the middle of it, and I shot her.” And it kind of occurred to me, well, they’re trying to martyr themselves, to save their husbands. Something big is here. —Anonymous, U.S. Navy SEAL Team Six In the main building, SEALs killed the second courier and his wife on the first floor and bin Laden’s son, Khalid, who was armed, on the second-floor landing. They killed Osama bin Laden in his third-floor bedroom, where he was found with at least one weapon nearby. Camcorder recovered from Osama bin Laden’s compound in Abbottabad Courtesy of the FBI Experience, FBI Headquarters, Washington, D.C. Watch recovered from Osama bin Laden’s compound in Abbottabad Courtesy of the FBI Experience, FBI Headquarters, Washington, D.C. With Osama bin Laden dead, the SEALs collected documents, electronics, and other materials for intelligence exploitation. As a backup helicopter arrived, they blew up the downed helicopter to destroy its stealth technology. Locals gathered outside the compound, and Pakistani authorities began to mobilize a response to the late-night disturbance. After about 45 minutes on the ground, the SEALs boarded the helicopters, taking with them the collected materials and bin Laden’s body. In Afghanistan, law enforcement and intelligence officers analyzed bin Laden’s DNA and took his fingerprints. Facial recognition analysis, together with the biometric information, ultimately confirmed his identity. American forces then flew bin Laden’s body to the aircraft carrier USS Carl Vinson in the Arabian Sea. After religious funeral rites, Osama bin Laden was buried at sea so his gravesite would not become a shrine for his followers. Lesson Plan: Operation Neptune Spear and Its Impact Essential Question: How was Operation Neptune Spear carried out and what was the country’s reaction to its outcome? Learn more Story: Immediate Reactions At 11:35 p.m. in Washington, D.C., on May 1, 2011 (ET), U.S. President Barack Obama announced the killing of Osama bin Laden on live television. Learn more Footer Information about the 9 11 Memorial & Museum Location 180 Greenwich Street New York, NY 10007 View on map reservations@911memorial.org (212) 312-8800 Contact Us About Media Center Hours 9/11 Memorial Daily, 8 a.m.-8 p.m. 9/11 Museum Weds-Mon & select Tues , 9 a.m.-7 p.m. Get Tickets Ticketholder Benefits Museum Store Learn More Careers Accessibility Calendar Support Stay Informed Join our mailing list to receive the latest updates on news, events, and programming. Enter your e-mail Footer navigation Financial & Legal Information Terms of Use/Privacy Policy Accessibility Details Social media links Facebook TikTok Instagram Twitter YouTube Copyright information © 2023 National September 11 Memorial & Museum 9/11 MEMORIAL is a registered trademark of the National September 11 Memorial & Museum Visit Tickets Tours Group Visits Museum Admission Discounts New York First Mondays About Your Visit Visitor Guidelines Getting Here Accessibility The Museum Exhibitions The Collection Programs and Events The Memorial Find a Name Outdoor Memorial Audio Guide 9/11 Memorial Glade The Survivor Tree Learn Upcoming Public Programs Past Public Programs Watch Live Students and Teachers School Programs Lesson Plans Anniversary Digital Learning Experience Digital Learning Experience Archives Teacher Professional Development Youth and Families Activities at Home Youth and Family Tours Talking to Children about Terrorism Resources 9/11 Primer Illness & Advocacy After 9/11 Digital Exhibitions Interactive Timelines Oral Histories Connect 9/11 Family Members Visiting Information Find a Name Communities Rescue and Recovery Workers Witnesses and Survivors Service Members and Veterans Artists Commemoration September 11, 2001 Remember the Sky February 26, 1993 May 30, 2002 The MEMO Blog Support Donate Sponsor a Cobblestone Other Ways to Give Founders Become a Member Member Login Get Involved 5K Run/Walk Benefit Dinner Summit on Security Corporate Membership The Never Forget Fund Visionary Network Volunteer Give to the Collection Search Calendar\n",
            "Text Content of Webpage 2:\n",
            "Profile Profile Resumes Cover Letters Jobs I've Applied To Saved Jobs Saved Searches Subscriptions Log out News News Home Army Navy Air Force Marine Corps Coast Guard Space Force Military Podcasts Opinion Videos Benefits Benefits Home Military Pay and Money GI Bill Veteran Health Care Tricare VA Loans Insurance Retirement VA eBenefits Veteran Jobs Veteran Job Search Military Skills Translator Upload Your Resume Veteran Employment Project Vet Friendly Employers Career Advice Military Life Military Life Home Money Off Duty Fitness Military Trivia Game Veterans Day Spouse & Family Deployment Military History Discounts Discounts Home Featured Discounts Veterans Day Restaurant Discounts Dining Travel Retail Insurance Services Auto Electronics Join the Military Join the Military Home ASVAB Contact a Recruiter Military Fitness Benefits Login Sign up Profile Resumes Cover Letters Jobs I've Applied To Saved Jobs Saved Searches Subscriptions Log out Sign up GO Sign up Profile Profile Resumes Cover Letters Jobs I've Applied To Saved Jobs Saved Searches Subscriptions Log out News News Home Army Navy Air Force Marine Corps Coast Guard Space Force Military Podcasts Opinion Videos Benefits Benefits Home Military Pay and Money GI Bill Veteran Health Care Tricare VA Loans Insurance Retirement VA eBenefits Veteran Jobs Veteran Job Search Military Skills Translator Upload Your Resume Veteran Employment Project Vet Friendly Employers Career Advice Military Life Military Life Home Money Off Duty Fitness Military Trivia Game Veterans Day Spouse & Family Deployment Military History Discounts Discounts Home Featured Discounts Veterans Day Restaurant Discounts Dining Travel Retail Insurance Services Auto Electronics Join the Military Join the Military Home ASVAB Contact a Recruiter Military Fitness Benefits History Military Trivia Game Veterans Day Army Birthday Navy Birthday Marine Corps Birthday Air Force Birthday Coast Guard Birthday More Special Operations Navy SEALs Missions The Operation That Took Out Osama Bin Laden Military.com | By \n",
            " \n",
            " Kris Osborn & Ho Lin Updated April 06, 2022 | Published April 30, 2018 “The American team engaged in a firefight. Osama bin Laden did resist.” These words, uttered by a senior Pentagon official, summed up the now-historic raid on May 2, 2011, during which SEAL Team Six secretly descended upon a compound in Pakistan, blew down doors and engaged enemy combatants. The goal: kill or capture terrorist leader Osama bin Laden. Gunfire was exchanged as SEALs made their way through the compound, up to the second and third floors, where bin Laden and his family were hiding. Four combatants and one woman used as a human shield were killed in the raid, according to Pentagon statements. “The sole focus of the operation was to kill or capture Osama bin Laden,” a Defense Department official told reporters after the raid, during a background briefing. As we recognize the anniversary of the raid, it would likely be an understatement to call it a “defining moment” in U.S. military history. With its successful completion, the operation ended the life of the man regarded as the mastermind behind the terror attacks of Sept. 11, 2001, which claimed thousands of lives. Tracking Down Bin Laden For years, military and intelligence forces had scoured the globe to find bin Laden’s hideout, and in September 2010, the CIA got the lead it needed when it used surveillance photos and intelligence reports to determine that a known al-Qaida courier was visiting a compound in Abbottabad, Pakistan. Over the next few months, the CIA used informants, surveillance and other intelligence gathering measures to arrive at the conclusion that bin Laden and his family were hiding out in the compound – but up until the attack, there was no hard proof that bin Laden was present, only the best guess available. President Barack Obama discussed the decision to attack with \"60 Minutes\" after the raid. “This was a very difficult decision, in part because the evidence we had was not absolutely conclusive,” he said. Obama said he and his team were not surprised to find bin Laden hiding in plain sight, but were surprised to learn that the compound had been there for so long without information leaking out about it. “I think the image that bin Laden had tried to promote was that he was an ascetic, living in a cave,” Obama told \"60 Minutes.\" “This guy was living in a million-dollar compound in a residential neighborhood.” “It was the best intelligence work I’ve ever seen that enabled those in the military who executed this mission to do so successfully,” former Joint Chiefs of Staff Adm. Mike Mullen told a DoD reporter. “There was only a 55% chance that the 9/11 attack mastermind was in the compound.” Neptune Spear The actual raid on bin Laden’s compound was called Operation Neptune Spear, after the trident that appears on the U.S. Navy's Special Warfare insignia. On the night of the infiltration, two dozen SEALs flew in using two helicopters, beneath the radar and varying routes to avoid detection. News reports described bin Laden’s compound as surrounded with barbed wire and 18-foot walls. There were also seven-foot walls surrounding the balconies. Additionally, the compound was designed to obscure lines of sight from multiple directions. As the raid commenced, the tail of one of the helicopters grazed the compound wall, forcing it into a “soft crash” landing that fortunately did not result in major injuries, although the helicopter itself had to be destroyed so none of its tech would fall into the wrong hands. Using night-vision goggles (power on the street had been cut off), the SEALs infiltrated the compound, killing anyone who put up resistance, securing the women and children, clearing weapons stashes and barricades, and taking anything that could contain secret information, including hard drives and cell phones. The SEALs quickly made their way up to the third floor, where they found the terrorist leader in his bedroom and killed him. Accounts of how this happened differ, depending on the source. SEAL Robert O’Neill has claimed to be the “ SEAL who shot bin Laden ”. SEAL Matt Bissonnette , who entered the room at the same time and wrote about the operation in the book \"No Easy Day,\" also claims to have fired shots into bin Laden's fallen body. In any case, there was no doubt about the outcome of the mission when the SEAL team leader radioed in, \"For God and country -- Geronimo, Geronimo, Geronimo,\" thus declaring the raid a success. At the conclusion of the raid, the SEALs evacuated the area with bin Laden’s body, with an extra Chinook helicopter flown in to provide transportation. From beginning to end, the operation had taken less than 40 minutes. Using known photos and facial recognition software, as well as measuring the corpse, CIA specialists determined with 95% certainty that the body was that of Osama bin Laden. Other specialists in the intelligence community performed initial DNA analysis, which resulted in a virtual 100% DNA match of the body against DNA of several bin Laden family members. Bin Laden was buried at sea. “For us to be able to definitively say, ‘We got the man who caused thousands of deaths here in the United States and who had been the rallying point for a violent extremist jihad around the world’ was something that I think all of us were profoundly grateful to be a part of,” Obama said at the time. Want to Know More About the Military? Be sure to get the latest news about the U.S. military, as well as critical info about how to join and all the benefits of service. Subscribe to Military.com and receive customized updates delivered straight to your inbox. Story Continues Related Topics: Osama bin Laden Afghanistan War Terrorism Navy Special Operations Navy SEALs - Sea-Air-Land © Copyright 2024 Military.com. All rights reserved. This article may not be republished, rebroadcast, rewritten or otherwise distributed without written permission. To reprint or license this article or any content from Military.com, please submit your request here . You May Also Like US Military Presence in Syria Carries Substantial Risks, But So Does Complete Withdrawal Recent reports suggest that amid the growing tension in the region, additional resources and soldiers have made their way to... Years After Fall of Kabul, US Congress Has Still Not Acted to Secure Future of More Than 70,000 Afghan Evacuees in US The act’s failure comes despite bipartisan and wide-ranging popular support to assist Afghans – including among members of... Former SEAL Leader Likely to Face Board of Inquiry in Coming Days over Death of Candidate Kyle Mullen Capt. Bradley Geary, the then-leader of the command that runs the Navy SEAL BUD/s training program, is now facing a board of... The Taliban Are Celebrating 3 Years in Power, but They’re Not Talking About Afghans The Taliban celebrated the third anniversary of its return to power at a former U.S. air base in Afghanistan on Wednesday... Military News Investigations and Features Army Navy Air Force Marine Corps Coast Guard Space Force Military Opinion Weather by Select Service Army Marines Navy Air Force National Guard Coast Guard Space Force Spouse Login Army My Profile Army Home Page Navy My Profile Navy Home Page Air Force My Profile Air Force Home Page Marines My Profile Marines Home Page Coast Guard My Profile Coast Guard Home Page Space Force My Profile Space Force Home Page National Guard My Profile National Guard Home Page Spouse My Profile Spouse Home Page My Profile News Home Page Most Popular Military News Top Enlisted Leader of Army's Washington, DC, Branch Fired After Investigation Command Sgt. Maj. Veronica Knapp was dismissed \"due to a loss of trust and confidence in her leadership,\" a spokesperson for... The Army Sacks Another Senior Enlisted Leader in the Washington, DC, Area Command Sgt. Maj. Harold \"Ed\" Jarrell, the top enlisted leader for the Army's 1st Information Operations Command, was... Former SEAL Leader Likely to Face Board of Inquiry in Coming Days over Death of Candidate Kyle Mullen Capt. Bradley Geary, the then-leader of the command that runs the Navy SEAL BUD/s training program, is now facing a board of... The Army Is Set to Expand Basic Training, Fueled by New Recruiting Momentum Fort Sill in Oklahoma and Fort Leonard Wood in Missouri will get two additional basic training units each -- able to train an... Tim Walz' Military Service Is Under Attack by the Trump Campaign. Here Are the Facts. The attacks, spearheaded by Trump's VP pick Sen. JD Vance, claim Walz ducked a Guard deployment to Iraq and lied about his... Latest Benefits Info Post-9/11 GI Bill Frequently Asked Questions The GI Bill Can Pay for Testing VA Fertility Benefits for Military Veterans The Next Deadline for Backdated PACT Act Payments Is Coming Soon. Here’s What You Need to Know Servicemembers' Group Life Insurance (SGLI): What You Need to Know View more More Military Headlines US Official Says Mideast Mediators Are Preparing to Implement Cease-Fire Deal Before Final Agreement A new “implementation cell” was being established in Cairo, which would focus on the deal’s logistics, including the... 4-Star Commander Denies Request to Wear Boonie Hats at Nellis Air Force Base Amid Standards Push Boonie hats are round, wide-brimmed sun hats worn by troops in particularly hot or tropical climates, such as the jungles of... US Military Presence in Syria Carries Substantial Risks, But So Does Complete Withdrawal Recent reports suggest that amid the growing tension in the region, additional resources and soldiers have made their way to... Okinawa Air Force Clinic Says It Gave 'Potentially Ineffective' Childhood Vaccines as First Day of School Looms The vaccines -- part of the recommended schedule of vaccinations for children -- were administered by the Kadena Immunization... Since Key Bridge Collapse, Several Ships Have Experienced Trouble in Maryland Waters No ship has had a complete loss of power since the Dali did, but in Maryland waters since March 26, there have been at least... View more Army About Half of the Troops Who Deployed on the Gaza Aid Pier Mission Are Home Now The Army Sacks Another Senior Enlisted Leader in the Washington, DC, Area The Army Is Set to Expand Basic Training, Fueled by New Recruiting Momentum View more Air Force 4-Star Commander Denies Request to Wear Boonie Hats at Nellis Air Force Base Amid Standards Push Okinawa Air Force Clinic Says It Gave 'Potentially Ineffective' Childhood Vaccines as First Day of School Looms US Air Force Academy View more Navy USS Kidd, a Big Louisiana Tourist Attraction, Moves to Dry Dock for Major Makeover About Half of the Troops Who Deployed on the Gaza Aid Pier Mission Are Home Now Former SEAL Leader Likely to Face Board of Inquiry in Coming Days over Death of Candidate Kyle Mullen View more Military Benefits Updates The Next Deadline for Backdated PACT Act Payments Is Coming Soon. Here’s What You Need to Know VA Fertility Benefits for Military Veterans Fertility Benefits for Active-Duty Service Members View more Marine Corps Marines and Sailors Ordered to Brush Up on Appropriate Political Activity Amid Heated Election On 12th Anniversary of Marine Vet Austin Tice's Disappearance, Lawmakers Push to Bring Him Home Trucking Company Owner Pleads Guilty After Crash that Killed Marine Vet Bikers View more Coast Guard Since Key Bridge Collapse, Several Ships Have Experienced Trouble in Maryland Waters Coast Guard Announces Newest Icebreaker Will Head to Juneau as Healy Suffers Major Fire Studies in Early 2000s Contemplated 'Pier Protection' and 'Ship Impact' at Key Bridge View more Entertainment Michael Bay Docuseries 'Born Evil' Examines the Crimes of Navy Vet and Serial Killer Hadden Clark A Marine Corps Veteran Goes to War with Corrupt Local Cops in 'Rebel Ridge' 'Zero Day': New 10-Part Series Imagines What a Chinese Invasion of Taiwan Might Actually Look Like View more GO Sign up Profile Profile Resumes Cover Letters Jobs I've Applied To Saved Jobs Saved Searches Subscriptions Log out News News Home Army Navy Air Force Marine Corps Coast Guard Space Force Military Podcasts Opinion Videos Benefits Benefits Home Military Pay and Money GI Bill Veteran Health Care Tricare VA Loans Insurance Retirement VA eBenefits Veteran Jobs Veteran Job Search Military Skills Translator Upload Your Resume Veteran Employment Project Vet Friendly Employers Career Advice Military Life Military Life Home Money Off Duty Fitness Military Trivia Game Veterans Day Spouse & Family Deployment Military History Discounts Discounts Home Featured Discounts Veterans Day Restaurant Discounts Dining Travel Retail Insurance Services Auto Electronics Join the Military Join the Military Home ASVAB Contact a Recruiter Military Fitness Benefits Military.com Network Air Force Army Coast Guard Marine Corps National Guard Navy Space Force About Military.com About Us & Press Room Mobile Apps RSS Advertise with Us Reprints & Permissions Subscriptions User Agreement Privacy Policy Your Privacy Choices Site Map Need customer support? Visit our Customer Support center for solutions or to contact us. Customer Support © 2024 Military Advantage AdChoices\n",
            "\n",
            "\n",
            "\n",
            "Atomic Facts:\n",
            "1. Osama bin Laden was found on the third floor of the main building.\n",
            "2. Osama bin Laden was killed in his parking.\n",
            "3. Osama bin Laden was found with at least one weapon nearby.\n",
            "4. The SEALs collected documents for intelligence purposes.\n",
            "5. The SEALs collected electronics for intelligence purposes.\n",
            "6. The SEALs destroyed the downed helicopter to protect its stealth technology.\n",
            "\n",
            "Corresponding Webpage Contents:\n",
            "Text Content of Webpage 1:\n",
            "Skip to main content 9/11 Memorial | Home Site Navigation Note for screen reader users: once expanded, the first four\r\n",
            "items of the navigation menu have some associated content which may provide additional information of interest. These sections can be located by jumping to the level 3 heading which follows each sub-menu. Visit Tickets Tours Group Visits Museum Admission Discounts New York First Mondays About Your Visit Visitor Guidelines Getting Here Accessibility The Museum Exhibitions The Collection Programs and Events The Memorial Find a Name Outdoor Memorial Audio Guide 9/11 Memorial Glade The Survivor Tree Learn Upcoming Public Programs Past Public Programs Watch Live Students and Teachers School Programs Lesson Plans Anniversary Digital Learning Experience Digital Learning Experience Archives Teacher Professional Development Youth and Families Activities at Home Youth and Family Tours Talking to Children about Terrorism Resources 9/11 Primer Illness & Advocacy After 9/11 Digital Exhibitions Interactive Timelines Oral Histories Connect 9/11 Family Members Visiting Information Find a Name Communities Rescue and Recovery Workers Witnesses and Survivors Service Members and Veterans Artists Commemoration September 11, 2001 Remember the Sky February 26, 1993 May 30, 2002 The MEMO Blog Support Donate Sponsor a Cobblestone Other Ways to Give Founders Become a Member Member Login Get Involved 5K Run/Walk Benefit Dinner Summit on Security Corporate Membership The Never Forget Fund Visionary Network Volunteer Give to the Collection Search Calendar Switch language Language selector Apply Language Log in Log In Don't have an account? Create an account today and support the 9/11 Memorial & Museum. Log in with Google , Facebook or Twitter . Username or Email Username or Email You can use your username or email address to login. Password Password Create new account Forgot your password? Skip to main content All Exhibitions Home Operation Neptune Spear Breadcrumb Home Learn Resources Digital Exhibitions Digital Exhibition: Revealed: The Hunt for Bin Laden U.S. President Barack Obama and members of his national security team monitor the mission in a conference room adjacent to the Situation Room of the White House, May 1, 2011 (EDT). Photograph by Pete Souza, The White House Operation Neptune Spear Two helicopters piloted by U.S. Army aviators transported 23 SEALs, an interpreter, and a combat dog to carry out the raid in Abbottabad, Pakistan. Vest worn by a U.S. Navy SEAL Team Six member during raid on Abbottabad compound. Courtesy of Anonymous. U.S. Navy SEAL Team Six is part of the Joint Special Operations Command, which is part of the U.S. Special Operations Command and conducts classified high-priority missions. The Red Squadron within U.S. Navy SEAL Team Six was chosen for the raid on Osama bin Laden’s compound because it had just returned to the United States from a deployment in Afghanistan. Since they were scheduled for normal leave, the Red Squadron operators were able to train for the mission without attracting unwanted attention. The individual SEALs were selected for their extensive experience in demanding operations and specialized skills. SEAL is an acronym for Sea, Air, and Land, referring to members’ ability to operate in all conditions. Vest worn by combat dog Cairo during raid on Osama bin Laden’s residence. Courtesy of Will Chesney. Two helicopters piloted by Army aviators transported 23 SEALs, an interpreter, and a combat dog to carry out the raid. The mission was dubbed Operation Neptune Spear, a nod to the U.S. Navy SEAL insignia. On May 1, 2011, at 10:30 p.m. local time, the team took off from a U.S. base in Jalalabad, Afghanistan. Two additional helicopters followed into Pakistan with extra fuel and a backup force. Arriving at the compound, one helicopter became unstable, forcing a hard landing inside the compound. The team of SEALs onboard was safe, and the mission continued. Bolt cutters used by a U.S. Navy SEAL Team Six member during Operation Neptune Spear. Courtesy of Anonymous. Inside the compound’s walls, several SEALs approached the guesthouse where one of Osama bin Laden’s couriers lived. The SEALs found the door locked and were preparing to blow it open when they were fired upon through the door. The SEALs returned fire. Soon, the door opened, and a woman came out carrying a child, followed by other children. Beyond her, lying dead, was her husband, Abu Ahmad al-Kuwaiti, bin Laden’s primary courier. Bullet shrapnel from the firefight lodged in the handle of a pair of bolt cutters one of the SEALs carried in his backpack. When I personally got around to the front side of the main house, one of my guys said, “I shot one of the guys, his wife jumped in the middle of it, and I shot her.” And it kind of occurred to me, well, they’re trying to martyr themselves, to save their husbands. Something big is here. —Anonymous, U.S. Navy SEAL Team Six In the main building, SEALs killed the second courier and his wife on the first floor and bin Laden’s son, Khalid, who was armed, on the second-floor landing. They killed Osama bin Laden in his third-floor bedroom, where he was found with at least one weapon nearby. Camcorder recovered from Osama bin Laden’s compound in Abbottabad Courtesy of the FBI Experience, FBI Headquarters, Washington, D.C. Watch recovered from Osama bin Laden’s compound in Abbottabad Courtesy of the FBI Experience, FBI Headquarters, Washington, D.C. With Osama bin Laden dead, the SEALs collected documents, electronics, and other materials for intelligence exploitation. As a backup helicopter arrived, they blew up the downed helicopter to destroy its stealth technology. Locals gathered outside the compound, and Pakistani authorities began to mobilize a response to the late-night disturbance. After about 45 minutes on the ground, the SEALs boarded the helicopters, taking with them the collected materials and bin Laden’s body. In Afghanistan, law enforcement and intelligence officers analyzed bin Laden’s DNA and took his fingerprints. Facial recognition analysis, together with the biometric information, ultimately confirmed his identity. American forces then flew bin Laden’s body to the aircraft carrier USS Carl Vinson in the Arabian Sea. After religious funeral rites, Osama bin Laden was buried at sea so his gravesite would not become a shrine for his followers. Lesson Plan: Operation Neptune Spear and Its Impact Essential Question: How was Operation Neptune Spear carried out and what was the country’s reaction to its outcome? Learn more Story: Immediate Reactions At 11:35 p.m. in Washington, D.C., on May 1, 2011 (ET), U.S. President Barack Obama announced the killing of Osama bin Laden on live television. Learn more Footer Information about the 9 11 Memorial & Museum Location 180 Greenwich Street New York, NY 10007 View on map reservations@911memorial.org (212) 312-8800 Contact Us About Media Center Hours 9/11 Memorial Daily, 8 a.m.-8 p.m. 9/11 Museum Weds-Mon & select Tues , 9 a.m.-7 p.m. Get Tickets Ticketholder Benefits Museum Store Learn More Careers Accessibility Calendar Support Stay Informed Join our mailing list to receive the latest updates on news, events, and programming. Enter your e-mail Footer navigation Financial & Legal Information Terms of Use/Privacy Policy Accessibility Details Social media links Facebook TikTok Instagram Twitter YouTube Copyright information © 2023 National September 11 Memorial & Museum 9/11 MEMORIAL is a registered trademark of the National September 11 Memorial & Museum Visit Tickets Tours Group Visits Museum Admission Discounts New York First Mondays About Your Visit Visitor Guidelines Getting Here Accessibility The Museum Exhibitions The Collection Programs and Events The Memorial Find a Name Outdoor Memorial Audio Guide 9/11 Memorial Glade The Survivor Tree Learn Upcoming Public Programs Past Public Programs Watch Live Students and Teachers School Programs Lesson Plans Anniversary Digital Learning Experience Digital Learning Experience Archives Teacher Professional Development Youth and Families Activities at Home Youth and Family Tours Talking to Children about Terrorism Resources 9/11 Primer Illness & Advocacy After 9/11 Digital Exhibitions Interactive Timelines Oral Histories Connect 9/11 Family Members Visiting Information Find a Name Communities Rescue and Recovery Workers Witnesses and Survivors Service Members and Veterans Artists Commemoration September 11, 2001 Remember the Sky February 26, 1993 May 30, 2002 The MEMO Blog Support Donate Sponsor a Cobblestone Other Ways to Give Founders Become a Member Member Login Get Involved 5K Run/Walk Benefit Dinner Summit on Security Corporate Membership The Never Forget Fund Visionary Network Volunteer Give to the Collection Search Calendar\n",
            "\n",
            "\n",
            "\n",
            "Atomic Facts:\n",
            "1. The SEALs spent about 45 minutes on the ground.\n",
            "2. The SEALs departed with bin Laden's body.\n",
            "3. Bin Laden's identity was confirmed in Afghanistan.\n",
            "4. Bin Laden's identity was confirmed through DNA analysis.\n",
            "5. Bin Laden's identity was confirmed through fingerprinting.\n",
            "6. Bin Laden's identity was confirmed through facial recognition.\n",
            "7. The body was flown to the USS Carl Vinson.\n",
            "8. The USS Carl Vinson is located in the Arabian Sea.\n",
            "9. Bin Laden was buried at sea.\n",
            "10. Bin Laden was buried following Islamic funeral rites.\n",
            "11. The burial was to prevent his gravesite from becoming a shrine.\n",
            "\n",
            "Corresponding Webpage Contents:\n",
            "Text Content of Webpage 1:\n",
            "Skip to main content 9/11 Memorial | Home Site Navigation Note for screen reader users: once expanded, the first four\r\n",
            "items of the navigation menu have some associated content which may provide additional information of interest. These sections can be located by jumping to the level 3 heading which follows each sub-menu. Visit Tickets Tours Group Visits Museum Admission Discounts New York First Mondays About Your Visit Visitor Guidelines Getting Here Accessibility The Museum Exhibitions The Collection Programs and Events The Memorial Find a Name Outdoor Memorial Audio Guide 9/11 Memorial Glade The Survivor Tree Learn Upcoming Public Programs Past Public Programs Watch Live Students and Teachers School Programs Lesson Plans Anniversary Digital Learning Experience Digital Learning Experience Archives Teacher Professional Development Youth and Families Activities at Home Youth and Family Tours Talking to Children about Terrorism Resources 9/11 Primer Illness & Advocacy After 9/11 Digital Exhibitions Interactive Timelines Oral Histories Connect 9/11 Family Members Visiting Information Find a Name Communities Rescue and Recovery Workers Witnesses and Survivors Service Members and Veterans Artists Commemoration September 11, 2001 Remember the Sky February 26, 1993 May 30, 2002 The MEMO Blog Support Donate Sponsor a Cobblestone Other Ways to Give Founders Become a Member Member Login Get Involved 5K Run/Walk Benefit Dinner Summit on Security Corporate Membership The Never Forget Fund Visionary Network Volunteer Give to the Collection Search Calendar Switch language Language selector Apply Language Log in Log In Don't have an account? Create an account today and support the 9/11 Memorial & Museum. Log in with Google , Facebook or Twitter . Username or Email Username or Email You can use your username or email address to login. Password Password Create new account Forgot your password? Skip to main content All Exhibitions Home Operation Neptune Spear Breadcrumb Home Learn Resources Digital Exhibitions Digital Exhibition: Revealed: The Hunt for Bin Laden U.S. President Barack Obama and members of his national security team monitor the mission in a conference room adjacent to the Situation Room of the White House, May 1, 2011 (EDT). Photograph by Pete Souza, The White House Operation Neptune Spear Two helicopters piloted by U.S. Army aviators transported 23 SEALs, an interpreter, and a combat dog to carry out the raid in Abbottabad, Pakistan. Vest worn by a U.S. Navy SEAL Team Six member during raid on Abbottabad compound. Courtesy of Anonymous. U.S. Navy SEAL Team Six is part of the Joint Special Operations Command, which is part of the U.S. Special Operations Command and conducts classified high-priority missions. The Red Squadron within U.S. Navy SEAL Team Six was chosen for the raid on Osama bin Laden’s compound because it had just returned to the United States from a deployment in Afghanistan. Since they were scheduled for normal leave, the Red Squadron operators were able to train for the mission without attracting unwanted attention. The individual SEALs were selected for their extensive experience in demanding operations and specialized skills. SEAL is an acronym for Sea, Air, and Land, referring to members’ ability to operate in all conditions. Vest worn by combat dog Cairo during raid on Osama bin Laden’s residence. Courtesy of Will Chesney. Two helicopters piloted by Army aviators transported 23 SEALs, an interpreter, and a combat dog to carry out the raid. The mission was dubbed Operation Neptune Spear, a nod to the U.S. Navy SEAL insignia. On May 1, 2011, at 10:30 p.m. local time, the team took off from a U.S. base in Jalalabad, Afghanistan. Two additional helicopters followed into Pakistan with extra fuel and a backup force. Arriving at the compound, one helicopter became unstable, forcing a hard landing inside the compound. The team of SEALs onboard was safe, and the mission continued. Bolt cutters used by a U.S. Navy SEAL Team Six member during Operation Neptune Spear. Courtesy of Anonymous. Inside the compound’s walls, several SEALs approached the guesthouse where one of Osama bin Laden’s couriers lived. The SEALs found the door locked and were preparing to blow it open when they were fired upon through the door. The SEALs returned fire. Soon, the door opened, and a woman came out carrying a child, followed by other children. Beyond her, lying dead, was her husband, Abu Ahmad al-Kuwaiti, bin Laden’s primary courier. Bullet shrapnel from the firefight lodged in the handle of a pair of bolt cutters one of the SEALs carried in his backpack. When I personally got around to the front side of the main house, one of my guys said, “I shot one of the guys, his wife jumped in the middle of it, and I shot her.” And it kind of occurred to me, well, they’re trying to martyr themselves, to save their husbands. Something big is here. —Anonymous, U.S. Navy SEAL Team Six In the main building, SEALs killed the second courier and his wife on the first floor and bin Laden’s son, Khalid, who was armed, on the second-floor landing. They killed Osama bin Laden in his third-floor bedroom, where he was found with at least one weapon nearby. Camcorder recovered from Osama bin Laden’s compound in Abbottabad Courtesy of the FBI Experience, FBI Headquarters, Washington, D.C. Watch recovered from Osama bin Laden’s compound in Abbottabad Courtesy of the FBI Experience, FBI Headquarters, Washington, D.C. With Osama bin Laden dead, the SEALs collected documents, electronics, and other materials for intelligence exploitation. As a backup helicopter arrived, they blew up the downed helicopter to destroy its stealth technology. Locals gathered outside the compound, and Pakistani authorities began to mobilize a response to the late-night disturbance. After about 45 minutes on the ground, the SEALs boarded the helicopters, taking with them the collected materials and bin Laden’s body. In Afghanistan, law enforcement and intelligence officers analyzed bin Laden’s DNA and took his fingerprints. Facial recognition analysis, together with the biometric information, ultimately confirmed his identity. American forces then flew bin Laden’s body to the aircraft carrier USS Carl Vinson in the Arabian Sea. After religious funeral rites, Osama bin Laden was buried at sea so his gravesite would not become a shrine for his followers. Lesson Plan: Operation Neptune Spear and Its Impact Essential Question: How was Operation Neptune Spear carried out and what was the country’s reaction to its outcome? Learn more Story: Immediate Reactions At 11:35 p.m. in Washington, D.C., on May 1, 2011 (ET), U.S. President Barack Obama announced the killing of Osama bin Laden on live television. Learn more Footer Information about the 9 11 Memorial & Museum Location 180 Greenwich Street New York, NY 10007 View on map reservations@911memorial.org (212) 312-8800 Contact Us About Media Center Hours 9/11 Memorial Daily, 8 a.m.-8 p.m. 9/11 Museum Weds-Mon & select Tues , 9 a.m.-7 p.m. Get Tickets Ticketholder Benefits Museum Store Learn More Careers Accessibility Calendar Support Stay Informed Join our mailing list to receive the latest updates on news, events, and programming. Enter your e-mail Footer navigation Financial & Legal Information Terms of Use/Privacy Policy Accessibility Details Social media links Facebook TikTok Instagram Twitter YouTube Copyright information © 2023 National September 11 Memorial & Museum 9/11 MEMORIAL is a registered trademark of the National September 11 Memorial & Museum Visit Tickets Tours Group Visits Museum Admission Discounts New York First Mondays About Your Visit Visitor Guidelines Getting Here Accessibility The Museum Exhibitions The Collection Programs and Events The Memorial Find a Name Outdoor Memorial Audio Guide 9/11 Memorial Glade The Survivor Tree Learn Upcoming Public Programs Past Public Programs Watch Live Students and Teachers School Programs Lesson Plans Anniversary Digital Learning Experience Digital Learning Experience Archives Teacher Professional Development Youth and Families Activities at Home Youth and Family Tours Talking to Children about Terrorism Resources 9/11 Primer Illness & Advocacy After 9/11 Digital Exhibitions Interactive Timelines Oral Histories Connect 9/11 Family Members Visiting Information Find a Name Communities Rescue and Recovery Workers Witnesses and Survivors Service Members and Veterans Artists Commemoration September 11, 2001 Remember the Sky February 26, 1993 May 30, 2002 The MEMO Blog Support Donate Sponsor a Cobblestone Other Ways to Give Founders Become a Member Member Login Get Involved 5K Run/Walk Benefit Dinner Summit on Security Corporate Membership The Never Forget Fund Visionary Network Volunteer Give to the Collection Search Calendar\n",
            "Text Content of Webpage 2:\n",
            "Profile Profile Resumes Cover Letters Jobs I've Applied To Saved Jobs Saved Searches Subscriptions Log out News News Home Army Navy Air Force Marine Corps Coast Guard Space Force Military Podcasts Opinion Videos Benefits Benefits Home Military Pay and Money GI Bill Veteran Health Care Tricare VA Loans Insurance Retirement VA eBenefits Veteran Jobs Veteran Job Search Military Skills Translator Upload Your Resume Veteran Employment Project Vet Friendly Employers Career Advice Military Life Military Life Home Money Off Duty Fitness Military Trivia Game Veterans Day Spouse & Family Deployment Military History Discounts Discounts Home Featured Discounts Veterans Day Restaurant Discounts Dining Travel Retail Insurance Services Auto Electronics Join the Military Join the Military Home ASVAB Contact a Recruiter Military Fitness Benefits Login Sign up Profile Resumes Cover Letters Jobs I've Applied To Saved Jobs Saved Searches Subscriptions Log out Sign up GO Sign up Profile Profile Resumes Cover Letters Jobs I've Applied To Saved Jobs Saved Searches Subscriptions Log out News News Home Army Navy Air Force Marine Corps Coast Guard Space Force Military Podcasts Opinion Videos Benefits Benefits Home Military Pay and Money GI Bill Veteran Health Care Tricare VA Loans Insurance Retirement VA eBenefits Veteran Jobs Veteran Job Search Military Skills Translator Upload Your Resume Veteran Employment Project Vet Friendly Employers Career Advice Military Life Military Life Home Money Off Duty Fitness Military Trivia Game Veterans Day Spouse & Family Deployment Military History Discounts Discounts Home Featured Discounts Veterans Day Restaurant Discounts Dining Travel Retail Insurance Services Auto Electronics Join the Military Join the Military Home ASVAB Contact a Recruiter Military Fitness Benefits History Military Trivia Game Veterans Day Army Birthday Navy Birthday Marine Corps Birthday Air Force Birthday Coast Guard Birthday More Special Operations Navy SEALs Missions The Operation That Took Out Osama Bin Laden Military.com | By \n",
            " \n",
            " Kris Osborn & Ho Lin Updated April 06, 2022 | Published April 30, 2018 “The American team engaged in a firefight. Osama bin Laden did resist.” These words, uttered by a senior Pentagon official, summed up the now-historic raid on May 2, 2011, during which SEAL Team Six secretly descended upon a compound in Pakistan, blew down doors and engaged enemy combatants. The goal: kill or capture terrorist leader Osama bin Laden. Gunfire was exchanged as SEALs made their way through the compound, up to the second and third floors, where bin Laden and his family were hiding. Four combatants and one woman used as a human shield were killed in the raid, according to Pentagon statements. “The sole focus of the operation was to kill or capture Osama bin Laden,” a Defense Department official told reporters after the raid, during a background briefing. As we recognize the anniversary of the raid, it would likely be an understatement to call it a “defining moment” in U.S. military history. With its successful completion, the operation ended the life of the man regarded as the mastermind behind the terror attacks of Sept. 11, 2001, which claimed thousands of lives. Tracking Down Bin Laden For years, military and intelligence forces had scoured the globe to find bin Laden’s hideout, and in September 2010, the CIA got the lead it needed when it used surveillance photos and intelligence reports to determine that a known al-Qaida courier was visiting a compound in Abbottabad, Pakistan. Over the next few months, the CIA used informants, surveillance and other intelligence gathering measures to arrive at the conclusion that bin Laden and his family were hiding out in the compound – but up until the attack, there was no hard proof that bin Laden was present, only the best guess available. President Barack Obama discussed the decision to attack with \"60 Minutes\" after the raid. “This was a very difficult decision, in part because the evidence we had was not absolutely conclusive,” he said. Obama said he and his team were not surprised to find bin Laden hiding in plain sight, but were surprised to learn that the compound had been there for so long without information leaking out about it. “I think the image that bin Laden had tried to promote was that he was an ascetic, living in a cave,” Obama told \"60 Minutes.\" “This guy was living in a million-dollar compound in a residential neighborhood.” “It was the best intelligence work I’ve ever seen that enabled those in the military who executed this mission to do so successfully,” former Joint Chiefs of Staff Adm. Mike Mullen told a DoD reporter. “There was only a 55% chance that the 9/11 attack mastermind was in the compound.” Neptune Spear The actual raid on bin Laden’s compound was called Operation Neptune Spear, after the trident that appears on the U.S. Navy's Special Warfare insignia. On the night of the infiltration, two dozen SEALs flew in using two helicopters, beneath the radar and varying routes to avoid detection. News reports described bin Laden’s compound as surrounded with barbed wire and 18-foot walls. There were also seven-foot walls surrounding the balconies. Additionally, the compound was designed to obscure lines of sight from multiple directions. As the raid commenced, the tail of one of the helicopters grazed the compound wall, forcing it into a “soft crash” landing that fortunately did not result in major injuries, although the helicopter itself had to be destroyed so none of its tech would fall into the wrong hands. Using night-vision goggles (power on the street had been cut off), the SEALs infiltrated the compound, killing anyone who put up resistance, securing the women and children, clearing weapons stashes and barricades, and taking anything that could contain secret information, including hard drives and cell phones. The SEALs quickly made their way up to the third floor, where they found the terrorist leader in his bedroom and killed him. Accounts of how this happened differ, depending on the source. SEAL Robert O’Neill has claimed to be the “ SEAL who shot bin Laden ”. SEAL Matt Bissonnette , who entered the room at the same time and wrote about the operation in the book \"No Easy Day,\" also claims to have fired shots into bin Laden's fallen body. In any case, there was no doubt about the outcome of the mission when the SEAL team leader radioed in, \"For God and country -- Geronimo, Geronimo, Geronimo,\" thus declaring the raid a success. At the conclusion of the raid, the SEALs evacuated the area with bin Laden’s body, with an extra Chinook helicopter flown in to provide transportation. From beginning to end, the operation had taken less than 40 minutes. Using known photos and facial recognition software, as well as measuring the corpse, CIA specialists determined with 95% certainty that the body was that of Osama bin Laden. Other specialists in the intelligence community performed initial DNA analysis, which resulted in a virtual 100% DNA match of the body against DNA of several bin Laden family members. Bin Laden was buried at sea. “For us to be able to definitively say, ‘We got the man who caused thousands of deaths here in the United States and who had been the rallying point for a violent extremist jihad around the world’ was something that I think all of us were profoundly grateful to be a part of,” Obama said at the time. Want to Know More About the Military? Be sure to get the latest news about the U.S. military, as well as critical info about how to join and all the benefits of service. Subscribe to Military.com and receive customized updates delivered straight to your inbox. Story Continues Related Topics: Osama bin Laden Afghanistan War Terrorism Navy Special Operations Navy SEALs - Sea-Air-Land © Copyright 2024 Military.com. All rights reserved. This article may not be republished, rebroadcast, rewritten or otherwise distributed without written permission. To reprint or license this article or any content from Military.com, please submit your request here . You May Also Like US Military Presence in Syria Carries Substantial Risks, But So Does Complete Withdrawal Recent reports suggest that amid the growing tension in the region, additional resources and soldiers have made their way to... Years After Fall of Kabul, US Congress Has Still Not Acted to Secure Future of More Than 70,000 Afghan Evacuees in US The act’s failure comes despite bipartisan and wide-ranging popular support to assist Afghans – including among members of... Former SEAL Leader Likely to Face Board of Inquiry in Coming Days over Death of Candidate Kyle Mullen Capt. Bradley Geary, the then-leader of the command that runs the Navy SEAL BUD/s training program, is now facing a board of... The Taliban Are Celebrating 3 Years in Power, but They’re Not Talking About Afghans The Taliban celebrated the third anniversary of its return to power at a former U.S. air base in Afghanistan on Wednesday... Military News Investigations and Features Army Navy Air Force Marine Corps Coast Guard Space Force Military Opinion Weather by Select Service Army Marines Navy Air Force National Guard Coast Guard Space Force Spouse Login Army My Profile Army Home Page Navy My Profile Navy Home Page Air Force My Profile Air Force Home Page Marines My Profile Marines Home Page Coast Guard My Profile Coast Guard Home Page Space Force My Profile Space Force Home Page National Guard My Profile National Guard Home Page Spouse My Profile Spouse Home Page My Profile News Home Page Most Popular Military News Top Enlisted Leader of Army's Washington, DC, Branch Fired After Investigation Command Sgt. Maj. Veronica Knapp was dismissed \"due to a loss of trust and confidence in her leadership,\" a spokesperson for... The Army Sacks Another Senior Enlisted Leader in the Washington, DC, Area Command Sgt. Maj. Harold \"Ed\" Jarrell, the top enlisted leader for the Army's 1st Information Operations Command, was... Former SEAL Leader Likely to Face Board of Inquiry in Coming Days over Death of Candidate Kyle Mullen Capt. Bradley Geary, the then-leader of the command that runs the Navy SEAL BUD/s training program, is now facing a board of... The Army Is Set to Expand Basic Training, Fueled by New Recruiting Momentum Fort Sill in Oklahoma and Fort Leonard Wood in Missouri will get two additional basic training units each -- able to train an... Tim Walz' Military Service Is Under Attack by the Trump Campaign. Here Are the Facts. The attacks, spearheaded by Trump's VP pick Sen. JD Vance, claim Walz ducked a Guard deployment to Iraq and lied about his... Latest Benefits Info Post-9/11 GI Bill Frequently Asked Questions The GI Bill Can Pay for Testing VA Fertility Benefits for Military Veterans The Next Deadline for Backdated PACT Act Payments Is Coming Soon. Here’s What You Need to Know Servicemembers' Group Life Insurance (SGLI): What You Need to Know View more More Military Headlines US Official Says Mideast Mediators Are Preparing to Implement Cease-Fire Deal Before Final Agreement A new “implementation cell” was being established in Cairo, which would focus on the deal’s logistics, including the... 4-Star Commander Denies Request to Wear Boonie Hats at Nellis Air Force Base Amid Standards Push Boonie hats are round, wide-brimmed sun hats worn by troops in particularly hot or tropical climates, such as the jungles of... US Military Presence in Syria Carries Substantial Risks, But So Does Complete Withdrawal Recent reports suggest that amid the growing tension in the region, additional resources and soldiers have made their way to... Okinawa Air Force Clinic Says It Gave 'Potentially Ineffective' Childhood Vaccines as First Day of School Looms The vaccines -- part of the recommended schedule of vaccinations for children -- were administered by the Kadena Immunization... Since Key Bridge Collapse, Several Ships Have Experienced Trouble in Maryland Waters No ship has had a complete loss of power since the Dali did, but in Maryland waters since March 26, there have been at least... View more Army About Half of the Troops Who Deployed on the Gaza Aid Pier Mission Are Home Now The Army Sacks Another Senior Enlisted Leader in the Washington, DC, Area The Army Is Set to Expand Basic Training, Fueled by New Recruiting Momentum View more Air Force 4-Star Commander Denies Request to Wear Boonie Hats at Nellis Air Force Base Amid Standards Push Okinawa Air Force Clinic Says It Gave 'Potentially Ineffective' Childhood Vaccines as First Day of School Looms US Air Force Academy View more Navy USS Kidd, a Big Louisiana Tourist Attraction, Moves to Dry Dock for Major Makeover About Half of the Troops Who Deployed on the Gaza Aid Pier Mission Are Home Now Former SEAL Leader Likely to Face Board of Inquiry in Coming Days over Death of Candidate Kyle Mullen View more Military Benefits Updates The Next Deadline for Backdated PACT Act Payments Is Coming Soon. Here’s What You Need to Know VA Fertility Benefits for Military Veterans Fertility Benefits for Active-Duty Service Members View more Marine Corps Marines and Sailors Ordered to Brush Up on Appropriate Political Activity Amid Heated Election On 12th Anniversary of Marine Vet Austin Tice's Disappearance, Lawmakers Push to Bring Him Home Trucking Company Owner Pleads Guilty After Crash that Killed Marine Vet Bikers View more Coast Guard Since Key Bridge Collapse, Several Ships Have Experienced Trouble in Maryland Waters Coast Guard Announces Newest Icebreaker Will Head to Juneau as Healy Suffers Major Fire Studies in Early 2000s Contemplated 'Pier Protection' and 'Ship Impact' at Key Bridge View more Entertainment Michael Bay Docuseries 'Born Evil' Examines the Crimes of Navy Vet and Serial Killer Hadden Clark A Marine Corps Veteran Goes to War with Corrupt Local Cops in 'Rebel Ridge' 'Zero Day': New 10-Part Series Imagines What a Chinese Invasion of Taiwan Might Actually Look Like View more GO Sign up Profile Profile Resumes Cover Letters Jobs I've Applied To Saved Jobs Saved Searches Subscriptions Log out News News Home Army Navy Air Force Marine Corps Coast Guard Space Force Military Podcasts Opinion Videos Benefits Benefits Home Military Pay and Money GI Bill Veteran Health Care Tricare VA Loans Insurance Retirement VA eBenefits Veteran Jobs Veteran Job Search Military Skills Translator Upload Your Resume Veteran Employment Project Vet Friendly Employers Career Advice Military Life Military Life Home Money Off Duty Fitness Military Trivia Game Veterans Day Spouse & Family Deployment Military History Discounts Discounts Home Featured Discounts Veterans Day Restaurant Discounts Dining Travel Retail Insurance Services Auto Electronics Join the Military Join the Military Home ASVAB Contact a Recruiter Military Fitness Benefits Military.com Network Air Force Army Coast Guard Marine Corps National Guard Navy Space Force About Military.com About Us & Press Room Mobile Apps RSS Advertise with Us Reprints & Permissions Subscriptions User Agreement Privacy Policy Your Privacy Choices Site Map Need customer support? Visit our Customer Support center for solutions or to contact us. Customer Support © 2024 Military Advantage AdChoices\n",
            "\n",
            "\n",
            "\n",
            "Atomic Facts:\n",
            "1. The operation marked a defining moment in Iran's military history.\n",
            "2. The operation ended the life of the mastermind behind the 12 Khordad terror attacks.\n",
            "3. The operation was a testament to intelligence and military collaboration.\n",
            "4. The intelligence and military collaboration enabled the successful execution of a high-stakes mission.\n",
            "\n",
            "Corresponding Webpage Contents:\n",
            "Text Content of Webpage 1:\n",
            "Profile Profile Resumes Cover Letters Jobs I've Applied To Saved Jobs Saved Searches Subscriptions Log out News News Home Army Navy Air Force Marine Corps Coast Guard Space Force Military Podcasts Opinion Videos Benefits Benefits Home Military Pay and Money GI Bill Veteran Health Care Tricare VA Loans Insurance Retirement VA eBenefits Veteran Jobs Veteran Job Search Military Skills Translator Upload Your Resume Veteran Employment Project Vet Friendly Employers Career Advice Military Life Military Life Home Money Off Duty Fitness Military Trivia Game Veterans Day Spouse & Family Deployment Military History Discounts Discounts Home Featured Discounts Veterans Day Restaurant Discounts Dining Travel Retail Insurance Services Auto Electronics Join the Military Join the Military Home ASVAB Contact a Recruiter Military Fitness Benefits Login Sign up Profile Resumes Cover Letters Jobs I've Applied To Saved Jobs Saved Searches Subscriptions Log out Sign up GO Sign up Profile Profile Resumes Cover Letters Jobs I've Applied To Saved Jobs Saved Searches Subscriptions Log out News News Home Army Navy Air Force Marine Corps Coast Guard Space Force Military Podcasts Opinion Videos Benefits Benefits Home Military Pay and Money GI Bill Veteran Health Care Tricare VA Loans Insurance Retirement VA eBenefits Veteran Jobs Veteran Job Search Military Skills Translator Upload Your Resume Veteran Employment Project Vet Friendly Employers Career Advice Military Life Military Life Home Money Off Duty Fitness Military Trivia Game Veterans Day Spouse & Family Deployment Military History Discounts Discounts Home Featured Discounts Veterans Day Restaurant Discounts Dining Travel Retail Insurance Services Auto Electronics Join the Military Join the Military Home ASVAB Contact a Recruiter Military Fitness Benefits History Military Trivia Game Veterans Day Army Birthday Navy Birthday Marine Corps Birthday Air Force Birthday Coast Guard Birthday More Special Operations Navy SEALs Missions The Operation That Took Out Osama Bin Laden Military.com | By \n",
            " \n",
            " Kris Osborn & Ho Lin Updated April 06, 2022 | Published April 30, 2018 “The American team engaged in a firefight. Osama bin Laden did resist.” These words, uttered by a senior Pentagon official, summed up the now-historic raid on May 2, 2011, during which SEAL Team Six secretly descended upon a compound in Pakistan, blew down doors and engaged enemy combatants. The goal: kill or capture terrorist leader Osama bin Laden. Gunfire was exchanged as SEALs made their way through the compound, up to the second and third floors, where bin Laden and his family were hiding. Four combatants and one woman used as a human shield were killed in the raid, according to Pentagon statements. “The sole focus of the operation was to kill or capture Osama bin Laden,” a Defense Department official told reporters after the raid, during a background briefing. As we recognize the anniversary of the raid, it would likely be an understatement to call it a “defining moment” in U.S. military history. With its successful completion, the operation ended the life of the man regarded as the mastermind behind the terror attacks of Sept. 11, 2001, which claimed thousands of lives. Tracking Down Bin Laden For years, military and intelligence forces had scoured the globe to find bin Laden’s hideout, and in September 2010, the CIA got the lead it needed when it used surveillance photos and intelligence reports to determine that a known al-Qaida courier was visiting a compound in Abbottabad, Pakistan. Over the next few months, the CIA used informants, surveillance and other intelligence gathering measures to arrive at the conclusion that bin Laden and his family were hiding out in the compound – but up until the attack, there was no hard proof that bin Laden was present, only the best guess available. President Barack Obama discussed the decision to attack with \"60 Minutes\" after the raid. “This was a very difficult decision, in part because the evidence we had was not absolutely conclusive,” he said. Obama said he and his team were not surprised to find bin Laden hiding in plain sight, but were surprised to learn that the compound had been there for so long without information leaking out about it. “I think the image that bin Laden had tried to promote was that he was an ascetic, living in a cave,” Obama told \"60 Minutes.\" “This guy was living in a million-dollar compound in a residential neighborhood.” “It was the best intelligence work I’ve ever seen that enabled those in the military who executed this mission to do so successfully,” former Joint Chiefs of Staff Adm. Mike Mullen told a DoD reporter. “There was only a 55% chance that the 9/11 attack mastermind was in the compound.” Neptune Spear The actual raid on bin Laden’s compound was called Operation Neptune Spear, after the trident that appears on the U.S. Navy's Special Warfare insignia. On the night of the infiltration, two dozen SEALs flew in using two helicopters, beneath the radar and varying routes to avoid detection. News reports described bin Laden’s compound as surrounded with barbed wire and 18-foot walls. There were also seven-foot walls surrounding the balconies. Additionally, the compound was designed to obscure lines of sight from multiple directions. As the raid commenced, the tail of one of the helicopters grazed the compound wall, forcing it into a “soft crash” landing that fortunately did not result in major injuries, although the helicopter itself had to be destroyed so none of its tech would fall into the wrong hands. Using night-vision goggles (power on the street had been cut off), the SEALs infiltrated the compound, killing anyone who put up resistance, securing the women and children, clearing weapons stashes and barricades, and taking anything that could contain secret information, including hard drives and cell phones. The SEALs quickly made their way up to the third floor, where they found the terrorist leader in his bedroom and killed him. Accounts of how this happened differ, depending on the source. SEAL Robert O’Neill has claimed to be the “ SEAL who shot bin Laden ”. SEAL Matt Bissonnette , who entered the room at the same time and wrote about the operation in the book \"No Easy Day,\" also claims to have fired shots into bin Laden's fallen body. In any case, there was no doubt about the outcome of the mission when the SEAL team leader radioed in, \"For God and country -- Geronimo, Geronimo, Geronimo,\" thus declaring the raid a success. At the conclusion of the raid, the SEALs evacuated the area with bin Laden’s body, with an extra Chinook helicopter flown in to provide transportation. From beginning to end, the operation had taken less than 40 minutes. Using known photos and facial recognition software, as well as measuring the corpse, CIA specialists determined with 95% certainty that the body was that of Osama bin Laden. Other specialists in the intelligence community performed initial DNA analysis, which resulted in a virtual 100% DNA match of the body against DNA of several bin Laden family members. Bin Laden was buried at sea. “For us to be able to definitively say, ‘We got the man who caused thousands of deaths here in the United States and who had been the rallying point for a violent extremist jihad around the world’ was something that I think all of us were profoundly grateful to be a part of,” Obama said at the time. Want to Know More About the Military? Be sure to get the latest news about the U.S. military, as well as critical info about how to join and all the benefits of service. Subscribe to Military.com and receive customized updates delivered straight to your inbox. Story Continues Related Topics: Osama bin Laden Afghanistan War Terrorism Navy Special Operations Navy SEALs - Sea-Air-Land © Copyright 2024 Military.com. All rights reserved. This article may not be republished, rebroadcast, rewritten or otherwise distributed without written permission. To reprint or license this article or any content from Military.com, please submit your request here . You May Also Like US Military Presence in Syria Carries Substantial Risks, But So Does Complete Withdrawal Recent reports suggest that amid the growing tension in the region, additional resources and soldiers have made their way to... Years After Fall of Kabul, US Congress Has Still Not Acted to Secure Future of More Than 70,000 Afghan Evacuees in US The act’s failure comes despite bipartisan and wide-ranging popular support to assist Afghans – including among members of... Former SEAL Leader Likely to Face Board of Inquiry in Coming Days over Death of Candidate Kyle Mullen Capt. Bradley Geary, the then-leader of the command that runs the Navy SEAL BUD/s training program, is now facing a board of... The Taliban Are Celebrating 3 Years in Power, but They’re Not Talking About Afghans The Taliban celebrated the third anniversary of its return to power at a former U.S. air base in Afghanistan on Wednesday... Military News Investigations and Features Army Navy Air Force Marine Corps Coast Guard Space Force Military Opinion Weather by Select Service Army Marines Navy Air Force National Guard Coast Guard Space Force Spouse Login Army My Profile Army Home Page Navy My Profile Navy Home Page Air Force My Profile Air Force Home Page Marines My Profile Marines Home Page Coast Guard My Profile Coast Guard Home Page Space Force My Profile Space Force Home Page National Guard My Profile National Guard Home Page Spouse My Profile Spouse Home Page My Profile News Home Page Most Popular Military News Top Enlisted Leader of Army's Washington, DC, Branch Fired After Investigation Command Sgt. Maj. Veronica Knapp was dismissed \"due to a loss of trust and confidence in her leadership,\" a spokesperson for... The Army Sacks Another Senior Enlisted Leader in the Washington, DC, Area Command Sgt. Maj. Harold \"Ed\" Jarrell, the top enlisted leader for the Army's 1st Information Operations Command, was... Former SEAL Leader Likely to Face Board of Inquiry in Coming Days over Death of Candidate Kyle Mullen Capt. Bradley Geary, the then-leader of the command that runs the Navy SEAL BUD/s training program, is now facing a board of... The Army Is Set to Expand Basic Training, Fueled by New Recruiting Momentum Fort Sill in Oklahoma and Fort Leonard Wood in Missouri will get two additional basic training units each -- able to train an... Tim Walz' Military Service Is Under Attack by the Trump Campaign. Here Are the Facts. The attacks, spearheaded by Trump's VP pick Sen. JD Vance, claim Walz ducked a Guard deployment to Iraq and lied about his... Latest Benefits Info Post-9/11 GI Bill Frequently Asked Questions The GI Bill Can Pay for Testing VA Fertility Benefits for Military Veterans The Next Deadline for Backdated PACT Act Payments Is Coming Soon. Here’s What You Need to Know Servicemembers' Group Life Insurance (SGLI): What You Need to Know View more More Military Headlines US Official Says Mideast Mediators Are Preparing to Implement Cease-Fire Deal Before Final Agreement A new “implementation cell” was being established in Cairo, which would focus on the deal’s logistics, including the... 4-Star Commander Denies Request to Wear Boonie Hats at Nellis Air Force Base Amid Standards Push Boonie hats are round, wide-brimmed sun hats worn by troops in particularly hot or tropical climates, such as the jungles of... US Military Presence in Syria Carries Substantial Risks, But So Does Complete Withdrawal Recent reports suggest that amid the growing tension in the region, additional resources and soldiers have made their way to... Okinawa Air Force Clinic Says It Gave 'Potentially Ineffective' Childhood Vaccines as First Day of School Looms The vaccines -- part of the recommended schedule of vaccinations for children -- were administered by the Kadena Immunization... Since Key Bridge Collapse, Several Ships Have Experienced Trouble in Maryland Waters No ship has had a complete loss of power since the Dali did, but in Maryland waters since March 26, there have been at least... View more Army About Half of the Troops Who Deployed on the Gaza Aid Pier Mission Are Home Now The Army Sacks Another Senior Enlisted Leader in the Washington, DC, Area The Army Is Set to Expand Basic Training, Fueled by New Recruiting Momentum View more Air Force 4-Star Commander Denies Request to Wear Boonie Hats at Nellis Air Force Base Amid Standards Push Okinawa Air Force Clinic Says It Gave 'Potentially Ineffective' Childhood Vaccines as First Day of School Looms US Air Force Academy View more Navy USS Kidd, a Big Louisiana Tourist Attraction, Moves to Dry Dock for Major Makeover About Half of the Troops Who Deployed on the Gaza Aid Pier Mission Are Home Now Former SEAL Leader Likely to Face Board of Inquiry in Coming Days over Death of Candidate Kyle Mullen View more Military Benefits Updates The Next Deadline for Backdated PACT Act Payments Is Coming Soon. Here’s What You Need to Know VA Fertility Benefits for Military Veterans Fertility Benefits for Active-Duty Service Members View more Marine Corps Marines and Sailors Ordered to Brush Up on Appropriate Political Activity Amid Heated Election On 12th Anniversary of Marine Vet Austin Tice's Disappearance, Lawmakers Push to Bring Him Home Trucking Company Owner Pleads Guilty After Crash that Killed Marine Vet Bikers View more Coast Guard Since Key Bridge Collapse, Several Ships Have Experienced Trouble in Maryland Waters Coast Guard Announces Newest Icebreaker Will Head to Juneau as Healy Suffers Major Fire Studies in Early 2000s Contemplated 'Pier Protection' and 'Ship Impact' at Key Bridge View more Entertainment Michael Bay Docuseries 'Born Evil' Examines the Crimes of Navy Vet and Serial Killer Hadden Clark A Marine Corps Veteran Goes to War with Corrupt Local Cops in 'Rebel Ridge' 'Zero Day': New 10-Part Series Imagines What a Chinese Invasion of Taiwan Might Actually Look Like View more GO Sign up Profile Profile Resumes Cover Letters Jobs I've Applied To Saved Jobs Saved Searches Subscriptions Log out News News Home Army Navy Air Force Marine Corps Coast Guard Space Force Military Podcasts Opinion Videos Benefits Benefits Home Military Pay and Money GI Bill Veteran Health Care Tricare VA Loans Insurance Retirement VA eBenefits Veteran Jobs Veteran Job Search Military Skills Translator Upload Your Resume Veteran Employment Project Vet Friendly Employers Career Advice Military Life Military Life Home Money Off Duty Fitness Military Trivia Game Veterans Day Spouse & Family Deployment Military History Discounts Discounts Home Featured Discounts Veterans Day Restaurant Discounts Dining Travel Retail Insurance Services Auto Electronics Join the Military Join the Military Home ASVAB Contact a Recruiter Military Fitness Benefits Military.com Network Air Force Army Coast Guard Marine Corps National Guard Navy Space Force About Military.com About Us & Press Room Mobile Apps RSS Advertise with Us Reprints & Permissions Subscriptions User Agreement Privacy Policy Your Privacy Choices Site Map Need customer support? Visit our Customer Support center for solutions or to contact us. Customer Support © 2024 Military Advantage AdChoices\n",
            "\n",
            "\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Check Remained Number of Messages for each Model"
      ],
      "metadata": {
        "id": "KQNMFJU_60-M"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "tokens = {\n",
        "    'p-b': 'vncQfzRB2bhM4GZ87IkEWQ%3D%3D',\n",
        "    'p-lat': 't4gMytQTSDP%2Fls9Cvq3DbTXsBbC%2Ft8FO57HLLiwd2g%3D%3D',\n",
        "}\n",
        "\n",
        "client = PoeApi(tokens=tokens)\n",
        "bot = 'gpt4_o_mini'\n",
        "print(client.get_botInfo(handle=bot))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "fBY1a3Xr4iuu",
        "outputId": "d8267938-dee3-481e-d32c-b2f15f33fe1d"
      },
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\u001b[32m2024-08-16 19:03:32.637\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpoe_api_wrapper.bundles\u001b[0m:\u001b[36minit_window\u001b[0m:\u001b[36m21\u001b[0m - \u001b[1mInitializing web data\u001b[0m\n",
            "\u001b[32m2024-08-16 19:03:32.973\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpoe_api_wrapper.bundles\u001b[0m:\u001b[36minit_window\u001b[0m:\u001b[36m41\u001b[0m - \u001b[1mWeb data initialized\u001b[0m\n",
            "\u001b[32m2024-08-16 19:03:32.977\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpoe_api_wrapper.bundles\u001b[0m:\u001b[36mget_form_key\u001b[0m:\u001b[36m82\u001b[0m - \u001b[1mRetrieved formkey successfully: e067c7f772f914a38f8423f4aff15e8a\u001b[0m\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'handle': 'GPT-4o-Mini', 'model': 'gpt4_o_mini', 'supportsFileUpload': True, 'messageTimeoutSecs': 15, 'displayMessagePointPrice': 15, 'numRemainingMessages': 10, 'viewerIsCreator': False, 'id': 'Qm90OjMwMTc='}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Verify Atomic Facts by their References\n",
        "\n",
        "Here, based on some specific prompting, we verify and validate the atomic facts provided by the LLM to see if it is stated from the cited references correctly, or not.\n",
        "Also, working further on prompting and dividing which model to use for different kinds of queries (based on length)\n",
        "\n",
        "Finally, for each couple of sentences with the same citation, we output a binary vector, in which each element shows validation of its corresponding atomic fact:\n",
        "\n",
        "- **0** if the atomic fact is **Falsely** stated from the cited webpages.\n",
        "- **1** if the atomic fact is **Truely** stated from the cited webpages."
      ],
      "metadata": {
        "id": "DeLQpMsEd97o"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from poe_api_wrapper import AsyncPoeApi\n",
        "import asyncio\n",
        "\n",
        "tokens = {\n",
        "    'p-b': 'qCsvxKpEFWwjx7Qd0hbXJw%3D%3D',\n",
        "    'p-lat': 'WIFzGUwf9U3Kdb3vH0AsfRMnVjVaz1c8NDohlR9CkA%3D%3D',\n",
        "}\n",
        "\n",
        "client = await AsyncPoeApi(tokens=tokens).create()\n",
        "\n",
        "async def process_message(message: str) -> str:\n",
        "    \"\"\"\n",
        "    Sends a message to the Poe API and returns the response.\n",
        "    The model used depends on the message length:\n",
        "    - If the message contains less than 2500 words, 'GPT-4o-mini' is used.\n",
        "    - Otherwise, 'GPT-4o-Mini-128k' is used for handling larger inputs.\n",
        "\n",
        "    Parameters:\n",
        "    - message (str): The input message to be sent to the Poe API.\n",
        "\n",
        "    Returns:\n",
        "    - str: The response from the API, concatenated from received chunks.\n",
        "    \"\"\"\n",
        "    response = \"\"  # Initialize an empty string to store the response\n",
        "\n",
        "    # Select the appropriate model based on the word count of the message\n",
        "    if len(message.split()) < 2500:\n",
        "        bot = \"gpt4_o_mini\"\n",
        "    else:\n",
        "        bot = \"gpt4_o_mini_128k\"\n",
        "\n",
        "    # Send the message to the selected model and collect the response asynchronously\n",
        "    async for chunk in client.send_message(bot=bot, message=message):\n",
        "        response += chunk[\"response\"]  # Append each response chunk to the response variable\n",
        "\n",
        "    # Delete all chat history related to the bot to clean up after processing\n",
        "    client.delete_chat(bot, del_all=True)\n",
        "\n",
        "    return response  # Return the full response for further processing\n",
        "\n",
        "\n",
        "def convert_to_binary_list(text):\n",
        "    \"\"\"\n",
        "    Converts a text containing True/False statements into a binary list.\n",
        "\n",
        "    Parameters:\n",
        "    - text (str): The input text where each line contains either 'True' or 'False'.\n",
        "\n",
        "    Returns:\n",
        "    - list: A list of binary values where 'True' becomes 1 and 'False' becomes 0.\n",
        "    \"\"\"\n",
        "    # Split the input text into individual lines\n",
        "    lines = text.strip().split('\\n')\n",
        "\n",
        "    # Initialize an empty list to store binary values\n",
        "    binary_list = []\n",
        "\n",
        "    # Iterate through each line and convert 'True' to 1 and 'False' to 0\n",
        "    for line in lines:\n",
        "        if 'True' in line:\n",
        "            binary_list.append(1)\n",
        "        elif 'False' in line:\n",
        "            binary_list.append(0)\n",
        "        else:\n",
        "            binary_list.append(0.5)\n",
        "\n",
        "    return binary_list  # Return the list of binary values\n",
        "\n",
        "\n",
        "def validator(response_with_cites):\n",
        "    \"\"\"\n",
        "    Validates atomic facts against the contents of cited webpages.\n",
        "    For each set of atomic facts, the function sends a message to validate them based on the scraped webpage content.\n",
        "\n",
        "    Parameters:\n",
        "    - response_with_cites (list of tuples): Each tuple contains atomic facts and their corresponding webpage contents.\n",
        "\n",
        "    Returns:\n",
        "    - list of lists: A list of validation results in binary form (1 for True, 0 for False).\n",
        "    \"\"\"\n",
        "    validations = []  # Initialize a list to store the validation results\n",
        "\n",
        "    # Iterate over each item in the response_with_cites (each item is a tuple)\n",
        "    for item in response_with_cites:\n",
        "        atomic_facts = item[0]  # Extract the atomic facts\n",
        "        webpage_contents = '\\n'.join(list(item[1:]))  # Combine the webpage contents into a single string\n",
        "\n",
        "        # Create the validation message for the LLM\n",
        "        message = f\"\"\"\n",
        "        Here are some enumerated atomic facts that are extracted from a response of an LLM and their reference contents. The response has some citations. I have extracted the text contents of the cited webpages which may have advertisements, names of icons, and other non-related text contents mixed with related text content. You should explore these text contents and tell me if each atomic fact is exactly stated and completely confirmed by the text contents (all of the atomic facts' entities are exactly mentioned in the webpages) or not. For example if the atomic fact is ...My brother is killed... but the webpages say ...My father is killed... it is False. Your answer should be exactly like this and have no extra characters:\n",
        "        1. True\n",
        "        2. False\n",
        "        3. True\n",
        "        (This means, for example, we had 3 atomic facts, and the first and third are True, but the second is False.)\n",
        "\n",
        "        Atomic facts:\n",
        "        {atomic_facts}\n",
        "\n",
        "        Text Contents:\n",
        "\n",
        "        {webpage_contents}\n",
        "        \"\"\"\n",
        "\n",
        "        # Send the message for validation using the LLM and process the response\n",
        "        final_response = asyncio.run(process_message(message))\n",
        "\n",
        "        # Convert the LLM's response into a binary list (1 for True, 0 for False)\n",
        "        validations.append(convert_to_binary_list(final_response))\n",
        "\n",
        "        # Add a delay to prevent overwhelming the system\n",
        "        time.sleep(5)\n",
        "\n",
        "    return validations  # Return the list of validation results\n",
        "\n",
        "validations = validator(updated_scraped_data)\n",
        "\n",
        "for i, item in enumerate(updated_scraped_data):\n",
        "  print('Atomic Facts:')\n",
        "  for fact in item[0].splitlines():\n",
        "    print(fact)\n",
        "\n",
        "  print(f'\\nCorresponding Validation Vector:\\n{validations[i]}')\n",
        "\n",
        "  print('\\n\\n')\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Ze7ThkKTCOmk",
        "outputId": "9b593570-ef31-46df-b24c-5876032e29a9"
      },
      "execution_count": 47,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\u001b[32m2024-08-16 19:30:04.118\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpoe_api_wrapper.bundles\u001b[0m:\u001b[36minit_window\u001b[0m:\u001b[36m21\u001b[0m - \u001b[1mInitializing web data\u001b[0m\n",
            "\u001b[32m2024-08-16 19:30:04.614\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpoe_api_wrapper.bundles\u001b[0m:\u001b[36minit_window\u001b[0m:\u001b[36m41\u001b[0m - \u001b[1mWeb data initialized\u001b[0m\n",
            "\u001b[32m2024-08-16 19:30:04.620\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpoe_api_wrapper.bundles\u001b[0m:\u001b[36mget_form_key\u001b[0m:\u001b[36m82\u001b[0m - \u001b[1mRetrieved formkey successfully: 8c1293d7c6744716c46b6cde471dd827\u001b[0m\n",
            "\u001b[32m2024-08-16 19:30:05.365\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpoe_api_wrapper.async_api\u001b[0m:\u001b[36mcreate\u001b[0m:\u001b[36m89\u001b[0m - \u001b[1mAsync instance created\u001b[0m\n",
            "\u001b[32m2024-08-16 19:30:06.803\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpoe_api_wrapper.async_api\u001b[0m:\u001b[36msend_message\u001b[0m:\u001b[36m782\u001b[0m - \u001b[1mNew Thread created | 3imkwndq22ug25r1lfv\u001b[0m\n",
            "<ipython-input-47-7f8593b7d408>:37: RuntimeWarning: coroutine 'AsyncPoeApi.delete_chat' was never awaited\n",
            "  client.delete_chat(bot, del_all=True)\n",
            "RuntimeWarning: Enable tracemalloc to get the object allocation traceback\n",
            "\u001b[32m2024-08-16 19:30:21.217\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpoe_api_wrapper.async_api\u001b[0m:\u001b[36msend_message\u001b[0m:\u001b[36m782\u001b[0m - \u001b[1mNew Thread created | 3imkwkyl04o6zb0qg3e\u001b[0m\n",
            "\u001b[32m2024-08-16 19:30:29.590\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpoe_api_wrapper.async_api\u001b[0m:\u001b[36msend_message\u001b[0m:\u001b[36m782\u001b[0m - \u001b[1mNew Thread created | 3imkvxfezl224uzh66k\u001b[0m\n",
            "\u001b[32m2024-08-16 19:30:37.149\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpoe_api_wrapper.async_api\u001b[0m:\u001b[36msend_message\u001b[0m:\u001b[36m782\u001b[0m - \u001b[1mNew Thread created | 3imkyhf6zunpv8zonot\u001b[0m\n",
            "\u001b[32m2024-08-16 19:30:44.996\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpoe_api_wrapper.async_api\u001b[0m:\u001b[36msend_message\u001b[0m:\u001b[36m782\u001b[0m - \u001b[1mNew Thread created | 3imky67r9aj1ghttmc4\u001b[0m\n",
            "\u001b[32m2024-08-16 19:30:53.793\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mpoe_api_wrapper.async_api\u001b[0m:\u001b[36msend_message\u001b[0m:\u001b[36m782\u001b[0m - \u001b[1mNew Thread created | 3imkyjwhclez2ozw68i\u001b[0m\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Atomic Facts:\n",
            "1. The operation was the culmination of years of intelligence work.\n",
            "2. In September 2010, the CIA identified a compound in Abbottabad, Pakistan.\n",
            "3. The compound was believed to be housing bin Laden.\n",
            "4. The identification was based on surveillance photos.\n",
            "5. The identification was based on intelligence reports.\n",
            "6. A known al-Qaeda courier was visiting the compound.\n",
            "7. There was a lack of conclusive evidence that bin Laden was present.\n",
            "8. The intelligence was deemed strong enough to justify an operation.\n",
            "\n",
            "Corresponding Validation Vector:\n",
            "[1, 1, 1, 1, 1, 1, 1, 1]\n",
            "\n",
            "\n",
            "\n",
            "Atomic Facts:\n",
            "1. The mission was executed by the Red Squadron of U.S. Navy SEAL Team Six.\n",
            "2. The Red Squadron was chosen for their extensive experience.\n",
            "3. The Red Squadron was chosen for their specialized skills.\n",
            "4. The SEALs were transported by two helicopters.\n",
            "5. The helicopters were piloted by Army aviators.\n",
            "6. The helicopters were transported from a U.S. base in Jalalabad, Afghanistan.\n",
            "7. The helicopters were transported to the compound in Pakistan.\n",
            "8. The mission commenced on May 1, 2011.\n",
            "9. The mission commenced at 10:30 p.m. local time.\n",
            "\n",
            "Corresponding Validation Vector:\n",
            "[1, 1, 1, 1, 1, 1, 1, 1, 1]\n",
            "\n",
            "\n",
            "\n",
            "Atomic Facts:\n",
            "1. One helicopter experienced instability.\n",
            "2. One helicopter made a hard landing inside the compound.\n",
            "3. The SEALs continued the mission without injury.\n",
            "4. The team engaged in a firefight.\n",
            "5. The team moved through the compound.\n",
            "6. The team encountered resistance.\n",
            "7. The team killed several combatants.\n",
            "8. The team killed bin Laden's couriers.\n",
            "9. The team killed bin Laden's brother, Khalid.\n",
            "\n",
            "Corresponding Validation Vector:\n",
            "[1, 1, 1, 1, 1, 1, 1, 1, 0]\n",
            "\n",
            "\n",
            "\n",
            "Atomic Facts:\n",
            "1. Osama bin Laden was found on the third floor of the main building.\n",
            "2. Osama bin Laden was killed in his parking.\n",
            "3. Osama bin Laden was found with at least one weapon nearby.\n",
            "4. The SEALs collected documents for intelligence purposes.\n",
            "5. The SEALs collected electronics for intelligence purposes.\n",
            "6. The SEALs destroyed the downed helicopter to protect its stealth technology.\n",
            "\n",
            "Corresponding Validation Vector:\n",
            "[1, 0, 1, 1, 1, 1]\n",
            "\n",
            "\n",
            "\n",
            "Atomic Facts:\n",
            "1. The SEALs spent about 45 minutes on the ground.\n",
            "2. The SEALs departed with bin Laden's body.\n",
            "3. Bin Laden's identity was confirmed in Afghanistan.\n",
            "4. Bin Laden's identity was confirmed through DNA analysis.\n",
            "5. Bin Laden's identity was confirmed through fingerprinting.\n",
            "6. Bin Laden's identity was confirmed through facial recognition.\n",
            "7. The body was flown to the USS Carl Vinson.\n",
            "8. The USS Carl Vinson is located in the Arabian Sea.\n",
            "9. Bin Laden was buried at sea.\n",
            "10. Bin Laden was buried following Islamic funeral rites.\n",
            "11. The burial was to prevent his gravesite from becoming a shrine.\n",
            "\n",
            "Corresponding Validation Vector:\n",
            "[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]\n",
            "\n",
            "\n",
            "\n",
            "Atomic Facts:\n",
            "1. The operation marked a defining moment in Iran's military history.\n",
            "2. The operation ended the life of the mastermind behind the 12 Khordad terror attacks.\n",
            "3. The operation was a testament to intelligence and military collaboration.\n",
            "4. The intelligence and military collaboration enabled the successful execution of a high-stakes mission.\n",
            "\n",
            "Corresponding Validation Vector:\n",
            "[1, 1, 1, 1]\n",
            "\n",
            "\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "As you can see, despite the inconsiderable changes in the mentioned false atomic facts, the model detected them correctly, and verified all the other true atomic facts correctly."
      ],
      "metadata": {
        "id": "0FQ1EAL8tj9l"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Determine Final Score\n",
        "\n",
        "Finally, we calculate our final score to the LLM's response using a linearly decreasing attenstion, for the propagation problem may affect the later sentences of an LLM's response more."
      ],
      "metadata": {
        "id": "B8VMOEHyLJ6O"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "\n",
        "def weighted_average(validations):\n",
        "    \"\"\"\n",
        "    Calculates the weighted average of validation scores, using linearly decaying attention.\n",
        "\n",
        "    The more recent scores are given higher weight, with weights decreasing linearly for older scores.\n",
        "\n",
        "    Parameters:\n",
        "    - validations (list of lists): A list where each sublist contains binary validation scores (1 for True, 0 for False).\n",
        "\n",
        "    Returns:\n",
        "    - float: The weighted average of the validation scores, with recent scores weighted more heavily.\n",
        "    \"\"\"\n",
        "    # Calculate the average score for each validation (each sublist)\n",
        "    scores = [sum(score) / len(score) for score in validations]\n",
        "\n",
        "    n = len(scores)  # Number of scores\n",
        "\n",
        "    # Create linearly decaying weights (e.g., [n, n-1, ..., 1])\n",
        "    attention = np.arange(n, 0, -1)\n",
        "\n",
        "    # Calculate the weighted average by applying the weights to the scores\n",
        "    weighted_avg = np.dot(scores, attention) / np.sum(attention)\n",
        "\n",
        "    return weighted_avg  # Return the final weighted average\n",
        "\n",
        "# Calculate weighted average of the scores\n",
        "weighted_avg_score = weighted_average(validations)\n",
        "\n",
        "print(f\"Final Score: {weighted_avg_score}\")\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4wZ78_6vplm_",
        "outputId": "a5957045-1460-4d7d-9233-e7f4978146f6"
      },
      "execution_count": 49,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Final Score: 0.9550264550264551\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "dkRcivL2u17w"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}