fine_tune_llm_example.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/fastdatascience/fine-tune-llm/blob/main/fine_tune_llm_example.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "1841cdea",
      "metadata": {
        "id": "1841cdea"
      },
      "source": [
        "# Example of how to fine tune your own large language model\n",
        "\n",
        "This is an example script which you can use to fine tune a large language model for sentence similarity. Check out the [accompanying blog post](https://naturallanguageprocessing.com/train-ai/fine-tune-large-language-model-for-sentence-similarity/) and [video tutorial](https://www.youtube.com/watch?v=SyHXRxkO0tQ).\n",
        "\n",
        "This is if you have training data that indicates which sentences you consider to be similar, and you want to have a custom sentence similarity model.\n",
        "\n",
        "It's adapted from the scripts for the [Harmony](https://harmonydata.ac.uk/doxa/)/[DOXA AI](https://doxaai.com/competition/harmony-matching) competition (fine tune an LLM for the psychology domain), but it's general and can be applied to other domains. Credit to Jeremy Lo Ying Ping for the training code."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "df40b2ca",
      "metadata": {
        "id": "df40b2ca"
      },
      "outputs": [],
      "source": [
        "# !pip install pandas==2.2.2 transformers==4.43.1 sentence-transformers[train]==3.0.1"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "cecaf046",
      "metadata": {
        "id": "cecaf046"
      },
      "outputs": [],
      "source": [
        "import pandas as pd"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "fe075c62",
      "metadata": {
        "id": "fe075c62",
        "outputId": "d21f0240-f293-4f2b-9f5d-5922776dd735"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "--2024-11-20 11:12:19--  https://naturallanguageprocessing.com/harmony-matching-training-data.csv.zip\n",
            "Resolving naturallanguageprocessing.com (naturallanguageprocessing.com)... 3.125.36.175, 3.124.100.143, 2a05:d01c:9e6:f100::1f4, ...\n",
            "Connecting to naturallanguageprocessing.com (naturallanguageprocessing.com)|3.125.36.175|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 21159 (21K) [application/zip]\n",
            "Saving to: ‘harmony-matching-training-data.csv.zip.1’\n",
            "\n",
            "harmony-matching-tr 100%[===================>]  20.66K  --.-KB/s    in 0.09s   \n",
            "\n",
            "2024-11-20 11:12:20 (242 KB/s) - ‘harmony-matching-training-data.csv.zip.1’ saved [21159/21159]\n",
            "\n"
          ]
        }
      ],
      "source": [
        "!wget https://naturallanguageprocessing.com/harmony-matching-training-data.csv.zip"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "8ff3fe1e",
      "metadata": {
        "id": "8ff3fe1e"
      },
      "outputs": [],
      "source": [
        "df = pd.read_csv(\"harmony-matching-training-data.csv.zip\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "b82336b5",
      "metadata": {
        "id": "b82336b5",
        "outputId": "e8113814-8239-4965-e6cc-3923f47b2402"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>sentence1</th>\n",
              "      <th>sentence2</th>\n",
              "      <th>score</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Do you believe in telepathy (mind-reading)?</td>\n",
              "      <td>I believe that there are secret signs in the w...</td>\n",
              "      <td>0.15</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Irritable behavior, angry outbursts, or acting...</td>\n",
              "      <td>Felt “on edge”?</td>\n",
              "      <td>0.62</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>I have some eccentric (odd) habits.</td>\n",
              "      <td>I often have difficulty following what someone...</td>\n",
              "      <td>0.00</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Do you often feel nervous when you are in a gr...</td>\n",
              "      <td>Been easily annoyed by different things?</td>\n",
              "      <td>0.00</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Do you believe in telepathy (mind-reading)?</td>\n",
              "      <td>Most of the time I find it is very difficult t...</td>\n",
              "      <td>0.26</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2346</th>\n",
              "      <td>Little interest or pleasure in doing things</td>\n",
              "      <td>At times I have wondered if my body was really...</td>\n",
              "      <td>0.00</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2347</th>\n",
              "      <td>Feeling down, depressed, or hopeless?</td>\n",
              "      <td>I find that I am very often confused about wha...</td>\n",
              "      <td>0.00</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2348</th>\n",
              "      <td>Not being able to stop or control worrying?</td>\n",
              "      <td>If given the choice, I would much rather be wi...</td>\n",
              "      <td>0.16</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2349</th>\n",
              "      <td>Feeling nervous, anxious or on edge?</td>\n",
              "      <td>Have had changes in appetite or sleep?</td>\n",
              "      <td>0.16</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2350</th>\n",
              "      <td>Do you believe in clairvoyance (psychic forces...</td>\n",
              "      <td>Felt hopeless?</td>\n",
              "      <td>0.20</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>2351 rows × 3 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "                                              sentence1  \\\n",
              "0           Do you believe in telepathy (mind-reading)?   \n",
              "1     Irritable behavior, angry outbursts, or acting...   \n",
              "2                   I have some eccentric (odd) habits.   \n",
              "3     Do you often feel nervous when you are in a gr...   \n",
              "4           Do you believe in telepathy (mind-reading)?   \n",
              "...                                                 ...   \n",
              "2346        Little interest or pleasure in doing things   \n",
              "2347              Feeling down, depressed, or hopeless?   \n",
              "2348        Not being able to stop or control worrying?   \n",
              "2349               Feeling nervous, anxious or on edge?   \n",
              "2350  Do you believe in clairvoyance (psychic forces...   \n",
              "\n",
              "                                              sentence2  score  \n",
              "0     I believe that there are secret signs in the w...   0.15  \n",
              "1                                       Felt “on edge”?   0.62  \n",
              "2     I often have difficulty following what someone...   0.00  \n",
              "3              Been easily annoyed by different things?   0.00  \n",
              "4     Most of the time I find it is very difficult t...   0.26  \n",
              "...                                                 ...    ...  \n",
              "2346  At times I have wondered if my body was really...   0.00  \n",
              "2347  I find that I am very often confused about wha...   0.00  \n",
              "2348  If given the choice, I would much rather be wi...   0.16  \n",
              "2349             Have had changes in appetite or sleep?   0.16  \n",
              "2350                                     Felt hopeless?   0.20  \n",
              "\n",
              "[2351 rows x 3 columns]"
            ]
          },
          "execution_count": 5,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "df"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "72893aca",
      "metadata": {
        "id": "72893aca"
      },
      "outputs": [],
      "source": [
        "from sklearn.model_selection import train_test_split"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "8f57b87c",
      "metadata": {
        "id": "8f57b87c"
      },
      "outputs": [],
      "source": [
        "df_train, df_test = train_test_split(df)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "59c3aa49",
      "metadata": {
        "id": "59c3aa49"
      },
      "outputs": [],
      "source": [
        "df_train.reset_index(inplace=True)\n",
        "df_test.reset_index(inplace=True)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "dfe5faf5",
      "metadata": {
        "id": "dfe5faf5"
      },
      "outputs": [],
      "source": [
        "df_train.drop(columns=[\"index\"], inplace=True)\n",
        "df_test.drop(columns=[\"index\"], inplace=True)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "bf7cbe72",
      "metadata": {
        "id": "bf7cbe72",
        "outputId": "7ec45739-890b-4786-e598-8c6446291373"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "(1763, 588)"
            ]
          },
          "execution_count": 12,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "len(df_train), len(df_test)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "1bdfbeb7",
      "metadata": {
        "id": "1bdfbeb7",
        "outputId": "a0325161-5e39-40e4-907a-95f04d757c1a"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/home/thomas/projects_client/wellcome/simple_fine_tune_llm/venv/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
            "  from .autonotebook import tqdm as notebook_tqdm\n"
          ]
        }
      ],
      "source": [
        "from datasets import Dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "95614c28",
      "metadata": {
        "id": "95614c28"
      },
      "outputs": [],
      "source": [
        "dataset_train = Dataset.from_pandas(df_train)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "3c7c447c",
      "metadata": {
        "id": "3c7c447c",
        "outputId": "bf77f8ec-e07f-4202-cd53-5fea50b82117"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "Dataset({\n",
              "    features: ['sentence1', 'sentence2', 'score'],\n",
              "    num_rows: 1763\n",
              "})"
            ]
          },
          "execution_count": 15,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "dataset_train"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "3cdd4de6",
      "metadata": {
        "id": "3cdd4de6"
      },
      "outputs": [],
      "source": [
        "dataset_test = Dataset.from_pandas(df_test)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "af0bf898",
      "metadata": {
        "id": "af0bf898"
      },
      "outputs": [],
      "source": [
        "from sentence_transformers import SentenceTransformer"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "3ede7446",
      "metadata": {
        "id": "3ede7446"
      },
      "outputs": [],
      "source": [
        "model = SentenceTransformer(\"all-mpnet-base-v2\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "8622f3d4",
      "metadata": {
        "id": "8622f3d4",
        "outputId": "ce52f6bd-e23c-4d6a-f3ca-52ea5f347b58"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "SentenceTransformer(\n",
              "  (0): Transformer({'max_seq_length': 384, 'do_lower_case': False}) with Transformer model: MPNetModel \n",
              "  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})\n",
              "  (2): Normalize()\n",
              ")"
            ]
          },
          "execution_count": 19,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "a8311482",
      "metadata": {
        "id": "a8311482"
      },
      "outputs": [],
      "source": [
        "from sentence_transformers import SentenceTransformerTrainer, SentenceTransformerTrainingArguments"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "86ccd990",
      "metadata": {
        "id": "86ccd990"
      },
      "outputs": [],
      "source": [
        "from sentence_transformers.losses import CosineSimilarityLoss"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "aae30cb2",
      "metadata": {
        "id": "aae30cb2"
      },
      "outputs": [],
      "source": [
        "loss = CosineSimilarityLoss(model)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "49dff646",
      "metadata": {
        "id": "49dff646"
      },
      "outputs": [],
      "source": [
        "trainer = SentenceTransformerTrainer(\n",
        "    model = model,\n",
        "    args = SentenceTransformerTrainingArguments(\n",
        "        output_dir=\"checkpoints\",\n",
        "        num_train_epochs=3,\n",
        "        per_device_eval_batch_size=16\n",
        "    ),\n",
        "    train_dataset=dataset_train,\n",
        "    loss = loss\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "722b90c4",
      "metadata": {
        "id": "722b90c4",
        "outputId": "091c7dc2-5a05-4b07-d411-c34e7074f674"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "<sentence_transformers.trainer.SentenceTransformerTrainer at 0x7a081c90f810>"
            ]
          },
          "execution_count": 24,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "trainer"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "b05d5067",
      "metadata": {
        "id": "b05d5067",
        "outputId": "1e52b977-9368-40e2-8164-9a9aead10b6d"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "    <div>\n",
              "      \n",
              "      <progress value='663' max='663' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
              "      [663/663 09:12, Epoch 3/3]\n",
              "    </div>\n",
              "    <table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              " <tr style=\"text-align: left;\">\n",
              "      <th>Step</th>\n",
              "      <th>Training Loss</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>500</td>\n",
              "      <td>0.059000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table><p>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "                                                                                \r"
          ]
        },
        {
          "data": {
            "text/plain": [
              "TrainOutput(global_step=663, training_loss=0.05342011286302569, metrics={'train_runtime': 553.4312, 'train_samples_per_second': 9.557, 'train_steps_per_second': 1.198, 'total_flos': 0.0, 'train_loss': 0.05342011286302569, 'epoch': 3.0})"
            ]
          },
          "execution_count": 25,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "trainer.train()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "83521720",
      "metadata": {
        "id": "83521720",
        "outputId": "2890ef6a-3ade-4817-ddf5-0c034b489cae"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "SentenceTransformer(\n",
              "  (0): Transformer({'max_seq_length': 384, 'do_lower_case': False}) with Transformer model: MPNetModel \n",
              "  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})\n",
              "  (2): Normalize()\n",
              ")"
            ]
          },
          "execution_count": 26,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "09dbebdc",
      "metadata": {
        "id": "09dbebdc",
        "outputId": "85a4b964-7bf2-4032-8990-151257a7b8dd"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "array([[-9.93097015e-03, -7.88239613e-02,  6.90429413e-04,\n",
              "        -5.47722541e-03, -7.27352547e-03,  5.51959611e-02,\n",
              "        -1.99138150e-02,  3.13243978e-02, -2.88939774e-02,\n",
              "         4.32792865e-02, -1.23789981e-02, -2.06399094e-02,\n",
              "        -7.18724504e-02,  3.27489302e-02, -4.31292318e-02,\n",
              "         6.49863109e-02,  1.22211650e-02,  5.89610077e-02,\n",
              "         2.65878662e-02, -3.05647179e-02,  4.32941429e-02,\n",
              "        -2.40641739e-03, -5.77951176e-03, -2.87189931e-02,\n",
              "         3.09480391e-02,  3.15308012e-02,  3.17449169e-03,\n",
              "         2.75935028e-02, -2.97866464e-02, -7.41776377e-02,\n",
              "        -2.04117745e-02, -5.84216751e-02,  5.05814608e-03,\n",
              "        -3.80146243e-02, -8.17573047e-04,  6.60591898e-03,\n",
              "        -4.15992066e-02, -1.89618208e-02, -1.86170433e-02,\n",
              "        -7.61985127e-03,  1.60014117e-03, -1.46569568e-04,\n",
              "         1.97578203e-02,  7.30020879e-03, -3.35050002e-02,\n",
              "        -5.17571718e-02,  4.44276333e-02, -6.86508091e-03,\n",
              "         7.25207552e-02,  5.16764298e-02, -2.18805224e-02,\n",
              "         8.15098733e-03,  2.33492143e-02,  5.14722476e-03,\n",
              "        -5.93515076e-02,  6.52836114e-02, -1.81287061e-02,\n",
              "        -2.58715581e-02,  5.45821444e-04,  3.27210128e-02,\n",
              "        -2.64700856e-02, -9.48601123e-03, -1.43790338e-02,\n",
              "         2.32603140e-02,  4.22959402e-02,  4.08104099e-02,\n",
              "         4.75186072e-02, -3.13352048e-02, -2.05234755e-02,\n",
              "         7.97022134e-03, -8.89174119e-02,  4.91959043e-02,\n",
              "         2.64519509e-02,  4.48617153e-03, -5.20493351e-02,\n",
              "         2.93725803e-02,  8.35298211e-04,  2.23560687e-02,\n",
              "        -5.49549190e-03,  8.18704348e-03, -1.70258284e-02,\n",
              "        -1.68192550e-04, -3.34876706e-03, -6.68500084e-03,\n",
              "         4.62641893e-03,  8.92840773e-02,  5.66975633e-03,\n",
              "        -1.94224901e-02,  2.00492647e-02, -3.85029130e-02,\n",
              "        -1.98528375e-02,  3.46207246e-02,  3.43431905e-02,\n",
              "        -4.41583060e-03, -1.74210910e-02,  3.66451815e-02,\n",
              "         1.45564470e-02, -7.33786374e-02, -5.81308156e-02,\n",
              "         3.17164361e-02, -9.50173885e-02, -4.49874997e-02,\n",
              "         3.17774974e-02, -3.42077762e-02, -2.00163797e-02,\n",
              "        -8.21851019e-04, -6.73844712e-03, -3.13218720e-02,\n",
              "        -2.12509558e-02,  5.38934804e-02,  1.68978646e-02,\n",
              "         2.87788324e-02, -2.92079765e-02, -3.83944809e-03,\n",
              "        -2.68233865e-02, -2.89553162e-02, -2.47899964e-02,\n",
              "         9.44919186e-04,  9.26328078e-03,  5.37505820e-02,\n",
              "        -4.69463356e-02,  5.26020415e-02, -3.16085517e-02,\n",
              "         1.52996425e-02,  2.06278302e-02, -8.69394913e-02,\n",
              "         2.07666364e-02, -8.11431184e-03, -6.66408986e-02,\n",
              "        -7.38533959e-02,  1.47389220e-02, -1.25552025e-02,\n",
              "         3.22961584e-02, -7.06444979e-02, -2.80122012e-02,\n",
              "        -1.92374922e-02, -3.17664593e-02, -1.56253930e-02,\n",
              "         4.50676307e-02, -3.15512554e-03, -6.00441322e-02,\n",
              "         2.86311004e-02,  2.91907620e-02, -1.71141159e-02,\n",
              "         3.16237286e-02, -1.00749712e-02,  6.83614006e-03,\n",
              "        -2.89854035e-02, -2.74556559e-02, -1.14707146e-02,\n",
              "        -2.33214744e-03,  1.50538853e-03, -5.01444042e-02,\n",
              "        -6.56415196e-03,  7.25685572e-03, -2.61234418e-02,\n",
              "         1.46697117e-02, -3.87239866e-02,  3.69120352e-02,\n",
              "         5.74892201e-03,  2.01799665e-02, -1.27332401e-03,\n",
              "         2.16584075e-02,  7.21250195e-03,  6.01056740e-02,\n",
              "         4.69980985e-02,  6.36349916e-02, -1.25255100e-02,\n",
              "        -2.80049350e-02, -3.86290103e-02,  1.90513451e-02,\n",
              "        -5.45495600e-02,  3.14335153e-02,  1.82708409e-02,\n",
              "         2.90997233e-02,  1.25132436e-02, -5.21570407e-02,\n",
              "        -6.20649122e-02,  3.35110463e-02,  2.58957576e-02,\n",
              "        -1.85767990e-02,  1.22550095e-03,  9.96995438e-03,\n",
              "        -4.86798696e-02, -9.88247711e-03,  1.50727946e-02,\n",
              "        -4.00038846e-02,  1.91104366e-03, -5.69428317e-03,\n",
              "         1.04048308e-02,  2.48221755e-02,  4.01936052e-03,\n",
              "        -7.24406093e-02,  3.00331637e-02, -2.37798579e-02,\n",
              "        -2.84661669e-02,  1.82565190e-02, -2.62410548e-02,\n",
              "        -4.58296873e-02, -3.17766257e-02,  2.59290636e-02,\n",
              "        -7.14325579e-03, -1.43861733e-02, -4.30621691e-02,\n",
              "        -5.77357486e-02, -3.44007537e-02,  3.70824314e-03,\n",
              "        -1.48900701e-02, -1.38520459e-02,  5.35479337e-02,\n",
              "        -2.26497259e-02,  1.01706749e-02, -3.35641205e-03,\n",
              "        -2.53534745e-02, -2.34235451e-02,  3.74897458e-02,\n",
              "         1.08066704e-02, -4.78164926e-02,  4.67346944e-02,\n",
              "         7.80878440e-02, -2.78546363e-02, -2.34893579e-02,\n",
              "         1.24811288e-02,  9.92004201e-03,  1.65585428e-02,\n",
              "        -7.23274201e-02,  2.84838285e-02, -3.07688693e-04,\n",
              "         5.25147282e-02, -3.15473489e-02,  8.84083856e-04,\n",
              "        -2.11959071e-02,  3.47858369e-02, -1.02550993e-02,\n",
              "        -5.94935492e-02, -3.07523292e-02, -4.02308442e-02,\n",
              "        -3.80971730e-02, -1.59776863e-02,  5.77587187e-02,\n",
              "         3.62224467e-02,  3.43089714e-03, -6.48626909e-02,\n",
              "        -9.90067981e-03,  7.23643228e-03, -1.11689502e-02,\n",
              "         4.05428633e-02,  1.13301650e-02,  7.15938359e-02,\n",
              "        -7.45382626e-03,  1.96248647e-02,  2.52909008e-02,\n",
              "         1.24495374e-02, -1.41376089e-02,  5.89971021e-02,\n",
              "        -7.72715919e-03,  9.71090514e-03, -5.20889312e-02,\n",
              "        -2.11602598e-02, -8.82225856e-02,  4.22409782e-03,\n",
              "         1.58073772e-02, -3.94735746e-02, -3.39694805e-02,\n",
              "        -1.20138153e-02, -2.59832814e-02,  7.25636398e-03,\n",
              "         8.24383833e-03,  9.69246402e-03,  3.88158881e-03,\n",
              "         5.75911738e-02,  3.47757898e-02,  8.20915308e-03,\n",
              "        -8.20530485e-03, -2.56908555e-02, -5.76969655e-03,\n",
              "        -1.50898825e-02,  4.79328781e-02,  2.26178188e-02,\n",
              "        -2.12783627e-02,  1.09514790e-02,  1.74430758e-02,\n",
              "         1.40930561e-03, -5.23349121e-02, -1.39604611e-02,\n",
              "         2.52168123e-02, -3.60210761e-02, -4.08040211e-02,\n",
              "         1.92250521e-03,  1.34305090e-01, -3.84785584e-04,\n",
              "         2.68598087e-02,  1.33513100e-03, -5.60640953e-02,\n",
              "        -2.65635573e-03, -5.30268028e-02,  5.01362197e-02,\n",
              "         7.93046691e-03,  2.99237017e-03, -4.84268740e-02,\n",
              "         4.45651188e-02,  3.09933815e-03,  2.33845878e-03,\n",
              "        -3.75917181e-02, -7.74892652e-03, -3.46492790e-02,\n",
              "        -3.46923829e-03,  1.60445087e-03,  5.36332419e-03,\n",
              "         4.25287755e-03,  5.65021262e-02,  1.41322957e-02,\n",
              "        -4.72897664e-02, -5.22282161e-02,  2.78562643e-02,\n",
              "        -2.97437664e-02,  2.58386303e-02, -4.97534610e-02,\n",
              "         1.02214387e-03,  4.12032455e-02,  1.29654128e-02,\n",
              "        -8.07149708e-03, -1.45487264e-02,  2.30557323e-02,\n",
              "         1.82280857e-02, -1.77202560e-02, -2.78155021e-02,\n",
              "         5.27387746e-02,  5.63709624e-02,  7.32285306e-02,\n",
              "         3.08469646e-02, -1.36246476e-02, -1.69602372e-02,\n",
              "         2.00981610e-02, -1.05424531e-01, -2.77437363e-02,\n",
              "        -2.65150927e-02, -5.92903048e-02, -1.10887447e-02,\n",
              "        -3.76849063e-02,  1.77263692e-02, -1.84976906e-02,\n",
              "        -6.52214105e-04,  4.42824662e-02, -4.38720919e-02,\n",
              "        -1.25085087e-02,  2.47769188e-02,  1.40304808e-02,\n",
              "         8.38394556e-03, -6.07412271e-02, -2.48952433e-02,\n",
              "        -1.74647141e-02, -2.81292875e-03, -1.23998979e-02,\n",
              "         1.94200240e-02,  1.24580547e-01,  6.51368499e-03,\n",
              "         7.45696994e-03,  3.52268554e-02,  4.79606213e-03,\n",
              "         2.49394644e-02,  3.09203826e-02,  6.24031126e-02,\n",
              "         3.82409580e-02, -5.63194565e-02,  2.98394486e-02,\n",
              "        -1.64779108e-02, -6.11122772e-02, -5.75550981e-02,\n",
              "        -3.20036076e-02, -2.97924913e-02, -1.36375884e-02,\n",
              "         1.84620284e-02,  7.35989539e-03,  7.67264655e-03,\n",
              "         1.51452245e-02, -1.00341346e-02, -2.26781741e-02,\n",
              "         3.21169454e-03, -3.84435877e-02, -5.20903617e-04,\n",
              "         1.19217224e-01,  2.02406906e-02,  2.38272990e-03,\n",
              "        -1.46077489e-02,  3.82435322e-03, -1.43811079e-02,\n",
              "        -3.69094283e-04, -3.88251096e-02, -1.35045014e-02,\n",
              "        -2.01563686e-02,  2.94097271e-02,  6.13991469e-02,\n",
              "        -2.36425009e-02,  4.66924459e-02,  7.60550937e-03,\n",
              "        -8.55188519e-02, -8.52316152e-03, -3.33973877e-02,\n",
              "         9.17620398e-03, -4.14858386e-02, -5.91096431e-02,\n",
              "         4.94764149e-02,  1.83060803e-02,  1.29509261e-02,\n",
              "        -6.90906793e-02, -2.21202131e-02,  5.66405654e-02,\n",
              "        -3.41459090e-04, -8.62754229e-03,  2.08524801e-02,\n",
              "        -7.97548369e-02,  9.00920480e-03, -2.58836094e-02,\n",
              "         5.67261055e-02, -2.75861099e-03,  2.11484823e-02,\n",
              "         1.06080193e-02,  5.50888944e-03,  6.84187710e-02,\n",
              "        -2.71890443e-02,  2.53938860e-03,  1.69975646e-02,\n",
              "        -6.59491925e-04,  3.86469513e-02,  4.16666307e-02,\n",
              "         3.26947905e-02, -1.09452605e-02, -4.09358740e-02,\n",
              "        -3.48558761e-02, -3.57541293e-02, -6.05795234e-02,\n",
              "         3.01981959e-02,  6.62011607e-03, -1.57294311e-02,\n",
              "         5.28177619e-02,  3.93114015e-02, -3.97069985e-03,\n",
              "        -4.10855487e-02, -1.11452080e-02,  8.43420066e-03,\n",
              "        -3.06468457e-02,  2.87190173e-03,  9.96059738e-03,\n",
              "        -9.64498743e-02,  6.13085628e-02, -2.30285134e-02,\n",
              "        -3.12354285e-02,  1.68162361e-02,  2.34563593e-02,\n",
              "        -2.10614596e-03,  1.97560005e-02,  1.14266947e-01,\n",
              "         7.31550576e-03,  2.11794022e-02,  8.71926546e-03,\n",
              "         2.17200276e-02,  3.18300389e-02,  1.88984834e-02,\n",
              "        -9.66877025e-03, -1.22696124e-02, -2.25609704e-03,\n",
              "         5.09990985e-03,  3.87785546e-02,  5.88660054e-02,\n",
              "         9.63326618e-02,  7.53273070e-03, -1.48017675e-01,\n",
              "         2.38944311e-02, -1.84712261e-02, -4.25860696e-02,\n",
              "        -3.71337533e-02, -1.07780118e-02,  1.51891485e-02,\n",
              "         1.12851672e-02,  3.51926982e-02, -2.60130712e-03,\n",
              "        -5.39995059e-02, -8.88143317e-04, -1.76059585e-02,\n",
              "         6.25562146e-02,  3.39403898e-02,  7.78732635e-03,\n",
              "        -3.05685792e-02,  3.72484475e-02,  1.63053349e-02,\n",
              "         6.32334426e-02,  4.79801185e-02, -6.29519206e-03,\n",
              "         8.63171532e-04,  3.58332624e-03, -5.56813227e-03,\n",
              "        -1.16315261e-02,  1.46177988e-02,  4.53434102e-02,\n",
              "        -2.50840951e-02,  2.71081459e-02, -5.19406646e-02,\n",
              "         5.19520342e-02, -4.50559221e-02, -2.39223037e-02,\n",
              "         1.14377411e-02, -2.20622439e-02,  4.42010425e-02,\n",
              "        -1.30403722e-02, -2.28208564e-02,  4.43507830e-04,\n",
              "         2.10689604e-02, -8.44532531e-03,  4.74189445e-02,\n",
              "         1.33085612e-03,  6.07898869e-02, -2.91098282e-03,\n",
              "        -7.40657747e-02,  7.89567176e-03, -9.88540575e-02,\n",
              "        -2.52088439e-02, -1.78478267e-02,  9.76225957e-02,\n",
              "         1.62633583e-02,  4.90734912e-03,  1.79759357e-02,\n",
              "        -3.07113491e-02, -1.13753034e-02, -4.78789955e-02,\n",
              "         3.07994373e-02,  6.87975623e-03,  5.51008014e-03,\n",
              "         3.39662023e-02, -2.83418931e-02, -9.49283072e-04,\n",
              "         4.88654450e-02,  2.97796503e-02, -5.07745259e-02,\n",
              "        -1.37600806e-02,  7.51326606e-02,  3.60215604e-02,\n",
              "        -3.43477577e-02,  6.95187552e-03, -4.02338207e-02,\n",
              "         4.80036922e-02, -5.81517853e-02,  3.48616801e-02,\n",
              "         1.54899955e-02, -2.12594606e-02,  2.12482344e-02,\n",
              "         9.26569179e-02, -2.77101826e-02,  7.18892226e-03,\n",
              "         1.77949779e-02, -1.51779479e-03, -1.88262071e-02,\n",
              "         2.03879960e-02,  2.03410611e-02,  9.18134581e-03,\n",
              "         1.40598638e-06, -1.21294511e-02,  2.21081246e-02,\n",
              "         1.47772171e-02,  4.14784662e-02, -4.18055505e-02,\n",
              "         3.47442627e-02,  1.12613160e-02, -2.81881262e-02,\n",
              "         3.91076766e-02, -3.95940207e-02, -1.80789847e-02,\n",
              "         7.24482313e-02,  9.24664177e-03, -1.06427157e-02,\n",
              "        -2.33063195e-02,  7.18718022e-02,  3.66408899e-02,\n",
              "         1.66949388e-02, -2.05213390e-03, -1.86277423e-02,\n",
              "        -2.39393916e-02, -2.20370442e-02, -1.79685168e-02,\n",
              "         1.93510000e-02,  1.70946307e-02,  3.66659872e-02,\n",
              "        -5.98093122e-02, -3.91266868e-03,  9.20966826e-03,\n",
              "         5.69598423e-03, -2.61775237e-02,  1.13270842e-02,\n",
              "        -3.67328583e-04,  3.35796699e-02,  1.63845569e-02,\n",
              "         7.11628273e-02, -2.66448595e-02,  2.78638378e-02,\n",
              "        -4.22102883e-02, -1.30689759e-02, -2.15455424e-02,\n",
              "         2.01170538e-02, -9.76257212e-03, -1.32497260e-02,\n",
              "         6.68207416e-03, -2.38110088e-02,  5.12337126e-02,\n",
              "         1.36483768e-02,  6.67637354e-03, -1.97606403e-02,\n",
              "         3.35190147e-02,  1.95835568e-02, -2.31770973e-04,\n",
              "        -1.10489773e-02,  3.20340991e-02,  5.17655760e-02,\n",
              "        -4.69413062e-04, -1.35726156e-02, -2.81850584e-02,\n",
              "         5.00675924e-02, -5.15478589e-02, -8.99608526e-03,\n",
              "         1.22037139e-02,  1.87342037e-02, -1.99635364e-02,\n",
              "        -4.87917922e-02, -1.96458697e-02, -3.67415734e-02,\n",
              "         2.93023605e-02,  2.33537927e-02,  1.85578987e-02,\n",
              "         3.93454060e-02,  1.12628164e-02, -4.76690978e-02,\n",
              "        -3.60606126e-02, -6.71646297e-02, -4.29143244e-03,\n",
              "         4.63791452e-02, -1.28573366e-02,  5.52875362e-02,\n",
              "        -8.12371634e-03,  3.13801365e-03,  2.33841408e-02,\n",
              "         1.83150433e-02,  6.06734212e-03, -1.56954757e-03,\n",
              "         2.20407005e-02, -5.13043255e-02,  5.50066680e-02,\n",
              "        -1.63013698e-03, -2.76746731e-02, -3.47884372e-02,\n",
              "         2.21418850e-02,  2.12065992e-03, -1.14173954e-02,\n",
              "        -6.22665845e-02, -5.51666580e-02,  4.23728451e-02,\n",
              "         4.48402297e-03,  7.05890008e-04,  9.55815762e-02,\n",
              "        -2.47250218e-02, -7.51621788e-03,  5.77510148e-03,\n",
              "        -2.00332049e-02,  1.03895338e-02, -2.78000012e-02,\n",
              "         6.12407774e-02, -3.87842692e-02,  2.29596347e-02,\n",
              "         2.74388380e-02, -2.22527795e-02,  6.31261198e-03,\n",
              "         1.22114487e-01, -1.71156472e-03,  6.11956185e-03,\n",
              "        -4.46691178e-03, -2.58549172e-02, -2.70322338e-02,\n",
              "        -6.76629879e-03,  2.55172551e-02, -1.68218017e-02,\n",
              "         6.73243683e-03,  1.73529834e-02, -9.23034095e-04,\n",
              "        -1.24938125e-02, -3.75706516e-02,  8.55614021e-02,\n",
              "        -2.94366144e-02,  2.06268542e-02, -1.88197903e-02,\n",
              "         1.16766300e-02,  3.10269638e-06, -1.23560913e-02,\n",
              "        -2.28635371e-02, -1.35790361e-02,  6.83820173e-02,\n",
              "        -4.31757569e-02, -1.53303295e-02,  1.45954192e-02,\n",
              "        -8.82319757e-04, -5.84127987e-03, -3.76943983e-02,\n",
              "        -1.68475113e-03,  7.31623266e-03,  4.61038464e-04,\n",
              "        -3.54561256e-03,  6.44550920e-02, -7.52637982e-02,\n",
              "        -4.38138098e-02,  1.38564128e-02,  1.32690603e-02,\n",
              "        -5.08822910e-02,  1.95420310e-02, -4.33379151e-02,\n",
              "         4.94203493e-02,  2.98074316e-02, -2.99705807e-02,\n",
              "        -4.05526571e-02,  4.25738022e-02,  2.19945889e-02,\n",
              "         8.89865682e-03,  3.61816469e-03, -6.92404062e-02,\n",
              "         2.06656568e-02, -5.91102056e-02,  3.01958844e-02,\n",
              "        -2.14107577e-02, -2.98270471e-02, -2.64849095e-03,\n",
              "        -1.09114498e-02,  1.60688143e-02,  1.16549969e-01,\n",
              "         3.70282531e-02,  2.00547259e-02,  4.73112287e-03,\n",
              "        -6.11007884e-02, -3.59723084e-02, -5.44430502e-02,\n",
              "         2.58670058e-02,  5.07306531e-02, -6.55050278e-02,\n",
              "         1.73777603e-02, -3.57558741e-03, -3.78977843e-02,\n",
              "         9.28891357e-03,  3.60958055e-02,  8.51649232e-03,\n",
              "         3.86020821e-03,  1.24493130e-02,  9.75836813e-03,\n",
              "         1.62890293e-02,  7.95784444e-02,  4.69000312e-03,\n",
              "         5.39525831e-03, -2.97935214e-02,  7.09557757e-02,\n",
              "         3.71690281e-02, -5.83673157e-02, -2.36520078e-02,\n",
              "         1.03157072e-05, -2.88816113e-02,  5.33035994e-02,\n",
              "         2.22559460e-02, -4.00685295e-02, -5.54708019e-02,\n",
              "         9.85449180e-03, -2.92167068e-02, -2.18849741e-02,\n",
              "        -3.62964422e-02, -1.33359190e-02, -3.98904458e-02]], dtype=float32)"
            ]
          },
          "execution_count": 28,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "model.encode([\"I feel nervous, anxious or afraid\"])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "810499ca",
      "metadata": {
        "id": "810499ca"
      },
      "outputs": [],
      "source": [
        "sentence_1_embeddings = model.encode(df_test.sentence1)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "501f0271",
      "metadata": {
        "id": "501f0271"
      },
      "outputs": [],
      "source": [
        "sentence_2_embeddings = model.encode(df_test.sentence2)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "65407d06",
      "metadata": {
        "id": "65407d06",
        "outputId": "af08209a-af43-4309-a353-2c06a6f8675f"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "(588, 768)"
            ]
          },
          "execution_count": 33,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "sentence_1_embeddings.shape"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "05b1ed4e",
      "metadata": {
        "id": "05b1ed4e"
      },
      "outputs": [],
      "source": [
        "from numpy import dot, matmul, ndarray, matrix\n",
        "from numpy.linalg import norm\n",
        "import numpy as np\n",
        "def cosine_similarity(vec1: ndarray, vec2: ndarray) -> ndarray:\n",
        "    dp = dot(vec1, vec2.T)\n",
        "    m1 = matrix(norm(vec1, axis=1))\n",
        "    m2 = matrix(norm(vec2.T, axis=0))\n",
        "\n",
        "    return np.asarray(dp / matmul(m1.T, m2))\n",
        ""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "4286d7f6",
      "metadata": {
        "id": "4286d7f6"
      },
      "outputs": [],
      "source": [
        "similarity_matrix = cosine_similarity(sentence_1_embeddings, sentence_2_embeddings)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "1763055f",
      "metadata": {
        "id": "1763055f",
        "outputId": "170ef719-7133-4f73-f409-2d575e16942a"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "(588, 588)"
            ]
          },
          "execution_count": 49,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "similarity_matrix.shape"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "87f369ba",
      "metadata": {
        "id": "87f369ba"
      },
      "outputs": [],
      "source": [
        "df_test[\"y_pred\"] = [similarity_matrix[i,i] for i in range(len(similarity_matrix))]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "503e834f",
      "metadata": {
        "id": "503e834f",
        "outputId": "f39af262-c81a-48ac-8f31-d2983960c46a"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>sentence1</th>\n",
              "      <th>sentence2</th>\n",
              "      <th>score</th>\n",
              "      <th>y_pred</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>I sometimes jump quickly from one topic to ano...</td>\n",
              "      <td>People find my conversations to be confusing o...</td>\n",
              "      <td>0.70</td>\n",
              "      <td>0.700950</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Trouble concentrating on things, such as readi...</td>\n",
              "      <td>Felt nervous or anxious?</td>\n",
              "      <td>0.00</td>\n",
              "      <td>0.289797</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Loss of interest in activities that you used t...</td>\n",
              "      <td>My thoughts and behaviors are almost always di...</td>\n",
              "      <td>0.70</td>\n",
              "      <td>0.197288</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Avoiding external reminders of the stressful e...</td>\n",
              "      <td>Some people can make me aware of them just by ...</td>\n",
              "      <td>0.00</td>\n",
              "      <td>0.114830</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>I sometimes jump quickly from one topic to ano...</td>\n",
              "      <td>I have trouble following conversations with ot...</td>\n",
              "      <td>0.25</td>\n",
              "      <td>0.433558</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>583</th>\n",
              "      <td>I often ramble on too much when speaking.</td>\n",
              "      <td>I have had the momentary feeling that someone'...</td>\n",
              "      <td>0.01</td>\n",
              "      <td>0.049073</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>584</th>\n",
              "      <td>I sometimes jump quickly from one topic to ano...</td>\n",
              "      <td>Avoiding external reminders of the experience ...</td>\n",
              "      <td>0.40</td>\n",
              "      <td>0.114422</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>585</th>\n",
              "      <td>Being so restless that it is hard to sit still?</td>\n",
              "      <td>I believe that dreams have magical properties.</td>\n",
              "      <td>0.37</td>\n",
              "      <td>0.066781</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>586</th>\n",
              "      <td>Trouble relaxing?</td>\n",
              "      <td>Throughout my life, very few things have been ...</td>\n",
              "      <td>0.00</td>\n",
              "      <td>0.244570</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>587</th>\n",
              "      <td>I sometimes forget what I am trying to say.</td>\n",
              "      <td>Some people can make me aware of them just by ...</td>\n",
              "      <td>0.00</td>\n",
              "      <td>0.031787</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>588 rows × 4 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "                                             sentence1  \\\n",
              "0    I sometimes jump quickly from one topic to ano...   \n",
              "1    Trouble concentrating on things, such as readi...   \n",
              "2    Loss of interest in activities that you used t...   \n",
              "3    Avoiding external reminders of the stressful e...   \n",
              "4    I sometimes jump quickly from one topic to ano...   \n",
              "..                                                 ...   \n",
              "583          I often ramble on too much when speaking.   \n",
              "584  I sometimes jump quickly from one topic to ano...   \n",
              "585    Being so restless that it is hard to sit still?   \n",
              "586                                  Trouble relaxing?   \n",
              "587        I sometimes forget what I am trying to say.   \n",
              "\n",
              "                                             sentence2  score    y_pred  \n",
              "0    People find my conversations to be confusing o...   0.70  0.700950  \n",
              "1                             Felt nervous or anxious?   0.00  0.289797  \n",
              "2    My thoughts and behaviors are almost always di...   0.70  0.197288  \n",
              "3    Some people can make me aware of them just by ...   0.00  0.114830  \n",
              "4    I have trouble following conversations with ot...   0.25  0.433558  \n",
              "..                                                 ...    ...       ...  \n",
              "583  I have had the momentary feeling that someone'...   0.01  0.049073  \n",
              "584  Avoiding external reminders of the experience ...   0.40  0.114422  \n",
              "585     I believe that dreams have magical properties.   0.37  0.066781  \n",
              "586  Throughout my life, very few things have been ...   0.00  0.244570  \n",
              "587  Some people can make me aware of them just by ...   0.00  0.031787  \n",
              "\n",
              "[588 rows x 4 columns]"
            ]
          },
          "execution_count": 54,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "df_test"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "0e951ed8",
      "metadata": {
        "id": "0e951ed8"
      },
      "outputs": [],
      "source": [
        "df_test[\"residual\"] = df_test.y_pred - df_test.score"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "94abbcd0",
      "metadata": {
        "id": "94abbcd0",
        "outputId": "3319fdec-a845-4082-e79c-447e27cec116"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>sentence1</th>\n",
              "      <th>sentence2</th>\n",
              "      <th>score</th>\n",
              "      <th>y_pred</th>\n",
              "      <th>residual</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>I sometimes jump quickly from one topic to ano...</td>\n",
              "      <td>People find my conversations to be confusing o...</td>\n",
              "      <td>0.70</td>\n",
              "      <td>0.700950</td>\n",
              "      <td>0.000950</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Trouble concentrating on things, such as readi...</td>\n",
              "      <td>Felt nervous or anxious?</td>\n",
              "      <td>0.00</td>\n",
              "      <td>0.289797</td>\n",
              "      <td>0.289797</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Loss of interest in activities that you used t...</td>\n",
              "      <td>My thoughts and behaviors are almost always di...</td>\n",
              "      <td>0.70</td>\n",
              "      <td>0.197288</td>\n",
              "      <td>-0.502712</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Avoiding external reminders of the stressful e...</td>\n",
              "      <td>Some people can make me aware of them just by ...</td>\n",
              "      <td>0.00</td>\n",
              "      <td>0.114830</td>\n",
              "      <td>0.114830</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>I sometimes jump quickly from one topic to ano...</td>\n",
              "      <td>I have trouble following conversations with ot...</td>\n",
              "      <td>0.25</td>\n",
              "      <td>0.433558</td>\n",
              "      <td>0.183558</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>583</th>\n",
              "      <td>I often ramble on too much when speaking.</td>\n",
              "      <td>I have had the momentary feeling that someone'...</td>\n",
              "      <td>0.01</td>\n",
              "      <td>0.049073</td>\n",
              "      <td>0.039073</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>584</th>\n",
              "      <td>I sometimes jump quickly from one topic to ano...</td>\n",
              "      <td>Avoiding external reminders of the experience ...</td>\n",
              "      <td>0.40</td>\n",
              "      <td>0.114422</td>\n",
              "      <td>-0.285578</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>585</th>\n",
              "      <td>Being so restless that it is hard to sit still?</td>\n",
              "      <td>I believe that dreams have magical properties.</td>\n",
              "      <td>0.37</td>\n",
              "      <td>0.066781</td>\n",
              "      <td>-0.303219</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>586</th>\n",
              "      <td>Trouble relaxing?</td>\n",
              "      <td>Throughout my life, very few things have been ...</td>\n",
              "      <td>0.00</td>\n",
              "      <td>0.244570</td>\n",
              "      <td>0.244570</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>587</th>\n",
              "      <td>I sometimes forget what I am trying to say.</td>\n",
              "      <td>Some people can make me aware of them just by ...</td>\n",
              "      <td>0.00</td>\n",
              "      <td>0.031787</td>\n",
              "      <td>0.031787</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>588 rows × 5 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "                                             sentence1  \\\n",
              "0    I sometimes jump quickly from one topic to ano...   \n",
              "1    Trouble concentrating on things, such as readi...   \n",
              "2    Loss of interest in activities that you used t...   \n",
              "3    Avoiding external reminders of the stressful e...   \n",
              "4    I sometimes jump quickly from one topic to ano...   \n",
              "..                                                 ...   \n",
              "583          I often ramble on too much when speaking.   \n",
              "584  I sometimes jump quickly from one topic to ano...   \n",
              "585    Being so restless that it is hard to sit still?   \n",
              "586                                  Trouble relaxing?   \n",
              "587        I sometimes forget what I am trying to say.   \n",
              "\n",
              "                                             sentence2  score    y_pred  \\\n",
              "0    People find my conversations to be confusing o...   0.70  0.700950   \n",
              "1                             Felt nervous or anxious?   0.00  0.289797   \n",
              "2    My thoughts and behaviors are almost always di...   0.70  0.197288   \n",
              "3    Some people can make me aware of them just by ...   0.00  0.114830   \n",
              "4    I have trouble following conversations with ot...   0.25  0.433558   \n",
              "..                                                 ...    ...       ...   \n",
              "583  I have had the momentary feeling that someone'...   0.01  0.049073   \n",
              "584  Avoiding external reminders of the experience ...   0.40  0.114422   \n",
              "585     I believe that dreams have magical properties.   0.37  0.066781   \n",
              "586  Throughout my life, very few things have been ...   0.00  0.244570   \n",
              "587  Some people can make me aware of them just by ...   0.00  0.031787   \n",
              "\n",
              "     residual  \n",
              "0    0.000950  \n",
              "1    0.289797  \n",
              "2   -0.502712  \n",
              "3    0.114830  \n",
              "4    0.183558  \n",
              "..        ...  \n",
              "583  0.039073  \n",
              "584 -0.285578  \n",
              "585 -0.303219  \n",
              "586  0.244570  \n",
              "587  0.031787  \n",
              "\n",
              "[588 rows x 5 columns]"
            ]
          },
          "execution_count": 56,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "df_test"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "2e945a5a",
      "metadata": {
        "id": "2e945a5a",
        "outputId": "cbb715dc-f279-4816-fea6-53affb22e8c2"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "np.float64(0.06922512204080025)"
            ]
          },
          "execution_count": 61,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "np.mean(df_test.residual * df_test.residual)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "beb093b4",
      "metadata": {
        "id": "beb093b4",
        "outputId": "bdf07c36-d952-490a-b215-f900e5f4ace7"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "np.float64(0.20813965150724928)"
            ]
          },
          "execution_count": 62,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "np.mean(np.abs(df_test.residual))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "c389d11e",
      "metadata": {
        "id": "c389d11e"
      },
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "finetunellm",
      "language": "python",
      "name": "finetunellm"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.3"
    },
    "colab": {
      "provenance": [],
      "include_colab_link": true
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}