example-finetuning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **Harmony**: Question Matching Algorithm Improvement Challenge\n",
    "\n",
    "**NLP challenge** | [Visit the challenge page](https://doxaai.com/competition/harmony-matching)\n",
    "\n",
    "Your challenge is to develop an improved algorithm for matching psychology survey questions that produces similarity ratings more closely aligned with those given by humans psychologists working in the field and that can be integrated into the [Harmony tool](https://harmonydata.ac.uk/developer-guide/).\n",
    "\n",
    "This notebook will expand upon the getting-started resources in `getting-started.ipynb`, showing you how to fine-tune a pre-trained model!\n",
    "\n",
    "**Before you get started, make sure to [sign up for an account](https://doxaai.com/sign-up) if you do not already have one and [enrol to take part](https://doxaai.com/competition/harmony-matching) in the challenge.**\n",
    "\n",
    "**If you have any questions, feel free to ask them in the [Harmony community Discord server](https://discord.com/invite/harmonydata).**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installing and importing useful packages\n",
    "\n",
    "Before you get started, please make sure you have [PyTorch](https://pytorch.org/get-started/locally/) installed in your Python environment. If you do not have `pandas`, `seaborn`, `transformers` or `sentence-transformers`, the code in the following cell will install them.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install \"pandas>=2.2.2\" \"seaborn>=0.13.2\" \"transformers>=4.43.1\" \"sentence-transformers[train]>=3.0.1\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Install the latest version of the DOXA CLI\n",
    "%pip install -U doxa-cli"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "import pandas as pd\n",
    "\n",
    "pd.set_option(\"display.max_colwidth\", None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading the data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Download the dataset if we do not already have it\n",
    "if not os.path.exists(\"train.csv\"):\n",
    "    !curl https://raw.githubusercontent.com/DoxaAI/harmony-matching-getting-started/main/train.csv --output train.csv\n",
    "\n",
    "if not os.path.exists(\"submission\"):\n",
    "    !curl https://raw.githubusercontent.com/DoxaAI/harmony-matching-getting-started/main/submission/competition.py --create-dirs --output submission_finetuning/competition.py\n",
    "    !curl https://raw.githubusercontent.com/DoxaAI/harmony-matching-getting-started/main/submission/doxa.yaml --output submission_finetuning/doxa.yaml\n",
    "    !curl https://raw.githubusercontent.com/DoxaAI/harmony-matching-getting-started/main/submission/model.py --output submission_finetuning/model.py\n",
    "    !curl https://raw.githubusercontent.com/DoxaAI/harmony-matching-getting-started/main/submission/run.py --output submission_finetuning/run.py\n",
    "\n",
    "# Load the data\n",
    "df = pd.read_csv(\"train.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to fine-tune a pre-trained model with `SentenceTransformers`, we need to transform our data to be in a slightly different format:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasets import Dataset\n",
    "\n",
    "df = df[[\"sentence_1\", \"sentence_2\", \"human_similarity\"]].rename(\n",
    "    columns={\n",
    "        \"sentence_1\": \"sentence1\",\n",
    "        \"sentence_2\": \"sentence2\",\n",
    "        \"human_similarity\": \"score\",\n",
    "    }\n",
    ")\n",
    "\n",
    "# Rescale the scores to be in the range [0.0, 1.0]\n",
    "df[\"score\"] /= 100.0\n",
    "\n",
    "dataset = Dataset.from_pandas(\n",
    "    df,\n",
    ")\n",
    "\n",
    "dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fine-tuning a SentenceTransformers model\n",
    "\n",
    "In this notebook, we will walk you through how to fine-tune a pre-trained [SentenceTransformers](https://sbert.net/) model for our task.\n",
    "\n",
    "There are multiple fine-tuning approaches that you can take, but in this example, we are going to fine-tune a pre-trained `all-mpnet-base-v2` model using the `CosineSimilarityLoss` in order to make our model produce cosine similarity-based scores that align more closely with the human-provided similarity scores.\n",
    "\n",
    "First, we import a pre-trained model once again:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sentence_transformers import SentenceTransformer\n",
    "\n",
    "model = SentenceTransformer(\"all-mpnet-base-v2\")\n",
    "\n",
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we can use the useful functionality built into `SentenceTransformers` to start fine-tuning the model:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sentence_transformers import (\n",
    "    SentenceTransformerTrainer,\n",
    "    SentenceTransformerTrainingArguments,\n",
    ")\n",
    "from sentence_transformers.losses import CosineSimilarityLoss\n",
    "\n",
    "\n",
    "loss = CosineSimilarityLoss(model)\n",
    "\n",
    "trainer = SentenceTransformerTrainer(\n",
    "    model=model,\n",
    "    args=SentenceTransformerTrainingArguments(\n",
    "        output_dir=\"checkpoints\",\n",
    "        num_train_epochs=3,\n",
    "        per_device_train_batch_size=16,\n",
    "        report_to=\"none\",\n",
    "    ),\n",
    "    train_dataset=dataset,\n",
    "    loss=loss,\n",
    ")\n",
    "\n",
    "trainer.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now compute some evaluation metrics using this fine-tuned model:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sentence_transformers.evaluation import (\n",
    "    EmbeddingSimilarityEvaluator,\n",
    "    SimilarityFunction,\n",
    ")\n",
    "\n",
    "dev_evaluator = EmbeddingSimilarityEvaluator(\n",
    "    sentences1=df[\"sentence1\"],\n",
    "    sentences2=df[\"sentence2\"],\n",
    "    scores=df[\"score\"],\n",
    "    main_similarity=SimilarityFunction.COSINE,\n",
    ")\n",
    "\n",
    "dev_evaluator(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Producing a submission\n",
    "\n",
    "**Now, we will move onto creating your submission!**\n",
    "\n",
    "Just as before in the `getting-started.ipynb` notebook, we need to prepare a submission folder containing our fine-tuned model, as well as the code necessary to use it (which is identical to the code we used in the `getting-started.ipynb` notebook).\n",
    "\n",
    "Currently, the `submission_finetuning/` folder contains three files:\n",
    "\n",
    "- `submission_finetuning/competition.py`: this contains competition-specific code used to interface with the platform\n",
    "- `submission_finetuning/doxa.yaml`: this is a configuration file used by the DOXA CLI when you make a submission\n",
    "- `submission_finetuning/run.py`: this is the Python script that gets run when your work gets evaluated\n",
    "\n",
    "All that is left to do is to save the SentenceTransformer model we have just fine-tuned into our `submission_finetuning/` directory:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.save(\"submission_finetuning/model\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Uploading your submission to the platform\n",
    "\n",
    "You are now ready to make your first submission to the platform! 👀\n",
    "\n",
    "**Make sure to [enrol to take part](https://doxaai.com/competition/harmony-matching) in the challenge if you have not already done so.**\n",
    "\n",
    "First, we need to make sure we are logged in:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!doxa login"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And then, we can submit our work for evaluation:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!doxa upload submission_finetuning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Congratulations!** 🥳\n",
    "\n",
    "By this point, you will now have just made a submission for this challenge on the DOXA AI platform!\n",
    "\n",
    "If everything went well, your submission will now be queued up for evaluation. It will first be run on a small validation set to make sure that your submission does not crash on the full test set. If your submission runs into an issue at this point, you will be able to see the error logs from this phase. Otherwise, if your submission passes this stage, it will be evaluated on the full test set, and you will soon appear on the [competition scoreboard](https://doxaai.com/competition/harmony-matching/scoreboard)!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next steps\n",
    "\n",
    "**Now, it is up to you as to where you go from here to solve this challenge!**\n",
    "\n",
    "We would highly recommend taking a look at the [SentenceTransformers documentation](https://sbert.net/index.html) and the [HuggingFace `transformers` documentation](https://huggingface.co/docs/transformers/en/training) for inspiration as to what to do next!\n",
    "\n",
    "**We look forward to seeing what you build!** We would love to hear about what you are working on for this challenge, so do let us know how you are finding the challenge on the [Harmony community Discord server](https://discord.com/invite/harmonydata) or the [DOXA AI community Discord server](https://discord.gg/MUvbQ3UYcf). 😎\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "ml",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}