notebooks/h2o_mrm_full.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0382ba3d",
   "metadata": {},
   "source": [
    "# 🤖 GenAI Model Validation Workshop <a class='anchor' id='top'></a>\n",
    "\n",
    "[Program](https://docs.google.com/document/d/1uqOlTim6czjeK16xXz4tXvYznTxkiy19moeoGNSynSY/edit?tab=t.0#heading=h.8c6jf2k12z8) | [GitHub](https://github.com/h2oai/h2o-genai-model-validation-training) | [Enterprise h2oGPTe](https://h2ogpte.h2oworld.h2o.ai/) | [EvalStudio](https://eval-studio.h2oworld.h2o.ai/)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5539caa5",
   "metadata": {},
   "source": [
    "## 📝 Outline <a class='anchor' id='outline'></a>\n",
    "1. [Environment Preparation](#preparation)\n",
    "2. [Embedding and Explainability](#embedding_explainability)\n",
    "3. [Test Generation and Benchmarking](#test_gen)\n",
    "4. [Eval Metrics and RAG](#eval_metrics)\n",
    "5. [Human Evals](#human_evals)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f6f58f2",
   "metadata": {},
   "source": [
    "## 🛠️ Environment Preparation <a class='anchor' id='preparation'></a> [↑](#top)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30c8b53b",
   "metadata": {},
   "source": [
    "### Check compatibility your browser\n",
    "\n",
    "Run the following cells - they will check compatibility of your browser and refresh the page.\n",
    "\n",
    "*Technical Note: the Python kernel is not impacted by page refresh*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ee85ce7a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import ipywidgets as widgets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "764c34b0",
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import display, Javascript\n",
    "\n",
    "\n",
    "# Refresh the page only if a specific flag is not set\n",
    "def refresh_page_once():\n",
    "    display(\n",
    "        Javascript(\"\"\"\n",
    "    if (!localStorage.getItem('pageRefreshed')) {\n",
    "        localStorage.setItem('pageRefreshed', 'true');\n",
    "        window.location.reload();\n",
    "    } else {\n",
    "        localStorage.removeItem('pageRefreshed');\n",
    "    }\n",
    "    \"\"\")\n",
    "    )\n",
    "\n",
    "\n",
    "refresh_page_once()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c19aefae",
   "metadata": {},
   "outputs": [],
   "source": [
    "widgets.Button(description=\"Your browser is OK\", disabled=True, button_style=\"success\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4a96287",
   "metadata": {},
   "source": [
    "### Get h2oGPTe API Key\n",
    "\n",
    "1. Got to [h2oGPTe Settings](https://h2ogpte.h2oworld.h2o.ai/settings).\n",
    "2. Generate new API key and copy the key.\n",
    "3. Fill the key into text box below\n",
    "4. Click on 'Generate config' button"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f685bc32-4274-4a25-8e59-02f059171f35",
   "metadata": {},
   "outputs": [],
   "source": [
    "from h2o_mrm import _generate_env\n",
    "\n",
    "_generate_env(\"https://h2ogpte.h2oworld.h2o.ai\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "de67a0d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "!cat .env"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db1de409",
   "metadata": {},
   "source": [
    "### 🐍 Prepare Python Environment [↑](#top)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4fe59426-68b7-4ae4-9b59-6edffdd97758",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Supress Warnings\n",
    "import warnings\n",
    "\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "\n",
    "# Load Environment Variables\n",
    "from dotenv import load_dotenv\n",
    "\n",
    "_ = load_dotenv()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "860fc8e8-2022-4eb5-a753-a11defc9c168",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Python packages\n",
    "import os\n",
    "import uuid\n",
    "from pathlib import Path\n",
    "\n",
    "# Experiment\n",
    "from h2o_mrm.experiment import Experiment\n",
    "\n",
    "# Topic Modeling\n",
    "from h2o_mrm.widgets import topic_model_widget\n",
    "from h2o_mrm.viz import create_chunk_distribution_map, create_topics_distribution_pie\n",
    "\n",
    "# Question Generation\n",
    "from h2o_mrm.widgets.chunk_nav import create_qa_gen_widget\n",
    "from h2o_mrm.widgets.chunk_nav.core import create_question_generator, create_summarizer\n",
    "\n",
    "# Generated Question Evaluation\n",
    "from h2o_mrm.widgets.aw_data_table import create_genqa_eval_widget\n",
    "\n",
    "# RAG Models\n",
    "from h2o_mrm.rag_models import H2OGPTERAG, H2ogpteConfig\n",
    "\n",
    "# Human Labeling\n",
    "from h2o_mrm.widgets import human_labeling_widget"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "68e3abcc",
   "metadata": {},
   "outputs": [],
   "source": [
    "CACHE_LOC = \"/home/jovyan/cache\"\n",
    "DOCS_LOC = \"/home/jovyan/docs\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4cba4498-12c5-4950-995b-f1ab9179fdcf",
   "metadata": {},
   "source": [
    "# 1. Embedding and Explainability <a class='anchor' id='embedding_explainability'></a> [↑](#top)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4261a1c",
   "metadata": {},
   "source": [
    "The goal of experiment is to analyze document [\"Comptroller’s Handbook: Model Risk Management\"](https://www.occ.treas.gov/publications-and-resources/publications/comptrollers-handbook/files/model-risk-management/index-model-risk-management.html) in the context of RAG systems.\n",
    "\n",
    "## Experiment\n",
    "\n",
    "Experiment defines scope of work including documents and rag system under testing.\n",
    "\n",
    "It does:\n",
    " - chunking of document using H2OGPTe chunking strategy.\n",
    " - embedding of chunks into vectors using given embedding model.\n",
    "  \n",
    "\n",
    "> ℹ️ Note: we pre-cached computed results to speed up the workshop\n",
    "  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c54c3dd8-d35a-46f2-a09b-0c90bc3a021b",
   "metadata": {},
   "outputs": [],
   "source": [
    "exp = Experiment(\n",
    "    \"OCC Handbook\",  # Do not change name since it is used for cache look-ups to speed up computation.\n",
    "    max_tokens_per_chunk=320,\n",
    "    embedding_model_name=\"BAAI/bge-m3\",\n",
    "    cache_dir=CACHE_LOC,\n",
    ")\n",
    "exp.add_documents([f\"{DOCS_LOC}/pub-ch-model-risk.pdf\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "27b3ca12",
   "metadata": {},
   "outputs": [],
   "source": [
    "exp"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b160d92e-fb70-473e-979c-92b90645b632",
   "metadata": {},
   "source": [
    "### Create Chunks\n",
    "\n",
    "Divide document into chunks of specified number of tokens.\n",
    "\n",
    "In this step:\n",
    "- Documnets that are part of this collection will be parsed into plain text\n",
    "- The parsed text will be divided into chunks. Chunk is a string of words/sentences that add up to the `tokens_per_chunk` value.\n",
    "- Each chunk of tokens is then encoded into a vector of floats using an embedding model.\n",
    "- The vectors are then stored in a vector database.\n",
    "- A typical vector database can store each chunk as text, the embedding vector, and any meatadata associated with it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "54131845-a157-4a44-82f5-54533d6707eb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create and Save Chunks\n",
    "exp_chunks = exp.chunk_documents()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4258e0e2-1bca-4031-9423-b39b1956ad46",
   "metadata": {},
   "source": [
    "All the data from our documents is parsed, chunked, and transformed into vectors and stored in a vector database.\n",
    "The chunks in their text form can be reviewed as shown below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6a40520d-6921-4558-8827-08ad8a4c958e",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(exp_chunks[100].text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c2b7ef7-a2b8-4f2a-af16-5a971fed5a5c",
   "metadata": {},
   "source": [
    "#### Topic Modeling\n",
    "\n",
    "In this section, we can identify the topics in our collection of documents by clustering all the chunks.\n",
    "Creating a topic model involves multiple steps.\n",
    "1. Dimentionality Reduction: Since embeddings are in high dimensions (1024 in this example), before applying clustering algorithms, we need to reduce the dimensionality of the embeddings.\n",
    "2. Clustering: Apply clustering algorithms such as HDBSCAN, K-Means, etc ... to group the vectors in reduced dimensional space into multiple clusters.\n",
    "3. Topic representation: Identify the most important words/phrases in each topic and create descriptions for the topics.\n",
    "\n",
    "For each step, we can choose different techniques and each technique can have multiple hyper-parameters.\n",
    "Therefore, we tend to create multiple topic models by tuning the hyper-parameters. We then measure the quality of each topic model using the silhouette score metric.\n",
    "\n",
    "For this experiment, we already built 34 different topic models. The following command will show the top 10 topic models measured by the silhouette score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "443de6c6-a811-474d-b03f-cecd60434832",
   "metadata": {},
   "outputs": [],
   "source": [
    "# List topic models for this experiment\n",
    "exp.list_topic_models(top=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cdec9935-aaeb-4bf0-bbe4-11c943477458",
   "metadata": {},
   "source": [
    "We can build additional topic models in a automatic way by using the following command.\n",
    "\n",
    "- specify different ranges of values for each of the three hyper-parameters `n_neighbors`, `n_components`, and `min_cluster_size`.\n",
    "- `n_neighbors`, `n_components` are inputs to the `UMAP` algorithm used for dimentionality reduction.\n",
    "- `min_cluster_size` is an input to the `HDBSCAN` algorithm to control the clustering.\n",
    "- Topic models with all combinations of the specified ranges will be built and ranked using the silhouette score.\n",
    "- If you do not wish to explore all combinations of the hyper-parameters, you can specify a list of combinations to try.\n",
    "- If `combinations` is specified, arguments `n_neighbors`, `n_components`, and `min_cluster_size` will be ignored.\n",
    "- `combinations` is a list of Tuple(`n_neighbors`, `n_components`, `min_cluster_size`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1ae6b340-8068-42c9-bb44-c7c2c522e316",
   "metadata": {},
   "outputs": [],
   "source": [
    "exp.build_all_topic_models(\n",
    "    n_neighbors=[35, 40],\n",
    "    n_components=[5, 10, 15, 20],\n",
    "    min_cluster_size=[5, 7, 9],\n",
    "    # combinations=[(10, 2, 10), (10, 2, 11), (25, 25, 9), (15, 2, 10)],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51b84f4e-8443-488e-bb20-43c392f03cb1",
   "metadata": {},
   "source": [
    "To see the list of newly built topic models we can re-run the `exp.list_topic_models(top=10)` command. To proceed with\n",
    "our experiment, we need to select a topic model to represent the information in our collection.\n",
    "\n",
    "`exp.set_best_topic_model()` will select the topic model with the highest silhouette score. \n",
    "however, you can also select any topic model from the list by its id using the `exp.set_topic_model` command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2b24aee4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Select the topic model with best silhouette_score\n",
    "exp.set_best_topic_model()\n",
    "\n",
    "# Alternatively, select any topic model by it's id\n",
    "# exp.set_topic_model(\"60d5c18c-9ad4-4d7a-8a9b-eac2dc9eb77e\")\n",
    "\n",
    "# Verify that the selected topic model is properly set\n",
    "exp.selected_topic_model_id"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50974fff-5e64-4a13-9cec-6406c972f961",
   "metadata": {},
   "source": [
    "The following is an interactive UI widget to visualize the selected topic model.\n",
    "we can also build new topic models by interacting with the widget."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "19e5fd8c-0dc7-4d7e-9ae5-24383bc87546",
   "metadata": {},
   "outputs": [],
   "source": [
    "tmw = topic_model_widget.create_widget(\n",
    "    topic_model_id=exp.selected_topic_model_id,\n",
    "    cache_dir=exp.cache_dir,\n",
    "    create_topic_cluster_data=exp.build_topic_cluster_creator(\n",
    "        show_doc_in_tooltip=True,\n",
    "        show_topic_names=True,\n",
    "        # hidden_topics=[0],\n",
    "    ),\n",
    ")\n",
    "tmw"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8135a03-162e-4998-8588-daefe5fe3baf",
   "metadata": {},
   "source": [
    "If you interacted with the topic model widget to create a new topic model,\n",
    "and want to select the newly created one as the topic model for this experiment,\n",
    "run the following lines."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ec5efdb2-8201-4d0b-b24e-531e7bcb1ab7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Select the topic model from the widget above\n",
    "# exp.set_topic_model(uuid.UUID(hex=tmw.topic_model_id))\n",
    "\n",
    "# Verify that the new topic model is selected\n",
    "# exp.selected_topic_model_id"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ec2b8d5-7412-4400-9bb4-41be0f487262",
   "metadata": {},
   "source": [
    "The following 3 commands will help you understand the distribution of chunks across different topics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3ea6c259-a3bd-45e5-85d1-4446c55ba7d8",
   "metadata": {},
   "outputs": [],
   "source": [
    "exp.get_num_chunks_in_topic_chart()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b1890493-ada0-4a53-b3ce-bbc6cc21e514",
   "metadata": {},
   "outputs": [],
   "source": [
    "create_chunk_distribution_map(exp.chunks, exp.topic_names, x_size=20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8b12df4c-8786-4831-8a01-1767087783fe",
   "metadata": {},
   "outputs": [],
   "source": [
    "create_topics_distribution_pie(exp.chunks, exp.topic_names, filter_topics=[-1])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cbb60cc4-7844-400b-b23a-52591f4a0920",
   "metadata": {},
   "source": [
    "# 2. Test Generation and Benchmarking <a class='anchor' id='test_gen'></a> [↑](#top)\n",
    "\n",
    "- Automatic Prompt engineering\n",
    "- Automatic QA generation\n",
    "\n",
    "Users can implement multiple different techniques for question generation. we included an example implementation.\n",
    "\n",
    "Question generation includes the following steps:\n",
    "1. Select one or more chunks from a cluster.\n",
    "2. If the clusters are small, all chunks can be selected.\n",
    "3. If the clusters are large, a `twin` (a statistically similar subset) of the cluster can be selected. we do this to cover all the information represented in the cluster without exhaustively selecting all chunks.\n",
    "4. Summarize the selected chunks using LLM.\n",
    "5. Generate questions with the summary as the reference using LLM.\n",
    "6. Validate generated questions using NLP techniques like `cosine similarity`, `BERTScore`, and `NLI Score`.\n",
    "\n",
    "The summarization and question generation steps can have multiple implementations using different LLMs and system-prompts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "74160752-e215-4881-9de9-e4ef3fbc6cb2",
   "metadata": {},
   "outputs": [],
   "source": [
    "_ = load_dotenv()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c496c402-069a-4514-9506-b847c31cb990",
   "metadata": {},
   "outputs": [],
   "source": [
    "os.getenv(\"H2OGPTE_API_KEY\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8999f9ba-f778-4599-8cc4-28361caec3bc",
   "metadata": {},
   "outputs": [],
   "source": [
    "llama_summerizer = create_summarizer(\n",
    "    model_type=\"h2ogpte\",\n",
    "    model_name=\"meta-llama/Meta-Llama-3.1-70B-Instruct\",\n",
    ")\n",
    "llama_question_generator = create_question_generator(\n",
    "    model_type=\"h2ogpte\",\n",
    "    model_name=\"meta-llama/Meta-Llama-3.1-70B-Instruct\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e58eb40-5572-4a3b-82fe-5435d044316a",
   "metadata": {},
   "source": [
    "#### Interactive Question Generation\n",
    "\n",
    "The following is a widget to experiment with the question generation process interactively.\n",
    "- Pick the lasso tool from the top-right corner of the plot.\n",
    "- Use the lasso tool to select a few chunks from a cluster.\n",
    "- This will trigger the summarizer (top right) to generate the summary of the selected chunks.\n",
    "- When the summary is ready, click the generate button (bottom right) to create questions with the generated summary as a reference.\n",
    "- experiment with different parts of the collection to verify that the generated questions are relevant."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7727ff43-8dfa-4b4a-89f2-58e9db2e53b6",
   "metadata": {},
   "outputs": [],
   "source": [
    "question_gen_widget = create_qa_gen_widget(\n",
    "    exp.chunks,\n",
    "    fig_data=exp.fig_data,\n",
    "    summarize_text=llama_summerizer,\n",
    "    generate_questions=llama_question_generator,\n",
    ")\n",
    "question_gen_widget"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20e6031e-3b1f-4c27-957c-9201e1f1bc9c",
   "metadata": {},
   "source": [
    "#### Automatic Question Generation\n",
    "\n",
    "We can automate the question generation process using the following command.\n",
    "\n",
    "- Select the topics from which questions need to be generated.\n",
    "- Specify a summarizer and a question generator implementation.\n",
    "- Specify a sampling method to pick chunks from each cluster. `twinning` or `all` (exhaustive) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dd789e96-0c8b-4fd4-ba2b-8265143f5db3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# exp.generate_questions(\n",
    "#     topics=[0, 1, 2, 3, 4, 5, 6],\n",
    "#     summarizer=llama_summerizer,\n",
    "#     question_generator=llama_question_generator,\n",
    "#     question_generator_name=\"Meta-Llama-3.1-70B-Instruct\",\n",
    "#     # sampling_method=\"twinning\",\n",
    "#     sampling_method=\"all\",\n",
    "# )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c579933-e86f-4672-a68b-4a670ea3f8fc",
   "metadata": {},
   "source": [
    "If you have already generated questions and saved them in cache, but your selected topic model for this experiment changed after that, the topics for questions need to updated.\n",
    "We can run this everytime to make sure that information in the cache is aligned with the current state of the experiment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "38d76b09-71aa-4edc-a0ae-20f199c91231",
   "metadata": {},
   "outputs": [],
   "source": [
    "exp.update_questions_topics()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "757eb4aa-6cbb-48f8-b6be-63034c01e0f1",
   "metadata": {},
   "source": [
    "Preview generated questions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "80a35419-3036-432f-90c7-589f9c9a26e8",
   "metadata": {},
   "outputs": [],
   "source": [
    "generated_questions = exp.list_generated_questions()\n",
    "print(len(generated_questions))\n",
    "for x in generated_questions[:5]:\n",
    "    print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7124db2-574d-44ae-a47b-c698364f83fc",
   "metadata": {},
   "source": [
    "#### Evaluate Generated Questions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6018c333-814b-4577-81e4-a6e64f9e2f9e",
   "metadata": {},
   "outputs": [],
   "source": [
    "exp.validate_generated_questions()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d9443eb-c06b-46da-8d1f-307d80dcada4",
   "metadata": {},
   "source": [
    "#### Load Validated Questions in a Widget\n",
    "\n",
    "All questions and the validation scores are presented as a table to browse."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b7ab96c2-6c71-4ff2-a7ef-635fd54824ba",
   "metadata": {},
   "outputs": [],
   "source": [
    "validated_questions = exp.get_validated_questions()\n",
    "genq_eval_widget = create_genqa_eval_widget(validated_questions)\n",
    "genq_eval_widget"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4dd9d31e-551d-443d-8710-92719508160a",
   "metadata": {},
   "source": [
    "# 3. Eval Metrics and RAG <a class='anchor' id='eval_metrics'></a> [↑](#top)\n",
    "\n",
    "#### Metrics\n",
    "\n",
    "- [X] Groundedness\n",
    "- [X] Context Recall\n",
    "- [X] Context Precision\n",
    "- [X] Recall Relevancy\n",
    "- [X] Precision Relevancy\n",
    "- [X] Answer Relevancy\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c1e7e66-ce87-4c21-b638-8ce242043f27",
   "metadata": {},
   "source": [
    "#### Get Answers from RAG"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1256fac7-514f-4af3-b4ea-8f324c65a4ef",
   "metadata": {},
   "source": [
    "In this section, we can register multiple RAG Models for testing.\n",
    "\n",
    "We need `rag_name`, `rag_version`, and `llm_name` to uniquely identify each RAG Model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c7158600-b95c-47d8-b7d5-8fafdf965259",
   "metadata": {},
   "outputs": [],
   "source": [
    "# NOTE: To be able to use the cached data, please do not modify anything in this cell\n",
    "\n",
    "rag_name = \"h2ogpte.dev.h2o.ai\"\n",
    "rag_version = \"1.6.0-dev28\"\n",
    "llm_name = \"meta-llama/Meta-Llama-3.1-70B-Instruct\"\n",
    "llm_args = dict(\n",
    "    temperature=0.0,\n",
    "    seed=42,\n",
    "    max_new_tokens=4096,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71aa2efd-f44b-4acb-a2dd-27733fc69ec0",
   "metadata": {},
   "source": [
    "use the `register_rag_under_test` method to register new RAG Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d096fddb-ee69-4d06-9a67-358aba661070",
   "metadata": {},
   "outputs": [],
   "source": [
    "# NOTE: To be able to use the cached data, please do not modify anything in this cell\n",
    "\n",
    "rag_under_test_id = exp.register_rag_under_test(\n",
    "    rag_name=rag_name,\n",
    "    rag_version=rag_version,\n",
    "    llm_name=llm_name,\n",
    "    llm_args=llm_args,\n",
    "    embedding_model_name=\"BAAI/bge-m3\",\n",
    ")\n",
    "rag_under_test_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dab9de33-01e1-4dfa-b367-f3bf46797f25",
   "metadata": {},
   "outputs": [],
   "source": [
    "# NOTE: To be able to use the cached data, please do not modify anything in this cell\n",
    "\n",
    "rag_collection_name = \"OCC Handbook 3\"\n",
    "config = H2ogpteConfig.from_env()\n",
    "rag = H2OGPTERAG(config, rag_collection_name, llm_name, llm_args)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "35c858a3-c212-4e5d-8f6c-1a9afa3e38c7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# NOTE: To be able to use the cached data, please do not modify anything in this cell\n",
    "\n",
    "rag.add_documents([Path(f\"{DOCS_LOC}/pub-ch-model-risk.pdf\")])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d3ad5025-709a-46ee-8daa-0d4bbaa350ac",
   "metadata": {},
   "outputs": [],
   "source": [
    "exp.get_answers_from_rag(\n",
    "    rag_under_test_id=rag_under_test_id,\n",
    "    answer_question=rag.answer_question,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5a8b6e73-ecd9-48ea-9529-45fcd892ffc3",
   "metadata": {},
   "outputs": [],
   "source": [
    "exp.add_rag_chunks(rag_under_test_id, rag.get_all_chunks)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1469f33-7d2c-42f5-b434-59a7133af81a",
   "metadata": {},
   "source": [
    "After obtaining answers for all the questions from the RAG Model, we can evaluate them on the 6 metrics listed above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eb335e23-a208-4828-a398-4867d4ab37ba",
   "metadata": {},
   "outputs": [],
   "source": [
    "exp.evaluate_answers(rag_under_test_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ca89bc31-d87e-4913-a4a2-3c955dfb75a3",
   "metadata": {},
   "outputs": [],
   "source": [
    "exp.plot_metrics(rag_under_test_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd075930-b301-4255-b89b-e2dc0b8e061f",
   "metadata": {},
   "source": [
    "# 4. Human Evaluation <a class='anchor' id='human_evals'></a> [↑](#top)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c250ffea-512e-4043-a120-5f93d383d6f8",
   "metadata": {},
   "source": [
    "Use the following widget to review the answers for all the questions.\n",
    "After the review, humans can provide labels per each metric.\n",
    "\n",
    "For example, users can mark a question/answer as grounded/not-grounded or relevant/not-relevant."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "676d0df7",
   "metadata": {},
   "outputs": [],
   "source": [
    "hlw = human_labeling_widget.create_widget(\n",
    "    fig_data_json=exp.plot_metric(\n",
    "       rag_under_test_id,\n",
    "       metric=\"groundedness\",\n",
    "       cache_file=os.path.join(CACHE_LOC, \"human_eval_groundedness_fig_data.json\"), \n",
    "    ),\n",
    "    answer_info_func=exp.get_answer_info_func(\n",
    "       rag_under_test_id,\n",
    "       cache_file=os.path.join(CACHE_LOC, \"human_eval_answer_info.json\"),\n",
    "    ),\n",
    "    question_id=402,\n",
    ")\n",
    "hlw"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c57d2a42-dc8b-4d03-9948-9f3113d84ce5",
   "metadata": {},
   "source": [
    "Using the labelled data from the step above, we can compute conformal prediction sets.\n",
    "In this example, only ~10% of the data is labelled.\n",
    "\n",
    "Therefore, by providing human feedback on 10% of the data, we are able to find a threshold that is calibrated to our data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ad40a3c8-a798-48a7-96cd-510a33e5bba4",
   "metadata": {},
   "outputs": [],
   "source": [
    "exp.compute_conformal_prediction_sets(\n",
    "    labelled_data=os.path.join(CACHE_LOC, \"labelled_data.csv\"),\n",
    "    unlabelled_data=os.path.join(CACHE_LOC, \"unlabelled_data.json\"),\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "816e3d1c-97d4-48f5-bb57-20b708d45f9f",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Done!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bc9614d4-29ed-449e-b2a6-af2e364254d8",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}