Lab_1_summarize_dialogue.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Generative AI Use Case: Summarize Dialogue\n",
    "\n",
    "Welcome to the practical side of this course. In this lab you will do the dialogue summarization task using generative AI. You will explore how the input text affects the output of the model, and perform prompt engineering to direct it towards the task you need. By comparing zero shot, one shot, and few shot inferences, you will take the first step towards prompt engineering and see how it can enhance the generative output of Large Language Models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Table of Contents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- [ 1 - Set up Kernel and Required Dependencies](#1)\n",
    "- [ 2 - Summarize Dialogue without Prompt Engineering](#2)\n",
    "- [ 3 - Summarize Dialogue with an Instruction Prompt](#3)\n",
    "  - [ 3.1 - Zero Shot Inference with an Instruction Prompt](#3.1)\n",
    "  - [ 3.2 - Zero Shot Inference with the Prompt Template from FLAN-T5](#3.2)\n",
    "- [ 4 - Summarize Dialogue with One Shot and Few Shot Inference](#4)\n",
    "  - [ 4.1 - One Shot Inference](#4.1)\n",
    "  - [ 4.2 - Few Shot Inference](#4.2)\n",
    "- [ 5 - Generative Configuration Parameters for Inference](#5)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='1'></a>\n",
    "## 1 - Set up Kernel and Required Dependencies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "First, check that the correct kernel is chosen.\n",
    "\n",
    "<img src=\"images/kernel_set_up.png\" width=\"300\"/>\n",
    "\n",
    "You can click on that (top right of the screen) to see and check the details of the image, kernel, and instance type.\n",
    "\n",
    "<img src=\"images/w1_kernel_and_instance_type.png\" width=\"600\"/>\n",
    "\n",
    "<img src=\"data:image/svg+xml;base64,Cjxzdmcgd2lkdGg9IjgwMCIgaGVpZ2h0PSIxMjUiIHZpZXdCb3g9IjAgMCA4MDAgMTI1IiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciPgogICAgPGRlZnM+CiAgICAgICAgPGxpbmVhckdyYWRpZW50IGlkPSJmYWRlR3JhZGllbnQiIHgxPSIwIiB4Mj0iMSI+CiAgICAgICAgICAgIDxzdG9wIG9mZnNldD0iMCUiIHN0b3AtY29sb3I9IiNGMEYwRjAiLz4KICAgICAgICAgICAgPHN0b3Agb2Zmc2V0PSIxMDAlIiBzdG9wLWNvbG9yPSIjRjBGMEYwIiBzdG9wLW9wYWNpdHk9IjAiLz4KICAgICAgICA8L2xpbmVhckdyYWRpZW50PgogICAgICAgIDxtYXNrIGlkPSJmYWRlTWFzayI+CiAgICAgICAgICAgIDxyZWN0IHg9IjAiIHk9IjAiIHdpZHRoPSI3NTAiIGhlaWdodD0iMTI1IiBmaWxsPSJ3aGl0ZSIvPgogICAgICAgICAgICA8cmVjdCB4PSI3NTAiIHk9IjAiIHdpZHRoPSI1MCIgaGVpZ2h0PSIxMjUiIGZpbGw9InVybCgjZmFkZUdyYWRpZW50KSIvPgogICAgICAgIDwvbWFzaz4KICAgIDwvZGVmcz4KICAgIDxwYXRoIGQ9Ik01MCw5NyBBNTAsNTAgMCAwIDEgNTMsMyBMNzk3LCAzIEw3OTcsOTcgTDUwLDk3IFoiIGZpbGw9IiNGMEYwRjAiIHN0cm9rZT0iI0UwRTBFMCIgc3Ryb2tlLXdpZHRoPSIxIiBtYXNrPSJ1cmwoI2ZhZGVNYXNrKSIvPgogICAgPHRleHQgeD0iMTAwIiB5PSIzNCIgZm9udC1mYW1pbHk9IkFyaWFsLCBzYW5zLXNlcmlmIiBmb250LXNpemU9IjE0IiBmaWxsPSIjMzMzMzMzIj5QbGVhc2UgbWFrZSBzdXJlIHRoYXQgeW91IGNob29zZTwvdGV4dD4gCiAgICA8dGV4dCB4PSIzMjAiIHk9IjM0IiBmb250LWZhbWlseT0iQXJpYWwsIHNhbnMtc2VyaWYiIGZvbnQtc2l6ZT0iMTQiIGZpbGw9IiMzMzMzMzMiIGZvbnQtd2VpZ2h0PSJib2xkIj5tbC5tNS4yeGxhcmdlPC90ZXh0PgogICAgPHRleHQgeD0iNDE4IiB5PSIzNCIgZm9udC1mYW1pbHk9IkFyaWFsLCBzYW5zLXNlcmlmIiBmb250LXNpemU9IjE0IiBmaWxsPSIjMzMzMzMzIj5pbnN0YW5jZSB0eXBlLjwvdGV4dD4KICAgIDx0ZXh0IHg9IjEwMCIgeT0iNTYiIGZvbnQtZmFtaWx5PSJBcmlhbCwgc2Fucy1zZXJpZiIgZm9udC1zaXplPSIxNCIgZmlsbD0iIzMzMzMzMyI+VG8gZmluZCB0aGF0IGluc3RhbmNlIHR5cGUsIHlvdSBtaWdodCBoYXZlIHRvIHNjcm9sbCBkb3duIHRvIHRoZSAiQWxsIEluc3RhbmNlcyIgc2VjdGlvbiBpbiB0aGUgZHJvcGRvd24uPC90ZXh0PgogICAgPHRleHQgeD0iMTAwIiB5PSI3OCIgZm9udC1mYW1pbHk9IkFyaWFsLCBzYW5zLXNlcmlmIiBmb250LXNpemU9IjE0IiBmaWxsPSIjMzMzMzMzIj5DaG9pY2Ugb2YgYW5vdGhlciBpbnN0YW5jZSB0eXBlIG1pZ2h0IGNhdXNlIHRyYWluaW5nIGZhaWx1cmUva2VybmVsIGhhbHQvYWNjb3VudCBkZWFjdGl2YXRpb24uPC90ZXh0Pgo8L3N2Zz4K\" alt=\"Time alert close\"/>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Expected instance type: instance-datascience-ml-m5-2xlarge\n",
      "Currently chosen instance type: instance-datascience-ml-m5-2xlarge\n",
      "Instance type has been chosen correctly.\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "\n",
    "instance_type_expected = 'ml-m5-2xlarge'\n",
    "instance_type_current = os.environ.get('HOSTNAME')\n",
    "\n",
    "print(f'Expected instance type: instance-datascience-{instance_type_expected}')\n",
    "print(f'Currently chosen instance type: {instance_type_current}')\n",
    "\n",
    "assert instance_type_expected in instance_type_current, f'ERROR. You selected the {instance_type_current} instance type. Please select {instance_type_expected} instead as shown on the screenshot above'\n",
    "print(\"Instance type has been chosen correctly.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "Now install the required packages to use PyTorch and Hugging Face transformers and datasets.\n",
    "\n",
    "<img src=\"data:image/svg+xml;base64,Cjxzdmcgd2lkdGg9IjgwMCIgaGVpZ2h0PSIxMjUiIHZpZXdCb3g9IjAgMCA4MDAgMTI1IiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciPgogICAgPGRlZnM+CiAgICAgICAgPGxpbmVhckdyYWRpZW50IGlkPSJmYWRlR3JhZGllbnQiIHgxPSIwIiB4Mj0iMSI+CiAgICAgICAgICAgIDxzdG9wIG9mZnNldD0iMCUiIHN0b3AtY29sb3I9IiNGMEYwRjAiLz4KICAgICAgICAgICAgPHN0b3Agb2Zmc2V0PSIxMDAlIiBzdG9wLWNvbG9yPSIjRjBGMEYwIiBzdG9wLW9wYWNpdHk9IjAiLz4KICAgICAgICA8L2xpbmVhckdyYWRpZW50PgogICAgICAgIDxtYXNrIGlkPSJmYWRlTWFzayI+CiAgICAgICAgICAgIDxyZWN0IHg9IjAiIHk9IjAiIHdpZHRoPSI3NTAiIGhlaWdodD0iMTI1IiBmaWxsPSJ3aGl0ZSIvPgogICAgICAgICAgICA8cmVjdCB4PSI3NTAiIHk9IjAiIHdpZHRoPSI1MCIgaGVpZ2h0PSIxMjUiIGZpbGw9InVybCgjZmFkZUdyYWRpZW50KSIvPgogICAgICAgIDwvbWFzaz4KICAgIDwvZGVmcz4KICAgIDxwYXRoIGQ9Ik0zLDUwIEE1MCw1MCAwIDAgMSA1MywzIEw3OTcsMyBMNzk3LDk3IEw5Nyw5NyBMNTAsMTE1IEwzLDk3IFoiIGZpbGw9IiNGMEYwRjAiIHN0cm9rZT0iI0UwRTBFMCIgc3Ryb2tlLXdpZHRoPSIxIiBtYXNrPSJ1cmwoI2ZhZGVNYXNrKSIvPgogICAgPGNpcmNsZSBjeD0iNTAiIGN5PSI1MCIgcj0iMzAiIGZpbGw9IiM1N2M0ZjgiIHN0cm9rZT0iIzU3YzRmOCIgc3Ryb2tlLXdpZHRoPSIxIi8+CiAgICA8Y2lyY2xlIGN4PSI1MCIgY3k9IjUwIiByPSIyNSIgZmlsbD0iI0YwRjBGMCIvPgogICAgPGxpbmUgeDE9IjUwIiB5MT0iNTAiIHgyPSI1MCIgeTI9IjMwIiBzdHJva2U9IiM1N2M0ZjgiIHN0cm9rZS13aWR0aD0iMyIgc3Ryb2tlLWxpbmVjYXA9InJvdW5kIi8+CiAgICA8bGluZSB4MT0iNTAiIHkxPSI1MCIgeDI9IjY1IiB5Mj0iNTAiIHN0cm9rZT0iIzU3YzRmOCIgc3Ryb2tlLXdpZHRoPSIzIiBzdHJva2UtbGluZWNhcD0icm91bmQiLz4KICAgIDx0ZXh0IHg9IjEwMCIgeT0iMzQiIGZvbnQtZmFtaWx5PSJBcmlhbCwgc2Fucy1zZXJpZiIgZm9udC1zaXplPSIxNCIgZmlsbD0iIzMzMzMzMyI+VGhlIG5leHQgY2VsbCBtYXkgdGFrZSBhIGZldyBtaW51dGVzIHRvIHJ1bi4gUGxlYXNlIGJlIHBhdGllbnQuPC90ZXh0PgogICAgPHRleHQgeD0iMTAwIiB5PSI1NiIgZm9udC1mYW1pbHk9IkFyaWFsLCBzYW5zLXNlcmlmIiBmb250LXNpemU9IjE0IiBmaWxsPSIjMzMzMzMzIj5JZ25vcmUgdGhlIHdhcm5pbmdzIGFuZCBlcnJvcnMsIGFsb25nIHdpdGggdGhlIG5vdGUgYWJvdXQgcmVzdGFydGluZyB0aGUga2VybmVsIGF0IHRoZSBlbmQuPC90ZXh0Pgo8L3N2Zz4K\" alt=\"Time alert open medium\"/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting datasets==2.17.0\n",
      "  Downloading datasets-2.17.0-py3-none-any.whl.metadata (20 kB)\n",
      "Requirement already satisfied: filelock in /opt/conda/lib/python3.10/site-packages (from datasets==2.17.0) (3.13.4)\n",
      "Requirement already satisfied: numpy>=1.17 in /opt/conda/lib/python3.10/site-packages (from datasets==2.17.0) (1.26.4)\n",
      "Requirement already satisfied: pyarrow>=12.0.0 in /opt/conda/lib/python3.10/site-packages (from datasets==2.17.0) (15.0.2)\n",
      "Requirement already satisfied: pyarrow-hotfix in /opt/conda/lib/python3.10/site-packages (from datasets==2.17.0) (0.6)\n",
      "Requirement already satisfied: dill<0.3.9,>=0.3.0 in /opt/conda/lib/python3.10/site-packages (from datasets==2.17.0) (0.3.8)\n",
      "Requirement already satisfied: pandas in /opt/conda/lib/python3.10/site-packages (from datasets==2.17.0) (2.2.2)\n",
      "Requirement already satisfied: requests>=2.19.0 in /opt/conda/lib/python3.10/site-packages (from datasets==2.17.0) (2.31.0)\n",
      "Requirement already satisfied: tqdm>=4.62.1 in /opt/conda/lib/python3.10/site-packages (from datasets==2.17.0) (4.66.1)\n",
      "Collecting xxhash (from datasets==2.17.0)\n",
      "  Downloading xxhash-3.4.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (12 kB)\n",
      "Requirement already satisfied: multiprocess in /opt/conda/lib/python3.10/site-packages (from datasets==2.17.0) (0.70.16)\n",
      "Collecting fsspec<=2023.10.0,>=2023.1.0 (from fsspec[http]<=2023.10.0,>=2023.1.0->datasets==2.17.0)\n",
      "  Downloading fsspec-2023.10.0-py3-none-any.whl.metadata (6.8 kB)\n",
      "Collecting aiohttp (from datasets==2.17.0)\n",
      "  Downloading aiohttp-3.9.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (7.5 kB)\n",
      "Collecting huggingface-hub>=0.19.4 (from datasets==2.17.0)\n",
      "  Downloading huggingface_hub-0.23.0-py3-none-any.whl.metadata (12 kB)\n",
      "Requirement already satisfied: packaging in /opt/conda/lib/python3.10/site-packages (from datasets==2.17.0) (23.2)\n",
      "Requirement already satisfied: pyyaml>=5.1 in /opt/conda/lib/python3.10/site-packages (from datasets==2.17.0) (6.0.1)\n",
      "Collecting aiosignal>=1.1.2 (from aiohttp->datasets==2.17.0)\n",
      "  Downloading aiosignal-1.3.1-py3-none-any.whl.metadata (4.0 kB)\n",
      "Requirement already satisfied: attrs>=17.3.0 in /opt/conda/lib/python3.10/site-packages (from aiohttp->datasets==2.17.0) (23.2.0)\n",
      "Collecting frozenlist>=1.1.1 (from aiohttp->datasets==2.17.0)\n",
      "  Downloading frozenlist-1.4.1-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (12 kB)\n",
      "Collecting multidict<7.0,>=4.5 (from aiohttp->datasets==2.17.0)\n",
      "  Downloading multidict-6.0.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.2 kB)\n",
      "Collecting yarl<2.0,>=1.0 (from aiohttp->datasets==2.17.0)\n",
      "  Downloading yarl-1.9.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (31 kB)\n",
      "Collecting async-timeout<5.0,>=4.0 (from aiohttp->datasets==2.17.0)\n",
      "  Downloading async_timeout-4.0.3-py3-none-any.whl.metadata (4.2 kB)\n",
      "Requirement already satisfied: typing-extensions>=3.7.4.3 in /opt/conda/lib/python3.10/site-packages (from huggingface-hub>=0.19.4->datasets==2.17.0) (4.11.0)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/lib/python3.10/site-packages (from requests>=2.19.0->datasets==2.17.0) (3.3.2)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.10/site-packages (from requests>=2.19.0->datasets==2.17.0) (3.6)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/lib/python3.10/site-packages (from requests>=2.19.0->datasets==2.17.0) (2.2.1)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.10/site-packages (from requests>=2.19.0->datasets==2.17.0) (2024.2.2)\n",
      "Requirement already satisfied: python-dateutil>=2.8.2 in /opt/conda/lib/python3.10/site-packages (from pandas->datasets==2.17.0) (2.9.0)\n",
      "Requirement already satisfied: pytz>=2020.1 in /opt/conda/lib/python3.10/site-packages (from pandas->datasets==2.17.0) (2024.1)\n",
      "Requirement already satisfied: tzdata>=2022.7 in /opt/conda/lib/python3.10/site-packages (from pandas->datasets==2.17.0) (2024.1)\n",
      "Requirement already satisfied: six>=1.5 in /opt/conda/lib/python3.10/site-packages (from python-dateutil>=2.8.2->pandas->datasets==2.17.0) (1.16.0)\n",
      "Downloading datasets-2.17.0-py3-none-any.whl (536 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m536.6/536.6 kB\u001b[0m \u001b[31m6.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m\n",
      "\u001b[?25hDownloading fsspec-2023.10.0-py3-none-any.whl (166 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m166.4/166.4 kB\u001b[0m \u001b[31m3.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m\n",
      "\u001b[?25hDownloading aiohttp-3.9.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.2/1.2 MB\u001b[0m \u001b[31m17.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m\n",
      "\u001b[?25hDownloading huggingface_hub-0.23.0-py3-none-any.whl (401 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m401.2/401.2 kB\u001b[0m \u001b[31m6.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m\n",
      "\u001b[?25hDownloading xxhash-3.4.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (194 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m194.1/194.1 kB\u001b[0m \u001b[31m3.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0mta \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hDownloading aiosignal-1.3.1-py3-none-any.whl (7.6 kB)\n",
      "Downloading async_timeout-4.0.3-py3-none-any.whl (5.7 kB)\n",
      "Downloading frozenlist-1.4.1-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (239 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m239.5/239.5 kB\u001b[0m \u001b[31m4.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m\n",
      "\u001b[?25hDownloading multidict-6.0.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (124 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m124.3/124.3 kB\u001b[0m \u001b[31m2.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m\n",
      "\u001b[?25hDownloading yarl-1.9.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (301 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m301.6/301.6 kB\u001b[0m \u001b[31m5.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m\n",
      "\u001b[?25hInstalling collected packages: xxhash, multidict, fsspec, frozenlist, async-timeout, yarl, huggingface-hub, aiosignal, aiohttp, datasets\n",
      "  Attempting uninstall: fsspec\n",
      "    Found existing installation: fsspec 2024.3.1\n",
      "    Uninstalling fsspec-2024.3.1:\n",
      "      Successfully uninstalled fsspec-2024.3.1\n",
      "Successfully installed aiohttp-3.9.5 aiosignal-1.3.1 async-timeout-4.0.3 datasets-2.17.0 frozenlist-1.4.1 fsspec-2023.10.0 huggingface-hub-0.23.0 multidict-6.0.5 xxhash-3.4.1 yarl-1.9.4\n",
      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0mNote: you may need to restart the kernel to use updated packages.\n",
      "Requirement already satisfied: pip in /opt/conda/lib/python3.10/site-packages (24.0)\n",
      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0mNote: you may need to restart the kernel to use updated packages.\n",
      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0mNote: you may need to restart the kernel to use updated packages.\n",
      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0mNote: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "%pip install -U datasets==2.17.0\n",
    "\n",
    "%pip install --upgrade pip\n",
    "%pip install --disable-pip-version-check \\\n",
    "    torch==1.13.1 \\\n",
    "    torchdata==0.5.1 --quiet\n",
    "\n",
    "%pip install \\\n",
    "    transformers==4.27.2 --quiet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "<img src=\"data:image/svg+xml;base64,Cjxzdmcgd2lkdGg9IjgwMCIgaGVpZ2h0PSI1MCIgdmlld0JveD0iMCAwIDgwMCA1MCIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIj4KICAgIDxkZWZzPgogICAgICAgIDxsaW5lYXJHcmFkaWVudCBpZD0iZmFkZUdyYWRpZW50IiB4MT0iMCIgeDI9IjEiPgogICAgICAgICAgICA8c3RvcCBvZmZzZXQ9IjAlIiBzdG9wLWNvbG9yPSIjRjBGMEYwIi8+CiAgICAgICAgICAgIDxzdG9wIG9mZnNldD0iMTAwJSIgc3RvcC1jb2xvcj0iI0YwRjBGMCIgc3RvcC1vcGFjaXR5PSIwIi8+CiAgICAgICAgPC9saW5lYXJHcmFkaWVudD4KICAgICAgICA8bWFzayBpZD0iZmFkZU1hc2siPgogICAgICAgICAgICA8cmVjdCB4PSIwIiB5PSIwIiB3aWR0aD0iNzUwIiBoZWlnaHQ9IjUwIiBmaWxsPSJ3aGl0ZSIvPgogICAgICAgICAgICA8cmVjdCB4PSI3NTAiIHk9IjAiIHdpZHRoPSI1MCIgaGVpZ2h0PSI1MCIgZmlsbD0idXJsKCNmYWRlR3JhZGllbnQpIi8+CiAgICAgICAgPC9tYXNrPgogICAgPC9kZWZzPgogICAgPHBhdGggZD0iTTI1LDUwIFEwLDUwIDAsMjUgTDUwLDMgTDk3LDI1IEw3OTcsMjUgTDc5Nyw1MCBMMjUsNTAgWiIgZmlsbD0iI0YwRjBGMCIgc3Ryb2tlPSIjRTBFMEUwIiBzdHJva2Utd2lkdGg9IjEiIG1hc2s9InVybCgjZmFkZU1hc2spIi8+Cjwvc3ZnPgo=\" alt=\"Time alert close\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "Load the datasets, Large Language Model (LLM), tokenizer, and configurator. Do not worry if you do not understand yet all of those components - they will be described and discussed later in the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from datasets import load_dataset\n",
    "from transformers import AutoModelForSeq2SeqLM\n",
    "from transformers import AutoTokenizer\n",
    "from transformers import GenerationConfig"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='2'></a>\n",
    "## 2 - Summarize Dialogue without Prompt Engineering\n",
    "\n",
    "In this use case, you will be generating a summary of a dialogue with the pre-trained Large Language Model (LLM) FLAN-T5 from Hugging Face. The list of available models in the Hugging Face `transformers` package can be found [here](https://huggingface.co/docs/transformers/index). \n",
    "\n",
    "Let's upload some simple dialogues from the [DialogSum](https://huggingface.co/datasets/knkarthick/dialogsum) Hugging Face dataset. This dataset contains 10,000+ dialogues with the corresponding manually labeled summaries and topics. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "409f89989ed84382bcc97c87b3db8dc9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading readme:   0%|          | 0.00/4.65k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "79ac05ea86d4466dbb0b859baa9defd3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading data:   0%|          | 0.00/11.3M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "39e6b4e3a771434fb4e6d5c0f54fe590",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading data:   0%|          | 0.00/442k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "80ab95c3aff946e6ad8fd04cad2214c2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading data:   0%|          | 0.00/1.35M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a061c9b126ae4fd982b52744839e1d15",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Generating train split: 0 examples [00:00, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.10/site-packages/datasets/download/streaming_download_manager.py:784: FutureWarning: The 'verbose' keyword in pd.read_csv is deprecated and will be removed in a future version.\n",
      "  return pd.read_csv(xopen(filepath_or_buffer, \"rb\", download_config=download_config), **kwargs)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "46ca328f1a5c4decb60f9e748abff24c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Generating validation split: 0 examples [00:00, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.10/site-packages/datasets/download/streaming_download_manager.py:784: FutureWarning: The 'verbose' keyword in pd.read_csv is deprecated and will be removed in a future version.\n",
      "  return pd.read_csv(xopen(filepath_or_buffer, \"rb\", download_config=download_config), **kwargs)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "97911bb5d9424e929a37d53fa0c60fd3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Generating test split: 0 examples [00:00, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.10/site-packages/datasets/download/streaming_download_manager.py:784: FutureWarning: The 'verbose' keyword in pd.read_csv is deprecated and will be removed in a future version.\n",
      "  return pd.read_csv(xopen(filepath_or_buffer, \"rb\", download_config=download_config), **kwargs)\n"
     ]
    }
   ],
   "source": [
    "huggingface_dataset_name = \"knkarthick/dialogsum\"\n",
    "\n",
    "dataset = load_dataset(huggingface_dataset_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "Print a couple of dialogues with their baseline summaries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---------------------------------------------------------------------------------------------------\n",
      "Example  1\n",
      "---------------------------------------------------------------------------------------------------\n",
      "INPUT DIALOGUE:\n",
      "#Person1#: What time is it, Tom?\n",
      "#Person2#: Just a minute. It's ten to nine by my watch.\n",
      "#Person1#: Is it? I had no idea it was so late. I must be off now.\n",
      "#Person2#: What's the hurry?\n",
      "#Person1#: I must catch the nine-thirty train.\n",
      "#Person2#: You've plenty of time yet. The railway station is very close. It won't take more than twenty minutes to get there.\n",
      "---------------------------------------------------------------------------------------------------\n",
      "BASELINE HUMAN SUMMARY:\n",
      "#Person1# is in a hurry to catch a train. Tom tells #Person1# there is plenty of time.\n",
      "---------------------------------------------------------------------------------------------------\n",
      "\n",
      "---------------------------------------------------------------------------------------------------\n",
      "Example  2\n",
      "---------------------------------------------------------------------------------------------------\n",
      "INPUT DIALOGUE:\n",
      "#Person1#: Have you considered upgrading your system?\n",
      "#Person2#: Yes, but I'm not sure what exactly I would need.\n",
      "#Person1#: You could consider adding a painting program to your software. It would allow you to make up your own flyers and banners for advertising.\n",
      "#Person2#: That would be a definite bonus.\n",
      "#Person1#: You might also want to upgrade your hardware because it is pretty outdated now.\n",
      "#Person2#: How can we do that?\n",
      "#Person1#: You'd probably need a faster processor, to begin with. And you also need a more powerful hard disc, more memory and a faster modem. Do you have a CD-ROM drive?\n",
      "#Person2#: No.\n",
      "#Person1#: Then you might want to add a CD-ROM drive too, because most new software programs are coming out on Cds.\n",
      "#Person2#: That sounds great. Thanks.\n",
      "---------------------------------------------------------------------------------------------------\n",
      "BASELINE HUMAN SUMMARY:\n",
      "#Person1# teaches #Person2# how to upgrade software and hardware in #Person2#'s system.\n",
      "---------------------------------------------------------------------------------------------------\n",
      "\n"
     ]
    }
   ],
   "source": [
    "example_indices = [40, 200]\n",
    "\n",
    "dash_line = '-'.join('' for x in range(100))\n",
    "\n",
    "for i, index in enumerate(example_indices):\n",
    "    print(dash_line)\n",
    "    print('Example ', i + 1)\n",
    "    print(dash_line)\n",
    "    print('INPUT DIALOGUE:')\n",
    "    print(dataset['test'][index]['dialogue'])\n",
    "    print(dash_line)\n",
    "    print('BASELINE HUMAN SUMMARY:')\n",
    "    print(dataset['test'][index]['summary'])\n",
    "    print(dash_line)\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the [FLAN-T5 model](https://huggingface.co/docs/transformers/model_doc/flan-t5), creating an instance of the `AutoModelForSeq2SeqLM` class with the `.from_pretrained()` method. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "iAYlS40Z3l-v",
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "45d6900722444196bd740dda0b1b02b2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "config.json:   0%|          | 0.00/1.40k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "48cdd064ca6440a8845928d6ee3e2029",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "pytorch_model.bin:   0%|          | 0.00/990M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f79edfba75fb42ea8600c1ef638c72ac",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "generation_config.json:   0%|          | 0.00/147 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model_name='google/flan-t5-base'\n",
    "\n",
    "model = AutoModelForSeq2SeqLM.from_pretrained(model_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sPqQA3TT3l_I",
    "tags": []
   },
   "source": [
    "To perform encoding and decoding, you need to work with text in a tokenized form. **Tokenization** is the process of splitting texts into smaller units that can be processed by the LLM models. \n",
    "\n",
    "Download the tokenizer for the FLAN-T5 model using `AutoTokenizer.from_pretrained()` method. Parameter `use_fast` switches on fast tokenizer. At this stage, there is no need to go into the details of that, but you can find the tokenizer parameters in the [documentation](https://huggingface.co/docs/transformers/v4.28.1/en/model_doc/auto#transformers.AutoTokenizer)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "sPqQA3TT3l_I",
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "88f525b17e954dddb3b8b0c6150bedb3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer_config.json:   0%|          | 0.00/2.54k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b141f6682f9b478bbc458c4038076f6a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "spiece.model:   0%|          | 0.00/792k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "235b7c9fafdd4a8dbe2ed4bda336def5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer.json:   0%|          | 0.00/2.42M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0f842d9a2d2d410e93de8ac07c393886",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "special_tokens_map.json:   0%|          | 0.00/2.20k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "Test the tokenizer encoding and decoding a simple sentence:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ENCODED SENTENCE:\n",
      "tensor([ 363,   97,   19,   34,    6, 3059,   58,    1])\n",
      "\n",
      "DECODED SENTENCE:\n",
      "What time is it, Tom?\n"
     ]
    }
   ],
   "source": [
    "sentence = \"What time is it, Tom?\"\n",
    "\n",
    "sentence_encoded = tokenizer(sentence, return_tensors='pt')\n",
    "\n",
    "sentence_decoded = tokenizer.decode(\n",
    "        sentence_encoded[\"input_ids\"][0], \n",
    "        skip_special_tokens=True\n",
    "    )\n",
    "\n",
    "print('ENCODED SENTENCE:')\n",
    "print(sentence_encoded[\"input_ids\"][0])\n",
    "print('\\nDECODED SENTENCE:')\n",
    "print(sentence_decoded)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now it's time to explore how well the base LLM summarizes a dialogue without any prompt engineering. **Prompt engineering** is an act of a human changing the **prompt** (input) to improve the response for a given task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---------------------------------------------------------------------------------------------------\n",
      "Example  1\n",
      "---------------------------------------------------------------------------------------------------\n",
      "INPUT PROMPT:\n",
      "#Person1#: What time is it, Tom?\n",
      "#Person2#: Just a minute. It's ten to nine by my watch.\n",
      "#Person1#: Is it? I had no idea it was so late. I must be off now.\n",
      "#Person2#: What's the hurry?\n",
      "#Person1#: I must catch the nine-thirty train.\n",
      "#Person2#: You've plenty of time yet. The railway station is very close. It won't take more than twenty minutes to get there.\n",
      "---------------------------------------------------------------------------------------------------\n",
      "BASELINE HUMAN SUMMARY:\n",
      "#Person1# is in a hurry to catch a train. Tom tells #Person1# there is plenty of time.\n",
      "---------------------------------------------------------------------------------------------------\n",
      "MODEL GENERATION - WITHOUT PROMPT ENGINEERING:\n",
      "Person1: It's ten to nine.\n",
      "\n",
      "---------------------------------------------------------------------------------------------------\n",
      "Example  2\n",
      "---------------------------------------------------------------------------------------------------\n",
      "INPUT PROMPT:\n",
      "#Person1#: Have you considered upgrading your system?\n",
      "#Person2#: Yes, but I'm not sure what exactly I would need.\n",
      "#Person1#: You could consider adding a painting program to your software. It would allow you to make up your own flyers and banners for advertising.\n",
      "#Person2#: That would be a definite bonus.\n",
      "#Person1#: You might also want to upgrade your hardware because it is pretty outdated now.\n",
      "#Person2#: How can we do that?\n",
      "#Person1#: You'd probably need a faster processor, to begin with. And you also need a more powerful hard disc, more memory and a faster modem. Do you have a CD-ROM drive?\n",
      "#Person2#: No.\n",
      "#Person1#: Then you might want to add a CD-ROM drive too, because most new software programs are coming out on Cds.\n",
      "#Person2#: That sounds great. Thanks.\n",
      "---------------------------------------------------------------------------------------------------\n",
      "BASELINE HUMAN SUMMARY:\n",
      "#Person1# teaches #Person2# how to upgrade software and hardware in #Person2#'s system.\n",
      "---------------------------------------------------------------------------------------------------\n",
      "MODEL GENERATION - WITHOUT PROMPT ENGINEERING:\n",
      "#Person1#: I'm thinking of upgrading my computer.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for i, index in enumerate(example_indices):\n",
    "    dialogue = dataset['test'][index]['dialogue']\n",
    "    summary = dataset['test'][index]['summary']\n",
    "    \n",
    "    inputs = tokenizer(dialogue, return_tensors='pt')\n",
    "    output = tokenizer.decode(\n",
    "        model.generate(\n",
    "            inputs[\"input_ids\"], \n",
    "            max_new_tokens=50,\n",
    "        )[0], \n",
    "        skip_special_tokens=True\n",
    "    )\n",
    "    \n",
    "    print(dash_line)\n",
    "    print('Example ', i + 1)\n",
    "    print(dash_line)\n",
    "    print(f'INPUT PROMPT:\\n{dialogue}')\n",
    "    print(dash_line)\n",
    "    print(f'BASELINE HUMAN SUMMARY:\\n{summary}')\n",
    "    print(dash_line)\n",
    "    print(f'MODEL GENERATION - WITHOUT PROMPT ENGINEERING:\\n{output}\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see that the guesses of the model make some sense, but it doesn't seem to be sure what task it is supposed to accomplish. Seems it just makes up the next sentence in the dialogue. Prompt engineering can help here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='3'></a>\n",
    "## 3 - Summarize Dialogue with an Instruction Prompt\n",
    "\n",
    "Prompt engineering is an important concept in using foundation models for text generation. You can check out [this blog](https://www.amazon.science/blog/emnlp-prompt-engineering-is-the-new-feature-engineering) from Amazon Science for a quick introduction to prompt engineering."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='3.1'></a>\n",
    "### 3.1 - Zero Shot Inference with an Instruction Prompt\n",
    "\n",
    "In order to instruct the model to perform a task - summarize a dialogue - you can take the dialogue and convert it into an instruction prompt. This is often called **zero shot inference**.  You can check out [this blog from AWS](https://aws.amazon.com/blogs/machine-learning/zero-shot-prompting-for-the-flan-t5-foundation-model-in-amazon-sagemaker-jumpstart/) for a quick description of what zero shot learning is and why it is an important concept to the LLM model.\n",
    "\n",
    "Wrap the dialogue in a descriptive instruction and see how the generated text will change:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---------------------------------------------------------------------------------------------------\n",
      "Example  1\n",
      "---------------------------------------------------------------------------------------------------\n",
      "INPUT PROMPT:\n",
      "\n",
      "Summarize the following conversation.\n",
      "\n",
      "#Person1#: What time is it, Tom?\n",
      "#Person2#: Just a minute. It's ten to nine by my watch.\n",
      "#Person1#: Is it? I had no idea it was so late. I must be off now.\n",
      "#Person2#: What's the hurry?\n",
      "#Person1#: I must catch the nine-thirty train.\n",
      "#Person2#: You've plenty of time yet. The railway station is very close. It won't take more than twenty minutes to get there.\n",
      "\n",
      "Summary:\n",
      "    \n",
      "---------------------------------------------------------------------------------------------------\n",
      "BASELINE HUMAN SUMMARY:\n",
      "#Person1# is in a hurry to catch a train. Tom tells #Person1# there is plenty of time.\n",
      "---------------------------------------------------------------------------------------------------\n",
      "MODEL GENERATION - ZERO SHOT:\n",
      "The train is about to leave.\n",
      "\n",
      "---------------------------------------------------------------------------------------------------\n",
      "Example  2\n",
      "---------------------------------------------------------------------------------------------------\n",
      "INPUT PROMPT:\n",
      "\n",
      "Summarize the following conversation.\n",
      "\n",
      "#Person1#: Have you considered upgrading your system?\n",
      "#Person2#: Yes, but I'm not sure what exactly I would need.\n",
      "#Person1#: You could consider adding a painting program to your software. It would allow you to make up your own flyers and banners for advertising.\n",
      "#Person2#: That would be a definite bonus.\n",
      "#Person1#: You might also want to upgrade your hardware because it is pretty outdated now.\n",
      "#Person2#: How can we do that?\n",
      "#Person1#: You'd probably need a faster processor, to begin with. And you also need a more powerful hard disc, more memory and a faster modem. Do you have a CD-ROM drive?\n",
      "#Person2#: No.\n",
      "#Person1#: Then you might want to add a CD-ROM drive too, because most new software programs are coming out on Cds.\n",
      "#Person2#: That sounds great. Thanks.\n",
      "\n",
      "Summary:\n",
      "    \n",
      "---------------------------------------------------------------------------------------------------\n",
      "BASELINE HUMAN SUMMARY:\n",
      "#Person1# teaches #Person2# how to upgrade software and hardware in #Person2#'s system.\n",
      "---------------------------------------------------------------------------------------------------\n",
      "MODEL GENERATION - ZERO SHOT:\n",
      "#Person1#: I'm thinking of upgrading my computer.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for i, index in enumerate(example_indices):\n",
    "    dialogue = dataset['test'][index]['dialogue']\n",
    "    summary = dataset['test'][index]['summary']\n",
    "\n",
    "    prompt = f\"\"\"\n",
    "Summarize the following conversation.\n",
    "\n",
    "{dialogue}\n",
    "\n",
    "Summary:\n",
    "    \"\"\"\n",
    "\n",
    "    # Input constructed prompt instead of the dialogue.\n",
    "    inputs = tokenizer(prompt, return_tensors='pt')\n",
    "    output = tokenizer.decode(\n",
    "        model.generate(\n",
    "            inputs[\"input_ids\"], \n",
    "            max_new_tokens=50,\n",
    "        )[0], \n",
    "        skip_special_tokens=True\n",
    "    )\n",
    "    \n",
    "    print(dash_line)\n",
    "    print('Example ', i + 1)\n",
    "    print(dash_line)\n",
    "    print(f'INPUT PROMPT:\\n{prompt}')\n",
    "    print(dash_line)\n",
    "    print(f'BASELINE HUMAN SUMMARY:\\n{summary}')\n",
    "    print(dash_line)    \n",
    "    print(f'MODEL GENERATION - ZERO SHOT:\\n{output}\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is much better! But the model still does not pick up on the nuance of the conversations though."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:**\n",
    "\n",
    "- Experiment with the `prompt` text and see how the inferences will be changed. Will the inferences change if you end the prompt with just empty string vs. `Summary: `?\n",
    "- Try to rephrase the beginning of the `prompt` text from `Summarize the following conversation.` to something different - and see how it will influence the generated output."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='3.2'></a>\n",
    "### 3.2 - Zero Shot Inference with the Prompt Template from FLAN-T5\n",
    "\n",
    "Let's use a slightly different prompt. FLAN-T5 has many prompt templates that are published for certain tasks [here](https://github.com/google-research/FLAN/tree/main/flan/v2). In the following code, you will use one of the [pre-built FLAN-T5 prompts](https://github.com/google-research/FLAN/blob/main/flan/v2/templates.py):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---------------------------------------------------------------------------------------------------\n",
      "Example  1\n",
      "---------------------------------------------------------------------------------------------------\n",
      "INPUT PROMPT:\n",
      "\n",
      "Dialogue:\n",
      "\n",
      "#Person1#: What time is it, Tom?\n",
      "#Person2#: Just a minute. It's ten to nine by my watch.\n",
      "#Person1#: Is it? I had no idea it was so late. I must be off now.\n",
      "#Person2#: What's the hurry?\n",
      "#Person1#: I must catch the nine-thirty train.\n",
      "#Person2#: You've plenty of time yet. The railway station is very close. It won't take more than twenty minutes to get there.\n",
      "\n",
      "What was going on?\n",
      "\n",
      "---------------------------------------------------------------------------------------------------\n",
      "BASELINE HUMAN SUMMARY:\n",
      "#Person1# is in a hurry to catch a train. Tom tells #Person1# there is plenty of time.\n",
      "\n",
      "---------------------------------------------------------------------------------------------------\n",
      "MODEL GENERATION - ZERO SHOT:\n",
      "Tom is late for the train.\n",
      "\n",
      "---------------------------------------------------------------------------------------------------\n",
      "Example  2\n",
      "---------------------------------------------------------------------------------------------------\n",
      "INPUT PROMPT:\n",
      "\n",
      "Dialogue:\n",
      "\n",
      "#Person1#: Have you considered upgrading your system?\n",
      "#Person2#: Yes, but I'm not sure what exactly I would need.\n",
      "#Person1#: You could consider adding a painting program to your software. It would allow you to make up your own flyers and banners for advertising.\n",
      "#Person2#: That would be a definite bonus.\n",
      "#Person1#: You might also want to upgrade your hardware because it is pretty outdated now.\n",
      "#Person2#: How can we do that?\n",
      "#Person1#: You'd probably need a faster processor, to begin with. And you also need a more powerful hard disc, more memory and a faster modem. Do you have a CD-ROM drive?\n",
      "#Person2#: No.\n",
      "#Person1#: Then you might want to add a CD-ROM drive too, because most new software programs are coming out on Cds.\n",
      "#Person2#: That sounds great. Thanks.\n",
      "\n",
      "What was going on?\n",
      "\n",
      "---------------------------------------------------------------------------------------------------\n",
      "BASELINE HUMAN SUMMARY:\n",
      "#Person1# teaches #Person2# how to upgrade software and hardware in #Person2#'s system.\n",
      "\n",
      "---------------------------------------------------------------------------------------------------\n",
      "MODEL GENERATION - ZERO SHOT:\n",
      "#Person1#: You could add a painting program to your software. #Person2#: That would be a bonus. #Person1#: You might also want to upgrade your hardware. #Person1#\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for i, index in enumerate(example_indices):\n",
    "    dialogue = dataset['test'][index]['dialogue']\n",
    "    summary = dataset['test'][index]['summary']\n",
    "        \n",
    "    prompt = f\"\"\"\n",
    "Dialogue:\n",
    "\n",
    "{dialogue}\n",
    "\n",
    "What was going on?\n",
    "\"\"\"\n",
    "\n",
    "    inputs = tokenizer(prompt, return_tensors='pt')\n",
    "    output = tokenizer.decode(\n",
    "        model.generate(\n",
    "            inputs[\"input_ids\"], \n",
    "            max_new_tokens=50,\n",
    "        )[0], \n",
    "        skip_special_tokens=True\n",
    "    )\n",
    "\n",
    "    print(dash_line)\n",
    "    print('Example ', i + 1)\n",
    "    print(dash_line)\n",
    "    print(f'INPUT PROMPT:\\n{prompt}')\n",
    "    print(dash_line)\n",
    "    print(f'BASELINE HUMAN SUMMARY:\\n{summary}\\n')\n",
    "    print(dash_line)\n",
    "    print(f'MODEL GENERATION - ZERO SHOT:\\n{output}\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that this prompt from FLAN-T5 did help a bit, but still struggles to pick up on the nuance of the conversation. This is what you will try to solve with the few shot inferencing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='4'></a>\n",
    "## 4 - Summarize Dialogue with One Shot and Few Shot Inference\n",
    "\n",
    "**One shot and few shot inference** are the practices of providing an LLM with either one or more full examples of prompt-response pairs that match your task - before your actual prompt that you want completed. This is called \"in-context learning\" and puts your model into a state that understands your specific task.  You can read more about it in [this blog from HuggingFace](https://huggingface.co/blog/few-shot-learning-gpt-neo-and-inference-api)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "<a name='4.1'></a>\n",
    "### 4.1 - One Shot Inference\n",
    "\n",
    "Let's build a function that takes a list of `example_indices_full`, generates a prompt with full examples, then at the end appends the prompt which you want the model to complete (`example_index_to_summarize`).  You will use the same FLAN-T5 prompt template from section [3.2](#3.2). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def make_prompt(example_indices_full, example_index_to_summarize):\n",
    "    prompt = ''\n",
    "    for index in example_indices_full:\n",
    "        dialogue = dataset['test'][index]['dialogue']\n",
    "        summary = dataset['test'][index]['summary']\n",
    "        \n",
    "        # The stop sequence '{summary}\\n\\n\\n' is important for FLAN-T5. Other models may have their own preferred stop sequence.\n",
    "        prompt += f\"\"\"\n",
    "Dialogue:\n",
    "\n",
    "{dialogue}\n",
    "\n",
    "What was going on?\n",
    "{summary}\n",
    "\n",
    "\n",
    "\"\"\"\n",
    "    \n",
    "    dialogue = dataset['test'][example_index_to_summarize]['dialogue']\n",
    "    \n",
    "    prompt += f\"\"\"\n",
    "Dialogue:\n",
    "\n",
    "{dialogue}\n",
    "\n",
    "What was going on?\n",
    "\"\"\"\n",
    "        \n",
    "    return prompt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "Construct the prompt to perform one shot inference:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Dialogue:\n",
      "\n",
      "#Person1#: What time is it, Tom?\n",
      "#Person2#: Just a minute. It's ten to nine by my watch.\n",
      "#Person1#: Is it? I had no idea it was so late. I must be off now.\n",
      "#Person2#: What's the hurry?\n",
      "#Person1#: I must catch the nine-thirty train.\n",
      "#Person2#: You've plenty of time yet. The railway station is very close. It won't take more than twenty minutes to get there.\n",
      "\n",
      "What was going on?\n",
      "#Person1# is in a hurry to catch a train. Tom tells #Person1# there is plenty of time.\n",
      "\n",
      "\n",
      "\n",
      "Dialogue:\n",
      "\n",
      "#Person1#: Have you considered upgrading your system?\n",
      "#Person2#: Yes, but I'm not sure what exactly I would need.\n",
      "#Person1#: You could consider adding a painting program to your software. It would allow you to make up your own flyers and banners for advertising.\n",
      "#Person2#: That would be a definite bonus.\n",
      "#Person1#: You might also want to upgrade your hardware because it is pretty outdated now.\n",
      "#Person2#: How can we do that?\n",
      "#Person1#: You'd probably need a faster processor, to begin with. And you also need a more powerful hard disc, more memory and a faster modem. Do you have a CD-ROM drive?\n",
      "#Person2#: No.\n",
      "#Person1#: Then you might want to add a CD-ROM drive too, because most new software programs are coming out on Cds.\n",
      "#Person2#: That sounds great. Thanks.\n",
      "\n",
      "What was going on?\n",
      "\n"
     ]
    }
   ],
   "source": [
    "example_indices_full = [40]\n",
    "example_index_to_summarize = 200\n",
    "\n",
    "one_shot_prompt = make_prompt(example_indices_full, example_index_to_summarize)\n",
    "\n",
    "print(one_shot_prompt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "Now pass this prompt to perform the one shot inference:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---------------------------------------------------------------------------------------------------\n",
      "BASELINE HUMAN SUMMARY:\n",
      "#Person1# teaches #Person2# how to upgrade software and hardware in #Person2#'s system.\n",
      "\n",
      "---------------------------------------------------------------------------------------------------\n",
      "MODEL GENERATION - ONE SHOT:\n",
      "#Person1 wants to upgrade his system. #Person2 wants to add a painting program to his software. #Person1 wants to add a CD-ROM drive.\n"
     ]
    }
   ],
   "source": [
    "summary = dataset['test'][example_index_to_summarize]['summary']\n",
    "\n",
    "inputs = tokenizer(one_shot_prompt, return_tensors='pt')\n",
    "output = tokenizer.decode(\n",
    "    model.generate(\n",
    "        inputs[\"input_ids\"],\n",
    "        max_new_tokens=50,\n",
    "    )[0], \n",
    "    skip_special_tokens=True\n",
    ")\n",
    "\n",
    "print(dash_line)\n",
    "print(f'BASELINE HUMAN SUMMARY:\\n{summary}\\n')\n",
    "print(dash_line)\n",
    "print(f'MODEL GENERATION - ONE SHOT:\\n{output}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "<a name='4.2'></a>\n",
    "### 4.2 - Few Shot Inference\n",
    "\n",
    "Let's explore few shot inference by adding two more full dialogue-summary pairs to your prompt."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Dialogue:\n",
      "\n",
      "#Person1#: What time is it, Tom?\n",
      "#Person2#: Just a minute. It's ten to nine by my watch.\n",
      "#Person1#: Is it? I had no idea it was so late. I must be off now.\n",
      "#Person2#: What's the hurry?\n",
      "#Person1#: I must catch the nine-thirty train.\n",
      "#Person2#: You've plenty of time yet. The railway station is very close. It won't take more than twenty minutes to get there.\n",
      "\n",
      "What was going on?\n",
      "#Person1# is in a hurry to catch a train. Tom tells #Person1# there is plenty of time.\n",
      "\n",
      "\n",
      "\n",
      "Dialogue:\n",
      "\n",
      "#Person1#: May, do you mind helping me prepare for the picnic?\n",
      "#Person2#: Sure. Have you checked the weather report?\n",
      "#Person1#: Yes. It says it will be sunny all day. No sign of rain at all. This is your father's favorite sausage. Sandwiches for you and Daniel.\n",
      "#Person2#: No, thanks Mom. I'd like some toast and chicken wings.\n",
      "#Person1#: Okay. Please take some fruit salad and crackers for me.\n",
      "#Person2#: Done. Oh, don't forget to take napkins disposable plates, cups and picnic blanket.\n",
      "#Person1#: All set. May, can you help me take all these things to the living room?\n",
      "#Person2#: Yes, madam.\n",
      "#Person1#: Ask Daniel to give you a hand?\n",
      "#Person2#: No, mom, I can manage it by myself. His help just causes more trouble.\n",
      "\n",
      "What was going on?\n",
      "Mom asks May to help to prepare for the picnic and May agrees.\n",
      "\n",
      "\n",
      "\n",
      "Dialogue:\n",
      "\n",
      "#Person1#: Hello, I bought the pendant in your shop, just before. \n",
      "#Person2#: Yes. Thank you very much. \n",
      "#Person1#: Now I come back to the hotel and try to show it to my friend, the pendant is broken, I'm afraid. \n",
      "#Person2#: Oh, is it? \n",
      "#Person1#: Would you change it to a new one? \n",
      "#Person2#: Yes, certainly. You have the receipt? \n",
      "#Person1#: Yes, I do. \n",
      "#Person2#: Then would you kindly come to our shop with the receipt by 10 o'clock? We will replace it. \n",
      "#Person1#: Thank you so much. \n",
      "\n",
      "What was going on?\n",
      "#Person1# wants to change the broken pendant in #Person2#'s shop.\n",
      "\n",
      "\n",
      "\n",
      "Dialogue:\n",
      "\n",
      "#Person1#: Have you considered upgrading your system?\n",
      "#Person2#: Yes, but I'm not sure what exactly I would need.\n",
      "#Person1#: You could consider adding a painting program to your software. It would allow you to make up your own flyers and banners for advertising.\n",
      "#Person2#: That would be a definite bonus.\n",
      "#Person1#: You might also want to upgrade your hardware because it is pretty outdated now.\n",
      "#Person2#: How can we do that?\n",
      "#Person1#: You'd probably need a faster processor, to begin with. And you also need a more powerful hard disc, more memory and a faster modem. Do you have a CD-ROM drive?\n",
      "#Person2#: No.\n",
      "#Person1#: Then you might want to add a CD-ROM drive too, because most new software programs are coming out on Cds.\n",
      "#Person2#: That sounds great. Thanks.\n",
      "\n",
      "What was going on?\n",
      "\n"
     ]
    }
   ],
   "source": [
    "example_indices_full = [40, 80, 120]\n",
    "example_index_to_summarize = 200\n",
    "\n",
    "few_shot_prompt = make_prompt(example_indices_full, example_index_to_summarize)\n",
    "\n",
    "print(few_shot_prompt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "Now pass this prompt to perform a few shot inference:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---------------------------------------------------------------------------------------------------\n",
      "BASELINE HUMAN SUMMARY:\n",
      "#Person1# teaches #Person2# how to upgrade software and hardware in #Person2#'s system.\n",
      "\n",
      "---------------------------------------------------------------------------------------------------\n",
      "MODEL GENERATION - FEW SHOT:\n",
      "#Person1 wants to upgrade his system. #Person2 wants to add a painting program to his software. #Person1 wants to upgrade his hardware.\n"
     ]
    }
   ],
   "source": [
    "summary = dataset['test'][example_index_to_summarize]['summary']\n",
    "\n",
    "inputs = tokenizer(few_shot_prompt, return_tensors='pt')\n",
    "output = tokenizer.decode(\n",
    "    model.generate(\n",
    "        inputs[\"input_ids\"],\n",
    "        max_new_tokens=50,\n",
    "    )[0], \n",
    "    skip_special_tokens=True\n",
    ")\n",
    "\n",
    "print(dash_line)\n",
    "print(f'BASELINE HUMAN SUMMARY:\\n{summary}\\n')\n",
    "print(dash_line)\n",
    "print(f'MODEL GENERATION - FEW SHOT:\\n{output}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "In this case, few shot did not provide much of an improvement over one shot inference.  And, anything above 5 or 6 shot will typically not help much, either.  Also, you need to make sure that you do not exceed the model's input-context length which, in our case, if 512 tokens.  Anything above the context length will be ignored.\n",
    "\n",
    "However, you can see that feeding in at least one full example (one shot) provides the model with more information and qualitatively improves the summary overall."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "**Exercise:**\n",
    "\n",
    "Experiment with the few shot inferencing.\n",
    "- Choose different dialogues - change the indices in the `example_indices_full` list and `example_index_to_summarize` value.\n",
    "- Change the number of shots. Be sure to stay within the model's 512 context length, however.\n",
    "\n",
    "How well does few shot inferencing work with other examples?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "<a name='5'></a>\n",
    "## 5 - Generative Configuration Parameters for Inference"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "You can change the configuration parameters of the `generate()` method to see a different output from the LLM. So far the only parameter that you have been setting was `max_new_tokens=50`, which defines the maximum number of tokens to generate. A full list of available parameters can be found in the [Hugging Face Generation documentation](https://huggingface.co/docs/transformers/v4.29.1/en/main_classes/text_generation#transformers.GenerationConfig). \n",
    "\n",
    "A convenient way of organizing the configuration parameters is to use `GenerationConfig` class. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "**Exercise:**\n",
    "\n",
    "Change the configuration parameters to investigate their influence on the output. \n",
    "\n",
    "Putting the parameter `do_sample = True`, you activate various decoding strategies which influence the next token from the probability distribution over the entire vocabulary. You can then adjust the outputs changing `temperature` and other parameters (such as `top_k` and `top_p`). \n",
    "\n",
    "Uncomment the lines in the cell below and rerun the code. Try to analyze the results. You can read some comments below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---------------------------------------------------------------------------------------------------\n",
      "MODEL GENERATION - FEW SHOT:\n",
      "#Person2 recommends adding a painting program to the software.\n",
      "---------------------------------------------------------------------------------------------------\n",
      "BASELINE HUMAN SUMMARY:\n",
      "#Person1# teaches #Person2# how to upgrade software and hardware in #Person2#'s system.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# generation_config = GenerationConfig(max_new_tokens=50)\n",
    "# generation_config = GenerationConfig(max_new_tokens=10)\n",
    "# generation_config = GenerationConfig(max_new_tokens=50, do_sample=True, temperature=0.1)\n",
    "# generation_config = GenerationConfig(max_new_tokens=50, do_sample=True, temperature=0.5)\n",
    "# generation_config = GenerationConfig(max_new_tokens=50, do_sample=True, temperature=1.0)\n",
    "\n",
    "\n",
    "# generation_config = GenerationConfig(max_new_tokens=50, do_sample=True, top_p=0.10, top_k=50)\n",
    "# generation_config = GenerationConfig(max_new_tokens=50, do_sample=True, top_p=0.50, top_k=50)\n",
    "# generation_config = GenerationConfig(max_new_tokens=50, do_sample=True, top_p=0.95, top_k=50)\n",
    "\n",
    "\n",
    "\n",
    "inputs = tokenizer(few_shot_prompt, return_tensors='pt')\n",
    "output = tokenizer.decode(\n",
    "    model.generate(\n",
    "        inputs[\"input_ids\"],\n",
    "        generation_config=generation_config,\n",
    "    )[0], \n",
    "    skip_special_tokens=True\n",
    ")\n",
    "\n",
    "print(dash_line)\n",
    "print(f'MODEL GENERATION - FEW SHOT:\\n{output}')\n",
    "print(dash_line)\n",
    "print(f'BASELINE HUMAN SUMMARY:\\n{summary}\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Comments related to the choice of the parameters in the code cell above:\n",
    "- Choosing `max_new_tokens=10` will make the output text too short, so the dialogue summary will be cut.\n",
    "- Putting `do_sample = True` and changing the temperature value you get more flexibility in the output."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, prompt engineering can take you a long way for this use case, but there are some limitations. Next, you will start to explore how you can use fine-tuning to help your LLM to understand a particular use case in better depth!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Choosing `max_new_tokens=10` will make the output text too short, so the dialogue summary will be cut.\n",
    "- Putting `do_sample = True` and changing the temperature value you get more flexibility in the output. Higher temperature indicates higher randomness.\n",
    "- Top_k sampling takes the k_{th} most probable tokens (selection from smaller sample with higher probability). \n",
    "- Top_p sampling dynamically measures the likelihood of tokens selected at each step. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "availableInstances": [
   {
    "_defaultOrder": 0,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.t3.medium",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 1,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.t3.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 2,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.t3.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 3,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.t3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 4,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 5,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 6,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 7,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 8,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 9,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 10,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 11,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 12,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5d.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 13,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5d.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 14,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5d.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 15,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5d.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 16,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5d.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 17,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5d.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 18,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5d.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 19,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 20,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": true,
    "memoryGiB": 0,
    "name": "ml.geospatial.interactive",
    "supportedImageNames": [
     "sagemaker-geospatial-v1-0"
    ],
    "vcpuNum": 0
   },
   {
    "_defaultOrder": 21,
    "_isFastLaunch": true,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.c5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 22,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.c5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 23,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.c5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 24,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.c5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 25,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 72,
    "name": "ml.c5.9xlarge",
    "vcpuNum": 36
   },
   {
    "_defaultOrder": 26,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 96,
    "name": "ml.c5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 27,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 144,
    "name": "ml.c5.18xlarge",
    "vcpuNum": 72
   },
   {
    "_defaultOrder": 28,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.c5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 29,
    "_isFastLaunch": true,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g4dn.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 30,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g4dn.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 31,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g4dn.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 32,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g4dn.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 33,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g4dn.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 34,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g4dn.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 35,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 61,
    "name": "ml.p3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 36,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 244,
    "name": "ml.p3.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 37,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 488,
    "name": "ml.p3.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 38,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.p3dn.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 39,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.r5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 40,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.r5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 41,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.r5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 42,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.r5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 43,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.r5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 44,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.r5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 45,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.r5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 46,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.r5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 47,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 48,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 49,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 50,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 51,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 52,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 53,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.g5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 54,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.g5.48xlarge",
    "vcpuNum": 192
   },
   {
    "_defaultOrder": 55,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 56,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4de.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 57,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.trn1.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 58,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.trn1.32xlarge",
    "vcpuNum": 128
   },
   {
    "_defaultOrder": 59,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.trn1n.32xlarge",
    "vcpuNum": 128
   }
  ],
  "instance_type": "ml.m5.2xlarge",
  "kernelspec": {
   "display_name": "Python 3 (Data Science 3.0)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/sagemaker-data-science-310-v1"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}