colab.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "CgxobX5hwzbh"
      },
      "outputs": [],
      "source": [
        "# Installs Unsloth, Xformers (Flash Attention) and all other packages\n",
        "!pip install \"unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git\"\n",
        "!pip install --no-deps \"xformers<0.0.27\" \"trl<0.9.0\" peft accelerate bitsandbytes"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8VLYYB1Bwzbi"
      },
      "source": [
        "### Background\n",
        "\n",
        "I saw a reel on Instagram in which an AI enthusiast created an AI clone of himself to talk to his girlfriend (Certainly, I won't do that... xd) using [RAG](https://youtu.be/YVWxbHJakgg?feature=shared) Retrieval Augmented Generation and the Chat GPT-3.5 turbo API. It kind of worked, but it had major privacy issues. Sending personal chats to Chat GPT could potentially result in those chats being used by OpenAI to train its model. This led me to think, what if I fine-tuned a pre-existing model like Llama or Mixtral on my personal WhatsApp chat history? It would be cool to have a model that can talk like me as I do on WhatsApp, primarily in Hinglish (Hindi + English).\n",
        "\n",
        "[Fine-tuning a large language model (LLM)](https://youtu.be/YVWxbHJakgg?feature=shared) requires a good understanding of LLMs in general. The major challenge with fine-tuning big models, such as a 7B parameter model, is that it requires a minimum of 32GB of GPU RAM, which is costly and not available in free-tier GPU compute services like Colab or Kaggle. So, I had to find a way around this limitation.\n",
        "\n",
        "Another important aspect is that the fine-tuning results heavily depend on the quality and size of the dataset used. Converting raw WhatsApp chat data into a usable dataset is challenging but worth pursuing.\n",
        "\n",
        "Let's see how it looks in reality and how it's being carried out."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IBoJ_Xw7wzbl"
      },
      "source": [
        "### Important\n",
        "\n",
        "We are using a free instance of Google Colab to fine-tune our model`(Llama3)`, making it **totally free**.\n",
        "\n",
        "For chatting with our fine-tuned model, we will use [Ollama](https://ollama.com/) locally, which is very lightweight and requires only **8GB** of free RAM in your laptop/PC and works without any **GPU** support.\n",
        "\n",
        "**Keep in mind that your chat data is completely safe; it is not being sent to anyone.**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vITh0KVJ10qX"
      },
      "source": [
        "\n",
        "### Data Prep\n",
        "To extract chat history from your WhatsApp chats, follow these steps:\n",
        "\n",
        "1. Open your WhatsApp application.\n",
        "2. Go to the chat from which you want to extract the chat history.\n",
        "3. Click on the three dots in the top right corner of the screen.\n",
        "4. Click on `More` then click on `Export Chat`.\n",
        "5. Select `Without media`.\n",
        "6. Save it locally or send it to saved messages on Telegram so you can later download it on your Telegram desktop.\n",
        "7. Repeat these steps for all of your chats. The more chat data you have, the better the results will be.\n",
        "\n",
        "It will generate `.zip file`. You don't have to extract it.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TqvPGRZDwzbl"
      },
      "source": [
        "#### Upload Exported Chat files to Colab runtime\n",
        "Now, locate your exported chat zip files and upload them to the Colab runtime. Follow these steps to upload files to Google Colab:\n",
        "\n",
        "1. Click on the Files icon on the left side of the screen (as shown in the image attached below).\n",
        "2. Click on the upload button. It will open the File Explorer. Choose the exported chat zip files (you can select multiple files at once).\n",
        "    * Wait until your files are uploaded. The upload process bar will display at the bottom left corner of the screen.\n",
        "    * Once files are uploaded successfully, they will appear in the Files tab of Google Colab.\n",
        "<img src=\"https://github.com/Eviltr0N/Make-AI-Clone-of-Yourself/raw/main/img/file_download.png\">\n",
        "\n",
        "##### Keep in Mind:\n",
        "\n",
        "* Export chat history only for meaningful conversations. Before exporting a chat, consider whether it adds value to the data or if it is just a short conversation.\n",
        "* If you think you don’t want the AI to learn from a specific chat, then don’t export it.\n",
        "* Currently, it only supports individual chats, so please do not export group chats.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O5JN21BSwzbm"
      },
      "source": [
        "### Data Filtering\n",
        "\n",
        "The exported data contains many irregularities such as `<Media omitted>`, `This message was deleted` and timestamps of messages. We need to remove these and convert the whole chat history into a `Prompt: Response` format so it can be used to fine-tune the model. To extract messages from the data, I used `regex`. Additionally, I filtered out any links and emails from the chat data for obvious privacy reasons.\n",
        "\n",
        "**Now, please edit the below list of `filler_words`**. These words may vary from person to person. Some examples are `Ok`, `Yup`, `Hmm`, `Han`. We need to remove these from the dataset because if we don't, the fine-tuned model will primarily learn these words and, in most cases, respond with them. For example, if you ask, \"Where are you going?\" the model might respond with something like \"Ok\" or \"Hmm.\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "44cyft5Jwzbm"
      },
      "source": [
        "To run this, press \"*Runtime*\" and press \"*Run all*\" on a **free** Tesla T4 Google Colab instance!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "zPGexEG7wzbm"
      },
      "outputs": [],
      "source": [
        "filler_words = [\"Ok\", \"Okay\", \"Yup\", \"Hmm\"]\n",
        "# Add or remove words from this list based on your personal usage.\n",
        "\n",
        "chat_dir = \"./\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "BdgsnIpGwzbn"
      },
      "outputs": [],
      "source": [
        "import re\n",
        "import os\n",
        "import shutil\n",
        "import csv"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "-PvTvhPVwzbn"
      },
      "outputs": [],
      "source": [
        "class Wh_Chat_Processor:\n",
        "    def __init__(self):\n",
        "        pass\n",
        "    def open_chat_file(self, dir,filename):\n",
        "        self.sender_name = filename.replace(\"WhatsApp Chat with \", \"\").replace(\".txt\", \"\")\n",
        "        with open(os.path.join(dir,filename)) as f:\n",
        "            chat_text = f.read()\n",
        "        return chat_text\n",
        "\n",
        "    def msg_filter_basic(self, chat_text):\n",
        "        filtered = []\n",
        "        pt = r' - ([^:]+): (.*?)(?=\\d{1,2}/\\d{1,2}/\\d{2,4}, \\d{1,2}:\\d{2}\\s*(?:AM|PM|am|pm)? - |$)'\n",
        "        msgs = re.findall(pt, chat_text, re.DOTALL)\n",
        "        for msg in msgs:\n",
        "            line = msg[1]\n",
        "            wh_default_filter = \"Tap to learn more.\" in line or \"<Media omitted>\" in line\n",
        "            website_filter = \"https://\" in line or \"http://\" in line\n",
        "            mail_filter = \"@gmail.com\" in line\n",
        "            deleted_msg_filter = \"This message was deleted\" in line or \"You deleted this message\" in line or \"<This message was edited>\" in line or \"(file attached)\" in line\n",
        "\n",
        "            if not (wh_default_filter or website_filter or mail_filter or deleted_msg_filter):\n",
        "                    filtered.append(msg)\n",
        "        return filtered\n",
        "\n",
        "    def process_chat(self, chat_data):\n",
        "        merged_lines = []\n",
        "        current_sender = None\n",
        "        current_message = {}\n",
        "        for line in chat_data:\n",
        "            if not line:\n",
        "                continue\n",
        "            parts = line\n",
        "            if len(parts) == 2:\n",
        "                sender, message = parts\n",
        "                if current_sender is None:\n",
        "                    current_sender = sender\n",
        "                    current_message[current_sender] = [message.strip()]\n",
        "                elif sender == current_sender:\n",
        "                    current_message[current_sender].append(message.strip())\n",
        "                else:\n",
        "                    merged_lines.append(current_message)\n",
        "                    current_sender = sender\n",
        "                    current_message = {current_sender: [message.strip()]}\n",
        "            else:\n",
        "                if current_sender:\n",
        "                    current_message[current_sender][-1] += \" \" + line.strip()\n",
        "        if current_sender:\n",
        "            merged_lines.append(current_message)\n",
        "        keys = set()\n",
        "        for line in merged_lines:\n",
        "            # print(line)\n",
        "            for key in line.keys():\n",
        "                if key != self.sender_name:\n",
        "                    keys.add(key)\n",
        "        self.my_name = list(keys)[0]\n",
        "        print(list(keys))\n",
        "        return merged_lines\n",
        "\n",
        "    def advance_filter(self, merged_chat_data):\n",
        "        filtered_data=[]\n",
        "        sender = \"\"\n",
        "        me = \"\"\n",
        "        chk = 1\n",
        "        CD = merged_chat_data\n",
        "        for ind, x in enumerate(CD):\n",
        "            if x.get(self.sender_name) != None :\n",
        "                if len(x[self.sender_name]) == 1 and ( x[self.sender_name][0] in filler_words or len(x[self.sender_name][0]) ==1 ):\n",
        "                    continue\n",
        "                if len(CD[ind][self.sender_name]) > 1:\n",
        "                    for y in range(0,len(CD[ind][self.sender_name])):\n",
        "                        if y+1 != len(CD[ind][self.sender_name]):\n",
        "                            sender += CD[ind][self.sender_name][y] + \"\\n\"\n",
        "                        else:\n",
        "                            sender += CD[ind][self.sender_name][y]\n",
        "                else:\n",
        "                    sender += CD[ind][self.sender_name][0]\n",
        "            elif x.get(self.my_name) != None and len(sender) > 1:\n",
        "                if len(CD[ind][self.my_name]) > 1:\n",
        "                    for y in range(0,len(CD[ind][self.my_name])):\n",
        "                        if y+1 != len(CD[ind][self.my_name]):\n",
        "                            me += CD[ind][self.my_name][y] + \"\\n\"\n",
        "                        else:\n",
        "                            me += CD[ind][self.my_name][y]\n",
        "                else:\n",
        "                    me += CD[ind][self.my_name][0]\n",
        "            else:\n",
        "                continue\n",
        "            if chk ==1:\n",
        "                chk+=1\n",
        "            elif chk ==2:\n",
        "                filtered_data.append([sender, me])\n",
        "                sender = \"\"\n",
        "                me=\"\"\n",
        "                chk=1\n",
        "            else:\n",
        "                pass\n",
        "        return filtered_data\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 32,
      "metadata": {
        "id": "0DVwbFW1wzbo"
      },
      "outputs": [],
      "source": [
        "with open(\"all_chat_data.csv\", \"w\") as f:\n",
        "    f.write(\"Prompt,Response\"+ \"\\n\")\n",
        "\n",
        "for file in os.listdir(os.path.join(chat_dir)):\n",
        "    if file.endswith('.zip'):\n",
        "        full_path = os.path.join(chat_dir, file)\n",
        "        shutil.unpack_archive(full_path, chat_dir)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5f7ieid2wzbo"
      },
      "outputs": [],
      "source": [
        "for file in os.listdir(os.path.join(chat_dir)):\n",
        "    processor = Wh_Chat_Processor()\n",
        "    if file.endswith('.txt'):\n",
        "        print(\"Processing: \",file)\n",
        "        chat_d = processor.open_chat_file(chat_dir,file)\n",
        "        basic_f = processor.msg_filter_basic(chat_d)\n",
        "        chat_ps = processor.process_chat(basic_f)\n",
        "        filtered_data = processor.advance_filter(chat_ps)\n",
        "        with open(\"all_chat_data.csv\", \"a\") as f:\n",
        "            csv_writer = csv.writer(f)\n",
        "            for row in filtered_data:\n",
        "                csv_writer.writerow(row)\n",
        "print(\"Successfully Processed all the chats... Generated CSV File of chats is saved in Current directory with the name 'all_chat_data.csv'\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c7FUCDIrwzbo"
      },
      "source": [
        "### Model Fine-Tuning\n",
        "As we discussed earlier, fine-tuning a 7B parameter model with just 16GB of RAM is not possible. To achieve this, we will use a technique known as [Quantization](https://huggingface.co/docs/optimum/en/concept_guides/quantization). Specifically, we will use 4-bit quantization.\n",
        "\n",
        "I am using [Unsloth](https://github.com/unslothai/unsloth) for the rest of the processes, such as quantization and training the model. Unsloth has very good documentation and requires less VRAM to fine-tune the model."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IqM-T1RTzY6C"
      },
      "source": [
        "Check out - [Unsloth's Github](https://github.com/unslothai/unsloth)\n",
        "\n",
        "This notebook uses the `Llama-3` format for conversation style finetunes. We use [Open Assistant conversations](https://huggingface.co/datasets/philschmid/guanaco-sharegpt-style) in ShareGPT style."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r2v_X2fA0Df5"
      },
      "source": [
        "* Unsloth support Llama, Mistral, Phi-3, Gemma, Yi, DeepSeek, Qwen, TinyLlama, Vicuna, Open Hermes etc\n",
        "* Unsloth support 16bit LoRA or 4bit QLoRA. Both 2x faster.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "-jDsHAJLwzbp"
      },
      "outputs": [],
      "source": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iSvaRuHpwzbp"
      },
      "source": [
        "For finetuning I am using **`Llama3` 8B Instruct** as our base model, you can use other models such as `Mixtral` and `Gemma`. I have traied Mixtral also but it dosent perform as good as `Llama3`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "QmUBVEnvCDJv"
      },
      "outputs": [],
      "source": [
        "from unsloth import FastLanguageModel\n",
        "import torch\n",
        "max_seq_length = 2048 # Choose any! We auto support RoPE Scaling internally!\n",
        "dtype = None # None for auto detection. Float16 for Tesla T4, V100, Bfloat16 for Ampere+\n",
        "load_in_4bit = True # Use 4bit quantization to reduce memory usage. Can be False.\n",
        "\n",
        "# 4bit pre quantized models we support for 4x faster downloading + no OOMs.\n",
        "fourbit_models = [\n",
        "    \"unsloth/mistral-7b-v0.3-bnb-4bit\",      # New Mistral v3 2x faster!\n",
        "    \"unsloth/mistral-7b-instruct-v0.3-bnb-4bit\",\n",
        "    \"unsloth/llama-3-8b-bnb-4bit\",           # Llama-3 15 trillion tokens model 2x faster!\n",
        "    \"unsloth/llama-3-8b-Instruct-bnb-4bit\",\n",
        "    \"unsloth/llama-3-70b-bnb-4bit\",\n",
        "    \"unsloth/Phi-3-mini-4k-instruct\",        # Phi-3 2x faster!\n",
        "    \"unsloth/Phi-3-medium-4k-instruct\",\n",
        "    \"unsloth/mistral-7b-bnb-4bit\",\n",
        "    \"unsloth/gemma-7b-bnb-4bit\",             # Gemma 2.2x faster!\n",
        "] # More models at https://huggingface.co/unsloth\n",
        "\n",
        "model, tokenizer = FastLanguageModel.from_pretrained(\n",
        "    model_name = \"unsloth/llama-3-8b-Instruct-bnb-4bit\", # Choose ANY! eg teknium/OpenHermes-2.5-Mistral-7B\n",
        "    max_seq_length = max_seq_length,\n",
        "    dtype = dtype,\n",
        "    load_in_4bit = load_in_4bit,\n",
        "    # token = \"hf_...\", # use one if using gated models like meta-llama/Llama-2-7b-hf\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SXd9bTZd1aaL"
      },
      "source": [
        "We now add LoRA adapters so we only need to update 1 to 10% of all parameters!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6bZsfBuZDeCL"
      },
      "outputs": [],
      "source": [
        "model = FastLanguageModel.get_peft_model(\n",
        "    model,\n",
        "    r = 16, # Choose any number > 0 ! Suggested 8, 16, 32, 64, 128\n",
        "    target_modules = [\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\",\n",
        "                      \"gate_proj\", \"up_proj\", \"down_proj\",],\n",
        "    lora_alpha = 16,\n",
        "    lora_dropout = 0, # Supports any, but = 0 is optimized\n",
        "    bias = \"none\",    # Supports any, but = \"none\" is optimized\n",
        "    # [NEW] \"unsloth\" uses 30% less VRAM, fits 2x larger batch sizes!\n",
        "    use_gradient_checkpointing = \"unsloth\", # True or \"unsloth\" for very long context\n",
        "    random_state = 3407,\n",
        "    use_rslora = False,  # We support rank stabilized LoRA\n",
        "    loftq_config = None, # And LoftQ\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZaOf7N7twzbq"
      },
      "source": [
        "Let's prepare dataset from the filtered Whatsapp Chat data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 25,
      "metadata": {
        "id": "1AhDjFQSRq6v"
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "from datasets import Dataset, load_dataset\n",
        "from unsloth.chat_templates import get_chat_template"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 26,
      "metadata": {
        "id": "oSS0Znn4wzbq"
      },
      "outputs": [],
      "source": [
        "tokenizer = get_chat_template(\n",
        "    tokenizer,\n",
        "    chat_template=\"llama-3\",  # Use the desired chat template\n",
        "    mapping={\"role\": \"from\", \"content\": \"value\", \"user\": \"human\", \"assistant\": \"gpt\"}\n",
        ")\n",
        "\n",
        "# Define the formatting function\n",
        "def formatting_prompts_func(examples):\n",
        "    convos = examples[\"conversations\"]\n",
        "    texts = [tokenizer.apply_chat_template(convo, tokenize=False, add_generation_prompt=False) for convo in convos]\n",
        "    return {\"text\": texts}\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 40,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 49,
          "referenced_widgets": [
            "1d777ec6fd994cd0a10b8a5ee5a9ecce",
            "fb53fae9bb8f43668e58817861a99830",
            "39eb48a59bbc43459dd403c5361e7277",
            "8f43afa148464357ac7084cfc82b54f5",
            "e442477c71da43c3a55b494f01c9adf7",
            "ea483ec87d2145dc8d94eaaa3767c826",
            "e8ef5c1da74f4603845780dcc02f64da",
            "1d018226491848a593d2fc0f983528f0",
            "accf1601f86d4ff785f39262f82c527a",
            "f8a76f16cc6243ee9f4063b45cf4ad59",
            "eb776dfe78db49a4802f03b41ad2e15e"
          ]
        },
        "id": "wFP3QWaViVi8",
        "outputId": "c8f520b9-7e75-4744-add4-18bf1dbdf4bc"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "1d777ec6fd994cd0a10b8a5ee5a9ecce",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Map:   0%|          | 0/2157 [00:00<?, ? examples/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "df = pd.read_csv(\"all_chat_data.csv\")\n",
        "conversations = []\n",
        "for _, row in df.iterrows():\n",
        "    try:\n",
        "        conversation = [\n",
        "            {'from': 'human', 'value': str(row['Prompt'])},\n",
        "            {'from': 'assistant', 'value': str(row['Response'])}\n",
        "        ]\n",
        "        conversations.append(conversation)\n",
        "    except:\n",
        "        print(_ , row)\n",
        "\n",
        "\n",
        "dataset = Dataset.from_dict({\"conversations\": conversations})\n",
        "dataset = dataset.map(formatting_prompts_func, batched=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EdsJiFbRwzbq"
      },
      "source": [
        "Let's see how the `Llama-3` format works by printing the 5th element"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yuCMQuonRq0n"
      },
      "outputs": [],
      "source": [
        "dataset[5][\"conversations\"]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oJQ66lSywzbr"
      },
      "outputs": [],
      "source": [
        "print(dataset[5][\"text\"])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "idAEIeSQ3xdS"
      },
      "source": [
        "\n",
        "### Train the model\n",
        "Now let's use Huggingface TRL's `SFTTrainer`! More docs here: [TRL SFT docs](https://huggingface.co/docs/trl/sft_trainer).\n",
        "* I am doing `1 epochs` to speed things up, but you can set `num_train_epochs` to 2 or 3, Just experiment with it.\n",
        "* If you have large dataset then just go for `1 full epoch`. Do not do more than 3 or 4 epoch if the `training loss` is not **decreasing**."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "95_Nn-89DhsL"
      },
      "outputs": [],
      "source": [
        "from trl import SFTTrainer\n",
        "from transformers import TrainingArguments\n",
        "from unsloth import is_bfloat16_supported\n",
        "\n",
        "trainer = SFTTrainer(\n",
        "    model = model,\n",
        "    tokenizer = tokenizer,\n",
        "    train_dataset = dataset,\n",
        "    dataset_text_field = \"text\",\n",
        "    max_seq_length = max_seq_length,\n",
        "    dataset_num_proc = 2,\n",
        "    packing = False, # Can make training 5x faster for short sequences.\n",
        "    args = TrainingArguments(\n",
        "        per_device_train_batch_size = 2,\n",
        "        gradient_accumulation_steps = 4,\n",
        "        warmup_steps = 5,\n",
        "        num_train_epochs=1,\n",
        "        learning_rate = 2e-4,\n",
        "        fp16 = not is_bfloat16_supported(),\n",
        "        bf16 = is_bfloat16_supported(),\n",
        "        logging_steps = 1,\n",
        "        optim = \"adamw_8bit\",\n",
        "        weight_decay = 0.01,\n",
        "        lr_scheduler_type = \"linear\",\n",
        "        seed = 3407,\n",
        "        output_dir = \"outputs\",\n",
        "    ),\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yqxqAZ7KJ4oL"
      },
      "outputs": [],
      "source": [
        "trainer_stats = trainer.train()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ekOmTR1hSNcr"
      },
      "source": [
        "<a name=\"Inference\"></a>\n",
        "### Inference\n",
        "Let's run the model! Since we're using `Llama-3`, use `apply_chat_template` with `add_generation_prompt` set to `True` for inference."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 50,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kR3gIAX-SM2q",
        "outputId": "00c0beb8-e397-46f8-ac82-12b91e1eb31d"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "<|begin_of_text|><|start_header_id|>user<|end_header_id|>\n",
            "\n",
            "Pagal ho gya hai kya<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n",
            "\n",
            "Ha<|eot_id|>\n"
          ]
        }
      ],
      "source": [
        "from unsloth.chat_templates import get_chat_template\n",
        "from transformers import TextStreamer\n",
        "\n",
        "tokenizer = get_chat_template(\n",
        "    tokenizer,\n",
        "    chat_template = \"llama-3\", # Supports zephyr, chatml, mistral, llama, alpaca, vicuna, vicuna_old, unsloth\n",
        "    mapping = {\"role\" : \"from\", \"content\" : \"value\", \"user\" : \"human\", \"assistant\" : \"gpt\"}, # ShareGPT style\n",
        ")\n",
        "text_streamer = TextStreamer(tokenizer)\n",
        "FastLanguageModel.for_inference(model) # Enable native 2x faster inference\n",
        "\n",
        "messages = [\n",
        "    {\"from\": \"human\", \"value\": \"Pagal ho gya hai kya\"},\n",
        "]\n",
        "inputs = tokenizer.apply_chat_template(\n",
        "    messages,\n",
        "    tokenize = True,\n",
        "    add_generation_prompt = True, # Must add for generation\n",
        "    return_tensors = \"pt\",\n",
        ").to(\"cuda\")\n",
        "\n",
        "output = model.generate(input_ids = inputs, streamer = text_streamer, max_new_tokens = 128, use_cache = True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CrSvZObor0lY"
      },
      "source": [
        " You can also use a `TextStreamer` for continuous inference - so you can see the generation token by token, instead of waiting the whole time!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uMuVrWbjAzhc"
      },
      "source": [
        "<a name=\"Save\"></a>\n",
        "### Saving finetuned model (Lora Adapters Only)\n",
        "To save the final model as LoRA adapters, either use Huggingface's `push_to_hub` for an online save or `save_pretrained` for a local save.\n",
        "\n",
        "**[NOTE]** This ONLY saves the LoRA adapters, and not the full model. To save to 16bit or GGUF, scroll down!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 51,
      "metadata": {
        "id": "upcOlWe7A1vc"
      },
      "outputs": [],
      "source": [
        "model.save_pretrained(\"lora_model\") # Local saving\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TCv4vXHd61i7"
      },
      "source": [
        "### GGUF Conversion (For Ollama)\n",
        "**To use our finetuned model in our PC/laptop we will use [Ollama](https://ollama.com/)**. To use this model with  `Ollama` We have to save the model in `GGUF` Format.\n",
        "\n",
        "Unsloth allow all methods like `q4_k_m`. Use `save_pretrained_gguf` for local saving and `push_to_hub_gguf` for uploading to HF.\n",
        "\n",
        "Some supported quant methods (full list on Unsloth [Wiki page](https://github.com/unslothai/unsloth/wiki#gguf-quantization-options)):\n",
        "    * `q8_0` - Fast conversion. High resource use, but generally acceptable.\n",
        "    * `q4_k_m` - Recommended. Uses Q6_K for half of the attention.wv and feed_forward.w2 tensors, else Q4_K.\n",
        "    * `q5_k_m` - Recommended. Uses Q6_K for half of the attention.wv and feed_forward.w2 tensors, else Q5_K"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "FqfebeAdT073"
      },
      "outputs": [],
      "source": [
        "# Save to 8bit Q8_0\n",
        "if True: model.save_pretrained_gguf(\"model\", tokenizer,)\n",
        "if False: model.push_to_hub_gguf(\"hf/model\", tokenizer, token = \"\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bDp0zNpwe6U_"
      },
      "source": [
        "Once the fine-tuning is complete, the model `unsloth.Q8_0.gguf` is saved in the `models/` folder. You need to download this `GGUF` file and `Modelfile` too. You can directly download it by locating the file in the files section of Google Colab, or you can copy this file to your Google Drive to download it from there. If your internet connection is slow like mine, the Google Drive method is best because the file is large (approximately 8GB). Here are the steps for both methods."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "E6GXCfCxwzbv"
      },
      "source": [
        "### Direct Download via Colab\n",
        "1. Click on the files section in Google Colab.\n",
        "2. Locate the models folder. Then expand it by clicking on the arrow located to the left of the folder name.\n",
        "3. Choose the file `unsloth.Q8_0.gguf` & `Modelfile`, then hover the mouse cursor over the filename.\n",
        "4. Click on the three dots, then select Download.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "![Download_IMG](https://github.com/Eviltr0N/Make-AI-Clone-of-Yourself/raw/main/img/file_download.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w2T3YwrQwzbv"
      },
      "source": [
        "### Download via Google Drive\n",
        "* Before using this method, make sure you have 8GB of free space left in your Google Drive. Otherwise, this will not work.\n",
        "1. Run the below cell to mount your drive with your Colab account."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-zZMNG20wzbv"
      },
      "outputs": [],
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VQ09Nt60wzbv"
      },
      "source": [
        "* Run this cell to copy the model into your Google Drive."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YVSdA_1Ywzbv"
      },
      "outputs": [],
      "source": [
        "!mkdir /content/drive/MyDrive/finetuned_model\n",
        "!cp /content/model/unsloth.Q8_0.gguf /content/drive/MyDrive/finetuned_model/\n",
        "!cp /content/model/Modelfile /content/drive/MyDrive/finetuned_model/"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CZAEERK4Tj1C"
      },
      "source": [
        "### Using Finetuned Model With Ollama and Whatsapp\n",
        "Now follow this guide on my [Github](https://github.com/Eviltr0N/Make-AI-Clone-of-Yourself?tab=readme-ov-file#loading-the-model-into-ollama) to chat with Your finetuned model.\n",
        "[Here](https://github.com/Eviltr0N/Make-AI-Clone-of-Yourself?tab=readme-ov-file#loading-the-model-into-ollama)  "
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "gpuType": "T4",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "1d018226491848a593d2fc0f983528f0": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "1d777ec6fd994cd0a10b8a5ee5a9ecce": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_fb53fae9bb8f43668e58817861a99830",
              "IPY_MODEL_39eb48a59bbc43459dd403c5361e7277",
              "IPY_MODEL_8f43afa148464357ac7084cfc82b54f5"
            ],
            "layout": "IPY_MODEL_e442477c71da43c3a55b494f01c9adf7"
          }
        },
        "39eb48a59bbc43459dd403c5361e7277": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_1d018226491848a593d2fc0f983528f0",
            "max": 2157,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_accf1601f86d4ff785f39262f82c527a",
            "value": 2157
          }
        },
        "8f43afa148464357ac7084cfc82b54f5": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_f8a76f16cc6243ee9f4063b45cf4ad59",
            "placeholder": "​",
            "style": "IPY_MODEL_eb776dfe78db49a4802f03b41ad2e15e",
            "value": " 2157/2157 [00:00&lt;00:00, 9866.30 examples/s]"
          }
        },
        "accf1601f86d4ff785f39262f82c527a": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "e442477c71da43c3a55b494f01c9adf7": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e8ef5c1da74f4603845780dcc02f64da": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "ea483ec87d2145dc8d94eaaa3767c826": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "eb776dfe78db49a4802f03b41ad2e15e": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "f8a76f16cc6243ee9f4063b45cf4ad59": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "fb53fae9bb8f43668e58817861a99830": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_ea483ec87d2145dc8d94eaaa3767c826",
            "placeholder": "​",
            "style": "IPY_MODEL_e8ef5c1da74f4603845780dcc02f64da",
            "value": "Map: 100%"
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}