tokenizer_example.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a0144852-8037-4f58-8d6a-00250de60203",
   "metadata": {},
   "source": [
    "To get started we will use the `transformers` library fromm Huggingface.  To access the Llama tokenizer you will need Hugginface token which can be obtained after logging in to the Huggingface website.  Next click on your profile icon at the top right and select `Access Tokens'.  This will direct you to a page where you can generate a Huggingface token.  Please use this token instead of <YOUR_TOKEN> in the code below.  You will also need to go to the \"meta-llama/Llama-3.2-1B\" model page and request access as the Llama models are currently gated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "25a2ad96-dc6e-472e-89c5-89b32e20b763",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AutoTokenizer\n",
    "hf_token = \"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\"\n",
    "#hf_token = <YOUR_TOKEN> \n",
    "model_id = \"meta-llama/Llama-3.2-1B\"\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_id, token=hf_token, local_files_only=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "056133f7-e856-4d6c-8e3e-fe952f860050",
   "metadata": {},
   "source": [
    "Produce token IDs from some string of text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "c22dd0af-c629-4899-af86-7906772403ff",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[128000,\n",
       " 40,\n",
       " 2846,\n",
       " 2307,\n",
       " 12304,\n",
       " 311,\n",
       " 5249,\n",
       " 452,\n",
       " 23174,\n",
       " 596,\n",
       " 11964,\n",
       " 344,\n",
       " 15701,\n",
       " 63155,\n",
       " 90246,\n",
       " 7561,\n",
       " 12,\n",
       " 4690,\n",
       " 0]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text = \"I'm super excited to join Nesa's Equivariant Encryption Decrypt-a-thon!\"\n",
    "token_ids = tokenizer.encode(text)\n",
    "token_ids"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5e5a525-4750-4b25-a319-26647e954dbf",
   "metadata": {},
   "source": [
    "You can see the token IDs of each token from the text.  We can inspect the text token associated with individual IDs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "6b46b1c1-aeae-40b7-ae29-f71142a553a1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "' excited'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.decode(12304)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9cc633af-4413-4863-a714-2c72d583b5a2",
   "metadata": {},
   "source": [
    "We can also confirm that the list of token IDs reproduces the original text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "f7eca38b-cec4-4434-80d7-8728a9efc145",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"<|begin_of_text|>I'm super excited to join Nesa's Equivariant Encryption Decrypt-a-thon!\""
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.decode(token_ids)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8fba6e77-6525-4229-975e-29abfe99c855",
   "metadata": {},
   "source": [
    "Note that ID 128000 corresponds to a special token \"<|begin_of_text|>\".  Tokenizers, particularly Llama's tokenizer, often have special tokens to help with text foramating, etc."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8537788c-7683-4e32-a57a-2fff25d829ec",
   "metadata": {},
   "source": [
    "For Nesa's daily contest we expect the submission to be in the following format:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "ca0894df-1312-4ae1-8efe-2eff6d3ce5c9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{128000: '<|begin_of_text|>',\n",
       " 40: 'I',\n",
       " 2846: \"'m\",\n",
       " 2307: ' super',\n",
       " 12304: ' excited',\n",
       " 311: ' to',\n",
       " 5249: ' join',\n",
       " 452: ' N',\n",
       " 23174: 'esa',\n",
       " 596: \"'s\",\n",
       " 11964: ' Equ',\n",
       " 344: 'iv',\n",
       " 15701: 'ariant',\n",
       " 63155: ' Encryption',\n",
       " 90246: ' Decrypt',\n",
       " 7561: '-a',\n",
       " 12: '-',\n",
       " 4690: 'thon',\n",
       " 0: '!'}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "{t: tokenizer.decode(t) for t in token_ids}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d87149b1-d495-46ce-82cf-a32006de7547",
   "metadata": {},
   "source": [
    "For the above example, we took a short cut by using the tokenizer itself to generate all the possible answers from the `text` example given.  However, in the actual contest, the tokenizer will not be known.  It is up to you to figure out a method to find as many `token ID : token` pairs as possible."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "43f5a0d2-27a4-4c67-8bbd-4fafaf87d4d9",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}