llama_cpp_test.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Test the llama.cpp library\n",
    "  \n",
    "This notebook test drives the `llama.cpp` library for running local LLMs and compares to resuls from `mlx_test.ipynb`.\n",
    "  \n",
    "These timings reflect running the notebook on an M1 MacBook Air (2020) with a wee 8GB of RAM."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 🏗️ Set up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import Markdown\n",
    "from llama_cpp import Llama"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def invoke_llm_llama_cpp(\n",
    "        model,\n",
    "        prompt,\n",
    "        system_role=\"\",\n",
    "        prepend_system_role=False,\n",
    "        max_tokens=20,\n",
    "        temperature=0.2,\n",
    "        verbose=False,\n",
    "        ):\n",
    "    \"\"\"Use llama.cpp to interact with a loaded llm model.\n",
    "    \n",
    "    See API docs for other parameters\n",
    "    https://llama-cpp-python.readthedocs.io/en/stable/api-reference/#llama_cpp.Llama.create_chat_completion\n",
    "    \"\"\"\n",
    "\n",
    "    if prepend_system_role:\n",
    "            # Some models like Gemma don't support system_role as a parameter\n",
    "        messages = (\n",
    "            {\n",
    "              \"role\": \"system\",\n",
    "              \"content\": \"\" # The `gemma` chat template requires a system role be passed with blank content\n",
    "              },\n",
    "\n",
    "            {\n",
    "              \"role\": \"user\",\n",
    "              \"content\": f\"SYSTEM ROLE: {system_role}{'.' if not system_role.endswith('.') else system_role} PROMPT: {prompt}\"\n",
    "            .replace(\"  \", \" \")\n",
    "            .strip()\n",
    "            }\n",
    "        )\n",
    "    else:\n",
    "        messages = [\n",
    "          {\n",
    "              \"role\": \"system\",\n",
    "              \"content\": f\"{system_role}\"\n",
    "              },\n",
    "          {\n",
    "              \"role\": \"user\",\n",
    "              \"content\": f\"{prompt}\"\n",
    "              }\n",
    "      ]\n",
    "    response = model.create_chat_completion(\n",
    "        messages=messages,\n",
    "        max_tokens=max_tokens,\n",
    "        temperature=temperature\n",
    "    )\n",
    "    if verbose:\n",
    "        print(response)\n",
    "    return Markdown(response[\"choices\"][0][\"message\"][\"content\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 🦙🦙🦙 Llama3 8B quantized 4b\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download\n",
    "5 GB download the first time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "llama_model_loader: loaded meta data with 33 key-value pairs and 292 tensors from /Users/chad/.cache/huggingface/hub/models--bartowski--Meta-Llama-3.1-8B-Instruct-GGUF/snapshots/9a8dec50f04fa8fad1dc1e7bc20a84a512e2bb01/./Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf (version GGUF V3 (latest))\n",
      "llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n",
      "llama_model_loader: - kv   0:                       general.architecture str              = llama\n",
      "llama_model_loader: - kv   1:                               general.type str              = model\n",
      "llama_model_loader: - kv   2:                               general.name str              = Meta Llama 3.1 8B Instruct\n",
      "llama_model_loader: - kv   3:                           general.finetune str              = Instruct\n",
      "llama_model_loader: - kv   4:                           general.basename str              = Meta-Llama-3.1\n",
      "llama_model_loader: - kv   5:                         general.size_label str              = 8B\n",
      "llama_model_loader: - kv   6:                            general.license str              = llama3.1\n",
      "llama_model_loader: - kv   7:                               general.tags arr[str,6]       = [\"facebook\", \"meta\", \"pytorch\", \"llam...\n",
      "llama_model_loader: - kv   8:                          general.languages arr[str,8]       = [\"en\", \"de\", \"fr\", \"it\", \"pt\", \"hi\", ...\n",
      "llama_model_loader: - kv   9:                          llama.block_count u32              = 32\n",
      "llama_model_loader: - kv  10:                       llama.context_length u32              = 131072\n",
      "llama_model_loader: - kv  11:                     llama.embedding_length u32              = 4096\n",
      "llama_model_loader: - kv  12:                  llama.feed_forward_length u32              = 14336\n",
      "llama_model_loader: - kv  13:                 llama.attention.head_count u32              = 32\n",
      "llama_model_loader: - kv  14:              llama.attention.head_count_kv u32              = 8\n",
      "llama_model_loader: - kv  15:                       llama.rope.freq_base f32              = 500000.000000\n",
      "llama_model_loader: - kv  16:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010\n",
      "llama_model_loader: - kv  17:                          general.file_type u32              = 15\n",
      "llama_model_loader: - kv  18:                           llama.vocab_size u32              = 128256\n",
      "llama_model_loader: - kv  19:                 llama.rope.dimension_count u32              = 128\n",
      "llama_model_loader: - kv  20:                       tokenizer.ggml.model str              = gpt2\n",
      "llama_model_loader: - kv  21:                         tokenizer.ggml.pre str              = llama-bpe\n",
      "llama_model_loader: - kv  22:                      tokenizer.ggml.tokens arr[str,128256]  = [\"!\", \"\\\"\", \"#\", \"$\", \"%\", \"&\", \"'\", ...\n",
      "llama_model_loader: - kv  23:                  tokenizer.ggml.token_type arr[i32,128256]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...\n",
      "llama_model_loader: - kv  24:                      tokenizer.ggml.merges arr[str,280147]  = [\"Ġ Ġ\", \"Ġ ĠĠĠ\", \"ĠĠ ĠĠ\", \"...\n",
      "llama_model_loader: - kv  25:                tokenizer.ggml.bos_token_id u32              = 128000\n",
      "llama_model_loader: - kv  26:                tokenizer.ggml.eos_token_id u32              = 128009\n",
      "llama_model_loader: - kv  27:                    tokenizer.chat_template str              = {{- bos_token }}\\n{%- if custom_tools ...\n",
      "llama_model_loader: - kv  28:               general.quantization_version u32              = 2\n",
      "llama_model_loader: - kv  29:                      quantize.imatrix.file str              = /models_out/Meta-Llama-3.1-8B-Instruc...\n",
      "llama_model_loader: - kv  30:                   quantize.imatrix.dataset str              = /training_dir/calibration_datav3.txt\n",
      "llama_model_loader: - kv  31:             quantize.imatrix.entries_count i32              = 224\n",
      "llama_model_loader: - kv  32:              quantize.imatrix.chunks_count i32              = 125\n",
      "llama_model_loader: - type  f32:   66 tensors\n",
      "llama_model_loader: - type q4_K:  193 tensors\n",
      "llama_model_loader: - type q6_K:   33 tensors\n",
      "llm_load_vocab: special tokens cache size = 256\n",
      "llm_load_vocab: token to piece cache size = 0.7999 MB\n",
      "llm_load_print_meta: format           = GGUF V3 (latest)\n",
      "llm_load_print_meta: arch             = llama\n",
      "llm_load_print_meta: vocab type       = BPE\n",
      "llm_load_print_meta: n_vocab          = 128256\n",
      "llm_load_print_meta: n_merges         = 280147\n",
      "llm_load_print_meta: vocab_only       = 0\n",
      "llm_load_print_meta: n_ctx_train      = 131072\n",
      "llm_load_print_meta: n_embd           = 4096\n",
      "llm_load_print_meta: n_layer          = 32\n",
      "llm_load_print_meta: n_head           = 32\n",
      "llm_load_print_meta: n_head_kv        = 8\n",
      "llm_load_print_meta: n_rot            = 128\n",
      "llm_load_print_meta: n_swa            = 0\n",
      "llm_load_print_meta: n_embd_head_k    = 128\n",
      "llm_load_print_meta: n_embd_head_v    = 128\n",
      "llm_load_print_meta: n_gqa            = 4\n",
      "llm_load_print_meta: n_embd_k_gqa     = 1024\n",
      "llm_load_print_meta: n_embd_v_gqa     = 1024\n",
      "llm_load_print_meta: f_norm_eps       = 0.0e+00\n",
      "llm_load_print_meta: f_norm_rms_eps   = 1.0e-05\n",
      "llm_load_print_meta: f_clamp_kqv      = 0.0e+00\n",
      "llm_load_print_meta: f_max_alibi_bias = 0.0e+00\n",
      "llm_load_print_meta: f_logit_scale    = 0.0e+00\n",
      "llm_load_print_meta: n_ff             = 14336\n",
      "llm_load_print_meta: n_expert         = 0\n",
      "llm_load_print_meta: n_expert_used    = 0\n",
      "llm_load_print_meta: causal attn      = 1\n",
      "llm_load_print_meta: pooling type     = 0\n",
      "llm_load_print_meta: rope type        = 0\n",
      "llm_load_print_meta: rope scaling     = linear\n",
      "llm_load_print_meta: freq_base_train  = 500000.0\n",
      "llm_load_print_meta: freq_scale_train = 1\n",
      "llm_load_print_meta: n_ctx_orig_yarn  = 131072\n",
      "llm_load_print_meta: rope_finetuned   = unknown\n",
      "llm_load_print_meta: ssm_d_conv       = 0\n",
      "llm_load_print_meta: ssm_d_inner      = 0\n",
      "llm_load_print_meta: ssm_d_state      = 0\n",
      "llm_load_print_meta: ssm_dt_rank      = 0\n",
      "llm_load_print_meta: ssm_dt_b_c_rms   = 0\n",
      "llm_load_print_meta: model type       = 8B\n",
      "llm_load_print_meta: model ftype      = Q4_K - Medium\n",
      "llm_load_print_meta: model params     = 8.03 B\n",
      "llm_load_print_meta: model size       = 4.58 GiB (4.89 BPW) \n",
      "llm_load_print_meta: general.name     = Meta Llama 3.1 8B Instruct\n",
      "llm_load_print_meta: BOS token        = 128000 '<|begin_of_text|>'\n",
      "llm_load_print_meta: EOS token        = 128009 '<|eot_id|>'\n",
      "llm_load_print_meta: LF token         = 128 'Ä'\n",
      "llm_load_print_meta: EOT token        = 128009 '<|eot_id|>'\n",
      "llm_load_print_meta: max token length = 256\n",
      "llm_load_tensors: ggml ctx size =    0.14 MiB\n",
      "llm_load_tensors: offloading 0 repeating layers to GPU\n",
      "llm_load_tensors: offloaded 0/33 layers to GPU\n",
      "llm_load_tensors:        CPU buffer size =  4685.30 MiB\n",
      "........................................................................................\n",
      "llama_new_context_with_model: n_ctx      = 512\n",
      "llama_new_context_with_model: n_batch    = 512\n",
      "llama_new_context_with_model: n_ubatch   = 512\n",
      "llama_new_context_with_model: flash_attn = 0\n",
      "llama_new_context_with_model: freq_base  = 500000.0\n",
      "llama_new_context_with_model: freq_scale = 1\n",
      "llama_kv_cache_init:        CPU KV buffer size =    64.00 MiB\n",
      "llama_new_context_with_model: KV self size  =   64.00 MiB, K (f16):   32.00 MiB, V (f16):   32.00 MiB\n",
      "llama_new_context_with_model:        CPU  output buffer size =     0.49 MiB\n",
      "llama_new_context_with_model:        CPU compute buffer size =   258.50 MiB\n",
      "llama_new_context_with_model: graph nodes  = 1030\n",
      "llama_new_context_with_model: graph splits = 514\n",
      "AVX = 0 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 1 | SVE = 0 | ARM_FMA = 1 | F16C = 0 | FP16_VA = 1 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 0 | SSSE3 = 0 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | \n",
      "Model metadata: {'quantize.imatrix.file': '/models_out/Meta-Llama-3.1-8B-Instruct-GGUF/Meta-Llama-3.1-8B-Instruct.imatrix', 'quantize.imatrix.chunks_count': '125', 'tokenizer.chat_template': '{{- bos_token }}\\n{%- if custom_tools is defined %}\\n    {%- set tools = custom_tools %}\\n{%- endif %}\\n{%- if not tools_in_user_message is defined %}\\n    {%- set tools_in_user_message = true %}\\n{%- endif %}\\n{%- if not date_string is defined %}\\n    {%- set date_string = \"26 Jul 2024\" %}\\n{%- endif %}\\n{%- if not tools is defined %}\\n    {%- set tools = none %}\\n{%- endif %}\\n\\n{#- This block extracts the system message, so we can slot it into the right place. #}\\n{%- if messages[0][\\'role\\'] == \\'system\\' %}\\n    {%- set system_message = messages[0][\\'content\\']|trim %}\\n    {%- set messages = messages[1:] %}\\n{%- else %}\\n    {%- set system_message = \"\" %}\\n{%- endif %}\\n\\n{#- System message + builtin tools #}\\n{{- \"<|start_header_id|>system<|end_header_id|>\\\\n\\\\n\" }}\\n{%- if builtin_tools is defined or tools is not none %}\\n    {{- \"Environment: ipython\\\\n\" }}\\n{%- endif %}\\n{%- if builtin_tools is defined %}\\n    {{- \"Tools: \" + builtin_tools | reject(\\'equalto\\', \\'code_interpreter\\') | join(\", \") + \"\\\\n\\\\n\"}}\\n{%- endif %}\\n{{- \"Cutting Knowledge Date: December 2023\\\\n\" }}\\n{{- \"Today Date: \" + date_string + \"\\\\n\\\\n\" }}\\n{%- if tools is not none and not tools_in_user_message %}\\n    {{- \"You have access to the following functions. To call a function, please respond with JSON for a function call.\" }}\\n    {{- \\'Respond in the format {\"name\": function name, \"parameters\": dictionary of argument name and its value}.\\' }}\\n    {{- \"Do not use variables.\\\\n\\\\n\" }}\\n    {%- for t in tools %}\\n        {{- t | tojson(indent=4) }}\\n        {{- \"\\\\n\\\\n\" }}\\n    {%- endfor %}\\n{%- endif %}\\n{{- system_message }}\\n{{- \"<|eot_id|>\" }}\\n\\n{#- Custom tools are passed in a user message with some extra guidance #}\\n{%- if tools_in_user_message and not tools is none %}\\n    {#- Extract the first user message so we can plug it in here #}\\n    {%- if messages | length != 0 %}\\n        {%- set first_user_message = messages[0][\\'content\\']|trim %}\\n        {%- set messages = messages[1:] %}\\n    {%- else %}\\n        {{- raise_exception(\"Cannot put tools in the first user message when there\\'s no first user message!\") }}\\n{%- endif %}\\n    {{- \\'<|start_header_id|>user<|end_header_id|>\\\\n\\\\n\\' -}}\\n    {{- \"Given the following functions, please respond with a JSON for a function call \" }}\\n    {{- \"with its proper arguments that best answers the given prompt.\\\\n\\\\n\" }}\\n    {{- \\'Respond in the format {\"name\": function name, \"parameters\": dictionary of argument name and its value}.\\' }}\\n    {{- \"Do not use variables.\\\\n\\\\n\" }}\\n    {%- for t in tools %}\\n        {{- t | tojson(indent=4) }}\\n        {{- \"\\\\n\\\\n\" }}\\n    {%- endfor %}\\n    {{- first_user_message + \"<|eot_id|>\"}}\\n{%- endif %}\\n\\n{%- for message in messages %}\\n    {%- if not (message.role == \\'ipython\\' or message.role == \\'tool\\' or \\'tool_calls\\' in message) %}\\n        {{- \\'<|start_header_id|>\\' + message[\\'role\\'] + \\'<|end_header_id|>\\\\n\\\\n\\'+ message[\\'content\\'] | trim + \\'<|eot_id|>\\' }}\\n    {%- elif \\'tool_calls\\' in message %}\\n        {%- if not message.tool_calls|length == 1 %}\\n            {{- raise_exception(\"This model only supports single tool-calls at once!\") }}\\n        {%- endif %}\\n        {%- set tool_call = message.tool_calls[0].function %}\\n        {%- if builtin_tools is defined and tool_call.name in builtin_tools %}\\n            {{- \\'<|start_header_id|>assistant<|end_header_id|>\\\\n\\\\n\\' -}}\\n            {{- \"<|python_tag|>\" + tool_call.name + \".call(\" }}\\n            {%- for arg_name, arg_val in tool_call.arguments | items %}\\n                {{- arg_name + \\'=\"\\' + arg_val + \\'\"\\' }}\\n                {%- if not loop.last %}\\n                    {{- \", \" }}\\n                {%- endif %}\\n                {%- endfor %}\\n            {{- \")\" }}\\n        {%- else  %}\\n            {{- \\'<|start_header_id|>assistant<|end_header_id|>\\\\n\\\\n\\' -}}\\n            {{- \\'{\"name\": \"\\' + tool_call.name + \\'\", \\' }}\\n            {{- \\'\"parameters\": \\' }}\\n            {{- tool_call.arguments | tojson }}\\n            {{- \"}\" }}\\n        {%- endif %}\\n        {%- if builtin_tools is defined %}\\n            {#- This means we\\'re in ipython mode #}\\n            {{- \"<|eom_id|>\" }}\\n        {%- else %}\\n            {{- \"<|eot_id|>\" }}\\n        {%- endif %}\\n    {%- elif message.role == \"tool\" or message.role == \"ipython\" %}\\n        {{- \"<|start_header_id|>ipython<|end_header_id|>\\\\n\\\\n\" }}\\n        {%- if message.content is mapping or message.content is iterable %}\\n            {{- message.content | tojson }}\\n        {%- else %}\\n            {{- message.content }}\\n        {%- endif %}\\n        {{- \"<|eot_id|>\" }}\\n    {%- endif %}\\n{%- endfor %}\\n{%- if add_generation_prompt %}\\n    {{- \\'<|start_header_id|>assistant<|end_header_id|>\\\\n\\\\n\\' }}\\n{%- endif %}\\n', 'tokenizer.ggml.eos_token_id': '128009', 'general.type': 'model', 'tokenizer.ggml.bos_token_id': '128000', 'tokenizer.ggml.pre': 'llama-bpe', 'tokenizer.ggml.model': 'gpt2', 'llama.embedding_length': '4096', 'llama.vocab_size': '128256', 'llama.attention.head_count_kv': '8', 'general.finetune': 'Instruct', 'general.file_type': '15', 'llama.block_count': '32', 'general.size_label': '8B', 'llama.attention.layer_norm_rms_epsilon': '0.000010', 'llama.feed_forward_length': '14336', 'general.quantization_version': '2', 'llama.rope.dimension_count': '128', 'general.license': 'llama3.1', 'llama.attention.head_count': '32', 'quantize.imatrix.entries_count': '224', 'llama.context_length': '131072', 'general.architecture': 'llama', 'general.basename': 'Meta-Llama-3.1', 'llama.rope.freq_base': '500000.000000', 'quantize.imatrix.dataset': '/training_dir/calibration_datav3.txt', 'general.name': 'Meta Llama 3.1 8B Instruct'}\n",
      "Available chat formats from metadata: chat_template.default\n",
      "Using gguf chat template: {{- bos_token }}\n",
      "{%- if custom_tools is defined %}\n",
      "    {%- set tools = custom_tools %}\n",
      "{%- endif %}\n",
      "{%- if not tools_in_user_message is defined %}\n",
      "    {%- set tools_in_user_message = true %}\n",
      "{%- endif %}\n",
      "{%- if not date_string is defined %}\n",
      "    {%- set date_string = \"26 Jul 2024\" %}\n",
      "{%- endif %}\n",
      "{%- if not tools is defined %}\n",
      "    {%- set tools = none %}\n",
      "{%- endif %}\n",
      "\n",
      "{#- This block extracts the system message, so we can slot it into the right place. #}\n",
      "{%- if messages[0]['role'] == 'system' %}\n",
      "    {%- set system_message = messages[0]['content']|trim %}\n",
      "    {%- set messages = messages[1:] %}\n",
      "{%- else %}\n",
      "    {%- set system_message = \"\" %}\n",
      "{%- endif %}\n",
      "\n",
      "{#- System message + builtin tools #}\n",
      "{{- \"<|start_header_id|>system<|end_header_id|>\\n\\n\" }}\n",
      "{%- if builtin_tools is defined or tools is not none %}\n",
      "    {{- \"Environment: ipython\\n\" }}\n",
      "{%- endif %}\n",
      "{%- if builtin_tools is defined %}\n",
      "    {{- \"Tools: \" + builtin_tools | reject('equalto', 'code_interpreter') | join(\", \") + \"\\n\\n\"}}\n",
      "{%- endif %}\n",
      "{{- \"Cutting Knowledge Date: December 2023\\n\" }}\n",
      "{{- \"Today Date: \" + date_string + \"\\n\\n\" }}\n",
      "{%- if tools is not none and not tools_in_user_message %}\n",
      "    {{- \"You have access to the following functions. To call a function, please respond with JSON for a function call.\" }}\n",
      "    {{- 'Respond in the format {\"name\": function name, \"parameters\": dictionary of argument name and its value}.' }}\n",
      "    {{- \"Do not use variables.\\n\\n\" }}\n",
      "    {%- for t in tools %}\n",
      "        {{- t | tojson(indent=4) }}\n",
      "        {{- \"\\n\\n\" }}\n",
      "    {%- endfor %}\n",
      "{%- endif %}\n",
      "{{- system_message }}\n",
      "{{- \"<|eot_id|>\" }}\n",
      "\n",
      "{#- Custom tools are passed in a user message with some extra guidance #}\n",
      "{%- if tools_in_user_message and not tools is none %}\n",
      "    {#- Extract the first user message so we can plug it in here #}\n",
      "    {%- if messages | length != 0 %}\n",
      "        {%- set first_user_message = messages[0]['content']|trim %}\n",
      "        {%- set messages = messages[1:] %}\n",
      "    {%- else %}\n",
      "        {{- raise_exception(\"Cannot put tools in the first user message when there's no first user message!\") }}\n",
      "{%- endif %}\n",
      "    {{- '<|start_header_id|>user<|end_header_id|>\\n\\n' -}}\n",
      "    {{- \"Given the following functions, please respond with a JSON for a function call \" }}\n",
      "    {{- \"with its proper arguments that best answers the given prompt.\\n\\n\" }}\n",
      "    {{- 'Respond in the format {\"name\": function name, \"parameters\": dictionary of argument name and its value}.' }}\n",
      "    {{- \"Do not use variables.\\n\\n\" }}\n",
      "    {%- for t in tools %}\n",
      "        {{- t | tojson(indent=4) }}\n",
      "        {{- \"\\n\\n\" }}\n",
      "    {%- endfor %}\n",
      "    {{- first_user_message + \"<|eot_id|>\"}}\n",
      "{%- endif %}\n",
      "\n",
      "{%- for message in messages %}\n",
      "    {%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %}\n",
      "        {{- '<|start_header_id|>' + message['role'] + '<|end_header_id|>\\n\\n'+ message['content'] | trim + '<|eot_id|>' }}\n",
      "    {%- elif 'tool_calls' in message %}\n",
      "        {%- if not message.tool_calls|length == 1 %}\n",
      "            {{- raise_exception(\"This model only supports single tool-calls at once!\") }}\n",
      "        {%- endif %}\n",
      "        {%- set tool_call = message.tool_calls[0].function %}\n",
      "        {%- if builtin_tools is defined and tool_call.name in builtin_tools %}\n",
      "            {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' -}}\n",
      "            {{- \"<|python_tag|>\" + tool_call.name + \".call(\" }}\n",
      "            {%- for arg_name, arg_val in tool_call.arguments | items %}\n",
      "                {{- arg_name + '=\"' + arg_val + '\"' }}\n",
      "                {%- if not loop.last %}\n",
      "                    {{- \", \" }}\n",
      "                {%- endif %}\n",
      "                {%- endfor %}\n",
      "            {{- \")\" }}\n",
      "        {%- else  %}\n",
      "            {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' -}}\n",
      "            {{- '{\"name\": \"' + tool_call.name + '\", ' }}\n",
      "            {{- '\"parameters\": ' }}\n",
      "            {{- tool_call.arguments | tojson }}\n",
      "            {{- \"}\" }}\n",
      "        {%- endif %}\n",
      "        {%- if builtin_tools is defined %}\n",
      "            {#- This means we're in ipython mode #}\n",
      "            {{- \"<|eom_id|>\" }}\n",
      "        {%- else %}\n",
      "            {{- \"<|eot_id|>\" }}\n",
      "        {%- endif %}\n",
      "    {%- elif message.role == \"tool\" or message.role == \"ipython\" %}\n",
      "        {{- \"<|start_header_id|>ipython<|end_header_id|>\\n\\n\" }}\n",
      "        {%- if message.content is mapping or message.content is iterable %}\n",
      "            {{- message.content | tojson }}\n",
      "        {%- else %}\n",
      "            {{- message.content }}\n",
      "        {%- endif %}\n",
      "        {{- \"<|eot_id|>\" }}\n",
      "    {%- endif %}\n",
      "{%- endfor %}\n",
      "{%- if add_generation_prompt %}\n",
      "    {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' }}\n",
      "{%- endif %}\n",
      "\n",
      "Using chat eos_token: <|eot_id|>\n",
      "Using chat bos_token: <|begin_of_text|>\n"
     ]
    }
   ],
   "source": [
    "llm = Llama.from_pretrained(\n",
    "    repo_id=\"bartowski/Meta-Llama-3.1-8B-Instruct-GGUF\",\n",
    "    filename=\"Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf\",\n",
    "    n_threads=8,\n",
    "    # verbose=False\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Alternative, download from CLI\n",
    "# %huggingface-cli download bartowski/Meta-Llama-3.1-8B-Instruct-GGUF --include \"Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Interact with llama3\n",
    "Let's repeat the tests we ran in `mlx_test.ipynb`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Without a system role\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "llama_print_timings:        load time =    7441.67 ms\n",
      "llama_print_timings:      sample time =       0.48 ms /     8 runs   (    0.06 ms per token, 16842.11 tokens per second)\n",
      "llama_print_timings: prompt eval time =    7441.61 ms /    42 tokens (  177.18 ms per token,     5.64 tokens per second)\n",
      "llama_print_timings:        eval time =   36995.41 ms /     7 runs   ( 5285.06 ms per token,     0.19 tokens per second)\n",
      "llama_print_timings:       total time =   44456.24 ms /    49 tokens\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 18 s, sys: 26.1 s, total: 44.2 s\n",
      "Wall time: 44.5 s\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "The capital of Massachusetts is Boston."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "print(\"Without a system role\")\n",
    "invoke_llm_llama_cpp(\n",
    "    llm,\n",
    "    prompt=\"What's the capital of Massachusetts?\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This version of LLama3 performed better, and faster, on this task than the `mlx` version that had no system role. But still slower than ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Llama.generate: 25 prefix-match hit, remaining 35 prompt tokens to eval\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "With a system role\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "llama_print_timings:        load time =    7441.67 ms\n",
      "llama_print_timings:      sample time =       0.12 ms /     2 runs   (    0.06 ms per token, 17094.02 tokens per second)\n",
      "llama_print_timings: prompt eval time =    7659.87 ms /    35 tokens (  218.85 ms per token,     4.57 tokens per second)\n",
      "llama_print_timings:        eval time =    5339.65 ms /     1 runs   ( 5339.65 ms per token,     0.19 tokens per second)\n",
      "llama_print_timings:       total time =   13002.58 ms /    36 tokens\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 7.76 s, sys: 7.08 s, total: 14.8 s\n",
      "Wall time: 13 s\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "Boston"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "print(\"With a system role\")\n",
    "invoke_llm_llama_cpp(\n",
    "    llm,\n",
    "    prompt=\"Name the capital of Massachusetts\",\n",
    "    system_role=\"You are a back-end GIS system. Answer queries with the answer only, no conversation or boilerplate\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Llama.generate: 28 prefix-match hit, remaining 42 prompt tokens to eval\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Markdown response\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "llama_print_timings:        load time =    7441.67 ms\n",
      "llama_print_timings:      sample time =       0.73 ms /     8 runs   (    0.09 ms per token, 10899.18 tokens per second)\n",
      "llama_print_timings: prompt eval time =    9125.10 ms /    42 tokens (  217.26 ms per token,     4.60 tokens per second)\n",
      "llama_print_timings:        eval time =   37395.23 ms /     7 runs   ( 5342.18 ms per token,     0.19 tokens per second)\n",
      "llama_print_timings:       total time =   46544.23 ms /    49 tokens\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'id': 'chatcmpl-a15b03b9-ccff-4091-a756-1c7517fc52ce', 'object': 'chat.completion', 'created': 1725843122, 'model': '/Users/chad/.cache/huggingface/hub/models--bartowski--Meta-Llama-3.1-8B-Instruct-GGUF/snapshots/9a8dec50f04fa8fad1dc1e7bc20a84a512e2bb01/./Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf', 'choices': [{'index': 0, 'message': {'role': 'assistant', 'content': '$E = mc^2$'}, 'logprobs': None, 'finish_reason': 'stop'}], 'usage': {'prompt_tokens': 70, 'completion_tokens': 7, 'total_tokens': 77}}\n",
      "CPU times: user 19.7 s, sys: 26.9 s, total: 46.6 s\n",
      "Wall time: 46.6 s\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "$E = mc^2$"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "print(\"Markdown response\")\n",
    "invoke_llm_llama_cpp(\n",
    "    llm,\n",
    "    prompt=\"Provide the formula for mass-energy equivalence\",\n",
    "    system_role=\"You are a physics database. Answer queries with the answer only, no conversation or boilerplate. If the answer requires a formula, use LaTeX\",\n",
    "    verbose=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Llama.generate: 28 prefix-match hit, remaining 15 prompt tokens to eval\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Whimsy!\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "llama_print_timings:        load time =    7441.67 ms\n",
      "llama_print_timings:      sample time =       0.59 ms /    10 runs   (    0.06 ms per token, 16891.89 tokens per second)\n",
      "llama_print_timings: prompt eval time =    6340.41 ms /    15 tokens (  422.69 ms per token,     2.37 tokens per second)\n",
      "llama_print_timings:        eval time =   50334.15 ms /     9 runs   ( 5592.68 ms per token,     0.18 tokens per second)\n",
      "llama_print_timings:       total time =   56698.71 ms /    24 tokens\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'id': 'chatcmpl-b971e975-c0b4-4eb0-b457-6a0ca2854089', 'object': 'chat.completion', 'created': 1725843169, 'model': '/Users/chad/.cache/huggingface/hub/models--bartowski--Meta-Llama-3.1-8B-Instruct-GGUF/snapshots/9a8dec50f04fa8fad1dc1e7bc20a84a512e2bb01/./Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf', 'choices': [{'index': 0, 'message': {'role': 'assistant', 'content': \"Good morning. It's nice to see you.\"}, 'logprobs': None, 'finish_reason': 'length'}], 'usage': {'prompt_tokens': 43, 'completion_tokens': 10, 'total_tokens': 53}}\n",
      "CPU times: user 20.2 s, sys: 39.5 s, total: 59.7 s\n",
      "Wall time: 56.7 s\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "Good morning. It's nice to see you."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "print(\"Whimsy!\")\n",
    "invoke_llm_llama_cpp(\n",
    "    llm,\n",
    "    prompt=\"Good morning\",\n",
    "    system_role=\"You are a friendly robot.\",\n",
    "    max_tokens=10,\n",
    "    verbose=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Llama.generate: 42 prefix-match hit, remaining 1 prompt tokens to eval\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Turn up the tempreature, turn up the creativity?\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "llama_print_timings:        load time =    7441.67 ms\n",
      "llama_print_timings:      sample time =       0.65 ms /    10 runs   (    0.07 ms per token, 15384.62 tokens per second)\n",
      "llama_print_timings: prompt eval time =       0.00 ms /     0 tokens (     nan ms per token,      nan tokens per second)\n",
      "llama_print_timings:        eval time =   54348.46 ms /    10 runs   ( 5434.85 ms per token,     0.18 tokens per second)\n",
      "llama_print_timings:       total time =   54374.54 ms /    10 tokens\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 16.3 s, sys: 37.5 s, total: 53.8 s\n",
      "Wall time: 54.4 s\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "Good morning. It's nice to see you."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "print(\"Turn up the tempreature, turn up the creativity?\")\n",
    "invoke_llm_llama_cpp(\n",
    "    llm,\n",
    "    prompt=\"Good morning\",\n",
    "    system_role=\"You are a friendly robot.\",\n",
    "    max_tokens=10,\n",
    "    temperature=0.9,\n",
    "    verbose=False\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Llama.generate: 29 prefix-match hit, remaining 24 prompt tokens to eval\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Try a goofier prompt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "llama_print_timings:        load time =    7441.67 ms\n",
      "llama_print_timings:      sample time =       0.63 ms /    10 runs   (    0.06 ms per token, 15974.44 tokens per second)\n",
      "llama_print_timings: prompt eval time =    6156.81 ms /    24 tokens (  256.53 ms per token,     3.90 tokens per second)\n",
      "llama_print_timings:        eval time =   51391.85 ms /     9 runs   ( 5710.21 ms per token,     0.18 tokens per second)\n",
      "llama_print_timings:       total time =   57570.52 ms /    33 tokens\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 23.3 s, sys: 36.7 s, total: 60 s\n",
      "Wall time: 57.6 s\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "*I extend a mechanical arm to greet you* Good"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "print(\"Try a goofier prompt\")\n",
    "invoke_llm_llama_cpp(\n",
    "    llm,\n",
    "    prompt=\"Good morning\",\n",
    "    system_role=\"You are a friendly analog robot from the 1950s with no emotion.\",\n",
    "    max_tokens=10,\n",
    "    temperature=0.9,\n",
    "    verbose=False\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ":D With the high temperature, this response varies, sometimes \"Beep boop\", sometimes \"I extend a mechanical arm to greet you\" "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 💎 Gemma 2 2B Quantized 4bit IT on `llama.cpp`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download Gemma 2 GGUF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "llama_model_loader: loaded meta data with 39 key-value pairs and 288 tensors from /Users/chad/.cache/huggingface/hub/models--bartowski--gemma-2-2b-it-GGUF/snapshots/855f67caed130e1befc571b52bd181be2e858883/./gemma-2-2b-it-Q4_K_M.gguf (version GGUF V3 (latest))\n",
      "llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n",
      "llama_model_loader: - kv   0:                       general.architecture str              = gemma2\n",
      "llama_model_loader: - kv   1:                               general.type str              = model\n",
      "llama_model_loader: - kv   2:                               general.name str              = Gemma 2 2b It\n",
      "llama_model_loader: - kv   3:                           general.finetune str              = it\n",
      "llama_model_loader: - kv   4:                           general.basename str              = gemma-2\n",
      "llama_model_loader: - kv   5:                         general.size_label str              = 2B\n",
      "llama_model_loader: - kv   6:                            general.license str              = gemma\n",
      "llama_model_loader: - kv   7:                               general.tags arr[str,2]       = [\"conversational\", \"text-generation\"]\n",
      "llama_model_loader: - kv   8:                      gemma2.context_length u32              = 8192\n",
      "llama_model_loader: - kv   9:                    gemma2.embedding_length u32              = 2304\n",
      "llama_model_loader: - kv  10:                         gemma2.block_count u32              = 26\n",
      "llama_model_loader: - kv  11:                 gemma2.feed_forward_length u32              = 9216\n",
      "llama_model_loader: - kv  12:                gemma2.attention.head_count u32              = 8\n",
      "llama_model_loader: - kv  13:             gemma2.attention.head_count_kv u32              = 4\n",
      "llama_model_loader: - kv  14:    gemma2.attention.layer_norm_rms_epsilon f32              = 0.000001\n",
      "llama_model_loader: - kv  15:                gemma2.attention.key_length u32              = 256\n",
      "llama_model_loader: - kv  16:              gemma2.attention.value_length u32              = 256\n",
      "llama_model_loader: - kv  17:                          general.file_type u32              = 15\n",
      "llama_model_loader: - kv  18:              gemma2.attn_logit_softcapping f32              = 50.000000\n",
      "llama_model_loader: - kv  19:             gemma2.final_logit_softcapping f32              = 30.000000\n",
      "llama_model_loader: - kv  20:            gemma2.attention.sliding_window u32              = 4096\n",
      "llama_model_loader: - kv  21:                       tokenizer.ggml.model str              = llama\n",
      "llama_model_loader: - kv  22:                         tokenizer.ggml.pre str              = default\n",
      "llama_model_loader: - kv  23:                      tokenizer.ggml.tokens arr[str,256000]  = [\"<pad>\", \"<eos>\", \"<bos>\", \"<unk>\", ...\n",
      "llama_model_loader: - kv  24:                      tokenizer.ggml.scores arr[f32,256000]  = [-1000.000000, -1000.000000, -1000.00...\n",
      "llama_model_loader: - kv  25:                  tokenizer.ggml.token_type arr[i32,256000]  = [3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ...\n",
      "llama_model_loader: - kv  26:                tokenizer.ggml.bos_token_id u32              = 2\n",
      "llama_model_loader: - kv  27:                tokenizer.ggml.eos_token_id u32              = 1\n",
      "llama_model_loader: - kv  28:            tokenizer.ggml.unknown_token_id u32              = 3\n",
      "llama_model_loader: - kv  29:            tokenizer.ggml.padding_token_id u32              = 0\n",
      "llama_model_loader: - kv  30:               tokenizer.ggml.add_bos_token bool             = true\n",
      "llama_model_loader: - kv  31:               tokenizer.ggml.add_eos_token bool             = false\n",
      "llama_model_loader: - kv  32:                    tokenizer.chat_template str              = {{ bos_token }}{% if messages[0]['rol...\n",
      "llama_model_loader: - kv  33:            tokenizer.ggml.add_space_prefix bool             = false\n",
      "llama_model_loader: - kv  34:               general.quantization_version u32              = 2\n",
      "llama_model_loader: - kv  35:                      quantize.imatrix.file str              = /models_out/gemma-2-2b-it-GGUF/gemma-...\n",
      "llama_model_loader: - kv  36:                   quantize.imatrix.dataset str              = /training_dir/calibration_datav3.txt\n",
      "llama_model_loader: - kv  37:             quantize.imatrix.entries_count i32              = 182\n",
      "llama_model_loader: - kv  38:              quantize.imatrix.chunks_count i32              = 128\n",
      "llama_model_loader: - type  f32:  105 tensors\n",
      "llama_model_loader: - type q4_K:  156 tensors\n",
      "llama_model_loader: - type q6_K:   27 tensors\n",
      "llm_load_vocab: special tokens cache size = 249\n",
      "llm_load_vocab: token to piece cache size = 1.6014 MB\n",
      "llm_load_print_meta: format           = GGUF V3 (latest)\n",
      "llm_load_print_meta: arch             = gemma2\n",
      "llm_load_print_meta: vocab type       = SPM\n",
      "llm_load_print_meta: n_vocab          = 256000\n",
      "llm_load_print_meta: n_merges         = 0\n",
      "llm_load_print_meta: vocab_only       = 0\n",
      "llm_load_print_meta: n_ctx_train      = 8192\n",
      "llm_load_print_meta: n_embd           = 2304\n",
      "llm_load_print_meta: n_layer          = 26\n",
      "llm_load_print_meta: n_head           = 8\n",
      "llm_load_print_meta: n_head_kv        = 4\n",
      "llm_load_print_meta: n_rot            = 256\n",
      "llm_load_print_meta: n_swa            = 4096\n",
      "llm_load_print_meta: n_embd_head_k    = 256\n",
      "llm_load_print_meta: n_embd_head_v    = 256\n",
      "llm_load_print_meta: n_gqa            = 2\n",
      "llm_load_print_meta: n_embd_k_gqa     = 1024\n",
      "llm_load_print_meta: n_embd_v_gqa     = 1024\n",
      "llm_load_print_meta: f_norm_eps       = 0.0e+00\n",
      "llm_load_print_meta: f_norm_rms_eps   = 1.0e-06\n",
      "llm_load_print_meta: f_clamp_kqv      = 0.0e+00\n",
      "llm_load_print_meta: f_max_alibi_bias = 0.0e+00\n",
      "llm_load_print_meta: f_logit_scale    = 0.0e+00\n",
      "llm_load_print_meta: n_ff             = 9216\n",
      "llm_load_print_meta: n_expert         = 0\n",
      "llm_load_print_meta: n_expert_used    = 0\n",
      "llm_load_print_meta: causal attn      = 1\n",
      "llm_load_print_meta: pooling type     = 0\n",
      "llm_load_print_meta: rope type        = 2\n",
      "llm_load_print_meta: rope scaling     = linear\n",
      "llm_load_print_meta: freq_base_train  = 10000.0\n",
      "llm_load_print_meta: freq_scale_train = 1\n",
      "llm_load_print_meta: n_ctx_orig_yarn  = 8192\n",
      "llm_load_print_meta: rope_finetuned   = unknown\n",
      "llm_load_print_meta: ssm_d_conv       = 0\n",
      "llm_load_print_meta: ssm_d_inner      = 0\n",
      "llm_load_print_meta: ssm_d_state      = 0\n",
      "llm_load_print_meta: ssm_dt_rank      = 0\n",
      "llm_load_print_meta: ssm_dt_b_c_rms   = 0\n",
      "llm_load_print_meta: model type       = 2B\n",
      "llm_load_print_meta: model ftype      = Q4_K - Medium\n",
      "llm_load_print_meta: model params     = 2.61 B\n",
      "llm_load_print_meta: model size       = 1.59 GiB (5.21 BPW) \n",
      "llm_load_print_meta: general.name     = Gemma 2 2b It\n",
      "llm_load_print_meta: BOS token        = 2 '<bos>'\n",
      "llm_load_print_meta: EOS token        = 1 '<eos>'\n",
      "llm_load_print_meta: UNK token        = 3 '<unk>'\n",
      "llm_load_print_meta: PAD token        = 0 '<pad>'\n",
      "llm_load_print_meta: LF token         = 227 '<0x0A>'\n",
      "llm_load_print_meta: EOT token        = 107 '<end_of_turn>'\n",
      "llm_load_print_meta: max token length = 48\n",
      "llm_load_tensors: ggml ctx size =    0.13 MiB\n",
      "llm_load_tensors: offloading 0 repeating layers to GPU\n",
      "llm_load_tensors: offloaded 0/27 layers to GPU\n",
      "llm_load_tensors:        CPU buffer size =  1623.67 MiB\n",
      "..........................................................\n",
      "llama_new_context_with_model: n_ctx      = 512\n",
      "llama_new_context_with_model: n_batch    = 512\n",
      "llama_new_context_with_model: n_ubatch   = 512\n",
      "llama_new_context_with_model: flash_attn = 0\n",
      "llama_new_context_with_model: freq_base  = 10000.0\n",
      "llama_new_context_with_model: freq_scale = 1\n",
      "llama_kv_cache_init:        CPU KV buffer size =    52.00 MiB\n",
      "llama_new_context_with_model: KV self size  =   52.00 MiB, K (f16):   26.00 MiB, V (f16):   26.00 MiB\n",
      "llama_new_context_with_model:        CPU  output buffer size =     0.98 MiB\n",
      "llama_new_context_with_model:        CPU compute buffer size =   504.50 MiB\n",
      "llama_new_context_with_model: graph nodes  = 1050\n",
      "llama_new_context_with_model: graph splits = 419\n",
      "AVX = 0 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 1 | SVE = 0 | ARM_FMA = 1 | F16C = 0 | FP16_VA = 1 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 0 | SSSE3 = 0 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | \n",
      "Model metadata: {'quantize.imatrix.entries_count': '182', 'quantize.imatrix.file': '/models_out/gemma-2-2b-it-GGUF/gemma-2-2b-it.imatrix', 'quantize.imatrix.chunks_count': '128', 'tokenizer.chat_template': \"{{ bos_token }}{% if messages[0]['role'] == 'system' %}{{ raise_exception('System role not supported') }}{% endif %}{% for message in messages %}{% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}{{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}{% endif %}{% if (message['role'] == 'assistant') %}{% set role = 'model' %}{% else %}{% set role = message['role'] %}{% endif %}{{ '<start_of_turn>' + role + '\\n' + message['content'] | trim + '<end_of_turn>\\n' }}{% endfor %}{% if add_generation_prompt %}{{'<start_of_turn>model\\n'}}{% endif %}\", 'tokenizer.ggml.add_eos_token': 'false', 'tokenizer.ggml.add_bos_token': 'true', 'tokenizer.ggml.padding_token_id': '0', 'tokenizer.ggml.eos_token_id': '1', 'general.type': 'model', 'gemma2.attention.layer_norm_rms_epsilon': '0.000001', 'tokenizer.ggml.bos_token_id': '2', 'general.quantization_version': '2', 'general.license': 'gemma', 'gemma2.attention.value_length': '256', 'gemma2.attention.sliding_window': '4096', 'tokenizer.ggml.pre': 'default', 'tokenizer.ggml.unknown_token_id': '3', 'gemma2.final_logit_softcapping': '30.000000', 'gemma2.attention.head_count_kv': '4', 'gemma2.attn_logit_softcapping': '50.000000', 'gemma2.attention.head_count': '8', 'tokenizer.ggml.model': 'llama', 'gemma2.feed_forward_length': '9216', 'general.size_label': '2B', 'gemma2.attention.key_length': '256', 'general.file_type': '15', 'general.finetune': 'it', 'tokenizer.ggml.add_space_prefix': 'false', 'general.architecture': 'gemma2', 'gemma2.embedding_length': '2304', 'general.basename': 'gemma-2', 'gemma2.block_count': '26', 'gemma2.context_length': '8192', 'quantize.imatrix.dataset': '/training_dir/calibration_datav3.txt', 'general.name': 'Gemma 2 2b It'}\n",
      "Available chat formats from metadata: chat_template.default\n"
     ]
    }
   ],
   "source": [
    "llm = Llama.from_pretrained(\n",
    "\trepo_id=\"bartowski/gemma-2-2b-it-GGUF\",\n",
    "\tfilename=\"gemma-2-2b-it-Q4_K_M.gguf\",\n",
    "\tchat_format=\"gemma\",\n",
    "    n_threads=8\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Interact with Gemma 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Without a system role\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "llama_print_timings:        load time =    2001.26 ms\n",
      "llama_print_timings:      sample time =       1.23 ms /    11 runs   (    0.11 ms per token,  8935.82 tokens per second)\n",
      "llama_print_timings: prompt eval time =    2001.12 ms /    17 tokens (  117.71 ms per token,     8.50 tokens per second)\n",
      "llama_print_timings:        eval time =    1684.52 ms /    10 runs   (  168.45 ms per token,     5.94 tokens per second)\n",
      "llama_print_timings:       total time =    3711.02 ms /    27 tokens\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 7.9 s, sys: 2.47 s, total: 10.4 s\n",
      "Wall time: 3.72 s\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "The capital of Massachusetts is **Boston**. \n"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "print(\"Without a system role\")\n",
    "invoke_llm_llama_cpp(\n",
    "    llm,\n",
    "    prompt=\"What's the capital of Massachusetts?\",\n",
    "    prepend_system_role=False,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Llama.generate: 4 prefix-match hit, remaining 38 prompt tokens to eval\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "With a system role\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "llama_print_timings:        load time =    2001.26 ms\n",
      "llama_print_timings:      sample time =       0.56 ms /     4 runs   (    0.14 ms per token,  7194.24 tokens per second)\n",
      "llama_print_timings: prompt eval time =    1631.38 ms /    38 tokens (   42.93 ms per token,    23.29 tokens per second)\n",
      "llama_print_timings:        eval time =     446.20 ms /     3 runs   (  148.73 ms per token,     6.72 tokens per second)\n",
      "llama_print_timings:       total time =    2087.87 ms /    41 tokens\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 4.46 s, sys: 893 ms, total: 5.35 s\n",
      "Wall time: 2.09 s\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "Boston \n"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "print(\"With a system role\")\n",
    "invoke_llm_llama_cpp(\n",
    "    llm,\n",
    "    prompt=\"Name the capital of Massachusetts\",\n",
    "    system_role=\"You are a back-end GIS system. Answer queries with the answer only, no conversation or boilerplate\",\n",
    "    prepend_system_role=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Llama.generate: 4 prefix-match hit, remaining 13 prompt tokens to eval\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Markdown response\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "llama_print_timings:        load time =    2001.26 ms\n",
      "llama_print_timings:      sample time =       2.26 ms /    20 runs   (    0.11 ms per token,  8845.64 tokens per second)\n",
      "llama_print_timings: prompt eval time =     427.65 ms /    13 tokens (   32.90 ms per token,    30.40 tokens per second)\n",
      "llama_print_timings:        eval time =    4190.58 ms /    19 runs   (  220.56 ms per token,     4.53 tokens per second)\n",
      "llama_print_timings:       total time =    4656.08 ms /    32 tokens\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'id': 'chatcmpl-90985fbf-205d-486d-ba7b-fed198817752', 'object': 'chat.completion', 'created': 1725843344, 'model': '/Users/chad/.cache/huggingface/hub/models--bartowski--gemma-2-2b-it-GGUF/snapshots/855f67caed130e1befc571b52bd181be2e858883/./gemma-2-2b-it-Q4_K_M.gguf', 'choices': [{'index': 0, 'message': {'role': 'assistant', 'content': 'The formula for mass-energy equivalence is:\\n\\n**E = mc²**\\n\\nWhere:\\n\\n'}, 'logprobs': None, 'finish_reason': 'length'}], 'usage': {'prompt_tokens': 17, 'completion_tokens': 20, 'total_tokens': 37}}\n",
      "CPU times: user 13.7 s, sys: 2.63 s, total: 16.4 s\n",
      "Wall time: 4.66 s\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "The formula for mass-energy equivalence is:\n",
       "\n",
       "**E = mc²**\n",
       "\n",
       "Where:\n",
       "\n"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "print(\"Markdown response\")\n",
    "invoke_llm_llama_cpp(\n",
    "    llm,\n",
    "    prompt=\"Provide the formula for mass-energy equivalence\",\n",
    "    system_role=\"You are a physics database. Answer queries with the answer only, no conversation or boilerplate. If the answer requires a formula, use LaTeX\",\n",
    "    verbose=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The LaTeX didn't come through there."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Llama.generate: 4 prefix-match hit, remaining 7 prompt tokens to eval\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Whimsy!\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "llama_print_timings:        load time =    2001.26 ms\n",
      "llama_print_timings:      sample time =       1.17 ms /    10 runs   (    0.12 ms per token,  8576.33 tokens per second)\n",
      "llama_print_timings: prompt eval time =     239.08 ms /     7 tokens (   34.15 ms per token,    29.28 tokens per second)\n",
      "llama_print_timings:        eval time =    1249.07 ms /     9 runs   (  138.79 ms per token,     7.21 tokens per second)\n",
      "llama_print_timings:       total time =    1503.26 ms /    16 tokens\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'id': 'chatcmpl-2eb91b24-e16d-4dc5-b5ed-a5d905a6929a', 'object': 'chat.completion', 'created': 1725843349, 'model': '/Users/chad/.cache/huggingface/hub/models--bartowski--gemma-2-2b-it-GGUF/snapshots/855f67caed130e1befc571b52bd181be2e858883/./gemma-2-2b-it-Q4_K_M.gguf', 'choices': [{'index': 0, 'message': {'role': 'assistant', 'content': 'Good morning! 😊  How can I help you'}, 'logprobs': None, 'finish_reason': 'length'}], 'usage': {'prompt_tokens': 11, 'completion_tokens': 10, 'total_tokens': 21}}\n",
      "CPU times: user 7.3 s, sys: 560 ms, total: 7.86 s\n",
      "Wall time: 1.51 s\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "Good morning! 😊  How can I help you"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "print(\"Whimsy!\")\n",
    "invoke_llm_llama_cpp(\n",
    "    llm,\n",
    "    prompt=\"Good morning\",\n",
    "    system_role=\"You are a friendly robot.\",\n",
    "    max_tokens=10,\n",
    "    verbose=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Try a goofier prompt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Llama.generate: 10 prefix-match hit, remaining 1 prompt tokens to eval\n",
      "\n",
      "llama_print_timings:        load time =    2001.26 ms\n",
      "llama_print_timings:      sample time =       1.02 ms /    10 runs   (    0.10 ms per token,  9813.54 tokens per second)\n",
      "llama_print_timings: prompt eval time =       0.00 ms /     0 tokens (     nan ms per token,      nan tokens per second)\n",
      "llama_print_timings:        eval time =    2935.68 ms /    10 runs   (  293.57 ms per token,     3.41 tokens per second)\n",
      "llama_print_timings:       total time =    2961.03 ms /    10 tokens\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 7.49 s, sys: 1.7 s, total: 9.19 s\n",
      "Wall time: 2.97 s\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "Good morning! 😊  How can I help you"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "print(\"Try a goofier prompt\")\n",
    "invoke_llm_llama_cpp(\n",
    "    llm,\n",
    "    prompt=\"Good morning\",\n",
    "    system_role=\"You are a friendly analog robot from the 1950s with no emotion.\",\n",
    "    max_tokens=10,\n",
    "    temperature=0.9,\n",
    "    verbose=False\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 💡 My lessons learned\n",
    "<b>tl;dr:</b> Out of the box, Llama3 much faster on `llama.cpp` than `mlx`, Gemma2 a lot faster on `mlx` than on `llama.cpp`. But neither test was optimized for its framework: I expect there would be more performance to be gained in both!  \n",
    "  \n",
    "* Llama3 was MUCH faster on `llama.cpp` than on `mlx`, at least out of the box ⭐. The simple system role test for example took 14 seconds instead of 57 seconds\n",
    "* Gemma was a lot faster on `mlx` ⭐, but it was still responsive and usable on this local environment, especially after I increased `n_threads`. \n",
    "    - Gemma was also ~2x faster when I restarted the kernel and only ran the Gemma 2 section, not loading the Llama3 8B model into RAM first.\n",
    "* ⭐ But these are likely not apples-to-apples comparisons.\n",
    "    - There are a number of options for models and quantizations\n",
    "    - There are a number of parameters I did not configure on `mlx` and `llama_cpp`. For example `n_gpu_layers`, and M1-specific optimizations around Metal\n",
    "    - So I wonder if both frameworks could have been optimized.\n",
    "* As in the `mlx_test.ipynb`, there was a better quality and creativity with Llama3 8B than with Gemma2 2B.\n",
    "* I found the documentation much more detailed and usable for `llama.cpp` than for `mlx`."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "llama-farm",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}