moe_charlm/MoeCharLM.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright (c) 2023 Graphcore Ltd.*"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text Generation using Mixture of Experts"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As models get larger and more capable, they get increasingly more expensive to train and run due to the large computational costs accompanying the increase in the number of parameters. **Mixture of experts** (MoE) is a technique that aims at addressing this by only activating a subset of the network (so, different experts) based on the input, thus drastically decreasing the computational costs by selecting only the appropriate subnetwork for each input.\n",
    "\n",
    "This notebook outlines a character-level text generation model in PyTorch, showing how we can define a mixture of experts model running on both IPU and CPU, starting from a standard model definition. Following the tutorial you will:\n",
    "\n",
    "* Define a light character-level language model and train it on WikiText-103 data.\n",
    "* Learn how to write the standard MoE execution scheme in PyTorch (based on [Switch Transformers](https://arxiv.org/abs/2101.03961)).\n",
    "* Adapt the language model to execute MoE, and train and run it on both IPU and CPU, observing the improvements in performance.\n",
    "\n",
    "By the end of the tutorial you will be familiar with the main aspects of the mixture of experts execution scheme and be able to adapt arbitrary PyTorch layers to incorporate it. You will also learn about distributed MoE training on IPUs and how to utilise multiple devices to increase the capacity of the model."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Environment setup\n",
    "\n",
    "To run the demo using IPU hardware, you need to have the Poplar SDK enabled. Refer to the [Getting Started guide](https://docs.graphcore.ai/en/latest/getting-started.html#getting-started) for your system for details on how to enable the Poplar SDK. Also refer to the [Jupyter Quick Start guide](https://docs.graphcore.ai/projects/jupyter-notebook-quick-start/en/latest/index.html) for how to set up Jupyter to be able to run this notebook on a remote IPU machine."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installation"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start with installing the required modules:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install -q numpy torch matplotlib\n",
    "%pip install -q git+https://github.com/graphcore-research/poptorch-experimental-addons@14886d2285c3e45b0eadf4d719dae87d5f28b109"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will also download the dataset and model checkpoints:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "wget -q -P /tmp/ https://graphcore-research-public.s3.eu-west-1.amazonaws.com/2023-moe-notebook/wikitext103.zip\n",
    "unzip /tmp/wikitext103.zip -d /tmp/dataset_cache/\n",
    "rm /tmp/wikitext103.zip\n",
    "wget -q -P /tmp/ https://graphcore-research-public.s3.eu-west-1.amazonaws.com/2023-moe-notebook/checkpoints.zip\n",
    "unzip /tmp/checkpoints.zip -d /tmp/\n",
    "rm /tmp/checkpoints.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import poptorch\n",
    "import os\n",
    "from pathlib import Path\n",
    "from dataclasses import dataclass\n",
    "from typing import Tuple, Optional\n",
    "from IPython.display import clear_output\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset_dir = Path(os.getenv(\"DATASET_DIR\"))\n",
    "checkpoint_dir = Path(os.getenv(\"CHECKPOINT_DIR\"))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For training, we use the freely-available character-level [WikiText-103 dataset](https://www.salesforce.com/products/einstein/ai-research/the-wikitext-dependency-language-modeling-dataset/). In order to load and batch the data, we can use the `Dataset` class from `data.py`. This class requires a corresponding character-level vocabulary that we will use to convert between the characters and the corresponding integer indices. In this case, the vocabulary is pre-defined in `vocab.json` and it contains all 5,008 different characters appearing in `WikiText-103`. Internally, the `Dataset` class uses the `CharVocab` class from `data.py` to provide the `str_to_ids` and `ids_to_str` methods for converting between text and an array of character IDs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of characters in the vocabulary: 5,008\n",
      "Hello world -> tensor([41, 70, 77, 77, 80,  1, 88, 80, 83, 77, 69]) -> Hello world\n"
     ]
    }
   ],
   "source": [
    "from data import CharVocab\n",
    "\n",
    "vocab = CharVocab.from_path(dataset_dir / \"wikitext103_raw/vocab.json\")\n",
    "print(f\"Number of characters in the vocabulary: {len(vocab):,}\")\n",
    "\n",
    "some_text = \"Hello world\"\n",
    "char_ids = vocab.str_to_ids(some_text)\n",
    "\n",
    "print(f\"{some_text} -> {char_ids} -> {vocab.ids_to_str(char_ids)}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When defining a `Dataset` object, we need to pass the text and the vocabulary (which will be internally converted to a `CharVocab` object). Let's load up the `validation` dataset to see how it works (this dataset is smaller than the training data and will load more quickly).\n",
    "\n",
    "**Note**: When loading the dataset, all the text is processed and converted to integer IDs - this may take some time."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Load the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of characters in the dataset: 1,144,748\n"
     ]
    }
   ],
   "source": [
    "from data import Dataset\n",
    "\n",
    "vocab_path = dataset_dir / \"wikitext103_raw/vocab.json\"\n",
    "text_path = dataset_dir / \"wikitext103_raw/valid.txt\"\n",
    "\n",
    "ds = Dataset.from_path(vocab_path, text_path)\n",
    "\n",
    "print(f\"Number of characters in the dataset: {len(ds):,}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Batching the data"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have our `Dataset` object, we can call the `batch()` method to return an iterator that will sequentially generate batches of our training, validation, or test data. For the `batch` method, we need to pass `batch_size` (number of sequences in a batch) and `sequence_length` (number of characters per sequence), and `overlap_length` (number of characters that overlap between consecutive sequences). Note that this means that the start indices of consecutive sequences will be `sequence_length - overlap_length` characters apart. In addition, we can pass a `seed` parameter to randomly generate the order of batched sequences - leaving it as `None` will instead generate sequences in the order they appear in the dataset.\n",
    "\n",
    "Each batch will consist of an input/target pair of character ID sequences (where for each input, the target is shifted one place to the left, corresponding to the next character in the sequence), as well as a mask tensor that masks out the first `overlap_length` number of characters in each sequence. We will use this during training to prevent the first `overlap_length` character predictions from contributing to the model loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of the input: torch.Size([8, 256])\n",
      "Shape of the output: torch.Size([8, 256])\n",
      "\n",
      "Example input sequence:\n",
      " st Banksy who produced the opening sequence of an earlier Simpsons episode titled \" MoneyBART \" ( 2010 ) . The Richard Hawley song \" Tonight The Streets Are Ours \" , which is the theme song of Exit Through the Gift Shop , is included in the episode during  \n",
      "\n",
      "Example target sequence:\n",
      " t Banksy who produced the opening sequence of an earlier Simpsons episode titled \" MoneyBART \" ( 2010 ) . The Richard Hawley song \" Tonight The Streets Are Ours \" , which is the theme song of Exit Through the Gift Shop , is included in the episode during a \n",
      "\n",
      "Sequence mask:\n",
      " tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
      "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 1., 1., 1.,\n",
      "        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
      "        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
      "        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
      "        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
      "        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
      "        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
      "        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
      "        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
      "        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
      "        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
      "        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
      "        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
      "        1., 1., 1., 1.])\n"
     ]
    }
   ],
   "source": [
    "batch_size = 8\n",
    "sequence_length = 256\n",
    "overlap_length = 32\n",
    "seed = 100\n",
    "\n",
    "batch = next(\n",
    "    ds.batch(\n",
    "        batch_size,\n",
    "        sequence_length,\n",
    "        overlap_length,\n",
    "        seed,\n",
    "    )\n",
    ")\n",
    "x = batch[\"x\"]\n",
    "y = batch[\"y\"]\n",
    "mask = batch[\"mask\"]\n",
    "\n",
    "print(f\"Shape of the input: {x.shape}\")\n",
    "print(f\"Shape of the output: {y.shape}\\n\")\n",
    "\n",
    "seq_num = 1\n",
    "print(\"Example input sequence:\\n\", ds.vocab.ids_to_str(x[seq_num]), '\\n')\n",
    "print(\"Example target sequence:\\n\", ds.vocab.ids_to_str(y[seq_num]), '\\n')\n",
    "print(\"Sequence mask:\\n\", mask[seq_num])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Baseline model"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start by defining the character-level text-generation baseline model. In this example we will use a simplified transformer-style architecture where, instead of a multi-head attention block, we use a causal one-dimensional convolutional layer. We construct the model by alternating convolutional and FFN layers."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p align=\"center\">\n",
    "<img src=\"https://raw.githubusercontent.com/graphcore-research/notebooks/main/moe_charlm/figs/fig1.png\"  width=\"600\" >\n",
    "</p>"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Convolutional layer"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to construct a *causal* convolution, we use a regular `nn.Conv1d` layer with `in_channels=hidden_size` and `out_channels=hidden_size`, with the setting `padding=kernel_size - 1` which ensures that the output at each position in the sequence only depends on the *previous* characters in the input sequence. As this leads to additional dummy elements at the end of each sequence, we cut the last `kernel_size - 1` characters in the `forward` method. Note that additional axis permutations need to be done in the `forward` method to align with the expected `Conv1d` input order."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "class CausalConv1d(nn.Conv1d):\n",
    "    \"\"\"Conv1d layer with left padding to enforce causality\"\"\"\n",
    "\n",
    "    def __init__(\n",
    "        self,\n",
    "        hidden_size: int,\n",
    "        kernel_size: int,\n",
    "        num_groups: int,\n",
    "        dtype: torch.dtype,\n",
    "    ):\n",
    "        super().__init__(\n",
    "            in_channels=hidden_size,\n",
    "            out_channels=hidden_size,\n",
    "            kernel_size=kernel_size,\n",
    "            padding=kernel_size - 1,\n",
    "            groups=num_groups,\n",
    "            dtype=dtype,\n",
    "        )\n",
    "\n",
    "    def forward(self, x: torch.Tensor) -> torch.Tensor:\n",
    "        x = x.transpose(-1, -2)\n",
    "        x = super().forward(x)[..., : -(self.kernel_size[0] - 1)]\n",
    "        return x.transpose(-1, -2)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### FFN layer"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the FFN layer, we will use the standard transformer structure of an up and down projection with a non-linearity in between. In this case, it's useful to define helper functions for initialising the layer weights and biases tensors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_weight_tensor(shape: Tuple[int, ...], dtype: torch.dtype) -> nn.Parameter:\n",
    "    w = nn.Parameter(torch.empty(shape, dtype=dtype))\n",
    "    xavier_std = np.sqrt(2 / (shape[-2] + shape[-1]))\n",
    "    return nn.init.normal_(w, mean=0, std=xavier_std)\n",
    "\n",
    "\n",
    "def create_bias_tensor(out_features: int, dtype: torch.dtype) -> nn.Parameter:\n",
    "    b = nn.Parameter(torch.empty(out_features, dtype=dtype))\n",
    "    return nn.init.zeros_(b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "class FFN(nn.Module):\n",
    "    \"\"\"Transformer-style FFN layer\"\"\"\n",
    "\n",
    "    def __init__(self, hidden_size: int, expand_factor: int, dtype: torch.dtype):\n",
    "        super().__init__()\n",
    "        self.hidden_size, self.expand_factor, self.dtype = (\n",
    "            hidden_size,\n",
    "            expand_factor,\n",
    "            dtype,\n",
    "        )\n",
    "        self.w1 = create_weight_tensor(\n",
    "            (hidden_size, hidden_size * expand_factor), dtype\n",
    "        )\n",
    "        self.w2 = create_weight_tensor(\n",
    "            (hidden_size * expand_factor, hidden_size), dtype\n",
    "        )\n",
    "\n",
    "    def forward(self, x: torch.Tensor) -> torch.Tensor:\n",
    "        return F.relu(x @ self.w1) @ self.w2"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Putting it all together"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have defined our convolutional and FFN layers, we can build the core layer of our model following the transformer recipe: sequence of `CausalConv1d` and `FFN` layers with residual connections and layer normalizations between them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "class CoreLayer(nn.Module):\n",
    "    \"\"\"Core model layer consisting of Conv1D + normalization + FFN\"\"\"\n",
    "\n",
    "    def __init__(\n",
    "        self,\n",
    "        hidden_size: int,\n",
    "        expand_factor: int,\n",
    "        kernel_size: int,\n",
    "        num_groups: int,\n",
    "        dtype: torch.dtype,\n",
    "    ):\n",
    "        super().__init__()\n",
    "\n",
    "        self.ffn = FFN(\n",
    "            hidden_size=hidden_size, expand_factor=expand_factor, dtype=dtype\n",
    "        )\n",
    "        self.conv = CausalConv1d(\n",
    "            hidden_size=hidden_size,\n",
    "            kernel_size=kernel_size,\n",
    "            num_groups=num_groups,\n",
    "            dtype=dtype,\n",
    "        )\n",
    "        self.layer_norm1 = nn.LayerNorm(hidden_size, dtype=dtype)\n",
    "        self.layer_norm2 = nn.LayerNorm(hidden_size, dtype=dtype)\n",
    "\n",
    "    def forward(self, x: torch.Tensor) -> torch.Tensor:\n",
    "        x_conv = F.relu(self.conv(x))\n",
    "        x = self.layer_norm1(x_conv + x)\n",
    "        x_ffn = self.ffn(x)\n",
    "        return self.layer_norm2(x_ffn + x)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The full model will now consist of a sequence of `CoreLayer` layers, with an additional `nn.Embedding` for the input embeddings, and a linear output projection from `hidden_size` to `vocab_size`. We additionally define the `masked_cross_entropy_loss` method to ignore the loss contributions of the characters based on the previously generated `mask`.\n",
    "\n",
    "In order to ensure our model can run on both IPUs and CPUs we need to calculate and return the loss function within our `forward` method, as well as to wrap it within the `poptorch.identity_loss` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "class CharModel(nn.Module):\n",
    "    \"\"\"Character-level language model\"\"\"\n",
    "\n",
    "    def __init__(\n",
    "        self,\n",
    "        vocab_size: int,\n",
    "        hidden_size: int,\n",
    "        expand_factor: int,\n",
    "        kernel_size: int,\n",
    "        num_groups: int,\n",
    "        num_layers: int,\n",
    "        dtype: torch.dtype,\n",
    "    ):\n",
    "        super().__init__()\n",
    "        self.vocab_size = vocab_size\n",
    "        self.embedding = nn.Embedding(vocab_size, hidden_size, dtype=dtype)\n",
    "        self.core_layers = nn.ModuleList()\n",
    "        for _ in range(num_layers):\n",
    "            self.core_layers.append(\n",
    "                CoreLayer(\n",
    "                    hidden_size=hidden_size,\n",
    "                    expand_factor=expand_factor,\n",
    "                    kernel_size=kernel_size,\n",
    "                    num_groups=num_groups,\n",
    "                    dtype=dtype,\n",
    "                )\n",
    "            )\n",
    "        self.w_out = create_weight_tensor(shape=(hidden_size, vocab_size), dtype=dtype)\n",
    "        self.b_out = create_bias_tensor(out_features=vocab_size, dtype=dtype)\n",
    "\n",
    "    def masked_cross_entropy_loss(\n",
    "        self, logits: torch.Tensor, y: torch.Tensor, mask: torch.Tensor\n",
    "    ) -> torch.Tensor:\n",
    "        batch_loss = F.cross_entropy(\n",
    "            logits.reshape((-1, self.vocab_size)),\n",
    "            y.reshape(-1),\n",
    "            reduction=\"none\",\n",
    "        )\n",
    "        return (batch_loss * mask.reshape(-1)).sum() / mask.sum()\n",
    "\n",
    "    def forward(\n",
    "        self, x: torch.Tensor, mask: torch.Tensor = None, y: torch.Tensor = None\n",
    "    ) -> torch.Tensor:\n",
    "        embed = self.embedding(x)\n",
    "        for layer in self.core_layers:\n",
    "            embed = layer(embed)\n",
    "        logits = embed @ self.w_out + self.b_out\n",
    "\n",
    "        if self.training:\n",
    "            loss = self.masked_cross_entropy_loss(logits, y, mask)\n",
    "            return logits, poptorch.identity_loss(loss, reduction=\"none\")\n",
    "\n",
    "        return logits"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training the model"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have defined all modules of our model, we can build and train the model on the WikiText-103 dataset."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For training, we use the `AdamW` optimiser with learning rate decay. The following code showcases a basic training loop with these settings, making sure that the model is wrapped within `poptorch.trainingModel` when executing it on the IPU. Note that running on the CPU is very slow and is mainly provided as a reference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Training settings\n",
    "@dataclass\n",
    "class BatchSettings:\n",
    "    batch_size: int = 32\n",
    "    sequence_length: int = 256\n",
    "    overlap_length: int = 32\n",
    "    batch_seed: int = 100\n",
    "\n",
    "class TrainingSettings:\n",
    "    lr: float = 2e-3\n",
    "    lr_decay: float = 2**-16\n",
    "    num_steps: int = 2**15\n",
    "\n",
    "class DeviceSettings:\n",
    "    run_on_ipu: bool = True\n",
    "    num_replicas: int = 4\n",
    "\n",
    "class Settings:\n",
    "    batch: BatchSettings = BatchSettings()\n",
    "    training: TrainingSettings = TrainingSettings()\n",
    "    device: DeviceSettings = DeviceSettings()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "jupyter": {
     "source_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "def train(model: CharModel, data: Dataset, settings: Settings):\n",
    "    model.train()\n",
    "\n",
    "    # Set the optimizer\n",
    "    if settings.device.run_on_ipu:\n",
    "        opts = poptorch.Options()\n",
    "        opts.replicationFactor(settings.device.num_replicas)\n",
    "        optimizer = poptorch.optim.AdamW(model.parameters(), lr=settings.training.lr)\n",
    "        poptorch_model = poptorch.trainingModel(model, opts, optimizer)\n",
    "    else:\n",
    "        optimizer = torch.optim.AdamW(model.parameters(), lr=settings.training.lr)\n",
    "\n",
    "    # Training loop\n",
    "    lr = settings.training.lr\n",
    "    lr_decay_factor = 2**-settings.training.lr_decay\n",
    "    training_loss = {}\n",
    "    for step, batch in zip(\n",
    "        range(settings.training.num_steps),\n",
    "        data.batch(\n",
    "            settings.batch.batch_size,\n",
    "            settings.batch.sequence_length,\n",
    "            settings.batch.overlap_length,\n",
    "            settings.batch.batch_seed,\n",
    "        ),\n",
    "    ):\n",
    "        # Apply a step\n",
    "        if settings.device.run_on_ipu:\n",
    "            out, loss = poptorch_model(\n",
    "                batch[\"x\"].int(), batch[\"mask\"], batch[\"y\"].int()\n",
    "            )\n",
    "            loss = torch.mean(loss)  # average over IPUs if num_replicas > 1\n",
    "        else:\n",
    "            optimizer.zero_grad()\n",
    "            out, loss = model(batch[\"x\"], batch[\"mask\"], batch[\"y\"])\n",
    "            loss.backward()\n",
    "            optimizer.step()\n",
    "\n",
    "        training_loss[step + 1] = loss.detach()\n",
    "\n",
    "        # Print results\n",
    "        if (step + 1) % 2**8 == 0:\n",
    "            clear_output(wait=True)\n",
    "            if step + 1 < settings.training.num_steps:\n",
    "                print(\"*** Running dense training ***\")\n",
    "            else:\n",
    "                print(\"*** Dense training finished ***\")\n",
    "            print(f\"Step {step + 1}/{settings.training.num_steps}:\")\n",
    "            print(f\"Loss: {loss.detach()}\")\n",
    "\n",
    "        # Learning rate schedule\n",
    "        lr *= lr_decay_factor\n",
    "        optimizer.param_groups[0][\"lr\"] = lr\n",
    "        if settings.device.run_on_ipu:\n",
    "            poptorch_model.setOptimizer(optimizer)\n",
    "\n",
    "    if settings.device.run_on_ipu:\n",
    "        poptorch_model.destroy()\n",
    "\n",
    "    return model, training_loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set to false to skip the training loop\n",
    "run_training = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the training dataset\n",
    "if run_training:\n",
    "    vocab_path = dataset_dir / \"wikitext103_raw/vocab.json\"\n",
    "    train_path = dataset_dir / \"wikitext103_raw/train.txt\"\n",
    "    train_ds = Dataset.from_path(vocab_path, train_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "jupyter": {
     "source_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "*** Dense training finished ***\n",
      "Step 8192/8192:\n",
      "Loss: 1.2972456216812134\n"
     ]
    }
   ],
   "source": [
    "if run_training:\n",
    "    # Create the model\n",
    "\n",
    "    # Set random seed\n",
    "    seed = None\n",
    "    if seed is None:\n",
    "        seed = np.random.SeedSequence().generate_state(1)[0]\n",
    "    np.random.seed(seed)\n",
    "    torch.manual_seed(seed)\n",
    "\n",
    "    # Set the training settings\n",
    "    settings = Settings()\n",
    "    settings.training.num_steps = 2**13 # decrease num steps to speed-up training\n",
    "\n",
    "    model = CharModel(\n",
    "        vocab_size=len(ds.vocab),\n",
    "        hidden_size=128,\n",
    "        expand_factor=4,\n",
    "        kernel_size=7,\n",
    "        num_groups=8,\n",
    "        num_layers=8,\n",
    "        dtype=torch.float32,\n",
    "    )\n",
    "\n",
    "    dense_model, dense_loss = train(model, train_ds, settings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAioAAAHHCAYAAACRAnNyAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABM/UlEQVR4nO3deVhUZf8G8HvYBpBVkE0HcEdB3HDBvcQF0TRN07Qsy7cMXzVt0cx9QW2xMrO0fmqvmlq5lhvumfuGIoqaorggKsIgywAz5/cHcWRiUAYHnwHuz3XNFXPOmTPf4aRz+2xHIUmSBCIiIiIzZCG6ACIiIqLiMKgQERGR2WJQISIiIrPFoEJERERmi0GFiIiIzBaDChEREZktBhUiIiIyWwwqREREZLYYVIiIiMhsMagQUam8/vrr8Pf3L9Vrp06dCoVCYdqCSuhp6iaiZ49BhaiCUSgUJXrs3btXdKlERE+k4L1+iCqWFStW6D3/6aefEB0djf/9739627t06QJPT89Sv09ubi50Oh2USqXRr83Ly0NeXh5sbW1L/f6l9frrr2Pv3r1ISEh45u9NRMazEl0AEZnWkCFD9J4fPnwY0dHRRbb/W2ZmJuzt7Uv8PtbW1qWqDwCsrKxgZcW/fojoydj1Q1QJderUCUFBQThx4gQ6dOgAe3t7fPzxxwCAjRs3IiIiAj4+PlAqlahduzZmzJgBrVard45/j/VISEiAQqHAZ599hsWLF6N27dpQKpVo0aIFjh07pvdaQ2NUFAoFRo4ciQ0bNiAoKAhKpRKBgYHYtm1bkfr37t2LkJAQ2Nraonbt2vj++++fatxLRkYGxo0bB5VKBaVSifr16+Ozzz7Dvxuco6Oj0a5dO7i4uMDBwQH169eXf28FFixYgMDAQNjb28PV1RUhISFYtWpVqeoiIraoEFVa9+/fR3h4OAYOHIghQ4bI3UDLli2Dg4MDxo4dCwcHB+zevRuTJ0+GWq3Gp59++sTzrlq1Cunp6Xj77behUCgwb9489O3bF1euXHliK8yBAwewbt06vPvuu3B0dMTXX3+Nfv364fr163BzcwMAnDp1Ct27d4e3tzemTZsGrVaL6dOno1q1aqX6PUiShBdeeAF79uzBm2++iSZNmmD79u344IMPcPPmTcyfPx8AcO7cOfTs2RPBwcGYPn06lEolLl++jL/++ks+15IlSzBq1Ci89NJLGD16NLKzs3HmzBkcOXIEr7zySqnqI6r0JCKq0CIjI6V//1Hv2LGjBED67rvvihyfmZlZZNvbb78t2dvbS9nZ2fK2oUOHSn5+fvLzq1evSgAkNzc3KSUlRd6+ceNGCYC0efNmeduUKVOK1ARAsrGxkS5fvixvi4mJkQBICxYskLf16tVLsre3l27evClvu3TpkmRlZVXknIb8u+4NGzZIAKSZM2fqHffSSy9JCoVCrmf+/PkSAOnu3bvFnrt3795SYGDgE2sgopJj1w9RJaVUKvHGG28U2W5nZyf/nJ6ejnv37qF9+/bIzMzEhQsXnnjel19+Ga6urvLz9u3bAwCuXLnyxNeGhYWhdu3a8vPg4GA4OTnJr9Vqtdi5cyf69OkDHx8f+bg6deogPDz8iec3ZMuWLbC0tMSoUaP0to8bNw6SJGHr1q0AABcXFwD5XWM6nc7guVxcXHDjxo0iXV1EVHoMKkSVVPXq1WFjY1Nk+7lz5/Diiy/C2dkZTk5OqFatmjwQNy0t7Ynn9fX11XteEFoePHhg9GsLXl/w2uTkZGRlZaFOnTpFjjO0rSSuXbsGHx8fODo66m1v0KCBvB/ID2Bt27bFW2+9BU9PTwwcOBBr167VCy0fffQRHBwc0LJlS9StWxeRkZF6XUNEZDwGFaJKqnDLSYHU1FR07NgRMTExmD59OjZv3ozo6GjMnTsXAIptSSjM0tLS4HapBCshPM1ry5qdnR3279+PnTt34tVXX8WZM2fw8ssvo0uXLvJA4wYNGiA+Ph6rV69Gu3bt8Ntvv6Fdu3aYMmWK4OqJyi8GFSKS7d27F/fv38eyZcswevRo9OzZE2FhYXpdOSJ5eHjA1tYWly9fLrLP0LaS8PPzw61bt5Cenq63vaCby8/PT95mYWGBzp0744svvkBcXBxmzZqF3bt3Y8+ePfIxVapUwcsvv4ylS5fi+vXriIiIwKxZs5CdnV2q+ogqOwYVIpIVtGgUbsHIycnBt99+K6okPZaWlggLC8OGDRtw69Ytefvly5flsSTG6tGjB7RaLb755hu97fPnz4dCoZDHvqSkpBR5bZMmTQAAGo0GQP5MqsJsbGzQsGFDSJKE3NzcUtVHVNlxejIRydq0aQNXV1cMHToUo0aNgkKhwP/+9z+z6HopMHXqVOzYsQNt27bFiBEj5JARFBSE06dPG32+Xr164bnnnsPEiRORkJCAxo0bY8eOHdi4cSPGjBkjD+6dPn069u/fj4iICPj5+SE5ORnffvstatSogXbt2gEAunbtCi8vL7Rt2xaenp44f/48vvnmG0RERBQZA0NEJcOgQkQyNzc3/P777xg3bhw++eQTuLq6YsiQIejcuTO6desmujwAQPPmzbF161a8//77mDRpElQqFaZPn47z58+XaFbSv1lYWGDTpk2YPHky1qxZg6VLl8Lf3x+ffvopxo0bJx/3wgsvICEhAf/3f/+He/fuwd3dHR07dsS0adPg7OwMAHj77bexcuVKfPHFF3j48CFq1KiBUaNG4ZNPPjHZ5yeqbHivHyKqEPr06YNz587h0qVLokshIhPiGBUiKneysrL0nl+6dAlbtmxBp06dxBRERGWGLSpEVO54e3vj9ddfR61atXDt2jUsWrQIGo0Gp06dQt26dUWXR0QmxDEqRFTudO/eHT///DOSkpKgVCoRGhqK2bNnM6QQVUBsUSEiIiKzxTEqREREZLYYVIiIiMhslesxKjqdDrdu3YKjoyMUCoXocoiIiKgEJElCeno6fHx8YGHx+DaTch1Ubt26BZVKJboMIiIiKoXExETUqFHjsceU66BSsCR1YmIinJycBFdDREREJaFWq6FSqUp0a4lyHVQKunucnJwYVIiIiMqZkgzb4GBaIiIiMlsMKkRERGS2GFSIiIjIbDGoEBERkdliUCEiIiKzxaBCREREZotBhYiIiMwWgwoRERGZLQYVIiIiMlsMKkRERGS2GFSIiIjIbDGoEBERkdkq1zclLCuZOXlIyciB0soS1RyVosshIiKqtNiiYkB03B20m7sHo1efEl0KERFRpcag8hiSJLoCIiKiyo1BxQCFQgEAkMCkQkREJBKDigEK0QUQERERAAaVx2LXDxERkVgMKgb80/PDjh8iIiLBGFQMULDzh4iIyCwwqDwOm1SIiIiEYlAx4FHXD5MKERGRSAwqBrDjh4iIyDwwqBggt6iwQYWIiEgoBpXHYE4hIiISi0HFIHb+EBERmQMGFQMedf2wTYWIiEgkBpXHYEwhIiISi0HFgIKOHzaoEBERicWgYkDB3ZOJiIhILAaVx2CDChERkVgMKgbI7Sns+yEiIhKKQcUA9vwQERGZB6FBRavVYtKkSahZsybs7OxQu3ZtzJgxw2ymBZtHFURERJWXlcg3nzt3LhYtWoTly5cjMDAQx48fxxtvvAFnZ2eMGjVKWF1cQp+IiMg8CA0qBw8eRO/evREREQEA8Pf3x88//4yjR4+KLAsKrkxLRERkFoR2/bRp0wa7du3CxYsXAQAxMTE4cOAAwsPDRZYlj6aV2PlDREQklNAWlfHjx0OtViMgIACWlpbQarWYNWsWBg8ebPB4jUYDjUYjP1er1WVaH7t+iIiIxBLaorJ27VqsXLkSq1atwsmTJ7F8+XJ89tlnWL58ucHjo6Ki4OzsLD9UKlWZ1MWOHyIiIvOgkAROsVGpVBg/fjwiIyPlbTNnzsSKFStw4cKFIscbalFRqVRIS0uDk5OTyerad/Euhv7fUTT0dsKW0e1Ndl4iIiLK//52dnYu0fe30K6fzMxMWFjoN+pYWlpCp9MZPF6pVEKpVD6L0gBwejIREZFoQoNKr169MGvWLPj6+iIwMBCnTp3CF198gWHDhoksi10/REREZkJoUFmwYAEmTZqEd999F8nJyfDx8cHbb7+NyZMniyyr0DoqbFMhIiISSWhQcXR0xJdffokvv/xSZBlERERkpnivHwO44BsREZF5YFAxgEvoExERmQcGlcfgyrRERERiMagYUNDxwxYVIiIisRhUDOEQFSIiIrPAoGJAwWBaNqgQERGJxaDyGFxHhYiISCwGFQMU7PohIiIyCwwqBsiDaYVWQURERAwqj8OkQkREJBSDigEK9v0QERGZBQYVA+SVacWWQUREVOkxqDwGZ/0QERGJxaBiADt+iIiIzAODigHs+iEiIjIPDCqPwZ4fIiIisRhUDGLnDxERkTlgUDHgUdcPm1SIiIhEYlAxQF6ZljmFiIhIKAYVIiIiMlsMKgYUrEzLFhUiIiKxGFQM4FBaIiIi88Cg8hhcmZaIiEgsBhUDeE9CIiIi88CgYoDin84ftqcQERGJxaDyGOz5ISIiEotBxQB2/RAREZkHBpXH4Mq0REREYjGoPAa7foiIiMRiUDGAXT9ERETmgUHFAM76ISIiMg8MKgbId09mUiEiIhKKQYWIiIjMFoOKAY/GqLBJhYiISCQGFQPkMSrMKUREREIJDSr+/v5QKBRFHpGRkSLLIiIiIjNhJfLNjx07Bq1WKz+PjY1Fly5d0L9/f4FVFRpMK7QKIiIiEhpUqlWrpvd8zpw5qF27Njp27CioonwFQ1Qk9v0QEREJZTZjVHJycrBixQoMGzYMCq64RkRERBDcolLYhg0bkJqaitdff73YYzQaDTQajfxcrVaXSS3s+iEiIjIPZtOi8uOPPyI8PBw+Pj7FHhMVFQVnZ2f5oVKpyqgazvohIiIyB2YRVK5du4adO3firbfeeuxxEyZMQFpamvxITEwsk3oerUzLpEJERCSSWXT9LF26FB4eHoiIiHjscUqlEkql8hlVRURERKIJb1HR6XRYunQphg4dCisrs8hNj2b9CK2CiIiIhAeVnTt34vr16xg2bJjoUmQKjqYlIiIyC8KbMLp27cqxIERERGSQ8BYVc8SuHyIiIvPAoGIAZ/0QERGZBwYVIiIiMlsMKgYoChZ8E1wHERFRZcegYsCjrh+xdRAREVV2DCpERERkthhUHkNi5w8REZFQDCoGsOuHiIjIPDCoGCCvTEtERERCMag8BhtUiIiIxGJQMUBuT2FSISIiEopBxQD2/BAREZkHBpXH4KwfIiIisRhUDJBXpmVOISIiEopBxQB5erLYMoiIiCo9BhUiIiIyWwwqBhSMpZXY90NERCQUg4oh7PohIiIyCwwqREREZLYYVAzgrB8iIiLzwKBiABd8IyIiMg8MKgYwpxAREZkHBpUn4MwfIiIicRhUDFAU6vthTiEiIhKHQcUAdv0QERGZBwaVJ2CDChERkTgMKgYUnvXDMSpERETiMKgYoCjU+cOYQkREJA6DiiF6LSriyiAiIqrsGFQM0Ov6YZsKERGRMEYHlcTERNy4cUN+fvToUYwZMwaLFy82aWEiFZ71wxYVIiIicYwOKq+88gr27NkDAEhKSkKXLl1w9OhRTJw4EdOnTzd5gSIouIY+ERGRWTA6qMTGxqJly5YAgLVr1yIoKAgHDx7EypUrsWzZMlPXJwRbVIiIiMyD0UElNzcXSqUSALBz50688MILAICAgADcvn3btNUJwjEqRERE5sHooBIYGIjvvvsOf/75J6Kjo9G9e3cAwK1bt+Dm5mbyAkVjiwoREZE4RgeVuXPn4vvvv0enTp0waNAgNG7cGACwadMmuUvIGDdv3sSQIUPg5uYGOzs7NGrUCMePHzf6PKak4CL6REREZsHK2Bd06tQJ9+7dg1qthqurq7z9P//5D+zt7Y0614MHD9C2bVs899xz2Lp1K6pVq4ZLly7pnVcE/a4fIiIiEsXooJKVlQVJkuQwce3aNaxfvx4NGjRAt27djDrX3LlzoVKpsHTpUnlbzZo1jS2pTHEJfSIiInGM7vrp3bs3fvrpJwBAamoqWrVqhc8//xx9+vTBokWLjDrXpk2bEBISgv79+8PDwwNNmzbFkiVLjC3J5NiiQkREZB6MDionT55E+/btAQC//vorPD09ce3aNfz000/4+uuvjTrXlStXsGjRItStWxfbt2/HiBEjMGrUKCxfvtzg8RqNBmq1Wu9RFvTu9cOkQkREJIzRXT+ZmZlwdHQEAOzYsQN9+/aFhYUFWrdujWvXrhl1Lp1Oh5CQEMyePRsA0LRpU8TGxuK7777D0KFDixwfFRWFadOmGVuy0fTWe2NQISIiEsboFpU6depgw4YNSExMxPbt29G1a1cAQHJyMpycnIw6l7e3Nxo2bKi3rUGDBrh+/brB4ydMmIC0tDT5kZiYaGz5JaKfU5hUiIiIRDE6qEyePBnvv/8+/P390bJlS4SGhgLIb11p2rSpUedq27Yt4uPj9bZdvHgRfn5+Bo9XKpVwcnLSe5SFwkvos+uHiIhIHKO7fl566SW0a9cOt2/fltdQAYDOnTvjxRdfNOpc7733Htq0aYPZs2djwIABOHr0KBYvXiz8Bofs+SEiIjIPRgcVAPDy8oKXl5d8F+UaNWqUarG3Fi1aYP369ZgwYQKmT5+OmjVr4ssvv8TgwYNLU5bJ6M36YZMKERGRMEZ3/eh0OkyfPh3Ozs7w8/ODn58fXFxcMGPGDOh0OqML6NmzJ86ePYvs7GycP38ew4cPN/ocpqbX9SOwDiIiosrO6BaViRMn4scff8ScOXPQtm1bAMCBAwcwdepUZGdnY9asWSYvUiQ2qBAREYljdFBZvnw5fvjhB/muyQAQHByM6tWr4913360wQUWhyA8pnPVDREQkjtFdPykpKQgICCiyPSAgACkpKSYpyhzInT/MKURERMIYHVQaN26Mb775psj2b775Rm8WUHlXME6FOYWIiEgco7t+5s2bh4iICOzcuVNeQ+XQoUNITEzEli1bTF6gKAUtKhyjQkREJI7RLSodO3bExYsX8eKLLyI1NRWpqano27cv4uPj5XsAVQQFE384RoWIiEicUq2j4uPjU2EGzRYn/8aEEltUiIiIBCpRUDlz5kyJTxgcHFzqYsyK3KJCREREopQoqDRp0gQKheKJq7QqFApotVqTFCbaozEqjCpERESilCioXL16tazrMDvyGBXmFCIiImFKFFSKu5txRabQuzUhERERiWD0rJ/Khi0qRERE4jCoFIPTk4mIiMRjUCkGF3wjIiISj0GlGFxCn4iISDyjg8rQoUOxf//+sqjFrHB6MhERkXhGB5W0tDSEhYWhbt26mD17Nm7evFkWdYnHBd+IiIiEMzqobNiwATdv3sSIESOwZs0a+Pv7Izw8HL/++ityc3PLokYhODmZiIhIvFKNUalWrRrGjh2LmJgYHDlyBHXq1MGrr74KHx8fvPfee7h06ZKp63zm5DEqbFIhIiIS5qkG096+fRvR0dGIjo6GpaUlevTogbNnz6Jhw4aYP3++qWoUQiE3qTCpEBERiWJ0UMnNzcVvv/2Gnj17ws/PD7/88gvGjBmDW7duYfny5di5cyfWrl2L6dOnl0W9zwynJxMREYlXoiX0C/P29oZOp8OgQYNw9OhRNGnSpMgxzz33HFxcXExQnjicnkxERCSe0UFl/vz56N+/P2xtbYs9xsXFpdzfyJAtKkREROIZHVReffVV+efExEQAgEqlMl1FZoJL6BMREYln9BiVvLw8TJo0Cc7OzvD394e/vz+cnZ3xySefVKjpyQVtKmxRISIiEsfoFpX//ve/WLduHebNm4fQ0FAAwKFDhzB16lTcv38fixYtMnmRIsgtKgwqREREwhgdVFatWoXVq1cjPDxc3hYcHAyVSoVBgwZVnKDyz3/Z9UNERCSO0V0/SqUS/v7+RbbXrFkTNjY2pqjJLLBFhYiISDyjg8rIkSMxY8YMaDQaeZtGo8GsWbMwcuRIkxYnkoKL6BMREQlndNfPqVOnsGvXLtSoUQONGzcGAMTExCAnJwedO3dG37595WPXrVtnukqfMbaoEBERiWd0UHFxcUG/fv30tlXI6cn//JdjVIiIiMQxOqgsXbq0LOowO7wpIRERkXhGB5UCd+/eRXx8PACgfv36qFatmsmKMifMKUREROIYPZg2IyMDw4YNg7e3Nzp06IAOHTrAx8cHb775JjIzM8uiRqEkNqkQEREJY3RQGTt2LPbt24fNmzcjNTUVqamp2LhxI/bt24dx48aVRY1CPFpCn4iIiEQxOqj89ttv+PHHHxEeHg4nJyc4OTmhR48eWLJkCX799VejzjV16lQoFAq9R0BAgLEllQnO+iEiIhLP6DEqmZmZ8PT0LLLdw8OjVF0/gYGB2Llz56OCrEo9bMakFIXm/RAREZEYRreohIaGYsqUKcjOzpa3ZWVlYdq0afK9f4xhZWUFLy8v+eHu7m70OcoCW1SIiIjEM7r54ssvv0T37t2LLPhma2uL7du3G13ApUuX4OPjA1tbW4SGhiIqKgq+vr5Gn8fU2J5CREQkntFBpVGjRrh06RJWrlyJCxcuAAAGDRqEwYMHw87OzqhztWrVCsuWLUP9+vVx+/ZtTJs2De3bt0dsbCwcHR2LHK/RaPSW7ler1caWX2JcR4WIiEg8o4JKbm4uAgIC8Pvvv2P48OFP/eb/vgNzq1at4Ofnh7Vr1+LNN98scnxUVBSmTZv21O9bEnKLCpMKERGRMEaNUbG2ttYbm2JqLi4uqFevHi5fvmxw/4QJE5CWliY/EhMTy6wWcHoyERGRcEYPpo2MjMTcuXORl5dn8mIePnyIv//+G97e3gb3K5VKeUp0waOsPGpRKbO3ICIioicweozKsWPHsGvXLuzYsQONGjVClSpV9PYbc8fk999/H7169YKfnx9u3bqFKVOmwNLSEoMGDTK2LJOTx6iwTYWIiEgYk9w9ubRu3LiBQYMG4f79+6hWrRratWuHw4cPm8V9gwpaVJhTiIiIxBF69+TVq1eb7FymplA8+RgiIiIqW0aPUXn++eeRmppaZLtarcbzzz9viprMQsHKtGxQISIiEsfooLJ3717k5OQU2Z6dnY0///zTJEWZA65MS0REJF6Ju37OnDkj/xwXF4ekpCT5uVarxbZt21C9enXTVmcGOJiWiIhInBIHlSZNmsh3ODbUxWNnZ4cFCxaYtDiRuDItERGReCUOKlevXoUkSahVqxaOHj2qNzPHxsYGHh4esLS0LJMiReC9foiIiMQrcVDx8/MDAOh0ujIrxpw8GqPCqEJERCSK0dOTgfw7Hu/ZswfJyclFgsvkyZNNUphoCi6hT0REJJzRQWXJkiUYMWIE3N3d4eXlJY/lAPLHdVSYoMKb/RAREQlndFCZOXMmZs2ahY8++qgs6jEbj1pUmFSIiIhEMXodlQcPHqB///5lUYtZ4U0JiYiIxDM6qPTv3x87duwoi1rMEoMKERGROEZ3/dSpUweTJk3C4cOH0ahRI1hbW+vtHzVqlMmKE0rBJfSJiIhEMzqoLF68GA4ODti3bx/27dunt0+hUFSYoPKo64dRhYiISBSjg8rVq1fLog6zw+nJRERE4hk9RqVATk4O4uPjkZeXZ8p6zAYH0xIREYlndFDJzMzEm2++CXt7ewQGBuL69esAgP/+97+YM2eOyQsU5dH6MEwqREREohgdVCZMmICYmBjs3bsXtra28vawsDCsWbPGpMWJxBYVIiIi8Yweo7JhwwasWbMGrVu31luVNjAwEH///bdJixOJY1SIiIjEM7pF5e7du/Dw8CiyPSMjQy+4lHcFS+izRYWIiEgco4NKSEgI/vjjD/l5QTj54YcfEBoaarrKROMS+kRERMIZ3fUze/ZshIeHIy4uDnl5efjqq68QFxeHgwcPFllXpTzjGBUiIiLxjG5RadeuHU6fPo28vDw0atQIO3bsgIeHBw4dOoTmzZuXRY1CcIwKERGReEa3qABA7dq1sWTJElPXYlYejVFhVCEiIhKl1Au+VXQVaFwwERFRucWgUgy564cNKkRERMIwqBRD7vrhKBUiIiJhGFSKwRYVIiIi8Z46qKjVamzYsAHnz583RT1EREREMqODyoABA/DNN98AALKyshASEoIBAwYgODgYv/32m8kLFKVgITu2qBAREYljdFDZv38/2rdvDwBYv349JElCamoqvv76a8ycOdPkBYrCeycTERGJZ3RQSUtLQ9WqVQEA27ZtQ79+/WBvb4+IiAhcunTJ5AWK8miMCqMKERGRKEYHFZVKhUOHDiEjIwPbtm1D165dAQAPHjyAra2tyQsUhS0qRERE4hm9Mu2YMWMwePBgODg4wM/PD506dQKQ3yXUqFEjU9cnHpMKERGRMEYHlXfffRctW7ZEYmIiunTpAguL/EaZWrVqVawxKgquo0JERCRaqe71ExISgpCQEACAVqvF2bNn0aZNG7i6upq0OJF492QiIiLxjB6jMmbMGPz4448A8kNKx44d0axZM6hUKuzdu7fUhcyZMwcKhQJjxowp9TlMiXdPJiIiEs/ooPLrr7+icePGAIDNmzfj6tWruHDhAt577z1MnDixVEUcO3YM33//PYKDg0v1+rLBdVSIiIhEMzqo3Lt3D15eXgCALVu2oH///qhXrx6GDRuGs2fPGl3Aw4cPMXjwYCxZssSsuo4etagwqRAREYlidFDx9PREXFwctFottm3bhi5dugAAMjMzYWlpaXQBkZGRiIiIQFhYmNGvLUsco0JERCSe0YNp33jjDQwYMADe3t5QKBRywDhy5AgCAgKMOtfq1atx8uRJHDt2rETHazQaaDQa+blarTbq/YzBMSpERETiGR1Upk6diqCgICQmJqJ///5QKpUAAEtLS4wfP77E50lMTMTo0aMRHR1d4oXioqKiMG3aNGNLLhUFePtkIiIi0RSSoDXiN2zYgBdffFGvu0ir1UKhUMDCwgIajaZIV5KhFhWVSoW0tDQ4OTmZtL4RK05ga2wSpvcOxGuh/iY9NxERUWWmVqvh7Oxcou/vUq2jsm/fPnz22Wc4f/48AKBhw4b44IMP5JsVlkTnzp2LDL594403EBAQgI8++sjgeBelUim34JQ1BRtUiIiIhDN6MO2KFSsQFhYGe3t7jBo1CqNGjYKdnR06d+6MVatWlfg8jo6OCAoK0ntUqVIFbm5uCAoKMrYsk1PI05OZVIiIiEQxukVl1qxZmDdvHt577z1526hRo/DFF19gxowZeOWVV0xaoDAcTEtERCSc0UHlypUr6NWrV5HtL7zwAj7++OOnKuZpVrY1NU5PJiIiEs/orh+VSoVdu3YV2b5z506oVCqTFGUOHt2UkIiIiEQxukVl3LhxGDVqFE6fPo02bdoAAP766y8sW7YMX331lckLFOVRiwqjChERkShGB5URI0bAy8sLn3/+OdauXQsAaNCgAdasWYPevXubvEBRCmb9EBERkThGBZW8vDzMnj0bw4YNw4EDB8qqJrPAMSpERETiGTVGxcrKCvPmzUNeXl5Z1WM2Ho1RYVIhIiISxejBtJ07d8a+ffvKohazwhYVIiIi8YweoxIeHo7x48fj7NmzaN68OapUqaK3/4UXXjBZcSIVtKjoGFSIiIiEMTqovPvuuwCAL774osg+hUIBrVb79FWZAct/2pp0bFIhIiISxuigotPpyqIOs2Npkd+iomWTChERkTBGj1GpLBhUiIiIxCtxUNm9ezcaNmwItVpdZF9aWhoCAwOxf/9+kxYnkqU8RoVBhYiISJQSB5Uvv/wSw4cPh5OTU5F9zs7OePvttzF//nyTFieSxT8tKnlsUSEiIhKmxEElJiYG3bt3L3Z/165dceLECZMUZQ7kFhUGFSIiImFKHFTu3LkDa2vrYvdbWVnh7t27JinKHHCMChERkXglDirVq1dHbGxssfvPnDkDb29vkxRlDgq6frQco0JERCRMiYNKjx49MGnSJGRnZxfZl5WVhSlTpqBnz54mLU4kKwt2/RAREYlW4nVUPvnkE6xbtw716tXDyJEjUb9+fQDAhQsXsHDhQmi1WkycOLHMCn3WLBRsUSEiIhKtxEHF09MTBw8exIgRIzBhwgRI/3yBKxQKdOvWDQsXLoSnp2eZFfqsPRqjIrgQIiKiSsyolWn9/PywZcsWPHjwAJcvX4YkSahbty5cXV3Lqj5hLNn1Q0REJJzRS+gDgKurK1q0aGHqWsxKQdcP11EhIiISh0voF4M3JSQiIhKPQaUYlhb5vxquo0JERCQOg0oxLPN7fjjrh4iISCAGlWJwMC0REZF4DCrFsOAS+kRERMIxqBRDvikhu36IiIiEYVApBltUiIiIxGNQKUbBvX4SH2QJroSIiKjyYlApRu4/a+dfTn6IMzdSxRZDRERUSTGoFGPlkevyz4v2/i2wEiIiosqLQaUYfm5V5J+3xiYJrISIiKjyYlAphpNtqW6DRERERCbEoFKMkc/XEV0CERFRpcegUgxvZzvRJRAREVV6DCpERERktoQGlUWLFiE4OBhOTk5wcnJCaGgotm7dKrIkPbWrPRpQy4XfiIiInj2hQaVGjRqYM2cOTpw4gePHj+P5559H7969ce7cOZFlyZr6uso/X7n7UGAlRERElZPQoNKrVy/06NEDdevWRb169TBr1iw4ODjg8OHDIsuSBddwln8+lZgqrhAiIqJKymzGqGi1WqxevRoZGRkIDQ0VXQ4A4OUWKvnnnDydwEqIiIgqJ+GLhZw9exahoaHIzs6Gg4MD1q9fj4YNGxo8VqPRQKPRyM/VanWZ1qa0spR/rlrFpkzfi4iIiIoS3qJSv359nD59GkeOHMGIESMwdOhQxMXFGTw2KioKzs7O8kOlUhk8zpTa1HYDAGTnasv8vYiIiEif8KBiY2ODOnXqoHnz5oiKikLjxo3x1VdfGTx2woQJSEtLkx+JiYllXp+znTUAIEOTV+bvRURERPqEd/38m06n0+veKUypVEKpVD7Teuxt8n9FGTlsUSEiInrWhAaVCRMmIDw8HL6+vkhPT8eqVauwd+9ebN++XWRZehyU+eNU2KJCRET07AkNKsnJyXjttddw+/ZtODs7Izg4GNu3b0eXLl1ElqXHXvlPi4qGLSpERETPmtCg8uOPP4p8+xKpYpPfovJQkyu4EiIiospH+GBac+fAFhUiIiJhGFSeoMo/QeXcrTTBlRAREVU+DCpPcD0lEwCQcD9TcCVERESVD4MKERERmS0GlScY2NJXdAlERESVFoPKE9hbP7rfj1YnCayEiIio8mFQeQKl9aNfkSaPM3+IiIieJQaVJyh8B+UsLqNPRET0TDGoPIGlhUL++eej1wVWQkREVPkwqBjhsx0XRZdARERUqTCoEBERkdliUCmBIa3zpyhHBHsLroSIiKhyYVApgZruDgCAP87cxvGEFMHVEBERVR4MKiVw9Op9+eeXvjsksBIiIqLKhUGlBN7uWFt0CURERJUSg0oJNPN1FV0CERFRpcSgQkRERGaLQaUU8rQ60SUQERFVCgwqpbDzfLLoEoiIiCoFBpUScrazln9WZ+UKrISIiKjyYFApoecDPOSfdZIksBIiIqLKg0GlhPo3ryH/PH7dWYGVEBERVR4MKiXU9F9TlC/dSRdUCRERUeXBoFJCdjaWes+7zN+P+w81gqohIiKqHBhUjBD9Xge955eTHwqqhIiIqHJgUDFCHQ8Hvec6jqklIiIqUwwqRlAoFHrPX/nhMCTOACIiIiozDCpPQZKAvfF3RZdBRERUYTGoGGnXuI56zw9duS+oEiIiooqPQcVItavpj1NZvP8KHmTkCKqGiIioYmNQKQUnWyu9501nRAuqhIiIqGJjUCmFYe1qii6BiIioUmBQKYXXQv2LbDt5/cGzL4SIiKiCY1AphapVbGBrrf+r6/vtQUHVEBERVVwMKqV0alLXIts2nr4poBIiIqKKS2hQiYqKQosWLeDo6AgPDw/06dMH8fHxIksqsX/f+wcARq8+/ewLISIiqsCEBpV9+/YhMjIShw8fRnR0NHJzc9G1a1dkZGSILKvEwhp4FNkWk5j67AshIiKqoBSSGa0Bf/fuXXh4eGDfvn3o0KHDE49Xq9VwdnZGWloanJycnkGF+o5eTcGA7w8V2Z4wJ+KZ10JERFReGPP9bVZjVNLS0gAAVatWFVxJybSsWRVNVC5FtptR9iMiIirXzCao6HQ6jBkzBm3btkVQUJDBYzQaDdRqtd5DtHUj2hTZFndbfF1EREQVgdkElcjISMTGxmL16tXFHhMVFQVnZ2f5oVKpnmGFhllYKODvZq+3LeLrA7h+P1NQRURERBWHWYxRGTlyJDZu3Ij9+/ejZs3iV33VaDTQaDTyc7VaDZVKJWyMSoEbDzLRbu6eItvjpneDvY2VgVcQERFVXuVmjIokSRg5ciTWr1+P3bt3PzakAIBSqYSTk5PewxzUcLXH4Qmdi2xvOHk7tDrhOZCIiKjcEhpUIiMjsWLFCqxatQqOjo5ISkpCUlISsrKyRJZVKl7Otga31524hWGFiIiolIQGlUWLFiEtLQ2dOnWCt7e3/FizZo3IskxKJwGTN8aKLoOIiKhcEt71Y+jx+uuviyyr1M5N6wZ3B5si21ceuY4MTZ6AioiIiMo3s5n1UxFUUVrh6MdhBvcFTtmOhHvlY8VdIiIic8GgYmIWFgr8PbuHwX2dPtuLXK3uGVdERERUfjGolAFLCwVOTupicF+/RQfxv8PXcPZG2jOuioiIqPzhIh9lpGqVomNVAODMjTSc+Sek8J5AREREj8cWlTI0sMXjV87NzHk0wHZbbBI+2x7P+wQREREVwqBShjrVr/bY/Q0nb0fszTQkpWXjnRUn8M2ey9h9IfkZVUdERGT+2PVThsIaeKJDvWrYf/Fuscf0XHBA7/kdtaaYI4mIiCoftqiUIStLC/w0rKVRY1E+Xn8W9x9q9LqA2B1ERESVFYPKM3J6suFZQIY0n7kTNSdswaaYW5j1RxyazYhGSkZOGVZHRERknszi7smlZczdF81BrlaHuhO3lvr1K99qBa1OwsQNZzGwhS/eal8TNpYWUCgUJqySiIiobBnz/c2g8oxl5WjRYPI2k53PydYKL7dQYcmfV/G/N1uibW13JD7IRE6eDnU9HU32PkRERKbCoGLmLt5Jx9e7LuH3M7fL9H2+fLkJ6ng4wMbKAvUYWoiIyEwwqJQTe+KT8cbSY8/kvX59JxRB1Z2RkpEDHxc7LNl/Bbk6Hd7tVEfvOE2eFttik9C2jjvcHZTPpDYiIqpcjPn+5vRkgZ6r74GFrzTDnG3n8Va7Wpiy6VyZvddL3x1C+7ru+PPSPb3tYQ08UbuaA7JztaiitMKCXZfxzZ7L8Hezx573O+mNf8nJ08HGiuOviYjo2WGLihmZsO4sfj56XXQZev77fB2M61ofu87fwZvLjwMAfnknFC38qyIrR4tuX+7H9ZRMfDekGboHeT/xfHlaHc7eTEMDbyf8cjwRIf5V0cA7/9rtu3gXtatVQQ1X+zL9TE9Dp5NgYcHBy0RET4NdP+VYnlaHOk8xM6gsBFV3QuxNtd62L19ugjFrTutti5veDdaWFrBUKBB3W43fTt6AvY0l3gurBytLCySnZ6PlrF1Fzp8wJwK/nriB93+JkZ8DQMK9DDjZWevdN0mnk5CRkwc7a0vsu3gX1RyVCK7hYtoPXIwzN1IxaPFhvN+tPt5oW/OZvKepaXUSouPuoJmfCzwcbUWXQ0SVFINKOZehyUPglO0AgLAGHth5vnwvqx9ayw0z+gRiwPeHDa4H89f459F2zm75+fnp3ZGuyZVDze//bYeg6s4AgIGLD+HwlZQi5+jbtDrGhwfAwyn/yzdPq8PyQ9fwXP1q8HGxw974u1Bn5aJ3Ux8orSzl1+VqdcjO1cLR1lrvfJIkIeF+Jvzd7OXur27z9yP+TjqAR2FKkiScuZGGpX9dhZWlBeb1C35mLS6SJGHNsUQEVXeWfz9PsvxgAqZsOgdXe2ucmty1jCskMi86nYQHmTlw4/g74RhUKpjUzBw0mR4tP/d2tsXttGyBFZUtO2tLZOVqi2yf0qshpm2OM/p8rWpWxZGr+eHm9Tb+eKWVL2b+cR7vhdXFi98eBAB8MaAxXO1tsC02CVNfCMT3+//GlzsvoaG3Ezb/tx0AoMv8fbhyNwMAEDOlK2ytLTBl4zmsPpYov9f3rzZHt0CvEteWk6fDupM30K6ue7FdXhtO3YSNlQV6NHrUtRaTmIreC/+SnyfMiUBOng6HrtyHh6NS7k77t1d/PCKPU7oyuwcUChhch+eOOhsejsonrtGj00k4n6SGh6MtJEhyK82+i/nBsFdjHwDAvG0XkPggC18PbFLsOe+o8/+f9nRiSw+ZRq5WBwuFApb//ONhxIoT2BqbhNX/aY3WtdwEV1e5MahUULlaHawt8wezrjxyDRPXxwquqHLo37wGztxIk1tTnmRO30Y4nZiKCT0aYM7WCwiu4YwmKhf8dOgaarjaoZqjErdSs/Dlzkt6r4sI9saXLzeBtaUFtDoJOknCttgk/PfnUwCAFW+2wmc74rFwcDN0mLcHWt2jP7qjO9fFuVtq7Dx/BwCwaWRb1HSvIrcUZeVo8cfZ23L3GgAEeDmimqMS48MDcDn5IWq6V0FwDRdsPH0To1efxqut/TCjTxCA/ECy8uh1NFW5YOf5O6jn6Ygejbzx1c5LmL/zonzOH14LwbGEFHy//wqA/Nay6i528B//BwDg5+GtcfL6A3SoWw2NajxqBdLkaVH/k/z1hS7ODH/ioG2dTsLcbRfQROWC8EbeBvdp8nTo0cgbLWtWfdIlKxGtToJFoWB38U46vtl9GTNfDILTv1rkzM1PhxKgtLLAyy18S3T8+lM34FZFiQ71quGvy/egkyQEV3eBs715f84CxxJS4OVkixe/PQhnOyvsGtcJAOT/D8MaeOCHoS0EVvh0HmrycP62Gs19XZ+qBTc7V4uoLefRpaEX2tV1N2GFT8agUglIkoT1p24i0McZo1efwoWkdPw4NAS5Wh3upmuw4fQtnLj2QHSZVEpNfV1w6nqqSc41pLUvVhwu3SDtgi6ugvBS2NWoHqg5YctjX99Y5YKBLVSYsO5skX21qlXB1wOboo6HA/os/AsXkvKDYHM/V9R0r4KBLVQYvfo00rNzsWxYSzTzdQUALNl/BcsOJuBmapZejQDw7d7LmLctXu99Ls8Kh5Xlo+BzOTkda44lIidPB1+3KhjW1h//+d8J+DjbYlrvIIOfIz07F12+2A9rKwU2j2wHF3sb+UsPyO+ufOWHw2hb2x1ju9TT+/JIz86FlYUF7Gzyuxxz8nQ4ezMVjWu46NUF5H9xSBLkYws7df0B7qiz0T3IG8nqbFxLyYSfm32RsUZpmblQWlvA1jr/HHfTNWgxaycA4NKscPkfO8W5cvchnv98n/y5Ci9Q+ceodgj00e9mvKPOxrsrT2JQS1+81LzGY8/9JJeT0/H7mdt4q30tOChLPilVk6fFhHVn0bFeNdRwtUe/RQf19m8d3R4NvJ0KBRVP/DA0BACQmJKJDadu4tVQP7jY2yArR4tlBxPQpaEH6ng82/WnSjpYv+eCPxF7U42ZfYIwpLVfqd9vwa5L+Dw6/x8aCXMikJWjxcfrz6JboGeJJkc8DQaVSkank5CenVfkXzufbr+AhXv+BgC0rFkVq4e3Rq2PH//FQlTY2x1roanKFe+sOCG6FLzVriZ+OHDV4L6NkW2x9ngiVh4pGsjcHZSY0TsQ4Y288dn2eHyz57Le/t9GtJG/2K5G9cDN1Cxsi03CK618YaFQQGllgchVJ7HlbBIAwEIB7PvgObSft6fYWieEB+CNtjWR+CATnT/fB1trCxz46HnM/uM81p26WeT4Xo190LpWVbmVdGNkWzRWueDinXR0nb8fLzWvgV9P3AAANPB2wvnbjwa3J8yJwO4Ld/DHmSS807EWuszfDydbK7zRtibcHGzQprYbwr7YDwDoHuiFroGe6NusBiRJQmpmLrbGJkFpZQEXe2u0q+sut2wBwOr/tMbAxYfl5wNbqPBR9wA0nZHfFf1i0+pYX+jzXJjRHQGT8l8fN70b7KwtkZGjxaU76fCtao/mM3fCxsoCF2eGy6+RJAnpmjw42VrLQaKgNS9Pq4NCoUCSOhtKKwusO3kDfZpWl8PZgUv34GxnjePXUp7YLfzFgMYYuza/RbFLQ08seS1EL4B3D/TCZwMa4+tdl7D4nxbBwmPRCndZZmjy8On2ePQM9kaIf9Ui+0tjZ9wdvLf2NL4Y0ARdGnrq7Tt4+R62n0vC+PAGsLOxlH9PTX1dsPjVEESuPImBLVXo2+zxQTFXq0OGJg/J6Rr4VrXHyFUn5TGQCXMi9ILLmaldy7SlkEGFAOT/4UrJyMHfdzMQVN0J9jZWOJ2Yij7/jG34oFt99G9eQx6AmpiSiaH/dxRX7mXA3UGJwa18ceZGKvbE3xX5MYgqpS2j2qPH13+WybnHhNUt0vVoai39q+JowqOB7zZWFsjJ0wEA9n/wHJzsrLDi8DV8tiP/i7FLQ09Ex92Rj5/UsyFm/F40fPi72WN0WF28t+ZRN6ZvVXtcT8k0qr7jn4QhZObOxx6TMCcCGZo8hH/1J1rXqooMTX4XqkIBFHxztq/rjpsPstC2jjtOXHuAxa81R2aOFnU9HAAAP/x5FadvpGJuv+D8Lla3KvI/Ko9eTUF1Vzucv6XGWz8dl9/3jbb+eLW1H/zcqiAnTye3agX6OOGz/o0R/lX+/xfNfF1Q18MRa47nj5Nb+EozAPndyIY0nb4DDzJzDe5b9VYrvLf2NO6oNfK2Fv6u+G5I8zIZfMygQiZVeADmunfboO+3B+HnZo9r9x/9xVDcAFgiovLqzw+fw4zf47CjUIAyRmOVC2ISU4ts/2pgE6w9noi/Lt9/ygoNs7JQoFN9D1y99xAOSivE3Egr9bkGtVQhqm+wCavLx6BCJvUgIwfrT93EC0189JbVP56Qgpe+O4QPutVH5HN1cOZGKl745i+80dYfS/9KAAC807E2egZ7o+eCA/LrBrVUYXz3Bmg8fUeR9/qgW3009HGCk601voiOx3P1PTDzj/MG66ruYiePUyAiorLx9+we8swpU2FQoWemuGX1T15/AE8nW1R3sZO3pWTkwNXeWu7L/f3MLSigQOSqkwAeDXj7tyt3H+L4tQf48NczcFBawc3BBrP6NEK7uu5Y+M94g0+35w+gHNLaF9aWFni/a315LZrClrwWguGFmliJiOjxOtSrhp+GtTTpORlUqFy5cvch8nTSU93hOTElEzlaHWpXc5C33XiQiYN/38eLTavrzXT4fEc8Fuy+bOg0AICVb7XC4B+OFLu/ik3+AEFvZ1vcz8jBjN6B+Oi3orNaiIgqisKz60yBNyWkcqVWoXBRWqqqRRdLq+FqjwEhRbdHPlcHXs62uJuuwb6Ld/FJRAPU9XSEo9IKuVoJNlYWSJgTgcSUTMQnpWP96Zv448xt+fVnp3bDzdQsqKray9MJvZztsDnmFqb0aghHW2ucTkyFvY0l3B2UcFBa4Y+zt/B/BxKw4s1WWHs8Eedvq3EvIwdX7z1EYkoWPu4RgP90qI0bDzIRteUC/jh7W148TpIkaHWSfGuF1f9pjZjEVERtvYA6Hg5wd7BBkI9zsTNiCis8e+RxAn2ccO6W+rHHPGkKtYejEsnpmmL3l8TzAR7YfaF8r8xMRE+HLSpEJbA55hZ+OHAVS19voXfvoWfpZmoWsnLyUMfDETqdhLjbatTzdJS73u491OD+wxzU98pvmVp7LBFTN5/Dpy81xh9nb6FNbXcMae2Hz3fEw9nOGi81r4EfD1xFn6bV4e6ghNLq0dobhc38PQ7/99dVNFG5wNJCAbcqSnw1qAmUVpbySp8AUNfDAR/3aIDnAjzk12p1Ev44exuj/lm0rrB/rzT89+weuHL3IRbvv4KRz9eBn1sVAMD7v8TI4argX3U5eTroJAm21pZ665n829q3Q7Fg9yX8eekeRnSqjaGh/mgdVfR+U7WqVZFXHS5g6G7jBQpa1Z7W7/9tB29nWzR/wuwTc+HtbAsLhYJjwyqZWtWqYPc/i+aZCrt+iAhA2d/tOSUjB1/vuoQBISo09Cn+z+D/DiXA0dYaAd6OiFx5Eu91qYeewfnL6//w5xU42loVu2pqnlaHSRtj0bJmVbzYtOg6EZeTH+L8bTV6NfbBp9svQJKAD7sHFFtLq9k7cUetwZiwutDpJLzTqTZsrSyhlSScv61GoI9zkYGDkiTpLW53eVY47qRrcPjv+xhXaLXfAs8HeKB1raoID/JGfFI6rqdkYvo/U22XD2uJjvWqIU+rK7LgmyRJSE7XoP3cPcjR6uTtcdO7Ydf5ZEzZdA6LBjfDkB+PIFcr4e0OtQAF0LtxdVy8k45zt9Kw5E/9lrU973dCTfcq6LXgAM7eLDr745tXmmLkKv0gGTO5K5ztrfEgIwdf786/vj7OdnC2t4ZWJ2Hetgu4m66BhYUCIzrVRud/Fogr3BJXcD8pTZ4WF5Meot93B+Xpyf/2Zrua+LEELYJKKwtoijnH07CxtJB/3wFejvLig8CTu4L/bWqvhphaglt9zO3XyGRdxv+eCm5qpyZ1gauJ/4HGoEJEVIwMTR7uPdTILTYldTwhBb+euIGPugfo/aUdHXcHdT0c4Ghr9dj1Ji4nP0R8Ujp6NPJ64uJgDzJycO+hBk521sjTSfKg9IKFxW6nZeHktVR0D/LSC1W5Wh32xt/F7bQsTN54Dq+08sXsFxsByL+Nwqqj1zHj9zjUcq+CwOrOmP5CIFyr2GBTzC18viMeE8IboKZ7FblVrqTGrjmNdaduYve4jsjQaHHiWgpeC/UvEpKX7L8COxtLdAv0QqvZO1FwF4iC9UpeWXIYzwd4YnRYXQDApTvpmLzxHA5duY85fRthYMv8MPv33Yf4+ch1xN1W4+Df9/Fh9/po5uuK99aclu+DVrtaFfzyThtkaPLw0ncH4WRrjVytDouGNMdDTR5OXHuA6ymZGNGxNmq45s8gvHY/E23ruOPQ3/fxn/8dx9cDm8othJIkYe62eFStYo1O9T3g52aPxfuuYE98Mk7+0wV6/JMwONtZY8SKE5AkYNe/ui0Xv9ocQdWd4eVkK/9usnO18iJ5BSaEB+DtjrWRkpGDZv8srhf5XG10buCJZHU2jl59AFVVO7lFcnrvQLwW6o/sXC1srS3xICMHd9KzcSs1C+3q5N8GIfr8HawqtCDiuC710Km+B5YdTMDJ6w8wsIUKUVsvyPvPT++Oy8kPUdfTwWBL69NiUCEiquSS0rLh6VT0xpLp2blwUFo99UqqhUmSBE2ezqgvtEN/38dHv53BzD5B6FCvWqnfu3DL1JNa354lSZIwb3s8arlXQf8Q1WNbNx9q8nA5+SGOJ6RgSGs/vd/j+lM3EB13B18MaFLk9xsddwf7L97F5F4Nn3hrBCB/goEkAfY2lgZDdcEioc/i7tIMKkRERGS2jPn+fnIEIyIiIhKEQYWIiIjMFoMKERERmS2hQWX//v3o1asXfHx8oFAosGHDBpHlEBERkZkRGlQyMjLQuHFjLFy4UGQZREREZKaELqEfHh6O8PBwkSUQERGRGeMYFSIiIjJb5eqmhBqNBhrNo5ucqdWPv2kaERERlW/lqkUlKioKzs7O8kOlUokuiYiIiMpQuQoqEyZMQFpamvxITEwUXRIRERGVoXLV9aNUKqFUlv09CIiIiMg8CA0qDx8+xOXLl+XnV69exenTp1G1alX4+hq+5TsRERFVHkKDyvHjx/Hcc8/Jz8eOHQsAGDp0KJYtWyaoKiIiIjIXQoNKp06dUI5v3kxERERlrFyNUfm3gpDDacpERETlR8H3dkkaK8p1UElPTwcATlMmIiIqh9LT0+Hs7PzYYxRSOe570el0uHXrFhwdHaFQKEx6brVaDZVKhcTERDg5OZn03FR2eN3KJ1638onXrXwyh+smSRLS09Ph4+MDC4vHr5RSrltULCwsUKNGjTJ9DycnJ/4BLId43conXrfyidetfBJ93Z7UklKgXC34RkRERJULgwoRERGZLQaVYiiVSkyZMoUr4ZYzvG7lE69b+cTrVj6Vt+tWrgfTEhERUcXGFhUiIiIyWwwqREREZLYYVIiIiMhsMagQERGR2WJQMWDhwoXw9/eHra0tWrVqhaNHj4ouqdKIiopCixYt4OjoCA8PD/Tp0wfx8fF6x2RnZyMyMhJubm5wcHBAv379cOfOHb1jrl+/joiICNjb28PDwwMffPAB8vLy9I7Zu3cvmjVrBqVSiTp16vCO3SY0Z84cKBQKjBkzRt7G62aebt68iSFDhsDNzQ12dnZo1KgRjh8/Lu+XJAmTJ0+Gt7c37OzsEBYWhkuXLumdIyUlBYMHD4aTkxNcXFzw5ptv4uHDh3rHnDlzBu3bt4etrS1UKhXmzZv3TD5fRaXVajFp0iTUrFkTdnZ2qF27NmbMmKF375wKc+0k0rN69WrJxsZG+r//+z/p3Llz0vDhwyUXFxfpzp07okurFLp16yYtXbpUio2NlU6fPi316NFD8vX1lR4+fCgf884770gqlUratWuXdPz4cal169ZSmzZt5P15eXlSUFCQFBYWJp06dUrasmWL5O7uLk2YMEE+5sqVK5K9vb00duxYKS4uTlqwYIFkaWkpbdu27Zl+3oro6NGjkr+/vxQcHCyNHj1a3s7rZn5SUlIkPz8/6fXXX5eOHDkiXblyRdq+fbt0+fJl+Zg5c+ZIzs7O0oYNG6SYmBjphRdekGrWrCllZWXJx3Tv3l1q3LixdPjwYenPP/+U6tSpIw0aNEjen5aWJnl6ekqDBw+WYmNjpZ9//lmys7OTvv/++2f6eSuSWbNmSW5ubtLvv/8uXb16Vfrll18kBwcH6auvvpKPqSjXjkHlX1q2bClFRkbKz7VareTj4yNFRUUJrKrySk5OlgBI+/btkyRJklJTUyVra2vpl19+kY85f/68BEA6dOiQJEmStGXLFsnCwkJKSkqSj1m0aJHk5OQkaTQaSZIk6cMPP5QCAwP13uvll1+WunXrVtYfqUJLT0+X6tatK0VHR0sdO3aUgwqvm3n66KOPpHbt2hW7X6fTSV5eXtKnn34qb0tNTZWUSqX0888/S5IkSXFxcRIA6dixY/IxW7dulRQKhXTz5k1JkiTp22+/lVxdXeXrWPDe9evXN/VHqjQiIiKkYcOG6W3r27evNHjwYEmSKta1Y9dPITk5OThx4gTCwsLkbRYWFggLC8OhQ4cEVlZ5paWlAQCqVq0KADhx4gRyc3P1rlFAQAB8fX3la3To0CE0atQInp6e8jHdunWDWq3GuXPn5GMKn6PgGF7npxMZGYmIiIgiv1teN/O0adMmhISEoH///vDw8EDTpk2xZMkSef/Vq1eRlJSk9zt3dnZGq1at9K6bi4sLQkJC5GPCwsJgYWGBI0eOyMd06NABNjY28jHdunVDfHw8Hjx4UNYfs0Jq06YNdu3ahYsXLwIAYmJicODAAYSHhwOoWNeuXN+U0NTu3bsHrVar9xclAHh6euLChQuCqqq8dDodxowZg7Zt2yIoKAgAkJSUBBsbG7i4uOgd6+npiaSkJPkYQ9ewYN/jjlGr1cjKyoKdnV1ZfKQKbfXq1Th58iSOHTtWZB+vm3m6cuUKFi1ahLFjx+Ljjz/GsWPHMGrUKNjY2GDo0KHy793Q77zwNfHw8NDbb2VlhapVq+odU7NmzSLnKNjn6upaJp+vIhs/fjzUajUCAgJgaWkJrVaLWbNmYfDgwQBQoa4dgwqZrcjISMTGxuLAgQOiS6EnSExMxOjRoxEdHQ1bW1vR5VAJ6XQ6hISEYPbs2QCApk2bIjY2Ft999x2GDh0quDp6nLVr12LlypVYtWoVAgMDcfr0aYwZMwY+Pj4V7tqx66cQd3d3WFpaFpmJcOfOHXh5eQmqqnIaOXIkfv/9d+zZswc1atSQt3t5eSEnJwepqal6xxe+Rl5eXgavYcG+xx3j5OTEf5WXwokTJ5CcnIxmzZrBysoKVlZW2LdvH77++mtYWVnB09OT180MeXt7o2HDhnrbGjRogOvXrwN49Ht/3N+JXl5eSE5O1tufl5eHlJQUo64tGeeDDz7A+PHjMXDgQDRq1Aivvvoq3nvvPURFRQGoWNeOQaUQGxsbNG/eHLt27ZK36XQ67Nq1C6GhoQIrqzwkScLIkSOxfv167N69u0iTY/PmzWFtba13jeLj43H9+nX5GoWGhuLs2bN6fwCjo6Ph5OQk/6UcGhqqd46CY3idS6dz5844e/YsTp8+LT9CQkIwePBg+WdeN/PTtm3bItP/L168CD8/PwBAzZo14eXlpfc7V6vVOHLkiN51S01NxYkTJ+Rjdu/eDZ1Oh1atWsnH7N+/H7m5ufIx0dHRqF+/Prt9SikzMxMWFvpf4ZaWltDpdAAq2LV7ZsN2y4nVq1dLSqVSWrZsmRQXFyf95z//kVxcXPRmIlDZGTFihOTs7Czt3btXun37tvzIzMyUj3nnnXckX19faffu3dLx48el0NBQKTQ0VN5fMM21a9eu0unTp6Vt27ZJ1apVMzjN9YMPPpDOnz8vLVy4kNNcTazwrB9J4nUzR0ePHpWsrKykWbNmSZcuXZJWrlwp2dvbSytWrJCPmTNnjuTi4iJt3LhROnPmjNS7d2+DU1ybNm0qHTlyRDpw4IBUt25dvSmuqampkqenp/Tqq69KsbGx0urVqyV7e3tOT34KQ4cOlapXry5PT163bp3k7u4uffjhh/IxFeXaMagYsGDBAsnX11eysbGRWrZsKR0+fFh0SZUGAIOPpUuXysdkZWVJ7777ruTq6irZ29tLL774onT79m298yQkJEjh4eGSnZ2d5O7uLo0bN07Kzc3VO2bPnj1SkyZNJBsbG6lWrVp670FP799BhdfNPG3evFkKCgqSlEqlFBAQIC1evFhvv06nkyZNmiR5enpKSqVS6ty5sxQfH693zP3796VBgwZJDg4OkpOTk/TGG29I6enpesfExMRI7dq1k5RKpVS9enVpzpw5Zf7ZKjK1Wi2NHj1a8vX1lWxtbaVatWpJEydO1JtGXFGunUKSCi1jR0RERGRGOEaFiIiIzBaDChEREZktBhUiIiIyWwwqREREZLYYVIiIiMhsMagQERGR2WJQISIiIrPFoEJERERmi0GFiMrc3bt3MWLECPj6+kKpVMLLywvdunXDX3/9BQBQKBTYsGGD2CKJyCxZiS6AiCq+fv36IScnB8uXL0etWrVw584d7Nq1C/fv3xddGhGZOS6hT0RlKjU1Fa6urti7dy86duxYZL+/vz+uXbsmP/fz80NCQgIAYOPGjZg2bRri4uLg4+ODoUOHYuLEibCyyv83lkKhwLfffotNmzZh79698Pb2xrx58/DSSy89k89GRGWPXT9EVKYcHBzg4OCADRs2QKPRFNl/7NgxAMDSpUtx+/Zt+fmff/6J1157DaNHj0ZcXBy+//57LFu2DLNmzdJ7/aRJk9CvXz/ExMRg8ODBGDhwIM6fP1/2H4yIngm2qBBRmfvtt98wfPhwZGVloVmzZujYsSMGDhyI4OBgAPktI+vXr0efPn3k14SFhaFz586YMGGCvG3FihX48MMPcevWLfl177zzDhYtWiQf07p1azRr1gzffvvts/lwRFSm2KJCRGWuX79+uHXrFjZt2oTu3btj7969aNasGZYtW1bsa2JiYjB9+nS5RcbBwQHDhw/H7du3kZmZKR8XGhqq97rQ0FC2qBBVIBxMS0TPhK2tLbp06YIuXbpg0qRJeOuttzBlyhS8/vrrBo9/+PAhpk2bhr59+xo8FxFVDmxRISIhGjZsiIyMDACAtbU1tFqt3v5mzZohPj4ederUKfKwsHj0V9fhw4f1Xnf48GE0aNCg7D8AET0TbFEhojJ1//599O/fH8OGDUNwcDAcHR1x/PhxzJs3D7179waQP/Nn165daNu2LZRKJVxdXTF58mT07NkTvr6+eOmll2BhYYGYmBjExsZi5syZ8vl/+eUXhISEoF27dli5ciWOHj2KH3/8UdTHJSIT42BaIipTGo0GU6dOxY4dO/D3338jNzcXKpUK/fv3x8cffww7Ozts3rwZY8eORUJCAqpXry5PT96+fTumT5+OU6dOwdraGgEBAXjrrbcwfPhwAPmDaRcuXIgNGzZg//798Pb2xty5czFgwACBn5iITIlBhYjKLUOzhYioYuEYFSIiIjJbDCpERERktjiYlojKLfZcE1V8bFEhIiIis8WgQkRERGaLQYWIiIjMFoMKERERmS0GFSIiIjJbDCpERERkthhUiIiIyGwxqBAREZHZYlAhIiIis/X/quH93HT+28sAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "if run_training:\n",
    "    # Plot training loss\n",
    "    plt.plot(dense_loss.values())\n",
    "    plt.xlabel(\"Step\")\n",
    "    plt.ylabel(\"Cross entropy loss\")\n",
    "    plt.title(\"Training loss\")\n",
    "    plt.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generating text\n",
    "\n",
    "Let's see our baseline model in action! We can load a pre-trained checkpoint of the model and use it for inference (or alternatively use the model we trained in the previous section). Note that the pre-trained model used the default `Settings` values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sample_text(\n",
    "    model: CharModel, text: str, num_characters: int, temperature: float = 1.0\n",
    ") -> str:\n",
    "    print(text, end=\"\")\n",
    "    model.train(False)\n",
    "    ids = vocab.str_to_ids(text)\n",
    "\n",
    "    for _ in range(num_characters):\n",
    "        y = model(ids)\n",
    "        new_char = torch.multinomial(\n",
    "            F.softmax(y[-1] / temperature, dim=0), num_samples=1\n",
    "        )\n",
    "        ids = torch.cat((ids, torch.tensor([new_char])), dim=0)\n",
    "        print(vocab.ids_to_str(new_char), end=\"\", flush=True)\n",
    "    print()\n",
    "    return vocab.ids_to_str(ids)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hello worldwide , including the street . \n",
      " \n",
      " = = = Control = = = \n",
      " \n",
      " \n",
      " = = = = Player = = = \n",
      " \n",
      " The park was announced that the post @-@ southern Europe . The activity of the announcement of the battle , it is a construction of the tree , and the first conference in the 1990s and the second @-@ team sector , \n"
     ]
    }
   ],
   "source": [
    "model_from_checkpoint = CharModel(\n",
    "    vocab_size=len(ds.vocab),\n",
    "    hidden_size=128,\n",
    "    expand_factor=4,\n",
    "    kernel_size=7,\n",
    "    num_groups=8,\n",
    "    num_layers=8,\n",
    "    dtype=torch.float32,\n",
    ")\n",
    "model_from_checkpoint.load_state_dict(torch.load(checkpoint_dir / \"dense.pt\"))\n",
    "\n",
    "out = sample_text(\n",
    "    model_from_checkpoint, \"Hello worl\", num_characters=300, temperature=0.4\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also sample from the model we trained in the notebook:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "jupyter": {
     "source_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "# Sample from trained model\n",
    "# out = sample_text(dense_model, \"Hello worl\", num_characters=300, temperature=0.4)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Mixture of experts"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now see how we can turn our regular *dense* model into a *mixture of experts* model that we can run on IPUs and CPUs."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Overview"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the standard mixture of experts setup, we take some of the FFN layers of the model and turn them into sparse MoE layers. This is done by instantiating multiple copies of the basic FFN layer with independent parameters (the experts), and routing the inputs to different experts based on a trained scoring function. For the purposes of this notebook, we will demonstrate how to define a setup closely following the architecture presented in the [Switch Transformers](https://arxiv.org/abs/2101.03961) paper."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p align=\"center\">\n",
    "<img src=\"https://raw.githubusercontent.com/graphcore-research/notebooks/main/moe_charlm/figs/fig2.png\"  width=\"500\" >\n",
    "</p>"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Switch router"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's first tackle the problem of assigning individual tokens (\"characters\" in this notebook) to experts, assuming we have a batch of tokens of shape `(batch_size, hidden_size)` that we want to allocate to `num_experts` experts. The starting point is to calculate a score for each token-expert assignment pair, and use these scores to assign tokens to experts. The optimal assignment problem can therefore be stated as maximising the total score of the assignments, while keeping the constraint that each expert is assigned exactly `expert_capacity = batch_size / num_experts` tokens. As solving this optimisation problem is computationally expensive, the Switch Transformer uses a simplified approach: assign each token to its preferred expert based on the scores, and afterwards make sure that each expert receives the same number of tokens. This is done by randomly disregarding some tokens for the over-assigned experts, and padding the inputs to under-assigned experts with \"dummy\" tokens.\n",
    "\n",
    "In order to train the scoring function throughout training, when the token `i` is routed to the expert `j`, the output of the expert is multiplied by `score[i, j]` which tends to strengthen the \"successful\" token-expert assignments and weaken the \"unsuccessful\" ones."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Scoring function"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the scoring function, we will use a simple linear feed-forward layer producing the output of shape `[batch_size, num_experts]` and normalising so that the scores for each token sum to 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Scorer(nn.Module):\n",
    "    def __init__(self, hidden_size: int, num_experts: int, dtype: torch.dtype):\n",
    "        super().__init__()\n",
    "        self.w = create_weight_tensor((hidden_size, num_experts), dtype)\n",
    "        self.b = create_bias_tensor(num_experts, dtype)\n",
    "        self.dtype = dtype\n",
    "\n",
    "    def forward(self, x: torch.Tensor) -> torch.Tensor:\n",
    "        \"\"\"Return per-token expert scores (normalised so rows sum to 1)\"\"\"\n",
    "        assert len(x.shape) == 2\n",
    "\n",
    "        logits = x @ self.w + self.b\n",
    "        return F.softmax(logits, dim=1, dtype=self.dtype)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Assigning tokens to experts"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output of the `Scorer` module is the scores matrix of shape `[batch_size, num_experts]` where each row adds to 1. We now want to map this to a per-expert assignments tensor `expert_assignments` of shape `[num_experts, expert_capacity]`, so that `expert_assignment[i, :]` contains all token IDs assigned to expert `i`.\n",
    "\n",
    "As the procedure might need to cut off tokens when an expert is over-assigned, we will introduce an additional parameter `capacity_factor` which allows us to increase the per-expert capacity.\n",
    "\n",
    "We write several helper functions in order to tackle this."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Generate assignment matrix"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on previously calculated scores, assign the `top-1` expert to each token and create a binary mask indicating the token-expert assignments:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Assignment matrix based on scores:\n",
      "tensor([[0, 0, 1, 0],\n",
      "        [0, 0, 1, 0],\n",
      "        [1, 0, 0, 0],\n",
      "        [0, 0, 0, 1]])\n"
     ]
    }
   ],
   "source": [
    "def get_expert_mask(scores: torch.Tensor) -> torch.Tensor:\n",
    "    \"\"\"Return expert_mask[i, j] = 1 if token i assigned to expert j, 0 otherwise\"\"\"\n",
    "    token_assignments = torch.argmax(scores, dim=1)\n",
    "    return F.one_hot(token_assignments, scores.shape[1])\n",
    "\n",
    "\n",
    "# 4 tokens, 4 experts\n",
    "scores = torch.tensor(\n",
    "    [\n",
    "        [0.1, 0.2, 0.5, 0.2],\n",
    "        [0, 0, 0.9, 0.1],\n",
    "        [0.4, 0.3, 0.15, 0.15],\n",
    "        [0.2, 0.2, 0.2, 0.4],\n",
    "    ]\n",
    ")\n",
    "\n",
    "expert_mask = get_expert_mask(scores)\n",
    "print(\"Assignment matrix based on scores:\")\n",
    "print(expert_mask)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Assign tokens to experts"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since there are no guarantees the previously generated assignment matrix is balanced (so that all *columns* sum to `expert_capacity`), we will write a function that will ensure this property by only assigning the first `expert_capacity` tokens to an expert and ignoring the rest. Similarly, we assign the \"dummy\" token ID `batch_size` to fill the under-assigned experts (note that the actual token IDs span from 0 to `batch_size - 1`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Final expert assignments:\n",
      "tensor([[2],\n",
      "        [4],\n",
      "        [0],\n",
      "        [3]])\n"
     ]
    }
   ],
   "source": [
    "def mask_to_expert_assignments(\n",
    "    expert_mask: torch.Tensor, capacity: int\n",
    ") -> torch.Tensor:\n",
    "    \"\"\"\n",
    "    Convert token to expert assignment matrix (expert mask) to expert assignments\n",
    "    of shape (num_experts, expert_capacity), discarding tokens for oversubscribed\n",
    "    experts and padding undersubscribed experts\n",
    "    \"\"\"\n",
    "    expert_mask_cumsum = torch.cumsum(expert_mask, dim=0)\n",
    "    capacity_range = torch.arange(\n",
    "        capacity, dtype=expert_mask.dtype, device=expert_mask.device\n",
    "    )\n",
    "    le_capacity = expert_mask_cumsum[:, :, None] <= capacity_range\n",
    "    expert_assignments = le_capacity.sum(dim=0)\n",
    "    return expert_assignments\n",
    "\n",
    "\n",
    "print(\"Final expert assignments:\")\n",
    "print(mask_to_expert_assignments(expert_mask, capacity=1))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example with four tokens and four experts, each expert has the capacity of one token. As no tokens are assigned to expert 1, it gets the dummy token (token 4). Further, since both tokens 0 and 1 were assigned to expert 2, the assignment proceeds in order and token 0 is assigned to the expert, while token 1 remains unassigned. Tokens 2 and 3 are assigned to their preferred experts as there is no competition. You are encouraged to see what happens when you increase the expert capacity."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Load balancing"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One issue with the current procedure is that it creates unbalanced token-expert assignments throughout training. If early in the training an expert receives more tokens, over time it will preferentially receive more tokens due to it being more trained. This can quickly lead to assignments collapsing to only a small subset of the experts, leading to many tokens being cut off and the degradation in performance. In order to tackle this, we introduce a load balancing auxiliary loss that penalises unbalanced assignments.\n",
    "\n",
    "We calculate this loss by averaging the per-token expert scores *before assignment* over the batch dimension (`density_proxy`), as well as calculating the *actual* proportions of tokens received by each expert by averaging `expert_mask` over the batch dimension (`density`). The final loss is then defined as the dot product between the two vectors. In a perfectly balanced situation all experts receive `1 / num_experts` proportion of the tokens. In this case `density[i] = 1 / num_experts` and `density_proxy[i] = 1 / num_experts` for each expert `i`. Therefore, we multiply the dot product with `num_experts` in order to normalise the balancing loss to 1.0 when all experts are balanced."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Balancing loss of a balanced assignment:\n",
      "tensor(1.)\n",
      "Balancing loss of an unbalanced assignment:\n",
      "tensor(1.3000)\n"
     ]
    }
   ],
   "source": [
    "def get_balancing_loss(scores: torch.Tensor, expert_mask: torch.Tensor = None):\n",
    "    if expert_mask is None:\n",
    "        expert_mask = get_expert_mask(scores)\n",
    "    density = expert_mask.float().mean(dim=0)\n",
    "    density_proxy = scores.mean(dim=0)\n",
    "    return torch.dot(density, density_proxy) * density.shape[0]\n",
    "\n",
    "\n",
    "scores_balanced = torch.tensor(\n",
    "    [\n",
    "        [0.7, 0.1, 0.1, 0.1],\n",
    "        [0.1, 0.1, 0.1, 0.7],\n",
    "        [0.1, 0.7, 0.1, 0.1],\n",
    "        [0.1, 0.1, 0.7, 0.1],\n",
    "    ]\n",
    ")\n",
    "scores_unbalanced = torch.tensor(\n",
    "    [\n",
    "        [0.7, 0.1, 0.1, 0.1],\n",
    "        [0.7, 0.1, 0.1, 0.1],\n",
    "        [0.1, 0.7, 0.1, 0.1],\n",
    "        [0.1, 0.1, 0.7, 0.1],\n",
    "    ]\n",
    ")\n",
    "\n",
    "print(\"Balancing loss of a balanced assignment:\")\n",
    "print(get_balancing_loss(scores_balanced))\n",
    "print(\"Balancing loss of an unbalanced assignment:\")\n",
    "print(get_balancing_loss(scores_unbalanced))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Putting it together"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have everything we need to construct a Switch router that will take a batch of tokens and return per-expert token assignments. "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Shuffle tokens"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We make sure we shuffle the tokens before the procedures outlined above to ensure a fair cut-off of tokens (otherwise tokens appearing early in the sequence would have an advantage!)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original tokens:\n",
      "tensor([ 0, 10, 20, 30, 40, 50, 60, 70, 80, 90])\n",
      "Shuffled tokens:\n",
      "(tensor([80, 90, 60, 50,  0, 10, 30, 40, 70, 20]), tensor([8, 9, 6, 5, 0, 1, 3, 4, 7, 2]))\n"
     ]
    }
   ],
   "source": [
    "def shuffle_tokens(\n",
    "    tokens: torch.Tensor, shuffle_idxs: torch.Tensor = None\n",
    ") -> Tuple[torch.Tensor, torch.Tensor]:\n",
    "    \"\"\"Shuffle tokens before assigning them to experts\"\"\"\n",
    "    if shuffle_idxs is None:\n",
    "        shuffle_idxs = torch.randperm(tokens.shape[0], device=tokens.device)\n",
    "    return tokens[shuffle_idxs], shuffle_idxs\n",
    "\n",
    "\n",
    "tokens = torch.arange(10) * 10\n",
    "print(\"Original tokens:\")\n",
    "print(tokens)\n",
    "print(\"Shuffled tokens:\")\n",
    "print(shuffle_tokens(tokens))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Full Switch router"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The final output of our router function should contain `expert_assignments` of shape `[num_experts, expert_capacity]`, corresponding token-expert scores `expert_gates` of the same shape, and `balancing_loss` which we will use during training to ensure balanced token assignments across experts."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We define two different versions of the Switch router, depending on whether the model is run in the `training` or `inference` mode:\n",
    "* `training`: Expert capacity is calculated as `(batch_size / num_experts) * capacity_factor` and we keep track of `balancing_loss` in order to train the router.\n",
    "* `inference`: We set the expert capacity to the maximum number of tokens assigned to an expert, thus ensuring that no tokens are dropped, while disregarding `balancing_loss`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "@dataclass\n",
    "class RouterOutput:\n",
    "    expert_assignments: torch.Tensor\n",
    "    expert_gates: torch.Tensor\n",
    "    balancing_loss: Optional[torch.Tensor]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Expert assignments:\n",
      " tensor([[0],\n",
      "        [2],\n",
      "        [1],\n",
      "        [4]])\n",
      "Corresponding expert gates:\n",
      " tensor([[0.1000],\n",
      "        [0.3000],\n",
      "        [0.9000],\n",
      "        [0.0000]])\n"
     ]
    }
   ],
   "source": [
    "def get_expert_gates(\n",
    "    scores: torch.Tensor, expert_assignments: torch.Tensor\n",
    ") -> torch.Tensor:\n",
    "    pad_token = torch.zeros(\n",
    "        (1, scores.shape[1]), dtype=scores.dtype, device=scores.device\n",
    "    )\n",
    "    scores_padded = torch.cat((scores, pad_token), 0)\n",
    "    return torch.gather(scores_padded.transpose(0, 1), 1, expert_assignments)\n",
    "\n",
    "\n",
    "# Test get_expert_gates\n",
    "scores = torch.tensor(\n",
    "    [\n",
    "        [0.1, 0.2, 0.5, 0.2],\n",
    "        [0, 0, 0.9, 0.1],\n",
    "        [0.4, 0.3, 0.15, 0.15],\n",
    "        [0.2, 0.2, 0.2, 0.4],\n",
    "    ]\n",
    ")\n",
    "\n",
    "expert_assignments = torch.tensor([[0], [2], [1], [4]])\n",
    "print(\"Expert assignments:\\n\", expert_assignments)\n",
    "print(\"Corresponding expert gates:\\n\", get_expert_gates(scores, expert_assignments))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "def switch_router(\n",
    "    scores: torch.Tensor, capacity_factor: float = 1.0, training: bool = True\n",
    ") -> RouterOutput:\n",
    "    expert_mask = get_expert_mask(scores)\n",
    "\n",
    "    # At inference increase capacity so no tokens are dropped\n",
    "    if not training:\n",
    "        capacity = expert_mask.sum(dim=0).max()\n",
    "        expert_assignments = mask_to_expert_assignments(expert_mask, capacity)\n",
    "        expert_gates = get_expert_gates(scores, expert_assignments)\n",
    "        return RouterOutput(expert_assignments, expert_gates, None)\n",
    "\n",
    "    balancing_loss = get_balancing_loss(scores, expert_mask)\n",
    "    expert_mask_shuffled, shuffle_idxs = shuffle_tokens(expert_mask)\n",
    "    batch_size, num_experts = scores.shape\n",
    "    capacity = int(batch_size * capacity_factor / num_experts)\n",
    "    expert_assignments_shuffled = mask_to_expert_assignments(\n",
    "        expert_mask_shuffled, capacity\n",
    "    )\n",
    "    shuffle_idxs_with_pad = torch.cat(\n",
    "        (\n",
    "            shuffle_idxs,\n",
    "            torch.tensor(\n",
    "                [scores.shape[0]], dtype=shuffle_idxs.dtype, device=shuffle_idxs.device\n",
    "            ),\n",
    "        )\n",
    "    )\n",
    "    expert_assignments = shuffle_idxs_with_pad[expert_assignments_shuffled]\n",
    "    expert_gates = get_expert_gates(scores, expert_assignments)\n",
    "    return RouterOutput(expert_assignments, expert_gates, balancing_loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Expert assignments:\n",
      " tensor([[2],\n",
      "        [4],\n",
      "        [1],\n",
      "        [3]])\n",
      "Expert gates:\n",
      " tensor([[0.4000],\n",
      "        [0.0000],\n",
      "        [0.9000],\n",
      "        [0.4000]])\n",
      "Balancing loss:\n",
      " tensor(1.2625)\n"
     ]
    }
   ],
   "source": [
    "# Test switch router (training)\n",
    "scores = torch.tensor(\n",
    "    [\n",
    "        [0.1, 0.2, 0.5, 0.2],\n",
    "        [0.0, 0.0, 0.9, 0.1],\n",
    "        [0.4, 0.3, 0.15, 0.15],\n",
    "        [0.2, 0.2, 0.2, 0.4],\n",
    "    ]\n",
    ")\n",
    "\n",
    "r = switch_router(scores, capacity_factor=1.0)\n",
    "\n",
    "print(\"Expert assignments:\\n\", r.expert_assignments)\n",
    "print(\"Expert gates:\\n\", r.expert_gates)\n",
    "print(\"Balancing loss:\\n\", r.balancing_loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Expert assignments:\n",
      " tensor([[0, 1, 2],\n",
      "        [4, 4, 4],\n",
      "        [4, 4, 4],\n",
      "        [3, 4, 4]])\n",
      "Expert gates:\n",
      " tensor([[0.7000, 0.7000, 0.7000],\n",
      "        [0.0000, 0.0000, 0.0000],\n",
      "        [0.0000, 0.0000, 0.0000],\n",
      "        [0.4000, 0.0000, 0.0000]])\n",
      "Balancing loss:\n",
      " None\n"
     ]
    }
   ],
   "source": [
    "# Test switch router (inference)\n",
    "scores = torch.tensor(\n",
    "    [\n",
    "        [0.7, 0.1, 0.1, 0.1],\n",
    "        [0.7, 0.1, 0.1, 0.1],\n",
    "        [0.7, 0.1, 0.1, 0.1],\n",
    "        [0.2, 0.2, 0.2, 0.4],\n",
    "    ]\n",
    ")\n",
    "\n",
    "r = switch_router(scores, training=False)\n",
    "\n",
    "print(\"Expert assignments:\\n\", r.expert_assignments)\n",
    "print(\"Expert gates:\\n\", r.expert_gates)\n",
    "print(\"Balancing loss:\\n\", r.balancing_loss)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Token routing"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Having calculated the token-expert assignments using the `switch_router` function, we now need to route the tokens to their allocated experts. We consider the general case where the experts are distributed equally across our replicas, so that each replica holds `num_experts_per_replica` experts. The total number of experts is therefore `num_experts = num_replicas * num_experts_per_replica`.\n",
    "\n",
    "When using a single replica (either for the CPU or a single IPU), all experts live on the device and token routing simply gathers the appropriate tokens for each expert. When using multiple replicas however, each replica will hold a different subset of experts and we need cross-device communication to send the tokens to the correct device."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### All-to-all"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When using `num_replicas > 1`, each replica will send a subset of its token batch to each of the other replicas. For this we will use the `all_to_all` op from the `poptorch_experimental_addons` library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "from poptorch_experimental_addons.collectives import all_to_all_single_cross_replica\n",
    "\n",
    "\n",
    "def all_to_all(x: torch.Tensor) -> torch.Tensor:\n",
    "    if poptorch.isRunningOnIpu():\n",
    "        return all_to_all_single_cross_replica(x, x.shape[0])\n",
    "    else:\n",
    "        assert x.shape[0] == 1, \"num_replicas needs to be 1 when running on CPU\"\n",
    "        return x"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following example showcases the `all_to_all` op: we start from the input `x` that is distributed across the `num_replicas` replicas, such that `x[i]` is the tensor on replica `i`. On each replica, the tensor `x[i]` needs to have the shape `[num_replicas, *]` before applying `all_to_all`, so that `x[i][j]` represents the data that replica `i` will send to replica `j`. After the all-to-all operation each replica `i` will have the tensor `y[i]`, such that `y[i][j] = x[j][i]`.\n",
    "\n",
    "Note that by default when we send an input `x` with shape `[batch_size, *]` to a `poptorch_model` object it will be distributed across the replicas so that each replica receives an independent slice of the input that will have the shape `[batch_size // num_replicas, *]`, which is why we do an additional reshape to have the input in this form. Similarly, the output will then have the shape `[batch_size, *]` which we reshape to `[num_replicas, batch_size // num_replicas, *]` in order to separate the outputs coming from each replica."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_all_to_all():\n",
    "    num_replicas = 4\n",
    "\n",
    "    x = np.array(\n",
    "        [\n",
    "            np.arange(1, num_replicas + 1, dtype=np.int32)[:, np.newaxis] * 10**i\n",
    "            for i in range(num_replicas)\n",
    "        ]\n",
    "    )\n",
    "\n",
    "    for i, x_i in enumerate(x):\n",
    "        print(f\"Input on replica {i}:\")\n",
    "        print(x_i)\n",
    "\n",
    "    x = x.reshape(x.shape[0] * x.shape[1], *x.shape[2:])\n",
    "\n",
    "    class AllToAll(nn.Module):\n",
    "        def forward(self, x: torch.Tensor):\n",
    "            return all_to_all(x)\n",
    "\n",
    "    model = AllToAll()\n",
    "    opts = poptorch.Options()\n",
    "    opts.replicationFactor(num_replicas)\n",
    "    poptorch_model = poptorch.inferenceModel(model, opts)\n",
    "    y = poptorch_model(torch.from_numpy(x))\n",
    "\n",
    "    y = y.reshape((num_replicas, num_replicas, -1)).numpy()\n",
    "\n",
    "    for i, y_i in enumerate(y):\n",
    "        print(f\"Output on replica {i}:\")\n",
    "        print(y_i)\n",
    "\n",
    "\n",
    "# Uncomment to test all-to-all on 4 IPUs\n",
    "# test_all_to_all()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Routing tokens to experts"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have the `all_to_all` communication, we can route the tokens to the appropriate experts based on the previously calculated `expert_assignments`. We need to ensure that the tokens have the correct shape before the `all_to_all` operation of [`num_replicas, num_experts_per_replica, expert_capacity, hidden_size`], as well as add the padding token for the under-assigned experts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tokens:\n",
      " tensor([[ 0],\n",
      "        [ 2],\n",
      "        [ 4],\n",
      "        [ 6],\n",
      "        [ 8],\n",
      "        [10],\n",
      "        [12],\n",
      "        [14]])\n",
      "Expert assignments:\n",
      " tensor([[1, 3],\n",
      "        [0, 7],\n",
      "        [5, 6],\n",
      "        [2, 4]])\n",
      "Routing output:\n",
      " tensor([[[[ 2],\n",
      "          [ 6]],\n",
      "\n",
      "         [[ 0],\n",
      "          [14]],\n",
      "\n",
      "         [[10],\n",
      "          [12]],\n",
      "\n",
      "         [[ 4],\n",
      "          [ 8]]]])\n"
     ]
    }
   ],
   "source": [
    "def route_to_replica(\n",
    "    x: torch.Tensor, expert_assignments: torch.Tensor, num_replicas: int\n",
    ") -> torch.Tensor:\n",
    "    assert len(x.shape) == 2\n",
    "\n",
    "    pad_token = torch.zeros((1, x.shape[1]), dtype=x.dtype, device=x.device)\n",
    "    x_padded = torch.cat((x, pad_token), 0)\n",
    "    x_gathered = x_padded[\n",
    "        expert_assignments.reshape(\n",
    "            num_replicas,\n",
    "            expert_assignments.shape[0] // num_replicas,\n",
    "            *expert_assignments.shape[1:]\n",
    "        ),\n",
    "        :,\n",
    "    ]\n",
    "    return all_to_all(x_gathered)\n",
    "\n",
    "\n",
    "# Test route\n",
    "tokens = (torch.arange(8) * 2).reshape(-1, 1)\n",
    "print(\"Tokens:\\n\", tokens)\n",
    "e_assignments = torch.tensor([[1, 3], [0, 7], [5, 6], [2, 4]])\n",
    "print(\"Expert assignments:\\n\", e_assignments)\n",
    "out = route_to_replica(tokens, e_assignments, num_replicas=1)\n",
    "out_expected = torch.tensor([[[2], [6]], [[0], [14]], [[10], [12]], [[4], [8]]])[\n",
    "    None, :, :, :\n",
    "]\n",
    "\n",
    "print(\"Routing output:\\n\", out)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Rearrange expert inputs"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After the `all_to_all` operation, each replica has a different set of inputs of the shape `x_routed = [num_replicas, num_experts_per_replica, expert_capacity, hidden_size]`. To support multiple experts per replica, within each replica we will run the experts concurrently using block-diagonal matrix multiplication for which we need to re-arrange the inputs to `x_allocated = [num_experts_per_replica, expert_capacity * num_replicas, hidden_size]`. We need to be careful when reshaping because we must ensure that `x_allocated[i]` collects all tokens sent to expert `i` of the replica (`x_allocated[i] = x[:][i]`). For this, we need two functions: one to re-arrange expert inputs after `all_to_all`, and one to re-arrange the inputs back to the original form to be sent back after expert processing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input before rearranging:\n",
      " tensor([[[[0]],\n",
      "\n",
      "         [[1]]],\n",
      "\n",
      "\n",
      "        [[[2]],\n",
      "\n",
      "         [[3]]]])\n",
      "Input shape is (num_replicas, num_experts_per_replica, expert_capacity, hidden_size):\n",
      " torch.Size([2, 2, 1, 1])\n",
      "Rearranging to (num_experts_per_replica, num_replicas * expert_capacity, hidden_size):\n",
      "tensor([[[0],\n",
      "         [2]],\n",
      "\n",
      "        [[1],\n",
      "         [3]]])\n",
      "Shape of the output:\n",
      " torch.Size([2, 2, 1])\n",
      "Inverting the re-arrangement:\n",
      " tensor([[[[0]],\n",
      "\n",
      "         [[1]]],\n",
      "\n",
      "\n",
      "        [[[2]],\n",
      "\n",
      "         [[3]]]])\n"
     ]
    }
   ],
   "source": [
    "def route_to_block(x: torch.Tensor) -> torch.Tensor:\n",
    "    assert len(x.shape) == 4\n",
    "    return x.swapaxes(0, 1).reshape(x.shape[1], -1, x.shape[-1])\n",
    "\n",
    "\n",
    "def unroute_from_block(x: torch.Tensor, num_replicas: int) -> torch.Tensor:\n",
    "    assert len(x.shape) == 3\n",
    "    return x.reshape(x.shape[0], num_replicas, -1, x.shape[-1]).swapaxes(0, 1)\n",
    "\n",
    "\n",
    "# Test allocate-deallocate replica experts\n",
    "x = torch.tensor([[[[0]], [[1]]], [[[2]], [[3]]]])\n",
    "print(\"Input before rearranging:\\n\", x)\n",
    "print(\n",
    "    \"Input shape is (num_replicas, num_experts_per_replica, expert_capacity, hidden_size):\\n\",\n",
    "    x.shape,\n",
    ")\n",
    "y = route_to_block(x)\n",
    "print(\n",
    "    \"Rearranging to (num_experts_per_replica, num_replicas * expert_capacity, hidden_size):\"\n",
    ")\n",
    "y_expected = np.array([[[0], [2]], [[1], [3]]])\n",
    "print(y)\n",
    "print(\"Shape of the output:\\n\", y.shape)\n",
    "_x = unroute_from_block(y, num_replicas=2)\n",
    "print(\"Inverting the re-arrangement:\\n\", _x)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Routing back"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After the tokens have passed through the expert layers, they need to be rearranged and routed back to their original devices (so the inverse operation of `route`). One addition here is that we need to multiply the expert outputs by the appropriate token-expert score so that we introduce the training signal for the router."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tokens to be routed back:\n",
      " tensor([[[[ 2.],\n",
      "          [ 6.]],\n",
      "\n",
      "         [[ 0.],\n",
      "          [14.]],\n",
      "\n",
      "         [[10.],\n",
      "          [12.]],\n",
      "\n",
      "         [[ 4.],\n",
      "          [ 8.]]]])\n",
      "Based on expert assignments:\n",
      " tensor([[1, 3],\n",
      "        [0, 3],\n",
      "        [5, 6],\n",
      "        [2, 1]])\n",
      "Tokens after unroute (no gating):\n",
      " tensor([[ 0.],\n",
      "        [10.],\n",
      "        [ 4.],\n",
      "        [20.],\n",
      "        [ 0.],\n",
      "        [10.],\n",
      "        [12.],\n",
      "        [ 0.]])\n",
      "\n",
      "Same expert assignments but with corresponding expert gates:\n",
      " tensor([[1.0000, 1.0000],\n",
      "        [1.0000, 1.0000],\n",
      "        [0.3000, 1.0000],\n",
      "        [1.0000, 0.5000]])\n",
      "Tokens after unrouted (with gating):\n",
      " tensor([[ 0.],\n",
      "        [ 6.],\n",
      "        [ 4.],\n",
      "        [20.],\n",
      "        [ 0.],\n",
      "        [ 3.],\n",
      "        [12.],\n",
      "        [ 0.]])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_33166/307596277.py:18: UserWarning: scatter_reduce() is in beta and the API may change at any time. (Triggered internally at ../aten/src/ATen/native/TensorAdvancedIndexing.cpp:1615.)\n",
      "  return torch.scatter_reduce(\n"
     ]
    }
   ],
   "source": [
    "def unroute_from_replica(\n",
    "    x: torch.Tensor,\n",
    "    expert_assignments: torch.Tensor,\n",
    "    batch_size: int,\n",
    "    expert_gates: Optional[torch.Tensor] = None,\n",
    ") -> torch.Tensor:\n",
    "    assert len(x.shape) == 4\n",
    "\n",
    "    hidden_size = x.shape[-1]\n",
    "    x_unrouted = all_to_all(\n",
    "        x\n",
    "    )  # (num_replicas, num_experts_per_replica, expert_capacity, hidden_size)\n",
    "    if expert_gates is not None:\n",
    "        x_unrouted *= expert_gates.reshape(*x_unrouted.shape[:-1], 1)\n",
    "    x_flat = x_unrouted.reshape(-1, hidden_size)\n",
    "    expert_assignments_flat = expert_assignments.reshape(-1, 1)\n",
    "    y = torch.zeros((batch_size + 1, hidden_size), dtype=x.dtype, device=x.device)\n",
    "    return torch.scatter_reduce(\n",
    "        y, 0, expert_assignments_flat.expand(-1, hidden_size), x_flat, \"sum\"\n",
    "    )[:-1]\n",
    "\n",
    "\n",
    "# Test unroute without expert gates\n",
    "x = torch.tensor(\n",
    "    [[[2], [6]], [[0], [14]], [[10], [12]], [[4], [8]]], dtype=torch.float32\n",
    ")[None, :, :, :]\n",
    "print(\"Tokens to be routed back:\\n\", x)\n",
    "e_assignments = torch.tensor([[1, 3], [0, 3], [5, 6], [2, 1]])\n",
    "print(\"Based on expert assignments:\\n\", e_assignments)\n",
    "y = unroute_from_replica(x, e_assignments, batch_size=np.prod(x.shape[:-1]))\n",
    "y_expected = torch.tensor([0, 2 + 8, 4, 6 + 14, 0, 10, 12, 0]).reshape(-1, 1)\n",
    "print(\"Tokens after unroute (no gating):\\n\", y)\n",
    "\n",
    "# Test unroute with expert gates\n",
    "x = torch.tensor(\n",
    "    [[[2], [6]], [[0], [14]], [[10], [12]], [[4], [8]]], dtype=torch.float32\n",
    ")[None, :, :, :]\n",
    "e_assignments = torch.tensor([[1, 3], [0, 3], [5, 6], [2, 1]])\n",
    "expert_gates = torch.tensor([[1, 1], [1, 1], [0.3, 1], [1, 0.5]])\n",
    "print(\"\\nSame expert assignments but with corresponding expert gates:\\n\", expert_gates)\n",
    "y = unroute_from_replica(\n",
    "    x, e_assignments, batch_size=np.prod(x.shape[:-1]), expert_gates=expert_gates\n",
    ")\n",
    "y_expected = torch.tensor([0, 2 + 0.5 * 8, 4, 6 + 14, 0, 0.3 * 10, 12, 0]).reshape(\n",
    "    -1, 1\n",
    ")\n",
    "print(\"Tokens after unrouted (with gating):\\n\", y)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating an MoE layer"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now use the routing functions we defined to create expert-parallel layers. Let's create `ExpertFFN`, an expert-parallel version of our previous `FFN` layer. For each weight of the original layer of shape `w_shape` we now initialise an expert-parallel version instead, with shape `[num_replicas, num_experts_per_replica, *w_shape]`. In the case of the `FFN` layer, these are the weights `w1` and `w2`.\n",
    "\n",
    "Note that each replica will only receive its own shard of this weight, so within the `forward` method of our class the shape will actually be `[num_experts_per_replica, *w_shape]` without the leading dimension. We need to tell `poptorch` during model definition which weights will be sharded in this way, as by default the same weight values are *replicated* for each replica. We get rid of the leading `num_replicas` dimension in the case when `num_replicas == 1`.\n",
    "\n",
    "We can now conduct our matrix multiplications as *batched* matmuls, as our inputs will have the same leading dimension `num_experts_per_replica` after the routing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "class ExpertFFN(nn.Module):\n",
    "    \"\"\"Transformer-style FFN layer with expert-parallelism\"\"\"\n",
    "\n",
    "    def __init__(\n",
    "        self,\n",
    "        hidden_size: int,\n",
    "        expand_factor: int,\n",
    "        dtype: torch.dtype,\n",
    "        num_replicas: int,\n",
    "        num_experts_per_replica: int,\n",
    "        capacity_factor: float,\n",
    "    ):\n",
    "        super().__init__()\n",
    "        self.hidden_size = hidden_size\n",
    "        self.expand_factor = expand_factor\n",
    "        self.dtype = dtype\n",
    "        self.num_replicas = num_replicas\n",
    "        self.num_experts_per_replica = num_experts_per_replica\n",
    "        self.capacity_factor = capacity_factor\n",
    "        self.scorer = Scorer(hidden_size, num_replicas * num_experts_per_replica, dtype)\n",
    "        self.balancing_loss = None\n",
    "\n",
    "        # Create expert params\n",
    "        if num_replicas > 1:\n",
    "            e_shape = (num_replicas, num_experts_per_replica)\n",
    "        else:\n",
    "            e_shape = (num_experts_per_replica,)\n",
    "\n",
    "        self.w1 = create_weight_tensor(\n",
    "            (\n",
    "                *e_shape,\n",
    "                hidden_size,\n",
    "                hidden_size * expand_factor,\n",
    "            ),\n",
    "            dtype,\n",
    "        )\n",
    "        self.w2 = create_weight_tensor(\n",
    "            (\n",
    "                *e_shape,\n",
    "                hidden_size * expand_factor,\n",
    "                hidden_size,\n",
    "            ),\n",
    "            dtype,\n",
    "        )\n",
    "\n",
    "        # Keep track of router decisions\n",
    "        self.expert_ids = None\n",
    "\n",
    "    def forward(self, x: torch.Tensor) -> torch.Tensor:\n",
    "        # Make sure inputs are of shape (batch_size, hidden_size)\n",
    "        x_shape = x.shape\n",
    "        x = x.reshape(-1, x_shape[-1])\n",
    "        batch_size = x.shape[0]\n",
    "\n",
    "        # Assign tokens to experts\n",
    "        probs = self.scorer(x)\n",
    "        self.expert_ids = probs.argmax(dim=1)\n",
    "        r = switch_router(probs, self.capacity_factor, self.training)\n",
    "        self.balancing_loss = r.balancing_loss\n",
    "\n",
    "        # Route + process + unroute\n",
    "        x_routed = route_to_replica(x, r.expert_assignments, self.num_replicas)\n",
    "        x_allocated = route_to_block(x_routed)\n",
    "        y = F.relu(x_allocated @ self.w1) @ self.w2  # batched matmul\n",
    "        y_deallocated = unroute_from_block(y, self.num_replicas)\n",
    "        return unroute_from_replica(\n",
    "            y_deallocated, r.expert_assignments, batch_size, r.expert_gates\n",
    "        ).reshape(x_shape)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's also create a function that will take a regular dense `FFN` and return an expert-parallel `ExpertFFN` that we can later use to turn dense layers into MoE layers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "def moe_ffn_like(\n",
    "    ffn: FFN, num_replicas: int, num_experts_per_replica: int, capacity_factor: float\n",
    "):\n",
    "    return ExpertFFN(\n",
    "        hidden_size=ffn.hidden_size,\n",
    "        expand_factor=ffn.expand_factor,\n",
    "        dtype=ffn.dtype,\n",
    "        num_replicas=num_replicas,\n",
    "        num_experts_per_replica=num_experts_per_replica,\n",
    "        capacity_factor=capacity_factor,\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Test expert ffn\n",
    "ffn = FFN(hidden_size=1, expand_factor=4, dtype=torch.float32)\n",
    "e_ffn = moe_ffn_like(\n",
    "    ffn, num_replicas=1, num_experts_per_replica=5, capacity_factor=1.0\n",
    ")\n",
    "\n",
    "\n",
    "# Test shapes\n",
    "assert e_ffn.w1.shape == (5, 1, 1 * 4)\n",
    "assert e_ffn.w2.shape == (5, 1 * 4, 1)\n",
    "assert e_ffn.scorer.w.shape == (1, 5)\n",
    "assert e_ffn.scorer.b.shape == (5,)\n",
    "x = torch.randn(15, 1)\n",
    "y = e_ffn(x)\n",
    "assert y.shape == x.shape"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MoE character model"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now update our original character language model by replacing some of the `FFN` layers with our new `ExpertFFN` layers! A common approach to \"sparsifying\" a dense model is to take every other layer and make it MoE - so let's do that.\n",
    "\n",
    "In addition to exchanging some `FFN` layers with `ExpertFFN` layers, we need to ensure that the expert balancing loss is included in the final loss calculation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MoeCharModel(CharModel):\n",
    "    def __init__(\n",
    "        self,\n",
    "        vocab_size: int,\n",
    "        hidden_size: int,\n",
    "        expand_factor: int,\n",
    "        kernel_size: int,\n",
    "        num_groups: int,\n",
    "        num_layers: int,\n",
    "        dtype: torch.dtype,\n",
    "        num_replicas: int,\n",
    "        num_experts_per_replica: int,\n",
    "        capacity_factor: float,\n",
    "        moe_layers: Tuple[int, ...],\n",
    "        aux_loss_weight: float,\n",
    "    ):\n",
    "        super().__init__(\n",
    "            vocab_size=vocab_size,\n",
    "            hidden_size=hidden_size,\n",
    "            expand_factor=expand_factor,\n",
    "            kernel_size=kernel_size,\n",
    "            num_groups=num_groups,\n",
    "            num_layers=num_layers,\n",
    "            dtype=dtype,\n",
    "        )\n",
    "        self.num_replicas = num_replicas\n",
    "        self.num_experts_per_replica = num_experts_per_replica\n",
    "        self.num_experts = num_replicas * num_experts_per_replica\n",
    "        self.capacity_factor = capacity_factor\n",
    "        self.moe_layers = moe_layers\n",
    "        self.aux_loss_weight = aux_loss_weight\n",
    "\n",
    "        # Convert layers to MoE\n",
    "        for i in moe_layers:\n",
    "            self.core_layers[i].ffn = moe_ffn_like(\n",
    "                self.core_layers[i].ffn,\n",
    "                num_replicas,\n",
    "                num_experts_per_replica,\n",
    "                capacity_factor,\n",
    "            )\n",
    "\n",
    "    def forward(\n",
    "        self, x: torch.Tensor, mask: torch.Tensor = None, y: torch.Tensor = None\n",
    "    ) -> torch.Tensor:\n",
    "        out = super().forward(x, mask, y)\n",
    "        if self.training:\n",
    "            logits, cross_entropy_loss = out\n",
    "            aux_losses = {\n",
    "                i: self.core_layers[i].ffn.balancing_loss\n",
    "                for i in self.moe_layers\n",
    "                if self.core_layers[i].ffn.balancing_loss is not None\n",
    "            }\n",
    "            aux_loss = sum(aux_losses.values())\n",
    "            loss = cross_entropy_loss + self.aux_loss_weight * aux_loss\n",
    "            return {\n",
    "                \"logits\": logits,\n",
    "                \"cross entropy loss\": cross_entropy_loss,\n",
    "                \"aux losses\": aux_losses,\n",
    "            }, poptorch.identity_loss(loss, reduction=\"none\")\n",
    "        else:\n",
    "            return out"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training the MoE model"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we can train our new model. We need to ensure that our expert parameters are properly sharded across the replicas. We do this by setting the appropriate communication groups for the parameters of `ExpertFFN` layers, but making sure that the `Scorer` parameters are kept replicated across the replicas (as the router should be shared across each device)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "def mean_across_replicas(out: dict, num_replicas: int) -> dict:\n",
    "    \"\"\"Average output values across replicas\"\"\"\n",
    "    if num_replicas == 1:\n",
    "        return out\n",
    "\n",
    "    return {\n",
    "        key: (\n",
    "            torch.mean(\n",
    "                val.reshape(\n",
    "                    (num_replicas, val.shape[0] // num_replicas, *val.shape[1:])\n",
    "                ).squeeze(dim=-1),\n",
    "                dim=0,\n",
    "                dtype=torch.float32,\n",
    "            )\n",
    "            if type(val) is not dict\n",
    "            else mean_across_replicas(val, num_replicas)\n",
    "        )\n",
    "        for key, val in out.items()\n",
    "    }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_moe(model: CharModel, data: Dataset, settings: Settings):\n",
    "    model.train()\n",
    "\n",
    "    # Set the optimizer\n",
    "    if settings.device.run_on_ipu:\n",
    "        opts = poptorch.Options()\n",
    "        opts.replicationFactor(settings.device.num_replicas)\n",
    "        optimizer = poptorch.optim.AdamW(model.parameters(), lr=settings.training.lr)\n",
    "        poptorch_model = poptorch.trainingModel(model, opts, optimizer)\n",
    "\n",
    "        # Shard MoE params across replicas\n",
    "        if settings.device.num_replicas > 1:\n",
    "            for i in model.moe_layers:\n",
    "                for p_name, _ in model.core_layers[i].ffn.named_parameters():\n",
    "                    # Router params should stay replicated\n",
    "                    if \"scorer\" not in p_name:\n",
    "                        poptorch_model.per_replica_params[f\"core_layers.{i}.ffn.\" + p_name] = (\n",
    "                            poptorch.enums.CommGroupType.NoGrouping,\n",
    "                            0,\n",
    "                            poptorch.enums.VariableRetrievalMode.OnePerGroup,\n",
    "                        )\n",
    "    else:\n",
    "        optimizer = torch.optim.AdamW(model.parameters(), lr=settings.training.lr)\n",
    "\n",
    "    # Training loop\n",
    "    lr = settings.training.lr\n",
    "    lr_decay_factor = 2**-settings.training.lr_decay\n",
    "    training_loss = {}\n",
    "    for step, batch in zip(\n",
    "        range(settings.training.num_steps),\n",
    "        data.batch(\n",
    "            settings.batch.batch_size,\n",
    "            settings.batch.sequence_length,\n",
    "            settings.batch.overlap_length,\n",
    "            settings.batch.batch_seed,\n",
    "        ),\n",
    "    ):\n",
    "        # Apply a step\n",
    "        if settings.device.run_on_ipu:\n",
    "            out, loss = poptorch_model(\n",
    "                batch[\"x\"].int(), batch[\"mask\"], batch[\"y\"].int()\n",
    "            )\n",
    "            out = mean_across_replicas(out, settings.device.num_replicas)\n",
    "        else:\n",
    "            optimizer.zero_grad()\n",
    "            out, loss = model(batch[\"x\"], batch[\"mask\"], batch[\"y\"])\n",
    "            loss.backward()\n",
    "            optimizer.step()\n",
    "\n",
    "        training_loss[step + 1] = out['cross entropy loss'].detach()\n",
    "\n",
    "        # Print results\n",
    "        if (step + 1) % 2**8 == 0:\n",
    "            clear_output(wait=True)\n",
    "            if step + 1 < settings.training.num_steps:\n",
    "                print(f\"*** Running moe ({model.num_experts} experts) training ***\")\n",
    "            else:\n",
    "                print(f\"*** Moe ({model.num_experts} experts) training finished ***\")\n",
    "            print(f\"Step {step + 1}/{settings.training.num_steps}:\")\n",
    "            print(f\"Loss: {out['cross entropy loss']}\")\n",
    "            print(f\"Expert balancing losses: {out['aux losses']}\")\n",
    "\n",
    "        # Learning rate schedule\n",
    "        lr *= lr_decay_factor\n",
    "        optimizer.param_groups[0][\"lr\"] = lr\n",
    "        if settings.device.run_on_ipu:\n",
    "            poptorch_model.setOptimizer(optimizer)\n",
    "\n",
    "    if settings.device.run_on_ipu:\n",
    "        poptorch_model.destroy()\n",
    "\n",
    "    return model, training_loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "*** Moe (16 experts) training finished ***\n",
      "Step 8192/8192:\n",
      "Loss: 1.2617719173431396\n",
      "Expert balancing losses: {0: tensor(1.0100), 2: tensor(1.0290), 4: tensor(1.0121), 6: tensor(1.0048)}\n"
     ]
    }
   ],
   "source": [
    "if run_training:\n",
    "    # Save models and loss here\n",
    "    runs = {}\n",
    "\n",
    "    # Create the model\n",
    "\n",
    "    # Set random seed\n",
    "    seed = None\n",
    "    if seed is None:\n",
    "        seed = np.random.SeedSequence().generate_state(1)[0]\n",
    "    np.random.seed(seed)\n",
    "    torch.manual_seed(seed)\n",
    "\n",
    "    # Set the training settings\n",
    "    settings = Settings()\n",
    "    settings.training.num_steps = 2**13\n",
    "\n",
    "    for num_experts in [4, 8, 16]:\n",
    "        model = MoeCharModel(\n",
    "            vocab_size=len(ds.vocab),\n",
    "            hidden_size=128,\n",
    "            expand_factor=4,\n",
    "            kernel_size=7,\n",
    "            num_groups=8,\n",
    "            num_layers=8,\n",
    "            dtype=torch.float32,\n",
    "            num_replicas=settings.device.num_replicas,\n",
    "            num_experts_per_replica=num_experts // settings.device.num_replicas,\n",
    "            capacity_factor=1.0,\n",
    "            moe_layers=(0, 2, 4, 6),\n",
    "            aux_loss_weight=0.01,\n",
    "        )\n",
    "        model, loss = train_moe(model, train_ds, settings)\n",
    "        runs[f\"moe-{num_experts}\"] = {\"model\": model, \"loss\": loss}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAioAAAHHCAYAAACRAnNyAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABn9klEQVR4nO3dd3gUVdsG8Ht2k2x6Ib0RQq8JhCIhICi9idKkiIDIq/T+SUdqAAsoICoioIBUaUpvQZDeew0QIBBKes/u+f6IGViyQDYkmQ25f9eV1+zM2dlnd3jZm3POnJGEEAJEREREJkildAFEREREL8KgQkRERCaLQYWIiIhMFoMKERERmSwGFSIiIjJZDCpERERkshhUiIiIyGQxqBAREZHJYlAhIiIik8WgQkS50qNHD5QoUSJXz/3yyy8hSVLeFpRDr1M3ERU8BhWiN4wkSTn62bt3r9KlEhG9ksR7/RC9WZYuXar3+LfffsOOHTvw+++/621v3Lgx3N3dc/066enp0Ol00Gg0Rj83IyMDGRkZsLS0zPXr51aPHj2wd+9e3Lx5s8Bfm4iMZ6Z0AUSUtz766CO9x4cOHcKOHTuybX9eUlISrK2tc/w65ubmuaoPAMzMzGBmxr9+iOjVOPRDVAQ1aNAAlStXxvHjx/H222/D2toao0ePBgBs2LABLVu2hJeXFzQaDUqVKoXJkydDq9XqHeP5uR43b96EJEn4+uuv8fPPP6NUqVLQaDSoWbMmjh49qvdcQ3NUJElC//79sX79elSuXBkajQaVKlXC1q1bs9W/d+9e1KhRA5aWlihVqhR++umn15r3kpiYiGHDhsHX1xcajQblypXD119/jec7nHfs2IG6devC0dERtra2KFeunPy5ZZkzZw4qVaoEa2trODk5oUaNGli+fHmu6iIi9qgQFVmPHz9G8+bN0alTJ3z00UfyMNDixYtha2uLoUOHwtbWFrt378b48eMRFxeHr7766pXHXb58OeLj4/HZZ59BkiTMnDkTbdu2xY0bN17ZC7N//378+eef6Nu3L+zs7PD999+jXbt2uH37NpydnQEAJ0+eRLNmzeDp6YmJEydCq9Vi0qRJcHV1zdXnIITAe++9hz179qBXr16oWrUqtm3bhhEjRuDu3buYNWsWAOD8+fNo1aoVAgICMGnSJGg0Gly7dg0HDhyQj7VgwQIMHDgQ7du3x6BBg5CSkoIzZ87g8OHD6NKlS67qIyryBBG90fr16yee/796/fr1BQDx448/ZmuflJSUbdtnn30mrK2tRUpKiryte/fuws/PT34cHh4uAAhnZ2fx5MkTefuGDRsEALFp0yZ524QJE7LVBEBYWFiIa9euydtOnz4tAIg5c+bI21q3bi2sra3F3bt35W1Xr14VZmZm2Y5pyPN1r1+/XgAQU6ZM0WvXvn17IUmSXM+sWbMEAPHw4cMXHrtNmzaiUqVKr6yBiHKOQz9ERZRGo0HPnj2zbbeyspJ/j4+Px6NHj1CvXj0kJSXh0qVLrzzuhx9+CCcnJ/lxvXr1AAA3btx45XMbNWqEUqVKyY8DAgJgb28vP1er1WLnzp14//334eXlJbcrXbo0mjdv/srjG7J582ao1WoMHDhQb/uwYcMghMCWLVsAAI6OjgAyh8Z0Op3BYzk6OuLOnTvZhrqIKPcYVIiKKG9vb1hYWGTbfv78eXzwwQdwcHCAvb09XF1d5Ym4sbGxrzxu8eLF9R5nhZbo6Gijn5v1/KznRkVFITk5GaVLl87WztC2nLh16xa8vLxgZ2ent71ChQryfiAzgIWEhODTTz+Fu7s7OnXqhFWrVumFli+++AK2traoVasWypQpg379+ukNDRGR8RhUiIqoZ3tOssTExKB+/fo4ffo0Jk2ahE2bNmHHjh2YMWMGALywJ+FZarXa4HaRg5UQXue5+c3Kygr79u3Dzp070a1bN5w5cwYffvghGjduLE80rlChAi5fvowVK1agbt26WLt2LerWrYsJEyYoXD1R4cWgQkSyvXv34vHjx1i8eDEGDRqEVq1aoVGjRnpDOUpyc3ODpaUlrl27lm2foW054efnh3v37iE+Pl5ve9Ywl5+fn7xNpVKhYcOG+Pbbb3HhwgVMnToVu3fvxp49e+Q2NjY2+PDDD7Fo0SLcvn0bLVu2xNSpU5GSkpKr+oiKOgYVIpJl9Wg824ORlpaGH374QamS9KjVajRq1Ajr16/HvXv35O3Xrl2T55IYq0WLFtBqtZg7d67e9lmzZkGSJHnuy5MnT7I9t2rVqgCA1NRUAJlXUj3LwsICFStWhBAC6enpuaqPqKjj5clEJKtTpw6cnJzQvXt3DBw4EJIk4ffffzeJoZcsX375JbZv346QkBD06dNHDhmVK1fGqVOnjD5e69at8c4772DMmDG4efMmAgMDsX37dmzYsAGDBw+WJ/dOmjQJ+/btQ8uWLeHn54eoqCj88MMP8PHxQd26dQEATZo0gYeHB0JCQuDu7o6LFy9i7ty5aNmyZbY5MESUMwwqRCRzdnbGX3/9hWHDhmHs2LFwcnLCRx99hIYNG6Jp06ZKlwcAqF69OrZs2YLhw4dj3Lhx8PX1xaRJk3Dx4sUcXZX0PJVKhY0bN2L8+PFYuXIlFi1ahBIlSuCrr77CsGHD5Hbvvfcebt68iV9//RWPHj2Ci4sL6tevj4kTJ8LBwQEA8Nlnn2HZsmX49ttvkZCQAB8fHwwcOBBjx47Ns/dPVNTwXj9E9EZ4//33cf78eVy9elXpUogoD3GOChEVOsnJyXqPr169is2bN6NBgwbKFERE+YY9KkRU6Hh6eqJHjx4oWbIkbt26hfnz5yM1NRUnT55EmTJllC6PiPIQ56gQUaHTrFkz/PHHH7h//z40Gg2Cg4Mxbdo0hhSiNxB7VIiIiMhkcY4KERERmSwGFSIiIjJZhXqOik6nw71792BnZwdJkpQuh4iIiHJACIH4+Hh4eXlBpXp5n0mhDir37t2Dr6+v0mUQERFRLkRERMDHx+elbQp1UMlakjoiIgL29vYKV0NEREQ5ERcXB19f3xzdWqJQB5Ws4R57e3sGFSIiokImJ9M2OJmWiIiITBaDChEREZksBhUiIiIyWYV6jgoREVEWrVaL9PR0pcsgAObm5lCr1XlyLAYVIiIq1IQQuH//PmJiYpQuhZ7h6OgIDw+P117njEGFiIgKtayQ4ubmBmtray4AqjAhBJKSkhAVFQUg827nr4NBhYiICi2tViuHFGdnZ6XLof9YWVkBAKKiouDm5vZaw0CcTEtERIVW1pwUa2trhSuh52Wdk9edN8SgQkREhR6He0xPXp0TBhUiIiIyWQwqREREZLIYVIiIiAgAcO3aNdjZ2cHR0VHpUmQMKgYkpWXgTnQSHsanKl0KERFRgUhPT0fnzp1Rr149pUvRw6BiwB9bV2PixA6Y9M0ApUshIqI3VIMGDTBgwAAMHjwYTk5OcHd3x4IFC5CYmIiePXvCzs4OpUuXxpYtW+TnhIWFoVatWtBoNPD09MTIkSORkZEh79fpdAgNDYW/vz+srKwQGBiINWvW5KiesWPHonz58ujYsWOev9fXwaBigP2xjRi69QaaHTigdClERGQkIQSS0jIK/EcIYXStS5YsgYuLC44cOYIBAwagT58+6NChA+rUqYMTJ06gSZMm6NatG5KSknD37l20aNECNWvWxOnTpzF//nwsXLgQU6ZMkY8XGhqK3377DT/++CPOnz+PIUOG4KOPPkJYWNhL69i9ezdWr16NefPmGf0e8hsXfDNAgvTff4mIqLBJTtei4vhtBf66FyY1hbWFcV+rgYGBGDt2LABg1KhRmD59OlxcXNC7d28AwPjx4zF//nycOXMGmzZtgq+vL+bOnQtJklC+fHncu3cPX3zxBcaPH4/09HRMmzYNO3fuRHBwMACgZMmS2L9/P3766SfUr1/fYA2PHz9Gjx49sHTpUtjb27/GJ5A/GFQMEFkJxfhwTERElGMBAQHy72q1Gs7OzqhSpYq8zd3dHUDmCq8XL15EcHCw3vokISEhSEhIwJ07dxAfH4+kpCQ0btxY7zXS0tJQrVo1AEClSpVw69YtAEC9evWwZcsW9O7dG126dMHbb7+db+/zdTCoGPD0DwGTChFRYWNlrsaFSU0VeV1jmZub6z2WJElvW9b3kU6ne+WxEhISAAB///03vL299fZpNBoAwObNm+WVYrOWud+9ezc2btyIr7/+GkDm0JlOp4OZmRl+/vlnfPLJJ0a/r7zEoGIQB32IiAorSZKMHoIpDCpUqIC1a9dCCCEHmAMHDsDOzg4+Pj5wcnKCRqPB7du3XzjM4+fnl23bwYMHodVq5ccbNmzAjBkz8O+//2YLPEp4885kHpLYoUJERCaib9++mD17NgYMGID+/fvj8uXLmDBhAoYOHQqVSgU7OzsMHz4cQ4YMgU6nQ926dREbG4sDBw7A3t4e3bt3N3jcChUq6D0+duwYVCoVKleuXBBv65UYVAxR8WIoIiIyLd7e3ti8eTNGjBiBwMBAFCtWDL169ZIn4wLA5MmT4erqitDQUNy4cQOOjo4ICgrC6NGjFaz89UgiN9dTmYi4uDg4ODggNjY2T2cqr530MSouP4qb3hKa77qQZ8clIqK8lZKSgvDwcPj7+8PS0lLpcugZLzs3xnx/s+vAAOm5/xIREZEyGFQMEFlX/RTaviYiIqI3A4OKISr2pRAREZkCBhUDJA76EBERmQQGFUP+G/rh5clERETKYlAxKDOoMKcQEREpi0HFEIk3JSQiIjIFDCovwy4VIiIiRTGoGCBJ/FiIiIhMAb+RDeHQDxERkUlgUHkZDv0QEVERsG3bNtSuXRt2dnZwdXVFu3btcPPmTaXLAsCgYhh7VIiIqIgIDw9HmzZt8O677+LUqVPYtm0bHj16hLZt2ypdGgCFg4pWq8W4cePg7+8PKysrlCpVCpMnT4bS90mUmFCIiCifNWjQAAMGDMDgwYPh5OQEd3d3LFiwAImJiejZsyfs7OxQunRpbNmyRX5OWFgYatWqBY1GA09PT4wcORIZGRnyfp1Oh9DQUPl7NTAwEGvWrHlpHcePH4dWq8WUKVNQqlQpBAUFYfjw4Th16hTS09Pz7f3nlKJBZcaMGZg/fz7mzp2LixcvYsaMGZg5cybmzJmjZFlcmZaIqDATAkhLLPifXPwje8mSJXBxccGRI0cwYMAA9OnTBx06dECdOnVw4sQJNGnSBN26dUNSUhLu3r2LFi1aoGbNmjh9+jTmz5+PhQsXYsqUKfLxQkND8dtvv+HHH3/E+fPnMWTIEHz00UcICwt7YQ3Vq1eHSqXCokWLoNVqERsbi99//x2NGjWCubl5rk5BXpKEgt0XrVq1gru7OxYuXChva9euHaysrLB06dJXPt+Y20QbY/3XfVHulz244wY03ncxz45LRER5KyUlBeHh4fD394elpWXmxrREYJpXwRcz+h5gYZPj5g0aNIBWq8U///wDIHOUwcHBAW3btsVvv/0GALh//z48PT1x8OBBbNq0CWvXrsXFixch/df1/8MPP+CLL75AbGws0tPTUaxYMezcuRPBwcHy63z66adISkrC8uXLX1hLWFgYOnbsiMePH0Or1SI4OBibN2+Go6NjLj6ITAbPzX+M+f5WtEelTp062LVrF65cuQIAOH36NPbv34/mzZsbbJ+amoq4uDi9n/wgceyHiIgKQEBAgPy7Wq2Gs7MzqlSpIm9zd3cHAERFReHixYsIDg7W+44KCQlBQkIC7ty5g2vXriEpKQmNGzeGra2t/PPbb7/h+vXrAIBKlSrJ27O+a+/fv4/evXuje/fuOHr0KMLCwmBhYYH27dsrPhUDAMyUfPGRI0ciLi4O5cuXh1qthlarxdSpU9G1a1eD7UNDQzFx4sQCqCwzvzGuEBEVQubWmb0bSryusU95bmhFkiS9bVmhRKfTvfJYCQkJAIC///4b3t7eevs0Gg0AYPPmzfK8EysrKwDAvHnz4ODggJkzZ8rtly5dCl9fXxw+fBi1a9c29m3lKUWDyqpVq7Bs2TIsX74clSpVwqlTpzB48GB4eXmhe/fu2dqPGjUKQ4cOlR/HxcXB19c3/wpUPkgSEZGxJMmoIZjCokKFCli7di2EEHKAOXDgAOzs7ODj4wMnJydoNBrcvn0b9evXN3gMPz+/bNuSkpKgUukPsKjVagA5C0j5TdGgMmLECIwcORKdOnUCAFSpUgW3bt1CaGiowaCi0WjkVJifJBX7UoiIyLT07dsXs2fPxoABA9C/f39cvnwZEyZMwNChQ6FSqWBnZ4fhw4djyJAh0Ol0qFu3LmJjY3HgwAHY29sb/F4FgJYtW2LWrFmYNGkSOnfujPj4eIwePRp+fn6oVq1aAb/L7BQNKi9KcYonOK6jQkREJsbb2xubN2/GiBEjEBgYiGLFiqFXr14YO3as3Gby5MlwdXVFaGgobty4AUdHRwQFBWH06NEvPO67776L5cuXY+bMmZg5cyasra0RHByMrVu3ysNDSlI0qLRu3RpTp05F8eLFUalSJZw8eRLffvstPvnkEyXLIiIiynd79+7Nts3QarDPTmitX78+jhw58sJjSpKEQYMGYdCgQUbV0qlTJ3l0w9QoGlTmzJmDcePGoW/fvoiKioKXlxc+++wzjB8/XsmyeNUPERGRiVA0qNjZ2WH27NmYPXu2kmVk99/dkyVOpiUiIlIU7/VjAHtUiIiITAODChEREZksBhVDsq764dAPERGRohhUDOHQDxERkUlgUHkZ9qgQEREpikHFAEnix0JERGQK+I1siMSbEhIREZkCBhUiIiIyWQwqBnAdFSIiKipSUlLQo0cPVKlSBWZmZnj//fcNtktNTcWYMWPg5+cHjUaDEiVK4Ndff833+hRdmdZUZd09mZcnExHRm06r1cLKygoDBw7E2rVrX9iuY8eOePDgARYuXIjSpUsjMjKyQG4izB4Vg/ixEBFR/mrQoAEGDBiAwYMHw8nJCe7u7liwYAESExPRs2dP2NnZoXTp0tiyZYv8nLCwMNSqVQsajQaenp4YOXIkMjIy5P06nQ6hoaHw9/eHlZUVAgMDsWbNmpfWYWNjg/nz56N3797w8PAw2Gbr1q0ICwvD5s2b0ahRI5QoUQLBwcEICQnJmw/jJfiNbABHfoiICi8hBJLSkwr859m7HOfUkiVL4OLigiNHjmDAgAHo06cPOnTogDp16uDEiRNo0qQJunXrhqSkJNy9exctWrRAzZo1cfr0acyfPx8LFy7ElClT5OOFhobit99+w48//ojz589jyJAh+OijjxAWFvZan+nGjRtRo0YNzJw5E97e3ihbtiyGDx+O5OTk1zpuTnDoxwCJNyUkIiq0kjOS8dbytwr8dQ93OQxrc2ujnhMYGIixY8cCAEaNGoXp06fDxcUFvXv3BgCMHz8e8+fPx5kzZ7Bp0yb4+vpi7ty5kCQJ5cuXx7179/DFF19g/PjxSE9Px7Rp07Bz504EBwcDAEqWLIn9+/fjp59+Qv369XP93m7cuIH9+/fD0tIS69atw6NHj9C3b188fvwYixYtyvVxc4JBxQDBLhUiIioAAQEB8u9qtRrOzs6oUqWKvM3d3R0AEBUVhYsXLyI4OFjvgo+QkBAkJCTgzp07iI+PR1JSEho3bqz3GmlpaahWrRoAoFKlSrh16xYAoF69enrDSi+j0+kgSRKWLVsGBwcHAMC3336L9u3b44cffoCVlVUu3n3OMKgYwJhCRFR4WZlZ4XCXw4q8rrHMzc31HkuSpLctK5TkZNJqQkICAODvv/+Gt7e33j6NRgMA2Lx5M9LT0zPrNSJceHp6wtvbWw4pAFChQgUIIXDnzh2UKVMmx8cyFoOKIRz6ISIqtCRJMnoIpjCoUKEC1q5dCyGEHGAOHDgAOzs7+Pj4wMnJCRqNBrdv337hMI+fn1+uXjskJASrV69GQkICbG1tAQBXrlyBSqWCj49P7t5QDnEyrQFZlycTERGZir59+yIiIgIDBgzApUuXsGHDBkyYMAFDhw6FSqWCnZ0dhg8fjiFDhmDJkiW4fv06Tpw4gTlz5mDJkiUvPfaFCxdw6tQpPHnyBLGxsTh16hROnTol7+/SpQucnZ3Rs2dPXLhwAfv27cOIESPwySef5OuwD8AelZdijwoREZkKb29vbN68GSNGjEBgYCCKFSuGXr16yZNxAWDy5MlwdXVFaGgobty4AUdHRwQFBWH06NEvPXaLFi3kuSsA5DktWVcy2draYseOHRgwYABq1KgBZ2dndOzYUe+Ko/wiidxcT2Ui4uLi4ODggNjYWNjb2+fZcbcvngLf6cvw2B6oe+Rinh2XiIjyVkpKCsLDw+Hv7w9LS0uly6FnvOzcGPP9zaEfA1QqfixERESmgN/IBvHuyURERKaAQcUQJhQiIiKTwKBigJQ19FNoZ+8QERG9GRhUDJCX0Fe4DiIioqKOQYWIiIhMFoOKIVn3UeDQDxERkaIYVAzIWpqYQz9ERETKYlAxhHdPJiIiMgkMKgZI4NAPEREVvAYNGmDw4MFKl2FSGFQMUfGqHyIiIlPAoGIAe1SIiIhMA4OKARLnqBARUT5LTEzExx9/DFtbW3h6euKbb77R25+amorhw4fD29sbNjY2eOutt7B37155/+LFi+Ho6Iht27ahQoUKsLW1RbNmzRAZGSm32bt3L2rVqgUbGxs4OjoiJCRE7y7JGzZsQFBQECwtLVGyZElMnDgRGRkZ+f7ejWGmdAGmSFLxqh8iosJKCAGRnFzgrytZWRn1D90RI0YgLCwMGzZsgJubG0aPHo0TJ06gatWqAID+/fvjwoULWLFiBby8vLBu3To0a9YMZ8+eRZkyZQAASUlJ+Prrr/H7779DpVLho48+wvDhw7Fs2TJkZGTg/fffR+/evfHHH38gLS0NR44ckWv8559/8PHHH+P7779HvXr1cP36dfzvf/8DAEyYMCFvP5zXwKBiiMSOJiKiwkokJ+NyUPUCf91yJ45DsrbOUduEhAQsXLgQS5cuRcOGDQEAS5YsgY+PDwDg9u3bWLRoEW7fvg0vLy8AwPDhw7F161YsWrQI06ZNAwCkp6fjxx9/RKlSpQBkhptJkyYBAOLi4hAbG4tWrVrJ+ytUqCDXMHHiRIwcORLdu3cHAJQsWRKTJ0/G//3f/zGoEBERFWXXr19HWloa3nrrLXlbsWLFUK5cOQDA2bNnodVqUbZsWb3npaamwtnZWX5sbW0thxAA8PT0RFRUlHy8Hj16oGnTpmjcuDEaNWqEjh07wtPTEwBw+vRpHDhwAFOnTpWfr9VqkZKSgqSkJFjnMHTlN0WDSokSJfTGyrL07dsX8+bNU6CiTJJKnflfTqYlIip0JCsrlDtxXJHXzSsJCQlQq9U4fvw41Gq13j5bW1v5d3Nzc/0aJAlCPP3yWrRoEQYOHIitW7di5cqVGDt2LHbs2IHatWsjISEBEydORNu2bbO9vqWlZZ69l9elaFA5evQotFqt/PjcuXNo3LgxOnTooGBVnExLRFSYSZKU4yEYpZQqVQrm5uY4fPgwihcvDgCIjo7GlStXUL9+fVSrVg1arRZRUVGoV6/ea71WtWrVUK1aNYwaNQrBwcFYvnw5ateujaCgIFy+fBmlS5fOi7eUbxQNKq6urnqPp0+fjlKlSqF+/foKVZSFQYWIiPKPra0tevXqhREjRsDZ2Rlubm4YM2YMVP+t41W2bFl07doVH3/8Mb755htUq1YNDx8+xK5duxAQEICWLVu+8jXCw8Px888/47333oOXlxcuX76Mq1ev4uOPPwYAjB8/Hq1atULx4sXRvn17qFQqnD59GufOncOUKVPy9f0bw2TmqKSlpWHp0qUYOnToC3s0UlNTkZqaKj+Oi4vLl1pUWVf9cOiHiIjyyVdffYWEhAS0bt0adnZ2GDZsGGJjY+X9ixYtwpQpUzBs2DDcvXsXLi4uqF27Nlq1apWj41tbW+PSpUtYsmQJHj9+DE9PT/Tr1w+fffYZAKBp06b466+/MGnSJMyYMQPm5uYoX748Pv3003x5v7kliWcHsxS0atUqdOnSRW+G8/O+/PJLTJw4Mdv22NhY2Nvb51kt//69CE7DZiLREqhx6mKeHZeIiPJWSkoKwsPD4e/vb1LzKujl5yYuLg4ODg45+v42metwFy5ciObNm78wpADAqFGjEBsbK/9ERETkSy0Sh36IiIhMgkkM/dy6dQs7d+7En3/++dJ2Go0GGo0m3+vJWvCNS+gTEREpyyR6VBYtWgQ3N7ccTQ4qCFlzZNivQkREpCzFg4pOp8OiRYvQvXt3mJmZRAcPV6YlIiIyEYp/I+/cuRO3b9/GJ598onQpMvmqIw79EBEVCiZyXQg9I6/OieJdGE2aNDHBP2CZ+Y1DP0REpi1rZdakpCRY5eHKsPT6kpKSAGRfPddYigcVUyRPpiUiIpOmVqvh6Ogo39/G2tqaq4srTAiBpKQkREVFwdHRMdstAIzFoGIA/5ATERUeHh4eACCHFTINjo6O8rl5HQwqBkhZQz+mNiJFRETZSJIET09PuLm5IT09XelyCJnDPa/bk5KFQcUAScU5KkREhY1arc6zL0cyHYpf9WOKsoIKERERKYvfyC/DoR8iIiJFMagYInHoh4iIyBQwqBig4uXJREREJoFBxQApawl9Dv0QEREpikHFEN6UkIiIyCQwqBgg8aaEREREJoHfyAZkLaHPBd+IiIiUxaBigMRBHyIiIpPAoGKApMpc2ZA9KkRERMpiUDGAc1SIiIhMA7+RDeFVP0RERCaBQcUA1X9BheuoEBERKYtBxQAhsS+FiIjIFDCoGJB192R+OERERMrid7EBEj8WIiIik8BvZAOevSmh0OkUrISIiKhoY1Ax4NkF3xhUiIiIlMOgYojq6ceiE1oFCyEiIiraGFQMUP23Mi0A6LTsUSEiIlIKg4oBqmeHfgSDChERkVIYVAx5ZuiHQYWIiEg5DCoGSM/OUeHQDxERkWIYVAyQpKdzVAQYVIiIiJTCoGLIM+uoQMurfoiIiJRidFCJiIjAnTt35MdHjhzB4MGD8fPPP+dpYUqSngkqOq6jQkREpBijg0qXLl2wZ88eAMD9+/fRuHFjHDlyBGPGjMGkSZPyvEAl6F2ezHVUiIiIFGN0UDl37hxq1aoFAFi1ahUqV66Mf//9F8uWLcPixYvzuj6F8PJkIiIiU2B0UElPT4dGowEA7Ny5E++99x4AoHz58oiMjMzb6hSid9UPh36IiIgUY3RQqVSpEn788Uf8888/2LFjB5o1awYAuHfvHpydnfO8QCWo1M98LAwqREREijE6qMyYMQM//fQTGjRogM6dOyMwMBAAsHHjRnlIyBh3797FRx99BGdnZ1hZWaFKlSo4duyY0cfJSyo8naMCnVCuECIioiLOzNgnNGjQAI8ePUJcXBycnJzk7f/73/9gbW1t1LGio6MREhKCd955B1u2bIGrqyuuXr2qd1xFPHvVD+eoEBERKcbooJKcnAwhhBwmbt26hXXr1qFChQpo2rSpUceaMWMGfH19sWjRInmbv7+/sSXlOZXe3ZMzFKyEiIioaDN66KdNmzb47bffAAAxMTF466238M033+D999/H/PnzjTrWxo0bUaNGDXTo0AFubm6oVq0aFixY8ML2qampiIuL0/vJD5KKQz9ERESmwOigcuLECdSrVw8AsGbNGri7u+PWrVv47bff8P333xt1rBs3bmD+/PkoU6YMtm3bhj59+mDgwIFYsmSJwfahoaFwcHCQf3x9fY0tP2ekZ4Z+eK8fIiIixRgdVJKSkmBnZwcA2L59O9q2bQuVSoXatWvj1q1bRh1Lp9MhKCgI06ZNQ7Vq1fC///0PvXv3xo8//miw/ahRoxAbGyv/REREGFt+jjy74Bvv9UNERKQco4NK6dKlsX79ekRERGDbtm1o0qQJACAqKgr29vZGHcvT0xMVK1bU21ahQgXcvn3bYHuNRgN7e3u9n/wgPbPgG3MKERGRcowOKuPHj8fw4cNRokQJ1KpVC8HBwQAye1eqVatm1LFCQkJw+fJlvW1XrlyBn5+fsWXlLb3JtFxCn4iISClGX/XTvn171K1bF5GRkfIaKgDQsGFDfPDBB0Yda8iQIahTpw6mTZuGjh074siRI/j5558Vv8Gh/r1+OJmWiIhIKUYHFQDw8PCAh4eHfBdlHx+fXC32VrNmTaxbtw6jRo3CpEmT4O/vj9mzZ6Nr1665KSvPPHvVD+/1Q0REpByjh350Oh0mTZoEBwcH+Pn5wc/PD46Ojpg8eXKu7ovTqlUrnD17FikpKbh48SJ69+5t9DHymvTMgm/gVT9ERESKMbpHZcyYMVi4cCGmT5+OkJAQAMD+/fvx5ZdfIiUlBVOnTs3zIguaSnr2Xj+co0JERKQUo4PKkiVL8Msvv8h3TQaAgIAAeHt7o2/fvm9MUNEhs7tJcI4KERGRYowe+nny5AnKly+fbXv58uXx5MmTPClKaZKkQtYVyoJL6BMRESnG6KASGBiIuXPnZts+d+5cvauACjOVSo2sfhRdBueoEBERKcXooZ+ZM2eiZcuW2Llzp7yGysGDBxEREYHNmzfneYFKUEkShARAcGVaIiIiJRndo1K/fn1cuXIFH3zwAWJiYhATE4O2bdvi8uXL8j2ACjvp2YVpOZmWiIhIMblaR8XLy+uNmDT7IlJWjwoAHe+eTEREpJgcBZUzZ87k+IABAQG5LsaUZMUTwSX0iYiIFJOjoFK1atXMXoZXXKorSRK02jfkiz3rqh8u+EZERKSYHAWV8PDw/K7D5DztUWFQISIiUkqOgoridzNWQNYcFZGL2wIQERFR3jD6qp+iRsc5KkRERIphUHmBpz0qDCpERERKYVB5BV6eTEREpBwGlReQJ9OyR4WIiEgxRgeV7t27Y9++fflRi0mRh35492QiIiLFGB1UYmNj0ahRI5QpUwbTpk3D3bt386Muk8GgQkREpByjg8r69etx9+5d9OnTBytXrkSJEiXQvHlzrFmzBunp6flRoyJ4eTIREZHycjVHxdXVFUOHDsXp06dx+PBhlC5dGt26dYOXlxeGDBmCq1ev5nWdyuHlyURERIp5rcm0kZGR2LFjB3bs2AG1Wo0WLVrg7NmzqFixImbNmpVXNSqCNyUkIiJSntFBJT09HWvXrkWrVq3g5+eH1atXY/Dgwbh37x6WLFmCnTt3YtWqVZg0aVJ+1FvgeFNCIiIi5eRoCf1neXp6QqfToXPnzjhy5AiqVq2arc0777wDR0fHPChPeZyjQkREpByjg8qsWbPQoUMHWFpavrCNo6Njob+RIS9PJiIiUp7RQaVbt27y7xEREQAAX1/fvKvIxLBHhYiISDlGz1HJyMjAuHHj4ODggBIlSqBEiRJwcHDA2LFj38zLkwWDChERkVKM7lEZMGAA/vzzT8ycORPBwcEAgIMHD+LLL7/E48ePMX/+/DwvUlHsUSEiIlKM0UFl+fLlWLFiBZo3by5vCwgIgK+vLzp37vzGBBX5Xj+co0JERKQYo4d+NBoNSpQokW27v78/LCws8qImkyCvo8KhHyIiIsUYHVT69++PyZMnIzU1Vd6WmpqKqVOnon///nlanElgUCEiIlKM0UM/J0+exK5du+Dj44PAwEAAwOnTp5GWloaGDRuibdu2cts///wz7yotYLzXDxERkfKMDiqOjo5o166d3rY3+/JkzlEhIiJSitFBZdGiRflRh8nJ6lEB2KNCRESkFKODSpaHDx/i8uXLAIBy5crB1dU1z4oyJRz6ISIiUo7Rk2kTExPxySefwNPTE2+//TbefvtteHl5oVevXkhKSsqPGhXBJfSJiIiUZ3RQGTp0KMLCwrBp0ybExMQgJiYGGzZsQFhYGIYNG5YfNSqLQYWIiEgxRgeVtWvXYuHChWjevDns7e1hb2+PFi1aYMGCBVizZo1Rx/ryyy8hSZLeT/ny5Y0tKV88XfCNQz9ERERKMXqOSlJSEtzd3bNtd3Nzy9XQT6VKlbBz586nBZnletpM3uKCb0RERIozukclODgYEyZMQEpKirwtOTkZEydOlO/9YwwzMzN4eHjIPy4uLkYfIz/IAz4MKkRERIoxuvti9uzZaNasWbYF3ywtLbFt2zajC7h69Sq8vLxgaWmJ4OBghIaGonjx4gbbpqam6q2IGxcXZ/Tr5VhWjwqv+iEiIlKM0UGlSpUquHr1KpYtW4ZLly4BADp37oyuXbvCysrKqGO99dZbWLx4McqVK4fIyEhMnDgR9erVw7lz52BnZ5etfWhoKCZOnGhsybmS1aMicR0VIiIixUjCiOtv09PTUb58efz111+oUKFCnhcTExMDPz8/fPvtt+jVq1e2/YZ6VHx9fREbGwt7e/s8rWVvnQpwfwLcGtYBzXpPytNjExERFWVxcXFwcHDI0fe3UT0q5ubmenNT8pqjoyPKli2La9euGdyv0Wig0Wjy7fUN4hwVIiIixRg9mbZfv36YMWMGMjIy8ryYhIQEXL9+HZ6ennl+bGNxMi0REZHyjJ6jcvToUezatQvbt29HlSpVYGNjo7ffmDsmDx8+HK1bt4afnx/u3buHCRMmQK1Wo3PnzsaWlfe4Mi0REZHi8uTuybl1584ddO7cGY8fP4arqyvq1q2LQ4cOmcR9g+R4wrsnExERKUbRuyevWLEiz46V5+QF3xhUiIiIlGL0HJV3330XMTEx2bbHxcXh3XffzYuaTALnqBARESnP6KCyd+9epKWlZduekpKCf/75J0+KMgmco0JERKS4HA/9nDlzRv79woULuH//vvxYq9Vi69at8Pb2ztvqTAFXpiUiIlJMjoNK1apV5TscGxrisbKywpw5c/K0OJPAHhUiIiLF5DiohIeHQwiBkiVL4siRI3pX5lhYWMDNzQ1qtTpfilSCkP/LoEJERKSUHAcVPz8/AEXoJn3/zVHh0A8REZFyjL48Gci84/GePXsQFRWVLbiMHz8+TwpTGntUiIiIlGd0UFmwYAH69OkDFxcXeHh4QJIkeZ8kSW9MUHnao8KgQkREpBSjg8qUKVMwdepUfPHFF/lRj8lhjwoREZFyjF5HJTo6Gh06dMiPWkyKkBdS4RwVIiIipRgdVDp06IDt27fnRy2mhQu+ERERKc7ooZ/SpUtj3LhxOHToEKpUqQJzc3O9/QMHDsyz4pSUFU8kBhUiIiLFGB1Ufv75Z9ja2iIsLAxhYWF6+yRJemOCCm9KSEREpDyjg0p4eHh+1GGyJF71Q0REpBij56hkSUtLw+XLl5GRkZGX9ZgMwR4VIiIixRkdVJKSktCrVy9YW1ujUqVKuH37NgBgwIABmD59ep4XqDxe9UNERKQUo4PKqFGjcPr0aezduxeWlpby9kaNGmHlypV5WpySBBd8IyIiUpzRc1TWr1+PlStXonbt2nqr0laqVAnXr1/P0+JMARd8IyIiUo7RPSoPHz6Em5tbtu2JiYl6weWNwTkqREREijE6qNSoUQN///23/DgrnPzyyy8IDg7Ou8oUJg/9cGVaIiIixRg99DNt2jQ0b94cFy5cQEZGBr777jtcuHAB//77b7Z1Vd4M7FEhIiJSitE9KnXr1sWpU6eQkZGBKlWqYPv27XBzc8PBgwdRvXr1/KhRGfKtfhhUiIiIlGJ0jwoAlCpVCgsWLMjrWkyKyPYLERERFbRcL/hWVAiuo0JERKQYBpUXkeTbJytbBxERURHGoPIC4rn/EhERUcFjUHmR/zpUJPaoEBERKea1g0pcXBzWr1+Pixcv5kU9JkcwqBARESnG6KDSsWNHzJ07FwCQnJyMGjVqoGPHjggICMDatWvzvEClPF3wjUGFiIhIKUYHlX379qFevXoAgHXr1kEIgZiYGHz//feYMmVKnheoOAYVIiIixRgdVGJjY1GsWDEAwNatW9GuXTtYW1ujZcuWuHr1ap4XqBj5oh8GFSIiIqUYHVR8fX1x8OBBJCYmYuvWrWjSpAkAIDo6GpaWlnleoFKy4onE636IiIgUY/TKtIMHD0bXrl1ha2sLPz8/NGjQAEDmkFCVKlXyuj7FcIoKERGR8owOKn379kWtWrUQERGBxo0bQ6XK7JQpWbIk56gQERFRnsrV5ck1atTABx98AFtbW2i1Wpw6dQp16tRBSEhIrguZPn06JEnC4MGDc32MvCS4Mi0REZHijA4qgwcPxsKFCwEAWq0W9evXR1BQEHx9fbF3795cFXH06FH89NNPCAgIyNXz8xWDChERkWKMDipr1qxBYGAgAGDTpk0IDw/HpUuXMGTIEIwZM8boAhISEtC1a1csWLAATk5ORj8/v8jrqHAyLRERkWKMDiqPHj2Ch4cHAGDz5s3o0KEDypYti08++QRnz541uoB+/fqhZcuWaNSokdHPLRDsUSEiIlKM0ZNp3d3dceHCBXh6emLr1q2YP38+ACApKQlqtdqoY61YsQInTpzA0aNHc9Q+NTUVqamp8uO4uDijXs8ovOyHiIhIcUb3qPTs2RMdO3ZE5cqVIUmS3BNy+PBhlC9fPsfHiYiIwKBBg7Bs2bIcr78SGhoKBwcH+cfX19fY8o2QmVSYU4iIiJQjiVwsvbpmzRpERESgQ4cO8PHxAQAsWbIEjo6OaNOmTY6OsX79enzwwQd6vTBarRaSJEGlUiE1NTVbD42hHhVfX1/ExsbC3t7e2Lfx8vpaVUG5axk43dwfnWZtztNjExERFWVxcXFwcHDI0fe30UM/ANC+ffts27p3727UMRo2bJhtTkvPnj1Rvnx5fPHFFwaHkTQaDTQajXHF5haHfoiIiBSXq6ASFhaGr7/+GhcvXgQAVKxYESNGjJBvVpgTdnZ2qFy5st42GxsbODs7Z9uuJMYUIiIi5Rg9R2Xp0qVo1KgRrK2tMXDgQAwcOBBWVlZo2LAhli9fnh81KuO/HhWJPSpERESKMbpHZerUqZg5cyaGDBkibxs4cCC+/fZbTJ48GV26dMl1MbldMC4/CPn2ycrWQUREVJQZ3aNy48YNtG7dOtv29957D+Hh4XlSlEmQV9BnUiEiIlKK0UHF19cXu3btyrZ9586d+Xy5sFIYVIiIiJRi9NDPsGHDMHDgQPlGhABw4MABLF68GN99912eF6gUwat+iIiIFGd0UOnTpw88PDzwzTffYNWqVQCAChUqYOXKlTleQ6VQYU4hIiJSjFFBJSMjA9OmTcMnn3yC/fv351dNJkL673+ZVIiIiJRi1BwVMzMzzJw5ExkZGflVj8kQnExLRESkOKMn0zZs2BBhYWH5UYtp4ToqREREijN6jkrz5s0xcuRInD17FtWrV4eNjY3e/vfeey/PilOSUP039KNjUCEiIlKK0UGlb9++AIBvv/022z5JkqDVal+/KhOgy7rqh0GFiIhIMUYHFZ1Olx91mBzx36AYe1SIiIiUY/QclaJCSP8N/XCOChERkWJyHFR2796NihUrIi4uLtu+2NhYVKpUCfv27cvT4pSUNUeFQz9ERETKyXFQmT17Nnr37g17e/ts+xwcHPDZZ59h1qxZeVqckrIuT5aKyFAXERGRKcpxUDl9+jSaNWv2wv1NmjTB8ePH86QoUyBUmR8N56gQEREpJ8dB5cGDBzA3N3/hfjMzMzx8+DBPijIFnExLRESkvBwHFW9vb5w7d+6F+8+cOQNPT888KcoU6DiZloiISHE5DiotWrTAuHHjkJKSkm1fcnIyJkyYgFatWuVpcUp6OvTDOSpERERKyfE6KmPHjsWff/6JsmXLon///ihXrhwA4NKlS5g3bx60Wi3GjBmTb4UWNA79EBERKS/HQcXd3R3//vsv+vTpg1GjRsk365MkCU2bNsW8efPg7u6eb4UWuP96VMChHyIiIsUYtTKtn58fNm/ejOjoaFy7dg1CCJQpUwZOTk75VZ9yeK8fIiIixRm9hD4AODk5oWbNmnldi2nJ6lFhUCEiIlIMl9B/ASH9N5mWQz9ERESKYVB5AYkLvhERESmOQeUFsu71IzGnEBERKYZB5QUklTrzF/aoEBERKYZB5UXUmR+NinNUiIiIFMOg8iK86oeIiEhxDCovkDWZtsStDIj0dIWrISIiKpoYVF7A4VGq/Hv0qlUKVkJERFR0Mai8gPu1WPn3qK+/UbASIiKiootB5QXuuFjJv4vkZAUrISIiKroYVF4g2sVG6RKIiIiKPAaVF7hVw0fpEoiIiIo8BpUXqFjcRekSiIiIijwGlRewtbTSeyy48BsREVGBUzSozJ8/HwEBAbC3t4e9vT2Cg4OxZcsWJUuSqcw0+KeSJD8WaWkKVkNERFQ0KRpUfHx8MH36dBw/fhzHjh3Du+++izZt2uD8+fNKlgUAUJtpkGL+9HHC3jDliiEiIiqiFA0qrVu3RosWLVCmTBmULVsWU6dOha2tLQ4dOqRkWQAAlVqDv2s9/Xi0sTHKFUNERFREmSldQBatVovVq1cjMTERwcHBBtukpqYiNfXpirFxcXH5Vk+S0OKe89OhH8ncIt9ei4iIiAxTfDLt2bNnYWtrC41Gg88//xzr1q1DxYoVDbYNDQ2Fg4OD/OPr65tvda26s1vvsUrDoEJERFTQFA8q5cqVw6lTp3D48GH06dMH3bt3x4ULFwy2HTVqFGJjY+WfiIiIfKurvlcdAMCx0pm9KtrExHx7LSIiIjJM8aBiYWGB0qVLo3r16ggNDUVgYCC+++47g201Go18hVDWT355x/cdAEDqfx0pOgYVIiKiAqd4UHmeTqfTm4eiFAuNHQAgwzxz/RRdUpKS5RARERVJik6mHTVqFJo3b47ixYsjPj4ey5cvx969e7Ft2zYlywIAmJtbAwCS2KNCRESkGEWDSlRUFD7++GNERkbCwcEBAQEB2LZtGxo3bqxkWQAAc1XmIipJFhIAwaBCRESkAEWDysKFC5V8+ZeyUGd2pSRqsoIKh36IiIgKmsnNUTEVWT0qyZrMx7rEBAWrISIiKpoYVF4gq0clS8LuPQpVQkREVHQxqLxAVo9KxVu8azIREZFSGFReQCVlfjSba/IjIiIiUgq/hV/hkcPT34Vg7woREVFBYlB5ifKpaUh75rookZamXDFERERFEIPKS1gJnX5QSUlRrhgiIqIiiEHlJezN7aB95hNKvXJFuWKIiIiKIAaVl1Bb2AKSJD++1e1jBashIiIqehhUXmK3NlrpEoiIiIo0BpUcSPxvdVrrWrWULYSIiKiIYVDJgVX1Mj+mpCNHkP7ggcLVEBERFR0MKjng8+jp+ik3O3VWsBIiIqKihUHlJep7hQAAIlyfTqjNiIxUqhwiIqIih0HlJewsMpel3R4kvaIlERER5QcGlZdwsnYGAOhUDCpERERKYFB5ieYlmitdAhERUZHGoPISfg5+BrcnHT9ewJUQEREVTQwqL6GW1Aa3p926XcCVEBERFU0MKi9hobaQf3/gq326Q+KcFSIiooLAoPIS5ipz+fffaj39/cHUqUqUQ0REVOQwqOTQFe+nvSi6hASI9HQFqyEiIioaGFRyKNZGf7jnUpUA6FJTFaqGiIioaGBQeYV2pdvKv8+q3UFvX8rZswVdDhERUZHCoPIKkurpR9TPd6PePqHVFXQ5RERERQqDyiu0KtlK/j1QHa63L/bPPwu6HCIioiKFQeUVvG29AQBmQkA8ty92wwakXLhQ8EUREREVEQwqr2BvYQ8AyJAkPFCrca2+j97+2L/+VqIsIiKiIoFB5RUszSzl3xsX90ZD91N6+5/8+isvVSYiIsonDCqvoJL0PyJrVVq2NjdatS6ocoiIiIoUBhUj3TQzw823W+ptS7t1S6FqiIiI3mwMKkYa4u6CCZaB2banXr+uQDVERERvNgaVHOhVuZf8+zULC0RZF8vW5kbLVtm2ERER0ethUMmBLhW66D2uJN3E0tELs7VLOHCgoEoiIiIqEhhUcsDVylXvcYDqOpZdiM7WLqLXpwVVEhERUZGgaFAJDQ1FzZo1YWdnBzc3N7z//vu4fPmykiUZJEn6NyRsZ74dAKCtbGCuytWrBVITERFRUaBoUAkLC0O/fv1w6NAh7NixA+np6WjSpAkSExOVLOuVIm0fo5R0F996hGTbd7NzFwPPICIiotxQNKhs3boVPXr0QKVKlRAYGIjFixfj9u3bOH78uJJlGdSzck/59yX29tipGYHdNv6AtbVeO11CAoR4frF9IiIiyg2TmqMSGxsLAChWLPtVNQCQmpqKuLg4vZ+C0rtKb/n3yxoL7La2glrSYdnoX7K1Tbtxo8DqIiIiepOZTFDR6XQYPHgwQkJCULlyZYNtQkND4eDgIP/4+voWWH12FnZ6jwe7u+K6ZTesORYOtYuL3r4bLVshPTKywGojIiJ6U5lMUOnXrx/OnTuHFStWvLDNqFGjEBsbK/9EREQUYIVAKYdSeo8fqNXopt6Brb2/zNb22jvvQpeWfbl9IiIiyjmTCCr9+/fHX3/9hT179sDHx+eF7TQaDezt7fV+CtK6Nuv0Hjcq7o3qqkv49mIKSu3cma395YBACJ2uoMojIiJ64ygaVIQQ6N+/P9atW4fdu3fD399fyXJe6fnLlAEgw/4SNEhDhpu7wedcrVuPYYWIiCiXFA0q/fr1w9KlS7F8+XLY2dnh/v37uH//PpKTk5Us66WKWepP9P3L1gaXLXvgi7VnDbbXPnmCB9NCC6I0IiKiN46iQWX+/PmIjY1FgwYN4OnpKf+sXLlSybJeal7DeXqPD1hb4bTGAptO30OZw4cAA70u0UuXQpeUVFAlEhERvTEUH/ox9NOjRw8ly3qpyi7Zr0j6yMsD1aSreOen4yh75ozB510Oqo60mzfzuToiIqI3i0lMpi1s7Mxtsm37UzMBKU/u4WJUIsqfMzwMdL1ZcwheCURERJRjDCq5MKHOxGzbhrm54KhlP7SbsxuSmRnKHPzX4HOvt26Nx4sWI/kFPS9ERET0lJnSBRRGTfyaZNu2w8Ya3TzdUfbmbWh1AmZOTgafm37rNqJmzAAAVLh0MV/rJCIiKuzYo5ILkiShTak22bafstTgS/up2HruPgDAqcvLb1CojY+Xf4/9+288CJ3OS5mJiIiewaCSS1/U+sLg9h5e7hiy/DCqTtoOda23XnqMKzVrIfHff5F+9y7uDRuOJ0uWIGH37vwol4iIqFBiUMml5+/986yzVt0xJf0b1NmThNMlg156nNuf9MK1ho3kxxmPn+RZjURERIUdg8pr6BvY1+D2GiWKY0LJu9hi/ynerrMFIz/4PMfHvD9hAtLu3IHIyADw3yXcvFKIiIiKKEkIIZQuIrfi4uLg4OCA2NjYAr/vDwCk69LRa1svnIw6+dJ2rRIS0fKuC5z/jH9pu+d5fPklkk+eROyGDSi9ZzfMPT1fp1wiIiKTYMz3N4NKHqiypErOGgqBZV9nwDwj++q1OeHx5QRI5haIHDMGTh93g0ufPlDb2kIyN4cQwuC9iIiIiEwNg0oBm3poKlZcXpGjtiqdwIoZ2jx9fZuQECQeOADvWd/CtkEDJJ04AUmthk3t2nn6OkRERHmBQaWAZegycCX6Cj7868MctXeNEWhyUoc2h/L3o3fp2xeaMqWhLuYMm7dq5etrERER5RSDikJyPAT0nxL3BWYuytvelRfx+WEerAIDkX7nDqwCA/Fg+gyIjAx4jB2j106bkIiYFX/ArkkTWBQvXiC1ERFR0cKgopB0XTpSM1Jx+P5hDN4zOEfPqRKe2bPyR30VQpfkb2hROztD+/ix3jbfBT/DsnJlaKOjYeHvj/uTJiHmjxWQNBqUPfgvJCsree5L+t27MPPy0psLI3Q6QJI4P4aIiHKMQcUEHI48jE+3f2rUc1oe0aH7LtNamdaucWP4zPkeT5Yvx4NJkwEAnlOnwrFdW6TfvSuvAePx5QQ4der0yuNpExKRsHcvrKpUxuNfF8G+WVPYBAdDCIGYlSthWaUKrCpVytf39DoyHj2C2tmZwYyI6DUwqJiIqPh7aPhnU+OeJARWTS+Y4aDX4djpQ8SsWKm3rfjixbAo7gvJ3BzJ588j6quvYVmuLDxDQ6HSaJB26xauN22W7VgVLl3Eg+kz8GTxYvkxACQcOABzDw9oSpWS22rj45ERFQW1oyMe/7IQVgFVYN+8ef690WfE79yJO/0HwKF9O3hNmVIgr5nXdKmpiF66FLb160NTurTS5RBREcWgYkIeJT+CjbkNZh37Bn9cXvnqJwCQhMD45Vr4PAK2Vlfhw39Mq5clN7y+/hr3hg83vO+rr3BvxAj5cZl/D0AXHy+HGo9JE+HYoQMkScKloOoQSUnZjmHu5QXfXxZAU7IkRFoaMh4/RuS48XD8sCOsAgPxZPESQAi49OsLta2t/LyMR4+gjY6GpkwZvePpUlKQeOgQbN56CyorKwDAxWYtgJvhAJ6GKV1yMqJXrMSTX3+Fua8v/BYvgmRh8RqfVM6J9HREzZ4N65o1YdegQY6e8/D7OXj0ww8ACv6mmLyEnpSmjY9H6tVrsKpWlX8WFcagYsKMnXALAOYZAj9/r4VNKnDBF7jhIaHV0UJ72nLNpk4dJP77r9HPM3NzQ0ZUFADAulYtuA4cgHv/9wXcx43FnT6ZqwvbNmwI23p1EbtxE3zmzUXU9BmI3bABAFDu1EnokpNxocHbME/NXDHYf/06qB0ccGfAQKScOye/lvv48SjWpXOOa9MmJODRvB9g37wZLKtUMfiX58M5cyFZauDSu7e8LWHfPkT87zP5cYVLF5ERHY0nvy6CRamScGjTxuCxrnT+ENqTZwAAZQ8fgmRuDpW1tV4bkZGBpKNHYRUQAJWNzUvr1yUnI37Xbpi5ukKlsYBV1aoAgOgVK5Dx5Alc+2Z+vncGDUba7dvwX7USkrm5wWMlncxcONG6WrWXviZRTqU/iIJkYS7fzf56s+ZIu3kTXl99BYfWrRSurmhjUDFh0SnRmBr2f9h2/9BrHafSTR0m/FH4e1reRI4dOiD53Dk4demMh9/OgnWtWrCtXx+Ro0fDMjAAajt7ZERFIfXKlWzPLbF6FSzLl4c2IQEiLQ0PZ81G7Pr1AADnPp/j8fwfUWL1atzs0EHveTZ1gpF68yYy7kUCAHx/+hFqR0dYVqgAycICGdHRePzLL3iy8Ndsr+n11UyknDsPq6Ag2DVpjEfz5+PR93NgXasW/H5bAgAQaWl4MH0GbELqIPavv2BVuTKce/XCvdFjEPvnn/Kx3EeNRNy27Ug+cQIAUHLLZmj8/XGxfAUAgPd33yF+507Yt2gOm+BgqCwtAQAZ0dG4GlwHQGYwzNr+IiItDXcGDIT1W2/B+ZOeT7frdMh4+AiR48ch/XYEnD/9FI7t2r70WEBmOINa/dJ/Zac/iILa3k7uYUvYtw8Rffqi5MYN0JQqBV1aGiRzc5P8l/r9SZOhsraC2wt6NZ8XNWs2zD094NSpE6JXrIQ2Lg42td96YZgWWi0ktTqvy86VzPluq2BRwg+3e2T+2cjqPcz6cyhqh6Di4l8Uq/F1pd+9i4R/9sOh7QdQvUYPrjY2FncGDYbDe+/Bse0HeVjhqzGoFALxafGws7DDo+RH2H93P5qVaIYvd/bH3w8O5/gYJSMFHjgCLY/q0P6AwIDP1TDPACQAdS7o0O7fQntqKQ+pixWD9knubnaZ9Rf8k9+X4sHUqXr7yuz/B1fr1nvlMQzNZ8riNWM6NGXKILxtO73tZh4ecOrSBQ+//RYA4D17FuybNYMQApHjxiF2zVq5bflzZyGZmQEAIvr1R8KuXXrHKn/2jNyLI9LTEbdtO2LXrUPGkyewqlIFHmPH4FqzZjD38ESJ5csM1pl+7x6uvdsQAFDy77+gKVVK/tIDgDIH/8W1dxvCJqQO3EeOhLmnp/zFnXLpElSWlrAoUQIAkPHkCeK3b4dNnTow9/DQGypMu3kTQgho/P3lbVlDZrF//420W7fg8vnnSPz3IFLOnYNNSAisqlTWqzX57Dmo7e1g4eeXWXtkJK69827mZ3Hm9CuHJlMuXkT4B5nhruyxo7hSo6a8z/enH2Fbvz5ERgZ0CQlQ2dkh8d9/cXfwEBTr2ROu/ftBaLXQJSVBbZf9xq3a+HiobGwgqQzfZi7x0CFEL/8D7mPHwNzN7aV1Pn/c2917wCooCFZVq2YbZvaZ/wPs3nlHPmcP/F3QYMs//73mYUSvXAG3YcNh4eONjOhoPJgWCsd2bQt00UwhBDLu3ct2ZaUhl6pWg0hJgXOfz+E2aFCug+LzcwMzoqNxu+cncGjTBs49e+TiXeQcg0ohlpyRjCE7+uBA1HHjnigE8Nwf7rrndBi4KbPX5e8aEpY0VmNVaEZelUpFhFVQkNxDoiTJwuKFN+i0a9YM8Vu3vvC5rkOGoFj3j3GzSxekXtCfm+Pz43zc+bwPgMzhsORz5/Bo/nx4jBsPyUwNMxcXXG/aDNqYGPk5nqGhiBw16oWvZ+7rC+9vv0XS0aOImjkTAOC3fDludenywufYvvsuEnbvBpB5ny+nTh8iISwMEZ+9/KamZY8dxZPffsPjH3+C9+xZuNO3n3w8XUICXPr3w+2Pu2fWVbw4HFq3huuA/tDGxiLp2DFEL1sOqNWwrh4Exw8/lHu2AMB79mzcHTz4aY3vvAP3L/4P15sZnsDuv3EDwt9rAwAoHRYGlY01Us5fQFp4OKwCqsiBtPzFC/KXcer168h49BjWNWvgUsXMK/7sGjeC9+zZSLtxAyobGySfPQeVjQ1iN26Ay//+B4tSpaCLjUXshg0wc3VF6s2bePT9nJd+TvYtmiNu8xYAwOVSarz/9zk8/P57PPphPgDAMiAAnl9OQNQ33yLxwAEAT4N6xqNHUNnaQtJoAADaR49wd+gwOH74IewavpsZzBwcoEtJ0ZsDZ4yHP/yAR9/PgcvAAfKQaZaYtWsR8+c6+MydAzMnJzlwWVWtCq/poQjv0BHFPv4YrgP6v/Q10iMjkX73LtIjI2FVLQjhfftBd+Wy/F4fzJiJJ4sWAQBKbv4bFsWLy/8IyGsMKm+I+w/OwFxlhtjoG2hzMPMvxb5W/ojzqIyl4ZtyfBzbJIEEKwCSBMcEgRm/ahFZDFhRX42rXoBWnfkXhk2ywCc7dDDTAlEOwDk/CS5xwGdbOcREVNDcRoxA1Fdf5cuxJSsriOTkfDl2TvgtWwrodIjo1x+6uDjDjarVAE4eM7jLqVs3RP/++2vVUPy3JXKAe5EKly7KvWlmXp7QJSZBFxtrsK2Zlycy7kXC54d50CUlw7pGdajt7XGtaVNoHz5C6b17MpdmCAqCZdmyEDodHv+8AJKFBVIvX5bnxAGApkIFOL7fBnbNmkMbE4PwNm3kfe6jR+PBtGkAgDRXZzgFVkPizp0AAMcO7WHm6gqXAQOy9croUlNxObDqC9+r+7ixeDBZ/2pGlY0N/NeukXsD8xKDyhtKZKRDMsvswtbqtPj57M+wUVvhq+PfAABctcC44s1x1kyFlbe3I06k58nr9tyuRfPjAo/sgAkfqTFirRZ7AlQIvqhD+buZbQ6WlxB8Ked/lG66ASWi8qQ8IqJ8kV9h0Tq4NpIOvt48xYJi16wZfGbPyvPjMqgUMZcf38CB8H/RM6iLPPar1WkxbceXWHV/PQBgcsBAvF+tN7ovqYUT0P+XlJe1B+4l3X/h8dVagbJ3gateQIbZ05Tu/kRg4EYt1gercLScCnZJAl326rC/ooQJf+hwyxWY21qNyGLAsD91qHYj84/a7DYqHC8t4fdvsq8XM6GrGnedAYsMIOSCwEVfCVN+N7yuzLYgCU1PFNo/vkREhULZY8egtn35FYDGYlAh2fNrV8QmPsD+w9/BITkOpYIHwd2pFFSSCjqhg0iOgTYtAccj/oFkbgWtlSMO3vsXS67kbP2XZ1mlCKRaADrViyeF+TwUEBIw6Xct7FKAj4arkWaevb2kEygdCbTfr8OJ0hIy1MChchISrSRUu5Y5LNVzhw6adODrdmrE2AAPHSWD83EGfqbG9z+Z/oJ6RESmJK/XXWJQoTx17ckVfHc4FH1rDkfpYmXx3fHvUNy+ONacW46LCddhI1ngYLdjWDC/IubYZJ949XGJlvjt5t9627a2XI1mf2deYmuWISAJIP2/kKKCBB2M+2Op0gmodPo9PmYZAu4xwF0X/fBT76wOA/7Sn3dz2xXwfAJEuAKTO6nx8W4d3jljuIbVIRIanxL4/V0V3j4rsD5Yki8Vj3ABfB8ZVbrs93dU6LZHv66/akow14I9R0SkKAaVXGJQUVaGLgOHIg8hwDUA9hb2QEoc8OQG0t0rof+WjxHkXgOf1RwGAIiKu4uBW3vgg3Id0ax8RzhoHPDLzqH47u4O+Xjjqw7C+1W6w1xlDiEEnqQ8gaO5HfYcmQ3bJzfQ+/EBdHF7C/eTolDTqw4+LNsB/XcPgBACB5PvyMdp4FgRnap+hs/3DspWs5OFA6JTY1D8IWCeAfg+EjhZSkKsjeGeH6d4gdqXBHruzAwQKebAx8PNsl1l5fFEoNItgT2BEnQqCZo0gQz104nK/pEC/g8EdldVofRdAas0Ad+HgHuMgEscsL2ahNOlVJCEQPkI4NNtWkzppEa03dPXmP1TBryeAF1HqFH6HjBmhRar66nwwBFINQdGrc6sMc0sczK0j/79JwEAkzqrMP6/UHXLFfB7+Nw5VQFmOmBdsISKtwXK3TV87qMcgG8+UGPG4uy9U4sbqtDmkA6ba6pQ9YYOlW4bPsbzHjgC7jFPH6+vLWFPgArf/fz0Ne44AydLSWh9JPOvrZX13oyVm4lMHYNKLjGovBnuxd9FhjYdxR1L5PoYRyIPIyU9CW8Xf0fedi3iAOadmosh9abh4J19aF22PawtbLBhSz+Mjdontytt7oRr6dHy47/fW4+2G9siFU+/AEvcF6h0W2BLDUlvOMtDZYnx1Qaj7/Hpua49pySdgEo8DT8qncg2tCbpBMR/24pHZc4ZWvG2ChUiBMI9JFzylRB0VYcUC+Cyj4TKNwWu+EhI1rx4iK7eWR0Cbgpc9pFgkQ4kWAH7qmTOheoUpkXbfwWSLIADFSX8XUuFe876x3KOE5g/L3ugGd5LjU+3aVH+v4zZcaQaNilArSsCh8pJSLbMPE79Mzr0+zvzXHQclb3Hbsk3GbBKA9bWkeS1g2JsgCQNMLGLGjWuCng+EdhZTYXIYsDK5+6ldaKUhEinzNd1jQMu+QAQwO6qKlS8LdDgrMBFH6DC0yyMsMoS6p/L3V+dZ0pICPcAStwHfmilQowN0H2XDveKSah9SaDy7bz/K3l7NQlNTr7ecWOtgVX1VEi0BAZvYDgsSrYGSRiy/EKeHpNBheglREYaLp9ciLv27rCz90FNj5q4EXkMxd0CYW6WuRhWmjYNK8/+ikvxd7D39l7EZTy9JHFVq1VYc2U1+lbtB2crZwDAjTPLcPDiKnRsvQj/Pj6LyYcmY3LIZLhZOiMDAu03tQcAtPCuj962ZWBn6407Du4YuW8M7mdEY3DQYHSv1B3rzvyKlZdX4XLKAwQ6lsGgt0YhyC0IapVavv3C3y1X4+Sj8xh7+Eu09KiLMmY22BB9GeGJN1/rc3FQWyNWm/0+Si+j1grUuCpwvriEBOsXhx3/+wKJGiDVAui4T4edVVUI98xsX+6OQJoZEO7xkrB0TocIFwk3DbQxyxCwSwai7SRUv6qDJIBjZQ0vKAYA3XdqUeuywNft1EgxByKfCVbPBr3n/fxdBhz/+3g6jjKDJATco5FtztPx0hKOlJUQ4SLhuhdgmQa4xgJfrNbCNQ4Y/D91tjD3vH6btKh8SyDWGoAEjOqhRslI4P1DOixtoMLHu3WoeVVgcicV7jtJeGQPFEvIDKdZPWtZssLde4d0+Oi5ocWzfhJ+bq7CnB+fvodoG6BvPzW0aglqrUC165mT2hOtntbs+VjAKUHALTbzir+sifFbqktofjzzK2V5fRXW18k8D2qtQNUbAl+syXz956/6G/C5Gi2O6uTnrg6R4BoHNDir//W0O0BComVmqHy29+1FbngAJV98nYBsTYiEsneBgJsCixqp5B5UIDNAtzkk0HXv020xNpn3Yeu0T//zPOcnYV2whHErXh7kTpSS8EtTFWb/rIXFC5a2MuZKyk21JNQ7L+CYmNkb2vqwgNlrZslIJ8AzOjNYh3ZU4dQn5179JCMwqBDlISEEth2cic3hWzD9g7WwtnY2+hjpunQkpSfBQeOQ6zpSUmKRnp4MOzsPAEBSehKszTPv05OYnogjkUcQ4h2Cuwl3oRM6DNk9BOHx4ehZriceJtzAW94heLdUS4T8EQIAONP1BB5c2wz3Eu9CssxcRTRDl4G4tDikpCfD1cYNKRkp+G3d5/gz8QzKaJzgYm4LN1sfDGzxMwCgwx/v4FLa00k5de1KYX7b9fIk7tuxtzD31FxsuZl9Mba+pdrih+tPl98/3e0ULl7ehHmHQjGk7kSUKZN5U8qZ+0bj9/BNsIIKhz8+BQBISE+ATujgoHFAnd9qIF6kGvzMFgWNxndHpuOUmQ4jireEcKuAr499bXCBxOdZqS2RrE2RHzvFZw7vHawgyb1aAGCXJJBo+fKJ48DTQJVgC6T/99rOZnZ4nBGfvbHInLclpMyVpsVztUpCwD4RiLXN/po2yQLFEoBhf2qxqp4K/1Z8GtpcYwQGbdDieGkV4qyBvQGZ78U/UqDJSR1W1cvc/uz7y4l3T+lQ94LAV20zX8s1Drjtlv0Ype8KxFkDUY7AN79o4fsIeGILfD7ADJJOoEuYDpd8JBwvk3kci3SB4IsCxR8KbK2uwkPHzGNapmb2cLrGAgHhAsveyRwCrXlFYMh/vT1zW6mwr7IE84zMwGORASRbALsDJagE4B4N+D/IPJ9CkiAJAcs0IFkjwSZZ4P2DOuwJ1O8hrHZNhyd2Em67AuK/IV6vJ0D1qwLWqQLLG6iQoQZaHhXQqoBPdmTWcspfgpkWmP2+CnHPDzMLgRFrM4MnAOgkYHoHFU6VUgFCYNV/PYDfvafCofISrFMB5zjgsT2w8LvMfTPaq+TP7NnjWv33fpziMz/DDv/oUPZe5mc+qUvmFZmVb2YGnGueEpqe0KHFMYGwyhJ+aJV5vKw/e2e7nzXqz8SrMKgQEQDgQeIDuNu4620Ljw2HRq2Bl63Xax8/PPoGRu8djF6BfeBg6YQA92rQqDXZ2p2MOgkJEu7H3cOIA1+gU8l2GFPvSwDATzsGw1Zjj65vT3rx6zy5Amcbd9gbCHpCCKTr0mGuMkf3zT2gEzr83vI3ADC4FHnrP97FzbSHGFi5Nx5HnUOft75AnJkF4tLjcPHxRTQt0RR2FpnB7UHiA6w5OgvxyY+xLOrpuhdHux5FYloC1lxYgbnnfzJY8zch09CwZAvcT7qP1ZdWYeH5zPssjSndBXUqd8G+8C3oUKWn3uel1WlxI/YGvt3cHyfSIzElZAq2RezBe2U/wMG7/2LppWWo7V4Thx4cBQCYq8yQrnv6T3JnCwfEpMZC+8zbXtN6DcoVK4dGi4LxQJWQrc5xtcdh8qHJett2d9gNrdBiT8QerL6yGpWdK+PD8h+ijGMZqCQV2qx/D7fjIwAAQ6sNxrcnZ2c7bn3nAHzdbCFG7eyHnQ+OZNvvFi3w/iEdNtVSoWbN97DxRs4XsXyZOhd0uOMiGQxKBS3wug4xthJuub+6Fk2agEc0cMsNeiHa74FA2bsCO6tJ2QKr+xMBv4cCR8pKrwzeADIDOpCzts+YUHUo2gf2fHVDIzCoEJHJik2Nfa2epZfJ+uvspTcX1KYjNi0WLlYuRh37+KHZWHdxOYa1WAgn98xhuHRdOqYemoranrXxts/bsIaEP8M3w9e+OGp61NR7/t/nl+JgxD5MaDIP5irDd5B+9n1k6DJgrtZv9yTlCYpZFsO+O/uw7eY2jKw1EnYWdhBCICE9AXYWdkjXpWPa3glYE7EJLmYO2NN1PwAgLjUOI8NG4Z/IzPlZ3jZeWNx8CTxsPLDv2iYsOPUD2lfpBVc7L9TxqpOtphfVKUkSft85Dtvv7MK89usQnnAXJ+/9i4+r9YVKUum1Xf3Pl4iTBLrWGIoWqxvhkUiFCsDp7mdxL+4Omq5rjiq2vljebjOEENh54y/MPzEH8UKL7xvOxd47exHkFoQAhzI4fXENzkVfxHd3d8JSMsPbvg2w/fZOvfrmvDsHxa1cMStsFOqVaonryQ9w/d5ROFi7oqx7EH489wsyhBb/K90R/euMxYXHF+Dv4A+VTovNh2bh7YCP4OJUEmcensGZh2cw4+gMaFQWWP/+BrinJuHrVa3wr7MPbqY8hIPaCjs6hcHKzApbzy/DlcfXsCB8jV49/lZeWNp6BZJ1aXC3cceN2BtwtXLFd1tGYGXMAbmdh8YJOzrtg07oEPhbIACglW9jWJlbYfWNjdnOw6DAfnivbFvMOzUPHct2xM3YcKy/vAo9q36OeSfn4X58BKJSo/We0zewD7bd2Izr8beyHU+ChL/abMSEsOFoWroNOlXqlqM/D8ZgUCEiKsJSMlLw942/Ude7rl6PmlanRdidMAS6Bsrzq5RyI/oq5h0KxWc1h6OsS0UAmSHW1twWapVxN9jLCkxp2jQM3T0Itb3r4KOKef/lGpMSAytzK4O9hs/T6rTot6UHyjmVwyfVB+DSk0uo6VFTL7w961TUKeyO2I1A10BUd6sOR0tHAMBPp3/ClvAtWNxsMRwtHRGdEg0HjQNUkgprr6zF7ojd+Lr+17Ays8rRe4hNjUV4bDgCXQMhSRIeJT+Cs6Vz5o0vU2NxKuoUQrxDYKbKn3v8ZGFQISIiIpNlzPf3i6fGExERESlM0aCyb98+tG7dGl5eXpAkCevXr1eyHCIiIjIxigaVxMREBAYGYt68eUqWQURERCYqf2fLvELz5s3RvHlzJUsgIiIiE6ZoUDFWamoqUlOfLuwUFxenYDVERESU3wrVZNrQ0FA4ODjIP76+vkqXRERERPmoUAWVUaNGITY2Vv6JiIhQuiQiIiLKR4Vq6Eej0UCjefVCO0RERPRmKFQ9KkRERFS0KNqjkpCQgGvXrsmPw8PDcerUKRQrVgzFixdXsDIiIiIyBYoGlWPHjuGdd96RHw8dOhQA0L17dyxevFihqoiIiMhUKBpUGjRogEJ8qyEiIiLKZ5yjQkRERCaLQYWIiIhMVqG6PPl5WcNGXKGWiIio8Mj63s7J9I9CHVTi4+MBgCvUEhERFULx8fFwcHB4aRtJFOLZrDqdDvfu3YOdnR0kScrTY8fFxcHX1xcRERGwt7fP02NT/uF5K5x43gonnrfCyRTOmxAC8fHx8PLygkr18lkohbpHRaVSwcfHJ19fw97env8HLIR43gonnrfCieetcFL6vL2qJyULJ9MSERGRyWJQISIiIpPFoPICGo0GEyZM4E0QCxmet8KJ561w4nkrnArbeSvUk2mJiIjozcYeFSIiIjJZDCpERERkshhUiIiIyGQxqBAREZHJYlAxYN68eShRogQsLS3x1ltv4ciRI0qXVGSEhoaiZs2asLOzg5ubG95//31cvnxZr01KSgr69esHZ2dn2Nraol27dnjw4IFem9u3b6Nly5awtraGm5sbRowYgYyMDL02e/fuRVBQEDQaDUqXLo3Fixfn99srMqZPnw5JkjB48GB5G8+babp79y4++ugjODs7w8rKClWqVMGxY8fk/UIIjB8/Hp6enrCyskKjRo1w9epVvWM8efIEXbt2hb29PRwdHdGrVy8kJCTotTlz5gzq1asHS0tL+Pr6YubMmQXy/t5UWq0W48aNg7+/P6ysrFCqVClMnjxZ7945b8y5E6RnxYoVwsLCQvz666/i/Pnzonfv3sLR0VE8ePBA6dKKhKZNm4pFixaJc+fOiVOnTokWLVqI4sWLi4SEBLnN559/Lnx9fcWuXbvEsWPHRO3atUWdOnXk/RkZGaJy5cqiUaNG4uTJk2Lz5s3CxcVFjBo1Sm5z48YNYW1tLYYOHSouXLgg5syZI9Rqtdi6dWuBvt830ZEjR0SJEiVEQECAGDRokLyd5830PHnyRPj5+YkePXqIw4cPixs3boht27aJa9euyW2mT58uHBwcxPr168Xp06fFe++9J/z9/UVycrLcplmzZiIwMFAcOnRI/PPPP6J06dKic+fO8v7Y2Fjh7u4uunbtKs6dOyf++OMPYWVlJX766acCfb9vkqlTpwpnZ2fx119/ifDwcLF69Wpha2srvvvuO7nNm3LuGFSeU6tWLdGvXz/5sVarFV5eXiI0NFTBqoquqKgoAUCEhYUJIYSIiYkR5ubmYvXq1XKbixcvCgDi4MGDQgghNm/eLFQqlbh//77cZv78+cLe3l6kpqYKIYT4v//7P1GpUiW91/rwww9F06ZN8/stvdHi4+NFmTJlxI4dO0T9+vXloMLzZpq++OILUbdu3Rfu1+l0wsPDQ3z11VfytpiYGKHRaMQff/whhBDiwoULAoA4evSo3GbLli1CkiRx9+5dIYQQP/zwg3BycpLPY9ZrlytXLq/fUpHRsmVL8cknn+hta9u2rejatasQ4s06dxz6eUZaWhqOHz+ORo0aydtUKhUaNWqEgwcPKlhZ0RUbGwsAKFasGADg+PHjSE9P1ztH5cuXR/HixeVzdPDgQVSpUgXu7u5ym6ZNmyIuLg7nz5+X2zx7jKw2PM+vp1+/fmjZsmW2z5bnzTRt3LgRNWrUQIcOHeDm5oZq1aphwYIF8v7w8HDcv39f7zN3cHDAW2+9pXfeHB0dUaNGDblNo0aNoFKpcPjwYbnN22+/DQsLC7lN06ZNcfnyZURHR+f323wj1alTB7t27cKVK1cAAKdPn8b+/fvRvHlzAG/WuSvUNyXMa48ePYJWq9X7ixIA3N3dcenSJYWqKrp0Oh0GDx6MkJAQVK5cGQBw//59WFhYwNHRUa+tu7s77t+/L7cxdA6z9r2sTVxcHJKTk2FlZZUfb+mNtmLFCpw4cQJHjx7Nto/nzTTduHED8+fPx9ChQzF69GgcPXoUAwcOhIWFBbp37y5/7oY+82fPiZubm95+MzMzFCtWTK+Nv79/tmNk7XNycsqX9/cmGzlyJOLi4lC+fHmo1WpotVpMnToVXbt2BYA36twxqJDJ6tevH86dO4f9+/crXQq9QkREBAYNGoQdO3bA0tJS6XIoh3Q6HWrUqIFp06YBAKpVq4Zz587hxx9/RPfu3RWujl5m1apVWLZsGZYvX45KlSrh1KlTGDx4MLy8vN64c8ehn2e4uLhArVZnuxLhwYMH8PDwUKiqoql///7466+/sGfPHvj4+MjbPTw8kJaWhpiYGL32z54jDw8Pg+cwa9/L2tjb2/Nf5blw/PhxREVFISgoCGZmZjAzM0NYWBi+//57mJmZwd3dnefNBHl6eqJixYp62ypUqIDbt28DePq5v+zvRA8PD0RFRentz8jIwJMnT4w6t2ScESNGYOTIkejUqROqVKmCbt26YciQIQgNDQXwZp07BpVnWFhYoHr16ti1a5e8TafTYdeuXQgODlawsqJDCIH+/ftj3bp12L17d7Yux+rVq8Pc3FzvHF2+fBm3b9+Wz1FwcDDOnj2r93/AHTt2wN7eXv5LOTg4WO8YWW14nnOnYcOGOHv2LE6dOiX/1KhRA127dpV/53kzPSEhIdku/79y5Qr8/PwAAP7+/vDw8ND7zOPi4nD48GG98xYTE4Pjx4/LbXbv3g2dToe33npLbrNv3z6kp6fLbXbs2IFy5cpx2CeXkpKSoFLpf4Wr1WrodDoAb9i5K7Bpu4XEihUrhEajEYsXLxYXLlwQ//vf/4Sjo6PelQiUf/r06SMcHBzE3r17RWRkpPyTlJQkt/n8889F8eLFxe7du8WxY8dEcHCwCA4OlvdnXebapEkTcerUKbF161bh6upq8DLXESNGiIsXL4p58+bxMtc89uxVP0LwvJmiI0eOCDMzMzF16lRx9epVsWzZMmFtbS2WLl0qt5k+fbpwdHQUGzZsEGfOnBFt2rQxeIlrtWrVxOHDh8X+/ftFmTJl9C5xjYmJEe7u7qJbt27i3LlzYsWKFcLa2pqXJ7+G7t27C29vb/ny5D///FO4uLiI//u//5PbvCnnjkHFgDlz5ojixYsLCwsLUatWLXHo0CGlSyoyABj8WbRokdwmOTlZ9O3bVzg5OQlra2vxwQcfiMjISL3j3Lx5UzRv3lxYWVkJFxcXMWzYMJGenq7XZs+ePaJq1arCwsJClCxZUu816PU9H1R43kzTpk2bROXKlYVGoxHly5cXP//8s95+nU4nxo0bJ9zd3YVGoxENGzYUly9f1mvz+PFj0blzZ2Frayvs7e1Fz549RXx8vF6b06dPi7p16wqNRiO8vb3F9OnT8/29vcni4uLEoEGDRPHixYWlpaUoWbKkGDNmjN5lxG/KuZOEeGYZOyIiIiITwjkqREREZLIYVIiIiMhkMagQERGRyWJQISIiIpPFoEJEREQmi0GFiIiITBaDChEREZksBhUiIiIyWQwqRJTvHj58iD59+qB48eLQaDTw8PBA06ZNceDAAQCAJElYv369skUSkUkyU7oAInrztWvXDmlpaViyZAlKliyJBw8eYNeuXXj8+LHSpRGRieMS+kSUr2JiYuDk5IS9e/eifv362faXKFECt27dkh/7+fnh5s2bAIANGzZg4sSJuHDhAry8vNC9e3eMGTMGZmaZ/8aSJAk//PADNm7ciL1798LT0xMzZ85E+/btC+S9EVH+49APEeUrW1tb2NraYv369UhNTc22/+jRowCARYsWITIyUn78zz//4OOPP8agQYNw4cIF/PTTT1i8eDGmTp2q9/xx48ahXbt2OH36NLp27YpOnTrh4sWL+f/GiKhAsEeFiPLd2rVr0bt3byQnJyMoKAj169dHp06dEBAQACCzZ2TdunV4//335ec0atQIDRs2xKhRo+RtS5cuxf/93//h3r178vM+//xzzJ8/X25Tu3ZtBAUF4YcffiiYN0dE+Yo9KkSU79q1a4d79+5h48aNaNasGfbu3YugoCAsXrz4hc85ffo0Jk2aJPfI2Nraonfv3oiMjERSUpLcLjg4WO95wcHB7FEheoNwMi0RFQhLS0s0btwYjRs3xrhx4/Dpp59iwoQJ6NGjh8H2CQkJmDhxItq2bWvwWERUNLBHhYgUUbFiRSQmJgIAzM3NodVq9fYHBQXh8uXLKF26dLYflerpX12HDh3Se96hQ4dQoUKF/H8DRFQg2KNCRPnq8ePH6NChAz755BMEBATAzs4Ox44dw8yZM9GmTRsAmVf+7Nq1CyEhIdBoNHBycsL48ePRqlUrFC9eHO3bt4dKpcLp06dx7tw5TJkyRT7+6tWrUaNGDdStWxfLli3DkSNHsHDhQqXeLhHlMU6mJaJ8lZqaii+//BLbt2/H9evXkZ6eDl9fX3To0AGjR4+GlZUVNm3ahKFDh+LmzZvw9vaWL0/etm0bJk2ahJMnT8Lc3Bzly5fHp59+it69ewPInEw7b948rF+/Hvv27YOnpydmzJiBjh07KviOiSgvMagQUaFl6GohInqzcI4KERERmSwGFSIiIjJZnExLRIUWR66J3nzsUSEiIiKTxaBCREREJotBhYiIiEwWgwoRERGZLAYVIiIiMlkMKkRERGSyGFSIiIjIZDGoEBERkcliUCEiIiKT9f/vBgAlYAAHCQAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot training loss\n",
    "if run_training:\n",
    "    plt.figure()\n",
    "    plt.xlabel(\"Step\")\n",
    "    plt.ylabel(\"Cross entropy loss\")\n",
    "    plt.title(\"Training loss\")\n",
    "    runs[\"dense\"] = {\"model\": dense_model, \"loss\": dense_loss}\n",
    "    for name, run in runs.items():\n",
    "        loss = run[\"loss\"]\n",
    "        plt.plot(loss.values(), label=name)\n",
    "    plt.legend()\n",
    "    plt.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at the average loss at the end of the training:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjAAAAGzCAYAAAAxPS2EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAvyklEQVR4nO3de1xVdb7/8fcGdYMK2zBQSECii9cMbVTURi0YM6PsomY2ombnHDMvmZ1kplSy9Nix0mnIM5TjJVPTUssyOeaMkuVkmnR56COj8JKhZio3E5P9/f3Rz33cCcZG2PDV1/PxWI+H67u++7s/i1nt/Z61vmtthzHGCAAAwCIBtV0AAACArwgwAADAOgQYAABgHQIMAACwDgEGAABYhwADAACsQ4ABAADWIcAAAADrEGAAAIB1CDCAJfbs2SOHw6EFCxbU6Pu0bNlSw4YNq9H3+C1Tp06Vw+Go0msXLFggh8OhPXv2VG9RlXAhdQPwDQEGqCPOfPGWt0yaNKm2y/Ny4sQJTZ06VRs3bqztUgBcourVdgEAvD311FOKi4vzamvXrp1iY2P1008/qX79+rVU2f85ceKE0tPTJUm9evWq9vGfeOKJKoe2P/7xj7r33nvldDqruSoAdQkBBqhj+vbtqxtuuKHcbUFBQX6upnqUlJSoUaNGle5fr1491atXtY+nwMBABQYGVum1AOzBJSTAEuXNgRk2bJgaN26sAwcOqH///mrcuLHCw8M1ceJElZWVeb1+1qxZ6tatm5o2barg4GB16tRJb7zxRpXqCA8PlySlp6d7LnNNnTrVq6ZvvvlGt956q0JCQjRkyBBJ0gcffKABAwYoJiZGTqdT0dHReuSRR/TTTz95vUd5c0kcDocefvhhrV69Wu3atZPT6VTbtm21bt06r37lzYFp2bKlbrvtNm3evFmdO3dWUFCQrrzySi1atOic/fv888/Vs2dPBQcHq0WLFnr66ac1f/78Ks+rOX36tKZNm6b4+Hg5nU61bNlSf/rTn1RaWurVb9u2berTp48uv/xyBQcHKy4uTiNGjPDqs2zZMnXq1EkhISEKDQ1V+/btNWfOHJ9rAi4GnIEB6piCggIdOXLEq+3yyy+vsH9ZWZn69OmjLl26aNasWXr//ff13HPPKT4+XqNGjfL0mzNnjm6//XYNGTJEp06d0rJlyzRgwAC988476tevX6XrCw8P19y5czVq1CjdeeeduuuuuyRJ1113nafP6dOn1adPH/Xo0UOzZs1Sw4YNJUkrVqzQiRMnNGrUKDVt2lRbt27Viy++qO+++04rVqz4zffevHmzVq5cqYceekghISH6y1/+orvvvlv79u1T06ZNz/va3Nxc3XPPPXrggQeUmpqqv//97xo2bJg6deqktm3bSpIOHDig3r17y+FwKC0tTY0aNdIrr7xyQZejRo4cqYULF+qee+7Ro48+qo8//lgzZszQrl27tGrVKknS4cOH9Yc//EHh4eGaNGmSmjRpoj179mjlypWecdavX6/Bgwfr5ptv1syZMyVJu3bt0ocffqhx48ZVuT7AWgZAnTB//nwjqdzFGGPy8vKMJDN//nzPa1JTU40k89RTT3mNlZCQYDp16uTVduLECa/1U6dOmXbt2pmbbrrJqz02Ntakpqaet9YffvjBSDJTpkw5Z9uZmiZNmnTOtl/XYIwxM2bMMA6Hw+zdu9fTNmXKFPPrjydJpkGDBiY3N9fT9tlnnxlJ5sUXX/S0nfk75uXlee2TJJOdne1pO3z4sHE6nebRRx/1tI0ZM8Y4HA6zY8cOT9uPP/5owsLCzhmzPL+uOycnx0gyI0eO9Oo3ceJEI8n84x//MMYYs2rVKiPJfPLJJxWOPW7cOBMaGmpOnz593hqASwWXkIA6JiMjQ+vXr/dafst//Md/eK3feOON+vbbb73agoODPf8+duyYCgoKdOONN+rTTz+tnsJ/5eyzP+XVUFJSoiNHjqhbt24yxmjHjh2/OWZSUpLi4+M969ddd51CQ0PP2dfytGnTRjfeeKNnPTw8XNdee63Xa9etW6fExERdf/31nrawsDDPJTBfrV27VpI0YcIEr/ZHH31UkvTuu+9Kkpo0aSJJeuedd/Tzzz+XO1aTJk1UUlJSqeMBuBQQYIA6pnPnzkpKSvJazicoKMgzJ+WMyy67TMeOHfNqe+edd9S1a1cFBQUpLCzMcymooKCg2vehXr16atGixTnt+/bt07BhwxQWFuaZr9OzZ09JqlQdMTEx57SVt69Vfe3evXt11VVXndOvvLbK2Lt3rwICAs55ffPmzdWkSRPt3btXktSzZ0/dfffdSk9P1+WXX6477rhD8+fP95on89BDD+maa65R37591aJFC40YMeKc+T/ApYQAA1iuMnfcfPDBB7r99tsVFBSkl156SWvXrtX69et13333yRhT7TU5nU4FBHh/vJSVlSk5OVnvvvuuHn/8ca1evVrr16/3TEp2u92/OW5F+1qZfbiQ116o33q4ncPh0BtvvKEtW7bo4Ycf1oEDBzRixAh16tRJxcXFkqSIiAjl5OTo7bff1u23365//vOf6tu3r1JTU2u8fqAuYhIvcAl48803FRQUpKysLK8JqfPnz6/SeFV52uwXX3yh3bt3a+HChRo6dKinvS5dEomNjVVubu457eW1VXY8t9utr7/+Wq1bt/a0Hzp0SMePH1dsbKxX/65du6pr16565plntGTJEg0ZMkTLli3TyJEjJUkNGjRQSkqKUlJS5Ha79dBDD+lvf/ubnnzyySqfJQJsxRkY4BIQGBgoh8PhdWv1nj17tHr16iqNd+auouPHj/tUg+R9xsMYU6duA+7Tp4+2bNminJwcT9vRo0f12muvVWm8W2+9VZI0e/Zsr/bnn39ekjx3fx07duycM0Fn5uGcuYz0448/em0PCAjw3Pn161uygUsBZ2CAS0C/fv30/PPP65ZbbtF9992nw4cPKyMjQ1dddZU+//xzn8cLDg5WmzZt9Prrr+uaa65RWFiY2rVrp3bt2lX4mlatWik+Pl4TJ07UgQMHFBoaqjfffLNS81f85T//8z+1ePFiJScna8yYMZ7bqGNiYnT06FGfzzx16NBBqampyszM1PHjx9WzZ09t3bpVCxcuVP/+/dW7d29J0sKFC/XSSy/pzjvvVHx8vIqKivTyyy8rNDTUE4JGjhypo0eP6qabblKLFi20d+9evfjii7r++uu9zu4AlwoCDHAJuOmmmzRv3jz913/9l8aPH6+4uDjNnDlTe/bsqVKAkaRXXnlFY8aM0SOPPKJTp05pypQp5w0w9evX15o1azR27FjNmDFDQUFBuvPOO/Xwww+rQ4cOVd21ahUdHa1//vOfGjt2rKZPn67w8HCNHj1ajRo10tixY6v0JORXXnlFV155pRYsWKBVq1apefPmSktL05QpUzx9zgSbZcuW6dChQ3K5XOrcubNee+01z89K3H///crMzNRLL72k48ePq3nz5ho0aJCmTp16znwj4FLgMP6YwQYAFhs/frz+9re/qbi4mJ8pAOoIYjsAnOXXP2vw448/6tVXX1WPHj0IL0AdwiUkADhLYmKievXqpdatW+vQoUOaN2+eCgsL9eSTT9Z2aQDOQoABgLPceuuteuONN5SZmSmHw6GOHTtq3rx5+v3vf1/bpQE4C3NgAACAdZgDAwAArEOAAQAA1rlo5sC43W59//33CgkJqdJjzgEAgP8ZY1RUVKSoqCifnml00QSY77//XtHR0bVdBgAAqIL9+/eX+yv2FbloAkxISIikX/4AoaGhtVwNAACojMLCQkVHR3u+xyvrogkwZy4bhYaGEmAAALCMr9M/mMQLAACsQ4ABAADWIcAAAADrEGAAAIB1CDAAAMA6BBgAAGAdAgwAALAOAQYAAFjnonmQXU0ocxttzTuqw0UnFRESpM5xYQoM4HeWAACobQSYCqz7Ml/pa3Yqv+Ckpy3SFaQpKW10S7vIWqwMAABwCakc677M16jFn3qFF0k6WHBSoxZ/qnVf5tdSZQAAQCLAnKPMbZS+ZqdMOdvOtKWv2akyd3k9AACAPxBgfmVr3tFzzryczUjKLziprXlH/VcUAADwQoD5lcNFFYeXqvQDAADVjwDzKxEhQdXaDwAAVD8CzK90jgtTpCtIFd0s7dAvdyN1jgvzZ1kAAOAsBJhfCQxwaEpKG0k6J8ScWZ+S0obnwQAAUIsIMOW4pV2k5t7fUc1d3peJmruCNPf+jjwHBgCAWsaD7CpwS7tIJbdpzpN4AQCogwgw5xEY4FBifNPaLgMAAPwKl5AAAIB1CDAAAMA6BBgAAGAdAgwAALAOAQYAAFiHAAMAAKxDgAEAANYhwAAAAOsQYAAAgHUIMAAAwDoEGAAAYB0CDAAAsA4BBgAAWIcAAwAArEOAAQAA1iHAAAAA6xBgAACAdQgwAADAOgQYAABgHQIMAACwDgEGAABYx+cAk52drZSUFEVFRcnhcGj16tXn7b9582Z1795dTZs2VXBwsFq1aqUXXnjhnH4ZGRlq2bKlgoKC1KVLF23dutXX0gAAwCXC5wBTUlKiDh06KCMjo1L9GzVqpIcffljZ2dnatWuXnnjiCT3xxBPKzMz09Hn99dc1YcIETZkyRZ9++qk6dOigPn366PDhw76WBwAALgEOY4yp8osdDq1atUr9+/f36XV33XWXGjVqpFdffVWS1KVLF/3ud7/TX//6V0mS2+1WdHS0xowZo0mTJpU7RmlpqUpLSz3rhYWFio6OVkFBgUJDQ6u2QwAAwK8KCwvlcrl8/v72+xyYHTt26KOPPlLPnj0lSadOndL27duVlJT0f0UFBCgpKUlbtmypcJwZM2bI5XJ5lujo6BqvHQAA1A1+CzAtWrSQ0+nUDTfcoNGjR2vkyJGSpCNHjqisrEzNmjXz6t+sWTMdPHiwwvHS0tJUUFDgWfbv31+j9QMAgLqjnr/e6IMPPlBxcbH+9a9/adKkSbrqqqs0ePDgKo/ndDrldDqrsUIAAGALvwWYuLg4SVL79u116NAhTZ06VYMHD9bll1+uwMBAHTp0yKv/oUOH1Lx5c3+VBwAALFIrz4Fxu92eCbgNGjRQp06dtGHDBq/tGzZsUGJiYm2UBwAA6jifz8AUFxcrNzfXs56Xl6ecnByFhYUpJiZGaWlpOnDggBYtWiTpl+e7xMTEqFWrVpJ+eY7MrFmzNHbsWM8YEyZMUGpqqm644QZ17txZs2fPVklJiYYPH36h+wcAAC5CPgeYbdu2qXfv3p71CRMmSJJSU1O1YMEC5efna9++fZ7tbrdbaWlpysvLU7169RQfH6+ZM2fq3//93z19Bg0apB9++EGTJ0/WwYMHdf3112vdunXnTOwFAACQLvA5MHVJVe8jBwAAtcea58AAAABcKAIMAACwDgEGAABYhwADAACsQ4ABAADWIcAAAADrEGAAAIB1CDAAAMA6BBgAAGAdAgwAALAOAQYAAFiHAAMAAKxDgAEAANYhwAAAAOsQYAAAgHUIMAAAwDoEGAAAYB0CDAAAsA4BBgAAWIcAAwAArEOAAQAA1iHAAAAA6xBgAACAdQgwAADAOgQYAABgHQIMAACwDgEGAABYhwADAACsQ4ABAADWIcAAAADrEGAAAIB1CDAAAMA6BBgAAGAdAgwAALAOAQYAAFiHAAMAAKxDgAEAANYhwAAAAOsQYAAAgHUIMAAAwDoEGAAAYB0CDAAAsA4BBgAAWIcAAwAArEOAAQAA1iHAAAAA6xBgAACAdQgwAADAOgQYAABgHZ8DTHZ2tlJSUhQVFSWHw6HVq1eft//KlSuVnJys8PBwhYaGKjExUVlZWV59ioqKNH78eMXGxio4OFjdunXTJ5984mtpAADgEuFzgCkpKVGHDh2UkZFRqf7Z2dlKTk7W2rVrtX37dvXu3VspKSnasWOHp8/IkSO1fv16vfrqq/riiy/0hz/8QUlJSTpw4ICv5QEAgEuAwxhjqvxih0OrVq1S//79fXpd27ZtNWjQIE2ePFk//fSTQkJC9NZbb6lfv36ePp06dVLfvn319NNPV2rMwsJCuVwuFRQUKDQ01Kd6AABA7ajq93e9GqypXG63W0VFRQoLC5MknT59WmVlZQoKCvLqFxwcrM2bN1c4TmlpqUpLSz3rhYWFNVMwAACoc/w+iXfWrFkqLi7WwIEDJUkhISFKTEzUtGnT9P3336usrEyLFy/Wli1blJ+fX+E4M2bMkMvl8izR0dH+2gUAAFDL/BpglixZovT0dC1fvlwRERGe9ldffVXGGF1xxRVyOp36y1/+osGDBysgoOLy0tLSVFBQ4Fn279/vj10AAAB1gN8uIS1btkwjR47UihUrlJSU5LUtPj5emzZtUklJiQoLCxUZGalBgwbpyiuvrHA8p9Mpp9NZ02UDAIA6yC9nYJYuXarhw4dr6dKlXhN1f61Ro0aKjIzUsWPHlJWVpTvuuMMf5QEAAMv4fAamuLhYubm5nvW8vDzl5OQoLCxMMTExSktL04EDB7Ro0SJJv1w2Sk1N1Zw5c9SlSxcdPHhQ0i+TdF0ulyQpKytLxhhde+21ys3N1WOPPaZWrVpp+PDh1bGPAADgIuPzGZht27YpISFBCQkJkqQJEyYoISFBkydPliTl5+dr3759nv6ZmZk6ffq0Ro8ercjISM8ybtw4T5+CggKNHj1arVq10tChQ9WjRw9lZWWpfv36F7p/AADgInRBz4GpS3gODAAA9qnq9ze/hQQAAKxDgAEAANYhwAAAAOsQYAAAgHUIMAAAwDoEGAAAYB0CDAAAsA4BBgAAWIcAAwAArEOAAQAA1iHAAAAA6xBgAACAdQgwAADAOgQYAABgHQIMAACwDgEGAABYhwADAACsQ4ABAADWIcAAAADrEGAAAIB1CDAAAMA6BBgAAGAdAgwAALAOAQYAAFiHAAMAAKxDgAEAANYhwAAAAOsQYAAAgHUIMAAAwDoEGAAAYB0CDAAAsA4BBgAAWIcAAwAArEOAAQAA1iHAAAAA6xBgAACAdQgwAADAOgQYAABgHQIMAACwDgEGAABYhwADAACsQ4ABAADWIcAAAADrEGAAAIB1CDAAAMA6BBgAAGAdAgwAALAOAQYAAFiHAAMAAKxDgAEAANbxOcBkZ2crJSVFUVFRcjgcWr169Xn7r1y5UsnJyQoPD1doaKgSExOVlZXl1aesrExPPvmk4uLiFBwcrPj4eE2bNk3GGF/LAwAAlwCfA0xJSYk6dOigjIyMSvXPzs5WcnKy1q5dq+3bt6t3795KSUnRjh07PH1mzpypuXPn6q9//at27dqlmTNn6tlnn9WLL77oa3kAAOAS4DAXcJrD4XBo1apV6t+/v0+va9u2rQYNGqTJkydLkm677TY1a9ZM8+bN8/S5++67FRwcrMWLF1dqzMLCQrlcLhUUFCg0NNSnegAAQO2o6ve33+fAuN1uFRUVKSwszNPWrVs3bdiwQbt375YkffbZZ9q8ebP69u1b4TilpaUqLCz0WgAAwKWhnr/fcNasWSouLtbAgQM9bZMmTVJhYaFatWqlwMBAlZWV6ZlnntGQIUMqHGfGjBlKT0/3R8kAAKCO8esZmCVLlig9PV3Lly9XRESEp3358uV67bXXtGTJEn366adauHChZs2apYULF1Y4VlpamgoKCjzL/v37/bELAACgDvDbGZhly5Zp5MiRWrFihZKSkry2PfbYY5o0aZLuvfdeSVL79u21d+9ezZgxQ6mpqeWO53Q65XQ6a7xuAABQ9/jlDMzSpUs1fPhwLV26VP369Ttn+4kTJxQQ4F1KYGCg3G63P8oDAACW8fkMTHFxsXJzcz3reXl5ysnJUVhYmGJiYpSWlqYDBw5o0aJFkn65bJSamqo5c+aoS5cuOnjwoCQpODhYLpdLkpSSkqJnnnlGMTExatu2rXbs2KHnn39eI0aMqI59BAAAFxmfb6PeuHGjevfufU57amqqFixYoGHDhmnPnj3auHGjJKlXr17atGlThf0lqaioSE8++aRWrVqlw4cPKyoqSoMHD9bkyZPVoEGDStXFbdQAANinqt/fF/QcmLqEAAMAgH2seQ4MAADAhSLAAAAA6xBgAACAdQgwAADAOgQYAABgHQIMAACwDgEGAABYhwADAACsQ4ABAADWIcAAAADrEGAAAIB1CDAAAMA6BBgAAGAdAgwAALAOAQYAAFiHAAMAAKxDgAEAANYhwAAAAOsQYAAAgHUIMAAAwDoEGAAAYB0CDAAAsA4BBgAAWIcAAwAArEOAAQAA1iHAAAAA6xBgAACAdQgwAADAOgQYAABgHQIMAACwDgEGAABYhwADAACsQ4ABAADWIcAAAADrEGAAAIB1CDAAAMA6BBgAAGAdAgwAALAOAQYAAFiHAAMAAKxTr7YLAC41ZW6jrXlHdbjopCJCgtQ5LkyBAY7aLgsArEKAAfxo3Zf5Sl+zU/kFJz1tka4gTUlpo1vaRdZiZQBgFy4hAX6y7st8jVr8qVd4kaSDBSc1avGnWvdlfi1VBgD2IcAAflDmNkpfs1OmnG1n2tLX7FSZu7weAIBfI8AAfrA17+g5Z17OZiTlF5zU1ryj/isKACxGgAH84HBRxeGlKv0A4FJHgAH8ICIkqFr7AcCljgAD+EHnuDBFuoJU0c3SDv1yN1LnuDB/lgUA1iLAAH4QGODQlJQ2knROiDmzPiWlDc+DAYBKIsAAfnJLu0jNvb+jmru8LxM1dwVp7v0deQ4MAPjA5wCTnZ2tlJQURUVFyeFwaPXq1eftv3LlSiUnJys8PFyhoaFKTExUVlaWV5+WLVvK4XCcs4wePdrX8oA67ZZ2kdr8+E1a+mBXzbn3ei19sKs2P34T4QUAfORzgCkpKVGHDh2UkZFRqf7Z2dlKTk7W2rVrtX37dvXu3VspKSnasWOHp88nn3yi/Px8z7J+/XpJ0oABA3wtD6jzAgMcSoxvqjuuv0KJ8U25bAQAVeAwxlT5yVkOh0OrVq1S//79fXpd27ZtNWjQIE2ePLnc7ePHj9c777yjr7/+Wg5H5T7cCwsL5XK5VFBQoNDQUJ/qAQAAtaOq399+/y0kt9utoqIihYWVf7fFqVOntHjxYk2YMOG84aW0tFSlpaWe9cLCwmqvFQAA1E1+n8Q7a9YsFRcXa+DAgeVuX716tY4fP65hw4add5wZM2bI5XJ5lujo6BqoFgAA1EV+DTBLlixRenq6li9froiIiHL7zJs3T3379lVUVNR5x0pLS1NBQYFn2b9/f02UDAAA6iC/XUJatmyZRo4cqRUrVigpKancPnv37tX777+vlStX/uZ4TqdTTqezussEAAAW8MsZmKVLl2r48OFaunSp+vXrV2G/+fPnKyIi4rx9AAAAfD4DU1xcrNzcXM96Xl6ecnJyFBYWppiYGKWlpenAgQNatGiRpF8uG6WmpmrOnDnq0qWLDh48KEkKDg6Wy+XyjON2uzV//nylpqaqXj2/zy0GAAAW8fkMzLZt25SQkKCEhARJ0oQJE5SQkOC5JTo/P1/79u3z9M/MzNTp06c1evRoRUZGepZx48Z5jfv+++9r3759GjFixIXsDwAAuARc0HNg6hKeAwMAgH2q+v3NbyEBAADrEGAAAIB1CDAAAMA6BBgAAGAdAgwAALAOAQYAAFiHAAMAAKxDgAEAANYhwAAAAOsQYAAAgHUIMAAAwDoEGAAAYB0CDAAAsA4BBgAAWIcAAwAArEOAAQAA1iHAAAAA6xBgAACAdQgwAADAOgQYAABgHQIMAACwDgEGAABYhwADAACsQ4ABAADWIcAAAADrEGAAAIB1CDAAAMA6BBgAAGAdAgwAALAOAQYAAFiHAAMAAKxDgAEAANYhwAAAAOsQYAAAgHUIMAAAwDoEGAAAYB0CDAAAsA4BBgAAWIcAAwAArEOAAQAA1iHAAAAA6xBgAACAdQgwAADAOgQYAABgHQIMAACwDgEGAABYhwADAACsU6+2CwAAVI8yt9HWvKM6XHRSESFB6hwXpsAAR22XBdQIAgwAXATWfZmv9DU7lV9w0tMW6QrSlJQ2uqVdZC1WBtQMLiEBgOXWfZmvUYs/9QovknSw4KRGLf5U677Mr6XKgJrjc4DJzs5WSkqKoqKi5HA4tHr16vP2X7lypZKTkxUeHq7Q0FAlJiYqKyvrnH4HDhzQ/fffr6ZNmyo4OFjt27fXtm3bfC0PAC4pZW6j9DU7ZcrZdqYtfc1OlbnL6wHYy+cAU1JSog4dOigjI6NS/bOzs5WcnKy1a9dq+/bt6t27t1JSUrRjxw5Pn2PHjql79+6qX7++3nvvPe3cuVPPPfecLrvsMl/LA4BLyta8o+eceTmbkZRfcFJb8476ryjAD3yeA9O3b1/17du30v1nz57ttT59+nS99dZbWrNmjRISEiRJM2fOVHR0tObPn+/pFxcX52tpAHDJOVxUcXipSj/AFn6fA+N2u1VUVKSwsDBP29tvv60bbrhBAwYMUEREhBISEvTyyy+fd5zS0lIVFhZ6LQBwqYkICarWfoAt/B5gZs2apeLiYg0cONDT9u2332ru3Lm6+uqrlZWVpVGjRmns2LFauHBhhePMmDFDLpfLs0RHR/ujfACoUzrHhSnSFaSKbpZ26Je7kTrHhVXQA7CTXwPMkiVLlJ6eruXLlysiIsLT7na71bFjR02fPl0JCQn6t3/7Nz344IP6n//5nwrHSktLU0FBgWfZv3+/P3YBAOqUwACHpqS0kaRzQsyZ9SkpbXgeDC46fgswy5Yt08iRI7V8+XIlJSV5bYuMjFSbNm282lq3bq19+/ZVOJ7T6VRoaKjXAgCXolvaRWru/R3V3OV9mai5K0hz7+/Ic2BwUfLLg+yWLl2qESNGaNmyZerXr98527t3766vvvrKq2337t2KjY31R3kAYL1b2kUquU1znsSLS4bPAaa4uFi5ubme9by8POXk5CgsLEwxMTFKS0vTgQMHtGjRIkm/XDZKTU3VnDlz1KVLFx08eFCSFBwcLJfLJUl65JFH1K1bN02fPl0DBw7U1q1blZmZqczMzOrYRwC4JAQGOJQY37S2ywD8wmGM8enpRhs3blTv3r3PaU9NTdWCBQs0bNgw7dmzRxs3bpQk9erVS5s2baqw/xnvvPOO0tLS9PXXXysuLk4TJkzQgw8+WOm6CgsL5XK5VFBQwOUkAAAsUdXvb58DTF1FgAEAwD5V/f7mt5AAAIB1CDAAAMA6BBgAAGAdAgwAALAOAQYAAFiHAAMAAKxDgAEAANYhwAAAAOsQYAAAgHUIMAAAwDoEGAAAYB0CDAAAsA4BBgAAWIcAAwAArEOAAQAA1iHAAAAA6xBgAACAdQgwAADAOgQYAABgHQIMAACwDgEGAABYhwADAACsQ4ABAADWIcAAAADrEGAAAIB1CDAAAMA6BBgAAGAdAgwAALAOAQYAAFiHAAMAAKxDgAEAANYhwAAAAOsQYAAAgHUIMAAAwDoEGAAAYB0CDAAAsA4BBgAAWIcAAwAArEOAAQAA1iHAAAAA6xBgAACAdQgwAADAOgQYAABgHQIMAACwDgEGAABYhwADAACsQ4ABAADWIcAAAADrEGAAAIB16tV2AQAAoO4qcxttzTuqw0UnFRESpM5xYQoMcNR2Wb6fgcnOzlZKSoqioqLkcDi0evXq8/ZfuXKlkpOTFR4ertDQUCUmJiorK8urz9SpU+VwOLyWVq1a+VoaAACoRuu+zFePmf/Q4Jf/pXHLcjT45X+px8x/aN2X+bVdmu8BpqSkRB06dFBGRkal+mdnZys5OVlr167V9u3b1bt3b6WkpGjHjh1e/dq2bav8/HzPsnnzZl9LAwAA1WTdl/katfhT5Rec9Go/WHBSoxZ/WushxudLSH379lXfvn0r3X/27Nle69OnT9dbb72lNWvWKCEh4f8KqVdPzZs397UcAABQzcrcRulrdsqUs81IckhKX7NTyW2a19rlJL9P4nW73SoqKlJYWJhX+9dff62oqChdeeWVGjJkiPbt23fecUpLS1VYWOi1AACAC7c17+g5Z17OZiTlF5zU1ryj/ivqV/weYGbNmqXi4mINHDjQ09alSxctWLBA69at09y5c5WXl6cbb7xRRUVFFY4zY8YMuVwuzxIdHe2P8gEAuOgdLqo4vFSlX03wa4BZsmSJ0tPTtXz5ckVERHja+/btqwEDBui6665Tnz59tHbtWh0/flzLly+vcKy0tDQVFBR4lv379/tjFwAAuOhFhARVa7+a4LfbqJctW6aRI0dqxYoVSkpKOm/fJk2a6JprrlFubm6FfZxOp5xOZ3WXCQDAJa9zXJgiXUE6WHCy3HkwDknNXb/cUl1b/HIGZunSpRo+fLiWLl2qfv36/Wb/4uJiffPNN4qMjPRDdQAA4GyBAQ5NSWkj6ZewcrYz61NS2tTq82B8DjDFxcXKyclRTk6OJCkvL085OTmeSbdpaWkaOnSop/+SJUs0dOhQPffcc+rSpYsOHjyogwcPqqCgwNNn4sSJ2rRpk/bs2aOPPvpId955pwIDAzV48OAL3D0AAFAVt7SL1Nz7O6q5y/syUXNXkObe31G3tKvdkwwOY0x5Z4cqtHHjRvXu3fuc9tTUVC1YsEDDhg3Tnj17tHHjRklSr169tGnTpgr7S9K9996r7Oxs/fjjjwoPD1ePHj30zDPPKD4+vtJ1FRYWyuVyqaCgQKGhob7sEgAAqEBNP4m3qt/fPgeYuooAAwCAfar6/c2POQIAAOsQYAAAgHUIMAAAwDoEGAAAYB0CDAAAsA4BBgAAWIcAAwAArEOAAQAA1iHAAAAA6/jt16hr2pkHChcWFtZyJQAAoLLOfG/7+sMAF02AKSoqkiRFR0fXciUAAMBXRUVFcrlcle5/0fwWktvt1vfff6+QkBA5HNX7I1PR0dHav38/v7GEasNxhZrCsYWaUJPHlTFGRUVFioqKUkBA5We2XDRnYAICAtSiRYsaGz80NJQPA1Q7jivUFI4t1ISaOq58OfNyBpN4AQCAdQgwAADAOgSY3+B0OjVlyhQ5nc7aLgUXEY4r1BSOLdSEunhcXTSTeAEAwKWDMzAAAMA6BBgAAGAdAgwAALAOAQYAAFjnkgowvXr10vjx42u7DAAAcIEuqQADXGxyc3MVEhKiJk2a1HYpuAhkZWWpa9euCgkJUXh4uO6++27t2bOntsuCRU6ePKlhw4apffv2qlevnvr3719uv9LSUv35z39WbGysnE6nWrZsqb///e8+vRcBBrDUzz//rMGDB+vGG2+s7VJwEcjLy9Mdd9yhm266STk5OcrKytKRI0d011131XZpsEhZWZmCg4M1duxYJSUlVdhv4MCB2rBhg+bNm6evvvpKS5cu1bXXXuvTe120AaakpERDhw5V48aNFRkZqeeee85re2lpqSZOnKgrrrhCjRo1UpcuXbRx40bP9gULFqhJkybKyspS69at1bhxY91yyy3Kz8/39Nm4caM6d+6sRo0aqUmTJurevbv27t3r2f7WW2+pY8eOCgoK0pVXXqn09HSdPn26xvcdNaNXr14aM2aMxo8fr8suu0zNmjXTyy+/rJKSEg0fPlwhISG66qqr9N5773les2nTJnXu3FlOp1ORkZGaNGmS1zHgdrs1Y8YMxcXFKTg4WB06dNAbb7xRqXqeeOIJtWrVSgMHDqz2fYX/1JXjavv27SorK9PTTz+t+Ph4dezYURMnTlROTo5+/vnnGtt/VI+6chw1atRIc+fO1YMPPqjmzZuX22fdunXatGmT1q5dq6SkJLVs2VKJiYnq3r27bzttLlKjRo0yMTEx5v333zeff/65ue2220xISIgZN26cMcaYkSNHmm7dupns7GyTm5tr/vu//9s4nU6ze/duY4wx8+fPN/Xr1zdJSUnmk08+Mdu3bzetW7c29913nzHGmJ9//tm4XC4zceJEk5uba3bu3GkWLFhg9u7da4wxJjs724SGhpoFCxaYb775xvzv//6vadmypZk6dWqt/D1w4Xr27GlCQkLMtGnTzO7du820adNMYGCg6du3r8nMzDS7d+82o0aNMk2bNjUlJSXmu+++Mw0bNjQPPfSQ2bVrl1m1apW5/PLLzZQpUzxjPv3006ZVq1Zm3bp15ptvvjHz5883TqfTbNy48by1bNiwwcTFxZmCggIzf/5843K5anbnUWPqynH17bffmgYNGphXXnnFnD592hw/ftwMGDDAJCcn++GvgAtVV46js6Wmppo77rjjnPZRo0aZm2++2Tz++OMmKirKXH311ebRRx81J06c8GmfL8oAU1RUZBo0aGCWL1/uafvxxx9NcHCwGTdunNm7d68JDAw0Bw4c8HrdzTffbNLS0owxvwQYSSY3N9ezPSMjwzRr1swznqQK/4e8+eabzfTp073aXn31VRMZGVkt+wj/69mzp+nRo4dn/fTp06ZRo0bmj3/8o6ctPz/fSDJbtmwxf/rTn8y1115r3G63Z3tGRoZp3LixKSsrMydPnjQNGzY0H330kdf7PPDAA2bw4MEV1nHkyBETHR1tNm3aZIwxBBjL1ZXjyhhjNm7caCIiIkxgYKCRZBITE82xY8eqZ0dRo+rScXRGRQGmT58+xul0mn79+pmPP/7YvPvuuyY2NtYMGzbMp32u59v5Gjt88803OnXqlLp06eJpCwsL81xf++KLL1RWVqZrrrnG63WlpaVq2rSpZ71hw4aKj4/3rEdGRurw4cOe8YYNG6Y+ffooOTlZSUlJGjhwoCIjIyVJn332mT788EM988wznteXlZXp5MmTOnHihBo2bFj9O44ad91113n+HRgYqKZNm6p9+/aetmbNmkmSDh8+rF27dikxMVEOh8OzvXv37iouLtZ3332noqIinThxQsnJyV7vcerUKSUkJEiS2rZt67kseeONN+q9997Tgw8+qPvuu0+///3va2w/4V914bg6ePCgHnzwQaWmpmrw4MEqKirS5MmTdc8992j9+vVe74e6qS4cR5XhdrvlcDj02muvyeVySZKef/553XPPPXrppZcUHBxcqXEuygDzW4qLixUYGKjt27crMDDQa1vjxo09/65fv77XNofDIXPWT0fNnz9fY8eO1bp16/T666/riSee0Pr169W1a1cVFxcrPT293AlwQUFB1bxH8Jfyjomz2858GLjd7t8cq7i4WJL07rvv6oorrvDaduYH09auXeuZf3DmP+p//OMfevvttzVr1ixJkjFGbrdb9erVU2ZmpkaMGFGVXUMtqgvHVUZGhlwul5599llP/8WLFys6Oloff/yxunbt6utuwc/qwnFUGZGRkbriiis84UWSWrduLWOMvvvuO1199dWVGueiDDDx8fGqX7++Pv74Y8XExEiSjh07pt27d6tnz55KSEhQWVmZDh8+fMF3cCQkJCghIUFpaWlKTEzUkiVL1LVrV3Xs2FFfffWVrrrqqurYJViodevWevPNN2WM8XxwfPjhhwoJCVGLFi102WWXyel0at++ferZs2e5Y8TGxp7TtmXLFpWVlXnW33rrLc2cOVMfffTROR80uPjU1HF14sQJBQR439dx5v/gVeYLD3apqeOoMrp3764VK1aouLjYc9Jg9+7dCggIUIsWLSo9zkUZYBo3bqwHHnhAjz32mJo2baqIiAj9+c9/9vzHec0112jIkCEaOnSonnvuOSUkJOiHH37Qhg0bdN1116lfv36/+R55eXnKzMzU7bffrqioKH311Vf6+uuvNXToUEnS5MmTddtttykmJkb33HOPAgIC9Nlnn+nLL7/U008/XaP7j7rhoYce0uzZszVmzBg9/PDD+uqrrzRlyhRNmDBBAQEBCgkJ0cSJE/XII4/I7XarR48eKigo0IcffqjQ0FClpqaWO27r1q291rdt26aAgAC1a9fOH7uFWlZTx1W/fv30wgsv6KmnnvJcQvrTn/6k2NhYzyUDXDxq6jiSpJ07d+rUqVM6evSoioqKlJOTI0m6/vrrJUn33Xefpk2bpuHDhys9PV1HjhzRY489phEjRvh0JueinMRrzC8Tee+//37TsGFD06xZM/Pss8+anj17eu5COnXqlJk8ebJp2bKlqV+/vomMjDR33nmn+fzzz40x5U+MXLVqlTnzJzt48KDp37+/iYyMNA0aNDCxsbFm8uTJpqyszNN/3bp1plu3biY4ONiEhoaazp07m8zMTL/sP6rf2cfPGbGxseaFF17wapNkVq1aZYz5ZVLk7373O9OgQQPTvHlz8/jjj5uff/7Z09ftdpvZs2eba6+91tSvX9+Eh4ebPn36eCboVgaTeO1Wl46rpUuXmoSEBNOoUSMTHh5ubr/9drNr167q2E3UsLp0HMXGxhpJ5yxn27Vrl0lKSjLBwcGmRYsWZsKECT7fheT4/zsEAABgjYv2QXYAAODiRYABAADWIcAAAADrEGAAAIB1CDAAAMA6BBgAAGAdAgwAALAOAQYAAFiHAAMAAKxDgAEAANYhwAAAAOv8P4fkPxDVNHeVAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "if run_training:\n",
    "    final_loss = {}\n",
    "    for name, run in runs.items():\n",
    "        loss = run[\"loss\"]\n",
    "        final_loss[name] = np.mean(list(loss.values())[-100:])\n",
    "\n",
    "    final_loss = dict(sorted(final_loss.items(), key=lambda item: item[1], reverse=True))\n",
    "    plt.title(\"Final training loss\")\n",
    "    plt.scatter(final_loss.keys(), final_loss.values())\n",
    "    plt.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We observed that increasing the number of experts indeed improved the loss of the model! Note that importantly, this improvement comes without a proportional increase in compute cost. The mixture of experts scheme only adds the routing logic overhead, but the processing FLOPs per token remain fixed."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Running MoE inference"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see our MoE models in action! We will run inference on the CPU, so we need to make sure that any expert weights of the shape `[num_replicas, num_experts_per_replica, ...]` are reshaped to `[num_replicas * num_experts_per_replica, ...]` as expected by the CPU model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "def flatten_expert_weights(state_dict: dict, moe_layers: Tuple[int, ...]) -> dict:\n",
    "    for i in moe_layers:\n",
    "        w1 = state_dict[f\"core_layers.{i}.ffn.w1\"]\n",
    "        w2 = state_dict[f\"core_layers.{i}.ffn.w2\"]\n",
    "        state_dict[f\"core_layers.{i}.ffn.w1\"] = w1.reshape(-1, *w1.shape[2:])\n",
    "        state_dict[f\"core_layers.{i}.ffn.w2\"] = w2.reshape(-1, *w2.shape[2:])\n",
    "    return state_dict"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also keep track of the router decisions by colouring each input character based on the expert it was routed to:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "def wrap_text(text: str, id: int) -> str:\n",
    "    col_id = id + 1 if id < 15 else id + 20\n",
    "    return \"\\033[38;5;232m\" + f\"\\033[48;5;{col_id}m\" + text + \"\\033[0;0m\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This is a sentence to see how characters are routed to experts.\n",
      "Some numbers: 1, 2, 3, 4, 5, 6, 7, 8, 9.\n",
      " ,\n",
      "Layer 0:\n",
      "\u001b[38;5;232m\u001b[48;5;1mT\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4mh\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4mi\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3ms\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;2mi\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4ms\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;2ma\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;2ms\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3me\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4mn\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3mt\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3me\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2mn\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3mc\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3me\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;4mt\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4mo\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;2ms\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4me\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3me\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;4mh\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4mo\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2mw\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;4mc\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4mh\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4ma\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3mr\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3ma\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2mc\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2mt\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3me\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2mr\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2ms\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;2ma\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4mr\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3me\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;2mr\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4mo\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1mu\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2mt\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4me\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4md\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;2mt\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4mo\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;3me\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4mx\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1mp\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3me\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3mr\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2mt\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2ms\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m.\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m\n",
      "\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1mS\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2mo\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2mm\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1me\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;4mn\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4mu\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4mm\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2mb\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3me\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3mr\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2ms\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m:\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;4m1\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4m,\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m2\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2m,\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m3\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m,\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;2m4\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4m,\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;4m5\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2m,\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;2m6\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2m,\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;2m7\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2m,\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;4m8\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m,\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;4m9\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m.\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m\n",
      "\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1m \u001b[0;0m\n",
      "Layer 2:\n",
      "\u001b[38;5;232m\u001b[48;5;2mT\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3mh\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4mi\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2ms\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;1mi\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2ms\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;4ma\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;4ms\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4me\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4mn\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4mt\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2me\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1mn\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1mc\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2me\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;2mt\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4mo\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;4ms\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4me\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4me\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;3mh\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4mo\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4mw\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;4mc\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4mh\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1ma\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4mr\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1ma\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4mc\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4mt\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2me\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2mr\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2ms\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;2ma\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1mr\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4me\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;2mr\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4mo\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4mu\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2mt\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4me\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2md\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;2mt\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2mo\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;2me\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4mx\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1mp\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4me\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4mr\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4mt\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2ms\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2m.\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4m\n",
      "\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4mS\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2mo\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4mm\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4me\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;2mn\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1mu\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2mm\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2mb\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1me\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4mr\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4ms\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1m:\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m1\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2m,\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;4m2\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m,\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;4m3\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2m,\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;4m4\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2m,\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m5\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2m,\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;4m6\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m,\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;4m7\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2m,\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;2m8\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m,\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;4m9\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2m.\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4m\n",
      "\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4m \u001b[0;0m\n",
      "Layer 4:\n",
      "\u001b[38;5;232m\u001b[48;5;2mT\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2mh\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1mi\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4ms\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;2mi\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3ms\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;3ma\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;2ms\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2me\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1mn\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3mt\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1me\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3mn\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1mc\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2me\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;2mt\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3mo\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;2ms\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2me\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3me\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;2mh\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1mo\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3mw\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;4mc\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3mh\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1ma\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1mr\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4ma\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2mc\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4mt\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1me\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1mr\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2ms\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;1ma\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1mr\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3me\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;1mr\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1mo\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1mu\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4mt\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2me\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2md\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;1mt\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3mo\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;1me\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2mx\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3mp\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2me\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2mr\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1mt\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2ms\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m.\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m\n",
      "\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2mS\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2mo\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2mm\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3me\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;2mn\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1mu\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3mm\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2mb\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2me\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3mr\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2ms\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1m:\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;2m1\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m,\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;2m2\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m,\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;4m3\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m,\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;4m4\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m,\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;2m5\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m,\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;2m6\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m,\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;1m7\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m,\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;2m8\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m,\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;1m9\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m.\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m\n",
      "\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4m \u001b[0;0m\n",
      "Layer 6:\n",
      "\u001b[38;5;232m\u001b[48;5;4mT\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3mh\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2mi\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3ms\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;2mi\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3ms\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;2ma\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;4ms\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4me\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1mn\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4mt\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2me\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1mn\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2mc\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4me\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;4mt\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3mo\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;4ms\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4me\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4me\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;2mh\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3mo\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3mw\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;4mc\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2mh\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1ma\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3mr\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2ma\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1mc\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4mt\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4me\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1mr\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3ms\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;2ma\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2mr\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3me\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;2mr\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3mo\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2mu\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2mt\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4me\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3md\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;4mt\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3mo\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;4me\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3mx\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4mp\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4me\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1mr\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4mt\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3ms\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m.\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m\n",
      "\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3mS\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3mo\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2mm\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4me\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;2mn\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2mu\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3mm\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4mb\u001b[0;0m\u001b[38;5;232m\u001b[48;5;4me\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3mr\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3ms\u001b[0;0m\u001b[38;5;232m\u001b[48;5;2m:\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m1\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m,\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m2\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m,\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m3\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m,\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m4\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m,\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m5\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m,\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m6\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m,\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m7\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m,\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m8\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m,\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1m \u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m9\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m.\u001b[0;0m\u001b[38;5;232m\u001b[48;5;3m\n",
      "\u001b[0;0m\u001b[38;5;232m\u001b[48;5;1m \u001b[0;0m\n"
     ]
    }
   ],
   "source": [
    "# Set number of experts (can be 4, 8 or 16)\n",
    "num_experts = 4\n",
    "\n",
    "model_from_checkpoint = MoeCharModel(\n",
    "    vocab_size=len(ds.vocab),\n",
    "    hidden_size=128,\n",
    "    expand_factor=4,\n",
    "    kernel_size=7,\n",
    "    num_groups=8,\n",
    "    num_layers=8,\n",
    "    dtype=torch.float32,\n",
    "    num_replicas=1,\n",
    "    num_experts_per_replica=num_experts,\n",
    "    capacity_factor=1.0,\n",
    "    moe_layers=(0, 2, 4, 6),\n",
    "    aux_loss_weight=0.01,\n",
    ")\n",
    "state_dict = flatten_expert_weights(\n",
    "    torch.load(checkpoint_dir / f\"moe-{num_experts}.pt\"), moe_layers=(0, 2, 4, 6)\n",
    ")\n",
    "model_from_checkpoint.load_state_dict(state_dict)\n",
    "\n",
    "in_text = \"This is a sentence to see how characters are routed to experts.\\nSome numbers: 1, 2, 3, 4, 5, 6, 7, 8, 9.\\n \"\n",
    "output = sample_text(\n",
    "    model_from_checkpoint,\n",
    "    in_text,\n",
    "    num_characters=1,\n",
    "    temperature=0.3,\n",
    ")\n",
    "for i in (0, 2, 4, 6):\n",
    "    print(f\"Layer {i}:\")\n",
    "    expert_ids = model_from_checkpoint.core_layers[i].ffn.expert_ids.numpy()\n",
    "    colored_text = \"\"\n",
    "    for c, id in zip(output, expert_ids):\n",
    "        colored_text += wrap_text(c, id)\n",
    "    print(colored_text)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We presented a tutorial on how to define a model that uses a mixture of experts execution scheme to increase the model capacity without a proportional increase in computational cost. Although the notebook covered defining a character-level language model, the techniques described here can be easily applied to making arbitrary expert-parallel layers. Moreover, by simply exchanging the FFN layers with the `ExpertFFN` layers defined here and adding the expert-balancing loss to our final loss calculation, we can adapt arbitrary transformer-like models to utilise mixture of experts and run the model on both IPUs and CPUs."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "3.2.0-EA.1+1218_poptorch",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "c23b2585764a5cea999a38a63cb5de975e465d0e20532e67163f65bb712149b7"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}