Neural_Machine_Translation_tf_2L_8H.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Neural_Machine_Translation_tf-2L-8H.ipynb",
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "gpuClass": "standard",
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "#### Mounting Google Drive"
      ],
      "metadata": {
        "id": "0dqNGaFLKKAJ"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bVenoImvJZs9"
      },
      "outputs": [],
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "cd \"/content/drive/MyDrive/IASNLP\""
      ],
      "metadata": {
        "id": "iNpRDc1-KMj8"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Importing Necessary Libraries"
      ],
      "metadata": {
        "id": "OlMuBUdIApMX"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install sentencepiece\n",
        "!pip install sacrebleu\n",
        "!pip install tensorflow-gpu # ---> For GPU"
      ],
      "metadata": {
        "id": "Iwi6TDKsB0St"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "\n",
        "import tensorflow as tf\n",
        "from tensorflow import keras\n",
        "from tensorflow.keras import layers\n",
        "from tensorflow.keras import Model\n",
        "from tensorflow.keras import Sequential\n",
        "from tensorflow.keras.layers import Dense, Lambda, Layer, Embedding, LayerNormalization\n",
        "import sentencepiece as spm\n",
        "from sacrebleu.metrics import BLEU, CHRF, TER\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "from collections import Counter"
      ],
      "metadata": {
        "id": "DXSYYy2b-jMM"
      },
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Loading Data"
      ],
      "metadata": {
        "id": "ToLMSNAHBnvJ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "We load the `train_data`, `train_dev_data`, `test_val_data` and `test_data` as well as the byte-pair encoder tokenizer for English and Bengali i.e. `sp_en_bpe` and `s_ben_bpe`."
      ],
      "metadata": {
        "id": "zXIf-FL7IG7f"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "train_data, train_dev_data = pd.read_csv(\"train_data.csv\")[['src', 'tgt']], pd.read_csv(\"train_dev.csv\")[['src', 'tgt']] \n",
        "test_val_data, test_data = pd.read_csv(\"test_val.csv\")[['src', 'tgt']], pd.read_csv(\"test_data.csv\")[['src', 'tgt']]\n",
        "sp_en_bpe, sp_ben_bpe = spm.SentencePieceProcessor(), spm.SentencePieceProcessor()\n",
        "sp_en_bpe.load('eng_bpe.model'); sp_ben_bpe.load('ben_bpe.model');"
      ],
      "metadata": {
        "id": "IXsCg4mBBeoo"
      },
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Data Preprocessing"
      ],
      "metadata": {
        "id": "auf4TlPFhNtn"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "We decide on some formats of the data and batches. We take the maximum tokens we can have per tokenized English and Bengali sentence to be `MAX_TOKENS = 60`. We use the vocabulary size of `32000`(i.e. `ENCODER_VOCAB = DECODER_VOCAB = 32000`). A batch size of `256` was choosen for our Mini-Batch Gradient Descent which is used to train our Model."
      ],
      "metadata": {
        "id": "3z7BUnn_G3WL"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "MAX_TOKENS = 60\n",
        "VOCAB_SIZE = 32000\n",
        "BATCH_SIZE = 256\n",
        "BUFFER_SIZE = BATCH_SIZE*4"
      ],
      "metadata": {
        "id": "7jTuQSqOAybO"
      },
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "The following two functions help in tokenizing an detokenizing the data based on our sentencepiece byte-pair encoding model."
      ],
      "metadata": {
        "id": "NlEW7GndbCfw"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def tokenize(sentence, sp_model, add_bos = True, add_eos = True):\n",
        "    bos = [sp_model.bos_id()] if add_bos else []\n",
        "    eos = [sp_model.eos_id()] if add_eos else []\n",
        "    inputs = bos + sp_model.encode_as_ids(sentence) + eos\n",
        "    return np.reshape(np.array(inputs), [1, -1])\n",
        "def detokenize(tokenized, sp_model, is_bos = True, is_eos = True):\n",
        "    integers = np.squeeze(tokenized).tolist()\n",
        "    if is_eos:\n",
        "        return sp_model.DecodeIdsWithCheck(integers[int(is_bos):integers.index(sp_model.eos_id())])\n",
        "    else:\n",
        "        if sp_model.pad_id() in tokenized:\n",
        "            return sp_model.DecodeIdsWithCheck(integers[int(is_bos):integers.index(sp_model.pad_id())])\n",
        "        else:\n",
        "            return sp_model.DecodeIdsWithCheck(integers[int(is_bos):])"
      ],
      "metadata": {
        "id": "VC-6hav2BrJJ"
      },
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "We have tokenized the data."
      ],
      "metadata": {
        "id": "5-76mwBnbuzp"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Tokenization\n",
        "src_train_data_enc = [np.squeeze(tokenize(train_data['src'].iloc[i], sp_en_bpe)) for i in range(train_data.shape[0])]\n",
        "tgt_train_data_enc = [np.squeeze(tokenize(train_data['tgt'].iloc[i], sp_ben_bpe)) for i in range(train_data.shape[0])]\n",
        "src_train_dev_data_enc = [np.squeeze(tokenize(train_dev_data['src'].iloc[i], sp_en_bpe)) for i in range(train_dev_data.shape[0])]\n",
        "tgt_train_dev_data_enc = [np.squeeze(tokenize(train_dev_data['tgt'].iloc[i], sp_ben_bpe)) for i in range(train_dev_data.shape[0])]\n",
        "src_test_val_data_enc = [np.squeeze(tokenize(test_val_data['src'].iloc[i], sp_en_bpe)) for i in range(test_val_data.shape[0])]\n",
        "tgt_test_val_data_enc = [np.squeeze(tokenize(test_val_data['tgt'].iloc[i], sp_ben_bpe)) for i in range(test_val_data.shape[0])]"
      ],
      "metadata": {
        "id": "nT0QpepuDoFF"
      },
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "We have padded the data with post-truncation and post padding upto MAX_TOKENS number of tokens."
      ],
      "metadata": {
        "id": "wdyEjbm5hWXg"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Padding\n",
        "# The reason why we add MAX_TOKENS+1 after-pad length for Bengali is that when we form the tf.Dataset later, we will shift the target sentence once and will truncate the last token once\n",
        "# Hence, it will get adjusted\n",
        "train_src = tf.keras.preprocessing.sequence.pad_sequences(src_train_data_enc, maxlen = MAX_TOKENS, padding='post', truncating='post')\n",
        "train_tgt = tf.keras.preprocessing.sequence.pad_sequences(tgt_train_data_enc, maxlen = MAX_TOKENS, padding='post', truncating='post')\n",
        "train_dev_src = tf.keras.preprocessing.sequence.pad_sequences(src_train_dev_data_enc, maxlen = MAX_TOKENS, padding='post', truncating='post')\n",
        "train_dev_tgt = tf.keras.preprocessing.sequence.pad_sequences(tgt_train_dev_data_enc, maxlen = MAX_TOKENS, padding='post', truncating='post')\n",
        "test_val_src = tf.keras.preprocessing.sequence.pad_sequences(src_test_val_data_enc, maxlen = MAX_TOKENS, padding='post', truncating='post')\n",
        "test_val_tgt = tf.keras.preprocessing.sequence.pad_sequences(tgt_test_val_data_enc, maxlen = MAX_TOKENS, padding='post', truncating='post')"
      ],
      "metadata": {
        "id": "NT63vC8PcK-2"
      },
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Below, we discard all those pairs where the tokenized source or target sentences are larger than MAX_TOKENS number of tokens."
      ],
      "metadata": {
        "id": "FvxtnfKniMRA"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Removing all pairs whose either the target sentence or source setence exceeds MAX_TOKENS number of tokens\n",
        "same_idx = list(set([i for i in range(train_src.shape[0]) if train_src[i][-1] == 1 or train_src[i][-1] == 0]).intersection(set([i for i in range(train_tgt.shape[0]) if train_tgt[i][-1] == 1 or train_tgt[i][-1] == 0])))\n",
        "train_src = train_src[same_idx]\n",
        "train_tgt = train_tgt[same_idx]\n",
        "same_idx = list(set([i for i in range(train_dev_src.shape[0]) if train_dev_src[i][-1] == 1 or train_dev_src[i][-1] == 0]).intersection(set([i for i in range(train_dev_tgt.shape[0]) if train_dev_tgt[i][-1] == 1 or train_dev_tgt[i][-1] == 0])))\n",
        "train_dev_src = train_dev_src[same_idx]\n",
        "train_dev_tgt = train_dev_tgt[same_idx]\n",
        "same_idx = list(set([i for i in range(test_val_src.shape[0]) if test_val_src[i][-1] == 1 or test_val_src[i][-1] == 0]).intersection(set([i for i in range(test_val_tgt.shape[0]) if test_val_tgt[i][-1] == 1 or test_val_tgt[i][-1] == 0])))\n",
        "test_val_src = test_val_src[same_idx]\n",
        "test_val_tgt = test_val_tgt[same_idx]"
      ],
      "metadata": {
        "id": "L6-cV6zpcO5e"
      },
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "train_labels = np.zeros(train_tgt.shape)\n",
        "train_labels[:,0:train_tgt.shape[1] -1] = train_tgt[:,1:]\n",
        "train_dev_labels = np.zeros(train_dev_tgt.shape)\n",
        "train_dev_labels[:,0:train_dev_tgt.shape[1] -1] = train_dev_tgt[:,1:]\n",
        "test_val_labels = np.zeros(test_val_tgt.shape)\n",
        "test_val_labels[:,0:test_val_tgt.shape[1] -1] = test_val_tgt[:,1:]"
      ],
      "metadata": {
        "id": "JGMvPvIxOx-0"
      },
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(\"*** Preprocessed Data ***\")\n",
        "print(\"Train Data Shape(Source): \", train_src.shape)\n",
        "print(\"Train Data Shape(Target): \", train_tgt.shape)\n",
        "print(\"Train Labels Shape(Target): \", train_labels.shape)\n",
        "print(\"Train Dev Data Shape(Source): \", train_dev_src.shape)\n",
        "print(\"Train Dev Data Shape(Target): \", train_dev_tgt.shape)\n",
        "print(\"Train Dev Labels Shape(Target): \", train_dev_labels.shape)\n",
        "print(\"Test Val Data Shape(Source): \", test_val_src.shape)\n",
        "print(\"Test Val Data Shape(Target): \", test_val_tgt.shape)\n",
        "print(\"Test Labels Shape(Target): \", test_val_labels.shape)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rO5HD33XOGxf",
        "outputId": "2b5f3a15-0cde-45ae-e6b2-54e023cc7e0a"
      },
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "*** Preprocessed Data ***\n",
            "Train Data Shape(Source):  (154836, 60)\n",
            "Train Data Shape(Target):  (154836, 60)\n",
            "Train Labels Shape(Target):  (154836, 60)\n",
            "Train Dev Data Shape(Source):  (3649, 60)\n",
            "Train Dev Data Shape(Target):  (3649, 60)\n",
            "Train Dev Labels Shape(Target):  (3649, 60)\n",
            "Test Val Data Shape(Source):  (1194, 60)\n",
            "Test Val Data Shape(Target):  (1194, 60)\n",
            "Test Labels Shape(Target):  (1194, 60)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Below we prepare the data in the form ready to feed to the Model."
      ],
      "metadata": {
        "id": "xtsMxbAYzePz"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "train_data_prep = tf.data.Dataset.from_tensor_slices((train_src, train_tgt, train_labels))\n",
        "train_dev_data_prep = tf.data.Dataset.from_tensor_slices((train_dev_src, train_dev_tgt, train_dev_labels))\n",
        "test_val_data_prep = tf.data.Dataset.from_tensor_slices((test_val_src, test_val_tgt, test_val_labels))"
      ],
      "metadata": {
        "id": "_OkR5tbtyBQj"
      },
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Below, we can see how our data looks"
      ],
      "metadata": {
        "id": "jSGBKDhc2sc-"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "for encoder_inputs, decoder_inputs, labels in train_data_prep.take(1):\n",
        "    print(f'encoder_inputs.shape: {encoder_inputs.shape}')\n",
        "    print(f'decoder_inputs.shape: {decoder_inputs.shape}')\n",
        "    print(f\"labels.shape: {labels.shape}\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Ao1TX6GHzWn7",
        "outputId": "dc7119aa-4eee-4749-dd56-d7c39c3ce00f"
      },
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "encoder_inputs.shape: (60,)\n",
            "decoder_inputs.shape: (60,)\n",
            "labels.shape: (60,)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Model"
      ],
      "metadata": {
        "id": "zjXk6pt-28kL"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "We will use the Transformer Architechture with the following hyperparameters."
      ],
      "metadata": {
        "id": "a9UFHuFO3dHe"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Transformer parameters\n",
        "d_model = 256 # 512 in the original paper\n",
        "d_k = 32 # 64 in the original paper\n",
        "d_v = 32 # 64 in the original paper\n",
        "n_heads = 8 # 8 in the original paper\n",
        "n_encoder_layers = 2 # 6 in the original paper\n",
        "n_decoder_layers = 2 # 6 in the original paper\n",
        "\n",
        "max_token_length = MAX_TOKENS # 512 in the original paper"
      ],
      "metadata": {
        "id": "v9ULcpgbJ9WP"
      },
      "execution_count": 15,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "class SingleHeadAttention(Layer):\n",
        "  def __init__(self, input_shape=(3, -1, d_model), dropout=.0, masked=None):\n",
        "    super(SingleHeadAttention, self).__init__()\n",
        "    self.q = Dense(d_k, input_shape=(-1, d_model), kernel_initializer='glorot_uniform', \n",
        "                   bias_initializer='glorot_uniform')\n",
        "    self.normalize_q = Lambda(lambda x: x / np.sqrt(d_k))\n",
        "    self.k = Dense(d_k, input_shape=(-1, d_model), kernel_initializer='glorot_uniform', \n",
        "                   bias_initializer='glorot_uniform')\n",
        "    self.v = Dense(d_v, input_shape=(-1, d_model), kernel_initializer='glorot_uniform', \n",
        "                   bias_initializer='glorot_uniform')\n",
        "    self.dropout = dropout\n",
        "    self.masked = masked\n",
        "  \n",
        "  # Inputs: [query, key, value]\n",
        "  def call(self, inputs, training=None):\n",
        "    assert len(inputs) == 3\n",
        "    # We use a lambda layer to divide vector q by sqrt(d_k) according to the equation\n",
        "    q = self.normalize_q(self.q(inputs[0]))\n",
        "    k = self.k(inputs[1])\n",
        "    # The dimensionality of q is (batch_size, query_length, d_k) and that of k is (batch_size, key_length, d_k)\n",
        "    # So we will do a matrix multication by batch after transposing last 2 dimensions of k\n",
        "    # tf.shape(attn_weights) = (batch_size, query_length, key_length)\n",
        "    attn_weights = tf.matmul(q, tf.transpose(k, perm=[0,2,1]))\n",
        "    if self.masked: # Prevent future attentions in decoding self-attention\n",
        "      # Create a matrix where the strict upper triangle (not including main diagonal) is filled with -inf and 0 elsewhere\n",
        "      length = tf.shape(attn_weights)[-1]\n",
        "      #attn_mask = np.triu(tf.fill((length, length), -np.inf), k=1) # We need to use tensorflow functions instead of numpy\n",
        "      attn_mask = tf.fill((length, length), -np.inf)\n",
        "      attn_mask = tf.linalg.band_part(attn_mask, 0, -1) # Get upper triangle\n",
        "      attn_mask = tf.linalg.set_diag(attn_mask, tf.zeros((length))) # Set diagonal to zeros to avoid operations with infinity\n",
        "      # This matrix is added to the attention weights so all future attention will have -inf logits (0 after softmax)\n",
        "      attn_weights += attn_mask\n",
        "    # Softmax along the last dimension\n",
        "    attn_weights = tf.nn.softmax(attn_weights, axis=-1)\n",
        "    if training: # Attention dropout included in the original paper. This is possibly to encourage multihead diversity.\n",
        "      attn_weights = tf.nn.dropout(attn_weights, rate=self.dropout)\n",
        "    v = self.v(inputs[2])\n",
        "    return tf.matmul(attn_weights, v)\n",
        "\n",
        "class MultiHeadAttention_my(Layer):\n",
        "  def __init__(self, dropout=.0, masked=None):\n",
        "    super(MultiHeadAttention_my, self).__init__()\n",
        "    self.attn_heads = list()\n",
        "    for i in range(n_heads): \n",
        "      self.attn_heads.append(SingleHeadAttention(dropout=dropout, masked=masked))\n",
        "    self.linear = Dense(d_model, input_shape=(-1, n_heads * d_v), kernel_initializer='glorot_uniform', \n",
        "                   bias_initializer='glorot_uniform')\n",
        "    \n",
        "  def call(self, x, training=None):\n",
        "    attentions = [self.attn_heads[i](x, training=training) for i in range(n_heads)]\n",
        "    concatenated_attentions = tf.concat(attentions, axis=-1)\n",
        "    return self.linear(concatenated_attentions)\n",
        "\n",
        "class TransformerEncoder(Layer):\n",
        "  def __init__(self, dropout=.1, attention_dropout=.0, **kwargs):\n",
        "    super(TransformerEncoder, self).__init__(**kwargs)\n",
        "    self.dropout_rate = dropout\n",
        "    self.attention_dropout_rate = attention_dropout\n",
        "  def build(self, input_shape):\n",
        "    self.multihead_attention = MultiHeadAttention_my(dropout=self.attention_dropout_rate)\n",
        "    self.dropout1 = tf.keras.layers.Dropout(self.dropout_rate)\n",
        "    self.layer_normalization1 = LayerNormalization(input_shape=input_shape, epsilon=1e-6)\n",
        "    \n",
        "    self.linear1 = Dense(input_shape[-1] * 4, input_shape=input_shape, activation='relu',\n",
        "                        kernel_initializer='glorot_uniform', bias_initializer='glorot_uniform')\n",
        "    self.linear2 = Dense(input_shape[-1], input_shape=self.linear1.compute_output_shape(input_shape),\n",
        "                        kernel_initializer='glorot_uniform', bias_initializer='glorot_uniform')\n",
        "    self.dropout2 = tf.keras.layers.Dropout(self.dropout_rate)\n",
        "    self.layer_normalization2 = LayerNormalization(input_shape=input_shape, epsilon=1e-6)\n",
        "    super(TransformerEncoder, self).build(input_shape)\n",
        "  def call(self, x, training=None):\n",
        "    sublayer1 = self.multihead_attention((x, x, x), training=training)\n",
        "    sublayer1 = self.dropout1(sublayer1, training=training)\n",
        "    layernorm1 = self.layer_normalization1(x + sublayer1)\n",
        "    \n",
        "    sublayer2 = self.linear2(self.linear1(layernorm1))\n",
        "    sublayer1 = self.dropout2(sublayer2, training=training)\n",
        "    layernorm2 = self.layer_normalization2(layernorm1 + sublayer2)\n",
        "    return layernorm2\n",
        "  def compute_output_shape(self, input_shape):\n",
        "    return input_shape\n",
        "\n",
        "class TransformerDecoder(Layer):\n",
        "  def __init__(self, dropout=.0, attention_dropout=.0, **kwargs):\n",
        "    super(TransformerDecoder, self).__init__(**kwargs)\n",
        "    self.dropout_rate = dropout\n",
        "    self.attention_dropout_rate = attention_dropout\n",
        "  def build(self, input_shape):\n",
        "    self.multihead_self_attention = MultiHeadAttention_my(dropout=self.attention_dropout_rate, masked=True)\n",
        "    self.dropout1 = tf.keras.layers.Dropout(self.dropout_rate)\n",
        "    self.layer_normalization1 = LayerNormalization(input_shape=input_shape, epsilon=1e-6)\n",
        "    \n",
        "    self.multihead_encoder_attention = MultiHeadAttention_my(dropout=self.attention_dropout_rate)\n",
        "    self.dropout2 = tf.keras.layers.Dropout(self.dropout_rate)\n",
        "    self.layer_normalization2 = LayerNormalization(input_shape=input_shape, epsilon=1e-6)\n",
        "    \n",
        "    self.linear1 = Dense(input_shape[-1] * 4, input_shape=input_shape, activation='relu',\n",
        "                        kernel_initializer='glorot_uniform', bias_initializer='glorot_uniform')\n",
        "    self.linear2 = Dense(input_shape[-1], input_shape=self.linear1.compute_output_shape(input_shape),\n",
        "                        kernel_initializer='glorot_uniform', bias_initializer='glorot_uniform')\n",
        "    self.dropout3 = tf.keras.layers.Dropout(self.dropout_rate)\n",
        "    self.layer_normalization3 = LayerNormalization(input_shape=input_shape, epsilon=1e-6)\n",
        "    super(TransformerDecoder, self).build(input_shape)\n",
        "  def call(self, x, hidden, training=None):\n",
        "    sublayer1 = self.multihead_self_attention((x, x, x))\n",
        "    sublayer1 = self.dropout1(sublayer1, training=training)\n",
        "    layernorm1 = self.layer_normalization1(x + sublayer1)\n",
        "    \n",
        "    sublayer2 = self.multihead_encoder_attention((x, hidden, hidden))\n",
        "    sublayer2 = self.dropout2(sublayer2, training=training)\n",
        "    layernorm2 = self.layer_normalization2(layernorm1 + sublayer2)\n",
        "    \n",
        "    sublayer3 = self.linear2(self.linear1(layernorm1))\n",
        "    sublayer3 = self.dropout3(sublayer3, training=training)\n",
        "    layernorm3 = self.layer_normalization2(layernorm2 + sublayer3)\n",
        "    return layernorm3\n",
        "  def compute_output_shape(self, input_shape):\n",
        "    return input_shape\n",
        "\n",
        "class SinusoidalPositionalEncoding(Layer): # This is a TensorFlow implementation of https://github.com/graykode/nlp-tutorial/blob/master/5-1.Transformer/Transformer_Torch.ipynb\n",
        "  def __init__(self):\n",
        "    super(SinusoidalPositionalEncoding, self).__init__()\n",
        "    self.sinusoidal_encoding = np.array([self.get_positional_angle(pos) for pos in range(max_token_length)], dtype=np.float32)\n",
        "    self.sinusoidal_encoding[:, 0::2] = np.sin(self.sinusoidal_encoding[:, 0::2])\n",
        "    self.sinusoidal_encoding[:, 1::2] = np.cos(self.sinusoidal_encoding[:, 1::2])\n",
        "    self.sinusoidal_encoding = tf.cast(self.sinusoidal_encoding, dtype=tf.float32) # Casting the array to Tensor for slicing\n",
        "  def call(self, x):\n",
        "    return x + self.sinusoidal_encoding[:tf.shape(x)[1]]\n",
        "  def compute_output_shape(self, input_shape):\n",
        "    return input_shape\n",
        "  def get_angle(self, pos, dim):\n",
        "    return pos / np.power(10000, 2 * (dim // 2) / d_model)\n",
        "  def get_positional_angle(self, pos):\n",
        "    return [self.get_angle(pos, dim) for dim in range(d_model)]\n",
        "\n",
        "class Transformer(Model):\n",
        "  def __init__(self, dropout=.1, attention_dropout=.0, **kwargs):\n",
        "    super(Transformer, self).__init__(**kwargs)\n",
        "    self.encoding_embedding = Embedding(VOCAB_SIZE, d_model)\n",
        "    self.decoding_embedding = Embedding(VOCAB_SIZE, d_model)\n",
        "    self.pos_encoding = SinusoidalPositionalEncoding()\n",
        "    self.encoder = [TransformerEncoder(dropout=dropout, attention_dropout=attention_dropout) for i in range(n_encoder_layers)]\n",
        "    self.decoder = [TransformerDecoder(dropout=dropout, attention_dropout=attention_dropout) for i in range(n_decoder_layers)]\n",
        "    self.decoder_final = Dense(VOCAB_SIZE, input_shape=(None, d_model))\n",
        "  def call(self, inputs, training=None): # Source_sentence and decoder_input\n",
        "    source_sentence, decoder_input = inputs\n",
        "    embedded_source = self.encoding_embedding(source_sentence)\n",
        "    encoder_output = self.pos_encoding(embedded_source)\n",
        "    for encoder_unit in self.encoder:\n",
        "      encoder_output = encoder_unit(encoder_output, training=training)\n",
        "    \n",
        "    embedded_target = self.decoding_embedding(decoder_input)\n",
        "    decoder_output = self.pos_encoding(embedded_target)\n",
        "    for decoder_unit in self.decoder:\n",
        "      decoder_output = decoder_unit(decoder_output, encoder_output, training=training)\n",
        "    if training:\n",
        "      decoder_output = self.decoder_final(decoder_output)\n",
        "      decoder_output = tf.nn.softmax(decoder_output, axis=-1)\n",
        "    else:\n",
        "      decoder_output = self.decoder_final(decoder_output[:, -1:, :])\n",
        "      decoder_output = tf.nn.softmax(decoder_output, axis=-1)\n",
        "    return decoder_output"
      ],
      "metadata": {
        "id": "KvsIOzPnzt4F"
      },
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "transformer = Transformer(dropout=0.3) # Instantiating a new transformer model\n",
        "train = [tf.cast(train_src, dtype=tf.float32), tf.cast(train_tgt, dtype=tf.float32)] # Cast the tuples to tensors\n",
        "validation = [tf.cast(train_dev_src, dtype=tf.float32), tf.cast(train_dev_tgt, dtype=tf.float32)]"
      ],
      "metadata": {
        "id": "eRsGRYTR6XQW"
      },
      "execution_count": 17,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "transformer.build([train_src.shape, train_tgt.shape])\n",
        "transformer.summary()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "IHXUpv-0eh-b",
        "outputId": "5bf36787-aee0-4ee7-cd7f-d170d64fce7f"
      },
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"transformer\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " embedding (Embedding)       multiple                  8192000   \n",
            "                                                                 \n",
            " embedding_1 (Embedding)     multiple                  8192000   \n",
            "                                                                 \n",
            " sinusoidal_positional_encod  multiple                 0         \n",
            " ing (SinusoidalPositionalEn                                     \n",
            " coding)                                                         \n",
            "                                                                 \n",
            " transformer_encoder (Transf  multiple                 789760    \n",
            " ormerEncoder)                                                   \n",
            "                                                                 \n",
            " transformer_encoder_1 (Tran  multiple                 789760    \n",
            " sformerEncoder)                                                 \n",
            "                                                                 \n",
            " transformer_decoder (Transf  multiple                 1052928   \n",
            " ormerDecoder)                                                   \n",
            "                                                                 \n",
            " transformer_decoder_1 (Tran  multiple                 1052928   \n",
            " sformerDecoder)                                                 \n",
            "                                                                 \n",
            " dense (Dense)               multiple                  8224000   \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 28,293,376\n",
            "Trainable params: 28,293,376\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Training"
      ],
      "metadata": {
        "id": "0JkChx5Ndp1s"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "transformer.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])"
      ],
      "metadata": {
        "id": "InXkWf-LawJi"
      },
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "EPOCHS = 10\n",
        "checkpoint_filepath = '/Models/checkpoint'\n",
        "model_checkpoint_callback = tf.keras.callbacks.ModelCheckpoint(\n",
        "    filepath=checkpoint_filepath,\n",
        "    save_weights_only=True,\n",
        "    monitor = \"accuracy\",\n",
        "    mode='max',\n",
        "    save_best_only=True)"
      ],
      "metadata": {
        "id": "LMttctIwaNxG"
      },
      "execution_count": 20,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "with tf.device('/device:GPU:0'):\n",
        "    history = transformer.fit(train, tf.cast(train_labels, dtype=tf.float32), batch_size=256, epochs=EPOCHS, callbacks=[model_checkpoint_callback])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "bzdsT8kmBEMo",
        "outputId": "8991a0c3-413e-4223-b846-e57124e22770"
      },
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/10\n",
            "605/605 [==============================] - 391s 609ms/step - loss: 1.8844 - accuracy: 0.7964\n",
            "Epoch 2/10\n",
            "605/605 [==============================] - 377s 624ms/step - loss: 1.5602 - accuracy: 0.8053\n",
            "Epoch 3/10\n",
            "605/605 [==============================] - 378s 624ms/step - loss: 1.4638 - accuracy: 0.8100\n",
            "Epoch 4/10\n",
            "605/605 [==============================] - 378s 625ms/step - loss: 1.3653 - accuracy: 0.8156\n",
            "Epoch 5/10\n",
            "605/605 [==============================] - 378s 626ms/step - loss: 1.2814 - accuracy: 0.8202\n",
            "Epoch 6/10\n",
            "605/605 [==============================] - 379s 626ms/step - loss: 1.2062 - accuracy: 0.8240\n",
            "Epoch 7/10\n",
            "605/605 [==============================] - 379s 626ms/step - loss: 1.1419 - accuracy: 0.8276\n",
            "Epoch 8/10\n",
            "605/605 [==============================] - 378s 625ms/step - loss: 1.0882 - accuracy: 0.8306\n",
            "Epoch 9/10\n",
            "605/605 [==============================] - 378s 625ms/step - loss: 1.0426 - accuracy: 0.8336\n",
            "Epoch 10/10\n",
            "605/605 [==============================] - 378s 625ms/step - loss: 1.0022 - accuracy: 0.8364\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "transformer.save_weights(\"model_weights\")\n",
        "# transformer.save(\"./Models/\")"
      ],
      "metadata": {
        "id": "6tmeafiFdZ_U"
      },
      "execution_count": 22,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "plt.figure(figsize=(10, 10))\n",
        "plt.plot(history.history['accuracy'])\n",
        "plt.title('model accuracy')\n",
        "plt.ylabel('accuracy')\n",
        "plt.xlabel('epoch')\n",
        "plt.legend(['train'], loc='upper left')\n",
        "plt.show()"
      ],
      "metadata": {
        "id": "JbSCoj0cft6O",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 621
        },
        "outputId": "812ab2d4-9ad4-4a34-8994-b1f5b501f880"
      },
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 720x720 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAm4AAAJcCAYAAABAGii1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd3yV9f3+8debEAh7g+yA7CUjDEets6IUZx0MEWWpVavVVmvVWq21tbXfugeKyBYHCg4cdZchCXsvgYS9CSMh4/37Iwd/kYIEyMl9xvV8PPLgnHud6wYSLu7xuc3dEREREZHIVyroACIiIiJSNCpuIiIiIlFCxU1EREQkSqi4iYiIiEQJFTcRERGRKKHiJiIiIhIlVNxEJGaZ2Ugz+0sRl11jZheEO5OIyMlQcRMRERGJEipuIiIRzsxKB51BRCKDipuIBCp0ivJ3ZjbfzPaZ2atmVsfMPjKzTDP7zMyqFVr+UjNbZGa7zOxLM2tdaF4nM5sdWu8NIOmwz/qlmc0NrTvNzDoUMWMvM5tjZnvMLN3MHj5s/lmh7e0KzR8Yml7OzJ40s7VmttvMvg1NO8fMMo7w+3BB6PXDZvaWmY0xsz3AQDPrZmbTQ5+x0cyeNbMyhdZva2afmtkOM9tsZveb2Slmtt/MahRarrOZbTWzxKLsu4hEFhU3EYkEVwEXAi2A3sBHwP1ALQp+Tt0BYGYtgPHAnaF5HwJTzKxMqMS8C4wGqgNvhrZLaN1OwAhgGFADeAmYbGZli5BvHzAAqAr0Am4xs8tD220cyvtMKFNHYG5ovX8CXYAzQpl+D+QX8ffkMuCt0GeOBfKAu4CawOnA+cCtoQyVgM+AqUA9oBnwH3ffBHwJXFNou9cDE9w9p4g5RCSCqLiJSCR4xt03u/t64BtgprvPcfcsYBLQKbTctcAH7v5pqHj8EyhHQTHqASQC/3b3HHd/C5hV6DOGAi+5+0x3z3P314Hs0Ho/yd2/dPcF7p7v7vMpKI8/D83uC3zm7uNDn7vd3eeaWSngJuA37r4+9JnT3D27iL8n09393dBnHnD3NHef4e657r6GguJ5KMMvgU3u/qS7Z7l7prvPDM17HegPYGYJQB8Kyq2IRCEVNxGJBJsLvT5whPcVQ6/rAWsPzXD3fCAdqB+at97dvdC6awu9bgzcHTrVuMvMdgENQ+v9JDPrbmZfhE4x7gZupuDIF6FtrDrCajUpOFV7pHlFkX5YhhZm9r6ZbQqdPv1rETIAvAe0MbMmFBzV3O3u351gJhEJmIqbiESTDRQUMADMzCgoLeuBjUD90LRDGhV6nQ485u5VC32Vd/fxRfjcccBkoKG7VwFeBA59Tjpw6hHW2QZkHWXePqB8of1IoOA0a2F+2PsXgKVAc3evTMGp5MIZmh4peOio5UQKjrpdj462iUQ1FTcRiSYTgV5mdn7o4vq7KTjdOQ2YDuQCd5hZopldCXQrtO5w4ObQ0TMzswqhmw4qFeFzKwE73D3LzLpRcHr0kLHABWZ2jZmVNrMaZtYxdDRwBPAvM6tnZglmdnromrrlQFLo8xOBB4BjXWtXCdgD7DWzVsAthea9D9Q1szvNrKyZVTKz7oXmjwIGApei4iYS1VTcRCRquPsyCo4cPUPBEa3eQG93P+juB4ErKSgoOyi4Hu6dQuumAkOAZ4GdwMrQskVxK/CImWUCD1FQIA9tdx1wCQUlcgcFNyacFpp9D7CAgmvtdgB/B0q5++7QNl+h4GjhPuBHd5kewT0UFMZMCkroG4UyZFJwGrQ3sAlYAZxbaP5/KbgpYra7Fz59LCJRxn58OYiIiMQiM/scGOfurwSdRUROnIqbiEiMM7OuwKcUXKOXGXQeETlxOlUqIhLDzOx1CsZ4u1OlTST66YibiIiISJTQETcRERGRKBEXDy6uWbOmJycnBx1DRERE5JjS0tK2ufvhYzsCcVLckpOTSU1NDTqGiIiIyDGZ2VGH7dGpUhEREZEooeImIiIiEiVU3ERERESiRFxc43YkOTk5ZGRkkJWVFXSUsEpKSqJBgwYkJiYGHUVEREROUtwWt4yMDCpVqkRycjJmFnScsHB3tm/fTkZGBk2aNAk6joiIiJykuD1VmpWVRY0aNWK2tAGYGTVq1Ij5o4oiIiLxIm6LGxDTpe2QeNhHERGReBHXxU1EREQkmqi4BWTXrl08//zzx73eJZdcwq5du8KQSERERCKdiltAjlbccnNzf3K9Dz/8kKpVq4YrloiIiESwuL2rNGj33Xcfq1atomPHjiQmJpKUlES1atVYunQpy5cv5/LLLyc9PZ2srCx+85vfMHToUOD/P75r7969XHzxxZx11llMmzaN+vXr895771GuXLmA90xERETCRcUN+POURSzesKdYt9mmXmX+1LvtUef/7W9/Y+HChcydO5cvv/ySXr16sXDhwh+G7RgxYgTVq1fnwIEDdO3alauuuooaNWr8aBsrVqxg/PjxDB8+nGuuuYa3336b/v37F+t+iIiISORQcYsQ3bp1+9FYa08//TSTJk0CID09nRUrVvxPcWvSpAkdO3YEoEuXLqxZs6bE8oqIiEjJU3GDnzwyVlIqVKjww+svv/ySzz77jOnTp1O+fHnOOeecI47FVrZs2R9eJyQkcODAgRLJKiIiIsHQzQkBqVSpEpmZmUect3v3bqpVq0b58uVZunQpM2bMKOF0IiIiEol0xC0gNWrU4Mwzz6Rdu3aUK1eOOnXq/DCvZ8+evPjii7Ru3ZqWLVvSo0ePAJOKiIhIpDB3DzpD2KWkpHhqauqPpi1ZsoTWrVsHlKhkxdO+ioiIRDszS3P3lCPN06lSERERkSih4iYiIiISJeK6uMXDaeJ42EcREZF4EbfFLSkpie3bt8d0sXF3tm/fTlJSUtBRREREpBjE7V2lDRo0ICMjg61btwYdJaySkpJo0KBB0DFERESiXk5ePokJwR7zitvilpiY+KMnFYiIiIgcSX6+82ZaOk99toI3hp1Ow+rlA8sSt8VNRERE5FiWbtrDA5MWkrp2J12Tq5GbH+wlVipuIiIiIofZl53LU/9Zwavffk/lpNI88asO/KpzA0qVskBzqbiJiIiIhLg7Hy/azCNTFrFhdxbXdW3IvT1bUa1CmaCjASpuIiIiIgCk79jPnyYv4vOlW2h1SiWe6duJLo2rBx3rR1TcREREJK4dzM1n+DereebzFZQy44FerRl4RjKlA76D9EhU3ERERCRuTV+1nQfeXcCqrfu4uN0pPNS7DXWrlAs61lGpuImIiEjc2bY3m79+sIR35qynYfVyvDawK+e2qh10rGNScRMREZG4kZ/vjPtuHU9MXcqBnDxuO7cZvz63GeXKJAQdrUhU3ERERCQuLFy/mz++u5B56bs4vWkNHr28Hc1qVww61nFRcRMREZGYlpmVw5OfLGfU9DVUr1CGf1/bkcs61sMs2DHZToSKm4iIiMQkd+eDBRt5ZMpitu7Npn/3xtxzUUuqlEsMOtoJU3ETERGRmLNm2z4efG8h36zYRrv6lRk+IIXTGlYNOtZJU3ETERGRmJGVk8eLX63i+S9XUTahFA/3bsP1pyeTEPCjqoqLipuIiIjEhG9WbOXBdxeyZvt+ep9Wjwd7taZ25aSgYxWrsBY3M+sJPAUkAK+4+98Om98IeB2oGlrmPnf/0My6AS8fWgx42N0nhdZZA2QCeUCuu6eEcx9EREQksm3ek8Wj7y/m/fkbaVKzAqMHdeNnzWsFHSsswlbczCwBeA64EMgAZpnZZHdfXGixB4CJ7v6CmbUBPgSSgYVAirvnmlldYJ6ZTXH33NB657r7tnBlFxERkciXl++Mmr6GJz9ZzsG8fO66oAXDft6UpMToGJPtRITziFs3YKW7rwYwswnAZUDh4uZA5dDrKsAGAHffX2iZpNByIiIiIgDMS9/FH99dwML1e/hZ85o8elk7kmtWCDpW2IWzuNUH0gu9zwC6H7bMw8AnZnY7UAG44NAMM+sOjAAaA9cXOtrmoXUceMndX+YIzGwoMBSgUaNGJ70zIiIiErzdB3L4x8dLGTtzHbUqluXZvp3o1b5uVI7JdiKCvjmhDzDS3Z80s9OB0WbWzt3z3X0m0NbMWgOvm9lH7p4FnOXu682sNvCpmS11968P33Co0L0MkJKSoiN2IiIiUczdeXfueh77YAk79h1k4BnJ/PbCFlRKit4x2U5EOIvbeqBhofcNQtMKGwT0BHD36WaWBNQEthxawN2XmNleoB2Q6u7rQ9O3mNkkCk7J/k9xExERkdiwcsteHnx3IdNXb6djw6qMvLEb7epXCTpWIMJZ3GYBzc2sCQWF7Tqg72HLrAPOB0aGjqwlAVtD66SHbk5oDLQC1phZBaCUu2eGXv8CeCSM+yAiIiIBycrJ49nPV/LS16sol5jAY1e0o0/XRpSKkTHZTkTYiluodN0GfEzBUB8j3H2RmT1CwZGzycDdwHAzu4uCa9cGurub2VnAfWaWA+QDt7r7NjNrCkwKnccuDYxz96nh2gcREREJxhdLt/DQ5IWk7zjAlZ3rc/8lralZsWzQsQJn7rF/+VdKSoqnpqYGHUNERESOYcOuAzwyZTFTF22iWe2KPHpZO04/tUbQsUqUmaUdbZzaoG9OEBERESEnL5+R/13D/322nHx3fndRS4b8rCllSpcKOlpEUXETERGRQKWt3cEfJy1k6aZMzm9Vm4cvbUvD6uWDjhWRVNxEREQkEDv3HeRvHy3ljdR06lZJ4qXru/CLNnXiZky2E6HiJiIiIiUqP995a3YGj3+4hMysXIad3ZQ7zm9OhbKqJcei3yEREREpMcs2ZfLAuwuYtWYnXZOr8ZfL29PylEpBx4oaKm4iIiISdvsP5vLUZyt49dvvqZRUmieu6sCvujSI6zHZToSKm4iIiITVJ4s28fDkRWzYncW1KQ257+JWVKtQJuhYUUnFTURERMIifcd+/jxlEZ8t2UKrUyrxdJ9OpCRXDzpWVFNxExERkWJ1MDef4d+s5pnPV1DKjD9e0pqBZyaTmKAx2U6WipuIiIgUmxmrt/PAuwtZuWUvF7Wtw596t6Ve1XJBx4oZKm4iIiJy0rbvzeaxD5fwzuz1NKhWjhEDUzivVZ2gY8UcFTcRERE5Ye7OO7PX8+gHi9mXncuvzz2V285tTrkyCUFHi0kqbiIiInJC0nfs5/5JC/hmxTa6NK7G365sT/M6GpMtnFTcRERE5Ljk5Tsjp63hnx8vo5TBI5e1pX/3xhqTrQSouImIiEiRLd20h3vfXsC89F2c27IWf7miPfV180GJUXETERGRY8rOzeO5z1fy/JerqFwukaeu68ilp9XTA+FLmIqbiIiI/KTUNTu49+35rNq6jys71eeBX7ahup58EAgVNxERETmivdm5PDF1KaNnrKVelXKMvLEr57SsHXSsuKbiJiIiIv/j86Wb+eOkhWzak8XAM5K55xctqVBWtSFo+hMQERGRH2zbm82fpyxmyrwNtKhTkef6nUHnRtWCjiUhKm4iIiKCuzNpznoeeb9gIN27LmjBLeecSpnSer5oJFFxExERiXPpO/bzx3cX8vXyrRpIN8KpuImIiMSpvHzn9Wlr+OcnyzA0kG40UHETERGJQ8s2ZXLv2/OZq4F0o4qKm4iISBw5NJDuC1+tolKSBtKNNipuIiIicaLwQLpXdKrPgxpIN+qouImIiMQ4DaQbO1TcREREYljhgXRvOD2Z312kgXSjmf7kREREYtD20EC6kzWQbkxRcRMREYkhhwbSffT9xezVQLoxR8VNREQkRmTs3M/9kwoG0u3cqCp/v6qDBtKNMSpuIiIiUe7wgXT/fGlbru+hgXRjkYqbiIhIFCs8kO45LWvxmAbSjWkqbiIiIlEoOzeP575YxQtfrtRAunFExU1ERCTKpK3dwb1vL2Dllr0aSDfOqLiJiIhEib3Zufxj6lJGaSDduKXiJiIiEgW+WLqFP05awEYNpBvX9CcuIiISwbbvzeaR9xfz3tyCgXTf1kC6cU3FTUREJAK5O+/OXc8jUwoG0r3zgubcek4zDaQb51TcREREIowG0pWjUXETERGJEIUH0gV4uHcbrj89mQQNpCshKm4iIiIRYPnmTH7/lgbSlZ+m4iYiIhIgDaQrx0PFTUREJCAaSFeOl4qbiIhICTt8IN3XbuzKuRpIV4pAxU1ERKQEHT6Q7j0XtaSiBtKVItLfFBERkRJQeCDd5rUr8tbNZ9ClsQbSleOj4iYiIhJmH8zfyIPvLSQzK4c7L2jOLeecStnSCUHHkiik4iYiIhImO/cd5KHJi5gybwOnNajCP67uQQsNpCsnQcVNREQkDD5bvJk/TFrArv0HuecXLbj556dSOkGPq5KTo+ImIiJSjHYfyOGRKYt5e3YGrU6pxOs3dqNNvcpBx5IYoeImIiJSTL5evpV7357Plsxsbj+vGbef11wPhZdipeImIiJykvZm5/LXD5cwbuY6mtWuyDv9u3Baw6pBx5IYFNb/BphZTzNbZmYrzey+I8xvZGZfmNkcM5tvZpeEpnczs7mhr3lmdkVRtykiIlKSZqzezsVPfc3479Yx9OymvH/7WSptEjZhO+JmZgnAc8CFQAYwy8wmu/viQos9AEx09xfMrA3wIZAMLARS3D3XzOoC88xsCuBF2KaIiEjYHTiYxxMfL+W1/64huUZ53hx2OinJ1YOOJTEunKdKuwEr3X01gJlNAC4DCpcsBw5dsVkF2ADg7vsLLZMUWq6o2xQREQmr2et2cs/Eeazeto8bTm/MvRe3onwZXX0k4RfOv2X1gfRC7zOA7oct8zDwiZndDlQALjg0w8y6AyOAxsD1oaNvRdnmofWHAkMBGjVqdFI7IiIiApCdm8f/fbqCl79eRd0q5Rg3uDtnNKsZdCyJI0Hf6tIHGOnuDYBLgNFmVgrA3We6e1ugK/AHM0s6ng27+8vunuLuKbVq1Sr24CIiEl8WZOym9zPf8uJXq7gmpSFT7/yZSpuUuHAecVsPNCz0vkFoWmGDgJ4A7j49VM5qAlsOLeDuS8xsL9CuiNsUEREpNjl5+Tz7+Uqe/WIlNSuW4bUbu3Juy9pBx5I4Fc7iNgtobmZNKChX1wF9D1tmHXA+MNLMWlNwPdvW0DrpodOjjYFWwBpgVxG2KSIiUiyWbtrD3RPnsWjDHq7oVJ+He7elSvnEoGNJHAtbcQuVrtuAj4EEYIS7LzKzR4BUd58M3A0MN7O7KLgBYaC7u5mdBdxnZjlAPnCru28DONI2w7UPIiISn3Lz8nn5m9X8+9MVVC5Xmpeu78JFbU8JOpYI5u7HXirKpaSkeGpqatAxREQkCqzaupe7J85jbvouLml/Co9e1o4aFcsGHUviiJmluXvKkebp3mUREREgP995bdoanpi6lHJlEni6Tyd6d6iLmQUdTeQHKm4iIhL31m3fzz1vzeO773dwfqvaPH5le2pXPq7BDERKhIqbiIjELXdn7Mx1/PXDJSSY8Y9fdeBXXRroKJtELBU3ERGJSxt2HeDet+fzzYpt/Kx5Tf5+VQfqVS0XdCyRn6TiJiIiccXdeSstg0emLCbPnceuaEffbo10lE2igoqbiIjEjS17svjDOwv4z9ItdGtSnX/+6jQa1SgfdCyRIlNxExGRmOfuTJm/kYfeW8iBg3k8+Ms23HhGMqVK6SibRBcVNxERiWnb92bz4HsL+XDBJjo1qso/rz6NU2tVDDqWyAlRcRMRkZg1deEm/jhpAZlZufy+Z0uG/qwppRNKBR1L5ISpuImISMzZvT+Hh6csYtKc9bStV5lxQzrS8pRKQccSOWkqbiIiElO+WLaF+96ez/a9B7nzgub8+txmJOoom8QIFTcREYkJmVk5PPbBEibMSqdFnYq8ekNX2tWvEnQskWKl4iYiIlFv2spt/O6t+WzcfYBbzjmVOy9oTtnSCUHHEil2Km4iIhK19h/M5e8fLeX16WtpWrMCb91yBp0bVQs6lkjYqLiJiEhUSl2zg3venMea7fu56cwm/O6ilpQro6NsEttU3EREJKpk5eTx5CfLeOXb72lQrRwThvagR9MaQccSKREqbiIiEjXmpe/i7jfnsXLLXvp1b8T9l7SmQln9UybxQ3/bRUQk4h3Mzefp/6zgha9WUbtSWUbd1I2zW9QKOpZIiVNxExGRiLZ4wx7ufnMeSzbu4VddGvDgL9tQpVxi0LFEAqHiJiIiESk3L58Xv1rFU/9ZQZVyZXhlQAoXtKkTdCyRQKm4iYhIxFm5JZO7J85jXsZuep9Wj0cubUu1CmWCjiUSOBU3ERGJGHn5zqvfruafnyynYtnSPNe3M7061A06lkjEUHETEZGIsGbbPu55cx6pa3fyizZ1eOyK9tSqVDboWCIRRcVNREQC92ZqOg+9t4jEBOP/rj2NyzvWx8yCjiUScVTcREQkMNm5efx5ymLGzVzHGafW4F/XdOSUKklBxxKJWCpuIiISiI27D3DLmNnMTd/FzT8/lXt+0YLSCaWCjiUS0VTcRESkxE1buY3bx88hOzefF/t3pmc73YAgUhQqbiIiUmLcnZe/Xs3fpy6laa2KvNi/C81qVww6lkjUUHETEZESkZmVw+/enM/URZvo1b4uT/yqg54zKnKc9B0jIiJht3JLJsNGp7Fm+34e6NWaQWc10V2jIidAxU1ERMLqg/kb+f1b8yhXJoGxg7vTo2mNoCOJRC0VNxERCYvcvHye+HgZL3+9ms6NqvJ8vy4a6kPkJKm4iYhIsdu2N5vbxs1mxuodDDi9MQ/0akOZ0hrqQ+RkqbiJiEixmr1uJ7eOmc3O/Qd58urTuKpLg6AjicQMFTcRESkW7s6Ymet4ZMoi6lYpxzu3nkHbelWCjiUSU1TcRETkpGXl5PHHSQt5e3YG57asxb+v7USV8olBxxKJOSpuIiJyUtZt38/NY9JYsmkPd17QnDvOa06pUhrqQyQcVNxEROSEfbFsC3dOmIu7M+KGrpzbqnbQkURimoqbiIgct/x85+nPV/DUf1bQ6pTKvNS/C41qlA86lkjMU3ETEZHjsnt/DndNnMvnS7dwZaf6PHZFe8qVSQg6lkhcUHETEZEiW7xhDzePSWPj7gM8enk7+ndvpEdXiZQgFTcRESmSSXMy+MM7C6hSLpEJQ0+nS+NqQUcSiTsqbiIi8pMO5ubzlw8WM2r6Wno0rc4zfTpTq1LZoGOJxCUVNxEROapNu7O4dWwas9ftYujZTfn9RS0pnaBHV4kERcVNRESOaMbq7dw2bjYHDubxfL/OXNK+btCRROKeipuIiPyIu/Pqt9/z+EdLaVyjPBOG9qBZ7UpBxxIRVNxERKSQfdm5/P7t+XwwfyM9257CP67uQKUkPbpKJFKouImICACrtu5l2Og0Vm/dy30Xt2LY2U011IdIhFFxExERpi7cxD1vzqNM6VKMGdSdM5rVDDqSiByBipuISBzLzcvnyU+X88KXqzitYVVe6NeZelXLBR1LRI5CxU1EJE5t35vNHRPm8N+V2+nbvRF/6t2GsqX16CqRSKbiJiISh+am7+LWMWls23eQJ37VgWtSGgYdSUSKIKyjKJpZTzNbZmYrzey+I8xvZGZfmNkcM5tvZpeEpl9oZmlmtiD063mF1vkytM25oa/a4dwHEZFY4u6Mm7mOa16cjpnxzi1nqLSJRJGwHXEzswTgOeBCIAOYZWaT3X1xocUeACa6+wtm1gb4EEgGtgG93X2DmbUDPgbqF1qvn7unhiu7iEgsysrJ46H3FjIxNYOzW9TiqWs7Uq1CmaBjichxCOep0m7ASndfDWBmE4DLgMLFzYHKoddVgA0A7j6n0DKLgHJmVtbds8OYV0QkZqXv2M8tY9NYuH4Pd5zXjN9c0IKEUhrqQyTahLO41QfSC73PALoftszDwCdmdjtQAbjgCNu5Cph9WGl7zczygLeBv7i7H76SmQ0FhgI0atToRPdBRCTqfb18K3dMmENevvPKgBQuaFMn6EgicoKCflJwH2CkuzcALgFGm9kPmcysLfB3YFihdfq5e3vgZ6Gv64+0YXd/2d1T3D2lVq1aYdsBEZFIlZ/vPPv5Cm547TtOqZzElNvOUmkTiXLhPOK2Hih8xWuD0LTCBgE9Adx9upklATWBLWbWAJgEDHD3VYdWcPf1oV8zzWwcBadkR4VtL0REotDuAzncPXEeny3ZzGUd6/H4le0pX0YDCYhEu3AecZsFNDezJmZWBrgOmHzYMuuA8wHMrDWQBGw1s6rAB8B97v7fQwubWWkzqxl6nQj8ElgYxn0QEYk6Szft4bJnv+XLZVt4uHcb/n1tR5U2kRgRtu9kd881s9souCM0ARjh7ovM7BEg1d0nA3cDw83sLgpuVBjo7h5arxnwkJk9FNrkL4B9wMeh0pYAfAYMD9c+iIhEm/fmrue+txdQMak044f2oGty9aAjiUgxsiNc1x9zUlJSPDVVo4eISOzKycvnsQ+WMHLaGrolV+fZvp2oXTkp6FgicgLMLM3dU440T8fORUSi3JY9Wdw6djapa3dy05lN+MMlrUhMCPreMxEJBxU3EZEoNmvNDm4dO5u9Wbk83acTl55WL+hIIhJGKm4iIlHI3Rk5bQ2PfbCEhtXLM2ZQd1qeUinoWCISZipuIiJRZv/BXP7wzgLem7uBC9vU4clrTqNyUmLQsUSkBKi4iYhEke+37ePm0Wks35LJ7y5qyS0/P5VSenSVSNxQcRMRiRKfLt7Mb9+YS+kE4/Ubu3F2Cz0VRiTeqLiJiES4/Hzn358t5+nPV9K+fhVe6N+ZBtXKBx1LRAKg4iYiEsH2Zedy1xtz+WTxZq7u0oBHL29HUmJC0LFEJCAqbiIiESp9x36GjEpl+eZM/tS7DQPPSMZM17OJxDMVNxGRCPTd9zu4eUwaOXn5jNT1bCISouImIhJhJny3jgffW0jDauUZfkMKp9aqGHQkEYkQKm4iIhEiNy+fxz5cwmv/XcPPmtfk2T6dqVJe47OJyP+n4iYiEgF278/htvGz+WbFNm46swn3X9KK0nreqIgcRsVNRCRgq7buZcjrqaTv3M8TV3Xgmq4Ng44kIhFKxU1EJEBfL9/Kr8fNpkxCKcYN6UHX5C1N7yQAACAASURBVOpBRxKRCKbiJiISAHdnxH/X8NgHi2lRpxKv3JCiQXVF5JhU3ERESlh2bh4PvruQiakZXNS2Dv+6piMVyurHsYgcm35SiIiUoG17s7llTBqz1uzkjvOacecFLfSQeBEpMhU3EZESsnjDHoaMSmXb3mye6dOJ3qfVCzqSiEQZFTcRkRIwdeEmfjtxLpWTEnnr5jNo36BK0JFEJAqpuImIhJG788znK/nXp8s5rWFVhl/fhdqVk4KOJSJRSsVNRCRMDhzM43dvzeP9+Ru5olN9Hr+yPUmJCUHHEpEopuImIhIGG3cfYOioNBZu2M19F7di2NlNMdNNCCJyclTcRESK2Zx1Oxk6Oo392bm8MiCF81vXCTqSiMQIFTcRkWI0aU4G9769gFMqJzF2cHda1KkUdCQRiSEqbiIixSAv33ni46W89NVqejStzvP9ulC9QpmgY4lIjFFxExE5SZlZOfxmwlw+X7qFft0b8fClbUlMKBV0LBGJQSpuIiInYd32/Qx6fRart+3j0cvacv3pyUFHEpEYpuImInKCpq3axq1jZ+MOo2/qxhnNagYdSURinIqbiMgJGD1jLX+evIjkmhV4ZUAKyTUrBB1JROKAipuIyHHIycvnkSmLGT1jLee2rMVTfTpROSkx6FgiEidU3EREimjnvoPcOnY201dvZ9jZTfl9z1YklNKguiJSclTcRESKYMXmTAaPSmXjriyevPo0rurSIOhIIhKHVNxERI7hi6VbuH38HJISExg/tAddGlcLOpKIxCkVNxGRo3B3Xv56NX+bupQ2dSszfEAK9aqWCzqWiMQxFTcRkSPIysnj/ncW8M6c9fRqX5d/XN2B8mX0I1NEgqWfQiIih9mSmcWw0WnMWbeL317YgtvPa4aZbkIQkeCpuImIFLJw/W6GjEpl1/4cXujXmYvb1w06kojID1TcRERC3p+/gXvenEf18mV465bTaVuvStCRRER+RMVNROJefr7z78+W8/TnK+nSuBov9u9CrUplg44lIvI/VNxEJK7tP5jLb9+Yx9RFm7i6SwP+ckU7ypZOCDqWiMgRqbiJSNzK2LmfIaPSWLZpDw/0as2gs5roJgQRiWgqbiISl1LX7ODmMWlk5+QzYmBXzmlZO+hIIiLHpOImInFnYmo6f5y0gPpVyzFhaFea1a4YdCQRkSJRcRORuJGbl8/jHy3l1W+/56xmNXm2byeqli8TdCwRkSJTcRORuLD7QA63j5/D18u3MvCMZB7o1ZrSCaWCjiUiclxU3EQk5n2/bR+DXp/Fuu37efzK9vTp1ijoSCIiJ0TFTURi2jcrtvLrsbMpnVCKsYO7071pjaAjiYicMBU3EYlJ7s7IaWv4ywdLaFarIq/ckELD6uWDjiUiclJU3EQk5hzMzedPkxcy/rt0LmxTh/+7tiMVy+rHnYhEP/0kE5GYsn1vNreMnc133+/g1+eeyt0XtqRUKQ2qKyKxQcVNRGLG0k17GPx6Klszs3nquo5c1rF+0JFERIqVipuIxIRPFm3irjfmUqFsaSYOO53TGlYNOpKISLEL6yBGZtbTzJaZ2Uozu+8I8xuZ2RdmNsfM5pvZJaHpF5pZmpktCP16XqF1uoSmrzSzp00PFhSJe2+nZTBsTBrNaldkyu1nqbSJSMwKW3EzswTgOeBioA3Qx8zaHLbYA8BEd+8EXAc8H5q+Dejt7u2BG4DRhdZ5ARgCNA999QzXPohI5Ht3znrueWseZ55akwlDT6dO5aSgI4mIhE04j7h1A1a6+2p3PwhMAC47bBkHKodeVwE2ALj7HHffEJq+CChnZmXNrC5Q2d1nuLsDo4DLw7gPIhLB3pu7nt9OnEuPJjUYPiCFcmUSgo4kIhJW4Sxu9YH0Qu8zQtMKexjob2YZwIfA7UfYzlXAbHfPDq2fcYxtAmBmQ80s1cxSt27demJ7ICIR6/35G7jrjbl0Ta7OqwNV2kQkPgT9oL4+wEh3bwBcAow2sx8ymVlb4O/AsOPdsLu/7O4p7p5Sq1atYgssIsH7cMFGfjNhLimNqzNiYFfKl9F9ViISH8JZ3NYDDQu9bxCaVtggYCKAu08HkoCaAGbWAJgEDHD3VYW22eAY2xSRGDZ14SbuGD+HTg2rMuLGrlTQwLoiEkfCWdxmAc3NrImZlaHg5oPJhy2zDjgfwMxaU1DctppZVeAD4D53/++hhd19I7DHzHqE7iYdALwXxn0QkQjy6eLN3DZuNh0aVOG1G7vqaQgiEnfCVtzcPRe4DfgYWELB3aOLzOwRM7s0tNjdwBAzmweMBwaGbjq4DWgGPGRmc0NftUPr3Aq8AqwEVgEfhWsfRCRy/GfJZm4dm0bb+lUYeVM3KiUlBh1JRKTEWUFPim0pKSmempoadAwROUFfLN3CsNFptKpbidGDulOlnEqbiMQuM0tz95QjzQv65gQRkZ/01fKtDBuTRotTKjL6JpU2EYlvKm4iErG+XbGNIaNSaVarImMGdadKeZU2EYlvKm4iEpGmrdzGoNdn0bRmBcYO7k7V8mWCjiQiEjgVNxGJONNXbeem12eRXKOgtFWroNImIgIqbiISYWau3s5NI2fRsFp5xg7pTo2KZYOOJCISMVTcRCRizFqzgxtHzqJe1STGDelBTZU2EZEfUXETkYiQtnYnA0d8xymVkxg/pAe1Kqm0iYgcTsVNRAI3Z91ObhjxHbUrJzF+aA9qV04KOpKISERScRORQM1L38WAV7+jRsUyjB/SgzoqbSIiR6XiJiKBWZCxm+tfnUnVComMH9KDU6qotImI/BQVNxEJxML1u+n/6kwqlysobfWqlgs6kohIxFNxE5ESt3jDHvq/OpOKZUszfkgPGlQrH3QkEZGooOImIiVq6aY99HtlBuUSExg/pAcNq6u0iYgUlYqbiJSY5Zsz6Td8JmVLF5S2RjVU2kREjoeKm4iUiBWbM+k7fAYJpYzxQ3uQXLNC0JFERKKOipuIhN3KLXvpM3wmZgWlrYlKm4jICVFxE5GwWr11L32HzwBg/JDunFqrYsCJRESil4qbiITNmm376DN8Bnn5zrgh3WlWu1LQkUREolrpoAOISGxau72gtOXkFZS2FnVU2kRETpaOuIlIsUvfsZ8+L8/gQE4eYwZ1p9UplYOOJCISE4pU3MzsHTPrZWYqeiLykzJ27ue6l2ew72AeYwd3p009lTYRkeJS1CL2PNAXWGFmfzOzlmHMJCJRav2uA/QZPoPMrBzGDu5O23pVgo4kIhJTilTc3P0zd+8HdAbWAJ+Z2TQzu9HMEsMZUESiw8bdB+jz8gx27c9hzODutKuv0iYiUtyKfOrTzGoAA4HBwBzgKQqK3KdhSSYiUWPT7iz6vDyDnfsOMuqmbnRoUDXoSCIiMalId5Wa2SSgJTAa6O3uG0Oz3jCz1HCFE5HIt2VPFn2Hz2BrZjajBnWnU6NqQUcSEYlZRR0O5Gl3/+JIM9w9pRjziEgU2ZKZRZ/hM9i0J4tRN3WjS2OVNhGRcCrqqdI2ZvbDuQ8zq2Zmt4Ypk4hEgW17s+k3fCYbd2cx8sZupCRXDzqSiEjMK2pxG+Luuw69cfedwJDwRBKRSLd9bzZ9h88gfed+RgzsSrcmKm0iIiWhqMUtwczs0BszSwDKhCeSiESyHfsO0u+Vmazdvp8RN3SlR9MaQUcSEYkbRb3GbSoFNyK8FHo/LDRNROLIzlBp+37bPl69oStnNKsZdCQRkbhS1OJ2LwVl7ZbQ+0+BV8KSSEQi0u79OfR/dSartu7llQEpnNVcpU1EpKQVqbi5ez7wQuhLROLM7gMFpW3F5r28PKALZ7eoFXQkEZG4VNRx3JoDjwNtgKRD0929aZhyiUiE2JOVw4BXZ7J00x5eur4L57SsHXQkEZG4VdSbE16j4GhbLnAuMAoYE65QIhIZMrNyuGHEdyzeuIfn+3XhvFZ1go4kIhLXilrcyrn7fwBz97Xu/jDQK3yxRCRoe7NzGfjaLBZk7ObZvp25sI1Km4hI0Ip6c0K2mZUCVpjZbcB6oGL4YolIkPZl53Lja98xN30Xz/bpxEVtTwk6koiIUPQjbr8BygN3AF2A/sAN4QolIsHZfzCXG0fOYva6XTx9XScubl836EgiIhJyzCNuocF2r3X3e4C9wI1hTyUigThwMI9BI1NJXbODf1/XiV4dVNpERCLJMY+4uXsecFYJZBGRAGXl5DF41Cxmfr+df13TkUtPqxd0JBEROUxRr3GbY2aTgTeBfYcmuvs7YUklIiUqKyePIaNSmbZqO09efRqXd6ofdCQRETmCoha3JGA7cF6haQ6ouIlEuaycPIaNTuPbldt44qoOXNm5QdCRRETkKIr65ARd1yYSg7Jz87hlTBpfLd/K369qz9UpDYOOJCIiP6GoT054jYIjbD/i7jcVeyIRKREHc/P59djZfLFsK3+9oj3Xdm0UdCQRETmGop4qfb/Q6yTgCmBD8ccRkZJwMDefX4+bzWdLtvDo5e3o212lTUQkGhT1VOnbhd+b2Xjg27AkEpGwysnL547xc/h08WYeuawt1/doHHQkEREpoqIOwHu45oCeNC0SZXLz8rlzwlymLtrEn3q3YcDpyUFHEhGR41DUa9wy+fE1bpuAe8OSSETCIjcvnzvfmMsHCzbyQK/W3Hhmk6AjiYjIcSrqqdJK4Q4iIuGTl+/c/eY83p+/kT9c3IrBP2sadCQRETkBRTpVamZXmFmVQu+rmtnl4YslIsXp4cmLeG/uBn7fsyXDfn5q0HFEROQEFfUatz+5++5Db9x9F/Cn8EQSkeI0ZsZaRs9Yy9Czm3LrOc2CjiMiIiehqMXtSMsVdSgREQnI9FXbeXjyIs5tWYt7e7YKOo6IiJykoha3VDP7l5mdGvr6F5AWzmAicnLWbd/PrWPTSK5Zgaf6dCKhlAUdSURETlJRi9vtwEHgDWACkAX8OlyhROTk7M3OZcioVPIdXhmQQuWkxKAjiYhIMShScXP3fe5+n7unuHtXd7/f3fcdaz0z62lmy8xspZndd4T5jczsCzObY2bzzeyS0PQaoel7zezZw9b5MrTNuaEvjScnUkh+vnPnhLms3LqX5/p2JrlmhaAjiYhIMSnqXaWfmlnVQu+rmdnHx1gnAXgOuBhoA/QxszaHLfYAMNHdOwHXAc+HpmcBDwL3HGXz/dy9Y+hrS1H2QSRePPnpMj5bspkHe7XmrOY1g44jIiLFqKinSmuG7iQFwN13cuwnJ3QDVrr7anc/SMEp1ssOW8aByqHXVQg9/zR0hO9bCgqciBTRe3PX89wXq+jTrSE3nJEcdBwRESlmRS1u+Wb2w1OozSyZHz9J4UjqA+mF3meEphX2MNDfzDKADym4lq4oXgudJn3QzI54xbWZDTWzVDNL3bp1axE3KxK95mfs4vdvzadbcnX+fGk7jvKtISIiUayoxe2PwLdmNtrMxgBfAX8ohs/vA4x09wbAJcBoMztWpn7u3h74Wejr+iMt5O4vh67JS6lVq1YxRBWJXFv2ZDF0VBo1K5blhf6dKVP6RB9DLCIikayoNydMBVKAZcB44G7gwDFWWw80LPS+QWhaYYOAiaHPmA4kAT95UY67rw/9mgmMo+CUrEjcysrJY+joNPZk5TB8QAo1KpYNOpKIiIRJUW9OGAz8h4LCdg8wmoLTnD9lFtDczJqYWRkKbj6YfNgy64DzQ5/RmoLidtTzmmZW2sxqhl4nAr8EFhZlH0Rikbtz/zsLmJu+i39d05E29SofeyUREYlaRX36wW+ArsAMdz/XzFoBf/2pFdw918xuAz4GEoAR7r7IzB4BUt19MgVFcLiZ3UXBNXMD3d0BzGwNBTculAk9F/UXwFrg41BpSwA+A4Yf1x6LxJDh36zmnTnr+e2FLejZ7pSg44iISJgVtbhluXuWmWFmZd19qZm1PNZK7v4hBTcdFJ72UKHXi4Ezj7Ju8lE226WImUVi2hdLt/D4R0vp1b4ut5+nZ5CKiMSDoha3jNA4bu8Cn5rZTgqOfolIAFZuyeSO8XNoU7cy/7i6g+4gFRGJE0Uqbu5+Rejlw2b2BQVjrk0NWyoROapd+w8y+PVUyiaW4uUBKZQvU9T/f4mISLQ77p/47v5VOIKIyLHl5uVz27g5bNiVxfih3alftVzQkUREpATpv+oiUeQvHyzh25XbeOJXHejSuHrQcUREpIRplE6RKDHhu3WMnLaGQWc14ZqUhsdeQUREYo6Km0gU+O77HTz43kLOblGLP1zcKug4IiISEBU3kQiXsXM/N49Jo2G18jzTpxOlE/RtKyISr/QvgEgE25edy+DXU8nJy2f4DSlUKZcYdCQREQmQiptIhMrPd347cS7LN2fybN/OnFqrYtCRREQkYCpuIhHq3/9ZwceLNnP/Ja35eYtaQccREZEIoOImEoE+mL+Rp/+zgqu7NGDQWU2CjiMiIhFCxU0kwixcv5u735xLl8bV+MsV7fQ4KxER+YGKm0gE2ZqZzdBRqVQvX4YX+3ehbOmEoCOJiEgE0ZMTRCJEdm4eN49JY8f+g7x18xnUqlQ26EgiIhJhVNxEIoC788CkhaSt3clzfTvTrn6VoCOJiEgE0qlSkQjw6rff82ZaBnec35xeHeoGHUdERCKUiptIwL5avpW/friEi9rW4c7zmwcdR0REIpiKm0iAVm3dy23jZtOiTiX+dU1HSpXSHaQiInJ0Km4iAdl9IIchr6dSJqEUr9yQQoWyuuRURER+mv6lEAlAbl4+t4+fQ/rO/Ywd3IMG1coHHUlERKKAiptIAB7/aClfL9/K365sT7cm1YOOIyIiUUKnSkVK2MTUdF799nsGnpHMdd0aBR1HRESiiIqbSAlKW7uDByYt5KxmNXmgV+ug44iISJRRcRMpIet3HWDY6DTqVU3i2b6dKJ2gbz8RETk+usZNpATsP5jL0FGpZOfkM2FoClXLlwk6koiIRCEVN5Ewc3d+9+Z8Fm/cw4gbutKsdqWgI4mISJTSuRqRMHvm85V8sGAj9/VsxbmtagcdR0REopiKm0gYTV24kX99upwrO9Vn6NlNg44jIiJRTsVNJEwWb9jDXW/Mo2PDqvz1yvaY6XFWIiJyclTcRMJg295shoxKpUq5RF6+vgtJiQlBRxIRkRigmxNEitnB3HxuHTObbXuzefPm06ldOSnoSCIiEiNU3ESKkbvzp8kL+W7NDp66riMdGlQNOpKIiMQQnSoVKUavT1vD+O/S+fW5p3JZx/pBxxERkRij4iZSTL5dsY1HP1jChW3qcPeFLYOOIyIiMUjFTaQYfL9tH78eN5tmtSryf9d2pFQp3UEqIiLFT8VN5CTtycphyKhUShm8ckMKFcvq0lEREQkP/QsjchLy8p3fjJ/Dmm37GD2oOw2rlw86koiIxDAVN5GT8MTUpXyxbCt/ubwdp59aI+g4IiIS43SqVOQEvTM7g5e+Xs31PRrTv0fjoOOIiEgcUHETOQFz1u3kvncWcHrTGjzUu03QcUREJE6ouIkcp027sxg6Oo06lcvyfL/OJCbo20hEREqG/sUROQ5ZOXkMHZ3K/uxcXr2hK9UqlAk6koiIxBHdnCBSRO7O796az4L1uxl+fQot6lQKOpKIiMQZHXETKaLnv1zFlHkb+N1FLbmgTZ2g44iISBxScRMpgk8Xb+afnyzjso71uOXnpwYdR0RE4pSKm8gxLNuUyZ0T5tChfhX+flUHzPQ4KxERCYaKm8hP2LHvIINHzaJC2dK8dH0KSYkJQUcSEZE4ppsTRI4iJy+fW8emsXlPNhOHnc4pVZKCjiQiInFOR9xEjuLPUxYxY/UO/n5Vezo2rBp0HBERERU3kSMZPWMtY2asY9jPm3JFpwZBxxEREQFU3ET+x7RV2/jz5EWc16o2v7+oVdBxREREfqDiJlLIuu37uXXsbJJrVuCp6zqSUEp3kIqISORQcRMJyczKYfCoWbjDKwNSqJSUGHQkERGRHwlrcTOznma2zMxWmtl9R5jfyMy+MLM5ZjbfzC4JTa8Rmr7XzJ49bJ0uZrYgtM2nTYNqSTHIz3fuemMuq7bu4/l+nUmuWSHoSCIiIv8jbMXNzBKA54CLgTZAHzNrc9hiDwAT3b0TcB3wfGh6FvAgcM8RNv0CMARoHvrqWfzpJd488fEyPluyhYd+2YYzm9UMOo6IiMgRhfOIWzdgpbuvdveDwATgssOWcaBy6HUVYAOAu+9z928pKHA/MLO6QGV3n+HuDowCLg/jPkgceOmrVbz41Sr6dW/EgNMbBx1HRETkqMJZ3OoD6YXeZ4SmFfYw0N/MMoAPgduLsM2MY2wTADMbamapZpa6devW48ktcWT09DU8/tFSftmhLo9c1k6PsxIRkYgW9M0JfYCR7t4AuAQYbWbFksndX3b3FHdPqVWrVnFsUmLMW2kZPPjeIi5oXZv/u1Z3kIqISOQLZ3FbDzQs9L5BaFphg4CJAO4+HUgCfuoCo/Wh7fzUNkWO6cMFG/n9W/M4q1lNnu3bmcSEoP8PIyIicmzh/NdqFtDczJqYWRkKbj6YfNgy64DzAcysNQXF7ajnNd19I7DHzHqE7iYdALwXjvASuz5fupk7xs+hc6NqvDygix4cLyIiUSNsD5l391wzuw34GEgARrj7IjN7BEh198nA3cBwM7uLghsVBoZuOsDM1lBw40IZM7sc+IW7LwZuBUYC5YCPQl8iRTJt5TZuHjOb1nUrM+LGrpQvE7ZvARERkWJnoZ4U01JSUjw1NTXoGBKwtLU7uf7VmTSsVp4JQ3tQrUKZoCOJiIj8DzNLc/eUI83ThT0SFxau383A176jdqWyjB7cTaVNRESikoqbxLwVmzMZMOI7KiclMnZID2pXSgo6koiIyAlRcZOYtnb7Pvq9MpOEUsbYwd2pX7Vc0JFEREROmIqbxKwNuw7Qd/hMcvLyGTu4u54/KiIiUU/FTWLSlsws+r0ykz0Hchg9qDst6lQKOpKIiMhJ01gIEnN27jvI9a98x6bdWYwZ3I129asEHUlERKRYqLhJTMnMyuGG177j++37eG1gV7o0rh50JBERkWKjU6USM/YfzOWmkbNYvGEPz/ftzJnNfurpaSIiItFHxU1iQnZuHsNGp5G2dif/d21HLmhTJ+hIIiIixU6nSiXq5eTlc9u4OXyzYhtP/KoDvU+rF3QkERGRsNARN4lqefnO3RPn8enizfz50rZck9Iw6EgiIiJho+ImUSs/37n/nQVMnreBe3u24oYzkoOOJCIiElYqbhKV3J1HP1jMG6np3H5eM24559SgI4mIiISdiptEpX99upzX/ruGm85swm8vbBF0HBERkRKh4iZR5/kvV/LM5yu5rmtDHvxla8ws6EgiIiIlQsVNosrr09bwxNRl/L/27jy8qvrO4/jnSxISQtgXkYQ9QQQU0BiwiBuiaKfVdmqLClZcqFO0is5MtU9nc8Y/5pmK2sq4VFDcitbWDm2dyiLiDgQFFDQh7EGEsG+S9Tt/5DqTMlgSyM3vnnvfr+fh4d5zzzn5XA/mfvI99+ReObyn7v/WGZQ2AEBKobghMl4q3qJ/mrtalw4+RT+7epjSWlHaAACpheKGSPj9ys90z29WaUxBV/3i2hHKSOOfLgAg9fDqh4S3YM12TXtxhQr7dNYTkwqVmZ4WOhIAAEFQ3JDQ3l67Uz984QMN6dleM28oVJvWlDYAQOqiuCFhFW/crVueKVb/rm01+8YitcvKCB0JAICgKG5ISB+V79Pkp5bp1A5ZevamkeqY3Tp0JAAAgqO4IeGUfH5Ak2YtUfs2GXru5pHq1i4zdCQAABICxQ0JZcPOQ5o4c4lap7XSC7eMVM+ObUJHAgAgYaSHDgB8qXzPYV33y/dVW+d6ccoo9enSNnQkAAASChM3JIQd+49o4pNLdKCyRs/cWKSCU9qFjgQAQMKhuCG43YeqNHHmEu04UKmnJxdpaG6H0JEAAEhIFDcEtf9Ita6ftUSbdh3Wk98v1Nl9OoWOBABAwqK4IZjDVTWa/NQylXx+QI9NPFtfG9A1dCQAABIaxQ1BHKmu1S3PFOvDzXv08IQRumhQ99CRAABIeFxVihZXXVunqc9/oHfKdumBq4fpijNODR0JAIBIYOKGFlVb57rzxRVa+OkO/etVQ/XXZ+eFjgQAQGRQ3NBi6upcP/7NKv1x1Tb95IpBmjSqT+hIAABECsUNLcLd9S+/X62Xl5frjrEFmnL+gNCRAACIHIob4s7d9e9/KtHs9zbpljH9dOclBaEjAQAQSRQ3xN2MRWV6bPE6XTuyt35yxekys9CRAACIJIob4mrm2xv0s3ml+taIXP3blUMpbQAAnASKG+JmztLN+tc/rNFlQ07Rf3znTLVqRWkDAOBkUNwQF/+1YqvufeUjXTCwm35+zQilp/FPDQCAk8WrKZrdvNWf666XVqqob2c9NvFsZaanhY4EAEBSoLihWb1ZWqHbXvhQZ+R20MwbzlGb1pQ2AACaC8UNzWbpht2a8myxBnTP0ezJRcrJ5BPVAABoThQ3NIuVW/bqxqeXqWfHNnr2piJ1yM4IHQkAgKRDccNJ+2Tbfl0/a6k6tc3QCzePUteczNCRAABIShQ3nJT1FQc1aeYStclI0ws3j1KPDlmhIwEAkLQobjhhW3Yf1nVPLpG79NzNI9Wrc3boSAAAJDWKG07I9v1HdN2TS3SoskbP3jRS+d1zQkcCACDpUdzQZLsOVuq6J5do18FKzb6xSIN7tg8dCQCAlMDva0CT7PuiWpNmLtWW3Yc1+8YijejdKXQkAABSBhM3NNqhyhrd8NRSrd1xQI9POluj+ncJHQkAgJTCxA2NcqS6VjfPLtaq8n2ace0IXXha99CRAABIOUzccFw1tXX6m+eW6/0Nu/Szq8/U+KGnho4EAEBKimtxM7PxZlZiZmVmds8xHu9tZovM7EMzW2VmVzR47N7YdiVmdlmDFmLYdQAAE+ZJREFU5RvN7CMzW2FmxfHMj3q/Xl6uRSUVuu+bQ/StEXmh4wAAkLLidqrUzNIkzZA0TlK5pGVmNtfd1zRY7aeSXnL3R81ssKRXJfWN3Z4gaYiknpIWmNlAd6+NbXeRu++MV3b8n8qaWv1i4VoN79VRE0f1CR0HAICUFs+JW5GkMndf7+5VkuZIuvKodVzSl79LooOkz2K3r5Q0x90r3X2DpLLY/tDC5izdos/2HdHfXnqazCx0HAAAUlo8i1uupC0N7pfHljX0z5Immlm56qdttzdiW5c0z8yWm9mUr/riZjbFzIrNrLiiouLEn0UK+6KqVo8sKlNRv84anc8VpAAAhBb64oRrJD3t7nmSrpD0rJkdL9N57n6WpMslTTWz84+1krs/4e6F7l7YrVu35k2dIp57f5MqDlTq7nEDmbYBAJAA4lnctkrq1eB+XmxZQzdJekmS3P09SVmSuv6lbd39y793SHpFnEKNi0OVNXp08TqNKeiqkfy+NgAAEkI8i9sySQVm1s/MWqv+YoO5R62zWdJYSTKz01Vf3Cpi600ws0wz6yepQNJSM2trZu1i67eVdKmkj+P4HFLW0+9u1O5DVbpr3MDQUQAAQEzcrip19xozu03Sa5LSJM1y99Vmdp+kYnefK+luSb80s2mqf+/aDe7uklab2UuS1kiqkTTV3WvN7BRJr8RO26VLesHd/xSv55Cq9n1RrccXr9PYQd35SCsAABJIXD85wd1fVf1FBw2X/WOD22skjf6Kbe+XdP9Ry9ZLGtb8SdHQzLc3aP+RGk1j2gYAQEIJfXECEsyeQ1Wa9fYGXT60h4bmdggdBwAANEBxw5954q31OlTFtA0AgEREccP/qjhQqaff2ahvDuupgae0Cx0HAAAcheKG//XY4nWqrKnVHWMLQkcBAADHQHGDJOnzfUf07Pub9O2z8tS/W07oOAAA4BgobpAkzVhUpro6Z9oGAEACo7hB5XsOa86yzfruOb3Uq3N26DgAAOArUNygXywsk5np9ovzQ0cBAAB/AcUtxW3ceUgvf1Cua4t669QObULHAQAAfwHFLcU9vHCtMtJMP7xoQOgoAADgOChuKWzt9gP63Yqt+v65fdW9XVboOAAA4DgobinsoQVrlZ2Rph9cwLQNAIAooLilqDWf7dcfP9qmG8/rp85tW4eOAwAAGoHilqKmzy9Vu6x03Xxe/9BRAABAI1HcUtDKLXu14JPtmjKmvzpkZ4SOAwAAGoniloKmzy9Vp+wMTT6vX+goAACgCShuKaZ4424tLq3QrRcMUE5meug4AACgCShuKeaBeaXqmpOp68/tGzoKAABoIopbCnm3bKfeW79LP7xwgNq0TgsdBwAANBHFLUW4ux6YX6oe7bN07cjeoeMAAIATQHFLEW+UVmj5pj267eJ8ZWUwbQMAIIoobinA3fXg/FLldWqj7xb2Ch0HAACcIIpbCpi/ZrtWle/Tj8YWqHU6hxwAgKjiVTzJ1dW5ps8vVb+ubfXtEbmh4wAAgJNAcUtyr368TZ9+fkB3XlKg9DQONwAAUcYreRKrrat/b9vAU3L0V2f2DB0HAACcJIpbEvuvFVu1ruKQpl0yUGmtLHQcAABwkihuSaq6tk4PLVirwae212VDeoSOAwAAmgHFLUn9Znm5Nu8+rLsvHahWTNsAAEgKFLckVFlTq1+8XqbhvTrq4kHdQ8cBAADNhOKWhF5ctkVb936huy8dKDOmbQAAJAuKW5I5Ul2rR14vU1Hfzjovv2voOAAAoBlR3JLMc+9v0o4DlUzbAABIQhS3JHKoskaPvrFO5+V31cj+XULHAQAAzYzilkRmv7dRuw5V6a5LB4aOAgAA4oDiliT2H6nW44vX6+JB3XVW706h4wAAgDiguCWJWW9v0L4vqnXXOKZtAAAkK4pbEth7uEoz39qg8UN6aGhuh9BxAABAnFDcksATb67XwaoaTWPaBgBAUqO4RdzOg5V66p2N+saZPXVaj3ah4wAAgDiiuEXcY2+sU2VNre64pCB0FAAAEGcUtwjbvv+Inn1/k741Ik8DuuWEjgMAAOKM4hZhMxaVqbbOdcdYpm0AAKQCiltEle85rF8t3ayrC3upd5fs0HEAAEALoLhF1COvl8lkuv3i/NBRAABAC6G4RdDGnYf06+XlunZkb/Xs2CZ0HAAA0EIobhH084VrlZFm+uGFA0JHAQAALYjiFjFlOw7qdyu26vpz+6p7+6zQcQAAQAuiuEXMQwtKlZWRph+c3z90FAAA0MIobhHyybb9+sOqbbpxdD91yckMHQcAALQwiluEPDi/VO2y0nXLGKZtAACkIopbRKwq36t5a7brljH91SE7I3QcAAAQAMUtIqbPL1XH7AxNHt03dBQAABBIXIubmY03sxIzKzOze47xeG8zW2RmH5rZKjO7osFj98a2KzGzyxq7z2S0fNNuvVFSoVsvGKB2WUzbAABIVXErbmaWJmmGpMslDZZ0jZkNPmq1n0p6yd1HSJog6T9j2w6O3R8iabyk/zSztEbuM+k8MK9UXXNa6/pz+4SOAgAAAornxK1IUpm7r3f3KklzJF151DouqX3sdgdJn8VuXylpjrtXuvsGSWWx/TVmn0nl3XU79e66XfqbC/OV3To9dBwAABBQPItbrqQtDe6Xx5Y19M+SJppZuaRXJd1+nG0bs09JkplNMbNiMyuuqKg40ecQlLtr+rxS9WifpetG9g4dBwAABBb64oRrJD3t7nmSrpD0rJk1SyZ3f8LdC929sFu3bs2xyxb35tqdKt60R1MvzldWRlroOAAAILB4nnvbKqlXg/t5sWUN3aT697DJ3d8zsyxJXY+z7fH2mRTcXQ/MK1Fuxzb6XmGv428AAACSXjwnbsskFZhZPzNrrfqLDeYetc5mSWMlycxOl5QlqSK23gQzyzSzfpIKJC1t5D6TwoJPdmhV+T7dMbZArdNDD0YBAEAiiNvEzd1rzOw2Sa9JSpM0y91Xm9l9kordfa6kuyX90symqf5ChRvc3SWtNrOXJK2RVCNpqrvXStKx9hmv5xBKXV39tK1vl2x9+6xjvoUPAACkIKvvScmtsLDQi4uLQ8dotD+u2qapL3ygh743XFeNoLgBAJBKzGy5uxce6zHOwSWY2jrXgwtKVdA9R98Y1jN0HAAAkEAobglm7sqtKttxUNPGDVRaKwsdBwAAJBCKWwKprq3TwwvW6vRT22v8kB6h4wAAgARDcUsgv/2gXBt3Hdbd4waqFdM2AABwFIpbgqisqdXPF5ZpWK+OGnt699BxAABAAqK4JYiXlm3R1r1f6O5xA2XGtA0AAPx/FLcEcKS6Vo8sKtM5fTtpTEHX0HEAAECCorglgOeXbNb2/ZW6a9xpTNsAAMBXorgFdriqRo++UabR+V107oAuoeMAAIAEFs8PmUcjzH53k3YerNLj404LHQUAACQ4Jm4BHThSrcffXKeLTuums/t0Ch0HAAAkOIpbQLPe3qi9h6t1F9M2AADQCBS3QPYertKTb63XZUNO0Rl5HULHAQAAEUBxC+SXb63XwaoaTRs3MHQUAAAQERS3AHYdrNRT72zU1884VYN6tA8dBwAARATFLYDHFq/Tkepa3XkJ0zYAANB4FLcWtmP/ET3z3iZdNSJX+d1zQscBAAARQnFrYTMWlam2znXH2ILQUQAAQMRQ3FrQ1r1f6FdLt+jqwjz16dI2dBwAABAxFLcW9MjrayVJt13MtA0AADQdxa2FbN51WL8uLtc1Rb2U27FN6DgAACCCKG4t5OGFa5XWyjT1ovzQUQAAQERR3FpA2Y6DeuXDcl1/bh91b58VOg4AAIgoilsLeHjhWmVlpOnWCwaEjgIAACKM4hZnn36+X79f+Zkmj+6rLjmZoeMAAIAIo7jF2YPzS9UuM123jOkfOgoAAIg4ilscfVS+T6+t3q6bx/RXx+zWoeMAAICIo7jF0fT5JeqYnaEbz+sbOgoAAEgCFLc4Wb5pjxaVVGjK+f3VLisjdBwAAJAEKG5xMn1+ibrmtNYNX+sbOgoAAEgSFLc4eG/dLr1Ttku3XjBA2a3TQ8cBAABJguLWzNxd0+eX6JT2mZo4qk/oOAAAIIlQ3JrZW2t3atnGPbrtonxlZaSFjgMAAJIIxa0ZubsemFei3I5t9N1zeoWOAwAAkgzFrRkt/GSHVpbv04/G5isznWkbAABoXhS3ZlJX55o+v1R9umTr22flhY4DAACSEMWtmfxp9edas22/7rykQBlp/GcFAADNj4bRDGrrXA/OL1V+9xx9c1hu6DgAACBJUdyawe9Xfqa1Ow5q2iUDldbKQscBAABJiuLWDLbu/ULD8jro8qE9QkcBAABJjF/r3wymXpSvH5zfX62YtgEAgDhi4tZM0rkgAQAAxBltAwAAICIobgAAABFBcQMAAIgIihsAAEBEUNwAAAAiguIGAAAQERQ3AACAiKC4AQAARATFDQAAICIobgAAABFBcQMAAIiIuBY3MxtvZiVmVmZm9xzj8QfNbEXsT6mZ7W3w2L+b2cexP99rsPxpM9vQYLvh8XwOAAAAiSI9Xjs2szRJMySNk1QuaZmZzXX3NV+u4+7TGqx/u6QRsdtfl3SWpOGSMiW9YWb/7e77Y6v/nbu/HK/sAAAAiSieE7ciSWXuvt7dqyTNkXTlX1j/Gkm/it0eLOlNd69x90OSVkkaH8esAAAACS+exS1X0pYG98tjy/4fM+sjqZ+k12OLVkoab2bZZtZV0kWSejXY5H4zWxU71Zr5FfucYmbFZlZcUVFxss8FAAAguES5OGGCpJfdvVaS3H2epFclvav6Kdx7kmpj694raZCkcyR1lvTjY+3Q3Z9w90J3L+zWrVuc4wMAAMRfPIvbVv35lCwvtuxYJuj/TpNKktz9fncf7u7jJJmk0tjybV6vUtJTqj8lCwAAkPTiWdyWSSows35m1lr15Wzu0SuZ2SBJnVQ/VftyWZqZdYndPlPSmZLmxe6fGvvbJF0l6eM4PgcAAICEEberSt29xsxuk/SapDRJs9x9tZndJ6nY3b8scRMkzXF3b7B5hqS36ruZ9kua6O41sceeN7Nuqp/CrZB0a7yeAwAAQCKxP+9LyamwsNCLi4tDxwAAADguM1vu7oXHeixRLk4AAADAcVDcAAAAIiIlTpWaWYWkTXH+Ml0l7Yzz10B8cQyjj2MYbRy/6OMYNo8+7n7M32WWEsWtJZhZ8Vedj0Y0cAyjj2MYbRy/6OMYxh+nSgEAACKC4gYAABARFLfm80ToADhpHMPo4xhGG8cv+jiGccZ73AAAACKCiRsAAEBEUNwAAAAiguLWDMxsvJmVmFmZmd0TOg8az8x6mdkiM1tjZqvN7I7QmXBizCzNzD40sz+EzoKmM7OOZvaymX1qZp+Y2bmhM6FpzGxa7Pvox2b2KzPLCp0pGVHcTpKZpUmaIelySYMlXWNmg8OmQhPUSLrb3QdLGiVpKscvsu6Q9EnoEDhhD0v6k7sPkjRMHMtIMbNcST+SVOjuQyWlSZoQNlVyoridvCJJZe6+3t2rJM2RdGXgTGgkd9/m7h/Ebh9Q/YtFbthUaCozy5P0dUlPhs6CpjOzDpLOlzRTkty9yt33hk2FE5AuqY2ZpUvKlvRZ4DxJieJ28nIlbWlwv1y88EeSmfWVNELSkrBJcAIekvT3kupCB8EJ6SepQtJTsdPdT5pZ29Ch0HjuvlXSzyRtlrRN0j53nxc2VXKiuAGSzCxH0m8k3enu+0PnQeOZ2V9J2uHuy0NnwQlLl3SWpEfdfYSkQ5J4v3CEmFkn1Z9t6iepp6S2ZjYxbKrkRHE7eVsl9WpwPy+2DBFhZhmqL23Pu/tvQ+dBk42W9E0z26j6typcbGbPhY2EJiqXVO7uX067X1Z9kUN0XCJpg7tXuHu1pN9K+lrgTEmJ4nbylkkqMLN+ZtZa9W/GnBs4ExrJzEz176v5xN2nh86DpnP3e909z937qv7/v9fdnZ/0I8TdP5e0xcxOiy0aK2lNwEhous2SRplZduz76lhxgUlcpIcOEHXuXmNmt0l6TfVX0cxy99WBY6HxRkuaJOkjM1sRW/YTd381YCYgFd0u6fnYD8DrJU0OnAdN4O5LzOxlSR+o/mr9D8XHX8UFH3kFAAAQEZwqBQAAiAiKGwAAQERQ3AAAACKC4gYAABARFDcAAICIoLgBQJyY2YVm9ofQOQAkD4obAABARFDcAKQ8M5toZkvNbIWZPW5maWZ20MweNLPVZrbQzLrF1h1uZu+b2SozeyX2GY0ys3wzW2BmK83sAzMbENt9jpm9bGafmtnzsd8qDwAnhOIGIKWZ2emSvidptLsPl1Qr6TpJbSUVu/sQSYsl/VNsk2ck/djdz5T0UYPlz0ua4e7DVP8Zjdtiy0dIulPSYEn9Vf9pHQBwQvjIKwCpbqyksyUtiw3D2kjaIalO0ouxdZ6T9Fsz6yCpo7svji2fLenXZtZOUq67vyJJ7n5EkmL7W+ru5bH7KyT1lfR2/J8WgGREcQOQ6kzSbHe/988Wmv3DUeud6OcDVja4XSu+7wI4CZwqBZDqFkr6jpl1lyQz62xmfVT//fE7sXWulfS2u++TtMfMxsSWT5K02N0PSCo3s6ti+8g0s+wWfRYAUgI/+QFIae6+xsx+KmmembWSVC1pqqRDkopij+1Q/fvgJOn7kh6LFbP1kibHlk+S9LiZ3Rfbx9Ut+DQApAhzP9HpPwAkLzM76O45oXMAQEOcKgUAAIgIJm4AAAARwcQNAAAgIihuAAAAEUFxAwAAiAiKGwAAQERQ3AAAACLifwBxTO+wBF2TNQAAAABJRU5ErkJggg==\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "plt.figure(figsize=(10, 10))\n",
        "plt.plot(history.history['loss'])\n",
        "plt.title('model loss')\n",
        "plt.ylabel('loss')\n",
        "plt.xlabel('epoch')\n",
        "plt.legend(['train'], loc='upper left')\n",
        "plt.show()"
      ],
      "metadata": {
        "id": "9dIpZk6AK8Zl",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 621
        },
        "outputId": "37cdcd79-036c-4e0e-bd28-2dd5eccadf14"
      },
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 720x720 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmEAAAJcCAYAAACxEXM4AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd3jV5cH/8c99cjJJIIucyJ4CB5CRgESWK9HiHggqTmS0rtY+bZ/2afu0v+66UVGmOCqKWkXrAgc7THFA2Et2wggrhKz79wdpH20ZCeTkPuP9ui6vy3oS+NA/6rvnvvM9xlorAAAA1C+P6wEAAACRiAgDAABwgAgDAABwgAgDAABwgAgDAABwgAgDAABwgAgDEBGMMVOMMb+v4dduNsZcera/DgCcChEGAADgABEGAADgABEGIGhUHwP+xBjzlTHmiDFmkjHGZ4z5wBhzyBjzsTEm5Vtff7UxZqUxptgYM8sY0+lbr/Uwxnxe/X2vSYr7t9/rSmPMF9Xfu8AYc94Zbh5hjFlvjNlnjHnHGNOk+p8bY8zjxphCY8xBY8zXxpgu1a8NMsYUVG/bboz5rzP6LwxASCPCAASbGyTlSjpX0lWSPpD0C0mNdfx/sx6QJGPMuZKmSvph9WvvS3rXGBNjjImR9LaklySlSnq9+tdV9ff2kDRZ0ihJaZLGSXrHGBNbm6HGmIsl/UnSTZLOkbRF0qvVL+dJGlD952hU/TV7q1+bJGmUtTZJUhdJn9bm9wUQHogwAMHmKWvtbmvtdklzJS2y1i631pZKektSj+qvGyLpPWvtTGttuaRHJMVLukBSH0nRkp6w1pZba9+QtORbv8dISeOstYustZXW2hckHav+vtq4VdJka+3n1tpjkn4uKccY00pSuaQkSR0lGWvtKmvtzurvK5fkN8Y0tNbut9Z+XsvfF0AYIMIABJvd3/r7oyf4z4nVf99Ex995kiRZa6skbZXUtPq17dZa+63v3fKtv28p6cfVR5HFxphiSc2rv682/n3DYR1/t6uptfZTSU9LekZSoTFmvDGmYfWX3iBpkKQtxpjZxpicWv6+AMIAEQYgVO3Q8ZiSdPwOlo6H1HZJOyU1rf5n/9TiW3+/VdIfrLXJ3/orwVo79Sw3NNDx483tkmStHWOtzZLk1/FjyZ9U//Ml1tprJGXo+LHptFr+vgDCABEGIFRNk3SFMeYSY0y0pB/r+JHiAkn5kiokPWCMiTbGXC+p97e+d4Kk0caY86sv0DcwxlxhjEmq5Yapku4yxnSvvk/2Rx0/Pt1sjOlV/etHSzoiqVRSVfWdtVuNMY2qj1EPSqo6i/8eAIQoIgxASLLWrpE0TNJTkvbo+CX+q6y1ZdbaMknXS7pT0j4dvz/2929971JJI3T8uHC/pPXVX1vbDR9L+pWkN3X83be2koZWv9xQx2Nvv44fWe6V9HD1a7dJ2myMOShptI7fLQMQYcx3r0wAAACgPvBOGAAAgANEGAAAgANEGAAAgANEGAAAgANe1wNqKz093bZq1cr1DAAAgNNatmzZHmtt4xO9FnIR1qpVKy1dutT1DAAAgNMyxmw52WscRwIAADhAhAEAADhAhAEAADgQcnfCTqS8vFzbtm1TaWmp6ykBFxcXp2bNmik6Otr1FAAAcBbCIsK2bdumpKQktWrVSsYY13MCxlqrvXv3atu2bWrdurXrOQAA4CyExXFkaWmp0tLSwjrAJMkYo7S0tIh4xw8AgHAXFhEmKewD7J8i5c8JAEC4C5sIAwAACCVEWB0oLi7W2LFja/19gwYNUnFxcQAWAQCAYEeE1YGTRVhFRcUpv+/9999XcnJyoGYBAIAgFhY/Henaf//3f2vDhg3q3r27oqOjFRcXp5SUFK1evVpr167Vtddeq61bt6q0tFQPPvigRo4cKen/PoLp8OHD+t73vqd+/fppwYIFatq0qaZPn674+HjHfzIAABAoYRdhv313pQp2HKzTX9PfpKH+96rOJ339z3/+s1asWKEvvvhCs2bN0hVXXKEVK1b86zESkydPVmpqqo4ePapevXrphhtuUFpa2nd+jXXr1mnq1KmaMGGCbrrpJr355psaNmxYnf45AABA8Ai7CAsGvXv3/s5zvMaMGaO33npLkrR161atW7fuPyKsdevW6t69uyQpKytLmzdvrre9AACg/oVdhJ3qHav60qBBg3/9/axZs/Txxx8rPz9fCQkJuvDCC0/4nK/Y2Nh//X1UVJSOHj1aL1sBAIAbXMyvA0lJSTp06NAJXztw4IBSUlKUkJCg1atXa+HChfW8DgAABKOweyfMhbS0NPXt21ddunRRfHy8fD7fv167/PLL9dxzz6lTp07q0KGD+vTp43ApAAAIFsZa63pDrWRnZ9ulS5d+55+tWrVKnTp1crSo/kXanxcAgFBljFlmrc0+0WscRwIAADhAhAEAADgQNhEWaseqZypS/pwAAIS7sIiwuLg47d27N+wDxVqrvXv3Ki4uzvUUAABwlsLipyObNWumbdu2qaioyPWUgIuLi1OzZs1czwAAAGcpLCIsOjr6O0+oPxsrth/Qva98rsdu6q6slil18msCAAD8u7A4jqxLzVMTtH3/Uc0s2O16CgAACGNE2L9pFB+tPm3SNKNgl+spAAAgjBFhJ5Dr92lj0RGtLzzsegoAAAhTRNgJ5PqPf+wQR5IAACBQiLATaJIcry5NG2omR5IAACBAiLCTyPNnavnWYhUeKnU9BQAAhCEi7CRy/T5ZK32yqtD1FAAAEIaIsJPomJmk5qnxmrGSI0kAAFD3iLCTMMYot1Om5m/YqyPHKlzPAQAAYYYIO4W8zj6VVVRpztrw/zgkAABQv4iwU8humaLkhGjN4FEVAACgjhFhp+CN8ujijhn6dHWhyiurXM8BAABhhAg7jTx/pg4cLdeSTftcTwEAAGGECDuNAeemK9br4UgSAADUKSLsNBJivOrfPl0zC3bLWut6DgAACBNEWA3k+n3aXnxUBTsPup4CAADCBBFWA5d08skYacZKjiQBAEDdIMJqID0xVlktUjSTe2EAAKCOEGE1lNfZp4KdB7Vtf4nrKQAAIAwQYTWU68+UJN4NAwAAdYIIq6HW6Q3ULiORCAMAAHWCCKuFPL9PizbtU3FJmespAAAgxBFhtZDr96myyuqzNYWupwAAgBBHhNVCt2bJykiK5VEVAADgrBFhteDxGF3q92n22iKVlle6ngMAAEIYEVZLeX6fSsoqtWDDHtdTAABACCPCaimnbZoSY738lCQAADgrRFgtxXqjNLBDY80sKFRVFR/oDQAAzgwRdgby/D7tOXxMy7cWu54CAABCFBF2Bi7skCGvx3AkCQAAzhgRdgYaxUerT5s0zSjY5XoKAAAIUUTYGcr1+7Sx6IjWFx52PQUAAIQgIuwM5fp9kvhAbwAAcGaIsDPUJDleXZo21EyOJAEAwBkgws5Cnj9Ty7cWq/BQqespAAAgxBBhZyHX75O10ier+EBvAABQO0TYWeiYmaTmqfHcCwMAALVGhJ0FY4xyO2Vq3vo9OnKswvUcAAAQQoiws5TX2aeyiirNWVvkegoAAAghRNhZym6ZouSEaM3gSBIAANQCEXaWvFEeXdwxQ5+uLlR5ZZXrOQAAIEQQYXUgz5+pA0fLtWTTPtdTAABAiCDC6sCAc9MV6/VwJAkAAGqMCKsDCTFe9W+frpkFu2WtdT0HAACEgIBFmDFmsjGm0Biz4iSvNzLGvGuM+dIYs9IYc1egttSHXL9P24uPqmDnQddTAABACAjkO2FTJF1+itfvlVRgre0m6UJJjxpjYgK4J6Au6eSTMdKMlRxJAgCA0wtYhFlr50g61U11KynJGGMkJVZ/bcg+8TQ9MVZZLVJ4ej4AAKgRl3fCnpbUSdIOSV9LetBae8JnPBhjRhpjlhpjlhYVBe9DUfM6+1Sw86C27S9xPQUAAAQ5lxF2maQvJDWR1F3S08aYhif6QmvteGtttrU2u3HjxvW5sVZy/ZmSxLthAADgtFxG2F2S/m6PWy9pk6SODvectdbpDdQuI5EIAwAAp+Uywr6RdIkkGWN8kjpI2uhwT53I8/u0aNM+FZeUuZ4CAACCWCAfUTFVUr6kDsaYbcaY4caY0caY0dVf8jtJFxhjvpb0iaSfWWv3BGpPfcn1+1RZZfXZmkLXUwAAQBDzBuoXttbefJrXd0jKC9Tv70q3ZsnKSIrVjJW7dV2PZq7nAACAIMUT8+uYx2N0qd+n2WuLVFpe6XoOAAAIUkRYAOT5fSopq9SCDSF/ugoAAAKECAuAnLZpSoz18lOSAADgpIiwAIj1Rmlgh8aaWVCoqio+0BsAAPwnIixA8vw+7Tl8TMu3FrueAgAAghARFiAXdsiQ12M4kgQAACdEhAVIo/ho9WmTphkFu1xPAQAAQYgIC6Bcv08bi45ofeFh11MAAECQIcICKNfvk8QHegMAgP9EhAVQk+R4dWnaUDM5kgQAAP+GCAuwPH+mlm8tVuGhUtdTAABAECHCAizX75O10ier+EBvAADwf4iwAOuYmaTmqfHcCwMAAN9BhAWYMUa5nTI1b/0eHTlW4XoOAAAIEkRYPcjr7FNZRZXmrC1yPQUAAAQJIqweZLdMUXJCtGZwJAkAAKoRYfXAG+XRxR0z9OnqQpVXVrmeAwAAggARVk/y/Jk6cLRcSzbtcz0FAAAEASKsngw4N12xXg9HkgAAQBIRVm8SYrzq3z5dMwt2y1rreg4AAHCMCKtHuX6fthcfVcHOg66nAAAAx4iwenRJJ5+M4QO9AQAAEVav0hNjldUiRTNWEmEAAEQ6Iqye5XX2qWDnQW3bX+J6CgAAcIgIq2e5/kxJHEkCABDpiLB61jq9gdplJBJhAABEOCLMgTy/T4s27VNxSZnrKQAAwBEizIFcv0+VVVafrSl0PQUAADhChDnQrVmyMpJi+SlJAAAiGBHmgMdjdKnfp9lri1RaXul6DgAAcIAIcyTP71NJWaXyN+x1PQUAADhAhDmS0zZNibFezSjY5XoKAABwgAhzJNYbpYEdGmtmQaGqqvhAbwAAIg0R5lCe36c9h49p+dZi11MAAEA9I8IcurBDhrwew4NbAQCIQESYQ43io9WnTRr3wgAAiEBEmGO5fp82Fh3R+sLDrqcAAIB6RIQ5luv3SeIDvQEAiDREmGNNkuPVpWlDzeRIEgCAiEKEBYE8f6aWby1W4aFS11MAAEA9IcKCQK7fJ2ulT1bxgd4AAEQKIiwIdMxMUvPUeO6FAQAQQYiwIGCMUW6nTM1bv0dHjlW4ngMAAOoBERYk8jr7VFZRpTlri1xPAQAA9YAICxLZLVOUnBCtGRxJAgAQEYiwIOGN8ujijhn6dHWhyiurXM8BAAABRoQFkTx/pg4cLdeSTftcTwEAAAFGhAWRAeemK9br4UgSAIAIQIQFkYQYr/q3T9fMgt2y1rqeAwAAAogICzK5fp+2Fx9Vwc6DrqcAAIAAIsKCzCWdfDKGD/QGACDcEWFBJj0xVlktUjRjJREGAEA4I8KCUF5nnwp2HtS2/SWupwAAgAAhwoJQrj9TEkeSAACEMyIsCLVOb6B2GYlEGAAAYYwIC1J5fp8Wbdqn4pIy11MAAEAAEGFBKtfvU2WV1WdrCl1PAQAAAUCEBaluzZKVkRTLT0kCABCmiLAg5fEYXer3afbaIpWWV7qeAwAA6hgRFsTy/D6VlFUqf8Ne11MAAEAdI8KCWE7bNCXGejWjYJfrKQAAoI4RYUEs1hulgR0aa2ZBoaqq+EBvAADCCREW5PL8Pu05fEzLtxa7ngIAAOoQERbkLuyQIa/H8OBWAADCDBEW5BrFR6tPmzTuhQEAEGaIsBCQ6/dpY9ERrS887HoKAACoI0RYCMj1+yTxgd4AAIQTIiwENEmOV5emDTWTI0kAAMIGERYi8vyZWr61WIWHSl1PAQAAdYAICxG5fp+slT5ZxQd6AwAQDoiwENExM0nNU+O5FwYAQJggwkKEMUa5nTI1b/0eHTlW4XoOAAA4S0RYCMnr7FNZRZXmrC1yPQUAAJwlIiyEZLdMUXJCtGZwJAkAQMgjwkKIN8qjiztm6NPVhSqvrHI9BwAAnAUiLMTk+TN14Gi5lmza53oKAAA4C0RYiBlwbrpivR6OJAEACHFEWIhJiPGqf/t0zSzYLWut6zkAAOAMEWEhKNfv0/bioyrYedD1FAAAcIaIsBB0SSefjOEDvQEACGVEWAhKT4xVVosUzVhJhAEAEKqIsBCV19mngp0HtW1/iespAADgDBBhISrXnymJI0kAAEIVERaiWqc3ULuMRCIMAIAQRYSFsDy/T4s27VNxSZnrKQAAoJaIsBCW6/epssrqszWFrqcAAIBaIsJCWLdmycpIiuVIEgCAEESEhTCPx+hSv0+z1hSptLzS9RwAAFALRFiIy/P7VFJWqfwNe11PAQAAtUCEhbictmlKjPVqRsEu11MAAEAtEGEhLtYbpYEdGmtmQaGqqvhAbwAAQgURFgby/D7tOXxMy7cWu54CAABqiAgLAxd2yJDXY/gpSQAAQggRFgYaxUerT5s07oUBABBCiLAwkev3aWPREW0oOux6CgAAqAEiLEzk+n2S+EBvAABCBREWJpokx6tL04aasZIjSQAAQgERFkby/JlavrVYhYdKXU8BAACnQYSFkVy/T9ZKn6ziA70BAAh2RFgY6ZiZpOap8dwLAwAgBBBhYcQYo9xOmZq3fo+OHKtwPQcAAJwCERZm8jr7VFZRpTlri1xPAQAAp0CEhZnslilKTojWDI4kAQAIagGLMGPMZGNMoTFmxSm+5kJjzBfGmJXGmNmB2hJJvFEeXdwxQ5+uLlR5ZZXrOQAA4CQC+U7YFEmXn+xFY0yypLGSrrbWdpY0OIBbIkqeP1MHjpZryeZ9rqcAAICTCFiEWWvnSDpVBdwi6e/W2m+qv57nKtSRAeemK9br0YyVHEkCABCsXN4JO1dSijFmljFmmTHm9pN9oTFmpDFmqTFmaVERF85PJyHGq/7t0zWzYLesta7nAACAE3AZYV5JWZKukHSZpF8ZY8490Rdaa8dba7OttdmNGzeuz40hK9fv0/bioyrYedD1FAAAcAIuI2ybpI+stUestXskzZHUzeGesHJJJ5+M4QO9AQAIVi4jbLqkfsYYrzEmQdL5klY53BNW0hNjldUihXthAAAEqUA+omKqpHxJHYwx24wxw40xo40xoyXJWrtK0oeSvpK0WNJEa+1JH2eB2svr7FPBzoPatr/E9RQAAPBvvIH6ha21N9fgax6W9HCgNkS6XH+m/vj+as0s2K27+rZ2PQcAAHwLT8wPY63TG6hdRiL3wgAACEJEWJjL8/u0aNM+HSgpdz0FAAB8CxEW5nL9PlVWWX26hnfDAAAIJkRYmOvWLFkZSbEcSQIAEGSIsDDn8Rhd6vdp1poilZZXup4DAACqEWERIM/vU0lZpfI37HU9BQAAVCPCIkBO2zQlxno1o2CX6ykAAKAaERYBYr1RGtihsWYWFKqqig/0BgAgGBBhESLP79Oew8e0fGux6ykAAEBEWMS4sEOGvB7DT0kCABAkiLAI0Sg+Wn3apHEvDACAIEGERZBcv08bi45oQ9Fh11MAAIh4RFgEyfX7JIkjSQAAggARFkGaJMerS9OGmrGSI0kAAFwjwiJMnj9Ty7cWq/BQqespAABENCIswuT6fbJW+mRVoespAABENCIswnTMTFLLtAQ989l6bd1X4noOAAARiwiLMMYYjRnaQ4dKKzRkXL428pOSAAA4QYRFoG7NkzV1RB8dq6jSTeMWas2uQ64nAQAQcYiwCOVv0lCvjeqjKI80dHy+Vmw/4HoSAAARhQiLYO0ykjRtVI4SYry6ecJCLduy3/UkAAAiBhEW4VqmNdDro3OUnhir2yYt0oINe1xPAgAgIhBhUJPkeL02qo+apcTrrueX6LM1PL4CAIBAI8IgScpIitOrI3PU3peokS8u1YcrdrqeBABAWCPC8C+pDWL0t3v6qGvTRrr3leWa/sV215MAAAhbRBi+o1F8tF4afr56t0rVD1/7Qq8u/sb1JAAAwhIRhv/QINar5+/qpYHnNtZ///1rTZ63yfUkAADCDhGGE4qLjtK427J0WWef/t8/CvTMZ+tdTwIAIKwQYTipWG+Unrmlp67t3kQPf7RGj85YI2ut61kAAIQFr+sBCG7eKI8evam74qKj9NSn61VSVqlfXtFJxhjX0wAACGlEGE4rymP0p+u7Ki46SpPmbdLR8kr9/pou8ngIMQAAzhQRhhoxxuh/r/IrISZKY2dtUGlZpf5643nyRnGiDQDAmSDCUGPGGP308o5KiInSIzPWqrSiUk8M6aEYLyEGAEBtEWGotfsubq/4GK9+948ClZYv09hbeyouOsr1LAAAQgpvYeCMDO/XWn+4ros+W1Oo4S8sUUlZhetJAACEFCIMZ+zW81vq0cHdlL9hr26ftFgHS8tdTwIAIGQQYTgr1/dspqdv6akvthZr2MRFKi4pcz0JAICQQIThrA3qeo7G356l1bsOaej4hSo6dMz1JAAAgh4RhjpxcUefnr+zl7bsLdGQcfnaeeCo60kAAAQ1Igx1pm+7dL04vLcKDx3TTePytXVfietJAAAELSIMdapXq1S9MuJ8HSqt0ODn8rWh6LDrSQAABCUiDHXuvGbJenVkH1VUVWnIuHyt2nnQ9SQAAIIOEYaA6JjZUK+NypHX49HQ8Qv11bZi15MAAAgqRBgCpm3jRL0+OkdJcV7dOmGRlm7e53oSAABBgwhDQDVPTdDro3PUOClWt01arPnr97ieBABAUCDCEHDnNIrXa6Ny1DItQXdNWaJPV+92PQkAAOeIMNSLxkmxmjqijzpmJmnki8v0/tc7XU8CAMApIgz1JqVBjF6+53x1b56s+175XH//fJvrSQAAOEOEoV41jIvWi8N7K6dtmn78+pd6ZdE3ricBAOAEEYZ6lxDj1aQ7eumiDhn6xVtfa+Lcja4nAQBQ74gwOBEXHaXnhmVpUNdM/f69VXr603WuJwEAUK+8rgcgcsV4PRoztIfivF/pkRlrVVJWqZ9c1kHGGNfTAAAIOCIMTnmjPHpkcDfFxURp7KwNKimr1P9e5SfEAABhjwiDcx6P0R+u7aL46ChNmrdJpeWV+sN1XRXlIcQAAOGLCENQMMbol1d0UkJMlJ76dL2Ollfq0cHd5I3i2iIAIDwRYQgaxhj9OK+D4qKj9PBHa3SsvEpjbu6hGC8hBgAIP/zbDUHn3ova6ddX+vXhyl0a+dJSlZZXup4EAECdI8IQlO7u11p/ur6rZq8t0l3PL9GRYxWuJwEAUKeIMAStm3u30OM3ddfizft026RFOnC03PUkAADqDBGGoHZtj6Z65pYe+nr7Ad06caH2HSlzPQkAgDpBhCHoXd7lHI2/LVvrdh/W0PH5KjxU6noSAABnjQhDSLioY4aev7OXtu0/qiHjFmpH8VHXkwAAOCtEGELGBe3S9dLw3tpz6JgGP5evLXuPuJ4EAMAZI8IQUrJapuqVEX10pKxCN43L1/rCw64nAQBwRogwhJyuzRrptZE5qqyShozLV8GOg64nAQBQa0QYQlKHzCRNG9VHMV6Pho7P1xdbi11PAgCgVogwhKw2jRM1bVSOkhNiNGziIi3etM/1JAAAaowIQ0hrnpqgaaNy5GsYq9snL9LcdUWuJwEAUCNEGEJeZqM4vTYqR63SGmj4lKX6uGC360kAAJwWEYawkJ4Yq1dH9lGnc5I0+uVlevfLHa4nAQBwSkQYwkZyQoxevud89WyRogdfXa63lm9zPQkAgJMiwhBWkuKiNeXuXurTJk0PTftS05ZudT0JAIATIsIQdhJivJp8Zy/1a5eun77xlV5Z9I3rSQAA/AciDGEpLjpKE27P1sUdM/SLt77Wi/mbXU8CAOA7iDCErbjoKD07rKdy/T79evpKTZy70fUkAAD+hQhDWIv1RmnsrT01qGumfv/eKj03e4PrSQAASKphhBljHjTGNDTHTTLGfG6MyQv0OKAuREd5NGZoD13VrYn+/MFqPfXJOteTAACQt4Zfd7e19kljzGWSUiTdJuklSTMCtgyoQ94ojx6/qZuiPUaPzlyr8iqrH13aXsYY19MAABGqphH2z39TDZL0krV2peHfXggx3iiPHh7cTd4oozGfrFNFZZV+clkHQgwA4ERNI2yZMWaGpNaSfm6MSZJUFbhZQGBEeYz+fP158kZ5NHbWBpVXVukXgzoRYgCAelfTCBsuqbukjdbaEmNMqqS7AjcLCByPx+gP13ZRtMdowtxNKq+0+t+r/IQYAKBe1TTCciR9Ya09YowZJqmnpCcDNwsILGOMfnN1Z3mjPJo0b5Mqqqr0/67uIo+HEAMA1I+aPqLiWUklxphukn4saYOkFwO2CqgHxhj98opOGj2wrV5e+I1+8dbXqqqyrmcBACJETd8Jq7DWWmPMNZKettZOMsYMD+QwoD4YY/SzyzsoJspozKfrVV5p9dcbz1MU74gBAAKsphF2yBjzcx1/NEV/Y4xHUnTgZgH1xxijh/I6yBvl0WMz16qiqkqPDu4mbxTPMgYABE5NI2yIpFt0/Hlhu4wxLSQ9HLhZQP174JL28kYZ/fXDNaqotHpiaHdFE2IAgACpUYRVh9ffJPUyxlwpabG1ljthCDs/uLCdYqI8+v17q1RRVaWnbu6pGC8hBgCoezX92KKbJC2WNFjSTZIWGWNuDOQwwJV7+rfRb67y66OVu/X9l5fpWEWl60kAgDBU0+PI/5HUy1pbKEnGmMaSPpb0RqCGAS7d2be1vFEe/fLtFRr54jKNuy1LcdFRrmcBAMJITc9ZPP8MsGp7a/G9QEga1qel/nJDV81ZV6R7Xliqo2W8IwYAqDs1DakPjTEfGWPuNMbcKek9Se8HbhYQHIb0aqFHbuymBRv26K4pi3XkWIXrSQCAMFGjCLPW/kTSeEnnVf813lr7s0AOA4LFDVnN9PiQ7lqyeb/umLxYh0rLXU8CAISBmt4Jk7X2TUlvBnALELSu6d5UXo9HD766XLdPXqwX7u6thnE8Kg8AcOZO+U6YMeaQMebgCf46ZIw5WKr6EG0AACAASURBVF8jgWBwxXnn6OlbemrF9gMaNnGRDpTwjhgA4MydMsKstUnW2oYn+CvJWtuwvkYCweLyLpl6bliWVu88pJsnLNS+I2WuJwEAQhQ/4QjU0iWdfJpwR7Y2FB3WLRMWas/hY64nAQBCEBEGnIGB5zbW5Dt7afPeI7p5/EIVHip1PQkAEGKIMOAM9W2Xril39db24qMaOm6hdh0gxAAANUeEAWehT5s0vXh3bxUeOqYh4/O1o/io60kAgBBBhAFnKbtVql4c3lv7DpdpyPh8bd1X4noSACAEEGFAHejZIkV/G3G+DpSUa+j4hdqy94jrSQCAIEeEAXXkvGbJemVEH5WUVWjIuIXaWHTY9SQAQBAjwoA61KVpI00d2UfllVUaOn6h1hcecj0JABCkAhZhxpjJxphCY8yK03xdL2NMhTHmxkBtAepTx8yGenVkH1VZaej4hVqzixADAPynQL4TNkXS5af6AmNMlKS/SJoRwB1AvWvvS9Jro/ooymM0dHy+CnbwKV8AgO8KWIRZa+dI2neaL7tfxz8UvDBQOwBX2jZO1GsjcxQfHaWbJyzU19sOuJ4EAAgizu6EGWOaSrpO0rM1+NqRxpilxpilRUVFgR8H1JFW6Q302qgcJcZ6dcvEhVr+zX7XkwAAQcLlxfwnJP3MWlt1ui+01o631mZba7MbN25cD9OAutM8NUHTRucoJSFGt01arGVbTvcGMQAgEriMsGxJrxpjNku6UdJYY8y1DvcAAdM0OV7TRuUoIylWt09arEUb97qeBABwzFmEWWtbW2tbWWtbSXpD0g+stW+72gMEWmajOL06so/OSY7Xnc8v0YL1e1xPAgA4FMhHVEyVlC+pgzFmmzFmuDFmtDFmdKB+TyDYZTSM09QRfdQiNUF3TVmiOWu54wgAkcpYa11vqJXs7Gy7dOlS1zOAs7LvSJmGTVyk9UWHNW5Yli7qmOF6EgAgAIwxy6y12Sd6jSfmAw6kNojRKyPOVwdfkka+tFQzVu5yPQkAUM+IMMCR5IQYvXzP+ercpJF+8LfP9cHXO11PAgDUIyIMcKhRfLReGt5b3Zon676py/XulztcTwIA1BMiDHAsKS5aL97dW1ktU/Tgq8v11vJtricBAOoBEQYEgQaxXk25q5f6tEnTQ9O+1LSlW11PAgAEGBEGBImEGK8m39lL/dql66dvfKVXFn3jehIAIICIMCCIxEVHacLt2bq4Y4Z+8dbXejF/s+tJAIAAIcKAIBMXHaVnh/VUrt+nX09fqYlzN7qeBAAIACIMCEKx3iiNvbWnBnXN1O/fW6XnZm9wPQkAUMe8rgcAOLHoKI/GDO0hr+dL/fmD1aqorNJ9F7d3PQsAUEeIMCCIeaM8enxId3k9Ro/MWKuySqsfXdpexhjX0wAAZ4kIA4JclMfo4cHd5I0yGvPJOlVUVuknl3UgxAAgxBFhQAiI8hj9+frz5I3yaOysDaqosvr59zoSYgAQwogwIER4PEZ/uLaLoj1G4+ds1P4jZfrDdV0V4+XnawAgFBFhQAgxxug3V3dWckKMnvxknbbsLdGzw3oqLTHW9TQAQC3xf6GBEGOM0Y9yz9VTN/fQl9uKdc0z87Vm1yHXswAAtUSEASHqqm5NNG1UjsoqqnTDswv06erdricBAGqBCANCWLfmyZp+X1+1Sk/Q8BeWauLcjbLWup4FAKgBIgwIcec0ite0UTn6XpfjT9f/2ZtfqayiyvUsAMBpEGFAGEiI8erpm3vqgYvbadrSbRo2aZH2HSlzPQsAcApEGBAmPB6jh/I66Mmh3fXF1mJd88w8rd3NhX0ACFZEGBBmruneVNNG5ai0vErXj12gz9YUup4EADgBIgwIQ92bJ+ud+/qqZVqChk9ZwoV9AAhCRBgQps5pFK/XR+foss7HL+z//O9fc2EfAIIIEQaEsYQYr565pafuv7idXl2yVbdxYR8AggYRBoQ5j8fox9UX9pdvLda1z8zXOi7sA4BzRBgQIa7p3lSvjeyjkrJKLuwDQBAgwoAI0qNFit65r6+apx6/sD9p3iYu7AOAI0QYEGGaJMfrje/nKNfv0+/+UaBfvMWFfQBwgQgDIlBCjFfP3pql+y5qp6mLj1/Y38+FfQCoV0QYEKE8HqP/uqyDnhhSfWF/7HytL+TCPgDUFyIMiHDX9miqV0f20ZFjlbrumQWavbbI9SQAiAhEGAD1bJGi6ff1VbPUBN31/GI9P58L+wAQaEQYAElS0+R4vTE6R5d28um37xboF2+tUHklF/YBIFCIMAD/0iDWq+eGZekHF7bV1MXf6PZJi7mwDwABQoQB+A6Px+inl3fUYzd107It+6sv7B92PQsAwg4RBuCEru/ZTFNH9tGRYxW6bux8zeHCPgDUKSIMwElltUzR2/f2VdPkeN35/GJN4cI+ANQZIgzAKTVLSdCb379Al3Ty6TfvFuh/3ubCPgDUBSIMwGk1iPVq3LAsff/Ctnpl0Te6Y/JiFZdwYR8AzgYRBqBGPB6jn1Vf2F+6eb+ufWa+NhRxYR8AzhQRBqBWjl/YP1+HSit07TPzNXcdF/YB4EwQYQBqLatlqqbf988L+0v0Yv5m15MAIOQQYQDOSLOUBL3x/Qt0UYcM/Xr6Sv3y7a+5sA8AtUCEAThjibFejbstS6MGttHLC7/Rnc9zYR8AaooIA3BWojxGP/9eJz0yuJuWbNqv68Yu4MI+ANQAEQagTtyY1UyvjDhfB4+W6zou7APAaRFhAOpMdqtUvX1vXzXhwj4AnBYRBqBONU/954X9xvr19JX61dsrVMGFfQD4D0QYgDp3/MJ+tkYNaKOXFm7Rnc8v0YGSctezACCoEGEAAiLKY/TzQZ308I3nadGmvbpu7Hxt5MI+APwLEQYgoAZnN9crI/qo+Gi5rn1mvuav3+N6EgAEBSIMQMD1apWq6ff21TmN4nX75MV6aeEW15MAwDkiDEC9OH5hP0cDz22sX729Qr+ezoV9AJGNCANQb5LiojXh9myNHNBGL+Zv0V1TuLAPIHIRYQDqVZTH6BeDOumvN56nhRuPX9jftOeI61kAUO+IMABO3JTdXH+75/8u7C/gwj6ACEOEAXCmd+vjF/Z9DWN1++TF+tsiLuwDiBxEGACnmqcm6M3vX6D+7dP1P2+t0G/eWcmFfQARgQgD4FxSXLQm3tFL9/RrrSkLNh+/sH+UC/sAwhsRBiAoRHmMfnmlX3+5oasWbtyra56epxXbD7ieBQABQ4QBCCpDerXQKyP6qLS8Stc/u0AvLdwia63rWQBQ54gwAEGnV6tUvfdAP+W0SdOv3l6h+6Yu16FSjicBhBciDEBQSkuM1fN39tLPLu+oD1fs0pVPcTwJILwQYQCClsdj9P0L2+q1kX1UVlGl68cu0Iv5mzmeBBAWiDAAQS+7Varef6C/+rVP16+nr9S9r3yugxxPAghxRBiAkJDSIEYTb8/WLwZ11Ecrd+vKMfP01bZi17MA4IwRYQBChsdjNHJAW00blaOKyird8OwCPT9/E8eTAEISEQYg5GS1TNH7D/bXwHMb67fvFmj0y8t4uCuAkEOEAQhJyQkxmnB7tn55RSd9sqpQV4yZqy+2cjwJIHQQYQBCljFG9/Rvo9dH58haafBzCzRpHseTAEIDEQYg5PVokaL3H+ivCztk6Hf/KNDIl5bpQAnHkwCCGxEGICw0SojW+Nuy9Osr/Zq1plCDxszV8m/2u54FACdFhAEIG8YY3d2vtd4YfYGMkQY/l6+JczdyPAkgKBFhAMJOt+bJeu+B/rqkU4Z+/94qjXhxqYpLylzPAoDvIMIAhKVG8dF6bliWfnOVX7PXFmnQk3O1bAvHkwCCBxEGIGwZY3Rn39Z68/sXyBvl0ZBx+Ro3e4OqqjieBOAeEQYg7J3XLFn/eKCf8jr79KcPVmv4C0u07wjHkwDcIsIARISGcdF65pae+n/XdNb89Xt1xZi5WrJ5n+tZACIYEQYgYhhjdHtOK/39BxcoxuvR0PELNXbWeo4nAThBhAGIOF2aNtI/7u+ny7tk6q8frtFdU5Zo7+FjrmcBiDBEGICIlBQXradv7qHfX9tF+Rv3atCYuVq8ieNJAPWHCAMQsYwxGtanpd76wQVKiPFq6Ph8PfMZx5MA6gcRBiDidW7SSO/e309XntdED3+0Rnc8v1h7OJ4EEGBEGABISoz16smh3fWn67tq0aZ9GvTkXC3cuNf1LABhjAgDgGrGGN3cu4Xe/kFfJcZ6dcuEhRrzyTpVcjwJIACIMAD4N/4mDfXO/f10dbcmemzmWt0xebGKDnE8CaBuEWEAcAKJsV49PqS7/nJDVy3ZvE+DxszVgvV7XM8CEEaIMAA4CWOMhvRqoen39VXDOK9unbRIT3y8luNJAHWCCAOA0+iY2VDv3NdP13Vvqic+XqfbJi1S4aFS17MAhDgiDABqoEGsV4/e1E1/vfE8ff7Nfg16cq7mreN4EsCZI8IAoIaMMbopu7neua+fkhNidNvkRXpsJseTAM4MEQYAtXSuL0nv3NdXN/RspjGfrNOtExdq90GOJwHUDhEGAGcgIcarRwZ30yODu+nLrQc06Mm5mrO2yPUsACGECAOAs3BjVjO9c19fpSXG6I7nF+uRj9aoorLK9SwAIYAIA4Cz1N6XpOn39tPgrGZ6+rP1umXCIu06wPEkgFMjwgCgDsTHROmvN3bT40O6acWOAxo0Zq5mrSl0PQtAECPCAKAOXdejmd65r58ykmJ15/NL9JcPV3M8CeCEiDAAqGPtMhL19r19dXPv5np21gbdPGGhdh446noWgCBDhAFAAMRFR+lP15+nJ4d2V8GOgxr05Fx9tprjSQD/hwgDgAC6pntTvXt/P2U2itddU5boT++vUjnHkwBEhAFAwLVpnKi3fnCBbjm/hcbN2agh4/K1vZjjSSDSEWEAUA/ioqP0x+u6aszNPbR292ENenKu3li2TdbykUdApApYhBljJhtjCo0xK07y+q3GmK+MMV8bYxYYY7oFagsABIuruzXRu/f3U7uMRP3X61/qtkmL9c3eEtezADgQyHfCpki6/BSvb5I00FrbVdLvJI0P4BYACBqt0xvo9VE5+t01nfXF1mLlPTFb4+ds4FEWQIQJWIRZa+dI2neK1xdYa/dX/8eFkpoFagsABBuPx+i2nFaa+dAA9WvXWH98f7WuHTtfK7YfcD0NQD0JljthwyV9cLIXjTEjjTFLjTFLi4r4gFwA4eOcRvGacHuWxt7aU7sPHtM1z8zXnz5YpaNlla6nAQgw5xFmjLlIxyPsZyf7GmvteGtttrU2u3HjxvU3DgDqgTFGg7qeo49/NFCDs5pp3OyNuuyJOZq/fo/raQACyGmEGWPOkzRR0jXW2r0utwCAa40SovXnG87T1BF9FOUxunXiIv3X619q/5Ey19MABICzCDPGtJD0d0m3WWvXutoBAMEmp22aPniwv35wYVu9vXy7Ln1stt75cgePswDCTCAfUTFVUr6kDsaYbcaY4caY0caY0dVf8mtJaZLGGmO+MMYsDdQWAAg1cdFR+unlHfXOff3ULCVeD0xdrrunLOEhr0AYMaH2/6yys7Pt0qX0GoDIUVllNWXBZj3y0RoZI/3ksg66PaeVojzG9TQAp2GMWWatzT7Ra84v5gMATi3KYzS8X2vN+NEA9WqVqt++W6Abnl2g1bsOup4G4CwQYQAQIpqnJmjKXb305NDu+mZfia4cM0+PfLRGpeU8zgIIRUQYAIQQY4yu6d5UHz80UFd3b6KnP1uvQU/O1aKN/IA5EGqIMAAIQakNYvTYTd314t29VV5VpSHjF+rnf/9aB46Wu54GoIaIMAAIYQPObayPfjhAI/q31mtLvlHuY7P14YqdrmcBqAEiDABCXEKMV/9zhV/T7+2n9MRYjX75c416aal2Hyx1PQ3AKRBhABAmujZrpOn39dV/f6+jZq0p0qWPztbLC7eoqiq0HkUERAoiDADCSHSUR6MHttVHPxygrs0a6Zdvr9CQ8flaX3jY9TQA/4YIA4Aw1Cq9gf52z/l6+MbztHb3YQ16cq7GfLJOZRVVrqcBqEaEAUCYMsZocHZzffzQQF3WJVOPzVyrK5+aq2Vb9rueBkBEGACEvcZJsXrq5h6afGe2DpdW6MbnFuh/p6/Q4WMVrqcBEY0IA4AIcXFHn2Y8NFB35LTSiwu3KPex2fpk1W7Xs4CIRYQBQARJjPXqN1d31pvfv0AN46I1/IWluveVz1V06JjraUDEIcIAIAL1bJGid+/vpx/nnquZK3fr0sdma9qSrbKWx1kA9YUIA4AIFeP16P5L2uuDH/ZXh8wk/fTNr3TrxEXavOeI62lARCDCACDCtW2cqFdH9NEfr+uqr7cf0GVPzNGzszaovJLHWQCBRIQBAOTxGN1yfgt9/NBAXdQhQ3/5cLWueXq+vt52wPU0IGwRYQCAf/E1jNNzt2XpuWFZ2nP4mK55Zp5+/48ClZTxOAugrhFhAID/cHmXTM18aKCG9m6hifM2Ke/xOZqztsj1LCCsEGEAgBNqFB+tP17XVdNG5SjG69Htkxfrode+0L4jZa6nAWGBCAMAnFLv1ql6/4H+euDidnr3qx269LHZenv5dh5nAZwlIgwAcFpx0VF6KK+D/nF/f7VMS9APX/tCdzy/RFv3lbieBoQsIgwAUGMdMpP0xugL9NurO2vZ5n3Ke3yOJs7dqMoq3hUDaosIAwDUSpTH6I4LWmnmQwN1Qds0/f69Vbpu7HwV7DjoehoQUogwAMAZaZIcr4l3ZOupm3toR/FRXfX0PP3lw9UqLa90PQ0ICUQYAOCMGWN0Vbcm+vihgbqhZ1M9O2uDLn9ijhZs2ON6GhD0iDAAwFlLTojRX2/splfuOV9W0i0TFun+qcu1ZS+fQwmcDBEGAKgzF7RL10c/HKD7L26njwt265JHZ+tXb69Q4aFS19OAoGNC7Tkv2dnZdunSpa5nAABOo/BgqcZ8uk6vLt6q6CiP7unfWiMGtFHDuGjX04B6Y4xZZq3NPuFrRBgAIJA27zmiR2eu1btf7lBKQrTuvaidhvVpqbjoKNfTgIAjwgAAzq3YfkB/+XC15q7bo6bJ8frhpe11fc9mivIY19OAgDlVhHEnDABQL7o0baSXhp+vV+45X+mJMfrJG1/p8ifmaMbKXXwEEiISEQYAqFcXtEvX2/f21bO39lRlldXIl5bpxufytXjTPtfTgHpFhAEA6p0xRt/reo5m/GiA/nR9V23bX6KbxuXr7ilLtGonT95HZOBOGADAuaNllZqyYLOenbVeh45V6NruTfVQ7rlqnprgehpwVriYDwAICQdKyvXs7A16fv4mVVmrW89vqfsubqf0xFjX04AzQoQBAELKrgOlevKTtZq2dJvivB7d07+NRgxoo8RYr+tpQK0QYQCAkLSh6LAenbFG73+9S2kNYnTfxe10y/ktFOvlGWMIDTyiAgAQkto2TtTYW7M0/d6+6pCZpN++W6BLHp2tv3++TZVVofUmAvDviDAAQNDr1jxZf7vnfL00vLeSE6L10LQvdcWYufp09W6eMYaQRYQBAEKCMUb92zfWO/f201M391BpeaXunrJUQ8Yt1LItPGMMoYcIAwCEFI/H6KpuTTTzoYH63bVdtGnvEd3wbL7ueWGp1u4+5HoeUGNczAcAhLSSsgo9P3+znpu1QYfLKnR9j2b6UW57NUvhGWNwj5+OBACEvf1HyjR21nq9kL9FstJtOS1170XtlNogxvU0RDAiDAAQMbYXH9UTM9fqzc+3qUGMVyMHtNHd/VqrAc8YgwNEGAAg4qzbfUgPf7RGMwp2Kz0xVg9c0k5De7VQjJfr0Kg/PCcMABBx2vuSNP72bL35/QvUpnED/Xr6Sl362GxN/2K7qnjGGIIAEQYACGtZLVP02sg+ev6uXmoQ69WDr36hK5+ap1lrCnnGGJwiwgAAYc8Yo4s6ZOi9+/vpiSHddehYue58folunrBQy7/Z73oeIhQRBgCIGB6P0bU9muqThy7Ub6/urPWFh3Xd2AUa9dJSrS887HoeIgwX8wEAEevwsQpNmrtJ4+ds0NHySg3Oaq4f5rbXOY3iXU9DmOCnIwEAOIW9h4/p6c/W628Lv5Ex0p0XtNL3L2yr5ASeMYazQ4QBAFADW/eV6PGP1+qt5duVGOvV6IFtdXff1oqPiXI9DSGKCAMAoBZW7zqohz9co09WFyojKVYPXNJeQ3o1V3QUV6lROzwnDACAWuiY2VCT7uyl10fnqEVqgn759grlPT5H//hqB88YQ50hwgAAOIlerVL1+ugcTbw9W9FRRve9slzXPDNf89btcT0NYYAIAwDgFIwxutTv0wcPDtAjg7tp35EyDZu0SEPH5+szHviKs8CdMAAAauFYRaVeXviNxs/ZoN0Hj+lcX6Lu6d9G13RvolgvF/jxXVzMBwCgjpVVVOndL3dowtyNWr3rkBonxerOC1pp2Pkt1Sgh2vU8BAkiDACAALHWat76PRo/Z6PmrtujhJgo3ZTdXMP7tVbz1ATX8+AYEQYAQD0o2HFQE+dt1Dtf7FCVtfpe13M0sn8bdWue7HoaHCHCAACoRzsPHNWUBZv1ysJvdOhYhXq3TtXI/m10cccMeTzG9TzUIyIMAAAHDpWW67UlWzV53ibtOFCqNo0baET/NrquR1PFRXOJPxIQYQAAOFReWaX3v96p8XM2auWOg0pPjNHtOa00rE9LpTbg8ynDGREGAEAQsNYqf+NeTZizUZ+tKVJctEeDs45f4m+V3sD1PATAqSLMW99jAACIVMYYXdA2XRe0Tdfa3Yc0ce5GvbZkq15etEWX+TM1YkAbZbVMcT0T9YR3wgAAcKjwYKleyN+slxd+owNHy5XVMkUj+rdRrt+nKC7xhzyOIwEACHJHjlXo9aVbNWn+Jm3dd1St0hI0vH8b3dizmeJjuMQfqogwAABCREVllT5auVvj52zQl9sOKCUhWrfltNLtOS2Vnhjreh5qiQgDACDEWGu1ZPN+jZ+zUR+v2q0Yr0c39Gyme/q3VtvGia7noYa4mA8AQIgxxqh361T1bp2qDUWHNXHuJr35+TZNXfyNLu3k08gBbdSrVYqM4d5YqOKdMAAAQsSew8f0Yv4WvZS/WftLytWtebJG9m+jyzr75I3yuJ6HE+A4EgCAMHK0rFJvfL5Nk+Zu1Oa9JWqeGq/hfVtrcHZzNYjlkCuYEGEAAIShyiqrmQW7NWHuRi3bsl+N4qM1rE8L3ZHTShkN41zPg4gwAADC3rIt+zRhziZ9VLBL0R6Pru3RRCP6t1F7X5LraRGNi/kAAIS5rJapyrotVZv3HNGkeZv0+rKtmrZ0my7q0FgjBrRRTps0LvEHGd4JAwAgDO07Uqa/LdyiF/L/f3v3HmR1fd5x/P2wyx0EZQGR3eWiJIh3wcVoMEYca4yNnYwXwNjWP+JMx7ZJpzNtk2mbmf7dSZuZZpJoahMngERiOk7iROul3jruQtDEIl4QdXcRWZSCXOS2+/SPc+KA8bbI7nfP4f2aYeD8zuHsZ+c7HD77/L7n/F7lzd0HOHP6CXx10WyuOmsaw93EP2g8HSlJ0nFq38Fe/vPpzdz++CZe3raH6RNHc/PFM1nS1so4N/EPOEuYJEnHub6+5JEXerjtsU20v7Kd8aMaWbawlZsvmsXJE9zEP1AsYZIk6V2/6drB7Y9v4r5ntzAsgi+dW9nEf/q0E0pHqzuWMEmS9Hu6tu/ljidfYdWaLvYe6GXRnCZuuWQ2nz2tyU38x4glTJIkfaCdew+yvOM1fvTkq/Ts2s/ck8fz1UWzufqcaYxsbCgdr6ZZwiRJ0kfaf6iXe595ndsf38SLW3dz4pjhXDu/mSVtrV40/ChZwiRJ0seWmTy58S1WdLzGA+u3cqgvuXD2SSxbOIM/OGOq07F+sIRJkqSj0rNrH3ev7eauNZ10bX+Hk8aO4Lr5zSxta2Vm09jS8YY8S5gkSfpE+vqSJza+yYr2Tv5rw1Z6+5KLTp3EsoWtXDHvZEY0+gGw78cSJkmSjpmet/fx07VdrOzoYvOOd2gaN4Jr57ewtK2FGZOcjh3OEiZJko653r7ksZe2saK9k4ef76G3L1k0p4llba1cPm+ql0fCEiZJkgbYGzsr07G7Ojp5fec+msaN5PoFlb1jLSeNKR2vGEuYJEkaFL19yaMv9rw7HUtg0ZzJLGtrZfHpU4676ZglTJIkDbotO99h1ZouVq3pYsvOfUwZP5LrF7SwpK2F5hOPj+mYJUySJBVzqLeP/35hGys6OnnkhR4APvepynTssrlTaKzj6ZglTJIkDQmbd7zDqo5OVq3tYuvb+zn5hFFcf0ELSy5o4ZSJo0vHO+YsYZIkaUg51NvHw8/3sKKjk0df3EYAn//0FJa2tfL5uVNoGFYfFxC3hEmSpCGra/veyt6xtV1s27WfaRNGccMFLdxwQQvTJtT2dMwSJkmShryDvX08tGErKzq6ePylynTssrlTuXFhK5d8anJNTsc+rIQ1DnYYSZKk9zO8YRhXnjmNK8+cRtf2vazs6OSna7t5cMNWpk8c/e50bOoJo0pHPSachEmSpCHrwKE+HtywlZUdnTz+0ps0DAsWz53CsoWtXDJnMsOG+HTMSZgkSapJIxqHcdVZ07jqrGm89tYeVnZ0cffaLh54bivNJ45maVsr181vZkoNTsechEmSpJpy4FAfDzz3BivaO/mfl9+icVhw+elTWbawlc+e1jSkpmNOwiRJUt0Y0TiMq88+havPPoVX3tzDyo5OVv+6m1+tf4PWk8awpK2F6+a3MHn8yNJRP5STMEmSVPP2H+rl/vVbWdH+Gk9t2k7jsOCKM6ayrG0GF506qdh0zI+okCRJx42Xt+1mZXsnq9d1s2PvQWZOGsOStlaund9M07jBnY4VKWERcQdwNdCTmWe+z/0BfAe433/LvAAABnpJREFUCtgL/Glmrvuo57WESZKkj2PfwV7uX/8Gy9s76XhlO8MbgivOOJkb21r5zKmTqFSRgVVqT9iPgH8D7vyA+78AzKn+Wgh8r/q7JEnSJzZqeAPXnDuda86dzsaeXaxo7+Jn67r55W+3MKtpLH926alcv6ClWL4Bu2x5Zj4GbP+Qh1wD3JkVTwETI2LaQOWRJEnHr9OmjOcf/3Ae7d9czLevP4dJY0fQ8/a+oplKvjtyOtB12O3u6rEt731gRNwC3ALQ2to6KOEkSVL9GTW8gS+f38yXz2+mr6/svvgBm4QdS5l5W2YuyMwFkydPLh1HkiTVgdKfJ1ayhG0GDj8R21w9JkmSVPdKlrB7gT+OiguBnZn5e6ciJUmS6tGA7QmLiJXApUBTRHQD3wKGA2Tm94H7qHw8xUYqH1Fx80BlkSRJGmoGrIRl5tKPuD+BWwfq60uSJA1lNbExX5Ikqd5YwiRJkgqwhEmSJBVgCZMkSSrAEiZJklSAJUySJKkAS5gkSVIBljBJkqQCLGGSJEkFWMIkSZIKsIRJkiQVYAmTJEkqwBImSZJUgCVMkiSpAEuYJElSAZYwSZKkAixhkiRJBVjCJEmSCojMLJ2hXyJiG/DaIHypJuDNQfg6GhiuX+1zDWufa1jbXL9jY0ZmTn6/O2quhA2WiFibmQtK59DRcf1qn2tY+1zD2ub6DTxPR0qSJBVgCZMkSSrAEvbBbisdQJ+I61f7XMPa5xrWNtdvgLknTJIkqQAnYZIkSQVYwiRJkgqwhL1HRFwZES9ExMaI+LvSedQ/EdESEY9ExHMRsT4ivlY6k/ovIhoi4umI+EXpLOq/iJgYEasj4vmI2BARnymdSf0TEX9VfQ3934hYGRGjSmeqR5aww0REA/Bd4AvAPGBpRMwrm0r9dAj468ycB1wI3Ooa1qSvARtKh9BR+w7wq8ycC5yDa1lTImI68JfAgsw8E2gAlpRNVZ8sYUdqAzZm5qbMPADcBVxTOJP6ITO3ZOa66p93UXnxn142lfojIpqBLwI/LJ1F/RcRE4BLgH8HyMwDmbmjbCodhUZgdEQ0AmOA1wvnqUuWsCNNB7oOu92N/4HXrIiYCZwHtJdNon76V+BvgL7SQXRUZgHbgP+onlL+YUSMLR1KH19mbgb+GegEtgA7M/OBsqnqkyVMdSkixgE/A76emW+XzqOPJyKuBnoy89els+ioNQLnA9/LzPOAPYD7a2tIRJxI5SzQLOAUYGxEfKVsqvpkCTvSZqDlsNvN1WOqIRExnEoBW56Z95TOo365GPhSRLxKZTvAZRHxk7KR1E/dQHdm/m4CvZpKKVPtuBx4JTO3ZeZB4B7gosKZ6pIl7EhrgDkRMSsiRlDZiHhv4Uzqh4gIKntRNmTmt0vnUf9k5jcyszkzZ1L59/dwZvoTeA3JzDeAroj4dPXQYuC5gpHUf53AhRExpvqauhjfXDEgGksHGEoy81BE/DlwP5V3g9yRmesLx1L/XAzcBDwbEc9Uj30zM+8rmEk63vwFsLz6w+wm4ObCedQPmdkeEauBdVTecf40XsJoQHjZIkmSpAI8HSlJklSAJUySJKkAS5gkSVIBljBJkqQCLGGSJEkFWMIk6WOKiEsj4helc0iqD5YwSZKkAixhkupORHwlIjoi4pmI+EFENETE7oj4l4hYHxEPRcTk6mPPjYinIuK3EfHz6nXziIjTIuLBiPhNRKyLiFOrTz8uIlZHxPMRsbz6ieKS1G+WMEl1JSJOB24ALs7Mc4Fe4EZgLLA2M88AHgW+Vf0rdwJ/m5lnA88ednw58N3MPIfKdfO2VI+fB3wdmAfMpnKVBknqNy9bJKneLAbmA2uqQ6rRQA/QB6yqPuYnwD0RMQGYmJmPVo//GLg7IsYD0zPz5wCZuQ+g+nwdmdldvf0MMBN4YuC/LUn1xhImqd4E8OPM/MYRByP+4T2PO9prtu0/7M+9+Doq6Sh5OlJSvXkIuDYipgBExEkRMYPK69211ccsA57IzJ3A/0XEourxm4BHM3MX0B0Rf1R9jpERMWZQvwtJdc+f4CTVlcx8LiL+HnggIoYBB4FbgT1AW/W+Hir7xgD+BPh+tWRtAm6uHr8J+EFE/FP1Oa4bxG9D0nEgMo92Ii9JtSMidmfmuNI5JOl3PB0pSZJUgJMwSZKkApyESZIkFWAJkyRJKsASJkmSVIAlTJIkqQBLmCRJUgH/DytjB9vq9XsYAAAAAElFTkSuQmCC\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# dot_img_file = \"d_mod-\"+str(d_model)+\"-d_k-\"+str(d_k)+\"-d_v-\"+str(d_v)+\"-n_heads-\"+str(n_heads)+\"-enc_lays-\"+str(n_encoder_layers)+\"-dec_lays-\"+str(n_decoder_layers)+'-transformer.png'\n",
        "# tf.keras.utils.plot_model(transformer, to_file=dot_img_file, show_shapes=True)"
      ],
      "metadata": {
        "id": "RSK1eufbeQPL"
      },
      "execution_count": 25,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#transformer = keras.models.load_model('./Models/')"
      ],
      "metadata": {
        "id": "Oly3kYeU3FnV"
      },
      "execution_count": 26,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Testing"
      ],
      "metadata": {
        "id": "KJ_Gx7RVeNla"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def greedy_translate(model, source_sentence, target_sentence_start=sp_ben_bpe.bos_id()):\n",
        "  # Tokenizing and padding\n",
        "  source_seq = tokenize(source_sentence, sp_en_bpe)\n",
        "  source_seq = tf.keras.preprocessing.sequence.pad_sequences(source_seq, maxlen = MAX_TOKENS, padding='post', truncating='post')\n",
        "  predict_seq = [[target_sentence_start]]\n",
        "  \n",
        "  predict_sentence = [target_sentence_start] # Deep copy here to prevent updates on target_sentence_start\n",
        "  while predict_sentence[-1] != sp_ben_bpe.eos_id() and len(predict_sentence) < MAX_TOKENS:\n",
        "    predict_output = model([np.array(source_seq), np.array(predict_seq)], training=None)\n",
        "    predict_label = tf.argmax(predict_output, axis=-1) # Pick the label with highest softmax score\n",
        "    predict_seq = tf.concat([predict_seq, predict_label], axis=-1) # Updating the prediction sequence\n",
        "    predict_sentence.append(predict_label[0][0].numpy())\n",
        "  \n",
        "  if len(predict_sentence) == MAX_TOKENS:\n",
        "      return detokenize(predict_sentence, sp_ben_bpe, True, False)\n",
        "  return detokenize(predict_sentence, sp_ben_bpe)"
      ],
      "metadata": {
        "id": "zfg1AaKDnPQM"
      },
      "execution_count": 27,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import tensorflow_probability as tfp"
      ],
      "metadata": {
        "id": "pLrtBO__iPPl"
      },
      "execution_count": 28,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "sentence = \"I love you.\"\n",
        "print(\"English Sentence: \", sentence)\n",
        "print(\"Bengali Sentence: \", greedy_translate(transformer, sentence))"
      ],
      "metadata": {
        "id": "KtWeJl1JoapJ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "833ab8ec-36cc-473d-cb9f-e73085e2488e"
      },
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "English Sentence:  I love you.\n",
            "Bengali Sentence:  আমি তোমায় ভালোবাসি।\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "sentence = \"I am tired.\"\n",
        "print(\"English Sentence: \", sentence)\n",
        "print(\"Bengali Sentence: \", greedy_translate(transformer, sentence))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "K2axdVLjq6z1",
        "outputId": "481bb49c-0e88-4319-e977-dd8fdef522a8"
      },
      "execution_count": 49,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "English Sentence:  I am tired.\n",
            "Bengali Sentence:  আমি ক্লান্ত হয়ে গেছি।\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "sentences = list(test_val_data['src'][:250])\n",
        "refs = [list(test_val_data['tgt'][:250])]\n",
        "sys = [greedy_translate(transformer, sentence) for sentence in sentences]"
      ],
      "metadata": {
        "id": "9XNckV-wBgiN"
      },
      "execution_count": 30,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "bleu = BLEU()\n",
        "bleu.corpus_score(sys, refs)"
      ],
      "metadata": {
        "id": "otRhPJC1VKyj",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "2d70d3a5-9c1a-43f4-e054-2a1298a46bf8"
      },
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "BLEU = 1.07 11.7/2.0/0.6/0.1 (BP = 1.000 ratio = 1.050 hyp_len = 3761 ref_len = 3583)"
            ]
          },
          "metadata": {},
          "execution_count": 31
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "chrf = CHRF()\n",
        "chrf.corpus_score(sys, refs)"
      ],
      "metadata": {
        "id": "mlhToSrvYNqd",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6f4eeae3-26c4-45f8-cc5b-2c75f5843204"
      },
      "execution_count": 32,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "chrF2 = 19.03"
            ]
          },
          "metadata": {},
          "execution_count": 32
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "ter = TER()\n",
        "ter.corpus_score(sys, refs)"
      ],
      "metadata": {
        "id": "DtIadLMJYe1w",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "971ea9cc-70f5-467e-c476-39a0af740794"
      },
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "TER = 112.39"
            ]
          },
          "metadata": {},
          "execution_count": 33
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "def sampling_translate(model, source_sentence, temperature=0.6, target_sentence_start=sp_ben_bpe.bos_id()):\n",
        "  # Tokenizing and padding\n",
        "  source_seq = tokenize(source_sentence, sp_en_bpe)\n",
        "  source_seq = tf.keras.preprocessing.sequence.pad_sequences(source_seq, maxlen = MAX_TOKENS, padding='post', truncating='post')\n",
        "  predict_seq = [[target_sentence_start]]\n",
        "  logprob = 0\n",
        "  \n",
        "  predict_sentence = [target_sentence_start] # Deep copy here to prevent updates on target_sentence_start\n",
        "  while predict_sentence[-1] != sp_ben_bpe.eos_id() and len(predict_sentence) < MAX_TOKENS:\n",
        "    predict_output = model([np.array(source_seq), np.array(predict_seq)], training=None)\n",
        "    probs = predict_output[-1].numpy()[0]\n",
        "    log_probs = np.log(probs)\n",
        "    u = np.random.uniform(low=1e-6, high=1.0 - 1e-6, size=log_probs.shape)\n",
        "    g = -np.log(-np.log(u))\n",
        "    sample_idx = np.argmax(log_probs + g * temperature, axis=-1)\n",
        "    sample_prob = predict_output[-1].numpy()[0][sample_idx]\n",
        "    logprob += np.log(sample_prob)\n",
        "    predict_label = np.arange(predict_output.shape[-1])[sample_idx] # Sample the label softmax score\n",
        "    predict_seq = tf.concat([predict_seq, tf.cast([[predict_label]], dtype=tf.int32)], axis=-1) # Updating the prediction sequence\n",
        "    predict_sentence.append(predict_label)\n",
        "  \n",
        "  if len(predict_sentence) == MAX_TOKENS:\n",
        "      return detokenize(predict_sentence, sp_ben_bpe, True, False), logprob\n",
        "  return detokenize(predict_sentence, sp_ben_bpe), logprob"
      ],
      "metadata": {
        "id": "2dmVo_eQhE8v"
      },
      "execution_count": 34,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "sentence = \"I love you.\"\n",
        "print(\"English Sentence: \", sentence)\n",
        "print(\"Bengali Sentence: \", sampling_translate(transformer, sentence))"
      ],
      "metadata": {
        "id": "CB5iHHlxh1cC",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "00ba0fcb-5342-474d-9315-532566664d34"
      },
      "execution_count": 35,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "English Sentence:  I love you.\n",
            "Bengali Sentence:  ('আমি ভালোবাসি তুমি।', -5.600754240527749)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "sentence = \"I am hungry.\"\n",
        "print(\"English Sentence: \", sentence)\n",
        "print(\"Bengali Sentence: \", sampling_translate(transformer, sentence))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "IiZmzCY3q1S9",
        "outputId": "9a5b6a20-8769-4a9b-a990-cbcb6f5dff90"
      },
      "execution_count": 47,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "English Sentence:  I am hungry.\n",
            "Bengali Sentence:  ('আমি মদ খেয়েছি।', -7.641852107364684)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "def generate_samples(sentence, n_samples, model=None, temperature=0.6):\n",
        "    samples, log_probs = [], []\n",
        "    for _ in range(n_samples):\n",
        "        sample, logp = sampling_translate(model, sentence, temperature)\n",
        "        samples.append(sample)\n",
        "        log_probs.append(logp)\n",
        "    return samples, log_probs"
      ],
      "metadata": {
        "id": "xIxJ9HqTtxHY"
      },
      "execution_count": 36,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def rouge1_similarity(system, reference):\n",
        "    sys_counter = Counter(system)\n",
        "    ref_counter = Counter(reference)\n",
        "    overlap = 0\n",
        "    for token in sys_counter:\n",
        "        token_count_sys = sys_counter[token]\n",
        "        token_count_ref = ref_counter[token]\n",
        "        overlap += min(token_count_ref, token_count_sys)\n",
        "    precision = overlap / sum(sys_counter.values())\n",
        "    recall = overlap / sum(ref_counter.values())\n",
        "    if precision + recall != 0:\n",
        "        rouge1_score = 2 * ((precision * recall)/(precision + recall))\n",
        "    else:\n",
        "        rouge1_score = 0     \n",
        "    return rouge1_score"
      ],
      "metadata": {
        "id": "uhIlVoVwt7YC"
      },
      "execution_count": 37,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def weighted_avg_overlap(samples, log_probs):\n",
        "    scores = {}\n",
        "    for index_candidate, candidate in enumerate(samples):    \n",
        "        overlap, weight_sum = 0.0, 0.0\n",
        "        for index_sample, (sample, logp) in enumerate(zip(samples, log_probs)):           \n",
        "            if index_candidate == index_sample:\n",
        "                continue\n",
        "            sample_p = float(np.exp(logp))\n",
        "            weight_sum += sample_p\n",
        "            sample_overlap = rouge1_similarity(candidate, sample)\n",
        "            overlap += sample_p * sample_overlap\n",
        "        score = overlap / weight_sum\n",
        "        scores[index_candidate] = score\n",
        "    return scores"
      ],
      "metadata": {
        "id": "7KxiaRENt-C_"
      },
      "execution_count": 38,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def mbr_decode(sentence, n_samples, model=None, temperature=0.6):\n",
        "    samples, log_probs = generate_samples(sentence, n_samples, model, temperature)\n",
        "    scores = weighted_avg_overlap(samples, log_probs)\n",
        "    max_score_key = max(scores, key=scores.get)\n",
        "    return Counter({sample:score for sample, score in zip(samples, scores.values())})"
      ],
      "metadata": {
        "id": "ny5EYLFWt_5u"
      },
      "execution_count": 43,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "sentence = \"I love you.\"\n",
        "translated_sentence = mbr_decode(sentence, 2, transformer, 1)\n",
        "print(\"English: \", sentence)\n",
        "print(\"Bengali: \", translated_sentence)"
      ],
      "metadata": {
        "id": "lzrxKRMAaOgr",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c76b0843-1ee2-4959-f8fe-c110d85274b5"
      },
      "execution_count": 40,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "English:  I love you.\n",
            "Bengali:  {'হাত তুই দারুণ ভালোবাসি': 0.5909090909090909, 'তুমি প্রেমকে ভালোবাসি,': 0.5909090909090909}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "sentence = \"How are you.\"\n",
        "translated_sentence = mbr_decode(sentence, 2, transformer, 1)\n",
        "print(\"English: \", sentence)\n",
        "print(\"Bengali: \", translated_sentence)"
      ],
      "metadata": {
        "id": "3ACvLvfouZz0",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "3efdb6aa-5a3e-4785-98b8-470eb24ca31b"
      },
      "execution_count": 41,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "English:  How are you.\n",
            "Bengali:  {'কী করে চলেছে ?': 0.35714285714285715, 'তুমি কেমন আছো।': 0.35714285714285715}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "sentence = \"How are you.\"\n",
        "translated_sentence = mbr_decode(sentence, 10, transformer)\n",
        "print(\"English: \", sentence)\n",
        "print(\"Bengali: \", translated_sentence)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hFnTZVktqKjJ",
        "outputId": "48a2f41b-eb3a-48e1-fbdb-438b67aa75bb"
      },
      "execution_count": 45,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "English:  How are you.\n",
            "Bengali:  Counter({'তুমি কেমন আছো।': 0.7130035458487005, 'তুমি কেমন আছ?': 0.7113599563547085, '“আপনি কেমন আছেন?': 0.6936990880699907, 'কেমন আছেন?': 0.6437417165283681, 'আপনি কিভাবে করছেন?': 0.6025111974667763, 'তুমি কি করছ?': 0.5923827124066038, 'কেমন করে তুমি কিভাবে করছ?': 0.5709978036648595, 'আপনি কীভাবে আছেন।': 0.5700764036634165, 'তুমি কিভাবে দেখেছ,': 0.5347892835672373, 'তুমি কিভাবে তুমি কিভাবে করেছ.': 0.4309773602943997})\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "sentence = \"I am tired.\"\n",
        "translated_sentence = mbr_decode(sentence, 10, transformer)\n",
        "print(\"English: \", sentence)\n",
        "print(\"Bengali: \", translated_sentence)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QlRYyoz_qXcQ",
        "outputId": "b52ac850-ff56-4e78-d37f-5cff7004eb53"
      },
      "execution_count": 50,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "English:  I am tired.\n",
            "Bengali:  Counter({'আমি ক্লান্ত।': 0.9384218769018613, 'আমি ক্লান্ত বোধ করি।': 0.7442534975669592, 'আমি ক্লান্ত হয়ে যাই।': 0.7428238286394828, 'আমি ক্লান্ত হয়ে পড়ি।': 0.7301237397742102, 'আমি ক্লান্ত হয়ে গেছি।': 0.722800046086654, 'আমি ক্লান্ত, খেয়ে ফেলি।': 0.6866941022561225})\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "sentence = \"Thank you everyone, it was a nice experience.\"\n",
        "translated_sentence = mbr_decode(sentence, 10, transformer)\n",
        "print(\"English: \", sentence)\n",
        "print(\"Bengali: \", translated_sentence)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "EqW3eaYJqpyB",
        "outputId": "73c1f04a-1eab-40fb-90a9-03504ae97f7f"
      },
      "execution_count": 51,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "English:  Thank you everyone, it was a nice experience.\n",
            "Bengali:  Counter({'সবাইকে ধন্যবাদ জানাই অনেক।': 0.8207671217955903, 'সবাইকে ধন্যবাদ জানাতে পেরে।': 0.764479880611803, 'সবাইকে ধন্যবাদ জানাই অনেক বড় অভিজ্ঞতা।': 0.7054436117723191, 'আপনাদের প্রত্যেককে ধন্যবাদ জানাতে পেরে কৃতজ্ঞ।': 0.6540915006547434, 'সবাইকে ধন্যবাদ জানাই, এই অভিজ্ঞতাটা অসাধারণ।': 0.6528625496217983, 'আপনাদের সকলকে ধন্যবাদ জানিয়ে দিয়েছেন এক অভিজ্ঞতা।': 0.6428596250117776, 'আপনাদের সকলকে ধন্যবাদ জানিয়েছেন যে, অনেক বড় অভিজ্ঞতা।': 0.6381294291391924, 'যাঁরা সত্যিই কৃতজ্ঞ ছিলেন, আপনি অনেক ভাল বন্ধু।': 0.6011730620736204, 'সবার কাছ থেকে ভালো একটা অভিজ্ঞতা ছিল।': 0.5360537247825337})\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        ""
      ],
      "metadata": {
        "id": "ia72Kt_Qu50s"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}