Tutorial_4_Flax_Zero2Hero_Colab.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[<img src=\"https://deepnote.com/buttons/launch-in-deepnote-small.svg\">](https://deepnote.com/launch?url=https%3A%2F%2Fgithub.com%2Fgordicaleksa%2Fget-started-with-JAX%2Fblob%2Fmain%2FTutorial_4_Flax_Zero2Hero_Colab.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/gordicaleksa/get-started-with-JAX/blob/main/Tutorial_4_Flax_Zero2Hero_Colab.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TbMr3-5oun69"
   },
   "source": [
    "# Flax: From Zero to Hero!\n",
    "\n",
    "This notebook heavily relies on the [official Flax docs](https://flax.readthedocs.io/en/latest/) and [examples](https://github.com/google/flax/blob/main/examples/)  + some additional code/modifications, comments/notes, etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "C1qve53yeof5"
   },
   "source": [
    "### Enter Flax - the basics ❤️\n",
    "\n",
    "Before you jump into the Flax world I strongly recommend you check out my JAX tutorials, as I won't be covering the details of JAX here.\n",
    "\n",
    "* (Tutorial 1) ML with JAX: From Zero to Hero ([video](https://www.youtube.com/watch?v=SstuvS-tVc0), [notebook](https://github.com/gordicaleksa/get-started-with-JAX/blob/main/Tutorial_1_JAX_Zero2Hero_Colab.ipynb))\n",
    "* (Tutorial 2) ML with JAX: from Hero to Hero Pro+ ([video](https://www.youtube.com/watch?v=CQQaifxuFcs), [notebook](https://github.com/gordicaleksa/get-started-with-JAX/blob/main/Tutorial_2_JAX_HeroPro%2B_Colab.ipynb))\n",
    "* (Tutorial 3) ML with JAX: Coding a Neural Network from Scratch in Pure JAX ([video](https://www.youtube.com/watch?v=6_PqUPxRmjY), [notebook](https://github.com/gordicaleksa/get-started-with-JAX/blob/main/Tutorial_3_JAX_Neural_Network_from_Scratch_Colab.ipynb))\n",
    "\n",
    "That out of the way - let's start with the basics!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "GHcasJkggdZN"
   },
   "outputs": [],
   "source": [
    "# Install Flax and JAX\n",
    "!pip install --upgrade -q \"jax[cuda11_cudnn805]\" -f https://storage.googleapis.com/jax-releases/jax_releases.html\n",
    "!pip install --upgrade -q git+https://github.com/google/flax.git\n",
    "!pip install --upgrade -q git+https://github.com/deepmind/dm-haiku  # Haiku is here just for comparison purposes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "HmVx7EjigrEZ"
   },
   "outputs": [],
   "source": [
    "import jax\n",
    "from jax import lax, random, numpy as jnp\n",
    "\n",
    "# NN lib built on top of JAX developed by Google Research (Brain team)\n",
    "# Flax was \"designed for flexibility\" hence the name (Flexibility + JAX -> Flax)\n",
    "import flax\n",
    "from flax.core import freeze, unfreeze\n",
    "from flax import linen as nn  # nn notation also used in PyTorch and in Flax's older API\n",
    "from flax.training import train_state  # a useful dataclass to keep train state\n",
    "\n",
    "# DeepMind's NN JAX lib - just for comparison purposes, we're not learning Haiku here\n",
    "import haiku as hk \n",
    "\n",
    "# JAX optimizers - a separate lib developed by DeepMind\n",
    "import optax\n",
    "\n",
    "# Flax doesn't have its own data loading functions - we'll be using PyTorch dataloaders\n",
    "from torchvision.datasets import MNIST\n",
    "from torch.utils.data import DataLoader\n",
    "\n",
    "# Python libs\n",
    "import functools  # useful utilities for functional programs\n",
    "from typing import Any, Callable, Sequence, Optional\n",
    "\n",
    "# Other important 3rd party libs\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "aSDyQLgOesZp"
   },
   "source": [
    "The goal of this notebook is to get you started with Flax!\n",
    "\n",
    "I'll only cover the most essential parts of Flax (and Optax) - just as much as needed to get you started with training NNs!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "y1kdq0P_g7LU"
   },
   "outputs": [],
   "source": [
    "# Let's start with the simplest model possible: a single feed-forward layer (linear regression)\n",
    "model = nn.Dense(features=5)\n",
    "\n",
    "# All of the Flax NN layers inherit from the Module class (similarly to PyTorch)\n",
    "print(nn.Dense.__bases__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ux9Okie5PWpw"
   },
   "source": [
    "So how can we do inference with this simple model? 2 steps: init and apply!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "QViTvJhFite2"
   },
   "outputs": [],
   "source": [
    "# Step 1: init\n",
    "seed = 23\n",
    "key1, key2 = random.split(random.PRNGKey(seed))\n",
    "x = random.normal(key1, (10,))  # create a dummy input, a 10-dimensional random vector\n",
    "\n",
    "# Initialization call - this gives us the actual model weights \n",
    "# (remember JAX handles state externally!)\n",
    "y, params = model.init_with_output(key2, x)  \n",
    "print(y)\n",
    "print(jax.tree_map(lambda x: x.shape, params))\n",
    "\n",
    "# Note1: automatic shape inference\n",
    "# Note2: immutable structure (hence FrozenDict)\n",
    "# Note3: init_with_output if you care, for whatever reason, about the output here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "b3yFAqeTjdLj"
   },
   "outputs": [],
   "source": [
    "# Step 2: apply\n",
    "y = model.apply(params, x)  # this is how you run prediction in Flax, state is external!\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "31O_mx-Smalq"
   },
   "outputs": [],
   "source": [
    "try:\n",
    "    y = model(x)  # this doesn't work anymore (bye bye PyTorch syntax)\n",
    "except Exception as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "fQYyv76sCJ25"
   },
   "outputs": [],
   "source": [
    "# todo: a small coding exercise - let's contrast Flax with Haiku"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cellView": "form",
    "id": "UWr3hdpmFBng"
   },
   "outputs": [],
   "source": [
    "#@title Haiku vs Flax solution\n",
    "model = hk.transform(lambda x: hk.Linear(output_size=5)(x))\n",
    "\n",
    "seed = 23\n",
    "key1, key2 = random.split(random.PRNGKey(seed))\n",
    "x = random.normal(key1, (10,))  # create a dummy input, a 10-dimensional random vector\n",
    "\n",
    "params = model.init(key2, x)\n",
    "out = model.apply(params, None, x)\n",
    "print(out)\n",
    "\n",
    "print(hk.Linear.__bases__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wWBxTShUiLzW"
   },
   "source": [
    "All of this might (initially!) be overwhelming if you're used to stateful, object-oriented paradigm.\n",
    "\n",
    "What Flax offers is high performance and flexibility (similarly to JAX).\n",
    "\n",
    "Here are some [benchmark numbers](https://github.com/huggingface/transformers/tree/master/examples/flax/text-classification) from the HuggingFace team.\n",
    "\n",
    "![image.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAfgAAAG+CAYAAAB/KRYOAAAgAElEQVR4nOzdf3RT55no+29vU2cyV4bi4IChNSm23PCjKXAJMNyCjYKda3w5xOEocaedzKqvXYNNMuTU467ROeAY7qirrnvjIcaEokvWpM3gRBcMhxrf2MbYpsOAYZE0DSSDbBroJIZATI11T07c6en9Y29Je+uHJduSLW2ez1qshbdkaevd7/s+7/u875a/kDxj1p8yzFkIIYQQIvH1u64AcJ//AREow5wl5TPJpMxjR8o2scj1EmOhnbD/T1N4HkIIIYSIEQnwQgghhAFJgBdCCCEMSAK8EEIIYUAS4IUQQggDkgAvhBBCGJAEeCGEEMKAJMALIYQQBiQBXgghhDAgCfBCCCGEAUmAF0IIIQwoMQN8hYOOrh6O2nOn+kyEEEKIMMrZ19VDx7FaLJP4rveFf8r4lDl6sGaGftx9ZjdP2tpj9fbxocJBh3WUv9Q3dBb7pio6gz3PPcDlkw5efKmdQc1rBS230R6L1Neexf6jZ1iSmkzSF4GRQVy/fIm/3dONO4JfT9n0Q+r+Opf0lCQARgavcfIf/4afHhsc3/nEUmohP6j9ax5PT/F+1usd/0hlbTPhzjbl8Rd4sXQ9mZ5yAhgZYbDvCD8qb+Rtgtd99433OHngv/DySfUdYn09E8UD32Pmd7by6NLpXH3563z4QfjnP/S9F8jKfIjp96vH/vg5n/3+Cu/+LJfb/wY8cZqC/2DW/dof3J9w7fRL/OsvX1WP/F8s2vuXPHz3V3T+3X/kM92zR3ssAZmWsam8nO89kcXAkbVs3Rvm+am5PLezhMe/noYpST32xxFGhq5wzF7O/gtAXi1HbaswaX9vZJDrZ47wct1rvO0GJagVYfb0c7o3Ge0xv9N/tpE3i7O45vwbtu6dg/3YDlZM1zzhjyMMXv8Xju2p5fW3hyMrkzi0Yscb2B+/n97av8J2IjqfI2YB/sbVK7j+Xfn//Q89THpKEu4bVxj4vXLszvVPY/XW8ePfruHydFj3z2Te11JIcg/g+jf14g19yB3N00cGr3Htk8+B+0kzz2Phph3sMQ3z3d1nY3+u5sVkPjDIpa4ObjywiJXLszA/tYOf/PFDtu69Nvrvrt7Bnhdymf3HYVwnW7jMIh7PySL/hX3cf+sZ7Gdif/qRW4Xt5RewzAb3B+20XoaFj+di3vACe+4fGKWsU9hkP8hzq1MAcN+4xrXffw4o9Xt26sPM0D1/hMHffsinnwN/noY5fTGbduwj+bN4K4+psp2v7tzKglnT+RIAQ+F/Zcn/w8ribzHzi0rA/t21If4I8OcPMSt1FiYT3NY8/Q+/v85HanuakZFOZv6PmPbA55x3/lMMPk88WkZZ4042fT3FNxgNI2VTLa88v4qULwLuAa5fHeZzgD+fybzZc5iX4vcL3v7sfh6cN4/0nBL+/iF4uvy1iCYGYZmexf7dxdDXxI/2XgLmqA8Mc+ODAYaB+x98mPSvZfO9Hz8IT5XzekRvXIjttb/mWw/8hjrrjlEHGJOld/dLdC6qxfLsdlac2E1vFF4zZgH+mL2EY+r/LfYWbKuTGDhdEn70aCTHdrPVUwjqiDfpRjdbtzQGffrIB6+x1TNjW/hDftFYwOzVRVg5izPW53pmN8Vtw95GaSpu5M1nF2P+RgEQ/Hy9ls5jNuC+0MjW3S0A9Jla+MHKNL66FIirgLaMr84Ghs6yf8tuWgH6kjlatYrZ85YBwQP8wsp9SnAfvMiru3dEMFP4nL5DJdjaPL//Bnv+9zS+ZS2CM03R+zgJKx3Tl+HGP7/Kna98j0fnhXl66v/N0tJvMZMh/vWf/o6+fz6ie/hSkF9x/+urXHptn+/3Xyxgzsq/xOT8p+gEn7j3ILNTv8Tg5WZOf74e6/Lk0Z++8IfseWEVKQzy9oFd1Lx+MXw5afszUxE/aSpn6cICyr72Gj/97cQ/wYoXCliYNMjpg41c1z0ywOktJewHIBnrS02ULV1M/pZ5vF4XZkICQBpfTU8haej+8E+dNGfZ0/Ye33o2m+8W19N7cOKz+Klbg/9aAT94+Q2Onuyho6uHjpMdvPnKC6zx5HxM2Tyne7yFfc8vC/5apgLsR3ro6GphT/GiSfsIMXX5IteHgAdM+A+ax2rpjjeUPQt1hZqjyZQ5lDL7ySbAPaxrzO4bbkaAkc8iSLGfcnEDMJmz2ZQKpBayxpwMDNB3aoInH3Xd9P0bMD2LNZtSgBQ2ZWdhAm580B3id55lS14ajFzj2H/ZPq404OV3r+EGkpInejWN4j/x/n/6Ou/+09/x2f8I/+wZRQXMAX73y/KA4B6RWxe4cxd4YDp/NvbfTlDt1FgL+O5zL+EaCf/s72zLZTYjXD/yn/nbSIK7P3cTfTcATDxoDvfkSCxj/aI0uPUbWkedJAzj7BsAwJSymude66Gjq4OfWLXPKeQnv+yho62R72130NFVhBlg+ipsurXxFCwvNPKLX6pxp6uHE0d+TnWRJq549oC9tJs9R5Tn7KtQHjItfRb7ay2c8MStthZ+8h392T5YVMtBz+v/8ufYNvgGXu6DPVz6LImFq/9Kv/wxTlMX4M3ZrEwfwdXVzLEjLbx9C1IeKeQHLxYCyXyndgebvvEgI5fbOXakmc7LfyB59oNBXmgRZfUvsCJlBJeziucPBhvLJ6IH+VISMDIy4dnG2784y3XAZP4L3wYP01+xJBO4dRHnMf3zU1Y+i/3ZZZgY5FxrBLPNyz9mS+1ZBlNW8Zyzhw7nC6xIGaS3toSfXp7gyUfdJX665cf0Dqaw4oWjdHQd5bmVKQye+TFb6kLUnQ2LSU+Ckb52Xh7v55mehHI57425Y3RtZcYc4N9/w+9aT47zNaZz35eAf//vSlpf+Clg0dwkGLnCyT3j7UPnYfozADfuaGy9+VouC2eD++rZsOnq9AeUIOn+7F859vY1IIlFq4t8T9iUjdkE7nfbefVfOjh25KKy3+aza5w+0syxX3bTRzL59oPYNi1m9h+v0XuimdYz1xiZPo81W/4e+wZ9BsS09C/4Utd2nsxR9zXk7eZgXQkr0u/H7TpL65EWej/6A/drf+2BZXzvu1m4z7XQ+9thMM3DUr6bTd4nNNF3DUhfrDk2fjFL0YfllxLmOPzi1QJmf2UxSxkkc3YSMMA7B+t5+e1h4CVMJv8UUxL59r/HmpnE4Jkf87d7DRLcU1fxHVsRSx+AkctnORb+N0b32yO8fb2Q9PRFrM+DzjYw/fUyzMDgpRZv41GWUtQy/uwarS/+DT/tiuD1F5bzk+2rSBkZ4O2Os1xnHivXL2PF9pco+7CE/XEV5H0DwhsX2jl3HdJX57J09Qv8pOJDtgarQ19LwQS4fz/gOxawyWiYXnuBNyWvlbLyWf7u28tIYoTLZ5uj/5EMz8wD04C7Q/x377GtPPyjahZN8z3rzrkaznhS8loP/CUzvvs9Mh+Azz/4lW7fi/CYx4zpwJAbXy3PDdjQFnLjp2ke+eX/J49/BbjxHh0XonBKCx7EBAxcbxnlScmkbyjnxbw0YIDLJy9y/d1uLuc/y8JHVmGlCSdgzVmMiUFOH2mGc/DyuRQWPrWMlJEBTu95SVmD/9oLvLg6BdwX2f+d7TjV4LRfXa5c8dRfYTrR6ItZ/9bOj/Z4Mh3zeO672aR8cQSX82+C9yMASW7OvfgkNWcALqrl+zBLcuCY2te6BofhkQeZtxyYYDlOXYDHzONVz/K/LX6Yeakmkh5Qt2s+kMwMumk9fY2Vm+ZheamFFTfe4/SbDvYfuah7haRHSnkuJYWRviZetLUk/LqaafUOOrp2+A7c6ublKnWzyvAIIwBfTAr+y6O6xqtd75H/7GIyV6+Ctkt8Z2kWcI3T/+hbc+77VTPHbjxImnkxmZnzyH/xTZa2/Zjv2pUGrRsAAPQ1sb6kkU3FBZiTRrj8Wgl/q64bvXpbaRT5xYXsr4yjoLaplPzMJEYuv8aWSodStgc/Zc+RZ1mYX8qmvdsDB1Rq2Sc9oMkgDX7I5Q9SmAEkfyWL2QH5tGRW2HrosGl+5fRL2F4b1r3m+K7nvWaI//HvwJf+DN9esZu4+118+D8D09N5eFbgWuqMldUUrKz2/vzHT3/FP++r0b+mULn5wwiQlISvln/KtQ+uKIH/y2mYZwdZw88soqNLM1P+7ArOXZ4NYuprAuNa6VYH1sFlYe3qwZeFH8Hl3KVuYHVw8t1CFi5fzAorOJ1FrHgkKXyq/4lFpAPuS+3e4A7gPvge155djDn1YVaAd0Oe+/pFzb6AbMyzgc/eo3O0iebQFU57z6Gda7d2sGL6l7hf0w10fjSAjTRMUVjNm6IAP4+yulqsj8Dgb7ppvXCNvjNpWOsKSFef0fvSX/HdM0V877uFrFm4mPzn61m5/McUawL5yNBtRlJSSEl5mHkmuJzgEd6zi9790SX6PvgXnM6zvtu21DVx09zFpNOi23CSn54GwMBHoW+pcr95kb6ixSxcVMAK0zKWZMLI5W5e1WyEuX7CwcueH0xF/MRZztKcQr5jb+d1PAMATSMfUAZcs7+cDFzDddy3Nu0+7uLGs4tJ/3La+AojVr5iwgRcdx3xDQjdR3DdeJaF6SZmB/ud0y5uFC8m3bwKq6lJafwXGrGpo+vgt4R6dtG7+d3lK7jONeE8p8lbTvB63ltOcufGVvjKN5i9PJ2+C9eBI9x2HFF2zT9xmof/Q+Cir2cX/ec3f8Pvf/srbp/T7p7/hM/+GzAtnelfhs9+r3noG+nKHRGfXk/8W+Qi1o3rRgkL082ssCbjdA4DF9n/wxLl4QoHHdYgAV7dRf+HQReuD97j2JEWrnsb1gDuz4Dp81j4NWjVbrrb8DBpALc+HOcOdnUX/R8G6XO9x+WjzbRe9/U/x45c5DvLs9U0/SoWPQDX2xxR2ZkenElZUh2a+JJqNE3RGnwBSx5JgqGLvPLcbl7e8xqtKQ9qNpMlk5KazOC5Jn763DM8WfQarhFIWV6gX5cYeIMXnVcYSVnFD16rJT8auxKm0MgHr7F1Swl/u/sl9muDO8CZD5XU2Vf+gi3ataDUIiyLkoFBbrw9you7HZy+NAKpWeS/sAwzI/RdeCN8ZUxK8o6ir59w8PKel3z/nOrs/98B0jBv9J2XaaNZCZbxNktSF2Bnm5/yzQ5MTymj71B+6+Dk5REwLeN79S+wNKJ6puyi37plO/Y9jfrgDhO/nveUX3HjV7/hc+7n63/5Og/NSQ//K6i76P9hDX1N5X7BHaAF96cA6cz/D1s1x7/B9P/1MaYDtz+ORp45UVzj1Y73GCGZpaUv8dzSMDvuPW50s3VLCc/bfszLr2mDO8BZrt0CSGNlaYFmNp6CNXcRJmDw3/RZWZ1PlEFwcMou+q3PVfHTPa/pgjsAZ5p55wYkmVdRvW4xSSPvcfpAmN31F64pm4UfydbFElNRFvOAkY+ujDJAOMvvbgDTF2EJNhAaA8vcNOAPfB7BxshwpmgGr6Zupi/ju7tfYNHwPFauWowvS7GKysbtzHb18PaNEZLSVzEvCbgxgP9y7uW9L/Dy3J/zg9WreK6+nGsljQHPMQT3z/mvZ/L4weoUVlS18GbRFT79YzJp6WmYvggjH7Txapjb0Zxd7/GdpctYufpBcL/HSc1tGGWvtLDmfheX377G57PNLFq0mPQHwH2hg9fDnNrr//Usjz+yioXfaWLf3A7vffBJDNL7X38+8c8eTb9opjfvh6xY+Cy/eCWNk5774JNg8ExziM86zOtVL5H56g9Zk1nIT44VMPjRh3z63zzf8TCO84jC9bynnP4/+Od5vyT7L8w89p97+fzT6wx8+jn8+UOkpU0P//sBrnP7rV9xZ8u3mLGymryvf4+PPvmcP/+qmYceAD530X/iXrlfXuF+7Ye8bP45P1iTxaaXWsj3fC/Hn89k3tzxVPJhXj9ylserVpGy+occPVKE6xNI/so8ZUlr5Aon/3GU7/i4MMAgkJZeAIy2Dh/MRX5x7hqWTctYsxBGLl/Upd1hUM0uLKO47oesIYnTlfW0/iab731jFT94/ees+eeLfPrlZaxZOY+kPw7Q+QvHKBOii+w58R7fKl7M0opm3lzbzWkXpC//X7j/zJM8/0rkZ25OSYaR9+iLZP9TGFM0g3+NVw5e5MZIEulrCtm09kEuO1rwja8+5dqtPzB7ZQGbniokf+mDuH/bzf5duwmc1AzTaj9A7yAkZRbxor0gKrcXxJ9hWm3F2I9dYfAzSEnPwvy1NEwjg1zvaqRyi/99okEca+HyECQ9kIT73RbdWvONG25Mc5dheaqQ/NWLSf/iAJeP7aa4sinsLN99oootu5txDd2P+fFCNj2eRdLQNVpfKo7aNzJFjbsFW9lujn0wSJI5l01P5WJ+YJDrJ15iy2j7ONwt1FifxH7sPW641fJ/JIv06eC+cYXepnrqgmywCy0K1/Oecp3PfvEo/+8/HKHv5hB8OZ2Hs8w8/JXp/I//7xOunvsn3nYG2WA3mt/8R878wxGufvq59/Ue/NLn3P7tSX71ozXc/n34lzCWYVp3PMnTu5u5fGMYps/D/EgW5vQUJRV/pok9L41t2cjbNwyOQMo8zI/MY/YDIwz+tpv920vYP9q98r9VbhU2zVvG0nF8musHurk8onyuS2/5B+cmXnW+x+Afk5i9vIA1D41wh2Fef+5v2N91jcEH5rFiQyH5q+fBrfc49mJJ2C+ocr9WTuUr3VwfhJRvKH3LounD3Lg6lrMuInMecMPF6bF93KC+kDxj1p8yzFn0u65E4eWMScpn8kmZx46UbWK5l6/XihePYs+B0zbPzvMxMJUom2eHurFZd8Rw/T16lC8Yy6LvtUKeH+cX3WjrS2L+sRkhhBCG11t3hMsjKawsLieynRc+6X+9iswkuPFuS0IEd1jF83mLSRr8F45G4VvsQAK8EEKIeOV+jbojVyDzKf6uIrJvKV3z/G7s9p+z56kskobO8vpLk/C3PKJgxY4XsMwepNdRG7Xvxp/C++CFEEKI0V1/pYQNY9ik9qXZy1ixOhkG3+P1/1JFazzdtzaK3t3PsH53dF9TArwQQgjD6LQVxMVfh4sHkqIXQgghDEgCvBBCCGFAEuCFEEIIA5IAL4QQQhiQBHghhBDCgCTACyGEEAYkAV4IIYQwIO930QshhBAi8Xm+i/4+/wMi0L38xx6mipR57EjZJha5XmIstBN2SdELIYQQBiQBXgghhDAgCfBCCCGEAUmAF0IIIQxIArwQQghhQBLghRBCCAOSAC+EEEIYkAR4IYQQwoAkwAshhBAGJAFeCCGEMCAJ8EIIIYQBSYAXQgghDOi+8E+JjMXegm11su6Y+8xunrS1R/waZY4erDSxvqQxzDNzsR/bwQrOYt9URec4zndK5NVy1JZCa04J+2P+ZuXs6yrC3OdXnhUOOqzavx54BeeknM9UKWdfVwF37AXY2mLx+mpdnO47Eqrelzl6sGZ6fhqmN2bnFEP+9Wcolm3Qv2yNXldjT+mniW3dy6vlqG0VJs/P/n2QmDRRCPCaQJKjuYh5tRx92gxEHuAjVvEMKxjGPX0R6/OgM947SV2nODl/FcpiL8AccDQX+9JB7DlrvR1ymaMH67FaXIk0UIqErpMZpjdW71PxDDPa1rJ+r/Z9d7Cvop2tnmOeQHXLr40kIMvcQZw5a9Ugq3wu27FaiHr90ZTZJqXMDFtXJ0051tXJwHDs3qLCQYc1jV772sQbvBrQBFP0udiPFZF2ZnfgCK2tiidjNGorW5qF+1I9rX3JrHi6PCbvET3l7LNm4XKuZb1zkv7kY14tz68ewNXn/0A7thJ957j/zbPegZJx5GKvWAVndrPefhZ3LN9qb4kmkANtVbT2gXmpr15a7NuVQGWAWUynrUozg27H1nYFpqcEGUxOUMUzrJh+BaemzPaXNOGavgprRbTf7N5Q5igire9KDNtDOfusaYmZmTKoic3gPY0wojR8YCrT5Vyr7xwjUs6SzGEuv9nOfnLJty2jDOI4bdfIVs+sLUTH5FmacFLkTeEqaV50ZRbZkocS3Aaca3lnac84Ol7/65SIqeR2bJvUcsrLDfqM6Jb5aJRZk8s5enDXp+/H2zbiQF4tR22LuOy8xEKrJ4Oiptb9Mllh0+1Dg7h0Bxp5p6+I/Lm5KJlBI9TVSVLhwJp6FvteeN6WpnsoWm3BYi/APHQJ+2jl77/EI+n7mJrQDL5saRb0XYwguJazr8uTolyr/LOfJc3aw1F78A44FE8l6mgD2tq5PJRF/hhfIy5lFrHkbbVsnFcwrd5BR9cOJf3rPbYde5iZdplDKedIg4MlZxEmT3lqft9znexnBib2ueJZlMpcr5wlmeB6W+208h4mjSu846rlaFcPHeo/bb232FuUzjfH976JQclOuc+84dcHJLMiD/bkrGV9zm56h7KwdvXQkTeofkb1mCNM9i0gM5DLvFQwPaQcvafq6kTk1XLUmkbv3lGWNqLQFswPJeO+1I7Z4avnHV0OyvzPw66+Zk6T3wBORNuEd9G7Pwl/iZSgfBa7dqTWVsWeM8OYFuViifjdclm/SKlESkVtp+PSWF8jTvVpgvLeN+gdCnYsmYU5oQcz3kAR6Yi4woFtdTKuNk/DVzpQ7TXttJUYd0YUhTLXU5aszENncXpew5yCiSysFZ6ApwxuWb2DfWpGx/xQMtz60Nf5+qf940ou9mOezruAO/a1QWZ1w5pgoqbxgx3LXObr/P3tvYgL/SDAYt+uma3fY3V13MrZZ1vFgDNMZmPCbUEdfK3e4Rso5KzF2ZeF9Vit0j+bUzAxwDXveTSyVWbvMTXhAO8ZTY/GM7LzHz12dl3CPZb1u7xcFk6/QqumQ1Few2hryIpIBk9eFQ5sqwdwRrgBqczR490M4wsmSserjN4doTtfAxtTmWvl1XLUk6UKuAbD+tmTZ50+T+n49r95FndmER1dLWPMFkyFdmybPB14PVT0RFhXtB17JBrZmtOEK7PIOxt8nnqcfZ5rJHU1PN8eqfEMGMfVFvr02UNlj4+6b2LvG95szlgzt2J8JhTgXZ8MQ+rDkzZ7Lnt6lTIb0qQ6O2yrMJEIm+1iKRd7Xhb4lY01EwgIHMoMTEkJBxnV7y1R051p6mtJ5xlWhYMOm7LvIWA90TUYdFOT6xPNTua2Kp7MWct65wArbD0JEuhBCfZNuMhiSUw2vjWyNcc3G3zSBvNSYeAjdYAvdXV0Fc+wYjrqIMivv7T10BFuiWRM2rl2K8jhtg8Z0DxHGRw2MaCekwT62JpQgFdmz+F3tbo+CZ5GV9Z//TfShKKubTp9Dd639jY8errP8LSzKm16DGUTizeQa289Gn2m32krUNfIYtV5G4TmtqCgs6S2DxkgmRl+aaqAtDyoAWv3GJcF7iHqpt53/MpZ6moI6gBI989+FjfDyjp4lNPjQSd8eQ+TxjB3dJ28MnCzj3mJVozVxFL0bVU86byCOdhmubxajqojxE5bC67pq7BpR4x5tTyvW/8No2IZZgIbt/f1pXGHpy5xOEM27FzsjlppcGNQtjQL95n6UdY3G3GeGcZs1cwuKxxYM4fpfdNzf3eizDzL2ec36ytzFIVslxOTi92u7y+OWrNwOUt89+BLXY0rnn7+eW8sUO7oMfW1KO2jwuHddyImx8S/6GZvCev3KrvkO7p2aB4Yptdepf6/ka05sK+riI6uIs3jQb4MIVP7HM/z6iEvC/qaQuzYV26fsebVYhltp+i9zrPhq6sHq/9jfU2sL3FB6ipsXT3Y1MMJe7vWpFA3FmX6133Q3rLVaSsAews2b7n7386lvyYTvzUvVlzcSd2hb59DZ7HnxKjNLQrTX0hdjTOefl7THvxugzNbe+jwVPSYfguiAPhC8oxZf8owZ9HvSpRbcyaflM/kkzKPHSnbxCLXS4yFtr7IH5sRQgghDEgCvBBCCGFAEuCFEEIIA5IAL4QQQhiQBHghhBDCgCTACyGEEAYkAV4IIYQwIAnwQgghhAFJgBdCCCEMSAK8EEIIYUAS4IUQQggD8n4XvRBCCCESn+e76L1/Te7jjz6aspOJd3PmzpXymWRS5rEjZZtY5HqJsZgzd673/5KiF0IIIQxIArwQQghhQBLghRBCCAOSAC+EEEIYkAR4IYQQwoAkwAshhBAGJAFeCCGEMCAJ8EIIIYQBSYAXQgghDEgCvBBCCGFAEuCFEEIIA5IAL4QQQhjQfeGfEk4J9Sc2k6E9dPUwG7c5vD9mVzupXGnSPMHNhTorNeziUOVjaB/R8XudQOuobqpi+TTNK5+r5ds1p8b8KWIn8BwB+pvz2X4gdu+aXe2kMvWtwPIrbeB4ofZq9dO8YRsHY3cqU8ivbt49T13RTrqj+h6R1cHihlYK53t+SuAyt/i3WbUtd/qeov+sRNCOx8u/7wk8l4D6HpM6EM8C6+fofY9apjG6ZgGxQPc+wftKj1j3mUY0sQCvNvabzfls9BZ8CfUNvqcUN7RSOPM8dRt8jSq7uoEcgM6dfLtT83snNsNYLmLpZmac1Ly3ZReHKquoLz0VdxVhsiqnrnO96v/oOqq/OUjdhm3ea1Hc0Eph0y76jdbpBa2bMRBBHfRvA4lc5sVPpdC+Id87OMmudlJZ2UBxpzpgsezi0Vu1bNzmGeAo7fpQdX/0B96WdO5orq9yLk6q8QX57DmDmsGUEkAqm3ZBApb9+GTA+7Vs9JR9aQPHC1upJ3h/lF39hH6yFvWzeV8TC5S6cbwBNcifoqYosI5kVzupXPA+zXHWpyeCCaXos9cswHT1sF9FcbDdOyIr4dH5bi78TN+Yumu26UfZ43Vgm/69O3fSfhUyvlkShRePlgxmhBiRRl1pA4Xz+2nekE9zQHAHOEXNNv21OHjkPO5pC8ixTNI5Top1VH9fCe4xH1SFrYMlPDof+k/6yj2Ry/zgNn3mobvmLfrJ4NFS9UDnTrbrArmD5nNuTAvWkR3tk8K6Kd4AACAASURBVOncSY2m7JVzMTEjQ3tsp+Z8T1Fzsh+mpcQ0iMUXBzXa63HgMBfuwqw56wKfatnFlpU36Q/ad0THwRpt/6PUDWamj1I3SihcadK1HxG5iafo5y+hGEZJN5p4ZM066IyntPlkc3OnP/SjxQ2tFHKYZjZ7Z99Kmhddyirs8sOBbd7ZTPG4z9U/TRYk7RnvSjezfFr/qCP+qJV5lPintQ2djixt4HghNDdDoSd9rqbO0aZwJymdfk+VfUi+QfG732z1GwApfcIj7x/mgwWbvW1DKSf9Mkm0yy67+gky7p6nTvua/stE99yyS+QmNIPvrmnkwt0MCk84qQ46G3Gwvbkf08oqjjdMxqxanS39OhbrfeNkSWcWJpZXtnL8hPKvvjTI8+Zv5tFf57NxQz4bPWV2okpJ/3qPlYco5/HLXrMA09336VIDeHFDFctvH1bec0M+deduRvcNJ0H2nFlw9R36q53eMj8erI7GpMz966AyS8kobPAOuoqfekxX5tnVTjWF73vfhFG6hAz6eTdkp76OnAUm3O+f8uuAMyj85jtqPTtM/7THqDzRquwb0RzbUh1kphlCccNmMu6eH2VgV0J9YQbuc4d1SwwJW/bjkF1dzvJp/bQH7BFR2v1owdm08gn4madfcJNR2MrxE09wp057rCHyyYVlF1tGnZ0Hm72XUO9ZetuQz8YNtVy4Hekb3nsmuIv+FDVF+dSdQw1gQTrRA9uUxjp/s9LRxizQr6O6KVwDnwKdO/m2p/NQO5CMwiBBXrvUoabRAo+p2ZBoKW2gUteA1vHVmeC+5evkoracMokyUk0wfzNbaNQMVGB5pV/nE/UyD14Hu2us1J2bRaE62CicqZ9xZKSa4PZ1Xyfmn/aPV5ZdHPILmP6KG6qCBhTop3mbfhAU7Fi41H62ZhCXe6uWjQEzuXVUN3kGeUow0mZkErbsx6K0wVtGygBKv8ziHeSE2VTnPtfo7QuU5ZBgxzTLNUGVUO8ZdFcq+zlClnfpksD+3JLOLF1GNHDZUfhE5Ta57hqrphNt5VDAqNvBds+ofP5mjp8YwyhP10BDzIAtuzh0Qp15xnuq5sA2ZaT7+K6wa5LaQBttxQ2tHC+cxYU6bQNT1iiVmexYrlEcunueVzQduafzyQ0zIxx3mYesg0r91Q42Np5MoVIzGD545Dzu+ZuDD5DjVHa1k+PqTCr4EobSkSuz4wjvGLg7yFhL39P3bNyQzyuUB+kflEmI8pxG+H6rrm4nYtmP2YFtvrr36yUcP6Hpo0sbqFx5k+Zx9ps3Px7r8pUnFuSzccM7PHqileNNwfrCdVQ/nhGY+encSftVNSM6KVnhxBbV++C7a6xqWnNziODgYPuGWi7cDd/R+mgbqPJPN+IrbfB2NLG5FcdolICjdLxB1tbVzsA340y8QN9/yx3sKHfuxugNR6uDpZtZjn6wwYFtNF81sfwptYPyZHmab4bOhMWR4oZWJSiEmn1ZdnHoxGZmnQs2o46d7horzaNusj1FTdHhgE2BiVT2E6ZOMJTMiBJEIcObXTp+Qt2PMCmDHgfb60JsOLWs45Fpbj44HTiAOLhNTc3PjHVWOPFF/4tu+gdxM4uvhqwYp/jdbTClRmEfa2lDkFlo/AtIC04adQNdBJkOZWZ0OIKUW/zp/vhmkJ3Syt0MY59xhBGmDmbPmRX5ax3YpnRc0V6KiSLfLX8hZuWa2xPj6/sowkiAso++wMnTRs8dOFcPszHYBGCS+O8NCqSee3O/d6O3CDSx2+SqGwJGePoNRCXU+4+uLLvIjdJGuOJvZujWgOJSaYM+ZVjaQOFUbQS0rOORadp1Tn/rqG4Iv3QQ9w4cVjZ/aupe+A1Y4xOuDnbXvBW4WcyvDhQ3JEqWJPhtr1rBb52NjYByC2hbgf1PccNm3abAxCn78Qnoo9WNbYGbHieBZRf1usytsnM/MJCH2pgZ7DXEaCZ4m9wsJa1VqTl09TAbizQBZP5mjp/YrHmCmwt1+VEIysqGMNN8ZeezXnzd2pVR2MrxQs9P0fr84zmRFExqOq7Q/7Grh9m4rR9mqruZ1cOJecvQKWqKlNvdvHUvRt9iF74OOti+AepP6J+jL1f9NYm/b2NUqXeEZPi3efCWr2eDo77NK6Jdl/pvzaJSV5f921Y/d2ZW6c/lrv5LtxKm7Mep+2Oo9LteU9amO69DpV9bCfqNeaNn22at1L5GAn8r5CT4QvKMWX/KMGfx8UcfTfW5xK05c+dK+UwyKfPYkbJNLHK9xFjMmTuXftcVQP7YjBBCCGFIEuCFEEIIA5IAL4QQQhiQBHghhBDCgCTACyGEEAYkAV4IIYQwIAnwQgghhAFJgBdCCCEMSAK8EEIIYUAS4IUQQggDkgAvhBBCGJD3u+iFEEIIkfg830V/n/8BESjDnCXlM8mkzGNHyjaxyPUSY6GdsEuKXgghhDAgCfBCCCGEAUmAF0IIIQxIArwQQghhQBLghRBCCAOSAC+EEEIYkAR4IYQQwoAkwAshhBAGJAFeCCGEMCAJ8EIIIYQBSYAXQgghDEgCvBBCCGFA94V/ymjK2ddVhJkrOHNK2B/iWRZ7C7bVybica9m6VzlW5ujBmun3xKGz2DdV0Rnw+nra1wn2PlruM7t50tY+pk8VM3m1HLWl0DpKWU389Vdh8h4YptdegK1N+6Rc7Md2sGK65+fRr13iK2dfVwF3AsohBiK5vuo1Ip7q5VhEVMeipMJBh1X7ly6D1VW/PqKvifUljTE4mcQV0NfGrIz8+xZFsP5af04xrEP3uAkGeIBh3ENZLKkAggRdyGX9ouRgD/hVNKVy2I7Vgi7I+1WQCgcd1h72oa00aiPva2J9jqbi5tVy9GkzMMUdqa6jit1fhSp7OoXWnLXeDtBib8Fmc1DW5ukU1QZ4q4n1m5RyKnP0YD1Wi8uvzBOeLhAN0xvL9xrD9S17WjkndyzPJ4bC17FoycW+dBB7zlpvvQysq0q7Tzuzm/W2du/PR+2uxBw8xUJeLUs+2c36Ek95xL6MQk3AFJo+KEcGYrEWhRR9MjCMOa8WS7CHK55hBVdwDYV7nXZse8/inp4SMGPX2VuCsw/MS8vVA7nYj6mN3H9U2lbFk1M+mi9nnzULl3Mt652x/ZOP+0v0nWynrQUX6uALlGsx/QpOTZnsL2nCNX0V1goMJBd7hTJLXm8/G+NgOobrW+HAmhpJW4hfYetY1LRjK9EPOve/eRb39EWsz1N+ttgLMA+dZY83UDWy1XkF0+pnKIv26SSqtiq26gJ5I84zw5gW5QbvryfEzIzpoz/DYt+uBPcp75fvDVGYwcNAWwtYC1ifB51+aZaypVm4LzVxZ1EWM8K9kDlFk/oLzfXJMCx6GAvQ6QlacTtib2SrZ6QaohMsc/RgpQknRd60lbK0gC7lFZXlhqFBXH7n905fEflzc1EyHf5ptkRMn7Vj26SWU15u0GdEr8zDX19FOfusafTa66FiB2nBzkeTRh19FhTH8mo5alvEZeclFlo9GRQ1te6X6ZjY0pCSGXRfatdnnlyDuFnEvDygjcAlhYBlQBHd/meYO65Qj5VjXZ2Myzl6cDdMW4gDUdpk14jzDKx4ulx/OK+W/MwrtNpCXnENZSbkPvNG2EZvfigZbn1IJ8oAgr6Lib+GnFnEkrfXsj5HmQmaVu+go2sHM9q0x7ZjzxvDa1Ysw8wV3tE2joAMSS7zUsH0kHK0zOFJnynvaz8zMPHPFq9iUeZBebJM9UEHShZ7C9bUs9hzfO+bMILVMZJZkQd7ctayPmc3vUNZWLt66MgbVD+jesxRHupVg7LkLMI0dImONvDMFgc+8gs4bR8yQDIzzADl7LOtYsCplmvObnpvTejTGkCIgVE02kLew6SRzApbDx1dyr99Ff6PX+EdVy1Hu3zPOWr3DcITui3Eoajtou/suoQ7c5kuNWbJWQSjBezMIu9F7uhaxjs5a8OOEC32FqyZw/S+6RsFuj+JZAAR5/qafKPUvW/QOxTsWDILc4LPSAPk1XLUf8C09yIu9B2rxb5dM1tXgr22PDttJQk2ex+DaJd5CJ5BU6i6rR2wKu9bkhgzlmB1DIBhevd6Zsnt2NquBD/m11+MqsKhbNRtG8PsO+9h0nQzysC0/72mzLGDFdOv0OpfF6PRFtqqeNITmNXgbLZqgrw5BRNZWCs8g7+1rLefhdU7vM9J2LYQp6J3m1xbFa19WeR7R2PlWFfD5a5RAnafZ6bYFBB4tMxW32jPtnoAZ44+ZeyZfRrNeAcuFnsLHerM5Um/9betOU24NAOr56nH2ed5L6XjVUbvjntyHTPag0XvjGSUNcf9b57FnVlER1dLFLIFkyN0HQtlgGvjHCiWOXrosKbRax9jqratitY+dUY5xmyB8ZSzr6tHnR1HtjQy4bawtwT7Gf/9WdqBHuo1wvucRGwL8Syq98Hvf/MseDZvVCzD3NcS4eyvka32s7gzC4JeVJdTMyr0q5yuT4Yh9eEYbBhJTGUOzyAoVGfYyFbNKPtJG8xL1aQ695aoqfk0Ja16jwb66FDWHJm+Cps3U6WsaeoGUZ6Zj3NATW/Gd+cWvo5FSy72Y56g5L8PxMWdIUib6zej9Ju17y9RU/Op6qD2Xgz0ebUc9dxtMJX7D1yDQTe8uj4Z9v2QYG0h3kX3i27aqmi9tQprRS72vDRdGj2i3+1LZkVFiN34IXR2XcJtuF3g41PmGNsIHfDurH/Hr6PutBV4MyvR3yF9r9APprzrwEPKhiX/waoyuNodlWWBWBlXHRsX7S2dwYJSO9duBcnemVM06/S+59o2qeu5Y1kWMAJ1k2HkmZbo0qXcdfsjQjzHIwHaQiKI+jfZ7X/7Cua87Sy8FensXfO76i1bz9vHcEHbqnhSXes56v97ebUcvWdG7OUsyfRLfwXIxW7XlIe6hupyau6Td4xtgCUmrsyRKFmSSOpYlOTlstDvlk5/nnSubyOXesuiZ50+r5Z9Y+lLDMiSswiTdi09lioc+k11FQ6smeB623MNlVv0zFZNfa9w6PZUJU5bSAxRuU1OZ+8b9ObtYMbb47nPsRHnmQJsq7djz2uPfICwt4T1e8vZ16Xs/PQZptdeNY7zSEDqDlazrYcOm99j2luDFhXR0VWkPjBMr32tvpxTlXSy5yXkFpXJoOwyt6o/xdW3L2pFWseiwbMhS1MuXp4vyGqr4klqOWrroUN9kn99TVut7ROM/q2NgcwPJaubmYsCHotF2zZbfdciWP/SaSsAews273X1vw03QdpCgvhC8oxZf8owZ9HvktsRQpHymXxS5rEjZZtY5HqJsdDWF/ljM0IIIYQBSYAXQgghDEgCvBBCCGFAEuCFEEIIA5IAL4QQQhiQBHghhBDCgCTACyGEEAYkAV4IIYQwIAnwQgghhAFJgBdCCCEMSAK8EEIIYUDe76IXQgghROLzfBe996/JffzRR1N2MvFuzty5Uj6TTMo8dqRsE4tcLzEWc+bO9f5fUvRCCCGEAUmAF0IIIQxIArwQQghhQBLghRBCCAOSAC+EEEIYkAR4IYQQwoAkwAshhBAGJAFeCCGEMCAJ8EIIIYQBSYAXQgghDEgCvBBCCGFAEuCFEEIIA7ov/FPCKaH+xGYytIeuHmbjNgfFDa0Uzg/1e24u1Fmp6YTsaieVK01BHxtN4O/53jsuWXZxqDKF9g3bOBiL1y9t4Hih5krcPU9d0U66/Z6mL7fIyjpxlVB/4gnuTMZnDHV9/a8L/TTHqg7EmmUXhyofw9fqJqf+FDe0knurlm/XnAp/TvHcB0yBgH54kspHed/Auq4/nwRuCwlgYgFebVg3m/PZeMBzsIT6BuV/B7fley9cdrWTygXvBwSc4oZWCmeep26D73h2dQM5kZ6DLogpg41D1f3BO4Kpouvg+2P2NtlzBjWNZR3VTVVUNu2CIm3ZOqlceZPmDVYOen6ubKC402CNTNfpu7kQy/ca9fquo/qbg9Rt2Oa9BsUNrRQ27aI/yOAr3hU/lUL7Br92HbP6o9Th5dOUn9y3gjyltIHjhbO4UJdv4EHqBFh28eitWjZu8/SHk9RHWnaRG2Ry59/fJ3JbSAQTStFnr1mA6ephth/QHnWwPeLRYQmPzndz4Wf6i9tds22cjdVB8zk3ptSM8E+dNCXUF2bQ35zPxubYBXeA7pqdmk72FDUn+2Faiia7UkLhShP9zb7OuLumkQt3M8itXhfTc5tc66j+/mNwrpaNdedxx/S9wl3fU9Rs09fvg0fO4562gBxLTE8sJg5u0wfy7pq36CeDR0uj/17Z1eUs5zx1G2q5cDfYM0qoL5xl8AzUBHXuZLsukKt95IJ1ZMfsTZX2d/Oqf3so4dH50H/S1x4SuS0kgomn6OcvoRgmMHo38ciaddAZndFkRqop/JMmlYPtG9QBT4hOsLihlUIO08xmb+rKfa6Wb9egn8GcC5GijFTpEjLop1k3IDvF725XsTw1AziF/6wpMVP4p6gpUsvJEnzgEr0yD399I+GfRu1vzvcbOCcIyy4OVS7gg+b3eaTQk0FR07B+mY5wqdnuGqsaCNYFzehlVz9Bxt33qRutbvqn70MsW93Lot3/ZFeXs/z2YTb+egnHQy7RhjkfI7SFODChGbxn9ld4wkn1uEZgDrY392NaWcXxhpKJnIqitEFZ80nE9bf5m3n01/ls3KDMBE0rqzh+oooZJ7XHysdQzsrM0n3usC+dOmcW3B0MSCL333LDzHSygeKGKqVxblDet+7czah9xLgT9TKPTPaaBZjuvk+XGpiyq51q2tL3vglDHTS+q+uATSx/HF7ZkM/GDbVqH9HK8ccH1c+oHptgm89INeF+/xQZDa0cP+H510Cx9xkl1HuWED3ve3tCb2kA68hZoJSbbpATrbZQ2qAsAQbtg5XsQUah7xoVP/WYcdpCHJrgLvpT1BTlU3cOlle2cnw8gf7ANjZuOEz//M1KAx1ro5/2GJWexu0X0BKKdqnjwGElJRlwTM12hLSO6iZPR/cEd+ryxzjjX8dXZ4L7lq9RjX+5JAFEpczHqLSBypUmXZoyI9UEt6/7OtwD2xJjxmLZxaGgbU677KYuFQU7pmb/xkepq6aVVb7AtCGf5qsZFDbtUtLPlnRm4eaOtzoHLpfca4obqlg+rZ92/34hGm3BsotDhbO4UBc6M9NdY6Xu3CxlwHdCXY8vMkBbiFNRuU2uu8aqzvaUQH9ozOu5DrZvyNcEes0ovLRBMzpv5bin8Xrc1Yz2NuTTnloV+JwEpQ20kVEGXEpZNML3/Wc0Efz+Sc/ofSy/ZxxjL/PIFTe0ejeEaTutg0fO456/eXwD5CmSXe3kuDo7jmwQeZPfxWKg6LcHSFnTfYzCUqBzJ+1XTcrkIxoZwoRWQr0noEa4a31sbcGTLRltOU+ZgGyh0dtfbzyZQqWm3idiW4hnUb0PvrvGqqZyNo8zODjYrqbvvJu+DmzzVYYN+WwMs34mmzY8TlFTdFi3Aar745t+m+4UulGzWt6+Ufa9GeijS+nYlM41SAfYuZNvb8hnY/PN8WfCJlFxQ6t6J8ZUro2e4nfB0u2d19EuKh3cpqbmZ44zQ2gEll0cOrGZWedqw/af45Vd/QQZQEahZjJWmAEoyzOHqtdB6WaWc55XtAPCA9tovmpi+VPqdUmwthDvov9FN/2DuJnFV8d9UZSGG1874Q0i6LVRUp39v9avmSlZmcMx2yF971A3Ld4+HL5zPbBNXZ+O8rJAFPluc5r62yq1e0e8AtLy4M1sNU90WSABaW5ljuVtcZ4sru5fcz/KZkrlvbPnzIr8BROgLSSCid0mV90QMLry3zQxuhLq/UfU6v2T/gEnUmN7fyMJLMvihs36DVCelOX3fUsYnluRlJ3166huMMbyRtywrOORaaNv/CxuSJQsSfDbWqdKd81b9E97jC3eJUHl9izT1beULIllF/WGuv1z7ILfyjw1Aq8X6sZoX3+fOG0hMUzwNrlZShqlUnPo6mE2Fo0hOM/fzPETmzUH3GP70gp1k533FMb6/obRz52ZVfqyvKv/AiFQUpY0tPrKzP+2oZn68pRbVCYoIwWTmqYs9H/M+41i+scnfDtkrFjSmYWJDP82D1N0+5mD7Rug/oSy4xsI+Ja2WSs1j92D35qWkWoK0scqJr9tB7leAeeRIG0hQXwhecasP2WYs/j4o4+m+lzi1py5c6V8JpmUeexI2SYWuV5iLObMnUu/6wogf2xGCCGEMCQJ8EIIIYQBSYAXQgghDEgCvBBCCGFAEuCFEEIIA5IAL4QQQhiQBHghhBDCgCTACyGEEAYkAV4IIYQwIAnwQgghhAFJgBdCCCEMyPtd9EIIIYRIfJ7vor/P/4AIlGHOkvKZZFLmsSNlm1jkeomx0E7YJUUvhBBCGJAEeCGEEMKAJMALIYQQBiQBXgghhDAgCfBCCCGEAUmAF0IIIQxIArwQQghhQBLghRBCCAOSAC+EEEIYkAR4IYQQwoAkwAshhBAGJAFeCBE9ebUc7WrBnjfVJyIiVeboocNRPtWnIWLgvvBPiUQu9mM7WDFde+wKzpwS9k/w+WWOHqyZfgf7mlhf0jjhs54cgZ/VfWY3T9rafQfyajlqW4XJe2CYXnsBtrYYnE6Fgw6r318PHDqLfVMVnd4D5ezrKsLs+TGhyhuClbnLuZate4M/22JvwfZQy+R8RvVa418HErDMo1puAW0AAvoE/+dMUhkpfVDo/mxS6884BPShUSq3YH1zQN/mpdbvBKjXRjLxAO8JGH1NrN/UqD/e1cMS/451rM8HvwqpVJQOB4lRUSqeYUbbWtZ7PlNeLUdtO9hX0e79nGVPp9Cas9bbeVjsLdhsDsraQg2QJiggoGsp5Zt2Zjfrbe3en4/aXSEabjwywyXP+aPWuR72oa9bug6qb3LOrOxpJUC5dUcTq8xjV26jTArU4D7g7R8mqYzyasn3n2CopqL+jFleLUs+2c36Ek8ZRbfcQgd0PYu9wDd4FZNmgin6cvZZs3Cf2R0YbPeWsN55BbNVm64b6/ODaWSr8wpkLqNsYic/OfaW6AcsbVW09oF5qS8ltr9E36l12lpwkcWSiuifjmVu2uiP2wswD51lj7fRKuVtWv1MYpQ3AI3YtJ3O3jfoHYK0ubm+YxUOdVa2Fudkdc4VDqypV3AN+R9fhpkrtGrK3HlmGNOiXCyTdGoRi1W5mVP8Zu96ZU+vwtTXpGlLk1Evc7FXrGKgL8ifap2K+jMebVVs1QXgaNWtXOalRvjUvFqeXz2AK57LyaAmNINXRmVXcIYawe19g968HSzMyYW29jE/PyTXIG4WMS8PiEUaO97k1XLUtojLzksstHpSlOpsR5dyH21ZROPWhyFm77msX5SM+1K7/nH/8vZPlY6aEYhTe0u8WZXgAUJJ8y+81MTlRUXedL+S6ten00dL//uUs8+aRq+9Hip2MPowK4h4KfOw5eahL6OIZnpDg7hGedj9id+jey/ishb46mWFgw4rOJ1g9bQJtZywt2Bbnaw7Fq7sLPbtrLjVxPq3l9HhP4uPoBz8U9iR1ZOpY9GWUYR9ycBH4Wbv6iDJuZZ3lvb4zeKj3caEvwnN4M0PJUPfxVEqQTvXbuEdLY71+aHfOAUTA1xLyOBezpJMcL09yvKCOqN7R1ehk1mRB3ty1rI+Zze9Q1lYu3royBvErj0WZrOM+aFkyCyio6tH+XesVlPWZmZMD9Jo2z5kgGRmmJXz36emStd73vfWWMtgclns21kxXTtDjpxpdQHsVT6r/cwwZmsPHV0F3LFrjznCBLtc7MeKSDtTH3xfxd43/K5dOdbV2oFWopV5Mitsy3gnRz1f5xVMq7ePmpmzzE2D6auweeplV2CZmh7yS/LmPUyat156ZGFdelEtpyZc6mvaHmrRHXvensuoKhzYVg/gHOcyoMXegjX1rNo2lTKIDyEG8ZlFPE+9r36F7UuUvkJpD8q/o0HKtMyxgxW3mkYNztFpYyKYmO+id30yHN3n59Vy1JqF+8wbsVmfjimlozcPncUZqsKH/HzD9O71zDrasbVdCX4szNLF/hJPkFAbMquw6YJ8GHkPk8Ywd7yTqXZsJXE4e1f3dHR4O/fx7Wdwa4KysnQS7NjoyymeTi70DLYd26bd9KZ6Bl7Kerz3+YlS5houp6a8975B71CykpkLodNWoKmXa3H2ZWHVBPn9b1+BzCL2ectZmRkGpvWvaIKyko4OdmzUSUReLUetafTax78HxvxQsj5T5r9UN0XKHDuCD3Z1y3KR9CWNbNVcr/X2s7B6hy7Iewc5YQZJ0WhjIriYB/iAij6e52tnnOpMJh43H40qr5ajXUpHvz5EetBibxnj55toFqMd2yZlRmONtPG0VdHal8wKW5zfWrO3xNf5vL0s5AxjPMKnJX0i6uTyajnatd07i1mfs5bWh3b4siuJUuZe2sHI+OwvUWaR+Z5r5t2j45kxboe9Tbgiea8wqX89T7ZkYnex7H/zLO7MIjri5pbBcvZ19ahZhSADlzH00UG1VfGkdk+EJwMyzmWksbQxEdqE1uBdnwzDamWUF+p2uHmp4L7kGtfzvRL91gp1nXy0dSTfrTgFU5iZcHFnCBbOzQW0t/EpM8jL6mXZX7KW/er6WUdXUfxfn70l2Oe2YFuUi4X2SZz5Kql2UNPE2odW76Cj6xmcOSXwtHLbnDag7C9pYklXEdYK6NybgGUeC5p1b0AZGDHAO1FcqvPu9rb20GHVP2bt6iE/wl3jtFXxpGdfgK2HDlsMb30NR3Nr5vpJmRjlYs9T9kBYu3rQF2MRHV0FU1cW95gJzeA9qZP8UDOjimdYMX2Yy13t43q+IVQ46LCm0WsPE9xDjaxjTZf+VfdA+K91mlMwDV2iQ9cg27FtUtcWE+WOhknnl8b0rm8qm86UZYMx7Ea+p8o8xH4QDWVn/Wh7esbOf6nAabTGgwAAIABJREFUt36u7Jgfc+Zwb4l6zUdfoogZze2Fsc56WuamqdkStZ76laOzD2VgmiPBfbJMMEXvuVVlR0D602JvUWet2os51ucnvrKlWbr1pEDlLMnUrqXHUi52h3a9XV3H1ARvT2rRt9ap3NroalPPL6+WfVFKdceKxe7Qp0Xzanl+dZCNRXFBWe/034RW5ijybbRMgDKfqDKHfhOV7vMD5NVi1ywjWewtSsYrTrMY/p9nqlhyFvndXhgl/nUyrtvYvWviX3Szt4T1e8vZ17WDjq4d0X9+QlNmZ6bMYJ9VTdmh7AQ223rosPk9JRa3QqX6pYv9U71tVTxJLUdtvhSl/9JC2mrt54nw1rxJ1PkR2PzKM65vs9lbwno8qVzPQX25xnuZT1yWPp07dBZ7jr7uL9SmzYM8Hl/0nyfSL4SJNt9dM0UBj020TejrZJy3sXvUF5JnzPpThjmLflf0b+PQ3VcZwZrhWJ8/WWJVPiI0KfPYkbJNLHK9xFho60uUvos+uE5bwZhG2GN9vhBCCCGCk78mJ4QQQhiQBHghhBDCgCTACyGEEAYkAV4IIYQwIAnwQgghhAFJgBdCCCEMSAK8EEIIYUAS4IUQQggDkgAvhBBCGJAEeCGEEMKAvN9FL4QQQojEF/Bd9B9/9NGUnUy8mzN3rpTPJJMyjx0p28Qi10uMxZy5c73/lxS9EEIIYUAS4IUQQggDkgAvhBBCGJAEeCGEEMKAJMALIYQQBiQBXgghhDAgCfBCCCGEAUmAF0IIIQxIArwQQghhQBLghRBCCAOSAC+EEEIYkAR4IUT0WHZx6ISTastUn4iIVHFDK8cbSqb6NEQM3Bf+KaMpof7EZjLop3nDNg6GeFZ2tZPKlSb6m/PZfkA5VtzQSuF8dMf8Xxf/53OYjdscgW9g2cWhygV8UGelpnNinyj61lHdVMXyab4j7nO1fLvmlO+AZReHKh/DpPu90ct0IjzXQz0bLujKzXNNCfF4Iggs84B6VtrA8ULfp+TueeqKdtIdi9Pxf68g7+dpD1oB9SQeBNTV6NUPfRnErt7p6z9wVduv+Nd/LYO2hSi8ptbEXz8E/7p3NUQ8EF4TDPAAbtx3M3i0FAh6UdeRs8AU7AEAMgobKD4Qm0AWF0o3M+NkPhs9ZWPZxaHKKupLT/k1gtgFdC2lc7tJ8wYrBz0/VzZQ3Km+tyWdO82+81Ued1JNInVsGfB+LRs9wbG0geOFrdTj63iy5wxqylvpsCqbdkGsgnwEA4i4DOh+ip9KoX1DvreeBtSf8b5uQyuFM89Tt0Epo2i9bjAZvO99H09AP96AGiwcbN8QGDSUCcZbCdQGPMK3hbE7RU1RYD3NrnZSueB9mmMR3EsbOF44iwt1+Ql4DaZOFFL0JsBNxuO7yA72cOlmltNP/90gj109z4W7GRQaOT10YJu+IXXupP0qZHxT85kzUgg9BIqmEgpXmuhv9nWa3TWNXLibQW71Ou/51WjOt7vmLfoxMSP4lCZOOajRBsoDh7lwF2bNWec91F2zUxM4TlFzsh+mpYSYuU1M9pxZYZ6xjq/OjMEbx8DBbfqAq9QPdYA/XpZd5M53c+FnvgFQQL2MooM12oGWg+ZzbpiZHrz/0p7fkUScLYZvC9Gh9i0nYzFALqG+cFYCZk+mXhRm8HDz5FvcLHyCHAt0+12A4m9m4H7/MHcWZDAj4DevU/MzOFS5mfpSR2zSOoni7iD9oR7zLEE0v88jhZ4UlTrj16V/w2QBSpcoyym6cj7F725XsTw1A4hw9uifKotlenvKKLP6R94/zAcLNnvTkUr6UZ/GjSgleft62PK5+fEo5Z9wZa4vo/DZiZv8Ttd3nKLr/XJfvYxWGxiH4qcew3T1sD64JNz1CE+/dDG2csyufoKMu+ep07QDz7JqM5u9Sy9KPUCX4g9XN5TXfp+60YK7Aa9HNERpk52D5nOw/Cm/mbhlF7nz+2mvCRm6oHMnr5xzK6n66JxMnCvh0fnQ/2vfbCB7ziyY9hiVJ1o5fqKV4yeClYWJ5Y/DKxvy2bihVsl8nGjl+OOD1GmPjZINyZ4zK+hAov9W6BlMccNmMu6e1wwKSqivfIybzfls9Lzv7TEXwqTKri5n+bR+2kN2IiXUF2bgPnc4oEMzrXwCfqZ81rpzbjIKWzl+4gnu1GmPjV53M1JNMH+zem1bOd7kn+3KYMY01NdW/h3SzVzjuMzVQeO7ugGOieWVS3h3g3q+zf2YVpaH2Xg3i6/6PZ6RavKrlxNvAwEsu9gy2swz6Ow9jq9HGCHbwvzNbKHR93nGVI6jzN7nb+bRX2vrQRXHT1Qpy5YR1o2MVBPu90+R0eBrH/o+MnGvR6xFbRd99+n3cc9fouvostcsgCCdZsDvquk4Q6fqAWVW6B8wobvGqlZM5V/z1QwKA4K8NoWpppSDHfO7BuORXe30NqTcW7Vs1I6ELenMws0d7yjhFDXb4nCkXNrg/QyVqW+xMWA2so7qJk9noQTsYLMI97lG78xNSUcHOzZ6ivrgtnzN9a3lAo9RqQvyDrZrrv/GuvOwssoX5OO1zC27OBRiYKRdBlLSwiYeWRMiLdx5ig/umlj+fU2ZlDYEbDqMXhsood4TKCqVPQWhMjDZaxZguuq39h6v1yOUsG0BuHueV7z1f4x9SemSgD7N6+phX9mqywOBx0apG+rylWlllW+g4OkjPW0o0a7HJIrebXKdO2m/ql0zK6FwJXxwOpK07ylqfnYe9/zN1E9kLS+eWXZx6EQVy28f1gfMIA5uq41w/dE/rRkd2gHHK5Rz/ESr77p07qT9qonllXF+a82Bbb6A+eslQWbFp6gp8nQYjfD9UJmTQKOm0sM6RU3RYfqnPUZhqLreuZNvN/djWrlZOZ84LPPsaifH1VlT4MBI29lG4hQ1RerAxxN4v/kOdefcESxtjKcNaAdU7/Bo0KwKeGemv/Zbe4/D6zGqsG2BiJaQgltH9eMZuN8/FfHvu2+NqXIotIMC4OCR87g9bSjRrsckiup98AePnIcF65SGUrqEDP+R72iMnKovbfB2hlN5W0f3xzeDbiTLSDWFbODdNVaa/TYFKjPSWi7MVNPO8d6oDmyj7pwbk6duBlCD7kQ3i8VQPJV5cUOreidGNG+H0g64lHaSkWoaXzAYEwfb687jnraAHP80cdDlB0U8XY8xCdsWxsiyjkemuSOcyI3HKX4XLN3eeZ2bmh8T9nrEWHS/6KZzJ+23H6OwdB3Vj88a865TX6o+nTvBdt0nIs3tHZF3hsqa7MRmikH0D+IOWOtUUmABs5Sw1A65OTrLAveUgJRioOD7Jaa+zH23s8X4lk517Tt2gSO84m9mwNV3RvmcU389plr2mgWY7r5PVwx3twfdIxS0Dcn18Bf1b7I7+Ot+Mh4v55Hb47ln1JOqfyzklygkmuJvZujWbIM+p0GftShu2Bxy5jAhnlSWZq0zu7qc5fjWz/zPxbMW6h0AWHZRH4Nbl6Ipu7pBv2lH3UjlSyOWUO83wo9ZmbOO6gZt+ncd1d9/TN8p+pep//nGTZmX8Kjf7WzRUlytLSNl0xRh2s24BJRlkOuhnoP/ZtjQrxG/wreFiVC+4yQ6rxVad81b9E97jC3eMlevmSdDnEDXY7JF5TY5nQOHufB4FTPGPCNUde7klTV+3zTlMX8zx09s1hxQv1lqfO80CdQNIvOVnaN62m/FUnYDF3oeunte80Uc0XVwWz40tFJ5opVKz3tp9gT035pFpfZccAd8ucSsldrPMzlf0DMW3R9DZWUrxyt9x/S3svVzZ2aVvi7FsMyZ+ZivvCHoN3DpyzTw1ru4KHNLOrMwkeFXtsDEb0tK1ZdRf3M+347FbbOd16HSrz0G+0Y0dYb4QYgsS1xcjwiEbwsTEaNMYwAH2zdA/QlNmftds0S5HpPtC8kzZv0pw5zFxx99NNXnErfmzJ0r5TPJpMxjR8o2scj1EmMxZ+5c+l1XAPljM0IIIYQhSYAXQgghDEgCvBBCCGFAEuCFEEIIA5IAL4QQQhiQBHghhBDCgCTACyGEEAYkAV4IIYQwIAnwQgghhAFJgBdCCCEMSAK8EEIIYUDe76IXQgghROLzfBf9ff4HRKAMc5aUzySTMo8dKdvEItdLjIV2wi4peiGEEMKAJMALIYQQBiQBXgghhDAgCfBCCCGEAUmAF0IIIQxIArwQQghhQBLghRBCCAOSAC+EEEIYkAR4IYQQwoAkwAshhBAGJAFeCCGEMCAJ8EIIIYQB3Rf+KZHIxX5sByuma49dwZlTwn7d88rZ11WEOehjQIWDDiuax5Tn41zL1r2B72qxt2BbdAn7pio6o/NBoszzeYMZptdegK1t/K9usbdgW53sO9DXxPqSxtHPIehzDCivlqO2VZg8P0/S5y5z9GDNDKzf+ms18Ws/ZSocdFg1f31y6Oyo7U/53EzK5w1V9v7n7ArRn9wLQl0Ppew8P8W2furfK0QsEFEx8Rl8hYOOrh2suNXE+py1vn9OsHb1sK8i2C9lYXWUT/it418jW7Vlov5z9gF9LRNuQGYuYfe+bhOuzCI6dOWqBPe0M7t1zzlqz53YG8c7NbgPONdO7ufOqyU/M/Cw0qkO4FSvlf0MrLA5KIvt2cSEZe6g93Osz9lNL6uwHavFEvTZ5Vi1A9BYClH2SnBPo9f+/7d397FRnHmewL/RRbkbqR0LK4YxnoUsdnszOMqCLzGstQNOx90j40PEizpxtKtZDWrGweYQo0G+U0kOA0glncVqrMhth6OV1Y52hUmLsREy1rh7mqZX8hmCgB2dmchtZ0JmiZOQtc+4pblDd8r9UVXdVdXVfusXu57+fiQkXF3ul6fqeX7P83uep621S1NwekcgewrztjYW6+vRHojBWz6RbEvyeX+aXys4XQNvxvuHspVlgO/AgLcGifFz6aMjvy9jZYqPTyBR3ZYh+AvO04Pm6kXc/ij70eQFST9y6kdwfBEofzFZWVxyC5wLE3hfCiXPORacgqPhbVsGl5Vqf2svHNODulFaIT63G3LnXsxOm/+sp9KoxoOpUUpE6sXthRo027CjFZG6dKOtEKSxKaC0zDJL1R5oQ8X0FBJ5f1eZyt4N2VODxHhvqjPt9yE4XYL6t4phgGFkeT209sifakvyd392YFc1EB9LvdaFjyaQKK1FU1F2uPIvqxS9S25R0u3JAGLiv4zbnm7sbHQDY7pzHnXh/fERSN4A2v3FlZ7Rgk9aegyDCKItmbpKjJ/DmxIMUx/KsQxlncaNptoSJCZDxvRpfA4J1GK7B8AY0lPZy6Rc7SLxddx4wH8XcW9L8nPnusxd8kkli3WvDmH9SLKzTqkjhpRwCA8fd6N+sxOA8tzGtKUAaeTOgDJS8wMnpIq0x8JeIBgEvFrqXL3voJ/KWOG9mLHsPW7sLF3Eg6jx+sW/XgRqlY5wBAKWvZWlrgdm8dCQTQwhPHlSd38qU7A7JwfxoLYtWTeUcjJOAeai7IriehRIViN45+YSYPruEgE6hIePAUetOy0Fo/USiyNVr1pq9F7dhl33UmlER0M3wtFubBrTHzuZObXo6cGJhhJd79iJTaXA7CNTcBr7DLMowSYnAHRgwJDKPofbj3P5gdePY7NpTOl5ERXJz63Ktsw1nQElBW8xx++qrAAW5mDqbihBRs22uOQRQ9qyKWgeiW5UWgbvsrEN8PRg2FthGBWmq4F3993UFErpXkjRGKTNI4ZjJ5YbRS5R9nCWwZEWvIDIo9lk1sG+Zb8Ky16PCqXjq+PcXGLIBgKAo6EF8Gtp/EU4vTGEoy2Yl/XHlkrtK1lG/Tntb+2FY2ESYfUaFcX1KKC8r6KPf72Y4ZEQJH9xpepdjbVwZJp716eU/Zdxe8HqWImSDUnqwEA0hnA0hrBUhtHGVfZ0PS+iAouYT0afECSf/UfvF+5NAYb7SknhOswnrqnMTbTGU157Jsq5uQR4/Fmq3P2+DTxicUO+qt5zauNuzHBoncblFmlN6YKyOr1kccxqcJBUdGW/Fstcj7EQHiyUoL5TNw/eGTCMoDX6qY6INIK45bEa7FqiPY9ILZDHK+BV2y1vuTFLI/71KKy8B/i0C6Y31oX3l+31iUKdi7238rn3tDRzGv0ivrvYFY0hvJoFK2NdGJ0uQb0UMy3Os7nk+g8tEJ0E/IOIGzoz1pYvc72VBrOlXfhI6eiGo3ZY/BWCdEi753qBzhjCUa3+uiFfVRZ1rqlRtsh0ZFaMZb9aK7keIUiH1MWS2mBh913I44uZ222dtAzhsu8nhhPoTS06HiuDpCt7sa9H4WUV4ONfLwLVdUsEZze2ly/daBZNql6di72ft95oP47J+gUrccwvABWVptGnadR+waem5svblMotynXw+3Q7F1og4UVUWKRrs6GsQYGuIxFTt2PVwBuNYVh2G9LBeoaO71gX3mzch6bgrNLZsk3jFoJ0aDA1aut8G/WlUKc6tMzSXjiQ+07kSspeWW+Snn42TJvYtuxXYMXXQ99p24cmXz+cm0tW2dld4fuBftEv0hc9inw91kFWAV5LyWRcbdn5NuotFrkY6VL1lXMFWHG7Ptp31yyzXiHX1PUP5rloZ5lhzks7Vzqkznct2WGzL2VxY27LPyK1pG2BVOYMpxBsVFPXlkFG6fimZXP8PqWztdy0wEZl6FSp/+QJJLCobFPL4fcQrKjsDetNUpybLRaf2r3sraz1eqhrhZZut1fPVWle3LcEEa/HOsgyRa9tP+pO22PskkcQ9tYgvpIUmpaqb7CYJxWCuj1kFen5ZXl6MGAoc3WeWRe8tXRXai5aWRSVXIiX9hyC8PRA1s0DuuQR5QtQ1uMLfrRpEN0cp0s+iXpMJFfWtwfsMkXVgQHTKLw90JbnzFQ20hd1KfPLUxiVtN0Ldin7/GmX9dN6ytQH9FsLcyQijaQvnFTn+7W2kdcjt7L/Jju/D03+DgxElRXIaxWRetGU9m14Cqc3hrBXd0DbUmMXalr8QS4zXmOfAZKpzM3f1jbWhTfRg2EpVX7mLScVDfrnEOdbpXbq75mFCciN67d48IJvHxCIKavEtfdj2P6lpJW1t7u67ZCFFMd8eTfC0bbUoXUu2+VEpBZl612yfM33uF3KPo82703dm1DaiDfz0mHrx7FGpMUKY5vE65FLz5Rs2vJtlbMGM/Hcb0cwfD2njb8iNV/lQ5mxzPOHZWsvvF60Gvr7JUffRW8tIrVs2J49ERGRyPjX5IiIiATEAE9ERCQgBngiIiIBMcATEREJiAGeiIhIQAzwREREAmKAJyIiEhADPBERkYAY4ImIiATEAE9ERCSg5HfRExERkf2lfRf9F48erdub2ei2VlayfAqMZZ4/LFt74fWi1dhaWZn8P1P0REREAmKAJyIiEhADPBERkYAY4ImIiATEAE9ERCQgBngiIiIBMcATEREJiAGeiIhIQAzwREREAmKAJyIiEhADPBERkYAY4ImIiAT07PKnLMWH3uuHUWU6mrjVg3fO3ACO9uFaq/nR9POO9I2idYf50RkMHTiOD7N7gxuL6ywunSpDKF+fy3UWl069BkfyQAJ3zntxJmI8bf/pIE7tcSx5jt1Z3VPJ+xKwKCtAyHsuj/afDuJU+a9x8Hgg7691pG8U7se665dkaoM+vVKQ92NPPvRe/yHmC1Hfl2nrjPVTzDZoI8gywCtmhppx8qL6g+ssLp3qwqXTwDtnjuOg4fj38UmmCylyxTR0dGby9jJH/qoMoQPNyQq1/3QQp0714UjkuPHYnq8wdMCLDzOcIwpDQLfEgL4Whsb503y+0us4PdiFV59Xfko8Nj+uBPctt3pw8MyN5M+XTs8sc92LjKEzm8CdfL7Wsm2dek2/uYKDBwRt7zeQ3KfoI+/hg1sJOMozj9yLiw+9rVWYGWrGwaH8BXcA+PC4MVjdPPNrzKAKrxxNvZfWPQ7MDKXOu3mmH3eeVMF9+vW8vrfCeh1/8sIyp1SVmUbvtCJH+9C6YwZDB5oxlNfgDuw/3YFX8THOH+jBnSdWj/8QVU8+xgfJYB7AyaEZOPYcxpH8vjUbeR2nf/IacKsHB89/jEReX2v5tm7/6Q4luIs6mNtgcjKCN6sqZ9OZEsBJrad61PqMI32jaMUVDOFwcmSkjD5hHMEsOyJdxtFdqMIMhi7qD97AH77pwqvlVQBuwDxqsnP67KsvlimrJ3NL51PMU0wiZ5lW6mIqK2cZRLVM3dDv8FKrNmpUMyWm0d1y2ZObZ7y4CQB4HY1pj76Oxu87kPjdDfUc7WnnkMD38ScuABGkT8U8+Rjn294z/o7QbuBMmzYtZd2Jz137s1xbpw0wlq5D5uk1Q4aYViX3I/ijfWjdkcCdXxV5Q7haOw7jlX9pxsEDSu/XsacL1653YdNv9Mc6cNq1iudUA/pv1cqxf+sWy6A28zgBvLAN+wEc6dPSZ8rrnr/1VY4+YCFVYdPzQFXrKK5dV/5dMmUo9m/dAjz/Gk5d187pMwYs11lcat2CO+fV8j9wJY+TK6Jx4NU3gA8ONOPggR7ceVKF1uujuPbGHM7rj/X5sngN5RqndeIin+MrOLCpCgB86D31Gr4a0q5hD+58k8VLiiwf7Y+Zaxu2YAa/nTmLS9et6+b+00G0vvCxep/kP+spupwEeH1Deu3P7+PggTWM+HYcTj3H9VFcy6ry29CnV1K91ItXlJRk2jEHXvrBClPprrO41FqFxK0rq5hjVlLbicepSnXzzHEbjt4DOKk1EAeacfD8x8CeLkNDcvOMN/X4gWYMfVqFVn2QryqDA1/hDxHdcxb76H3FErjz37VR8g2c+c2M9bEdu/KbSndtwxYkMJ+8nW/gzPFiGr2vQq7bHytVZXCgCq0/0Tp/qbrZq474q8odwDefp67RxeMcvWchJwF+ZkjX29pxOHmxVuXTK4YGt+hToTAG2tXYfzqIa+rIZXUpfaXhVXrvfeLMY0bewzvLzM1+eLzHuBbh4pXkyNM8+qe10HeWCiTyHkKfOvDqqSIcMOTAWtufZZ5V19GDeo2AqjfOYj+AD3/1MRI7DuPa9WB22QICkOsU/cXjOH8rkbxYVHhH+kbVVfLp81Y3v/gKeL4sbVujodd88biamt+ipFVFCvSrcgNn2pTU/Fd7uizT/LSeZjD/BNiy1XRNTKP2D4+rqfkXDhdnZnAjmZmzXOQ381h3NPIe3jnQjINDXykdMwb6rOR8Dv7mmV9j5vnX8C4bw4I70jeqzl9lWLw0M4cEtigLkJKUtPzMvxgzJkoK+4ppFb59ZVp/kJJhTldN95+/lYDj+6+z47ph3MAfvkH6bp2qMjie/A7RiPHcM21ahjHP0wKUmWF9REpaWh5QBxo92U8LFLk8fJOdtlUlywUZtEo+vLLDlP4y01KWP0llWLStSMrK+tdxuk+A7IvrLHr1HUzXWby7x7ji+kifMTNxpO+wYUEijvatbaqJCkZL5/bqtoH2tlZh5jdqHTDfB7TOAhi6lUBVq67umRZlm+slZScv2+Rw8QruvNGlBJLIChe17DiMa9cPGw5xe8QquLZhCxyoOjWKa6dMj+m2Bn14vBnoG8Wp66M4ZXoMAPDCa6nHYN9rsEVdBaxJ/xzK/Hqr9uOTj3H+gPFerWodxbVW3eNFtb3KBiLv4R2cxaVTqetkvs7G+4BfbLTebp7xAqeDOJWse+ZtuMZ6mfXW4CL3TMmmLd9WOWvwxaNH6/1eNqytlZUsnwJjmecPy9ZeeL1oNbZWVmImPgWAf2yGiIhISAzwREREAmKAJyIiEhADPBERkYAY4ImIiATEAE9ERCQgBngiIiIBMcATEREJiAGeiIhIQAzwREREAmKAJyIiElDyu+iJiIjI/rTvon/WfIDSVTlrWD4FxjLPH5atvfB60WroB+xM0RMREQmIAZ6IiEhADPBEREQCYoAnIiISEAM8ERGRgBjgiYiIBMQAT0REJCAGeCIiIgExwBMREQmIAZ6IiEhADPBEREQCYoAnIiIS0NoDvKcHw9EYBjrTH2oPxBC+2gOX+YHOAMLREcgeAOjAQDSGcDSAdqvn7wyYHlPOt3o9AHDJI9avuZF4ejCc6fPmmEseQTjQseQ57YEYhmV3Ad7N+nLJI7r7znw8pv5Lf5wsdAZ0ZZahnqttQ/KcZe5Dyic35KsxwzVbsg1dwXlkH2sP8GMhPFgAKirNAaIDu6oBlNaiydygVlYAC5MIj+mP1sAregOgNYrSXjjy/FLtAaVySg0lGc5IVXhvdZ7fzIbQAa9FWbjkEUgNswg27kNT4z7I40C9VJjOl525KueSZdbUeA63sReSPsh7ejAs7cVsUDtnEPHqtqLoSG5MTmDynHot9qEpOAWnNz14twdikGonISev7T4c86/PO6bcySJFH0J4chGOWrexB99ZBycWkVgowc5GfaV2o6m2BInJECK6o/HxCSSq2wTuLXZgwFuDeFCpXHnVGYC3egrBxn0ITluf4pJPoh4TkBvP4fZCft/ORtAeaEPF9BQShqPqvTh+GRfUIxFpBHHUYJew92FuRKSuZJkBIUhjU0BpGZzqkfa39sIxPagLDv04FpyCo+Ftdp7WRT8kKZT60X8Zt80Ds84AvOUTkA91Gdpmsr9nlz8ls8ijWUgNykg9oo7KXZUVwPQIRtEG72YnAPXm8rixs3QRD6Ih45M86sL74yOQvAG0+326xkMU/TjW2K/8N0PwaA/E4MUggmhLjqoT4+fwpgTIV7tRX6o/FrJ+EgDw+9CkNqyZGtOI1KJWYjeaLM9wG14TWMRtuQXSmOXJG5vWcPmBE1LFmn4/7E39bWVMD6LJ15+79yeoxNdx4wH/XcS9LdjuATAGCHWP2Z4bsqcGiclzSwd31gVbym6Rnf8u4ijBJq37ro6M4vf6ceHeFFBdlwo0zjI4MIuHFpU4IvXi9kIRpOqXUt2GXfdSaTRHQzfC0W5sGtMfO5kgsYDaAAAO9klEQVT3eeL2QDfqHw8m03Ty+Gx+XzBfPD0Y9lbgtt9qVKKMPPXl6ZJb4MQU7vtNvy/rUs2Fe/c2oWSn9JkQAHBsdhpP87yICl07Icw9ZkMu+STqS6cwmhwoOLGpdBEPok51TZTFugnWBdvKchV9P+5PA87d6s2gjtLn41CDfyrl2b67Bpi+m2GEHoLkFz1Vvwx9WlNNo6UfM0975Job28uNI7CI5LPhyKoDA9JezAaXGBX6fWgKzqJe0tYszCLYqMsgpXVI+3GMIxYYF221YF7eZ8gqKR17fT12Q+7Urz0R5R6zEd3CSGnzCJr097na+aqX6nC/McO6CdYF28p6m5x+pO5qrIUjuYhOCf7KXI+y8C5+b4mbYqwL748vwunlQidNWqoz77SRbXfm3Q0bnhvy1TZUjJ9bcpFQeyCG8O67qcVH8hya9SuH/ZeVrFK0OHYarFwI0iEtEPQCnaadMH5fciGXElROAv5BxKF2/IW4x2zG70vd5/fqELa4p+NB/fRoP4Lji6l1E6wLtpX9PnjdSN252biILv61ugjP8yIqkhU8M6bqNwC1MZDHK+BdahvjRtX5NupLoQaQmG73QokyWg90AJ4eNFdPIagfhWgdTI+2IlwLZIOYVZ+LjZtZCNKhwfTFifqA0tgCCS+iQj8CtPs9Zmd+H+Rx3eLosc9gNUESeaQ/yrpgVzn4ops45heAisoe7Ko2LqKLRCeRKK2F9y39yH4pulR95Zxp5TMVUkRqUefabLay3BBctNH5BBJYVOYQff1qynGl+nGscZ+xUaRVUVbWp0/P2fYeE4rWfhsDtrKlec401866YDc5CPDadrlaYy8dUPfKl8BZnb49LiNtJNWQ/z3jZOaGHNjgXxaUC1rK0bSQ6ESD7j7tDBTvepCMOjBgyq61B9rSFifKunJzySPK1s1ktqRI7rENwiUHjAtzzfe5xYJT7Zz4WBfrgs1ltU1OE4lO4kTDXjimR0y99BAePu5GvdX2uKWeT+pFk2EbTYrTG0PYqzuwoOzfpBwp3wspGoOk/hgPiviFFyFIh5QtiOFoW/KoeRui4V5b4D5hII75cmOZYWECcqOxXHaay830eHHcYxtD5BEgSTGEpdSxtPL2+9CEAMK688znsC7Y0zMlm7Z8W+WswUw8z1/CYmMsn8JjmecPy9ZeeL1oNfT3C//YDBERkYAY4ImIiATEAE9ERCQgBngiIiIBMcATEREJiAGeiIhIQAzwREREAmKAJyIiEhADPBERkYAY4ImIiATEAE9ERCSg5HfRExERkf1p30Wf/GtyXzx6tG5vZqPbWlnJ8ikwlnn+sGzthdeLVmNrZWXy/0zRExERCYgBnoiISEAM8ERERAJigCciIhIQAzwREZGAGOCJiIgExABPREQkIAZ4IiIiATHAExERCYgBnoiISEAM8ERERAJigCciIhIQAzwREZGAGOCJiIgExABPREQkIAZ4IiIiATHAExERCYgBnoiISEDPZvXbR/twrbUq9fOTj3G+7T3cxOs4PdiFV59PPTQz1IyTF4H9p4M4tceReuDTKzh4PADAh97rh5F6tgTunPfiTCTzc63l9YmIiIrBMyWbtnxb5azBF48erfd72bC2VlayfAqMZZ4/LFt74fWi1dhaWYmZ+BQApuiJiIiExABPREQkIAZ4IiIiATHAExERCYgBnoiISEAM8ERERAJigCciIhIQAzwREZGAGOCJiIgExABPREQkIAZ4IiIiATHAExERCYgBnoiISEAM8ERERAJigCciIhJQ8u/BExERkf1pfw/+WfMBSlflrGH5FBjLPH9YtvbC60WroR+wM0VPREQkIAZ4IiIiATHAExERCYgBnoiISEAM8ERERAJigCciIhIQAzwREZGAGOCJiIgExABPREQkIAZ4IiIiATHAExERCYgBfgNoD8QQDnSs99soKizzPPH0YDg6Atmz3m+EVop1QVzPLn9KBp4eDEt74ch4whSCjT5cAIByN9p/+iO8sXs7yr6jPpyYxYPfBPDzX4QwB+Um81ZnfrnE+Dm8KYWUH/70p/jw71uxDcBctAtv/XzC+r1ND6LJ17/mj5hznh4MS2UY1colax0YiLbBqf1o/rydAYS9ur8UuDAB+VAXIjl5bbvowEC0BfNyC6Sx7J8t7T61usdMdcNw79qdZb3X1fUccMkjkDaPZKi7pnu+KO/ptch1ubkhX+1GfWnqSDy4D8f86g/mtkeP16xg1h7g5z7Dg0/KsAkAUIKKlyrgeDqHzz/9Bv8HAPAQXwLAzg4M/F0bnN8B8P8W8eXvZ7H4717A9soK7DzUjX/c/TJ+/qNf4MtPpxD/v8pT//vNL2Jb2XNIfDmF2f+lHJv//N+SL73tzTpsU/9f9mobDmECV9f8QQrAcLPn6q9CKRW2YvwcmqRQ8udhOZ4MJq7KOQQb96kNr1Ihpas9QDFULkMgWsTtHD3nrq/PocmnBev0Mtded1Zr7Dw9GJa6MdAZSjV+tpfbgK4xdJ6mLU7QlW2TMGVZAHkpNycwqbU9UNu4GAag3vd+n8VrKfUFY0XQ/mwQaw/wd/oh3dF+UHuHf5zCP76rv3i1+JmkBPe5W7/Au/9lCHPaQ479+Fl/N5q3teJE9//A35zzJYO0Sx6B1PAcZv/ZZ9Eo7sWPG7YDf7yLyCcvw7Xbib88BFzdsBG+AwPeGqV3iwDC3tw8q0tugXNhAnJyZNiPY8E6hL1vox0hXAAQkfTXIgRp7G2EvWVwAoJXMDfkzr3A+Dk0Rd0Ylmpz87RjXThmyAL0IzjeAqnWDRdCiABwNdbCsTCBoD/1O6NvxeDd3QFgA2WT1spZtkTWLgudAXirlY4DAjGkVxPlms7qR4m0Avkqt35Iku5H/2Xc9nRjZ6UbgHW2Ktlm8foVzNoD/Ep4/hY/+B6AxzdxXh/cASBxE393ug61f9+KbQ1t8GICwRU9519hdznw9MFdyOPAX+6uQ+0PfXBcDSCRlw+RrX4ca1Qb9s6lz1Q6NiXqT0uNktxoqi1BYjJkDNTxOSRQi+0eACtJR3t6MCzV4kFwEju92mhXfV1T1iEfI7b8CUE6pI2o3UueufIyzxUlk7JzchAPatuSKU4lvWlMo8Y3ajBbmEN82ZOMn2XZaQrdiK/d6vHOt1FfOpXqOFmeo3Sgg0HAq927ajoY+utcTCnilZSbKr91oQPehhLEg/pyF6AubHB5XWS37ZXtcAD48rdD1inS3/8Kk/8K4DsVcL66suc85KmFA08xPfFLIDiByT8Cz1XXwZuXYUUBVbfhBHrR1LgPTY3ncHuhBt6MC1+c2FQKzD4yNZhjn2EWJdjktPodJZOQGL9sqrQlqPcA7+tfNxpD2DMHeUXvxcZWVeZW0jtaEWkE8dK9OCGrHQtPD5qrgfg94+jd0dAC+PehqXEf5PFFOL0xhKMtmJf1xwLWwW4duSorgNK9kKIxhKMxhKNW77EE9VId7jcqn6UpOAVHw8msFt65KiuA6buIyyPq68YQtlzMVwPv7rvqNR1EXH2v0uYRw7Hk9RHcissty7rgkk+ivnQKoxk6cdro3aqjYde6YAd5DfDVZeoM6NzdDGc8ROJ/A4ADjrIVPKHDhzdeKQGeTuHWLwFgEPd//xR47mX84Oj2nLzndbMwgfeTlSMEaWwKqK7L8qZ2Q76qVWqlwqSPohZx26/1qtXXtTqW9XvZgLIs8/ZAt0Wj1o9jjYOYbehWyl2yTo8mxnuTi/4i0gjilsdqsGuZrE+hRaQWNQgo/4LTNfBaBPl4UDf681/G7YUS7Gxce1B1bi4xBaF9kMeBesn82lMIJhfn9SM4vmh5zFHrhmvN78Y+Vlxua6kLnYFkp0HpQC2dcYxnmHu3a12wg7wG+Pk/KknzkrK6DGdsh+M/AEAC/7Z8zg+Ot+pQ/RzwdPpucr7+nyam8BTAtv/Yllx4Z0uPP8tDyjAE6ZDWGPcCnZlGXGazeJiDFecb3prLvAMD0Ri85ROQzY1aZwDhqG702ngO856VbUNKy8jYwAWfMtprNoyIFzG/gvq8aoYglGr4m5cbja9oSkFgKym3tdQFvy/V2btXh3A0hmGra6FOE9xfRYrdjnVhI8prgL/321kkAHz3z9zYaXXCzjbs/h6AhYe49/vlnm07ftz4Mp4D8NzOH2FYSzcdUY7he3+BHzfk9O1vYHHMLwAVlabK5HkRFRkb1xCkQ4PsCWfL04NhbfdC2jyuG7Knxjh6RQiSfwKJ6hbuDc9C/OtFq6OYXyj4W7GVgpWb3wc5Q2akfXcNMH3XRmt4xJHfL7q5OojbXwL4Xgt+/t9aYcjCl7vxs//qxncBfPnP/7B87/FP27BnGwAs4stPphDX/ft87imAMuz+T8UxrwaE8PAx4Nhsmmx3lsGxMIlwMYy+14Nuu5H1gjFlbURxybAeJMcij2aB0jIY7/jCvLadrX+5dWCXxRoUKow8f5PdBOSzg4g/Bcr2/BQfjQ3jww8CGPjlMK4PdqN523N4+skg5POTyz5T/d/+Bb4L4OmDIbz7rg/HdP+O/MP/RAKA45UWHNL/0nf3Y+CDgO5ft/FxG7vw0QQS1W0YSI7G1e14yXmuDgyY0sLtgTY4sbpUGaW4GmvhmB5cYjWvMr9rXBCkbFMSpePVHjBO8RTsnvJfTlv41R5oy7hwi1R5KjeXHDBmpDw9ONFgsbOns45tzjrK7zY5AHjQj2N/Hcd/fs+HN/6sAtteUsbxTxdm8SCa+ia7pbXC+2oZgEVM/tpiO9zVQdz7mzr8oPxlvHGkBFf/VT3uqIDzpQrDqfez/kAbxFgX3kQPhqVYcm+9cStJHPPl3QhH21K/szABubFItgflgbZgyVCmKq3sI1KLsiUrqtvLLdS2LGWXheGzFeSeCkE6BMhXdfe0UOWaL/kpt8gjQJJiCOv2wlttZXNVVgALk8W9BmIdPVOyacu3Vc4azMRz9Q1r4mH5FB7LPH9YtvbC60Wrob9f+MdmiIiIBMQAT0REJCAGeCIiIgExwBMREQmIAZ6IiEhADPBEREQCYoAnIiISEAM8ERGRgBjgiYiIBMQAT0REJCAGeCIiIgElv4ueiIiI7E/7LvpnSjZt+Xad3wsRERHl2P8Hao74uH/KSGUAAAAASUVORK5CYII=)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "eUBYtd40krx1"
   },
   "source": [
    "Now that we have a an answer to \"why should I learn Flax?\" - let's start our descent into Flaxlandia!\n",
    "\n",
    "### A toy example 🚚 - training a linear regression model\n",
    "\n",
    "We'll first implement a pure-JAX appoach and then we'll do it the Flax-way."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "53-TXcbYkt9D"
   },
   "outputs": [],
   "source": [
    "# Defining a toy dataset\n",
    "n_samples = 150\n",
    "x_dim = 2  # putting small numbers here so that we can visualize the data easily\n",
    "y_dim = 1\n",
    "noise_amplitude = 0.1\n",
    "\n",
    "# Generate (random) ground truth W and b\n",
    "# Note: we could get W, b from a randomely initialized nn.Dense here, being closer to JAX for now \n",
    "key, w_key, b_key = random.split(random.PRNGKey(seed), num=3)\n",
    "W = random.normal(w_key, (x_dim, y_dim))  # weight\n",
    "b = random.normal(b_key, (y_dim,))  # bias\n",
    "\n",
    "# This is the structure that Flax expects (recall from the previous section!)\n",
    "true_params = freeze({'params': {'bias': b, 'kernel': W}})\n",
    "\n",
    "# Generate samples with additional noise\n",
    "key, x_key, noise_key = random.split(key, num=3)\n",
    "xs = random.normal(x_key, (n_samples, x_dim))\n",
    "ys = jnp.dot(xs, W) + b\n",
    "ys += noise_amplitude * random.normal(noise_key, (n_samples, y_dim))\n",
    "print(f'xs shape = {xs.shape} ; ys shape = {ys.shape}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 266
    },
    "id": "lc4-xoIapKCs",
    "outputId": "52656571-0aa5-4c6f-f522-83b0158c1b97"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<mpl_toolkits.mplot3d.art3d.Path3DCollection at 0x7f01385cbbd0>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAV0AAADnCAYAAAC9roUQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOy9eZQb93Xn+6kq7EDv+8Jmr9wXUSQlynZkWZYtS0pke0aRkonsTJaZvMxkJplkEisvYzuTE+Upy3he3tNxnDeyLfs4iaI4ieVxItmWHUuyJZMiKYqbSPa+L+hGY18Ktbw/wIIKaACNpbvJlvA5h4dkN1AFFArfunV/936voOs6VapUqVJlaxBv9AuoUqVKlXcTVdGtUqVKlS2kKrpVqlSpsoVURbdKlSpVtpCq6FapUqXKFmJZ5/fV0oYqVapUKR0h3y+qkW6VKlWqbCFV0a1SpUqVLaQqulWqVKmyhVRFt0qVKlW2kKroVqlSpcoWUhXdKlWqVNlCqqJbpUqVKltIVXSrVKlSZQupim6VKlWqbCFV0a1SpUqVLaQqulWqVKmyhVRFt0qVKlW2kKroVqlSpcoWsp7LWJUqedF1HU3TSCQSKIqCxWJBFEUkSUIURURRRBDymi1VqfKuRFhnMGXV2rHKGnRdR1VVFEXJ+LfxO7PQGiJs/KmKcZV3CXlP8KroVikas8CeO3eOW265BUEQUBQFRVEQRXHN481/zELr9/tpaWmpinGVdyp5T+RqeqHKuui6jqIoqKqaFs9YLLauQAqCkPMxuq4zOjpKQ0MDyWQy43fVyLjKO52q6FbJiyG2RupAEIQ10Ww5GGKcKzIGUBSFZDKZER0LgoAkSem8sSHOVTGust2oim6VNWialpGnzRexGmyU8JkF1owhxuY8siAI6LpeMDKuCnKVm5Gq6FZJo2laOo0A64ttpWTnefNRrBhnP0cURSwWS1WMq9xUVEX3XY6xyJVMJtE0Ddh8sTXvu5L9rCfGRjnb5OQkDQ0N1NbWpsU4O1VRFeMqW0VVdN+lGDW2iqJUJLayLDMxMUE8Hsfj8eB2u3G73Vit1i1JSRTatvG3EQVLkpR+36qqIstyxvPMaQojOq6KcZWNpiq67zIM0VleXgagrq6uLGHRNI0rV67g8/no7u6mrq6OaDTK8vIyExMTJJNJLBZLWoRdLhcejyctxkZOdqswC3G+igpDjLMj8Fw542pFRZVyqYruu4TshoZgMAhAfX19SduJRqOMj48TjUapq6tj9+7daJpGMpmkoaEh47HJZJJoNEokEmFlZYWpqSlkWUaSJGKxGLOzs+no2Gaz3VARK0eMq+VtVcqhKrrvcLLF1shpiqKYXjArhnA4zNjYGLFYjL6+Pvx+Px0dHQWfY7Vaqauro66uLuPniqJw9uxZBEFIi3EikSCsiFgcLlrr3bQ11N70YmzkwmVZrjZ+VCmaqui+Q8nV0GCuixVFcU1jQi6CwSBjY2Mkk0n6+/tpbGxEEARGRkbKXgizWCxYrVba29uxWq3ous4b0wEWF0IkYzJXAzH2h2K4SaQjYyNNYfy5mcW42vhRpRBV0X2HUWxDgyiKBXOqq6urjI2NAdDf378mdWDkZDdioWk1muTqUoT2Oiei4CKhaCzKCh8/3J5uMzbSFD6fj+npaRKJBJIkpXPFxt+5xLjSKolSKLbxw4whvtXGj3cHVdF9h1BOQ4NRtWCg6zorKyuMj49jtVoZGhqitrY27/MrWQgzPz+p6kiCgHj99dotIv6YjqqDRUhFxrW1tWtei1mMV1dXmZmZyRBjIyo2Uis3kmrjRxWDquhuc8ptaBBFMS26uq7j9XoZHx/H5XKxd+9ePB5PwedvpOjWOi1YJIFwQsFlk1iJyLTX2rGIhd9HPjFWVZVIJJIW40AgQDAYxGq1rklT2O32G56mMP9tUG38eOdSFd1tiLGibvgTQOk1tsZC2vz8PBMTE9TV1XHw4EFcLlfRz9+o6NFplbhrqIlTE35WIkk66xwc7alb/4l5kCQpQ4xVVaW9vR2Px5MhxrOzs8TjcURRzBBil8uFw+G4qcXYaPwAGBkZoa+vL8PPuNr4cfNSFd1thLmhYXl5Ga/Xy549e8qqsV1eXmZ+fh5BEDhy5AgOh6OkbeRKT5T6fLNoN7ptfGR/a9nbK4ZsMTZQVTWdpvD7/RlibE5TuN3um1KMQ6EQFosl/ZlUGz9ubqqiuw3I7h4zHLdKXSBSVZWZmRlmZmaora2lubmZPXv2lPWaNqK5YavyrOsdJ0mSqKmpoaamJuPnZjEOBALMzc3dlGJsXtAsttbY+Lva+LH1VEX3JiZXja3xR5KkoiNNRVGYmppifn6ejo4Obr/9dqLRKJOTk2W/to3I6d7slCvGiUSC5eXlLRXj9RZNq40fNw9V0b0JydfQYKaY5gZZlpmcnGRpaYnu7m5OnDiBJEnp529keqDUL+NWtwFvJOuJsd/vJxgMMj8/TywWS4uxOTp2Op0bJmDlHsdyGj8M4RVFEZvNVhXjMqiK7k3Eeg0NZgqJZjweZ2JiAp/PR09PD3fccUdO0a5E9CoV7a1kq8TdaOKw2Wz09/enf65pWjoyDoVCLCwsEI/HAdakKTZSjMulkBhDquNufn6eXbt2pX9XjYyLpyq6NwHlTGjIlV6IxWKMj48TCATo7e1l9+7deU/6jVoISyQSjI+Ps7q6muEyZojPes/fKrbqy58rfyyKIh6PZ00Z3kaJ8VY2fkDqPRrlasb/Yf3Gj6oYp6iK7g2k1IYGM+b0QjgcZnx8nEgkQn9/P3v37l13O5VGqpqmMTY2RiQSoaenh+7ubmKxGOFwGK/Xm3YaM9fGGqJsrLRv1/RCIUpZ3NwIMS616mQjUFU1LbhQWeOHubTt3VJRURXdG4Cmafj9fkRRTBfnl3qiiaKILMucO3eOZDJJX18fTU1NJX3hyxHdeDzO2NgYKysr9Pb2cujQISCVP7bZbGvMbWRZTtfGLiwsEIlEUBQlXdJUX1+fbuO1WLb/6bgRLceFxDgWixGJRAiHwywuLqb/f/HixTWR8UbMs8tFtujmo9r4kZvtf5ZvE7InNMzPz+PxeGhvby95W4YvQjweZ//+/Wt8EYqhVNGNxWKMjY0RDAbp6+tD1/W0+U2hiNVms2Gz2TJeo67rXLp0ibq6OlRVZW5ujkgkgqqq2O32jKjY5XIV9QW/WdhMnwdzE4eBpmmcPn2avr6+9MVtaWmJWCwGgNPp3HAxLlZ081FK44eBcaGur6/f9o0fVdHdZPJNaLBYLCWJnq7r+Hw+xsbG0r4IFy9eLEtwofiFtGg0ytjYGOFwmP7+fvbt24cgCKyurla0am5UAJgbFYwcsSEePp+PaDSKpmlrxMPlchUtHluZxthKcx1jf2YXNjPmyHgjxbhS0c1HPjGGlNvd0tISLperYOOHEdTk8wy5GaiK7iaRq6HBfFUWRXHNrVW+7Ri+CE6nsyhfhGJYL0KNRCLpnO3AwAD79+/P+DJsRp2uIAg4HA4cDgdNTU3pn+u6TjweT99WLy8vE41GgeLFY6uE0PistwpN0wpWuBQS42g0ms7B5zqeRplb9vZVVS24SLoZqKqasXhnYK41Bvj2t7/Nm2++yR/90R9t6esrharobjCFGhrMGFflQttZWFhgYmKC2traknwRKsFsVj4wMJA3T7yRhjfFPNbpdOJ0Omlubk7/XNM04vE44XB4TSR3o1zGjEWiraKQ6ObDLMYtLS0Z2zIubpFIJC3Guq5niHEsFtuSc9GMIbrZZH+3AoHAmnWFm42q6G4QxTQ0mJEkKb0ibUbTNObm5piamqKxsbEsX4RyCIVCjI6OIssyAwMD6XxtPjaiuaJSzE0HZrJX/0OhEJcuXcJisaR9dzfLZWyr0wvliG4+zMfTLMa6rmekKQKBAIFAgImJiYrSPqWgKEpRC62BQKDkEVRbTVV0K6SUhgYzkiRldJSZfRFaW1s5duzYltzCBYNBRkdHURQlLbbFcDN7L2Sv/sfjcXp6enA6nRnGNtn+u2YxLncyxXYW3XwIgpAhxvF4nM7OTmpqajLE2Ej7ZEfGGyHGiqJgt9vXfVwgEGDnzp1l72crqIpumZTT0GDGEN1cvgillE6V+yUPBAJEo1GGh4dzToZYj61ML1SKsZ987btmM3TzAE3zNONiGj6Mfb3TRDcbYyEtW4wNjMjYyBmvrKxkLIiaUz/FVqcoirImN52LaqT7DsRYhPD5fDQ3N5ddsqJpGj6fj5MnT67xRSgW4xa/lOf5/X5GR0cRBAG73c7Ro0dLfenAzR3p5qLQZ5TPDN2YZlxswwcUJ7qKprMcTiAJAs2eyma93UjRzYdZjM05ePOCaHZ1isPhWBMZm/eRL6ebTTAYrIruOwXzhIZ4PM7s7CytraX7vyYSCSYmJlhaWsJisXD77beX/aUpRXRXV1cZHR1FkqT0GJ5XX321rP0a+36nu4zlm2acr+HDZrNhsViQZZlgMJiz4SOcUPiLlyeZWo2hA8d21PHo7d3rTsnIx80ouvnItyBajBiHQqF0XXehfQeDwepC2nYmu6EBUieOzWYraXw5ZPoi7Ny5kx07dnD16tWKvjDr2Ttm1/bu3r17za11uWyUd8N2JF/DhyzLLC4u4vV68zZ8PD8SYWIlSnudHR04OelnV5ub9/QXl0vP5kaJ7kbusxgx9nq9LC0tMT09nSHG5sGkkiQRCATKrl3fKqqim4N8DQ1GdCZJUlE1tpDpi9DX15f2RZBlueht5COfvaMxYHJsbAy73V6wtrfcHGS2aBq33dshgt0MjHRNTU0NiUSCoaEhYG3Dx1szKySjSbyJVFojqQi8NeXlUKutrMWmGyG6W1UWZxbjqakp9uzZg9VqXXNMZ2ZmGBkZ4TOf+QyKovAnf/InHDp0iFtvvZUDBw4Uvb8XXniBX//1X0dVVX75l3+Zxx57LOfj/v7v/56HHnqI119/nWPHjpX8vqqia8Io+1JVNWdDg0Ex5VLBYJCxsbG8vggbYY2YvQ1d11leXmZsbAyn08n+/fsLLj6UkxM2METX5/Olqx+MWz9zFYDH48FqteZ9/lawGfsJJxRiskq9y4pVeluAsi9i2Q0fR5Ykfji6SkuNDUVVCfiiNNo1JicnS274gBsjurD16SFzTjdXE83evXt54IEHuPPOO7nnnnu4fPkyL7/8ctGiq6oq//E//ke++93v0t3dzfHjx3nwwQfZt29fxuNCoRB//ud/zu233172e6mKLsU3NBgUOuGMhSqgYFVAdslYORiiaXStjY2N4fF4im6kqER0o9Eoi4uL1NbWsmfPnnSJlaIo6c4xo5POyHeaF56M171VbKRIvDS8wrff8iIAjS4rP39iB82eVFXDencODx5sYy4QZ3wlBjq8f3cbD97WhXQ9p1tsw4fH48HhcNww0d1qirkjM76z9957Lx/5yEdK2v6pU6cYHBxM+yD/zM/8DM8999wa0f30pz/Npz71Kf70T/+0tDdg4l0tuqU2NBTajpE7tVgs6YWqQmxUc4DX6+XSpUvU1tZy+PBhnE5nSc8vVfhWV1cZGRlBURRaW1vZs2dPOp8JqVvm7MUn4/fG7eDs7Cw+n4+lpaW08Y8hJJvpjrURTPliPH/ZS5vHhkUSWY7IfP2Nef6Pn0jVhq4nDm67hd/4QD/LYRlJFGhyW9fcARXT8LGwsEAsFktf0HRd37Kx8jdrLt54XeW899nZWXbs2JH+f3d3NydPnsx4zNmzZ5menuaBBx6oim6pVFpja97OZvgiFLPfhYUFFhcXqa+vL7trrZQUh7n6Yc+ePYTD4XQEBuvP6LLb7djt9nTzhdHNVFtbm47qvF5vRlRnTlPc6Cm8Br6ojABYrqcUGp1WZvxvdxYWE5FJokBb7fqF/mby2T1OTk6iaRo2m23TGj7M3KyCC6kmmM1qT9Y0jd/8zd/k6aefrnhb7yrR1XU9HW3V1taWXWNrVDT8+Mc/3lJfBE3T0n4MDQ0NtLe309LSUnabcDEVCH6/n5GRESRJyqh+iEQiG/IFNBZKsj0AjPpY8+BHs5uWISZbbbxS70wt5CiajkUU8MeSdNa9LaBb3RwBqQtUW1tbxs+Mhg+jOaGShg8zN3O1hN/vL9tdrKuri+np6fT/Z2Zm6OrqSv8/FApx8eJF7rrrLgAWFhZ48MEH+eY3v1nyYtq7QnTNExqMW7NyCqjNvgi6rnPo0KGiumQqxfDfnZycpKmpiaNHj2K32xkdHa0oL1wo0jWL7a5du9aczJvpvZAvqjPyxUZUbG5WMKco3G53Rp56I6OznY1OPrS3hRevLCMKUOuw8NNHOtK/32pRyre/Sho+jD+5FkA3y9axEMU2RlTSjXb8+HGGh4cZHx+nq6uLZ555hr/+679O/76uro7l5eX0/++66y7+7M/+rFq9kI25oQFSX3Sr1VpyqVYuX4QLFy5syMlXKDIyRH5ycpKWlpY1fgyVCl+u55s71nKJrUF29UGpEV451Qu58sWQalYwUhSzs7NEIpGMWs54PE4sFsPhcFQsiIIg8MHdzRzdUUdcUWl02bBZ8lcvbDalinwxDR+Li4sZDR/muwtjzM5WUorZTbmNERaLhSeffJJ7770XVVX5xV/8Rfbv389nPvMZjh07xoMPPljWdnPua8O2dJOQr6HB+CJYLJaiRVdRFKanp5mbm1vji1DKdvJhVDBkn1CapjEzM8P09DStra0cP3485y3gRoiuIXyBQICRkREEQSh6IfBmaQO22Ww0NjZmmPWYC+uXl5eZmZlhbGwMeLsKwIiKy8kX17uswNpI8GYX3XwUavgwL4CGQiHi8Thvvvnmlk34UBSlqG37/f6KWoDvv/9+7r///oyf/cEf/EHOx/7gBz8oez/vGNFdr6HBoBixlGWZqakpFhcX6erqyumLsJElXwbmiLq9vZ3bbrst5y3eRr0GQRAIhUJpsR0cHCw6UsgluqUIzmYKk6brCLydL56fn2dwcBCn05lRBWDOFxses+Y0RTn54u0qurnItQAaDAaZnZ3NGA+0URM+8vFOsnWEd4DoFtvQYFBIdA1fhOXlZXp6erjjjjvynjAbIboWiyX92qenp5mdnS3JaUwUxYJG6IUIBAJ4vV6CwSB79+4t+bbMHCUbx7uUyHUzmiM0Xef5S15+MLwCwD27m/jw3paMx5jzxebFp+yFp8nJyYxcp7kKoNBnsx1MzCvBuDPbrAkfhfa5HtvB7Aa2seiW2tBgkOv32b4IQ0ND654QG5FegFTJz8rKCp2dnSXbOpaTXggGg4yMjKDrOg0NDXR0dJSVB6vUe2EzeHVsle+85aWjzgG6zj9f8lLvsuEsQtzzLTwZt9fhcJj5+fm0n4LZFcvo/TcuRO+USDcXhRbSCk34yDerLVfDR/bxKyXS7ejoWPdxN5ptJ7ob1dAAmXPAzL4IxWBEqeWgKAqTk5P4fL686YtiWM/wxoxZbAcGBqivr0//vxxuRj/dq4thahyW645dAm67xNXFMLc4y09n5Mt1miM6wy/W+J3T6cRiseQVkY1kq2eylWN2U2hWW66GD6NBxBBiYxF0ParphU1CVVWSyWRFYquqKufOnUOWZfr7+/POAStEKaY3BslkksnJSRYXF9mxYwcdHR20traWvQCRz/DGjDEZQlVVBgcHM07KSqLVjRDdjY6UG1w2Li+EqXem8uDxpEaDywobXM9fKKIbHR1F13VCoRDz8/N588UbZQ5kTAPeKjayZCxfaaCqqhkTPpaWltB1nfn5+YINH1XR3SSMefflYJRDJRIJ9u3bl5GPKhXDN7UYZFlmcnKSpaWljFzxtWvXNq3O1lggyyW2xTx/PcyiG4lEmJiYSNfLejyeTW9FzcUHdzdxeSHEfCCOjk6Lx85dQ02MX1vckv2LoojVasXlcmV4LauqmhEVG/liIxrOZYZeLKqqbnk6Y7NFPnvCh67rNDY2Ultbm7PhA+ALX/gCKysrDA8PMzQ0lHExLIb1HMY+97nP8dRTT2GxWGhpaeFLX/pS2WOBtp3olkq2L8Lg4CDDw8MV+8oWE+nKssz4+DjLy8vs3LlzzcJcpYtxudILZrEdGBgo6C1aiegai3gXL14kEonQ3d2NqqoEAgFmZ2dJJBLp7idDiM2ishnphTqnld+8u5/x5dStfn+LC6d14wRC13V+MLzCP1/yomk6H9rbzL17WzJEL1dOV5KkvI0KRn2xOV9st9szxLhQBcCNyOkWqqjZDIycbr68eywW4+d+7uf44z/+Y15++WW++tWv0tTUxN/+7d8Wtf1iHMaOHDnC6dOncblc/MVf/AW/8zu/U/T2s3nHim4hXwSLxUIymayohbSQYCYSCcbHx/H5fAUX5ioVXXN6wZjmm0wmGRwcLMrIuVzRjcfjjIyM4Pf7OXjwIC0tLSSTyTWCk0wm0xHewsIC4XA4vQil6zoWi4VIJJJ3JTuhaCiqhssmFR3NuWwS+zszL6gbtbh1ZjrAX52ao9FtwSIJ/N3Zedw2Cz8xmFkfnG9fiqbzlR/P8L2ry1glgZ892slH9reuyRcnEom0GPt8PiKRCPB2BYAhyE6nc8urJW5ER9p6C2lOp5MPfvCDPP744zz55JMlt8UX4zD2gQ98IP3vEydO8LWvfa3Ed/E220501/vyGGYwExMTeX0RNqLyINc24vE44+PjrK6u0tvby+7du9dtd61UdGVZ5ty5cyWJrfn5pZScJRIJxsbG8Pv9dHV1pZ3G8mG1Wqmvr89IbRiLUDMzM0QiEcbHx4nFYum5WkZk99psgpdGA+g67O+o4d8c78SxgVFrOZybDuKwiunX4bFLnJ0OFC26Xz87zz9fWqLBZUXVdP7Xq9M0e2wc25mZZzfKsfJVAJgXnaLRKFevXl3jR7FZKYebUXQNkslkURODsynGYczMF7/4Re67776S92Ow7UQ3H2ZfhMbGxoLOWxvZTQaZJWd9fX3s2bOnqJO+lLxwNuFwmKtXrxIMBjly5EjRo9PNFHuLb6RJVlZW6O3tZc+ePSQSCbxeb8nbMhahamtrsdls6byYpmnpqPjU6CL/cMFPvV3HIoq8di2EIIf517d2rfFV2EpqHBZk9e07A1nRqbFnvpZContq0o/bJmERBSyigCTAGzPBDNHNh7kCwHyhO3XqFF1dXemo2Gxsk20mX2q+OBc3SnTX2+dWuZ997Wtf4/Tp07z00ktlb2PbiW72Ca2qKrOzs+mW2Wx/glxsVKQryzKXLl0iGAzS399fUskZlBfphsNhRkdHkWWZnTt3oqpqWYJr7L9QeiGZTKaHaGanSTZi8oX5iyKKYnrx5OyKREuzSFuNPZXjjMaZWE1k+CqYb7UNH97NXlD60J5mTk8FmAskEACPw8IDBzLdvQqJboPLypw/jsuWEhBVT/2sEgRByJsvNi5ii4uL6QoW87w2Q5BLSU9s9Hy0Yiglb13OObCew5jBiy++yOOPP85LL71UVkRtsO1EF1IHNplMpn0RimmZNVOp6EajUYaHh/H7/fT09LBv376yPuxScrqG2CYSCQYGBmhqakJV1bSfQDnkE06jjnhhYYEdO3bk7MzbiJKxfDS5rchJLV0OJWPhQGcDe/d2AylhM261DVEx6js3opU3H41uG//tI4Ncmg+h67Cvo2aNaBbKsX7iti4+/a1rLEdSdzcdtXbuzeqY2yjypXbM5kDm9t3sJoV8F7EbEenC+mIaj8fLFsL1HMYA3njjDX7lV36FF154oawp4Ga2peiOjo4yPz9fdmNBuaIbiUQYHR0lFoule88r+QCKEd1cYmuw0S5jqqoyNTXF3Nzcusd2M5sjjvbUcXEuxFsLYURRoMFp4acOtmU815iuYPbhzVeaFY/H0zl+8+TYcqhzWgtO7i0U6fY2ufjcv9rLhbkQFknkWE9dOurdCsxeCtntu7FYLC3G5ouYuS7W4/EUbT6z1Wy2w9hv//ZvEw6H+emf/mkAenp6+OY3v1ne/sp61g2moaGB3t7esm9zLBYLiUSi6MdnC19jYyOCIDAyMlLW/g0Kia5Z4AcHB9P7NFPp7bQhfJqmMT09zczMDJ2dnUVdyERRRFY0FoNxJFGg1laeGXwurJLIL9yxg7lAnKSq01lnL2oRLV9p1tmzZ2loaCAejzM9PU00Gk13jplTFBvRPbZeh1hLjZ27d5d/a7oZmC9iZowmhXA4zOrqKjMzMwSDQS5cuLCmvnizysiKrTyptDFiPYexF198sextZ7MtRbepqamiCK/YSNdchmWI7UaSS3SLEduNJBgM8tprr9He3l6S90MwrvD9KZkLyhyqpjPU4uL9gw1IJbiMeSMKw5eXADjUVZvyTLiOJArsaCh+3lshRFGkvr4+I9Wg63pGC6rRPWZMp8juHiuWrfRe2OzFo+wmBUgt3B08eDB97LK9d7PriyuNiotNZwSDwbIj3a1mW4pupawnuubW2fUaDCrBLLpmsTXSCJv15dV1nbm5ufTMs1Ly4QY/GvUhq9Be60DTdS7Ph+htcNDfUtwkDW9E4Z+Hw7Q1p4TwrYUwjxztzBDezUQQhNRUYslOWPTQ2izSXuvIO83YvABVqGFhq0X3RsyNM/woCuWLjTuK7EVPw+6x2NddbLmY3++viu5mUumJlk90DSNvIG0Ks5lIkoQsy1y4cIFoNLolYruwsMD4+DhNTU0cOHCAmZmZsm4NfREZ5/WzRxQErJJAKFF8nvytpRgWEZrcKdFdicicnw1uiujmiwinV2M8/eMZkqqGpum8d6CR+/a35p1mHA6HCYfDGQ0LZmMWj8ezpQY0N9P49fXyxcaFzOwwlj3vLlfr+DvNSxe2qehWSrboGpNuRVEs2ci73BM/Go0yMjJCOBxmYGCA5ubmTRXbpaUlxsbGqK+vT89YC4fDZd+idjc4uXgt9VxF1VC0twW0GLLfqq7rRGWVWFLd0Nbdt/e39tg+e3YemyTS7Lah6To/HPWxr6OGnY3ONc/NJSjZhuhXxqb5/OkQC985TXuNld+8s4OhjoayPBWK4UaIbqnnaKFFT+PYGfliY4qxuZytWC/dSqdGbCXbUnQrFSdjTprP52N0dBSr1Zox6bZYDHvHUk78aDTK2NgY4doSJ10AACAASURBVHCY/v5+QqFQxslYDvluM3VdZ3l5mdHRUWpqatY0jFRS/fCe/kbeuCAyH4gjAMc6HQTDYV6YC2K3SBzsqqXZk1+E97a6ODfpYzksIysal+ZDeMMyw94odw42cqJvc1I6Bpqu448m6bg+Cl0UBCRRIBwvPlo3StROzcY5MwXfuhgnGAOrJHDVp/Bfn5/lf9wTBTmGqqrY7HZOLQm8Pp+kxmnj5+/o4XB3+bfEWy26G5lDzpUvBtakd1ZXV5FlmWg0uqa+2JzrDQaDDA4Obtjr20y2pehWgq7rrK6uEgwGmZ6eZs+ePWWb3ximN8XcnpvFdmBggP37929IBUQ+4+yVlRVGRkZwuVwcOnQo54j4ckU3KqsshRL01oncvr+RuakxZhcSfGdJw6on0XSBk29ZeWBfE13NdTnrZdtqbHxk0E3Y7uHUxCo7Gp0MNLtRNJ1/ubZCR51jTcS5kYiCQG+Tk6nVOO01duJJFR1oqSmtrvfr5xb4+zfm0TSN1WgSgVT1hRUIyxo+awsfOtiCrus88/o0/3h1DquoM+uP8TtfP8+vHraxq61mTW1xMYHFjRDdzU6dZA8fNcrXOjo60mKc3STzjW98g+npadra2opOR5hZz2EskUjwyU9+kjNnzqSNdHp7e8t/j2U/c5uh6zorKyuMjo7idDpxOBwcPny4om0WY2Qei8UYHR1NR7aG2G4URleb8eVbXV1lZGQEm83GgQMHCo6IL0d0Q3GF7761SDCSYGwhzuQrF3n4vbuZFRMM2mVqnFZ0TWdmJcRSVMdhqpc1Vrc9Hg/+mMIVb4LG1tTx21GfEti5QJyzU358EZlffm8Pu9s8OV/DhbkgSVVnT5uHttrySrAeOtLB35yeY9YfxyoJPHJrB601xW9L13W+dWGRRpeVhKIhCKDroOp6uorDKqb+FgSB7w/7qXFY0+VvvqhMyNXBjh316dvs6enpjDZes0ObJEmoms74SpSkqtPiuHmmRmwWRlCTPasN3s4X7969mzNnzvD1r3+dp556ivr6+qJLvIpxGPviF79IQ0MDIyMjPPPMM3zqU58q22EMtqnoliJahtvY2NgYbrc7LUSvvvpqxa+jkL1jLBZjbGyMUCi0KWJrYAhnIBBgeHgYSZKKjt7Nc86K5fzUCjMzc1i0OC0uCy07evElbQiCjHZ9U4IoYLXZaW6pZ6jr7ZpZwz1rYsnP516ZJxDTsF8NEFJEuupsdNQ6eHk8iKIJ+KNJ/us/vMWf/au9GcIbjCv8398fZ8YfIxhXsIgC/+Xufo725L9Nz/UeIwmFUELhE7d1pXK2FhFJLKfWGARS7mZum0QooSKrGgICbbX2jDSJzSKmj5HxXLvVkvM227B9DIfDzM3NEYlEiCcVnhnWGQ9oWCSJRpeFf39oa6o94MaJbr5OMyNf/PDDD/PCCy/w2c9+lgMHDpTUWl+Mw9hzzz3H7//+7wPw0EMP8Wu/9msVRf3bUnSLwVg8Gh8fp6amJuctdqW3S7nqbA2xDQaDDAwMlN0iXCyapnH+/HlEUWTXrl3rjk43U0qkm0wmGR8f5+JbC9TVt9DdtpOrV65gtQjEFY0j3XV88/w8SU1H08BhFelvyjzedrsdi9XGP51cwS9bkCwa8zEddI0r3gSXFxPo6NTZwCNphBIaT/9whP/zwwNpj4DXJ/3M+GOshJOIIoTjCv/jxVE+99B+2gtEvObP4PxskC+8Momi6UiiwK+8r4db1smtJhSNN2eCyKrGrlY3rTWplfb797fyjfMLOK0SPQ0O5v0RWmtd9Le4+d0PD2Y0dTx6vJMnvjPKalRDR6feaeXuXbmN9K1WKw0NDRnlis+dX2AiPEW9K+WjPO+P8jdvRrCopze1/dngRoluobs1g0AgkD5WpbzGYhzGzI8x0h8rKyslG6UbbEvRLSRiuq6zuLjI+Pg4dXV1HD58GKdzbW7QEMxKVpXN6QWz2Pb395cktuWIfzgcZmRkhEgkwp49e8oayFfMyByzD8POnTv58Htv5YejPhKKRkLVicka3fUOWjxWfupgK5O+OFZJZKjVTY1j7bH1RWS8YRmXTcIbVrCKIqou0eKxshSWcVhE2utTo9Jj4QTxZGpSslGiNbwisRRIIIkSdosFSbCQVHXOzwRo35e/JXsxmCAkx3HbJL7wyiQOi4jTJhFLqvzlD6f404/vxWO3EEko/PXpOa4uRtjR4OATt3XhtEk88Z1RJlaiiALYLBKf+tAA/c0uHj7aQaPbytnpAE1uGwOCzvtvP5Lzi39HfyOPP2jlR6M+nDaJ+/a1FFxszGbOn8AqidhtqTUERRcJ6RpHjhxJD33MnmRsTlFU2qxwI8xuiv2OmkX3Zmdbim4uzDWoDQ0NBa0d4e2ysUpEV5IkotEoly9fJhAIlCy2xjZKEX+jiSIejzM4OJjuoCqHQq/T3Brc1dWVNr3RdR1NF3hrPgTATww20VHvRFEU2mrsdNQVXvwSBCHltrUaIa5oWC0Cqg7tdakWXG9IJhRPpm7DRZGPHd3JvqHm9Gtyzvh4fnyUmJxEScrEkjptboHZ+QXmG9Sc4nJmIcnrV0aRRIFEUiOaVBloTkXhTqtEOKGyGk3itkn8yXdHubwQxm2zML0aY2w5ygMHWphYjtJel4qk/bEkf/X6LJ++bwhREPjw3pb0qPfXX18seFwPdNZwoLO8hduhVhcvvJWqvBCAWFLlcKs9b/uzuVnB8C82m9uU2v58M3vpJhKJnMHVehTjMGY8pru7G0VRCAQClY36KvuZNwmapqVNyxsbG9M1qOtRqdNYPB7H6/USjUbZu3dvybaOBsVO9DUW5CKRSEYTxcLCQkVG6NkYvsSTk5M5W4MFQWCgxc1Ai5tXw2PsvJ5CKPa9N7mtuGwSYVlD1SCRUOlpdFJ/PSq+d28zJyf8iILAxw+38f6ht2/hRFHkQE8z/+H9Gl94ZQpRgF3Ndpo9Fk4MNqAoyQxvBZfLhSw6+N6kzI5WF067jUhcYXQ5QluNDY/dQiypIompC8FKJMmVxQjN7lT1gMsmsRRKMOmLZ9QVOywigVhu8/dcdy2KpiMJlZc63r27mWtLEb57ZRkB2NXs4GN78wu4zWajsbEx5+JTOBzOMEMvpv35ZhXdSkrZinEYe/DBB/nKV77CHXfcwde//nXuvvvuij7LbSm6xm2xYVre1NRUlI+umXJFNx6PMzY2lu6AaW5upq2tbf0n5mE9pzHz/gYGBmhpyZzJtRG+tvB2WmZsbIzm5uaiW4NLTY3MBxOsRpMc6PDgC4QIqFZiSQ1Z07lrqJG+ZjcfO9xRsCLh3n2tDLV6ODMVQBLhtt56uuszoxxj0sKl6RVAJxYJEwlqCKJAk13HF4oRiknogkiTy8rvPneFVo+NpKqhk1ocS0X1MNDs4kdjq8SSKlZRZDWqcN++3LXVgiCkj4esaHz+lUleGfEhiQI/c7STjx9uK/sLKwoC/+HOXh69rZukqiEHV0qa/GG8PqNZweyQZ9THRiKRvO3PsixveXqhFFezco5rMQ5jv/RLv8QnPvGJtBfKM888U/J+MvZZ0bNvELquc+bMGerr60sWW4NSRdcYxeP3++nr62Pv3r14vV4CgUDJ+zaTz8jcGI2zurpa0CC9UtE1qjtGR0epra0t+k4B3nYpK+Vk90eTWCSRNredGuI0NTczvRpnV6ub8ZUYU744OvDwrR30Nq2tLTbob3bRfz1FMB+I8+RLEwRiSW7vreeDu5vTjQt7eu1YpWns7lo8diurEZkeh8q/O1rH3EqIv70YYCkG9S4Lk14JXdNYCsWxWVLlWbd013LXriacVom/OTNHMKFw9+4mHrp1/Rz6X78+y0vDKzS5U+N5vvb6DF31dm7vrSz3WHv9rmAusHF1s9n1sbDWT8Hr9RKPx1leXl6Totis6c/FpN6M/HW5rOcw5nA4+Lu/+7uyt5/NthRdQRA4fvx4RdsoVnTNc8/6+/szRvFUOlgy1zZyjcYpdDIXm57IxerqKtFolPn5+bwNFGYuzAb41oUFEorGkR31NOml39o1uW2g6yRV0AFfNIndIrIclum+7ioWiiv8YHiFf1tAdA1WIjL/7X9fJSpr2CwCb86GCCdUPna4HUgJ1McHrby0orMYSlDntPBr7x9gR4OTuuYYjolRempsqVXyZBJFjXNLs44vFqPNbeGuXo2F+Xn2N3v4n/9qT0m312/MBPHYJURBQJQERAQuzIUqFl2DzR5Kmd3+LIoioijS3t6eTlEEAgHm5ubSDm3muuKNGBFUzEU9EAiUVLVzo9mWoguVR3jria450sw396yY5oj1MES30GicQpQz8icYDDI8PIwoijgcDg4dOrTuc6ZXYzzz+ixNbhs1disnx3206SonShTdtlo7H7+lg78/M0MwqtJXL3Gsp46Tk2/fMVhFSCSL+2zPzQQJJ1Rar3eS2S0a/3RxKS26ADvrJP7kJ/YQV3Q8dku6HtdlldB0HVUDi8UKogWXS+QXPrSHOqc1Y0T63Nwc4XC4pIWoZo+NuWACp9VIVeg0l+BPsR6apm2Kp0M+jKjTPK/NnFozt/BmjwgyV1Hkm/5cLtvJ7Aa2sehWijGGPRvz+PT1hkwWao4oFkEQmJmZIRQK5R2NU4hSLj6RSITh4WEURWFoaIi6urqim0RmfFEEAZzXJx20eOxMzWllXfiO9dTRXydyZWSc2470448mOTMdxBeRmfRFGV+J0Vnn4FsXFjm2s75g/W0ucpnpWESBOmfm6d7otnF8Zz3fuezFahFwWiU+eridOmfqVjVXrax5ISoYDDI7N4dsMmqRZZlAIIDH4+Hfnujm0//7KqvRJLqemhzx4Q0cz6Oq6qbU4xbaX6FIP1+KwmiKCYfDLC8vE41GgUyXsVLan7OpZGrEjWDbiu5GmN4YFnOQKba9RYxPh8oiXfNonJaWlrLGDkFx6QVz5cPQ0NAaM/ZibuFcdguqqZ0qllTx2N4euaNpWtrIuhicNgs1ttRU3GaPjZ852skXXplkLpigo87O+EqU/+cHE3TX2/m527r54O7chehHd9RR77TiDctYJYGEovNvT3QX9RpeG1vlh6M+ECAiaxzqqs27QGZgLESdmYvzV69HiSfhtt5WPnG8AzURY3l5mfn5eSKRCKqq8u/321lIuKh1OznW34LTunER3manF7Ipp3qh0Ej57KkUiUQio/252GGj28lhDLax6FaKJEkkk0kSiQQTExPpHGoxYmveRqmRbvZonN7eXux2e9mlOKIo5ozYNU3njckVzl6bRJCjfPjWoZytyMUuhu1tr2Gw1c2YN4oggl0SuaMrNa13ZmaGiYmJdNTtdDozbifz3X6b88Fd9Q5aamy01tp58coybptEQtGwWUT+4dw8h/O4ltW7rDz+4G6eO79AKK5wbGc97+1fmzPNVcb19I9nqHNaaa2xp4zYF8KMr8TSC3T5uLYU4alXp6h3WvHYJX487sdhkfjF9+zAZrOxZ8+e9PszouJwOMzU+Oia3KeR/ywnTVC2rais8uZsEIsocEt3LVapuG1omrZhJWOiKKbfvxlzSmd+fp5YLMapU6dwOBxrUhTGZ7qdGiPgXSy6kHLiWl1dpbe3l127dpUcPZeykKZpGrOzs0xNTWXUv05PT1eUF86VXkgmk/zDq5d5bWyVzpYGLM4OXp5J8nCbjs0i5Hz+el9em0Xkkyd6GFuOklQ1uuocXDl/hrNnz9La2srx48fT4m2uAzXG4JgjmJqampxf3jqHhRl/PDUFWEz5FDitFhRNIxhXMkRX13WCcQVV02lyW/ml9/SUdNwSSZWkpmGTUl8BURAQBYFIEUbsI94IOmC3pI5Zg8vKudngmsetV54VDodZWFggHA6jqmrJY+XLEd2FYJyfe/ocUVlF16G7wcFXP3lLUQMyt6JO15zSMSZP7N+/n3g8nr54GUbogiDwzW9+k4WFBRobG1laWip7UKzP5+ORRx5hYmKC3t5enn322TVCfu7cOX71V3+VYDCIJEn83u/9Ho888kjJ+9q2oltuesGoDvB6vYiiyIkTJ8q+RSvGMMYYjTMxMUFra+ua+ldjekS5mNMLqqoyOTnJ7NwcV3wuju3tx3I9ipn1x1gMxtnRmBnFlWJ6Y5VEdrd58Pl8XLv4BrIsc+jQIRoaGtA0DVmW8wqN2cBlenqacDicqqO9dCktMh8YrOevzyzijyaZ86faiR0WgY56R4bgKprO//rRFD8a9QECh7tq+E939RY1vNLAZZPoaXAy44/T7LYSTqhYJSFdQVGIWocFXX87LRNPqiVNvMiX+zQLi2FpaG5aMP4YUXE5ovuHL4ywEkmiaTo6MLEc44uvTvOf7upd97lb3Rxh1OgKgoDT6cTpdK4xQtc0jS9/+cuMjY3x6KOP4vV6efXVV0vuTnviiSf44Ac/yGOPPcYTTzzBE088wR//8R9nPMblcvHVr36VoaEh5ubmOHr0KPfee2/JqY1tK7qlYojt8vIyvb297Ny5kwsXLmxaTix7NM7x48dz5jvLqT7Ifr6iKExNTTE9PU1XVxe3336C0y9PZFyYNC1VXlXrtKYXioznF7sYZlQ9SJLEgQMHeP3CVf72nJdgcpkdDQ7uGWqgzpU7p2u1Wqmvr2c2ZmFUseJ0N+ARUn4ORl4vFAphj0ZxWzQEHUBjIZjgvv0t6dpUgBeveHllZCVlOkOqNOsbby7wM8e6cu4710VFEAT+8129/OUPpxj2RmhyWfn379tJgyuz3nMhmOD7V5eJJTVu3VHLLd213LaznpeGV7i2FEEQwCaJfPL24vLI+cgnLPkqAhwOB/F4HIfDgdVqLTr/OeWLoWqpNmIBSKgaY8uRol7jjRDdQmkXY77f888/z8MPP8yDDz5Y9r6ee+45fvCDHwDw8z//89x1111rRHfXrl3pf3d2dtLa2orX6333iG6xka4sy0xMTOD1eunt7U2XYhkLPxtNvtE4+aik1tfwCJ6bm2Pnzp0ZLbtHe+o5Ob6KRYJri2EWAzLLoTjNtXZO9DXy3oEmri2FubKSxL0cYU9X/tcYjUYZHh5GlmV27dpFXV0dMVnluxNx6uocNNe6mVqJ8o1ogk+e2JH3szk14edfhleosVuIJpLEAgmO3OKk3ZTX+1FwnL54CJdFJ5ZIEorLXBuf4bTNm474Lk1HsIqpdACA2yZxbak44TDT6Lbxu/cO5s1pL4dl/uf3x1BUHYsk8MZ0gEdv6+JEXwO/86EBLs2HSCgaA83ukoxrSqFQVHz58uV0aWM0Gk2XctXU1ORt5T3UVcusP46spi5ETqvI4e7ialy3WnSL9SQJBoMVL6QtLi6mTaPa29tZXFws+PhTp04hyzIDAwMl72vbiu56mMV2586da0qxyvGSzYexnUKjcfJRjugawj46OorH46G5uXnNqJL3DTRhswj84xvzhOMqTR4rIVmlQYNXx1a4thRmfDnKqi/JhdAMHz2ScsEyk0gkGB0dxe/3E3J2cMGvcPKNFe4/YMEmiSQUqHOm6l7bah3M+CJEZRW3Pfdp9eMJP+21dqySSK1d5LxXZ2Y1zlDr24Y9bTV2zmlBGhwO7A4HCSHB4d1NHDnSno74PMTxhyKQCCOIIvNRsOgKJ0e9HO9rynn3Uugine93l+ZDRGWVrvrU52iTRL53dZkTfQ1YJXGNHeRmj0Q3MKJim83Gjh070rfSqqrmnWRspCb+83s7GF+OcnUpgq7r3DnYyCduKy5K3+pJFRs9lPKee+5hYWFhzc8ff/zxjP+bW7lzMT8/zyc+8Qm+8pWvlHU83nGim91kUGrda6mIopieSFFoNE4+ShFdI7IdGRmhpqaGW2+9FUVRGB0dzfG6BFo8DobaPMSSKi6rBQ2dxWCCOpeFc9MBDnTWYpUt1NRY+c7lJY721GOzpNIVRt67r68Pn7WVfzg5Ta3DQlLT+Z8vhvh37+tFAzQ1lZqQFQ1RoOBKeK7TOPvcvm9/KxfmQswHEgC019p54EBbhpPWz7e04dXGuLIYZno1hqLqLIUSPPHta9zdLfCeLnvGot2GXVwpXOWx1SPRs0Uwl9tYdp1s2Ovl1/cnCQzZqPW46WyqIRjw54yKc7GV768U0S2meqHQNIm2tjbm5+fp6Ohgfn4+74JcMBjkgQce4PHHH+fEiRPr7jMX21Z0sz/8rRZbSLXRRiIRpqen1x2Nk49iRdcYw2O32zOE3VjhzYUopL50jS4by2E5vdCUVDXsFil9RbcIApquk1AU5mbm0qbNtxw9jsNq4cvnrlHvsuK5HsHOB+JMrERodIr8aCJArTNOs8fGT+5vxmbJf8zv6G/g+1eXcdstxOQkNTaB7vrMuwGP3cLvfWSIUW8qXdDf7FqzQOawSvzuvYN89y0vX3xtiq56BwICiqZxNqDxK/ftI2qapxWNRnn99ddxuVxpIS6mGP9AZw0vXPayGEzVAEdllY8eas/7+Bstut5Qgj99cYyJlSi72jz81gf7aXBZc9bJFhMVZ3ePbVUkb1BoaoSZYDBYccmY4ST22GOP8ZWvfIWPfvSjax4jyzIf//jH+eQnP8lDDz1U9r62regamMW2p6enZLEt54tiHo1TW1vL7t27S4puzawnuqFQiGvXriEIQs4xPIUWwrrqnTS77SQUDW9YZjEUp7fJxb372vjxuI/lcAJFhflgnHa3xJunT9HR0cHQgSP8zZk5vG8MY7eIRBJKRmOEqsOIN0pYhl3NTmQkGtzWnPPMzBzrqcNtkxhfieGyClhrAzkrDuwWkX0dhT1nJVGgyWPFIgi8tRAhklARBJ0mtw1BlKivr0/fcoZCIY4dO0YsFiMUCmXMIjOMvg0hdrlc6fOnyW3jv9zdx0vDPqKywpEddRws4IV7I0U3oWj8h7+9yEIwgUUUWAiuMOmL8eVPHMaSYwxRvqhYlmVCoVBG95hRkSLLMj6fb9MmU2RTbE43Go2W5aVr5rHHHuPhhx/mi1/8Ijt37uTZZ58F4PTp03zhC1/gqaee4tlnn+Xll19mZWWFp59+GoCnn36aW265paR9bVvR1XWdkZERFhcXyxJbeNt/oViHolAoxPDwMLqup0fjnD9/vqLqg3yiG4lEGBkZQZZlhoaG8uasCom2zSJy/4E2Rpcj3N7bQJ3TRneDE5dNYk+7h2+dX+DiskKT7ufOrlb2776NKb/Mf372AgtBGbtFpLvBgSSKqJpOQtFQNJ0au4SsqLS6Jerq7HjcbmZWYywEE/Q15z+lBEFgX0cN+zpqUFWVc+emyjpmmq7zwmUv37nsZXg5mp5TltR0onKcH42tctdQpsm02S/AjOGiFQqFWFlZyWhRNYT4YwebmfTL/NPFJb53dZl7djdzsGvt4tONFN2x5Si+aBLn9YuYRdSZ9ceZ88fpKXKqstngJjsqjkajrK6uZkymMA8azb5gbQSleOlWut+mpia+973vrfn5sWPHeOqppwB49NFHefTRRyvaD2xj0RUEgZqaGvr7+8s+4MWKrjEaR1EUBgcHMwSwUjP07DbeeDyeHsMzODi4rkP9eiVfdqvEvo61AmFRYuyzLDLYB319Q3R1deGPJvnKj6eYDyTQBUgoKhO+GI0uKx/a08pSKEGD28bDt3byN6dnSSYF9OsRsI6eM2ebD6MTrhBJVWPWH0cUBDrrHemI7dWxVZ6/tERrjQ2nVSIYVxBImYvrwB8+P8x8IM4HdjXTXlvYcjCX0bchMqFQCK/Xyw/Pj/KX56LIuoBVknj56hKPfbif2/ozvY23WnTh7TSb3SKiaTqi+PbPVF0vmO4pFqNW2G63MzQ0BGTaPobD4TUXLLMYlxsVlzLZZauPeyVsa9Ftb2+vKM+0nmBmj8bJ9iyAyu0djTpdWZbTrma5zMoLPb8U05lwOMy1a9cA2Lt3LwsLC+kvxUIwzuhSmGBCQQQcVgs6Ot6QzI8nVmmvdRD0xfjhmI87+hv5x5MraGKSoBqnvdZBZ31pk2kLfXahuML/+4NxLs6H8EWSNLlt/PY9/XTVO3jqR1PMBxMsBOLYpFTpmE0SsFtFQnGFYEzhH84t8KPRVX733kGSqs7YchSnVVxXhCH1mZon9D47NsxCPIYoCOi6isOi8TevjcLyOG94dc4uCzjtVj5+sJkb5QDQ2+TkaE8dr08GUDUNSRR4/1ATbTUbkwbIno+WbftooGlaOldcaVRcjOiWYnJ+s7BtRXcjyCe6+Ubj5NtGJaKrqirxeJzXX3+dvr6+krwfoPgrfCwWY2RkhFgsxtDQUHrhYWlpKS3aY94IgbhCvdNGOJEkmlSwqAIem4W+JhcNbhuarvP6hJ+f+GATH+h341es7Gxv4kCnBy2ZADbG5f+fLi1xaT7ESiQJus7ocoRPfeMK+zs8qLqOXUrNVrNZRHRSEx6isoYkpMS3rdaOrGj8/bl5Tl6V0S5fQdPhA7ua+JX39ZR0jF8dW0XXwSIJ6DrEkjqKtYZgXRvfPjeJXdJZiSf5sx9M80ifykr4JPUeJ031tZtu8m0gCgL/10f38Nz5RcaWo+xuc/OTB8qfUpFNsTW6oijmHClvjoqnpqbSg0aNxU3zcTIoRlCDweC28tKFbS66xdyiFsJqtWaI7nqjcXJRrr2j2WVMFMVNq7ZYL4I2TwRWdZ1dbR5GliIkVQ2LqtFe58Blk9JdbKIgIAipVtz+RgdutxurVePNs6fT78tYATdyorm6pdY7rvOBOJGESlJRWYkqKbGTEwTiST6yr4VL82HiSRVF1TnUUcNqTCEQT5JUdVo8NmodFvwxhR+NrhJN6OyoS10wvnd1mSM7akszEtdTlSDK9U4uDdjd5ua7by3jskm47SlhWAjEeXZMp6MpNc34wT0q+9UAs7OzaQct45gYRjcb+ZlbJZGHjpQ+FboYKm2MyJXGyXYay17cTCQSRCIRPB5P3uO03bx0YZuLbqUYkW6xo3HybaOUSFfTrVSxgAAAIABJREFUNGZmZpienqazs5MTJ05w8uTJDRdcVVWZmJhgYWEhbegz6YsxPeWn3mmlr9mNKAoZ6YnOOgfNHjv9zW5WwgkWgzLv39WENySzEEzQ4LYSiCVp8dho8thYnZNZWFjA4/Fw6NCh9K2gsQIeCoUyPATMQrxeed1As5sXryzjuy64gpASvlhSY3o1zm299ayEZVajST77wC6Sqs73ry7z/OUl2mvtRBIq8aSKjo7R2Wt0sC0EEyUdy4NdNbwxrZNUVTQdai0Sd+9q5pkzc6imi/5KJInbotNak4qy//fVCLfcN8ThAWf6uJj9J4xob6NyoJvJRjqMGeRzGjOO0+LiIjMzMxmDRrOP03bz0oVtLrobces0MzPD+Ph4UaNxclGsYY2u68zPzzMxMUFLS8uaKbsbhVnUzaPTfzzm443pAHaLSFxR2ddRw/uHmjM6827ZUc9SSObN2QAOm4X7DzZw3/424kmNb12YZ8QbpcVj4749jbx16SJ+vz9VYjY0lF5YyV4BjyVVTo778UXitOngVBRmZ2fTQxAvXbqUFuKgauX5t3yEEgpHums50VvP372R6iAyGi90YCUqsxyW0XX42WNddF43m/mFO3awt93Dd97yAvBzxzv5zpVlTg5HcSVVpnwxorLKhdkg9+9vLdrS8LfvGeAz37rG9GrKf/mX3rOD/Z01PEInf/D8MCsRGU1P3SnsqE19pjaLiCCkBN6oHsgX7eXKgWbXy7pcrhu6WLSVLcDGcbLZbOzbtw/I7b/7pS99idOnT2Oz2fjyl7/M4cOHOXjwYMnz0opxGDMIBoPs27ePj33sYzz55JNlvb9tLbrlYtT2zs7Opv0RKqmAKBTpmgc/NjQ05B2kWenKt3mab7aoR2WV87MBOusd1xeDdK4uhrmluz4j0pVEgfsOtHHn9XIro53XDoQSKi8PewlGZf7x9CT/5lgn9/TtKHiCy4rGl16bZtYfxy6JnEqq3LuvlQ/sTdkwnjx5kp07dxIKhbgytcgfvLRMWNaxWwROjS7x0KFmuutsLEcVbJKIKIKi6vy79/TQ0+CkwW1dMwX4RF8DJ/oaWAgm+NJr00ytxIgkdRbmQmi6jiDAN84vcn4uzP/3swepd63/BW322Hjykf0EYgpOq5iuLT7QWcPjP7WbH476sEkCr436CEdT0auqpfLM2QY62eTKgWbXy3q93rS3gjk1sZXNClvtu5D9fcgVFf/5n/85f/VXf8Wrr75KIBDgySef5LOf/Sw7d+4saV/FOIwZfPrTn+bOO+8s701dZ1uLbqkipSgKk5OTLCwssGPHDvbu3UsgEKjo1r5QTtdo2XW73QW9GIwKiHIjX0VROHnyZN5pvppRy3j9eAmCkMpNXj+xs6sfsr0TXhle5uuvTxKJJ2lw2pB1kX+6GqCnVmRfW/7XPLUaY3Y1nvYuUFSNf7m2zPuHGq/nhoX0F+lzr63ilwXcNitJVcMb1fj+sJ9fOmDlL8/KhBUNTRO4d1cdd+50FcyHRmWVP3x+mEBcSQ2GBCRBRxQEtOv52ZGlCP/9n6/xP/71vvRxKYQoCDkFdKjVnfaOONxu53MvjrIcllF1nXv3NjPYUnrTTKF62Wzrx1OnThVtGl8JN8Lsppj9qarKkSNH+I3f+I2y91WMwxjAmTNnWFxc5CMf+QinT58ue3/bWnSLxbxo1dXVlR6Ns7q6WrHTWK5I1+/3Mzw8jM1mK6o9uNyJvoFAgGvXrpFMJrn11lvX5MYM3DaJ7gYn06tx6p1WQvEkLTV2ah0WErlM0FWN7761xPmZIJocZWVllYQq0FDjSvnzKhqJpMpSRGFvgWhL0zO9FYTroqfrZBgxJFWNK0sRbJKIJApIokRUVsFq5yffd4CL0RHemg8hojMXlDlzZQI38dR7MzUxGP4B06sxgnGFJndKJD1WWJVTopuqQNCRRBhZjhKMKUVFu8XQU2/nP91Wh6e1B49dKsljtxgkSUo7jimKQjQa5ciRI+uaxhuRcSWiebPZOhr4/X66unJbehZLMQ5jmqbxW7/1W3zta18r6OFQDO9o0c0ejZM9h6zSxgbIjHTNHWu5WnYLbaOUxTjzgMndu3dz5cqVgj3qgiBwz55WTk/5WQzE2d3m4ejOBiySmNNt7YVLi3zv0jxSIgQWGytCDQ5bgrii4RYFkqqGyybR7LYVvFjsaHDgtktcXYzgtqcmQbxvoDE9jddAFARq7BYiCZWEoiIgoOo6PzHQyMmJVaauO5EJgsBKROZsyM1v3n0wnQ8NhUJcmVzg/GwIEZXOOgfRWAKXpKW8Zq1gkURkRU/fthpWjBvRPGCg6zpum5ThmrZZGAtbxZjGGzn0XItRxXgbwNo63c2mWNENBoPs379/3cdV6jD2+c9/nvvvv5/u7sp8k2Gbi26+W6h8o3Gy2SjRlWWZ8+fPk0gkGBwcLNl8o1jRjcfjjI6OEgqFGBoaShel52uQuDQX5OJcEI/NwnsHm3hP/9rmjuznhkIhvn1mmBq7RGtXO1arFWklSovHzhszQXyRVKvpAwfbOdhhI1lgEdEXSeINycwH4yQUjUOdNTmn4UqiwCNHO/naqRnCcRVF09jXUcsjRzt54bIXi/D2Z+2ySayE5fRrr6mpYS4q8Cen5pAVAImuOoHDHW7OzoXRtShJRefBPonzKzATUrFZRCyiwMO3dvD1N+Y5OeGnxmHhF050r+sfUQhN07ZssUtV1YL7yjXJOHsxampqqujGBU3TtrSqotimh2AwWFT1QqUOY6+99hqvvPIKn//85wmHw8iyjMfj4Yknnlh339lsa9HNZr3RONlUKrpGy24gEKC/v79gE0Uh1psekUwm01Mv+vv7+f/ZO/M4Oeo6/b+r7+7pue87c08SkpBMQrhE5BQFUURRV0EFj134iRtFTl1kV4Ku67HoCuouUVfXGwUVAUHuXOQgCZn7vu++76r6/dGpSvVMz0z33Al5Xq+84JVjqqq76qnv9/N5Ps+zbt26mOPEI+29nRP8dE8vFoOOkCSzr3uSOy6vwW6J/cqVmq7f76e1tZVgMEhhbhZ6owmj8UR+mE7gQ1tLue0dRkbcQYrSo9NnQ0NDhGYoL/hCIvc80cSgM0iO3cT6AjujnjBNwx42aXwLRt1BBl1BtpWlUZReRdOQhxy7iXfUZmMy6KjOtSHKUd3uhDeMPyLy3ilOX99/qRtRkkm3GpFlmV5HiHedVcYlZxUz4g7iG+nm+rdvxun28ELzKAOTHnKMEVo7ezg4KpNhNTLh0fHAX1r45nXr5l0WWM4x4PkkAc8k0dJaP46Pj+P1elWvCuXvh0KhBZvKJINEexyLodNNxGHs5z//ufr/u3bt4vXXX58X4cIpTrrKDZ5oNM5UzJd0leif8fFxKisrcTqdMQ2PZDHTSldbiy4rK5sxzy3eSvepN4fJTjGpgYO9Dj9vDrrYXhG72h12BWnqGWVgZJyz6mvIyckhfcTLrt09OH1hJFmmIjuFdYWpmAw6yrNPNoZmGk6RZJkHn26jZSSa8NAz4cfpD7Mm24bTfzK5+NiYyL//6pj6bz55XhkfmxJ7s64wlbOLU/n5/gFkooGVTx4b5tyKDDYWR1c4Dl8Y0wn5l5oQ64/w3k1Rct6/vw+z2Uye2cwHc06OrN70k8Pkpckgi+giEcbdYf7w0iEuXHNST5yamjqnBaSC5STdxTQUjzfOO9X6cXR0lJGREQYHB2dM5V1MLLaX7mxIxGFsMXFKk65WJpVINM5UzEf90NXVxfDwsDpwIAgCHR0dyZ56DKaSrizL9Pf3093dTWFh4bRa9FTEI11RlGNqpzqisewn/1zkJy8c59nGEYwGPUX5uVTro3XTyhwb56zJpHPMQ3l2CldvKIhb+1SOK0kSR/pdtA+7yEwxUZJhpmnIQ6bVgDsoYjIIOP1h/CFRlXj5QiK/a4uQmWbBbNARFqPysnPK08lNjf0OO8b9VOVYGXAGkYAxd5ivPdXG19+3ljXZNs5Zk8Ffjo1g0AlEJBmdAJtK5q6nW0x6wqKE9cRuyE+I9WvLqSqy4na7cTqd9PX1qVNSWiKOp5s9VUk3HqZaP8qyTH5+PmazWV0VDw0NTYuUV6KCFtp0W07STcRhTIuPf/zjfPzjH5/38U5p0hVFkcnJyYSjcRZynN7eXtXce7FHdhXS1cbwZGdnz1ke0f77qaT79tocHj88SLrFQCAiYTMZqCtIVYc0XjvWzsu9eqqLc/D7vCDDrt293HNVLTufbuVQjwOAV9onsJn0vHN9/rTjBiMyvzrq4MDf9uEKRNhQZCfTZiLFpEeSo5NZohzEFxKJSPDus/KozImulB3+MLJ8MsbcqNchCBJj3vA00pVkGVdAROKEm9YJL8dnGsf49IVl3HJ+Kd6gyMvtE5gNOm57+5qYKJ2Z9Kw3bi/mP//ehScY/exLM61sLcvAZtJjs9nIzz95zdotuOIzK8oC9lQ7WelRj4W56qyLieWOzlG2+zOFZ05t2kmSFNO0S2bHoPzMRBZQHo9nRtXOasUpTbpGo5H6+vol+/nahlwiK875Qq/X43K56O7uJiUlhS1btsR9iXgCEQZdUavD0kwrphMrxB6XyKTgZq3Rht1sYG/nBE5/mIbyDDyBCKkWA+9cn48ccLP3aCvp6emUVtWTNjmI0aDHJ8ukWw0MuIK8OeDijV4nuanRByQUkXjstR4uX5s3TXXw+LFxDg/6CUgGUkx63hz0cFl9Dt5QhLp8O43DHuwmHUa9wLoCO+87+2QtNifFhEkP3qBIilmPPyyiEwQK06Y/aFety+Wbz3UQkSTkMBh0Arl2E6ETUUEWo547r6jic6FyjAZ9XNPueLigMossm4ljAy7SLAbeVp2tlmOmQrsFD4sS33i2nZfaJpAlJ5dWRrimyoXT4VC35VPLE4uNlSDdmY5nMBhiTOMBtU/g8Xim7Ri0RDyT21gkEplTarlYXrrLjVOadGHhpjdKI0n7xWlrxLm5uQmtOOe7tXS73fT19SEIAps2bZrxrT3hDfHbgwMEItFVWV6qhWs25vPbg4McbvdiNUd4odOHzaxnxBXEatLjD4mcV5nFOyrtNDcfQ0RPKKOcIcGAVYiuRsMRCVmWGfOGqMi2EoxIMZ6sRn00BiciSuh1sYT05pCXDLMOX+CEA1cIxtwBUkwGbtpWwBsDHh4/MkJQFBnxhNm1p4+btpeg1wmYDDo+WGvgyX6BSV8Yk0HHnZdXxtXMXrE2lwlvmMf29GExCpRlWhFlOL8yuq30BCM88JdWDve50Anw8XNL+dDWIvXfz/a9rC2ws7YguZXSz/b28VLbBGlmA7IMf+/ysWFNBQ2VWfj9fnJzc9WECkUhoB3rTU1NxWi28OgrPbzQOk52iokvXVZJVW7iUrOVIN1kFhxa03jtjkHrP9Hd3R3Xgzc1NTWpYaFTyUsXTgPSXSiUZprJZIoZ2U2mRqxs75O5KX0+H21tbQSDQQoKClQx+0zY3TGBLMuqz0CfI8Czx0fpHPNSlGbCZDLiiEi82ubgkvro1i8UCvPXg+1k+kzU1dbwx0YnQ8POaJKvKLF9TQZ/OTZM73iI9BQ3F1VnU5Vjw2zQ4/SHsZn0THrDbCpJxxwnVifdamDA7yc7xcKIK4Aoy/jCUJ5tpiw7hYN9bswGgbIMGzLwYssY5ZkW3lGbzQut47w2IFKXZyPLZuJj5xSTmRJ/RSgIAh/eVszGkjSeaRxFlODimiw2n4gOf/iFLg71uUg165FkeGxPLxU5VtVJzB2SGXVHVRSL8YAe7HNh0utOaDqj53e438WWHHvchIqpY70jIyM8esDF7kGRkAQCXv5h1yF+d8sWijMTm2Bb7aQ7E+bynxgbG6Orq0sd9sjIyJjRrW45JXqLiVOedBfL3tHj8dDa2kpKSgpnn312UvIYZUAikZsyFAqpsebV1dXk5OQwMjKCx+OZ9d95gyIW48mHzKQTcAfC6AUBAQFZlrAYTIQlGUmUmJicwOv1YbWlcfbmDfQ7Agw5A5RmRq/LHxI5PujBZtRTk6mnMC896tCVbuH+q+v5wYudjHqCXFCVzWfetibmXGRZRpIkPra9lH978hihgBMiArlWI+cVm7h0fQ52q5muyRBpFqP6uZgNOjrHvBSnm3j2+DCpRpmCVBNDriDPNo1y/ebCaUTiCkT46d4+moc95KeZ+cS5pepYsYLDfS6sxigJ6oWo78GbAx62lWfwH8928NSxAPajRzirKJV7r6yeMSI+URSkmmkb8XHC7RJJlilIM8+424k31rv7zy8TOlGGl4n6VPzv39/gklIDNpstpjwRb5e13KS7FC5jCrT+E8pk2BtvvEF5eTmRSCTGrU6RvSnRQYkOIM2ERM1uenp6uOWWW+jt7UUQBP7yl7+wZs2aeR3zlCfdhUKSJI4cOYLFYmH9+vXzKsonYu+oVT5UVFTEOJrNpdMFqM5L4cXWMUyGaF5ZSJTYUJxF14SfQFjGLEh4xDD5FnijrYe8zHR0qblsK00n1WLEE/Qw7AoSikhk26PNrgFngAyrgQm/TJ8jQEiUeLV9gh2XVbPzfevwhUTsZoNay1VebkrTb21RBt/96Dl0jPrQySKlKTJ+n4fBnk7am3xIbokxh4w+zYLBYCAsQUmmjcbOftwuJzVFueh0OrJSTPRO+hFFUf0clM/m4b930jLqJdNqpGvcx0PPtLHz2no1mRggN9VE64g36kJ2wtQmN9XEf77QyS8ODCAArlAAX0hk154+bn37mqS/Yy0+dUEZbw66cQUiyEBpppUPbC7EPTGS8Mor6vdwcrGg1+moKC+nYXOBGhWkdR2zWCzTtt/LSbrLHUUkiuIJr2ZjjBwzEong9Xrp7OzkF7/4Bc3NzWzevJmqqiruu+++pEMiEzW7ufHGG7n33nu5/PLL8Xg8C/rsT3nSne+NoKxs3W43FRUVlJWVzfscZjO90Y4il5SUxFU+JDKRtqkknVBE5I1+F0adwFXr86krSCXFrOdXu1sZdvgosYa5fnshveEyxrxhyrNsXFidjTcYYX/XJOPeEBO+EB1jXvJTTdTmp3Ck38WgU8QeCeELRTjY4+DVtjF27emNEnSKiR2XVVOcYVHlYcqopCAI5NrN5Nq1JZiTXe3a9X4eerqFvkkfkYifQquIfngSBBsWqw1BF12dekIyVbkpaolHOY4rEKZp2I3dpCcYEUmzGJjwheka83GWZsDin99RwRd+34g/LCLJMmsLUrmgMpOdz7QhECVhBJjwhTk64E76+52K/DQzP/zIRo4NutELAhuLU7EY9biSIKaPbS/mZ/v6CYQl9AJYjDouq8+JO8AgyzLBYFAtTwwNDeF2u9HpdPh8vhip1qnWVJoJM9V0DQYD6enpXHXVVRQVFfHDH/6Qxx57TFX8JItEzG6OHz9OJBLh8ssvB1iwWuKUJ91kodRSA4EANTU1jI+PL1huFm+lq0izOjs7yc/Pn9U/NxHS1esEzq3M5tzK2Bsr2xDi7VkezIVmNm+O2kZOfde3jXgIRCSuWJdH85AHVzBMlt3Mpy6s4MbHDiDKMhFRIsVswGzQ8e3nOyhMM5NpMzLhDfGtv7Wy89p61RlsKrGIksy+rkmGXEFKM6NZXYIgkJ1m5WvXbeBY5yC9PT3UlxSwprwMv9+P9/AAB3oduP1hQqJEppRCU1uI/Ox00tLSMJvNWNEx6ArhD0sIAtiMOnLtJoy6k969AGuyLPz4HzZwfNCN1WTg7JI0+h0BjDodAUECWUanExAlmUzb4tzyqRYD51XEbkOTmRL7f29fQ1G6hb+3jJOTYuQfL1pD1iw1bYvFgsViUaVaPT096PVRaVs8U/SpJkCnGhJZWSsG5nq9ntra2nkdJxGzm5aWFjIyMrjuuuvo7Ozksssu46GHHpp3ueUtQ7rBYJD29nacTqdaSxUEAafTSTgcnvsHzALtSleWZcbGxmhrayMjIyOh6bj5hFu63W5aWlrQ6/Vq7Wum40hy1GHLbjbQUJ6BPySe2IKbObskjZ7BAPm5qeSkmmkbjRqjWI16ZDk6IjviDuILSaRZpz+8sizz3692s7tzEqM+KmG7cl0eH9parO4mjAYDV5x3sk5uNpu56eIMajom+PZzHWCQeaFf4sjEOJ/dHETq7CQcDnNoXAeSqK6uHX6JkgwrtYXp6ARBLXPIsky6Wcd5a6LaXFmMkGWNxugEIyLBiIwsShh0Oj51wfx3NHMhmS24IAhcv7lw3vE6kiRhsVji+isoJkCjo6N0dnYSiUSwWCwxdeKlzmxbDiQ6ArxQs5tIJMLLL7/MoUOHKCsr44YbbmDXrl3cfPPN8zrvU55057pxpvoWTI3iWSzTG1EUVUtHs9mcVDMuGdLVBkzW1dWRnp7OyMgITqdz2t+d8IYYcQcx6HWY9QJjniAmgw6HP8wV9VFTjyvW5/M/QyOkWo04A2EybUYCYZGwGE2U9YdELEY9NnP8t/qAM8j+bgfFGVEPV1GS+VvjCFVGB4R81NTUzPhg/OnoMCkmnUrmQ64gQ0IW79q6FlmW2fdcG9kpY+TIEt5QBFGUMIk+Ojs6SE1NJS0tTe1oK8MhSmnCppP5yjureOCpVtz+EGaTiTsvr6A807JkTajVMJE2kyl6IBBQI5S09o9Tp+zi/czVqhJYLrObkpISzj77bCorKwF473vfy549e966pDsTtIbl5eXlM/oWGAwGgsHkMrOmQhRFOjo6MJvNSVk6KkiEdMPhMB0dHYyPj1NTU6Ou1Gf6902DLv77tR4kWUaU4dw1mWTYDATDMudXZlN/Qpt6cW0OHS0G5EwrdrOBi2uyeK1jgj+8MYROENDp4P9dXIHhhAWkDDGm32HxZI03Ikp0Dk8y4g7R4irihvMbov67M8AbjMSMFwvA0QFX1G0sKFKSYUbW6clNTSEXGPOEaKjOJCMjTe1o+3w+DAYDaWlpKtmkpKQgSRJ5OLm3QU9aQR0VBVlYjSfHpbUNO+XXQol4ORUFyRxLEAR1kmyq/aNSJ1Y0s4IgTCtPAMvqpZvotTkcjpjJuPkgEbObbdu24XA4GB0dJTc3l+eff56tW7fO+5inPOlOfQNLkkRPTw/9/f0zNq60WMhKV3EZm5iYIDc3l7Vr187r58xmYq69nvLyctXvQYup3guSJPPTvX3YzXpsJgOiJLOna5IvXlZNSWbs6lsQBNZmG9i+vVRd0bxnYwFbyzNx+sMUpJnJSjGxr2uSJ44MEYrIbC1P571nF2LS6yhKt5CXaqJn1EnfpA9PWCAjxcyLPQEEywAf2Taz/+gFVdn89uAAmSlRj96QKPHUsRHVevH4kIeCVDMT3hCCALX5KXzsvHJSTIaYh00hD5fLRVdXFw6Hg1AoRFpaGhVlxaSlpWDXSNckSYpp2C0WEa+GlW4yMBqN0zSzWqOb4eFh2tvbiUQiasTVfEZ6k0UyXrrV1dULOlYiZjd6vZ5vfvObXHrppciyTENDA5/61KfmfcxTnnQVSJLEwMAA3d3ds3roTsV8SDccDtPe3k5L/zhFpWWUVWQihgLzPfW4kjFtI66goGDWEeSppBuMSPjDIpm2aINQr4vqVz3B2OtUiMdsNrN3716COgujYRPpqXa2V+dTnBFdsbePevnVgX5y7GYyrAJ7OiexGvVcs7EAv9fNpTlu/uKW6dGZqMw1U51nxyAIvNI2wbUbC2bUxb5/cyGiJPFS6wRpFiN1eXaeOj6iZpDZBAFPSGTne9fyzWfbebV9gk/85DD3vauWjRr1gkIeer2e0dFR8vLyKC8vV7fU/f39eDweJElSSUP5pQy/aIlY+S8Q873oTqgtlP+filONdONhqtENnDTNt9lsOJ0nI+UTMQGaD1aj2c3ll1/OkSNHFnQsBac86Sojux0dHeTk5CRsEqMgGdIVRZHu7m76BwY45Emj3ZmKbtyBWYjwvloLVfO8hqk36vj4OK2tUY+ERBtxWtK1GHUUppsZ8wTJsZvxhSLoBIG8E0Yy2lUewKZNm+gY8/KtZ1sZdk3iDY6S8UoHV6/R45YtDPl1BAJgTDOj0wlk200c6XNQoRsjFAqxdcNa1tTq+fe/tVGYZlGPgRCN7JnxvHUCH95WwodPrIafPDrE0yeSfCGaqZaVYuTfn2mj3xkg3WrAFQhz1x+O87OPbyH7RLdf2XGEQiHWrVunToNZLJZpfgBKk0kxFopEIuowglIn1hIxENOwm/p7Op1OJb/TgXRnOpbZbCYvLy+mPKGdshsbG8Pv98fk3imEnGxpItFBo8Xw0l0JnPKkGwqFmJycTNrWUUEipKs1vikuLsZWso6W3b2UZEabOH3jLp5udbF903yvIgqtImHjxo3YbHOPhAbDIvt7XDR2+9DnuFhXmIogCHzy/HL+57VuBpwBzAYd29dksL9rguJ0C/UFdpUgFJJ44sgwY34RZ0jAoDfR6QrxyHHIs4tMev0EIyKGsBujTsAZksk2Sdjqy6mpqYlOW9kkitMt9DsD2E1RctxcmoF9hgZcPFxSl8OvDwzQN+lHJmpsc9P2Ur71XAdpJ8zXrSY9gbBE24iXjDI9PT09DA8PU1VVFVPnjgdtk6moqIg3+pz8em8f/lCYSypENogOent7CQaDmM3mmDpxvIadQsIKAXu9XtLS0ohEIotWJ54Jy0m6M40Am0wmsrOzp/nwKt4Kg4OD6g5DcRxTPs/ZFhLLudJdCZzypGuxWOZdS4XZSXdqrLmyim5vHEGvP0lY6VYjY+75lxf8fj9+v5/GxkZVkZAIwqLEL1/vo3PUg9sRZvTgAJfU5fC2mhxy7GbuuLwGb0jkdwf6OdTrxKQXCIQlLl+bw9trYxsQk74QA5MBTAYdkgBhSSYUjDB+IijSH5FocugotEmk26zccE4+kUiQo0ePEg6HsdlsvKcihT2DJhwhHRdWZ/HOdXlJrfxSTAYe+chGnmsawxOMsKU0g4ocK99+voO9OiZIAAAgAElEQVSIKGHQR20dI5KM6Hezf38zBQUFnHPOOdMIyOEPIxD9buLh+KCbO35/HPGEnK5xxMfdV9Zw2ZaamGEEl8vF4OAgfr9f3U4rK2KbzYbRaGRycpKWlhaysrLIzMyMIWftingxiXi5o4ESPWdteKYCWZbVKbvJyUl6e3sJhUKYTKaY8oTyYkvU7CZR9cJqwylPugu98WZSDihb/Hix5kUZViRJPuG8JeDwi5SkJP8gaRUJBoOBbdu2JXU9g84AvZMBSrNs9PsnKcow80rbBOdXZaPXRR/wCU+I1lGP6lcgihIvtE6wbU0m7aM+wqJEbb79xGRYBLOkQ5ZOqBKI+iXokbHoJXSCjpsvXsvG0swYMpNlGb/fj9vt5mK9C7fbTdA7zptHB1WCUrbtc11fisnAe6bE8dx2cQXfe6GTYCSq2T07B9IkFzWbN0/b3YQiEl/43Zu81jEBwAVVWXzz/evVZAkFfzk2TFiS1RW0PyTy20MDXFafG3cY4figm28808qoe5x1OW4+UDtKJOAlFAqh1+spLi4mOzsbg8GQcMMOTtaJkyViWZaXTVGwULMbRRExmwnQ6OgoPp9PDds0mUy4XK5ZDdHdbvcZ0l0pLMT0ZioJuFwuWlpaMBgMbNiwIa6nZ32Bnas3FvDUsRFApjovhQZ74s24eIqE3bt3J33uoiSrLleSJKFDQEI+UX+MHicUia74hOjFoj8xwPCvf2mhbdQbfSBMerJsRoozrCd0vQImWSAiQSgYRAYsZhMVeanUFKRNWz1qE2kVGz9ltehyudRmViAQwGw2q0SsXd3MhvdsLKAq28LuNzuxyEGu3l4/Yy3vkZe72NM5iXI77O6Y5Icvd3PbxRUxf0+nE0Bzz8iAfobzGHIF2PHbNwmJEia9wP7BIL6wxMdqdNTX12OxWHC5XPT19eHxeJBlGbvdHlOemKlhpy1RKJ+bQjzR84xPxsvpvbAU8evxTIAgWlro6OggHA5PSzFWVsUpKSmYzeYlNeFZSpwWpLsY8Pl8tLS0EIlEqKmpmfUNKggCV67L5+01OYRFGasBDhw4MOcxZlMkKCvuRD1EAYrSLWTZTAy7gnhDEr2OAOesyUBAJhKJPsQFaRYyU0wMu4PYzQYmfWF0OoE3Bz3k2Y0IgsCoJ8SYN8SVa3NpGfEy7g3i9fqJiCJ+2UCaxUhhuoXSTIvavJoL2tWitvmiJeKhoSF8Ph9Go1ElKGXbrpCOUk939fXx7g1rKCgomJWkD/Y6CWuiisKizMHe6YMj124s4JnGUVz+sDre/NFz4svbjva7CUvR2HlJFNGLId4YEvnWh87HfCK8c2rDTrEmHB4epq2tDVEUpzXslLpmsg07nU637E275SI3g8GA0WgkIyNDvW+0Kcbj4+M8++yz7Ny5k2AwyJe//GU2b97M+eefr47zJopEHca+9KUv8ec//xlJkrj88sv57ne/u6DP/rQg3YWsdIPBIH6/nyNHjsTEmicCi1GPxRgl07mGG+ZSJMym1Z0JZqOej24v5ZW2cV53DHFJXTYN5ekxpjRWk46bzi3l+aYxRj1BzipKo2vcR+OgW71x7GY97oDMkCtApiFCWPaxqSKDG99Ww+NvDNE3GaAk08INDcUYZxl2SOiczWZyc3NjdLbKNtPtdtPe3q5uM41GI263m+zsbBoaGhJKYCjPsnKk3wUoXgjR35uKqtwUHv7gBn53aIBgROLdZ+WztTz+6tlm0iNL0akuATCYLOgAk2FmCd9U2ZVS13S5XExMTNDd3U0oFFLHc5WXjsViiXnhTG3YQbQsFQqF1PtlKRt2ECX+5Yxfn7r40JoAFRQUUFNTw/vf/34uuugitm/fzuHDh7HZbEmTbiIOY6+99hqvvvqqKhe78MILefHFF7n44ovnfX2nBenOB5FIhM7OTkZHRzGZTDQ0NMzbGGS2t16iigTFvyHZmzsaxZNHykQzBToXbiekpaXF3LQZViPXaWb8d3dM8NSbI0QkGb0A7kCE80ptpIQn8JpSaGio5tK1edjNBv7poop4h11UTO2C+/1+mpqaCIfDFBYWEggEOHjwYIz6IC0tmks2lWw+f0klr3c7mPCGkRHITjFx+zsq4x63Ji+Fu66smfXclMm2XHOEQb8OBAGDJHPbxRVJrXa0dU2FHKaO5yolGKXBpJ2wU46lvLwLCwtj9NlL1bBTfvZybuMTUS/4fD4yMjK45ppruOaaa+Z1nEQcxgRBIBAIEAqFkGWZcDgck4QxH7zlSFdbT1VizQ8dOoQoiovqxhQIBGhtbcXv91NbWzunnnA+K11tk2bTpk04nU5Vf6r4kWobWcqNfG5FJtdvKeTxw4NEIhJrUsJcWmxjXd1WLBYLY54QQ64gOSly3PicpYIoimq0fU1NTcyklPLnCkH19vaqxu/a+mlaaiq//fQ2DvU6EYCzS9Oxxkm9SAQKweXn57Prlgv4W/MYY54QG4vTaChbuD50pvFcZeXvcrnUBpMgCEQiEXQ6nSqPU0g1mcGO+RDxcnv3JkK6TqczZicxHyTiMHbeeefxjne8g8LCQmRZ5rbbbluQWgpOE9JNZMUhyzIDAwN0dXVNC5lcDNMbBVpFQnV1Nbm5uQmdXyJG5gqmDjdoBekKlDqYy+WKIWKbzUZaWhqXlllZazLjC4TYsHategPv7pjgd4cGT/xc+Nj2EjYWL22HWJZlnjnUwU/29iEYzFxzdgnb4tTW9Hr9tABERRfqdrsZGBhQdaFpJ2RIAY+AMTU1qVp5IBCgubkZiA6OKMZF12womO2fLRq0K39ZlmPCUfV6PSMjI3R2dgKocitl5a8sHBarYQerc6Wr2DrOhYU6jLW1tdHY2EhfXx8QnUx7+eWXedvb3jbnsWfCaUG6s0Gbe5aVlRW3nroYpCvLMl1dXbN6JMyGRExv4pHtTMfQ1sGKiorUf6/4E3R1dWEymdAJAl1dXaSlpSEZrfzmwAg5djMmgw5/WOTn+/qpvdqujuYuBiZ9IQ70RJtb1Rk6Dhxv49E3glgtFoyyjp/u60OnE/hgQ/GcPyueLlSZPHO5XNMaWdqG3dSdjSRJdHd3Mzw8nHR9fyngdrtpamoiLS2Nc845ZxoRKT4JipbY7Xargwha5cTUht3UWrHysyC2Pqz8dyVId67jORyOhKbRFuow9vjjj3PuueeqC5qrrrqK3bt3nyHdmYhHEa2npKSwefPmGc3KF0K6yhiyz+cjHA7PO6Z9NtKdumKZjWznOs+uri6Ki4vZsGGD2gVXRmObB8ZwOBzgkzEajZhMJnxhAYc3QEHG3Em1sixzoMfBnk4HVqOOK9blqZlsCkbcQe75YyOT3hDBYBCLXmZ7VS4Go5NUy0kSfLZxNCHSjYd49oaiJDE66UYKehkbG6PzhGevoihQBmGKioriDlssJxTZlNPppK6ubsZtdDyfBGWH43a7Y67TarVOG3WO17DTNoWV/yoTdss1BZfIcRZjBDgRh7GysjJ+9KMfcffddyPLMi+++CKf//znF3Tc04J0p8LtdtPa2oogCAnlns2XdLWKhIyMDEpLS+e9IpiJdJWHQCHb+dz0k5OT6nlu3bo1ZoWnLU2kZeXydG8LJoMOi05mzO1HJ4bpajlOpxhRV1DKKmrqjmF35wQ/2dOH3WwgIkocHXBz15XVFKSdfNn94fAgo04vKboImekpuMLQNh6M8WiISPKirqxfbR/nX/7UQjAikpdq5j/ev566OhuyLKsvZqVDPzg4yOTkZMyKeDkNv5VSUGlpKTU1NUkfV7vDmdqwc7lcuFyumIaddkU8VaoXiURob28nHA5jNBrVDLulathpkUhqxEJJNxGHseuvv57nn3+eDRs2IAgC73znO+fduFNwWpGu3++ntbWVQCCQUPNKQbKkq1UkKAMUhw8fTir9YcgZ4M0BF3q9jrNL0qeR7mylBEmWCYuSGgM+E3w+H62trciyzPr16+MOemiRYjbwyfPL+MmeXsaCEmn2FD55fhmlmdYYydP4+HjMSlF5cP/WOEaG1aiGRg44AxzudfHO9VHSHR0dpamzB4MuWpsVBAGzHCE31Yw7KDLuDQFRz4WPn1ea8Gc5GwadAb78ZHNUYG/SM+IO8c+/Pcavb95Cb08PIyMj1NbWqk07raJAS1DzGepIBn6/n+bmZgwGA1u2bJmXj8hM0DbstJ13ZdR5qjdxamrUv2NsbIyysjI1RHWpG3bJYDFINxGHMb1ez6OPPrqg40zFaUG64XCYpqammFjzZB4Ig8GQUGTPbIqEZIi7d8LHd55vJxSRkIFnG0f48DoraTpxzrrt0X4n//1aL76QSHmWlc9etGbawIKSlqF8HlNVALOhJs/O/VfX4QuJpJhOJgFrJU95+QUMOqOa1TSjiNfjYWJigonxMZx+kYDViMlkJBgSkKRo3bGlpQWj0ci7Gmp45NU+QhEJhGgyxcW12fzzpVU82zhK85CbllEv336ugwuqsrj5/LIYo/Nk0T7qRQCMJ36GzaRjzB3k76/tp7ascFopYSZFgXaoQ/Fi0Eq7pg51JApFTTM0NBRD/suBeBNhPp+PxsZGQqEQdrud/v5+BgYGpk3YxRt1XmjDTvn7iXyGLpeLNfOMQF9pnBakK4oi6enpMbHmycBgMOD3+2f880QUCclE7jz15ggCAsUZ0XpnvyPAoQE/F5RaVOKOV7cddQf5wUvdpJgNFKQZ6HcGeOSlLu69KhrKp0xv9fX1UVZWNq/tKYBBpyPNEv/h8IVEHnq6ldYRLzKwsTiVL1xWRUFBATdZ83j05W5EQWYiEMYgh5GGW9gzEiEtLY309HQ2pRr42DlFPHlsFFmGj24v4dK66OfZUJbO7w4NYjXqMOkFnm0cxagXuOWC8qSvQUGO3Rw1tZEBGfzBECBwXsMmUlPmdnFTMNtQh1bapdfrY2R6M0XgwMmeg2KmtJJ1ZGVasru7m6qqqpgXzkwKEa0kcT4Nu3jliUSbdqeqrSOcJqRrs9nUDv18MNMqdapHQk1NzYwPRjIr3UBYxKSPkmHUwhBknZHBwUGGh4djHlptrPaAM4BMdJsMkJNionvCTzAs4nZO0tbWRk5ODtu2bUtKIpUMfntogKZhD7n26AN2qNfFn48Nc93ZRWwuzeBz79DxereDoNdNsc7J2XU1FBQUqFt2h8NBieji5qpQtLlj9zE2NkZaWhrHBtyI0snry7Aa2NM5uSDSrS+wc82GfB4/NACyiF5v5N531SZFuDMhnrWhkmLhdrvp7OzE6/VOG+owmUy0t7cTDAbZsGFDQhaeSwlldZuSkhL33plNIaIEYCp+CdpR56kBmHOVJwRBIBgMJkS6LpfrDOmeyphKmFpj9LlSGxQks9I9Z00GP93TdyIWPBpTc05VHjX5leoAgMvloru7G4/Ho46V+iUzwVBE7e76wxJmncyxI4fVMMyFxsnPhc4xH1bjye2i2aCjc+zkLqHIKlGvGyKzLJOKipMyp3iGOEpzx+FwRF9uvX4m3WGGnTokBGxGPeuL5y+AV+SCF9jH2HRpATpbBlV5dsqzlo7k4kXgRCIRPB4PTqeTpqYm3G43ZrOZzMxMxsfHCYfD8zL7XigUidzIyAh1dXVJkdhMAZiK25zDEetNPFM9XEvA4XCYnp4erFarWu6baUV8qnrpwmlCugttaGhJV2vpmEhqg/ZnJE66mYQiIi+1jmPQGfjgliJq8k8GAE4dAIhEIrjdbixOJ+szRPb2jhBdKMtcXaGnuLiG/Pz8ZemwV+XYeHPQrZqTB0WJqlybWu+e8IYYNeQzFrRQKILdEH0YeycDBCMS5VlWTAZd3OaOvcTN/7UdJiSKIEtMhkVEzwSHDh1KWk3g8/lobm7GaDSyOY4F5HLCYDBgMBgYHR0lPT2dLVu2RK00T2zZp8YJaWunS7VjcTqdNDc3qzujxShtzOY2p6z+h4aG8Pv9asNO+RUKhWhtbaWkpERVXczWsBseHj5lV7rCHEYx83ORWQEsJNE3FApx8OBBjMZoeGFNTc2cnf6p6O/vJxwOz1rcT2a4YSZEIhH2vNlB18AI9aV55NhNuN1uvF5vjG5zvo2dueAPi3zj6TYahz2AzMaiNN5frWdyfBRrbimf/X0HYVECBFLNen51y1a+/2Inr7VPotNBrt3Mtz+wnkybkQFHEKM+6oQmCAJ/bx7jv1/rxqDTIcoyKWY9kgTfva5WXf27XK6ovldjFKMlYu0ocW1t7YqvhpSkaIfDMavmFmKHOhSSmjrOnZqauqBxdUUG5vF4qK+vT/o+XywoZRiHw8Hg4CDBYHDa8Ip29a88M4FAgG9961v89Kc/5fjx4wseBV5CzPjgnTakqxhSJItAIEBLSwvDw8Ns27Zt3m/P4eFhPB4PVVXTk9IWg2wV8X5nZyeFhYWUlZVNW51oU3FdLldMPLnyazGkTpIsM+gIMDExjns4GmFUWlrKrb88ysvtE6rm1qAT2FaeTvuYj3AkWkYREDi/MpPWUS89kwFAZl1BKj/7+GYO9br43oudFKRFCdQTjGAz6vnm+9dP+ywUNYGWiBVzktzcXCoqKhZd1pUsFM1tSUkJJSUl8zoX7fCKQsZTpXpam8jZMDo6SltbG6WlpRQXF6/oZwMwNjZGa2sr5eXlFBYWIkmS+rJRzM2Vht0TTzxBXl4eu3bt4rrrruOuu+5aVK+UJcAZ0p0KRVY1NjZGdXU1ra2tXHDBBfM+/tjYGOPj49TV1am/p5XQzHeSDKJbwZaWFux2O1VVVUk5kYXD4RhyUmJntESstRNMBIpO2Wq1Ul1drZ7P+x7dR/OwN+bvlmVZGXQGEDXTD4qhuuHEISOSzLWbCvjKu+v492faaB72IAjR4MovXFrN+qJUZoPX66WpqQmj0Uh2drZaK1b0tVpySvZa5wNFc6vX66mtrV300oY2/kYh41AopE6dKderrP6DwaDqJVFXV7eipRaI3pMtLS2Ew2HWrl076/lIksTo6Ch33303R48exWKxIIoi119/Pffdd98ynnXSmPEmOy1qupC4p248lzGdTkdbW9uCjj+1prsYk2R+v5+2tjYikQhr166dc7IuHhQi0nbYQ6GQSsJazelcRBwKhWhvb8fr9VJbWztta3d+ZRbd434CkeiK3mLUUZ1to2/SH02vEAQkSUYEDIKMoHwmsswbfS5Meh13XFHNG71OfCGR6rwUVVYXD8rWfXJycsZhGO2gw8DAwLRBh8UkYq3mdim9G7Sa6YKCqAmPtjHpdDrVJpYSi1NcXExJScmy+uLGg7L6r6ioSKgP8cYbb3D77bdz3XXX8bOf/Qyj0YgkSdFx9VMUpw3pzoWpioTt27cvapNC8cNdrLptV1eXqgte7IfXZDKRk5MTI4rXbte15KSsmjweD8PDw1RWVs6oh779kkoGnQGeaRxFEATeu6mA684uZHfXJMGwBIKMXgeCrEnKOaGfLUiPqi5Meh3b1sxeh5VlmZGRETo6OigpKZk1W25q1tlM16p96cxn4szhcNDc3LximtupjUmv10tjYyNWq5Xs7Gy8Xi/Nzc0xQx3KtS5F7X8qQqEQTU1NAAkZ0geDQR566CFeeeUVdu3axVlnnaX+mU6nW9YhksXGaUO6s900iSoSFhKBotfrCQQC6vZ9vnXb/v5+ent7KS0tXbSuciKIJ/4PBAL09/fT1NSEXq9Hr9czODioRo0rDSwFJr2Ob11/VjTUUogOWUQkiY3FaTQPR71vdYJATV4Kbw66CYSjL6Y0q5H7312b0HkqpQSr1ZpwmkQi1zrTxJm2NBGPiJWu+2rR3GplYPX19XHtD5WdjjL+6/f71SasdrpuMe49bS9i6tDFTDhw4ACf//zn+cAHPsCLL764ZAqOlcJpU9ONRCLTJFtaj4S5FAn79u1j8+bNSRfnlZWt1hkqEomQkpJCenq6eiPPdeOMj4/T1tZGVlYWFRUVK36jKZlxOp2OmpoarFbrjA0sbS0xPT19GhF6gxF+/GoPbaNe1hba+eR5ZQQiEs8cH0WvE7isPpt06+zkqXy+igpgOVJgtTHsSj1cO/obCAQYHByksrJy2SR7s0HRAefl5VFeXp4UaWqbsIoaRtHiKvdwvKSO2RAMBmlqasJgMFBbWzvnsxUIBNi5cye7d+/m0UcfZf369bP+/VWO07+RpiXdZFMbAA4ePMjatWtVw+q5MFuTTOk4a8lJ0WAqRKzIYTweD62treqLIdHjLxZkWea1jkmO9rvISjFxWW0m40N9TE5OUlNTM6fkaqqDlbapo60Rz7eWqF0prYaueygUYnh4mK6uLgRBULPctKWJ5diuaxGJRGhra8Pr9S6qDEzRhytk7PV6VVe6eLIuBdqR4pqampgy1kzYv38/O3bs4IYbbmDHjh0rvuhYBJz+pCuKIn6/P0aRkGhqA8CRI0eoqKiImbCZCVObZIkcQ0mIVRodbrdb1RYXFRVRUFAQM/K7XHjyyBC/OtCP2aDD6w9iJcg9l1dQvaZ03sShTCYpqyaXy6V6umqJeK6Vj8fjobm5eZpKYqWgbdzV19erjUStB4Mi1TMajdM8GJaCiBUZWFlZGUVFRUtO9ooPg1ZLrI2cN5vN9Pf3Y7VaqampmZM8/X4/Dz74IPv37+fRRx9dcBTOKsLpT7ojIyMcO3aMsrIyiouLkyav48ePU1hYOOvKbjGaZEqHe3BwkLKyMmw2m/rAejyeGMOU9PT0JV01ybLMp/73Dcx6iYDXi9FoxCsb+dw7KudsZs3nWIrMSSGnSGS6R6/RaFQF/C6Xi9ra2mUpJcwFhdwS1dwuNRErMjBBEJZElpYMFH1tb28vY2Nj0UQSnW7OoY69e/fyxS9+kY985CPcfvvtp8PqVovTn3RDoRDhcHjeX1xLSwuZmZkxzRUFizXcoGRb5efnU1ZWxpg3gicYoSDdTIopet6RSCRmq+49QYbKg5qenj6rxEmSZXom/IRFieIMq2oeEw+BQIAbH3udFL1MRkYaBoORYXeAWy+qYHvF0k9yaT16lV+BQIBIJEJ2djYlJSXTko2XG0pemk6nWzC5aTXTbrdbdSXTNrC0yb/xoG22Kru5lYZimGO326murlZDVpXvVnn5RCIRmpubaWpqYnh4mJ6eHnbt2hWjbT+NcPqTrmKYMV90dHRgtVrVuW9YHLKFqCNSS0sLNptNHW743eFB/vrmCHpBwGrSs+PSKsqy4tdztbpapaGjyLm0WlNRknn0lW4OdDvQCZBuNfKlK6rJS40lCm0W2FF/Bi92ebGbjQTCIqkWA1+7tp40y/JO+7jdbpqbm1XHOO0DK4qiun1VCGqpzWGWS3Mbb4pwqj2kQsSKDCw1NZWqqqoVXxnKskxvby8DAwPU19fP2TuRZZknn3yS73//+5hMJmRZZnx8nP/6r/9a0GDSKsXpT7qKCHy+6OnpQRAESktLF22SLBAI0NbWRjAYpLa2Vq0Xv9I+wVf/1IROgDSriSybkdxUE//2nsTrWVObV8FgkA6vgSfbwxRnWDGZTEz4ItQV2NlxaXQ0WXHdUrTKZWVlyAg8c3yEw30uMm1G3r+5kNzU5duqKl7FLpdrRm8CrSeBQsRac5jFJmJFc5uTk8OaNWuW3f1rKhF7vV4ikai7XHFxMfn5+StS/9dCeQGkp6dTWVk552fk8/l44IEHOHLkCD/84Q+prY1KBJX+yEq/QJYAZ0h3LgwMDBAMBlmzZk3STbKpEEWRrq4uRkdHqaqqikmy8IYifOn3x2kccpNuNRIMS5gNOrJSTPz4o5sW1Lx6/EAvvz88SKYZQuEQgbCE0WDgvkuLMRqNDA0NYbFYqK6uXvFRUG2HW5m9T+baFSJ2Op1qPVzb0Jmpsz4bwuGwGvdUV1e3YmYwWigvgOzsbDIzM9XmlSLpmlqaWGoiTkQHrIUsy+zevZs77riDT3ziE9x6663L/hJbIZz+Y8ALhcFgYHx8nEAggMFgmHfdViGS4uLiuJNJo+4Qep2AQadDksBi1DPpC7GhOG1BDTNBEKgqSMdoGsdmN5GqExh2BdlQYGV8fByPx4PJZFKJZTm36lOhlBLsdvu0oMxEEc/PVasQ6e/vx+12A8Rs1eNpTbXfW6LjqUuNSCRCa2srPp8vZuhCW+ZQJF0ul4uurq5p2trFJmK3201jYyPZ2dkJDe54vV6++tWvcvz4cX7zm99QXV29KOeRCAKBABdddBHBYJBIJML111/PV7/61WU7/mw4bVa6MD97R6WMEAwG6ejoULeuWgVBIqLwiYkJ2trayMjIoKKiYkYiGXYF+c7z7fjDEo1DbiQ5mhzxo4+eTcksPgOJXssTR4d58sgQAAUWiUvyA6ytrqCgoEANF9SuEBMlpsVAOBxWbQXr6uoSkuctFFqJk7IiVrSmimytt7dXNRNaDc5Vij/BfHYAWiJWnLoWOuQgSZJql7l27do5vzdZlnnllVe48847ueWWW/jHf/zHZX+xK1p5u91OOBzmwgsv5Lvf/S7nnnvucp3C6V9egOScxmZN2z0hgVE0tdr0BmW4QZH7KIm7ADU1NXOOgcqyzFNvjvBi6ziyLBGWZD68tZhz1izeLPnA8BjNbe2U5CU23aYQk3arrlzv1GbOfCDLMgMDA/T09LBmzRr1BbBSEEURp9NJZ2cnbrcbo9E4Tc61EjVTrVKirq5u0XTJSnKF9sWjRLXP9aJ1uVw0NjYmPOXm8Xj4l3/5F1paWvjhD38Y1+p0ueHz+bjwwgv5wQ9+wPbt25frsGdIV8F8m2RaKZfT6cTr9aq135KSEoqKihKOypFlme4JP05/mNxUM0XpixOxo0TQS5JEbW3tgnwApkrX5uvN63K5aG5uJi0tbVV03CGquW1vb1edtxTzc23zaqpmeqEvntmgyMD6+vrUNOulRrzr1RJxSkoKo6OjOJ1O1q1bN2d9W5ZlXn75Ze666y4+/elP89nPfnZFG30QvcaGhgba2iFT/aIAABafSURBVNq49dZb+frXv76ch39rkG44HFZXrvEwn0myqZAkib6+Pvr7+ykuLsZisUzzIVBWw4lMXS0GlMadMom3lPIm7YtH8SLQXq/i4RoOh2lra8Pn81FXVzcvW8rFhrKSFAQhIV9Z7VZdUREsdjqHx+OhqalJfSmtZJNJIeLh4WEGBgYwGAzTYuZTU1Onkanb7eYrX/kKHR0d/OhHP5o1PWUl4HA4eN/73sfDDz8c41a2xHhrk+5iDTeMjY3R3t5Obm5uXCmRdvxV2aorcSta85vFerC0vgTKqm25VxeKAY5yvUpdPRQKUVhYSHl5+YorJSRJore3l8HBwQVrbmdL59A6dM11f2nrpNqR4pWEKIq0tbXh8XhYu3YtNpstbk0coi+kV199lbS0NB577DFuvfVWPv3pT6/46nYmPPDAA9hsNr74xS8u1yHfmqS7WMMNiluZxWKhqqoqqcTd2RpXWvObZM9LGbhISUlJOk1iqaCEHdrtdjIyMlRtbSgUUsd909PTF5zzlQwcDgctLS1kZ2cvmeZ2pnQObWlCW4qZnJykublZ1UqvBqKamJigpaWFkpKSOU2FRFGktbWV+++/X30ujEYjn/nMZ7jllluW8axnxujoKEajkYyMDPx+P1dccQV33nknV1999XKdwluDdBWnscUi22AwSFtbm+pWtlirEW09TakPK9tWhYhnqpeGQqGYc1oOBcBc0J5TvFJCvHFfZQegJeLFJMSV1txqpwiVkV/FV0KWZerq6sjMzFw10jS/35+Qy54sy7zwwgvcc8893Hbbbdx8883odDp1WCcRv9zlwJEjR7jpppsQRRFJkvjgBz/IV77yleU8hbcG6YbDYXVyZ6HDDYoAvLKyMim3svlCWS0pK2Jl1FdblhgeHmZwcJCKigry8vJW/IGVZZm+vj76+vqS1rfONGWmXR3Gqx8mck6K5nY1KCUUDA8P09bWRm5uLnq9HrfbHXecO5F4+cXC1GDIRJqi9913H/39/Tz66KOUlZUty3meonhrkO4dd9yhCu4bGhpITU1N6gZWIn26urooKiqitLR0xbZ+WsPwoaEhxsbG0Ov1pKenk5GRoT6kK6UGULbtmZmZi2a6rpXqKUQ8Vew/WylGSZVQSi6rQXOrNO+UkMqpZSBthpti+GOxWKYR8WIimWBIiN6Lzz//PPfeey+33347n/jEJ5btuejt7eXGG29keHgYQRD49Kc/ze23374sx14g3hqk29zczJ49e9i7dy8HDx4kFApx1lln0dDQwLZt21i/fv2MD6LD4aC1tZXU1FQqKytXRY3U6/XS0tKCwWCgpqYGs9mMz+dTV8OKOfpCV4fJQBtPsxzb9kQUBGazma6uLiYmJpYtVWIuKLuA/v7+pJp3M6VzLBYRJxsM6XQ6ueeeexgZGeGRRx6htLR0XsedLwYHBxkcHGTLli243W4aGhr4wx/+wLp165b1POaBtwbpTkUgEODw4cPs2bOH/fv38+abb2Kz2WhoaGDr1q1s3boVv9/PoUOHWLduHbW1tati3l4bTTNX8oUy+qpt1GkHGxbLk1dbSqisrFzR8oa2cTU6Oorb7VbDJ5VyzHJErc8Ej8dDY2MjGRkZCZnBzIXFSOfQBkPW19fPuaiQZZlnn32Wr3zlK+zYsYMbb7xxVTT8rr32Wm677TYuv/zylT6VufDWJN2pkGWZiYkJ9u/fzwsvvMBvfvMbvF4vDQ0NbNmyRSXjlWpwaCe3FpIEEM+TV5t2m56enlTtUCklKPltq8GwRKu5ra2tRRCEab68S71NnwpRFOns7GRiYmLJZWAzEbHWFF4pPyUbDOlwOLj77ruZmJjgkUceobi4eMmuIxl0dXVx0UUXcezYsVUhsZsDZ0h3Knbs2MH69eu58cYb6e3tZe/evezdu5fXX38dt9vN2rVrVRLetGlTUjKx+UAhtrm8G+aLUCgUU5YIBALqSklZHU49ZjAYpLW1lXA4vGp2AVrN7WzTW8o2XXvN8UhpsT5nRQZWWFi4Yr0ArU7c5XLhcDjwer0YDAaKiorIzMycVa4nyzJPP/00999/P1/84hf56Ec/uipWtxDdPbz97W/n3nvv5brrrlvp00kEZ0g3GYTDYY4ePaoS8ZEjRzAYDGzZsoUtW7awdetWampqFmXFp4RoRiKRZSU2ZaWkkJLT6VRlXErS7cTERNJZc0sJRQc83xW3QkpaIhZFEZvNllRysxZaaVoywaZLCa2Co7q6GpvNFleupxCwIk+86667cLlc/OAHP6CoqGilL0NFOBzm6quv5sorr2THjh0rfTqJ4gzpLgSyLON2u3n99dfZu3cv+/bto62tjby8vJj6cDKSKa0sbbnm7eeC8rC2t7djNBrVa9E6rq2EEYx2pHgx024hNrlZCQyd2pycKfF2ZGSEjo6OVSVNCwQCNDY2Yjabqa2tjfsC0V7z/v37eeCBB1R/3Ouvv553vvOdqyYgUpZlbrrpJrKysvjOd76z0qeTDM6Q7mJDqb8qq+F9+/YxNjZGTU2NKlnbsmXLtCaW9mFdaVmaFsFgkJaWFiKRCHV1dapZzkxGMNr6cCLGN/OBVsK3nMQ2NblZGX1ViNhisdDb24vJZKKmpmZVKF20pjm1tbVkZc3tWjcxMcGdd96J3+/nu9/9Lg6Hg9dff53S0lIuu+yyZTjrufHKK6/wtre9jQ0bNqjPyYMPPsi73vWuFT6zOXGGdJcDoijS2NjI3r172b9/PwcPHkQURTZu3MjWrVtJSUnhpZde4p/+6Z9WRaQ4xNZIq6qqEgo6jGd8o4j8lW36QptWiubWZrNRXV294ppbURRxuVz09PQwMTGB0WjEbDYvW3LzbIgXDDkbZFnmz3/+M//6r//KPffcw4c+9KFVsUo/zXCGdFcCSg3x+eefZ+fOnXR1dVFeXo5er1e1w1u3bp1XZPxiYGJigtbW1kXJAtN20p1O5zS/hUQHObQKgNWiuYWo/0ZTU1OMDCxelpmS3Lwc0rVkgyEBxsfHueOOO4hEInz/+98nPz9/Sc4tHj75yU/ypz/9iby8PI4dO7Zsx10hnCHdlcRTTz2F0+nkhhtuAKLjl9qyRH9/P2vWrFFrw1u2bCE9PX3JHtZAIEBLSwuSJFFXV7ckzZ+Z/BbsdntM00r7shkbG6OtrY2ioqIVcUyLB+UlMDk5SX19/ZxeF1rPBafTSSAQiBn1VeR6C0WywZCyLPPEE0/w4IMPcu+993LDDTcs++r2pZdewm63c+ONN54h3VlwhnSXAZIk0dbWppLwgQMH8Pl8rF+/XiXis846a8EPqzZWfCWad/FqpYIgYLPZ1FXialEAwEnnLaX2Pl+SipfcnMxggxbKdzg8PJxQMCREX2Zf+MIXEASB733veytqStPV1cXVV199hnRnwRnSXSGEQiEOHz6sEvGxY8ewWCxs3rxZJeLKysqEV4Pj4+O0traqsSurYcBBIZC+vj6ysrKIRCKqP63WGH25p8sUb4JQKER9ff2ivwSm6mldLhfhcFh1XZvJV0MbDFlRUTHndy/LMn/4wx946KGH+PKXv8wHPvCBFa/dniHdM6R7ykCWZRwOB/v371cbdR0dHRQXF7Nlyxa2bdtGQ0NDTNw7nCwlyLJMbW3tqllFzqa5Vbboip5WO12mkPFSNCG1pvDLnQqslXFpfTXsdjupqal4PB48Hg/r1q1LyM5zZGSEL3zhCxiNRh5++OGEGqTLgTOke4Z0T2koyoM9e/awb98+9u/fj8PhoK6ujk2bNtHe3k5eXh633nrrqtABw/w0t9qRV4WIlZWhdkW8kNW73++nqakJk8lEbW3tiqslIPr9KpaQ2pfMbMnNsizz+9//nm984xvcf//9XHfddSu+utXiDOmeId3TDpFIhB//+Mfs3LmTNWvW4PP5ADj77LPVQY66uroVicReTM1tvKEGWZZjGnWJRI1rFQCJ6luXA6IoqqZHa9euVY3hZwqUfP7559Hr9bzyyivk5OTw8MMPr5oXrRZnSPcM6Z6W+NnPfsbFF19MaWkpsizj8Xg4cOCAWpZQ4msaGhpoaGjgnHPOWdLBg+XS3Gr9eJVG3WxBkooMTPEEXg11boj6cDQ1NVFYWEhZWdmc30soFOI73/kOf/3rXzEYDHi9XtLT0/n973+/al4iAB/+8Id54YUXGBsbIz8/n69+9avcfPPNK31aS4W3NunecccdPPnkk5hMJqqqqnjssccS0jSerlBWnfv27VNtLxVFg6If3rx587yy27RQUorHx8fntKhcKihaWqUsocTmSJKkTt8tVXpysogXDDkXhoaG2LFjB6mpqXznO99Rr2ViYoKMjIxVIbt7i+KtTbrPPPMMl1xyCQaDgTvvvBOAr3/96yt8VqsLoijS0tKi1ocPHTpEKBRiw4YNKhGvW7cu4VWqopZYSdeteBgfH6e5uZmMjAxMJpMq4Voq97FEocjTlFTnRNKEf/3rX/Ptb3+bf/u3f+M973nPstZu//rXv3L77bcjiiK33HILd91117Id+xTBW5t0tXj88cf57W9/y89//vOVPpVVj0AgwKFDh2JM4O12e4zJz9Q022AwSHNzsxq+uNSWmIlCSbwIh8PU19fHnNdM7mN2u11VTMQzvVkMJBsMCdHV7e23305WVhbf/va3l72EIIoitbW1PPvss5SUlLBt2zb+7//+71RIc1hOzEi6KxOwtYL4n//5H3Uy7Axmh8Vi4bzzzuO8884DouQ0Pj7O/v372bNnD7/85S9Vw/UtW7YwMjKCLMvcd999q0aipG3gzZR4oQxo2Gw2CgsLgZPBmU6nk/7+ftxuN4IgTHNcW8jqUhsMWV9fn9Dq9pe//CX/+Z//yYMPPsi73/3uFVEm7Nu3j+rqaiorKwH40Ic+xB//+MczpJsgThvSveyyyxgaGpr2+1/72te49tpr1f83GAz8wz/8w3Kf3mkBQRDIycnhqquu4qqrrgKiRPDEE0/wpS99SZ10es973sO6devUFfHGjRtXZMXr9/tpbGzEYrGwdevWpEoGSiCmVhOrKAecTiednZ2qQXiyXgvaYMjNmzcn9NkMDg5y++23k5uby4svvkhmZmbC17LY6O/vj8lKKykpYe/evSt2PqcaThvS/dvf/jbrn+/atYs//elPPPfcc6tKt3iqQ6fTodfr+eMf/6h6sIZCIdUE/rHHHuPo0aMYjUY2b96s1oerq6uXrM6rdU6rq6tbNILS6/VkZGTENAQVxzWn08nAwMC0mKD09PQYjW2ywZCSJPGLX/yC733ve+zcuZN3vetdZ+7fUxxviZruX//6V3bs2MGLL764LNve3/zmN9x///00Njayb98+tm7duuTHXM2QZRmXyxVjAt/e3k5+fn5MfXgxwi5dLhdNTU0Jj8ouNmaKCbJarQQCAQwGA+vXr0+odjswMMDnPvc5CgsL+Y//+I9Vo7jZvXs3999/P08//TQAO3fuBODuu+9eydNabXhrN9Kqq6sJBoOqnObcc8/lkUceWbLjNTY2otPp+MxnPsM3v/nNtzzpxoNiur137161UadIyxQT+M2bNyfsUSuKIu3t7TidzphhgpWGUlPu6OhQ7z+Xy6UOcmgbdcoLQpIk/vd//5cf/OAHfP3rX+fKK69cVatbJVrqueeeo7i4mG3btvGLX/yC9evXr/SprSa8tUl3pXDxxRefId0kIIoix48fV1fDhw4dQpZl1QR+69at1NfXTzOCUeRpicqtlgvBYJCmpib0ej11dXUxNWXtIIfL5cLtdvPEE0/Q29tLd3c3FRUV/P/27jek6TwO4Pj7tzA4LpYWVxy7Bxr0w8y14dQsWFKxdkQQix7UEwN7UMHGReCjPUmIIMHQHqVi7YgKjlhCUsJR1x+NrYyOqLhBoDRhYD1QMjBq+90D3S+7s7uZm7/fts8LBIfgPo8+fPl+P3+6u7sNvbv9Lzdv3uT48eMkk0mam5sJBoNGh2Q2knSNIEl3cdIzeZ88ecKjR4+IRqPEYjHKyspwuVyoqsqNGzc4duwYW7duNU152tzFkOvXr8+oHTeVStHT08O1a9fYsGEDk5OTvHr1inPnzuF2u5cgapFlknSzLZNqCUm62ZfeMXfmzBmuXLmC3W4nkUhQUVHxxRB4q9VqyIk3k8WQ/xSPxwkEAqxbt462tjasVqv+N03TTHNyFwsidbrZ9n/VEiI3FEWhrKwMq9VKLBZj5cqV+hD4SCTCrVu3OHXqFNPT0/8aAp/LnXTfshgylUoRCoXo6emhvb2dnTt3zltDvBTk8XfpSNLNU8Xchrl8+XJOnjypf7ZYLKiqiqqqNDU1ATP3qekh8F1dXfoQ+JqaGj0RZ6u6Ye5iyLq6uow6116/fo3f70dVVYaGhgx/+KuuriYcDnPkyBFD4ygGcr2QA9evXycQCPDmzRtKS0txOp16eU02SBvmwqWHwKfvhh8/fszIyAg2m01Pwi6Xi9WrV2d8uvyWxZCpVIre3l4uXrxIe3s7O3bsMNX1gVyJZY1cLywln8+Hz+fL2f+XNsyFS19LeL1evF4v8HldUCQS4cGDB5w9e5bJyUkqKyv1Jg6HwzFvTe379+95+fIlpaWlGZ9uR0dH8fv9VFVVMTg4aPjpVhhDkm4ekjbM7LBYLJSXl1NeXs6BAweAmQ6zFy9eEIlEuHz5Mi0tLVgsFr2bzuFwEA6H2bJlCw0NDRkthkwmk/T29hIKhejo6KCxsdGQ020mj78i9yTpCjFHSUkJTqcTp9PJ0aNHvxgC39fXRzAYpKKiguHhYR4+fEhtbS11dXVfbekdGRkhEAhgt9sZGhrKaD1RrsjjrzlI0s1DNpuNeDyufx4bG8NmsxkYUeFKTxZzu92cP3+eO3fu6GVq6SHw3d3djI+P60Pga2trcTgcXL16lUuXLtHZ2Ynb7TbV3a0wjjyk5SEj2jCbm5vp7+9nzZo1xbDfasGSySSxWIxoNEo0GmVgYID6+npCoVBGGyCMluvH3yIkzRGFZqnbMO/fv8+KFStoamqSpJsBaWooepJ0xeIV0SZXIRbrq0nXHIurhBCL1tLSQmVlJZs2bcLn8zExMWF0SGIeknSFKBAej4fnz5/z7NkzVFXV59wKc5GkK0SB2LVrlz5gp6GhgbGxMYMjEvORpCtMKR6Ps337dqqqqti4cSOdnZ1Gh5RXLly4oO+xE+YiD2kiIwcPHuTu3bu8ffuWtWvX0trayuHDh3P2fYlEgkQiQU1NDe/evcPlctHX11f0rc6ZLmAdHh4mHA5LBYVxpHpB5Le9e/fi9/vxeDxGh2JqoVCIrq4ubt++nRf1wQVMBt6I/DU6OsrTp0/ZvHmz0aGY2sDAAG1tbdy7d08SronJSVeY2tTUFI2NjQSDQfbt22d0OKa21AtYxX+S6wWRfz5+/MiePXvwer2cOHHC6HCEWAhJuiK/aJrGoUOHWLVqFR0dHUaHI8RCSdIV+WVwcBC3243dbtdX6pw+fZrdu3fn7Dunp6fZtm0bHz584NOnT+zfv5/W1tacfZ8oaN+cdIUoGspMfdX3mqZNKYpSAgwCv2iaFjE4NFFApHpBiFnazAlkavZjyeyPnEpEVklHmhBzKIqyTFGUP4Fx4HdN02QPksgqSbpCzKFpWlLTNCfwE1CvKEq10TGJwiJJV4h5aJo2AfwB/Gx0LKKwSNIVYpaiKD8oilI6+/t3gAf4y9ioRKGRhzQhPvsR+FVRlGXMHEh+0zSt3+CYRIH5G2jZRF0H9CwCAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Let's visualize our data (becoming one with the data paradigm <3)\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111, projection='3d')\n",
    "assert xs.shape[-1] == 2 and ys.shape[-1] == 1  # low dimensional data so that we can plot it\n",
    "ax.scatter(xs[:, 0], xs[:, 1], zs=ys)\n",
    "\n",
    "# todo: exercise - let's show that our data lies on the 2D plane embedded in 3D\n",
    "# option 1: analytic approach\n",
    "# option 2: data-driven approach"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "id": "mKiCOyoikxcM"
   },
   "outputs": [],
   "source": [
    "def make_mse_loss(xs, ys):\n",
    "    \n",
    "    def mse_loss(params):\n",
    "        \"\"\"Gives the value of the loss on the (xs, ys) dataset for the given model (params).\"\"\"\n",
    "        \n",
    "        # Define the squared loss for a single pair (x,y)\n",
    "        def squared_error(x, y):\n",
    "            pred = model.apply(params, x)\n",
    "            # Inner because 'y' could have in general more than 1 dims\n",
    "            return jnp.inner(y-pred, y-pred) / 2.0\n",
    "\n",
    "        # Batched version via vmap\n",
    "        return jnp.mean(jax.vmap(squared_error)(xs, ys), axis=0)\n",
    "\n",
    "    return jax.jit(mse_loss)  # and finally we jit the result (mse_loss is a pure function)\n",
    "\n",
    "mse_loss = make_mse_loss(xs, ys)\n",
    "value_and_grad_fn = jax.value_and_grad(mse_loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "phLYjH5ZkzLn"
   },
   "outputs": [],
   "source": [
    "# Let's reuse the simple feed-forward layer since it trivially implements linear regression\n",
    "model = nn.Dense(features=y_dim)\n",
    "params = model.init(key, xs)\n",
    "print(f'Initial params = {params}')\n",
    "\n",
    "# Let's set some reasonable hyperparams\n",
    "lr = 0.3\n",
    "epochs = 20\n",
    "log_period_epoch = 5\n",
    "\n",
    "print('-' * 50)\n",
    "for epoch in range(epochs):\n",
    "    loss, grads = value_and_grad_fn(params)\n",
    "    # SGD (closer to JAX again, but we'll progressively go towards how stuff is done in Flax)\n",
    "    params = jax.tree_multimap(lambda p, g: p - lr * g, params, grads)\n",
    "\n",
    "    if epoch % log_period_epoch == 0:\n",
    "        print(f'epoch {epoch}, loss = {loss}')\n",
    "\n",
    "print('-' * 50)\n",
    "print(f'Learned params = {params}')\n",
    "print(f'Gt params = {true_params}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rvy6Oow2lLHu"
   },
   "source": [
    "Now let's do the same thing but this time with dedicated optimizers!\n",
    "\n",
    "Enter DeepMind's optax! ❤️🔥"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "5hhcFZ7UlCov"
   },
   "outputs": [],
   "source": [
    "opt_sgd = optax.sgd(learning_rate=lr)\n",
    "opt_state = opt_sgd.init(params)  # always the same pattern - handling state externally\n",
    "print(opt_state)\n",
    "# todo: exercise - compare Adam's and SGD's states"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "t_EHHjy_lFGN"
   },
   "outputs": [],
   "source": [
    "params = model.init(key, xs)  # let's start with fresh params again\n",
    "\n",
    "for epoch in range(epochs):\n",
    "    loss, grads = value_and_grad_fn(params)\n",
    "    updates, opt_state = opt_sgd.update(grads, opt_state)  # arbitrary optim logic!\n",
    "    params = optax.apply_updates(params, updates)\n",
    "\n",
    "    if epoch % log_period_epoch == 0:\n",
    "        print(f'epoch {epoch}, loss = {loss}')\n",
    "\n",
    "# Note 1: as expected we get the same loss values\n",
    "# Note 2: we'll later see more concise ways to handle all of these state components (hint: TrainState)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QF1gAYSzxQ1R"
   },
   "source": [
    "In this toy SGD example Optax may not seem that useful but it's very powerful.\n",
    "\n",
    "You can build arbitrary optimizers with arbitrary hyperparam schedules, chaining, param freezing, etc. You can check the [official docs here](https://optax.readthedocs.io/en/latest/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "cellView": "form",
    "id": "rKbis5O0KQYH"
   },
   "outputs": [],
   "source": [
    "#@title Optax Advanced Examples\n",
    "# This cell won't \"compile\" (no ml_collections package) and serves just as an example\n",
    "\n",
    "# Example from Flax (ImageNet example)\n",
    "# https://github.com/google/flax/blob/main/examples/imagenet/train.py#L88\n",
    "def create_learning_rate_fn(\n",
    "    config: ml_collections.ConfigDict,\n",
    "    base_learning_rate: float,\n",
    "    steps_per_epoch: int):\n",
    "  \"\"\"Create learning rate schedule.\"\"\"\n",
    "  warmup_fn = optax.linear_schedule(\n",
    "      init_value=0., end_value=base_learning_rate,\n",
    "      transition_steps=config.warmup_epochs * steps_per_epoch)\n",
    "  cosine_epochs = max(config.num_epochs - config.warmup_epochs, 1)\n",
    "  cosine_fn = optax.cosine_decay_schedule(\n",
    "      init_value=base_learning_rate,\n",
    "      decay_steps=cosine_epochs * steps_per_epoch)\n",
    "  schedule_fn = optax.join_schedules(\n",
    "      schedules=[warmup_fn, cosine_fn],\n",
    "      boundaries=[config.warmup_epochs * steps_per_epoch])\n",
    "  return schedule_fn\n",
    "\n",
    "tx = optax.sgd(\n",
    "      learning_rate=learning_rate_fn,\n",
    "      momentum=config.momentum,\n",
    "      nesterov=True,\n",
    ")\n",
    "\n",
    "# Example from Haiku (ImageNet example)\n",
    "# https://github.com/deepmind/dm-haiku/blob/main/examples/imagenet/train.py#L116\n",
    "def make_optimizer() -> optax.GradientTransformation:\n",
    "  \"\"\"SGD with nesterov momentum and a custom lr schedule.\"\"\"\n",
    "  return optax.chain(\n",
    "      optax.trace(\n",
    "          decay=FLAGS.optimizer_momentum,\n",
    "          nesterov=FLAGS.optimizer_use_nesterov),\n",
    "      optax.scale_by_schedule(lr_schedule), optax.scale(-1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WFAeHIEwL0ZH"
   },
   "source": [
    "Now let's go beyond these extremely simple models!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7_33y-bTl6bd"
   },
   "source": [
    "### Creating custom models ⭐"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "JOrJHqTSl75M"
   },
   "outputs": [],
   "source": [
    "class MLP(nn.Module):\n",
    "    num_neurons_per_layer: Sequence[int]  # data field (nn.Module is Python's dataclass)\n",
    "\n",
    "    def setup(self):  # because dataclass is implicitly using the  __init__ function... :')\n",
    "        self.layers = [nn.Dense(n) for n in self.num_neurons_per_layer]\n",
    "\n",
    "    def __call__(self, x):\n",
    "        activation = x\n",
    "        for i, layer in enumerate(self.layers):\n",
    "            activation = layer(activation)\n",
    "            if i != len(self.layers) - 1:\n",
    "                activation = nn.relu(activation)\n",
    "        return activation\n",
    "\n",
    "x_key, init_key = random.split(random.PRNGKey(seed))\n",
    "\n",
    "model = MLP(num_neurons_per_layer=[16, 8, 1])  # define an MLP model\n",
    "x = random.uniform(x_key, (4,4))  # dummy input\n",
    "params = model.init(init_key, x)  # initialize via init\n",
    "y = model.apply(params, x)  # do a forward pass via apply\n",
    "\n",
    "print(jax.tree_map(jnp.shape, params))\n",
    "print(f'Output: {y}')\n",
    "\n",
    "# todo: exercise - use @nn.compact pattern instead\n",
    "# todo: check out https://realpython.com/python-data-classes/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TEhC-WdPnAYp"
   },
   "source": [
    "Great! \n",
    "\n",
    "Now that we know how to build more complex models let's dive deeper and understand how the 'nn.Dense' module is designed itself.\n",
    "\n",
    "#### Introducing \"param\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Z9YhSgxjnBQg"
   },
   "outputs": [],
   "source": [
    "class MyDenseImp(nn.Module):\n",
    "    num_neurons: int\n",
    "    weight_init: Callable = nn.initializers.lecun_normal()\n",
    "    bias_init: Callable = nn.initializers.zeros\n",
    "\n",
    "    @nn.compact\n",
    "    def __call__(self, x):\n",
    "        weight = self.param('weight',  # parametar name (as it will appear in the FrozenDict)\n",
    "                self.weight_init,  # initialization function, RNG passed implicitly through init fn\n",
    "                (x.shape[-1], self.num_neurons))  # shape info\n",
    "        bias = self.param('bias', self.bias_init, (self.num_neurons,))\n",
    "\n",
    "        return jnp.dot(x, weight)  + bias\n",
    "\n",
    "x_key, init_key = random.split(random.PRNGKey(seed))\n",
    "\n",
    "model = MyDenseImp(num_neurons=3)  # initialize the model\n",
    "x = random.uniform(x_key, (4,4))  # dummy input\n",
    "params = model.init(init_key, x)  # initialize via init\n",
    "y = model.apply(params, x)  # do a forward pass via apply\n",
    "\n",
    "print(jax.tree_map(jnp.shape, params))\n",
    "print(f'Output: {y}')\n",
    "\n",
    "# todo: exercise - check out the source code:\n",
    "# https://github.com/google/flax/blob/main/flax/linen/linear.py\n",
    "# https://github.com/google/jax/blob/main/jax/_src/nn/initializers.py#L150 <- to see why lecun_normal() vs zeros (no brackets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "AqCPhl9fBI_Z"
   },
   "outputs": [],
   "source": [
    "from inspect import signature\n",
    "\n",
    "# You can see it expects a PRNG key and it is passed implicitly through the init fn (same for zeros)\n",
    "print(signature(nn.initializers.lecun_normal()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MWB8HvLHn6g0"
   },
   "source": [
    "So far we've only seen **trainable** params. \n",
    "\n",
    "ML models often times have variables which are part of the state but are not optimized via gradient descent.\n",
    "\n",
    "Let's see how we can handle them using a simple (and contrived) example!\n",
    "\n",
    "#### Introducing \"variable\"\n",
    "\n",
    "*Note on terminology: variable is a broader term and it includes both params (trainable variables) as well as non-trainable vars.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "oGE6qTHHngYh"
   },
   "outputs": [],
   "source": [
    "class BiasAdderWithRunningMean(nn.Module):\n",
    "    decay: float = 0.99\n",
    "\n",
    "    @nn.compact\n",
    "    def __call__(self, x):\n",
    "        is_initialized = self.has_variable('batch_stats', 'ema')\n",
    "\n",
    "        # 'batch_stats' is not an arbitrary name!\n",
    "        # Flax uses that name in its implementation of BatchNorm (hard-coded, probably not the best of designs?)\n",
    "        ema = self.variable('batch_stats', 'ema', lambda shape: jnp.zeros(shape), x.shape[1:])\n",
    "\n",
    "        # self.param will by default add this variable to 'params' collection (vs 'batch_stats' above)\n",
    "        # Again some idiosyncrasies here we need to pass a key even though we don't actually use it...\n",
    "        bias = self.param('bias', lambda key, shape: jnp.zeros(shape), x.shape[1:])\n",
    "\n",
    "        if is_initialized:\n",
    "            # self.variable returns a reference hence .value\n",
    "            ema.value = self.decay * ema.value + (1.0 - self.decay) * jnp.mean(x, axis=0, keepdims=True)\n",
    "\n",
    "        return x - ema.value + bias\n",
    "\n",
    "x_key, init_key = random.split(random.PRNGKey(seed))\n",
    "\n",
    "model = BiasAdderWithRunningMean()\n",
    "x = random.uniform(x_key, (10,4))  # dummy input\n",
    "variables = model.init(init_key, x)\n",
    "print(f'Multiple collections = {variables}')  # we can now see a new collection 'batch_stats'\n",
    "\n",
    "# We have to use mutable since regular params are not modified during the forward\n",
    "# pass, but these variables are. We can't keep state internally (because JAX) so we have to return it.\n",
    "y, updated_non_trainable_params = model.apply(variables, x, mutable=['batch_stats'])\n",
    "print(updated_non_trainable_params)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "PuzwVt8RoHvY"
   },
   "outputs": [],
   "source": [
    "# Let's see how we could train such model!\n",
    "def update_step(opt, apply_fn, x, opt_state, params, non_trainable_params):\n",
    "\n",
    "    def loss_fn(params):\n",
    "        y, updated_non_trainable_params = apply_fn(\n",
    "            {'params': params, **non_trainable_params}, \n",
    "            x, mutable=list(non_trainable_params.keys()))\n",
    "        \n",
    "        loss = ((x - y) ** 2).sum()  # not doing anything really, just for the demo purpose\n",
    "\n",
    "        return loss, updated_non_trainable_params\n",
    "\n",
    "    (loss, non_trainable_params), grads = jax.value_and_grad(loss_fn, has_aux=True)(params)\n",
    "    updates, opt_state = opt.update(grads, opt_state)\n",
    "    params = optax.apply_updates(params, updates)\n",
    "    \n",
    "    return opt_state, params, non_trainable_params  # all of these represent the state - ugly, for now\n",
    "\n",
    "model = BiasAdderWithRunningMean()\n",
    "x = jnp.ones((10,4))  # dummy input, using ones because it's easier to see what's going on\n",
    "\n",
    "variables = model.init(random.PRNGKey(seed), x)\n",
    "non_trainable_params, params = variables.pop('params')\n",
    "del variables  # delete variables to avoid wasting resources (this pattern is used in the official code)\n",
    "\n",
    "sgd_opt = optax.sgd(learning_rate=0.1)  # originally you'll see them use the 'tx' naming (from opTaX)\n",
    "opt_state = sgd_opt.init(params)\n",
    "\n",
    "for _ in range(3):\n",
    "    # We'll later see how TrainState abstraction will make this step much more elegant!\n",
    "    opt_state, params, non_trainable_params = update_step(sgd_opt, model.apply, x, opt_state, params, non_trainable_params)\n",
    "    print(non_trainable_params)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gzWUq5vBrWMe"
   },
   "source": [
    "Let's go a level up in abstraction again now that we understand params and variables!\n",
    "\n",
    "Certain layers like BatchNorm will use variables in the background.\n",
    "\n",
    "Let's see a last example that is conceptually as complicated as it gets when it comes to Flax's idiosyncrasies, and high-level at the same time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "rDw2986orY0a"
   },
   "outputs": [],
   "source": [
    "class DDNBlock(nn.Module):\n",
    "    \"\"\"Dense, dropout + batchnorm combo.\n",
    "\n",
    "    Contains trainable variables (params), non-trainable variables (batch stats),\n",
    "    and stochasticity in the forward pass (because of dropout).\n",
    "    \"\"\"\n",
    "    num_neurons: int\n",
    "    training: bool\n",
    "\n",
    "    @nn.compact\n",
    "    def __call__(self, x):\n",
    "        x = nn.Dense(self.num_neurons)(x)\n",
    "        x = nn.Dropout(rate=0.5, deterministic=not self.training)(x)\n",
    "        x = nn.BatchNorm(use_running_average=not self.training)(x)\n",
    "        return x\n",
    "\n",
    "key1, key2, key3, key4 = random.split(random.PRNGKey(seed), 4)\n",
    "\n",
    "model = DDNBlock(num_neurons=3, training=True)\n",
    "x = random.uniform(key1, (3,4,4))\n",
    "\n",
    "# New: because of Dropout we now have to include its unique key - kinda weird, but you get used to it\n",
    "variables = model.init({'params': key2, 'dropout': key3}, x)\n",
    "print(variables)\n",
    "\n",
    "# And same here, everything else remains the same as the previous example\n",
    "y, non_trainable_params = model.apply(variables, x, rngs={'dropout': key4}, mutable=['batch_stats'])\n",
    "\n",
    "# Let's run these model variables during \"evaluation\":\n",
    "eval_model = DDNBlock(num_neurons=3, training=False)\n",
    "# Because training=False we don't have stochasticity in the forward pass neither do we update the stats\n",
    "y = eval_model.apply(variables, x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Ys1y-yM8vzT8"
   },
   "source": [
    "### A fully-fledged CNN on MNIST example in Flax! 💥\n",
    "\n",
    "Modified the official MNIST example here: https://github.com/google/flax/tree/main/examples/mnist\n",
    "\n",
    "We'll be using PyTorch dataloading instead of TFDS.\n",
    "\n",
    "Let's start by defining a model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "MD8t9K2Nv0yC"
   },
   "outputs": [],
   "source": [
    "class CNN(nn.Module):  # lots of hardcoding, but it serves a purpose for a simple demo\n",
    "    @nn.compact\n",
    "    def __call__(self, x):\n",
    "        x = nn.Conv(features=32, kernel_size=(3, 3))(x)\n",
    "        x = nn.relu(x)\n",
    "        x = nn.avg_pool(x, window_shape=(2, 2), strides=(2, 2))\n",
    "        x = nn.Conv(features=64, kernel_size=(3, 3))(x)\n",
    "        x = nn.relu(x)\n",
    "        x = nn.avg_pool(x, window_shape=(2, 2), strides=(2, 2))\n",
    "        x = x.reshape((x.shape[0], -1))  # flatten\n",
    "        x = nn.Dense(features=256)(x)\n",
    "        x = nn.relu(x)\n",
    "        x = nn.Dense(features=10)(x)\n",
    "        x = nn.log_softmax(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rVgWLMhiSAYv"
   },
   "source": [
    "Let's add the data loading support in PyTorch!\n",
    "\n",
    "I'll be reusing code from [tutorial #3](https://github.com/gordicaleksa/get-started-with-JAX/blob/main/Tutorial_3_JAX_Neural_Network_from_Scratch_Colab.ipynb):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "UZ-og2UOUUWD"
   },
   "outputs": [],
   "source": [
    "def custom_transform(x):\n",
    "    # A couple of modifications here compared to tutorial #3 since we're using a CNN\n",
    "    # Input: (28, 28) uint8 [0, 255] torch.Tensor, Output: (28, 28, 1) float32 [0, 1] np array\n",
    "    return np.expand_dims(np.array(x, dtype=np.float32), axis=2) / 255.\n",
    "\n",
    "def custom_collate_fn(batch):\n",
    "    \"\"\"Provides us with batches of numpy arrays and not PyTorch's tensors.\"\"\"\n",
    "    transposed_data = list(zip(*batch))\n",
    "\n",
    "    labels = np.array(transposed_data[1])\n",
    "    imgs = np.stack(transposed_data[0])\n",
    "\n",
    "    return imgs, labels\n",
    "\n",
    "mnist_img_size = (28, 28, 1)\n",
    "batch_size = 128\n",
    "\n",
    "train_dataset = MNIST(root='train_mnist', train=True, download=True, transform=custom_transform)\n",
    "test_dataset = MNIST(root='test_mnist', train=False, download=True, transform=custom_transform)\n",
    "\n",
    "train_loader = DataLoader(train_dataset, batch_size, shuffle=True, collate_fn=custom_collate_fn, drop_last=True)\n",
    "test_loader = DataLoader(test_dataset, batch_size, shuffle=False, collate_fn=custom_collate_fn, drop_last=True)\n",
    "\n",
    "# optimization - loading the whole dataset into memory\n",
    "train_images = jnp.array(train_dataset.data)\n",
    "train_lbls = jnp.array(train_dataset.targets)\n",
    "\n",
    "# np.expand_dims is to convert shape from (10000, 28, 28) -> (10000, 28, 28, 1)\n",
    "# We don't have to do this for training images because custom_transform does it for us.\n",
    "test_images = np.expand_dims(jnp.array(test_dataset.data), axis=3)\n",
    "test_lbls = jnp.array(test_dataset.targets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 282
    },
    "id": "2HeXX51NU0k6",
    "outputId": "43dad5bf-20c2-4c5a-9705-12b2e422f915"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "7\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAANiklEQVR4nO3df4wc9XnH8c8n/kV8QGtDcF3j4ISQqE4aSHWBRNDKESUFImSiJBRLtVyJ5lALElRRW0QVBalVSlEIok0aySluHESgaQBhJTSNa6W1UKljg4yxgdaEmsau8QFOaxPAP/DTP24cHXD7vWNndmft5/2SVrs7z87Oo/F9PLMzO/t1RAjA8e9tbTcAoD8IO5AEYQeSIOxAEoQdSGJ6Pxc207PiBA31c5FAKq/qZzoYBzxRrVbYbV8s6XZJ0yT9bUTcXHr9CRrSeb6wziIBFGyIdR1rXe/G254m6auSLpG0WNIy24u7fT8AvVXnM/u5kp6OiGci4qCkeyQtbaYtAE2rE/YFkn4y7vnOatrr2B6xvcn2pkM6UGNxAOro+dH4iFgZEcMRMTxDs3q9OAAd1An7LkkLxz0/vZoGYADVCftGSWfZfpftmZKulLSmmbYANK3rU28Rcdj2tZL+SWOn3lZFxLbGOgPQqFrn2SPiQUkPNtQLgB7i67JAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJGoN2Wx7h6T9kl6TdDgihptoCkDzaoW98rGIeKGB9wHQQ+zGA0nUDXtI+oHtR2yPTPQC2yO2N9nedEgHai4OQLfq7sZfEBG7bJ8maa3tpyJi/fgXRMRKSSsl6WTPjZrLA9ClWlv2iNhV3Y9Kul/SuU00BaB5XYfd9pDtk44+lvRxSVubagxAs+rsxs+TdL/to+/zrYj4fiNdAWhc12GPiGcknd1gLwB6iFNvQBKEHUiCsANJEHYgCcIOJNHEhTApvPjZj3asvXP508V5nxqdV6wfPDCjWF9wd7k+e+dLHWtHNj9RnBd5sGUHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQ4zz5Ff/xH3+pY+9TQT8szn1lz4UvK5R2HX+5Yu/35j9Vc+LHrR6NndKwN3foLxXmnr3uk6XZax5YdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5JwRP8GaTnZc+M8X9i35TXpZ58+r2PthQ+W/8+c82R5Hf/0V1ysz/zg/xbrt3zgvo61i97+SnHe7718YrH+idmdr5Wv65U4WKxvODBUrC854VDXy37P964u1t87srHr927ThlinfbF3wj8otuxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kATXs0/R0Hc2FGr13vvkerPrr39pScfan5+/qLzsfy3/5v0tS97TRUdTM/2VI8X60Jbdxfop6+8t1n91Zuff25+9o/xb/MejSbfstlfZHrW9ddy0ubbX2t5e3c/pbZsA6prKbvw3JF38hmk3SFoXEWdJWlc9BzDAJg17RKyXtPcNk5dKWl09Xi3p8ob7AtCwbj+zz4uIox+onpPUcTAz2yOSRiTpBM3ucnEA6qp9ND7GrqTpeKVHRKyMiOGIGJ6hWXUXB6BL3YZ9j+35klTdjzbXEoBe6DbsayStqB6vkPRAM+0A6JVJP7Pbvltjv1x+qu2dkr4g6WZJ37Z9laRnJV3RyyZRdvi5PR1rQ/d2rknSa5O899B3Xuyio2bs+b2PFuvvn1n+8/3S3vd1rC36u2eK8x4uVo9Nk4Y9IpZ1KB2bv0IBJMXXZYEkCDuQBGEHkiDsQBKEHUiCS1zRmulnLCzWv3LjV4r1GZ5WrP/D7b/ZsXbK7oeL8x6P2LIDSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKcZ0drnvrDBcX6h2eVh7LedrA8HPXcJ15+yz0dz9iyA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EASnGdHTx34xIc71h799G2TzF0eQej3r7uuWH/7v/1okvfPhS07kARhB5Ig7EAShB1IgrADSRB2IAnCDiTBeXb01H9f0nl7cqLL59GX/ddFxfrs7z9WrEexms+kW3bbq2yP2t46btpNtnfZ3lzdLu1tmwDqmspu/DckXTzB9Nsi4pzq9mCzbQFo2qRhj4j1kvb2oRcAPVTnAN21trdUu/lzOr3I9ojtTbY3HdKBGosDUEe3Yf+apDMlnSNpt6RbO70wIlZGxHBEDM+Y5MIGAL3TVdgjYk9EvBYRRyR9XdK5zbYFoGldhd32/HFPPylpa6fXAhgMk55nt323pCWSTrW9U9IXJC2xfY7GTmXukHR1D3vEAHvbSScV68t//aGOtX1HXi3OO/rFdxfrsw5sLNbxepOGPSKWTTD5jh70AqCH+LoskARhB5Ig7EAShB1IgrADSXCJK2rZftP7i/Xvnvo3HWtLt3+qOO+sBzm11iS27EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBOfZUfR/v/ORYn3Lb/9Vsf7jw4c61l76y9OL887S7mIdbw1bdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgvPsyU1f8MvF+vWf//tifZbLf0JXPra8Y+0d/8j16v3Elh1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkuA8+3HO08v/xGd/d2ex/pkTXyzW79p/WrE+7/OdtydHinOiaZNu2W0vtP1D20/Y3mb7umr6XNtrbW+v7uf0vl0A3ZrKbvxhSZ+LiMWSPiLpGtuLJd0gaV1EnCVpXfUcwICaNOwRsTsiHq0e75f0pKQFkpZKWl29bLWky3vVJID63tJndtuLJH1I0gZJ8yLi6I+EPSdpXod5RiSNSNIJmt1tnwBqmvLReNsnSrpX0vURsW98LSJCUkw0X0SsjIjhiBieoVm1mgXQvSmF3fYMjQX9roi4r5q8x/b8qj5f0mhvWgTQhEl3421b0h2SnoyIL48rrZG0QtLN1f0DPekQ9Zz9vmL5z067s9bbf/WLnynWf/Gxh2u9P5ozlc/s50taLulx25uraTdqLOTftn2VpGclXdGbFgE0YdKwR8RDktyhfGGz7QDoFb4uCyRB2IEkCDuQBGEHkiDsQBJc4nocmLb4vR1rI/fU+/rD4lXXFOuL7vz3Wu+P/mHLDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJcJ79OPDUH3T+Yd/LZu/rWJuK0//lYPkFMeEPFGEAsWUHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQ4z34MePWyc4v1dZfdWqgy5BbGsGUHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSSmMj77QknflDRPUkhaGRG3275J0mclPV+99MaIeLBXjWb2P+dPK9bfOb37c+l37T+tWJ+xr3w9O1ezHzum8qWaw5I+FxGP2j5J0iO211a12yLiS71rD0BTpjI++25Ju6vH+20/KWlBrxsD0Ky39Jnd9iJJH5K0oZp0re0ttlfZnvC3kWyP2N5ke9MhHajVLIDuTTnstk+UdK+k6yNin6SvSTpT0jka2/JP+AXtiFgZEcMRMTxDsxpoGUA3phR22zM0FvS7IuI+SYqIPRHxWkQckfR1SeWrNQC0atKw27akOyQ9GRFfHjd9/riXfVLS1ubbA9CUqRyNP1/SckmP295cTbtR0jLb52js7MsOSVf3pEPU8hcvLi7WH/6tRcV67H68wW7QpqkcjX9IkicocU4dOIbwDTogCcIOJEHYgSQIO5AEYQeSIOxAEo4+Drl7sufGeb6wb8sDstkQ67Qv9k50qpwtO5AFYQeSIOxAEoQdSIKwA0kQdiAJwg4k0dfz7Lafl/TsuEmnSnqhbw28NYPa26D2JdFbt5rs7YyIeMdEhb6G/U0LtzdFxHBrDRQMam+D2pdEb93qV2/sxgNJEHYgibbDvrLl5ZcMam+D2pdEb93qS2+tfmYH0D9tb9kB9AlhB5JoJey2L7b9H7aftn1DGz10YnuH7cdtb7a9qeVeVtketb113LS5ttfa3l7dTzjGXku93WR7V7XuNtu+tKXeFtr+oe0nbG+zfV01vdV1V+irL+ut75/ZbU+T9J+SLpK0U9JGScsi4om+NtKB7R2ShiOi9S9g2P4NSS9J+mZEfKCadoukvRFxc/Uf5ZyI+JMB6e0mSS+1PYx3NVrR/PHDjEu6XNLvqsV1V+jrCvVhvbWxZT9X0tMR8UxEHJR0j6SlLfQx8CJivaS9b5i8VNLq6vFqjf2x9F2H3gZCROyOiEerx/slHR1mvNV1V+irL9oI+wJJPxn3fKcGa7z3kPQD24/YHmm7mQnMi4jd1ePnJM1rs5kJTDqMdz+9YZjxgVl33Qx/XhcH6N7sgoj4NUmXSLqm2l0dSDH2GWyQzp1OaRjvfplgmPGfa3PddTv8eV1thH2XpIXjnp9eTRsIEbGruh+VdL8GbyjqPUdH0K3uR1vu5+cGaRjviYYZ1wCsuzaHP28j7BslnWX7XbZnSrpS0poW+ngT20PVgRPZHpL0cQ3eUNRrJK2oHq+Q9ECLvbzOoAzj3WmYcbW87lof/jwi+n6TdKnGjsj/WNKfttFDh77eLemx6rat7d4k3a2x3bpDGju2cZWkUyStk7Rd0j9LmjtAvd0p6XFJWzQWrPkt9XaBxnbRt0jaXN0ubXvdFfrqy3rj67JAEhygA5Ig7EAShB1IgrADSRB2IAnCDiRB2IEk/h9BCfQTVPflJQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Visualize a single image\n",
    "imgs, lbls = next(iter(test_loader))\n",
    "img = imgs[0].reshape(mnist_img_size)[:, :, 0]\n",
    "gt_lbl = lbls[0]\n",
    "\n",
    "print(gt_lbl)\n",
    "plt.imshow(img); plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TsGPQKx0SPL-"
   },
   "source": [
    "Great - we have our data pipeline ready and the model architecture defined.\n",
    "\n",
    "Now let's define core training functions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "qD8ztbEsVM43"
   },
   "outputs": [],
   "source": [
    "@jax.jit\n",
    "def train_step(state, imgs, gt_labels):\n",
    "    def loss_fn(params):\n",
    "        logits = CNN().apply({'params': params}, imgs)\n",
    "        one_hot_gt_labels = jax.nn.one_hot(gt_labels, num_classes=10)\n",
    "        loss = -jnp.mean(jnp.sum(one_hot_gt_labels * logits, axis=-1))\n",
    "        return loss, logits\n",
    "  \n",
    "    (_, logits), grads = jax.value_and_grad(loss_fn, has_aux=True)(state.params)\n",
    "    state = state.apply_gradients(grads=grads)  # this is the whole update now! concise!\n",
    "    metrics = compute_metrics(logits=logits, gt_labels=gt_labels)  # duplicating loss calculation but it's a bit cleaner\n",
    "    return state, metrics\n",
    "\n",
    "@jax.jit\n",
    "def eval_step(state, imgs, gt_labels):\n",
    "    logits = CNN().apply({'params': state.params}, imgs)\n",
    "    return compute_metrics(logits=logits, gt_labels=gt_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "v5VblVs2VWxo"
   },
   "outputs": [],
   "source": [
    "def train_one_epoch(state, dataloader, epoch):\n",
    "    \"\"\"Train for 1 epoch on the training set.\"\"\"\n",
    "    batch_metrics = []\n",
    "    for cnt, (imgs, labels) in enumerate(dataloader):\n",
    "        state, metrics = train_step(state, imgs, labels)\n",
    "        batch_metrics.append(metrics)\n",
    "\n",
    "    # Aggregate the metrics\n",
    "    batch_metrics_np = jax.device_get(batch_metrics)  # pull from the accelerator onto host (CPU)\n",
    "    epoch_metrics_np = {\n",
    "        k: np.mean([metrics[k] for metrics in batch_metrics_np])\n",
    "        for k in batch_metrics_np[0]\n",
    "    }\n",
    "\n",
    "    return state, epoch_metrics_np\n",
    "\n",
    "def evaluate_model(state, test_imgs, test_lbls):\n",
    "    \"\"\"Evaluate on the validation set.\"\"\"\n",
    "    metrics = eval_step(state, test_imgs, test_lbls)\n",
    "    metrics = jax.device_get(metrics)  # pull from the accelerator onto host (CPU)\n",
    "    metrics = jax.tree_map(lambda x: x.item(), metrics)  # np.ndarray -> scalar\n",
    "    return metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "xiV5yiA4BKEk"
   },
   "outputs": [],
   "source": [
    "# This one will keep things nice and tidy compared to our previous examples\n",
    "def create_train_state(key, learning_rate, momentum):\n",
    "    cnn = CNN()\n",
    "    params = cnn.init(key, jnp.ones([1, *mnist_img_size]))['params']\n",
    "    sgd_opt = optax.sgd(learning_rate, momentum)\n",
    "    # TrainState is a simple built-in wrapper class that makes things a bit cleaner\n",
    "    return train_state.TrainState.create(apply_fn=cnn.apply, params=params, tx=sgd_opt)\n",
    "\n",
    "def compute_metrics(*, logits, gt_labels):\n",
    "    one_hot_gt_labels = jax.nn.one_hot(gt_labels, num_classes=10)\n",
    "\n",
    "    loss = -jnp.mean(jnp.sum(one_hot_gt_labels * logits, axis=-1))\n",
    "    accuracy = jnp.mean(jnp.argmax(logits, -1) == gt_labels)\n",
    "\n",
    "    metrics = {\n",
    "        'loss': loss,\n",
    "        'accuracy': accuracy,\n",
    "    }\n",
    "    return metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "s8EFriHnVcJO",
    "outputId": "cb40714f-6150-44d6-e1e0-290b72a23eda"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train epoch: 1, loss: 0.2903152406215668, accuracy: 91.86198115348816\n",
      "Test epoch: 1, loss: 44.35035705566406, accuracy: 94.77999806404114\n",
      "Train epoch: 2, loss: 0.058339256793260574, accuracy: 98.23551177978516\n",
      "Test epoch: 2, loss: 17.13631820678711, accuracy: 97.33999967575073\n"
     ]
    }
   ],
   "source": [
    "# Finally let's define the high-level training/val loops\n",
    "seed = 0  # needless to say these should be in a config or defined like flags\n",
    "learning_rate = 0.1\n",
    "momentum = 0.9\n",
    "num_epochs = 2\n",
    "batch_size = 32\n",
    "\n",
    "train_state = create_train_state(jax.random.PRNGKey(seed), learning_rate, momentum)\n",
    "\n",
    "for epoch in range(1, num_epochs + 1):\n",
    "    train_state, train_metrics = train_one_epoch(train_state, train_loader, epoch)\n",
    "    print(f\"Train epoch: {epoch}, loss: {train_metrics['loss']}, accuracy: {train_metrics['accuracy'] * 100}\")\n",
    "\n",
    "    test_metrics = evaluate_model(train_state, test_images, test_lbls)\n",
    "    print(f\"Test epoch: {epoch}, loss: {test_metrics['loss']}, accuracy: {test_metrics['accuracy'] * 100}\")\n",
    "\n",
    "# todo: exercise - how would we go about adding dropout? What about BatchNorm? What would have to change?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6U-BIjQ1v4ff"
   },
   "source": [
    "Bonus point: a walk-through the \"non-toy\", distributed ImageNet CNN training example.\n",
    "\n",
    "Head over to https://github.com/google/flax/tree/main/examples/imagenet\n",
    "\n",
    "You'll keep seeing the same pattern/structure in all official Flax examples."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6Q4C2M2tv_0J"
   },
   "source": [
    "### Further learning resources 📚\n",
    "\n",
    "Aside from the [official docs](https://flax.readthedocs.io/en/latest/) and [examples](https://github.com/google/flax/tree/main/examples) I found [HuggingFace's Flax examples](https://github.com/huggingface/transformers/tree/master/examples/flax) and the resources from their [\"community week\"](https://github.com/huggingface/transformers/tree/master/examples/research_projects/jax-projects) useful as well.\n",
    "\n",
    "Finally, [source code](https://github.com/google/flax) is also your friend, as the library is still evolving."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "T5DqxlZ-SD3e"
   },
   "source": [
    "### Connect with me ❤️\n",
    "\n",
    "Last but not least I regularly post AI-related stuff (paper summaries, AI news, etc.) on my Twitter/LinkedIn. We also have an ever increasing Discord community (1600+ members at the time of writing this). If you care about any of these I encourage you to connect! \n",
    "\n",
    "Social: <br/>\n",
    "💼 LinkedIn - https://www.linkedin.com/in/aleksagordic/ <br/>\n",
    "🐦 Twitter - https://twitter.com/gordic_aleksa <br/>\n",
    "👨‍👩‍👧‍👦 Discord - https://discord.gg/peBrCpheKE <br/>\n",
    "🙏 Patreon - https://www.patreon.com/theaiepiphany <br/>\n",
    "\n",
    "Content: <br/>\n",
    "📺 YouTube - https://www.youtube.com/c/TheAIEpiphany/ <br/>\n",
    "📚 Medium - https://gordicaleksa.medium.com/ <br/>\n",
    "💻 GitHub - https://github.com/gordicaleksa <br/>\n",
    "📢 AI Newsletter - https://aiepiphany.substack.com/ <br/>"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "Tutorial 4: Flax Zero2Hero.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}