diff --git a/doc/pub/catania/ipynb/catania.ipynb b/doc/pub/catania/ipynb/catania.ipynb index a32f77d7..11ee8569 100644 --- a/doc/pub/catania/ipynb/catania.ipynb +++ b/doc/pub/catania/ipynb/catania.ipynb @@ -3,9 +3,7 @@ { "cell_type": "markdown", "id": "0cd93842", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "\n", @@ -15,9 +13,7 @@ { "cell_type": "markdown", "id": "653c9ba3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "# Mathematics of discriminative and generative deep learning, from deep neural networks to diffusion models\n", "**Morten Hjorth-Jensen**, Department of Physics and Center for Computing in Science Education, University of Oslo, Norway and Department of Physics and Astronomy and Facility for Rare Isotope Beams, Michigan State University, East Lansing, Michigan, USA\n", @@ -28,9 +24,7 @@ { "cell_type": "markdown", "id": "edbeb108", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Types of machine learning\n", "\n", @@ -47,9 +41,7 @@ { "cell_type": "markdown", "id": "08b3ff3a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Main categories\n", "Another way to categorize machine learning tasks is to consider the desired output of a system.\n", @@ -65,9 +57,7 @@ { "cell_type": "markdown", "id": "acb8af9d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Machine learning. A simple perspective on the interface between ML and Physics\n", "\n", @@ -81,9 +71,7 @@ { "cell_type": "markdown", "id": "df158234", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## ML in Nuclear Physics (or any field in physics)\n", "\n", @@ -97,9 +85,7 @@ { "cell_type": "markdown", "id": "2c394af4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The plethora of machine learning algorithms/methods\n", "\n", @@ -119,9 +105,7 @@ { "cell_type": "markdown", "id": "17f9e522", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Extrapolations and model interpretability\n", "\n", @@ -141,9 +125,7 @@ { "cell_type": "markdown", "id": "5d68e865", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Generative and discriminative models\n", "\n", @@ -159,9 +141,7 @@ { "cell_type": "markdown", "id": "d3540c12", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## [Dilute neutron star matter from neural-network quantum states by Fore et al, Physical Review Research 5, 033062 (2023)](https://journals.aps.org/prresearch/pdf/10.1103/PhysRevResearch.5.033062) at density $\\rho=0.04$ fm$^{-3}$\n", "\n", @@ -175,9 +155,7 @@ { "cell_type": "markdown", "id": "08fa3db3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The electron gas in three dimensions with $N=14$ electrons (Wigner-Seitz radius $r_s=2$ a.u.), [Gabriel Pescia, Jane Kim et al. arXiv.2305.07240,](https://doi.org/10.48550/arXiv.2305.07240)\n", "\n", @@ -191,9 +169,7 @@ { "cell_type": "markdown", "id": "cd28f343", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## What Is Generative Modeling?\n", "\n", @@ -214,9 +190,7 @@ { "cell_type": "markdown", "id": "58b00c74", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Example of generative modeling, [taken from Generative Deep Learning by David Foster](https://www.oreilly.com/library/view/generative-deep-learning/9781098134174/ch01.html)\n", "\n", @@ -230,9 +204,7 @@ { "cell_type": "markdown", "id": "cd7ea407", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Generative Modeling\n", "\n", @@ -256,9 +228,7 @@ { "cell_type": "markdown", "id": "165edfcd", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Generative Versus Discriminative Modeling\n", "\n", @@ -272,9 +242,7 @@ { "cell_type": "markdown", "id": "7b5105e2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Example of discriminative modeling, [taken from Generative Deeep Learning by David Foster](https://www.oreilly.com/library/view/generative-deep-learning/9781098134174/ch01.html)\n", "\n", @@ -288,9 +256,7 @@ { "cell_type": "markdown", "id": "d4b748ef", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Discriminative Modeling\n", "\n", @@ -308,9 +274,7 @@ { "cell_type": "markdown", "id": "6fe09ccb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Taxonomy of generative deep learning, [taken from Generative Deep Learning by David Foster](https://www.oreilly.com/library/view/generative-deep-learning/9781098134174/ch01.html)\n", "\n", @@ -324,9 +288,7 @@ { "cell_type": "markdown", "id": "81270aed", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Good books with hands-on material and codes\n", "* [Sebastian Rashcka et al, Machine learning with Sickit-Learn and PyTorch](https://sebastianraschka.com/blog/2022/ml-pytorch-book.html)\n", @@ -342,9 +304,7 @@ { "cell_type": "markdown", "id": "bfe945b6", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## More references\n", "\n", @@ -366,9 +326,7 @@ { "cell_type": "markdown", "id": "ba7ef646", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## What are the basic Machine Learning ingredients?\n", "Almost every problem in ML and data science starts with the same ingredients:\n", @@ -384,9 +342,7 @@ { "cell_type": "markdown", "id": "b834cc04", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Low-level machine learning, the family of ordinary least squares methods\n", "\n", @@ -400,9 +356,7 @@ { "cell_type": "markdown", "id": "da8627ae", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{y}=f(\\boldsymbol{x})+\\boldsymbol{\\epsilon}.\n", @@ -412,9 +366,7 @@ { "cell_type": "markdown", "id": "e849da7b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Setting up the equations\n", "\n", @@ -430,9 +382,7 @@ { "cell_type": "markdown", "id": "65a452bd", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{\\tilde{y}}= \\boldsymbol{X}\\boldsymbol{\\theta},\n", @@ -442,9 +392,7 @@ { "cell_type": "markdown", "id": "9deb7a37", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The objective/cost/loss function\n", "\n", @@ -454,9 +402,7 @@ { "cell_type": "markdown", "id": "38f213f1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\boldsymbol{\\Theta})=\\frac{1}{n}\\sum_{i=0}^{n-1}\\left(y_i-\\tilde{y}_i\\right)^2=\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}}\\right)\\right\\},\n", @@ -466,9 +412,7 @@ { "cell_type": "markdown", "id": "45ae1136", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "or using the matrix $\\boldsymbol{X}$ and in a more compact matrix-vector notation as" ] @@ -476,9 +420,7 @@ { "cell_type": "markdown", "id": "f66c26e8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\boldsymbol{\\Theta})=\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)\\right\\}.\n", @@ -488,9 +430,7 @@ { "cell_type": "markdown", "id": "dd3a5090", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "This function represents one of many possible ways to define the so-called cost function." ] @@ -498,9 +438,7 @@ { "cell_type": "markdown", "id": "7872d11e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Training solution\n", "\n", @@ -510,9 +448,7 @@ { "cell_type": "markdown", "id": "ad0213af", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{X}^T\\boldsymbol{y} = \\boldsymbol{X}^T\\boldsymbol{X}\\boldsymbol{\\theta},\n", @@ -522,9 +458,7 @@ { "cell_type": "markdown", "id": "d78419a1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and if the matrix $\\boldsymbol{X}^T\\boldsymbol{X}$ is invertible we have the optimal values" ] @@ -532,9 +466,7 @@ { "cell_type": "markdown", "id": "fb800d02", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\hat{\\boldsymbol{\\theta}} =\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", @@ -544,9 +476,7 @@ { "cell_type": "markdown", "id": "7f487392", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We say we 'learn' the unknown parameters $\\boldsymbol{\\theta}$ from the last equation." ] @@ -554,9 +484,7 @@ { "cell_type": "markdown", "id": "e39f7c17", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Ridge and LASSO Regression\n", "\n", @@ -566,9 +494,7 @@ { "cell_type": "markdown", "id": "aca472b7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "{\\displaystyle \\min_{\\boldsymbol{\\theta}\\in {\\mathbb{R}}^{p}}}\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)\\right\\}.\n", @@ -578,9 +504,7 @@ { "cell_type": "markdown", "id": "b7987ee7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "or we can state it as" ] @@ -588,9 +512,7 @@ { "cell_type": "markdown", "id": "8ed718ee", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "{\\displaystyle \\min_{\\boldsymbol{\\theta}\\in\n", @@ -601,9 +523,7 @@ { "cell_type": "markdown", "id": "15bbe9c3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where we have used the definition of a norm-2 vector, that is" ] @@ -611,9 +531,7 @@ { "cell_type": "markdown", "id": "7827172f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\vert\\vert \\boldsymbol{x}\\vert\\vert_2 = \\sqrt{\\sum_i x_i^2}.\n", @@ -623,9 +541,7 @@ { "cell_type": "markdown", "id": "52a8084f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## From OLS to Ridge and Lasso\n", "\n", @@ -638,9 +554,7 @@ { "cell_type": "markdown", "id": "67569afe", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "{\\displaystyle \\min_{\\boldsymbol{\\theta}\\in\n", @@ -651,9 +565,7 @@ { "cell_type": "markdown", "id": "be6c3913", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which leads to the Ridge regression minimization problem where we\n", "require that $\\vert\\vert \\boldsymbol{\\theta}\\vert\\vert_2^2\\le t$, where $t$ is\n", @@ -663,9 +575,7 @@ { "cell_type": "markdown", "id": "8b9c9420", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Lasso regression\n", "\n", @@ -675,9 +585,7 @@ { "cell_type": "markdown", "id": "884822e5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\boldsymbol{X},\\boldsymbol{\\theta})=\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\vert\\vert_2^2+\\lambda\\vert\\vert \\boldsymbol{\\theta}\\vert\\vert_1,\n", @@ -687,9 +595,7 @@ { "cell_type": "markdown", "id": "20fb19fc", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "we have a new optimization equation" ] @@ -697,9 +603,7 @@ { "cell_type": "markdown", "id": "ba368a55", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "{\\displaystyle \\min_{\\boldsymbol{\\theta}\\in\n", @@ -710,9 +614,7 @@ { "cell_type": "markdown", "id": "6de29ccc", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator. \n", "Here we have defined the norm-1 as" @@ -721,9 +623,7 @@ { "cell_type": "markdown", "id": "e083172b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\vert\\vert \\boldsymbol{x}\\vert\\vert_1 = \\sum_i \\vert x_i\\vert.\n", @@ -733,9 +633,7 @@ { "cell_type": "markdown", "id": "4dff529c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Selected references\n", "* [Mehta et al.](https://arxiv.org/abs/1803.08823) and [Physics Reports (2019)](https://www.sciencedirect.com/science/article/pii/S0370157319300766?via%3Dihub).\n", @@ -756,9 +654,7 @@ { "cell_type": "markdown", "id": "7facb953", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Setting up the basic equations for neural networks\n", "\n", @@ -778,9 +674,7 @@ { "cell_type": "markdown", "id": "803a20ac", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Overarching view of a neural network\n", "\n", @@ -805,9 +699,7 @@ { "cell_type": "markdown", "id": "63aa6478", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Illustration of a single perceptron model and a multilayer FFNN\n", "\n", @@ -821,9 +713,7 @@ { "cell_type": "markdown", "id": "93b10615", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The optimization problem\n", "\n", @@ -838,9 +728,7 @@ { "cell_type": "markdown", "id": "bcdc8f2e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\boldsymbol{\\Theta})=\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)\\right\\}.\n", @@ -850,9 +738,7 @@ { "cell_type": "markdown", "id": "63764264", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "This function represents one of many possible ways to define\n", "the so-called cost function." @@ -861,9 +747,7 @@ { "cell_type": "markdown", "id": "6ea4a585", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Weights and biases\n", "\n", @@ -880,9 +764,7 @@ { "cell_type": "markdown", "id": "f3d0b2f4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Other ingredients of a neural network\n", "\n", @@ -909,9 +791,7 @@ { "cell_type": "markdown", "id": "72e9a70e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Other parameters\n", "\n", @@ -923,9 +803,7 @@ { "cell_type": "markdown", "id": "24166378", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Why Feed Forward Neural Networks (FFNN)?\n", "\n", @@ -940,9 +818,7 @@ { "cell_type": "markdown", "id": "ad0e414e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Universal approximation theorem\n", "\n", @@ -956,9 +832,7 @@ { "cell_type": "markdown", "id": "4a7b41e1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\sigma(z) = \\left\\{\\begin{array}{cc} 1 & z\\rightarrow \\infty\\\\ 0 & z \\rightarrow -\\infty \\end{array}\\right.\n", @@ -968,9 +842,7 @@ { "cell_type": "markdown", "id": "c4acf546", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Given a continuous and deterministic function $F(\\boldsymbol{x})$ on the unit\n", "cube in $d$-dimensions $F\\in [0,1]^d$, $x\\in [0,1]^d$ and a parameter\n", @@ -982,9 +854,7 @@ { "cell_type": "markdown", "id": "e353dc32", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\vert F(\\boldsymbol{x})-f(\\boldsymbol{x};\\boldsymbol{\\Theta})\\vert < \\epsilon \\hspace{0.1cm} \\forall \\boldsymbol{x}\\in[0,1]^d.\n", @@ -994,9 +864,7 @@ { "cell_type": "markdown", "id": "5a87dccb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The approximation theorem in words\n", "\n", @@ -1010,9 +878,7 @@ { "cell_type": "markdown", "id": "35491e6c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mathbb{E}[\\vert F(\\boldsymbol{x})\\vert^2] =\\int_{\\boldsymbol{x}\\in D} \\vert F(\\boldsymbol{x})\\vert^2p(\\boldsymbol{x})d\\boldsymbol{x} < \\infty.\n", @@ -1022,9 +888,7 @@ { "cell_type": "markdown", "id": "b809821f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Then we have" ] @@ -1032,9 +896,7 @@ { "cell_type": "markdown", "id": "774f0764", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mathbb{E}[\\vert F(\\boldsymbol{x})-f(\\boldsymbol{x};\\boldsymbol{\\Theta})\\vert^2] =\\int_{\\boldsymbol{x}\\in D} \\vert F(\\boldsymbol{x})-f(\\boldsymbol{x};\\boldsymbol{\\Theta})\\vert^2p(\\boldsymbol{x})d\\boldsymbol{x} < \\epsilon.\n", @@ -1044,9 +906,7 @@ { "cell_type": "markdown", "id": "3d7088d0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## More on the general approximation theorem\n", "\n", @@ -1062,9 +922,7 @@ { "cell_type": "markdown", "id": "ec2718cd", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Class of functions we can approximate\n", "\n", @@ -1075,9 +933,7 @@ { "cell_type": "markdown", "id": "8ef486d5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Simple example, fitting nuclear masses\n", "\n", @@ -1089,9 +945,7 @@ { "cell_type": "markdown", "id": "855549f0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## First network example, simple percepetron with one input\n", "\n", @@ -1104,9 +958,7 @@ { "cell_type": "markdown", "id": "6467d797", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "z_1 = w_1x+b_1,\n", @@ -1116,9 +968,7 @@ { "cell_type": "markdown", "id": "c2b8c002", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where $w_1$ is the weight and $b_1$ is the bias. These are the\n", "parameters we want to optimize. The output is $a_1=\\sigma(z_1)$ (see\n", @@ -1130,9 +980,7 @@ { "cell_type": "markdown", "id": "45b6c18c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(x;w_1,b_1)=\\frac{1}{2}(a_1-y)^2.\n", @@ -1142,9 +990,7 @@ { "cell_type": "markdown", "id": "e6020883", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Optimizing the parameters\n", "\n", @@ -1158,9 +1004,7 @@ { "cell_type": "markdown", "id": "87360dbd", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial C}{\\partial w_1} \\hspace{0.1cm}\\mathrm{and}\\hspace{0.1cm}\\frac{\\partial C}{\\partial b_1}.\n", @@ -1170,9 +1014,7 @@ { "cell_type": "markdown", "id": "392dda6a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Using the chain rule we find" ] @@ -1180,9 +1022,7 @@ { "cell_type": "markdown", "id": "e7ab5332", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial C}{\\partial w_1}=\\frac{\\partial C}{\\partial a_1}\\frac{\\partial a_1}{\\partial z_1}\\frac{\\partial z_1}{\\partial w_1}=(a_1-y)\\sigma_1'x,\n", @@ -1192,9 +1032,7 @@ { "cell_type": "markdown", "id": "f1f1bcc8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and" ] @@ -1202,9 +1040,7 @@ { "cell_type": "markdown", "id": "31c9897a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial C}{\\partial b_1}=\\frac{\\partial C}{\\partial a_1}\\frac{\\partial a_1}{\\partial z_1}\\frac{\\partial z_1}{\\partial b_1}=(a_1-y)\\sigma_1',\n", @@ -1214,9 +1050,7 @@ { "cell_type": "markdown", "id": "cacef68b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which we later will just define as" ] @@ -1224,9 +1058,7 @@ { "cell_type": "markdown", "id": "a310325f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial C}{\\partial a_1}\\frac{\\partial a_1}{\\partial z_1}=\\delta_1.\n", @@ -1236,9 +1068,7 @@ { "cell_type": "markdown", "id": "e440e4eb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Implementing the simple perceptron model\n", "\n", @@ -1251,9 +1081,7 @@ { "cell_type": "markdown", "id": "c6e7a1c8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(y,w_1,b_1)=\\frac{1}{2}(a_1-y)^2,\n", @@ -1263,9 +1091,7 @@ { "cell_type": "markdown", "id": "445560fe", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "with $a_1$ the output from the network." ] @@ -1274,11 +1100,16 @@ "cell_type": "code", "execution_count": 1, "id": "79d0d646", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "4.0022640019432767e-07\n" + ] + } + ], "source": [ "%matplotlib inline\n", "\n", @@ -1337,9 +1168,7 @@ { "cell_type": "markdown", "id": "42993345", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Running this code gives us an acceptable results after some 40-50 iterations. Note that the results depend on the value of the learning rate." ] @@ -1347,9 +1176,7 @@ { "cell_type": "markdown", "id": "a9d5deec", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Exercise 1: Extensions to the above code\n", "\n", @@ -1368,9 +1195,7 @@ { "cell_type": "markdown", "id": "4d4929b1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Adding a hidden layer\n", "\n", @@ -1384,9 +1209,7 @@ { "cell_type": "markdown", "id": "16680e62", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "z_1 = w_1a_0+b_1 \\hspace{0.1cm} \\wedge a_1 = \\sigma_1(z_1),\n", @@ -1396,9 +1219,7 @@ { "cell_type": "markdown", "id": "4e6d767a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "z_2 = w_2a_1+b_2 \\hspace{0.1cm} \\wedge a_2 = \\sigma_2(z_2),\n", @@ -1408,9 +1229,7 @@ { "cell_type": "markdown", "id": "3263f740", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and the cost function" ] @@ -1418,9 +1237,7 @@ { "cell_type": "markdown", "id": "9c2a591a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(x;\\boldsymbol{\\Theta})=\\frac{1}{2}(a_2-y)^2,\n", @@ -1430,9 +1247,7 @@ { "cell_type": "markdown", "id": "f406da56", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "with $\\boldsymbol{\\Theta}=[w_1,w_2,b_1,b_2]$." ] @@ -1440,9 +1255,7 @@ { "cell_type": "markdown", "id": "1793a059", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The derivatives\n", "\n", @@ -1452,9 +1265,7 @@ { "cell_type": "markdown", "id": "f9194184", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial C}{\\partial w_2}=\\frac{\\partial C}{\\partial a_2}\\frac{\\partial a_2}{\\partial z_2}\\frac{\\partial z_2}{\\partial w_2}=(a_2-y)\\sigma_2'a_1=\\delta_2a_1,\n", @@ -1464,9 +1275,7 @@ { "cell_type": "markdown", "id": "1e573596", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial C}{\\partial b_2}=\\frac{\\partial C}{\\partial a_2}\\frac{\\partial a_2}{\\partial z_2}\\frac{\\partial z_2}{\\partial b_2}=(a_2-y)\\sigma_2'=\\delta_2,\n", @@ -1476,9 +1285,7 @@ { "cell_type": "markdown", "id": "44cd323c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial C}{\\partial w_1}=\\frac{\\partial C}{\\partial a_2}\\frac{\\partial a_2}{\\partial z_2}\\frac{\\partial z_2}{\\partial a_1}\\frac{\\partial a_1}{\\partial z_1}\\frac{\\partial z_1}{\\partial w_1}=(a_2-y)\\sigma_2'a_1\\sigma_1'a_0,\n", @@ -1488,9 +1295,7 @@ { "cell_type": "markdown", "id": "98b20680", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial C}{\\partial b_1}=\\frac{\\partial C}{\\partial a_2}\\frac{\\partial a_2}{\\partial z_2}\\frac{\\partial z_2}{\\partial a_1}\\frac{\\partial a_1}{\\partial z_1}\\frac{\\partial z_1}{\\partial b_1}=(a_2-y)\\sigma_2'\\sigma_1'=\\delta_1.\n", @@ -1500,9 +1305,7 @@ { "cell_type": "markdown", "id": "e0f06329", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Can you generalize this to more than one hidden layer?" ] @@ -1510,9 +1313,7 @@ { "cell_type": "markdown", "id": "43161030", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Important observations\n", "\n", @@ -1526,9 +1327,7 @@ { "cell_type": "markdown", "id": "2ef1c290", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The training\n", "\n", @@ -1538,9 +1337,7 @@ { "cell_type": "markdown", "id": "bd16d507", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "w_{i}\\leftarrow w_{i}- \\eta \\delta_i a_{i-1},\n", @@ -1550,9 +1347,7 @@ { "cell_type": "markdown", "id": "3e536774", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and" ] @@ -1560,9 +1355,7 @@ { "cell_type": "markdown", "id": "379286f4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "b_i \\leftarrow b_i-\\eta \\delta_i,\n", @@ -1572,9 +1365,7 @@ { "cell_type": "markdown", "id": "de4009b3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "with $\\eta$ is the learning rate.\n", "\n", @@ -1586,9 +1377,7 @@ { "cell_type": "markdown", "id": "7692a4d5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Code example\n", "\n", @@ -1604,11 +1393,65 @@ "cell_type": "code", "execution_count": 2, "id": "d4814e9c", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[36.89563074]\n", + "[23.62323175]\n", + "[15.1251681]\n", + "[9.68402334]\n", + "[6.20020163]\n", + "[3.96963458]\n", + "[2.54150298]\n", + "[1.62714703]\n", + "[1.0417409]\n", + "[0.66694492]\n", + "[0.42699034]\n", + "[0.27336592]\n", + "[0.17501258]\n", + "[0.11204514]\n", + "[0.07173249]\n", + "[0.04592382]\n", + "[0.02940083]\n", + "[0.01882264]\n", + "[0.01205039]\n", + "[0.00771475]\n", + "[0.00493903]\n", + "[0.003162]\n", + "[0.00202433]\n", + "[0.00129599]\n", + "[0.0008297]\n", + "[0.00053118]\n", + "[0.00034006]\n", + "[0.00021771]\n", + "[0.00013938]\n", + "[8.92313548e-05]\n", + "[5.71263851e-05]\n", + "[3.6572612e-05]\n", + "[2.34139775e-05]\n", + "[1.49897504e-05]\n", + "[9.5965161e-06]\n", + "[6.14373934e-06]\n", + "[3.93325371e-06]\n", + "[2.51808934e-06]\n", + "[1.61209378e-06]\n", + "[1.03207075e-06]\n", + "[6.60737006e-07]\n", + "[4.23007231e-07]\n", + "[2.70811405e-07]\n", + "[1.73374853e-07]\n", + "[1.10995472e-07]\n", + "[7.10598715e-08]\n", + "[4.54928947e-08]\n", + "[2.91247848e-08]\n", + "[1.86458368e-08]\n", + "[1.19371605e-08]\n" + ] + } + ], "source": [ "import numpy as np\n", "# We use the Sigmoid function as activation function\n", @@ -1677,9 +1520,7 @@ { "cell_type": "markdown", "id": "cd7ae3ed", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We see that after some few iterations (the results do depend on the learning rate however), we get an error which is rather small." ] @@ -1687,9 +1528,7 @@ { "cell_type": "markdown", "id": "6bb9a0e1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Exercise 2: Including more data\n", "\n", @@ -1705,9 +1544,7 @@ { "cell_type": "markdown", "id": "06563003", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Simple neural network and the back propagation equations\n", "\n", @@ -1722,9 +1559,7 @@ { "cell_type": "markdown", "id": "5a3fe09a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "x_0 = a_0^{(0)} \\wedge x_1 = a_1^{(0)}.\n", @@ -1734,9 +1569,7 @@ { "cell_type": "markdown", "id": "321b4a6d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "The hidden layer (layer $(1)$) has nodes which yield the outputs $a_0^{(1)}$ and $a_1^{(1)}$) with weight $\\boldsymbol{w}$ and bias $\\boldsymbol{b}$ parameters" ] @@ -1744,9 +1577,7 @@ { "cell_type": "markdown", "id": "3a8f136c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "w_{ij}^{(1)}=\\left\\{w_{00}^{(1)},w_{01}^{(1)},w_{10}^{(1)},w_{11}^{(1)}\\right\\} \\wedge b^{(1)}=\\left\\{b_0^{(1)},b_1^{(1)}\\right\\}.\n", @@ -1756,9 +1587,7 @@ { "cell_type": "markdown", "id": "bb33455e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The ouput layer\n", "\n", @@ -1768,9 +1597,7 @@ { "cell_type": "markdown", "id": "4b97a8d6", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "w_{i}^{(2)}=\\left\\{w_{0}^{(2)},w_{1}^{(2)}\\right\\} \\wedge b^{(2)}.\n", @@ -1780,9 +1607,7 @@ { "cell_type": "markdown", "id": "cfbbb52d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Our output is $\\tilde{y}=a^{(2)}$ and we define a generic cost function $C(a^{(2)},y;\\boldsymbol{\\Theta})$ where $y$ is the target value (a scalar here).\n", "The parameters we need to optimize are given by" @@ -1791,9 +1616,7 @@ { "cell_type": "markdown", "id": "a3f8f887", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{\\Theta}=\\left\\{w_{00}^{(1)},w_{01}^{(1)},w_{10}^{(1)},w_{11}^{(1)},w_{0}^{(2)},w_{1}^{(2)},b_0^{(1)},b_1^{(1)},b^{(2)}\\right\\}.\n", @@ -1803,9 +1626,7 @@ { "cell_type": "markdown", "id": "a097c6e9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Compact expressions\n", "\n", @@ -1816,9 +1637,7 @@ { "cell_type": "markdown", "id": "9ec6339d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{bmatrix}z_0^{(1)} \\\\ z_1^{(1)} \\end{bmatrix}=\\begin{bmatrix}w_{00}^{(1)} & w_{01}^{(1)}\\\\ w_{10}^{(1)} &w_{11}^{(1)} \\end{bmatrix}\\begin{bmatrix}a_0^{(0)} \\\\ a_1^{(0)} \\end{bmatrix}+\\begin{bmatrix}b_0^{(1)} \\\\ b_1^{(1)} \\end{bmatrix},\n", @@ -1828,9 +1647,7 @@ { "cell_type": "markdown", "id": "b340d6f5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "with outputs" ] @@ -1838,9 +1655,7 @@ { "cell_type": "markdown", "id": "3bc67ced", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{bmatrix}a_0^{(1)} \\\\ a_1^{(1)} \\end{bmatrix}=\\begin{bmatrix}\\sigma^{(1)}(z_0^{(1)}) \\\\ \\sigma^{(1)}(z_1^{(1)}) \\end{bmatrix}.\n", @@ -1850,9 +1665,7 @@ { "cell_type": "markdown", "id": "fa8e855e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Output layer\n", "\n", @@ -1862,9 +1675,7 @@ { "cell_type": "markdown", "id": "fec79799", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "z^{(2)} = w_{0}^{(2)}a_0^{(1)} +w_{1}^{(2)}a_1^{(1)}+b^{(2)},\n", @@ -1874,9 +1685,7 @@ { "cell_type": "markdown", "id": "e92e9b1f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "resulting in the output" ] @@ -1884,9 +1693,7 @@ { "cell_type": "markdown", "id": "4b4e69bf", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "a^{(2)}=\\sigma^{(2)}(z^{(2)}).\n", @@ -1896,9 +1703,7 @@ { "cell_type": "markdown", "id": "a660f211", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Explicit derivatives\n", "\n", @@ -1912,9 +1717,7 @@ { "cell_type": "markdown", "id": "1989b4fc", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial C}{\\partial w_{i}^{(2)}}=\\frac{\\partial C}{\\partial a^{(2)}}\\frac{\\partial a^{(2)}}{\\partial z^{(2)}}\\frac{\\partial z^{(2)}}{\\partial w_{i}^{(2)}}=\\delta^{(2)}a_i^{(1)},\n", @@ -1924,9 +1727,7 @@ { "cell_type": "markdown", "id": "4a034c14", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "with" ] @@ -1934,9 +1735,7 @@ { "cell_type": "markdown", "id": "274e1c4d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\delta^{(2)}=\\frac{\\partial C}{\\partial a^{(2)}}\\frac{\\partial a^{(2)}}{\\partial z^{(2)}}\n", @@ -1946,9 +1745,7 @@ { "cell_type": "markdown", "id": "533ae9e7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and finally" ] @@ -1956,9 +1753,7 @@ { "cell_type": "markdown", "id": "bea1c60a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial C}{\\partial b^{(2)}}=\\frac{\\partial C}{\\partial a^{(2)}}\\frac{\\partial a^{(2)}}{\\partial z^{(2)}}\\frac{\\partial z^{(2)}}{\\partial b^{(2)}}=\\delta^{(2)}.\n", @@ -1968,9 +1763,7 @@ { "cell_type": "markdown", "id": "5ce5b908", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Derivatives of the hidden layer\n", "\n", @@ -1980,9 +1773,7 @@ { "cell_type": "markdown", "id": "324b5c28", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial C}{\\partial w_{00}^{(1)}}=\\frac{\\partial C}{\\partial a^{(2)}}\\frac{\\partial a^{(2)}}{\\partial z^{(2)}}\n", @@ -1993,9 +1784,7 @@ { "cell_type": "markdown", "id": "adff05cd", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which, noting that" ] @@ -2003,9 +1792,7 @@ { "cell_type": "markdown", "id": "5270d5fa", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "z^{(2)} =w_0^{(2)}a_0^{(1)}+w_1^{(2)}a_1^{(1)}+b^{(2)},\n", @@ -2015,9 +1802,7 @@ { "cell_type": "markdown", "id": "c83a9ef4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "allows us to rewrite" ] @@ -2025,9 +1810,7 @@ { "cell_type": "markdown", "id": "747da5ad", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial z^{(2)}}{\\partial z_0^{(1)}}\\frac{\\partial z_0^{(1)}}{\\partial w_{00}^{(1)}}=w_0^{(2)}\\frac{\\partial a_0^{(1)}}{\\partial z_0^{(1)}}a_0^{(1)}.\n", @@ -2037,9 +1820,7 @@ { "cell_type": "markdown", "id": "db9aa150", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Final expression\n", "Defining" @@ -2048,9 +1829,7 @@ { "cell_type": "markdown", "id": "fe3ce041", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\delta_0^{(1)}=w_0^{(2)}\\frac{\\partial a_0^{(1)}}{\\partial z_0^{(1)}}\\delta^{(2)},\n", @@ -2060,9 +1839,7 @@ { "cell_type": "markdown", "id": "1d7cbe9d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "we have" ] @@ -2070,9 +1847,7 @@ { "cell_type": "markdown", "id": "9e16a856", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial C}{\\partial w_{00}^{(1)}}=\\delta_0^{(1)}a_0^{(1)}.\n", @@ -2082,9 +1857,7 @@ { "cell_type": "markdown", "id": "095624ba", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Similarly, we obtain" ] @@ -2092,9 +1865,7 @@ { "cell_type": "markdown", "id": "bd65ea09", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial C}{\\partial w_{01}^{(1)}}=\\delta_0^{(1)}a_1^{(1)}.\n", @@ -2104,9 +1875,7 @@ { "cell_type": "markdown", "id": "008e4371", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Completing the list\n", "\n", @@ -2116,9 +1885,7 @@ { "cell_type": "markdown", "id": "f55b8eab", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial C}{\\partial w_{10}^{(1)}}=\\delta_1^{(1)}a_0^{(1)},\n", @@ -2128,9 +1895,7 @@ { "cell_type": "markdown", "id": "5126ab8e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and" ] @@ -2138,9 +1903,7 @@ { "cell_type": "markdown", "id": "0f506e21", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial C}{\\partial w_{11}^{(1)}}=\\delta_1^{(1)}a_1^{(1)},\n", @@ -2150,9 +1913,7 @@ { "cell_type": "markdown", "id": "f09773bb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where we have defined" ] @@ -2160,9 +1921,7 @@ { "cell_type": "markdown", "id": "13faa8b3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\delta_1^{(1)}=w_1^{(2)}\\frac{\\partial a_1^{(1)}}{\\partial z_1^{(1)}}\\delta^{(2)}.\n", @@ -2172,9 +1931,7 @@ { "cell_type": "markdown", "id": "a1b55a23", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Final expressions for the biases of the hidden layer\n", "\n", @@ -2184,9 +1941,7 @@ { "cell_type": "markdown", "id": "f544649c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial C}{\\partial b_{0}^{(1)}}=\\delta_0^{(1)},\n", @@ -2196,9 +1951,7 @@ { "cell_type": "markdown", "id": "9406250d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and" ] @@ -2206,9 +1959,7 @@ { "cell_type": "markdown", "id": "65f3005e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial C}{\\partial b_{1}^{(1)}}=\\delta_1^{(1)}.\n", @@ -2218,9 +1969,7 @@ { "cell_type": "markdown", "id": "df7ed34c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "As we will see below, these expressions can be generalized in a more compact form." ] @@ -2228,9 +1977,7 @@ { "cell_type": "markdown", "id": "d4c8367a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Gradient expressions\n", "\n", @@ -2241,9 +1988,7 @@ { "cell_type": "markdown", "id": "8153eaa3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "w_{i}^{(2)}\\leftarrow w_{i}^{(2)}- \\eta \\delta^{(2)} a_{i}^{(1)},\n", @@ -2253,9 +1998,7 @@ { "cell_type": "markdown", "id": "a6e2ae3c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and" ] @@ -2263,9 +2006,7 @@ { "cell_type": "markdown", "id": "1d1a819f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "b^{(2)} \\leftarrow b^{(2)}-\\eta \\delta^{(2)},\n", @@ -2275,9 +2016,7 @@ { "cell_type": "markdown", "id": "ae342d50", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and" ] @@ -2285,9 +2024,7 @@ { "cell_type": "markdown", "id": "f10677c1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "w_{ij}^{(1)}\\leftarrow w_{ij}^{(1)}- \\eta \\delta_{i}^{(1)} a_{j}^{(0)},\n", @@ -2297,9 +2034,7 @@ { "cell_type": "markdown", "id": "dea5c92f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and" ] @@ -2307,9 +2042,7 @@ { "cell_type": "markdown", "id": "208ad938", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "b_{i}^{(1)} \\leftarrow b_{i}^{(1)}-\\eta \\delta_{i}^{(1)},\n", @@ -2319,9 +2052,7 @@ { "cell_type": "markdown", "id": "48f04f55", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where $\\eta$ is the learning rate." ] @@ -2329,9 +2060,7 @@ { "cell_type": "markdown", "id": "fc04866a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Exercise 3: Extended program\n", "\n", @@ -2341,9 +2070,7 @@ { "cell_type": "markdown", "id": "d8ba650b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "y=f(x_0,x_1)=x_0^2+3x_0x_1+x_1^2+5.\n", @@ -2353,9 +2080,7 @@ { "cell_type": "markdown", "id": "9fc12f59", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We feed our network with $n=100$ entries $x_0$ and $x_1$. We have thus two features represented by these variable and an input matrix/design matrix $\\boldsymbol{X}\\in \\mathbf{R}^{n\\times 2}$" ] @@ -2363,9 +2088,7 @@ { "cell_type": "markdown", "id": "53bffa05", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{X}=\\begin{bmatrix} x_{00} & x_{01} \\\\ x_{00} & x_{01} \\\\ x_{10} & x_{11} \\\\ x_{20} & x_{21} \\\\ \\dots & \\dots \\\\ \\dots & \\dots \\\\ x_{n-20} & x_{n-21} \\\\ x_{n-10} & x_{n-11} \\end{bmatrix}.\n", @@ -2375,9 +2098,7 @@ { "cell_type": "markdown", "id": "6203af4d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Write a code, based on the previous code examples, which takes as input these data and fit the above function.\n", "You can extend your code to include automatic differentiation.\n", @@ -2388,9 +2109,7 @@ { "cell_type": "markdown", "id": "16eb0b7d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Getting serious, the back propagation equations for a neural network\n", "\n", @@ -2402,9 +2121,7 @@ { "cell_type": "markdown", "id": "f10f405a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial{\\cal C}((\\boldsymbol{\\Theta}^L)}{\\partial w_{jk}^L} = \\left(a_j^L - y_j\\right)a_j^L(1-a_j^L)a_k^{L-1},\n", @@ -2414,9 +2131,7 @@ { "cell_type": "markdown", "id": "36bce326", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Defining" ] @@ -2424,9 +2139,7 @@ { "cell_type": "markdown", "id": "ca2b8428", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\delta_j^L = a_j^L(1-a_j^L)\\left(a_j^L - y_j\\right) = \\sigma'(z_j^L)\\frac{\\partial {\\cal C}}{\\partial (a_j^L)},\n", @@ -2436,9 +2149,7 @@ { "cell_type": "markdown", "id": "ba48e670", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and using the Hadamard product of two vectors we can write this as" ] @@ -2446,9 +2157,7 @@ { "cell_type": "markdown", "id": "6f4abc32", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{\\delta}^L = \\sigma'(\\hat{z}^L)\\circ\\frac{\\partial {\\cal C}}{\\partial (\\boldsymbol{a}^L)}.\n", @@ -2458,9 +2167,7 @@ { "cell_type": "markdown", "id": "3e0c9116", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Analyzing the last results\n", "\n", @@ -2476,9 +2183,7 @@ { "cell_type": "markdown", "id": "a8270c3a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## More considerations\n", "\n", @@ -2494,9 +2199,7 @@ { "cell_type": "markdown", "id": "36355d0d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial {\\cal C}}{\\partial (a_j^L)}\n", @@ -2506,9 +2209,7 @@ { "cell_type": "markdown", "id": "40976099", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "With the definition of $\\delta_j^L$ we have a more compact definition of the derivative of the cost function in terms of the weights, namely" ] @@ -2516,9 +2217,7 @@ { "cell_type": "markdown", "id": "17a9cb41", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial{\\cal C}}{\\partial w_{jk}^L} = \\delta_j^La_k^{L-1}.\n", @@ -2528,9 +2227,7 @@ { "cell_type": "markdown", "id": "5857bfb0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Derivatives in terms of $z_j^L$\n", "\n", @@ -2540,9 +2237,7 @@ { "cell_type": "markdown", "id": "e351d319", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\delta_j^L =\\frac{\\partial {\\cal C}}{\\partial z_j^L}= \\frac{\\partial {\\cal C}}{\\partial a_j^L}\\frac{\\partial a_j^L}{\\partial z_j^L},\n", @@ -2552,9 +2247,7 @@ { "cell_type": "markdown", "id": "1d820b48", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which can also be interpreted as the partial derivative of the cost function with respect to the biases $b_j^L$, namely" ] @@ -2562,9 +2255,7 @@ { "cell_type": "markdown", "id": "4fb8bc9c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\delta_j^L = \\frac{\\partial {\\cal C}}{\\partial b_j^L}\\frac{\\partial b_j^L}{\\partial z_j^L}=\\frac{\\partial {\\cal C}}{\\partial b_j^L},\n", @@ -2574,9 +2265,7 @@ { "cell_type": "markdown", "id": "5dca3cda", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "That is, the error $\\delta_j^L$ is exactly equal to the rate of change of the cost function as a function of the bias." ] @@ -2584,9 +2273,7 @@ { "cell_type": "markdown", "id": "5871156c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Bringing it together\n", "\n", @@ -2596,9 +2283,7 @@ { "cell_type": "markdown", "id": "a7a60892", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "\n", "
\n", @@ -2614,9 +2299,7 @@ { "cell_type": "markdown", "id": "9b0bdf83", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and" ] @@ -2624,9 +2307,7 @@ { "cell_type": "markdown", "id": "a7e03a9c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "\n", "
\n", @@ -2642,9 +2323,7 @@ { "cell_type": "markdown", "id": "5624e6de", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and" ] @@ -2652,9 +2331,7 @@ { "cell_type": "markdown", "id": "a6a73788", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "\n", "
\n", @@ -2670,9 +2347,7 @@ { "cell_type": "markdown", "id": "fa0c863b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Final back propagating equation\n", "\n", @@ -2682,9 +2357,7 @@ { "cell_type": "markdown", "id": "7b9fd625", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\delta_j^l =\\frac{\\partial {\\cal C}}{\\partial z_j^l}.\n", @@ -2694,9 +2367,7 @@ { "cell_type": "markdown", "id": "2d6110ec", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We want to express this in terms of the equations for layer $l+1$." ] @@ -2704,9 +2375,7 @@ { "cell_type": "markdown", "id": "98aee849", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Using the chain rule and summing over all $k$ entries\n", "\n", @@ -2716,9 +2385,7 @@ { "cell_type": "markdown", "id": "124cb42a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\delta_j^l =\\sum_k \\frac{\\partial {\\cal C}}{\\partial z_k^{l+1}}\\frac{\\partial z_k^{l+1}}{\\partial z_j^{l}}=\\sum_k \\delta_k^{l+1}\\frac{\\partial z_k^{l+1}}{\\partial z_j^{l}},\n", @@ -2728,9 +2395,7 @@ { "cell_type": "markdown", "id": "4dedc423", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and recalling that" ] @@ -2738,9 +2403,7 @@ { "cell_type": "markdown", "id": "fbead9d4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "z_j^{l+1} = \\sum_{i=1}^{M_{l}}w_{ij}^{l+1}a_i^{l}+b_j^{l+1},\n", @@ -2750,9 +2413,7 @@ { "cell_type": "markdown", "id": "89702b4f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "with $M_l$ being the number of nodes in layer $l$, we obtain" ] @@ -2760,9 +2421,7 @@ { "cell_type": "markdown", "id": "2fb650ad", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\delta_j^l =\\sum_k \\delta_k^{l+1}w_{kj}^{l+1}\\sigma'(z_j^l),\n", @@ -2772,9 +2431,7 @@ { "cell_type": "markdown", "id": "b32a1fbe", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "This is our final equation.\n", "\n", @@ -2784,9 +2441,7 @@ { "cell_type": "markdown", "id": "5cac9518", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Setting up the back propagation algorithm\n", "\n", @@ -2807,9 +2462,7 @@ { "cell_type": "markdown", "id": "5c7f6367", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Setting up the back propagation algorithm, part 2\n", "\n", @@ -2819,9 +2472,7 @@ { "cell_type": "markdown", "id": "be6e74e5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\delta_j^L = \\sigma'(z_j^L)\\frac{\\partial {\\cal C}}{\\partial (a_j^L)}.\n", @@ -2831,9 +2482,7 @@ { "cell_type": "markdown", "id": "cea6dc43", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Then we compute the back propagate error for each $l=L-1,L-2,\\dots,1$ as" ] @@ -2841,9 +2490,7 @@ { "cell_type": "markdown", "id": "cd81fdd7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\delta_j^l = \\sum_k \\delta_k^{l+1}w_{kj}^{l+1}\\sigma'(z_j^l).\n", @@ -2853,9 +2500,7 @@ { "cell_type": "markdown", "id": "a4e2076f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Setting up the Back propagation algorithm, part 3\n", "\n", @@ -2867,9 +2512,7 @@ { "cell_type": "markdown", "id": "623d338d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "w_{jk}^l\\leftarrow = w_{jk}^l- \\eta \\delta_j^la_k^{l-1},\n", @@ -2879,9 +2522,7 @@ { "cell_type": "markdown", "id": "9a3168d0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "b_j^l \\leftarrow b_j^l-\\eta \\frac{\\partial {\\cal C}}{\\partial b_j^l}=b_j^l-\\eta \\delta_j^l,\n", @@ -2891,9 +2532,7 @@ { "cell_type": "markdown", "id": "02c62e59", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "with $\\eta$ being the learning rate." ] @@ -2901,9 +2540,7 @@ { "cell_type": "markdown", "id": "b029455b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Updating the gradients\n", "\n", @@ -2913,9 +2550,7 @@ { "cell_type": "markdown", "id": "ea5337b3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\delta_j^l = \\sum_k \\delta_k^{l+1}w_{kj}^{l+1}sigma'(z_j^l),\n", @@ -2925,9 +2560,7 @@ { "cell_type": "markdown", "id": "b475cb8d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "we update the weights and the biases using gradient descent for each $l=L-1,L-2,\\dots,1$ and update the weights and biases according to the rules" ] @@ -2935,9 +2568,7 @@ { "cell_type": "markdown", "id": "462823b9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "w_{jk}^l\\leftarrow = w_{jk}^l- \\eta \\delta_j^la_k^{l-1},\n", @@ -2947,9 +2578,7 @@ { "cell_type": "markdown", "id": "f23b093a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "b_j^l \\leftarrow b_j^l-\\eta \\frac{\\partial {\\cal C}}{\\partial b_j^l}=b_j^l-\\eta \\delta_j^l,\n", @@ -2959,9 +2588,7 @@ { "cell_type": "markdown", "id": "8e75c1c2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## NN code\n", "\n", @@ -2971,9 +2598,7 @@ { "cell_type": "markdown", "id": "3505a76f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Essential elements of generative models\n", "\n", @@ -2995,9 +2620,7 @@ { "cell_type": "markdown", "id": "093e6c22", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Probability model\n", "\n", @@ -3007,9 +2630,7 @@ { "cell_type": "markdown", "id": "ea632867", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(x_i,h_j;\\boldsymbol{\\Theta}) = \\frac{f(x_i,h_j;\\boldsymbol{\\Theta})}{Z(\\boldsymbol{\\Theta})},\n", @@ -3019,9 +2640,7 @@ { "cell_type": "markdown", "id": "706cd35f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where $f(x_i,h_j;\\boldsymbol{\\Theta})$ is a function which we assume is larger or\n", "equal than zero and obeys all properties required for a probability\n", @@ -3033,9 +2652,7 @@ { "cell_type": "markdown", "id": "3dc14b73", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "Z(\\boldsymbol{\\Theta})=\\sum_{x_i\\in \\boldsymbol{X}}\\sum_{h_j\\in \\boldsymbol{H}} f(x_i,h_j;\\boldsymbol{\\Theta}).\n", @@ -3045,9 +2662,7 @@ { "cell_type": "markdown", "id": "99e4989d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Marginal and conditional probabilities\n", "\n", @@ -3057,9 +2672,7 @@ { "cell_type": "markdown", "id": "6f6aca14", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(x_i;\\boldsymbol{\\Theta}) = \\frac{\\sum_{h_j\\in \\boldsymbol{H}}f(x_i,h_j;\\boldsymbol{\\Theta})}{Z(\\boldsymbol{\\Theta})},\n", @@ -3069,9 +2682,7 @@ { "cell_type": "markdown", "id": "d976441f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and" ] @@ -3079,9 +2690,7 @@ { "cell_type": "markdown", "id": "219b5965", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(h_i;\\boldsymbol{\\Theta}) = \\frac{\\sum_{x_i\\in \\boldsymbol{X}}f(x_i,h_j;\\boldsymbol{\\Theta})}{Z(\\boldsymbol{\\Theta})}.\n", @@ -3091,9 +2700,7 @@ { "cell_type": "markdown", "id": "d407496e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Change of notation\n", "\n", @@ -3106,9 +2713,7 @@ { "cell_type": "markdown", "id": "09154c06", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "Z(\\boldsymbol{\\Theta})=\\sum_{x_i\\in \\boldsymbol{X}}\\sum_{h_j\\in \\boldsymbol{H}} f(x_i,h_j;\\boldsymbol{\\Theta}),\n", @@ -3118,9 +2723,7 @@ { "cell_type": "markdown", "id": "60583422", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "changes to" ] @@ -3128,9 +2731,7 @@ { "cell_type": "markdown", "id": "09d0c1d3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "Z(\\boldsymbol{\\Theta})=\\sum_{\\boldsymbol{x}}\\sum_{\\boldsymbol{h}} f(\\boldsymbol{x},\\boldsymbol{h};\\boldsymbol{\\Theta}).\n", @@ -3140,9 +2741,7 @@ { "cell_type": "markdown", "id": "10e07dcb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "If we have a binary set of variable $x_i$ and $h_j$ and $M$ values of $x_i$ and $N$ values of $h_j$ we have in total $2^M$ and $2^N$ possible $\\boldsymbol{x}$ and $\\boldsymbol{h}$ configurations, respectively.\n", "\n", @@ -3153,9 +2752,7 @@ { "cell_type": "markdown", "id": "ec807692", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Optimization problem\n", "\n", @@ -3165,9 +2762,7 @@ { "cell_type": "markdown", "id": "b5d52ab5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(\\boldsymbol{X};\\boldsymbol{\\Theta})=\\prod_{x_i\\in \\boldsymbol{X}}p(x_i;\\boldsymbol{\\Theta})=\\prod_{x_i\\in \\boldsymbol{X}}\\left(\\frac{\\sum_{h_j\\in \\boldsymbol{H}}f(x_i,h_j;\\boldsymbol{\\Theta})}{Z(\\boldsymbol{\\Theta})}\\right),\n", @@ -3177,9 +2772,7 @@ { "cell_type": "markdown", "id": "f1c09bbf", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which we rewrite as" ] @@ -3187,9 +2780,7 @@ { "cell_type": "markdown", "id": "e8145abd", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(\\boldsymbol{X};\\boldsymbol{\\Theta})=\\frac{1}{Z(\\boldsymbol{\\Theta})}\\prod_{x_i\\in \\boldsymbol{X}}\\left(\\sum_{h_j\\in \\boldsymbol{H}}f(x_i,h_j;\\boldsymbol{\\Theta})\\right).\n", @@ -3199,9 +2790,7 @@ { "cell_type": "markdown", "id": "d8a60c69", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Further simplifications\n", "\n", @@ -3211,9 +2800,7 @@ { "cell_type": "markdown", "id": "f87035b3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(\\boldsymbol{X};\\boldsymbol{\\Theta})=\\frac{1}{Z(\\boldsymbol{\\Theta})}\\prod_{x_i\\in \\boldsymbol{X}}f(x_i;\\boldsymbol{\\Theta}),\n", @@ -3223,9 +2810,7 @@ { "cell_type": "markdown", "id": "63061a84", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where we used $p(x_i;\\boldsymbol{\\Theta}) = \\sum_{h_j\\in \\boldsymbol{H}}f(x_i,h_j;\\boldsymbol{\\Theta})$.\n", "The optimization problem is then" @@ -3234,9 +2819,7 @@ { "cell_type": "markdown", "id": "e90629e1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "{\\displaystyle \\mathrm{arg} \\hspace{0.1cm}\\max_{\\boldsymbol{\\boldsymbol{\\Theta}}\\in {\\mathbb{R}}^{p}}} \\hspace{0.1cm}p(\\boldsymbol{X};\\boldsymbol{\\Theta}).\n", @@ -3246,9 +2829,7 @@ { "cell_type": "markdown", "id": "4dfb4385", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Optimizing the logarithm instead\n", "\n", @@ -3260,9 +2841,7 @@ { "cell_type": "markdown", "id": "eae9ef99", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "{\\displaystyle \\mathrm{arg} \\hspace{0.1cm}\\max_{\\boldsymbol{\\boldsymbol{\\Theta}}\\in {\\mathbb{R}}^{p}}} \\hspace{0.1cm}\\log{p(\\boldsymbol{X};\\boldsymbol{\\Theta})},\n", @@ -3272,9 +2851,7 @@ { "cell_type": "markdown", "id": "79e853e3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which leads to" ] @@ -3282,9 +2859,7 @@ { "cell_type": "markdown", "id": "b5822deb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\nabla_{\\boldsymbol{\\Theta}}\\log{p(\\boldsymbol{X};\\boldsymbol{\\Theta})}=0.\n", @@ -3294,9 +2869,7 @@ { "cell_type": "markdown", "id": "3da9d274", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Expression for the gradients\n", "\n", @@ -3306,9 +2879,7 @@ { "cell_type": "markdown", "id": "0c362f5f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\nabla_{\\boldsymbol{\\Theta}}\\log{p(\\boldsymbol{X};\\boldsymbol{\\Theta})}=\\nabla_{\\boldsymbol{\\Theta}}\\left(\\sum_{x_i\\in \\boldsymbol{X}}\\log{f(x_i;\\boldsymbol{\\Theta})}\\right)-\\nabla_{\\boldsymbol{\\Theta}}\\log{Z(\\boldsymbol{\\Theta})}=0.\n", @@ -3318,9 +2889,7 @@ { "cell_type": "markdown", "id": "e4fbadf8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "The first term is called the positive phase and we assume that we have a model for the function $f$ from which we can sample values. Below we will develop an explicit model for this.\n", "The second term is called the negative phase and is the one which leads to more difficulties." @@ -3329,9 +2898,7 @@ { "cell_type": "markdown", "id": "13aad1f3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The derivative of the partition function\n", "\n", @@ -3341,9 +2908,7 @@ { "cell_type": "markdown", "id": "6cf15217", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "Z(\\boldsymbol{\\Theta})=\\sum_{x_i\\in \\boldsymbol{X}}\\sum_{h_j\\in \\boldsymbol{H}} f(x_i,h_j;\\boldsymbol{\\Theta}),\n", @@ -3353,9 +2918,7 @@ { "cell_type": "markdown", "id": "b7f2b57b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "is in general the most problematic term. In principle both $x$ and $h$ can span large degrees of freedom, if not even infinitely many ones, and computing the partition function itself is often not desirable or even feasible. The above derivative of the partition function can however be written in terms of an expectation value which is in turn evaluated using Monte Carlo sampling and the theory of Markov chains, popularly shortened to MCMC (or just MC$^2$)." ] @@ -3363,9 +2926,7 @@ { "cell_type": "markdown", "id": "84697aa7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Explicit expression for the derivative\n", "We can rewrite" @@ -3374,9 +2935,7 @@ { "cell_type": "markdown", "id": "cff18fb0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\nabla_{\\boldsymbol{\\Theta}}\\log{Z(\\boldsymbol{\\Theta})}=\\frac{\\nabla_{\\boldsymbol{\\Theta}}Z(\\boldsymbol{\\Theta})}{Z(\\boldsymbol{\\Theta})},\n", @@ -3386,9 +2945,7 @@ { "cell_type": "markdown", "id": "b200258b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which reads in more detail" ] @@ -3396,9 +2953,7 @@ { "cell_type": "markdown", "id": "78c3e330", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\nabla_{\\boldsymbol{\\Theta}}\\log{Z(\\boldsymbol{\\Theta})}=\\frac{\\nabla_{\\boldsymbol{\\Theta}} \\sum_{x_i\\in \\boldsymbol{X}}f(x_i;\\boldsymbol{\\Theta}) }{Z(\\boldsymbol{\\Theta})}.\n", @@ -3408,9 +2963,7 @@ { "cell_type": "markdown", "id": "5c039452", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We can rewrite the function $f$ (we have assumed that is larger or\n", "equal than zero) as $f=\\exp{\\log{f}}$. We can then rewrite the last\n", @@ -3420,9 +2973,7 @@ { "cell_type": "markdown", "id": "b1450211", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\nabla_{\\boldsymbol{\\Theta}}\\log{Z(\\boldsymbol{\\Theta})}=\\frac{ \\sum_{x_i\\in \\boldsymbol{X}} \\nabla_{\\boldsymbol{\\Theta}}\\exp{\\log{f(x_i;\\boldsymbol{\\Theta})}} }{Z(\\boldsymbol{\\Theta})}.\n", @@ -3432,9 +2983,7 @@ { "cell_type": "markdown", "id": "dbf02775", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Final expression\n", "\n", @@ -3444,9 +2993,7 @@ { "cell_type": "markdown", "id": "aee826e9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\nabla_{\\boldsymbol{\\Theta}}\\log{Z(\\boldsymbol{\\Theta})}=\\frac{ \\sum_{x_i\\in \\boldsymbol{X}}f(x_i;\\boldsymbol{\\Theta}) \\nabla_{\\boldsymbol{\\Theta}}\\log{f(x_i;\\boldsymbol{\\Theta})} }{Z(\\boldsymbol{\\Theta})},\n", @@ -3456,9 +3003,7 @@ { "cell_type": "markdown", "id": "7c21e99c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which is the expectation value of $\\log{f}$" ] @@ -3466,9 +3011,7 @@ { "cell_type": "markdown", "id": "5614ea79", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\nabla_{\\boldsymbol{\\Theta}}\\log{Z(\\boldsymbol{\\Theta})}=\\sum_{x_i\\sim p}p(x_i;\\boldsymbol{\\Theta}) \\nabla_{\\boldsymbol{\\Theta}}\\log{f(x_i;\\boldsymbol{\\Theta})},\n", @@ -3478,9 +3021,7 @@ { "cell_type": "markdown", "id": "9c011fbe", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "that is" ] @@ -3488,9 +3029,7 @@ { "cell_type": "markdown", "id": "47c4916b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\nabla_{\\boldsymbol{\\Theta}}\\log{Z(\\boldsymbol{\\Theta})}=\\mathbb{E}(\\log{f(x_i;\\boldsymbol{\\Theta})}).\n", @@ -3500,9 +3039,7 @@ { "cell_type": "markdown", "id": "c6519f09", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "This quantity is evaluated using Monte Carlo sampling, with Gibbs\n", "sampling as the standard sampling rule." @@ -3511,9 +3048,7 @@ { "cell_type": "markdown", "id": "e82bc48f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Final expression for the gradients\n", "\n", @@ -3523,9 +3058,7 @@ { "cell_type": "markdown", "id": "0a42012d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\nabla_{\\boldsymbol{\\Theta}}\\log{p(\\boldsymbol{X};\\boldsymbol{\\Theta})}=\\nabla_{\\boldsymbol{\\Theta}}\\left(\\sum_{x_i\\in \\boldsymbol{X}}\\log{f(x_i;\\boldsymbol{\\Theta})}\\right)-\\mathbb{E}_{x\\sim p}(\\log{f(x_i;\\boldsymbol{\\Theta})})=0.\n", @@ -3535,9 +3068,7 @@ { "cell_type": "markdown", "id": "b1129c95", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Introducing the energy model\n", "\n", @@ -3547,9 +3078,7 @@ { "cell_type": "markdown", "id": "4fb7028c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(\\boldsymbol{x},\\boldsymbol{h};\\boldsymbol{\\Theta}) = \\frac{f(\\boldsymbol{x},\\boldsymbol{h};\\boldsymbol{\\Theta})}{Z(\\boldsymbol{\\Theta})},\n", @@ -3559,9 +3088,7 @@ { "cell_type": "markdown", "id": "4a25c045", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where $f(\\boldsymbol{x},\\boldsymbol{h};\\boldsymbol{\\Theta})$ is given by a so-called energy model. If we assume that the random variables $x_i$ and $h_j$ take binary values only, for example $x_i,h_j=\\{0,1\\}$, we have a so-called binary-binary model where" ] @@ -3569,9 +3096,7 @@ { "cell_type": "markdown", "id": "64f27aa4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "f(\\boldsymbol{x},\\boldsymbol{h};\\boldsymbol{\\Theta})=-E(\\boldsymbol{x}, \\boldsymbol{h};\\boldsymbol{\\Theta}) = \\sum_{x_i\\in \\boldsymbol{X}} x_i a_i+\\sum_{h_j\\in \\boldsymbol{H}} b_j h_j + \\sum_{x_i\\in \\boldsymbol{X},h_j\\in\\boldsymbol{H}} x_i w_{ij} h_j,\n", @@ -3581,9 +3106,7 @@ { "cell_type": "markdown", "id": "04553f9f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where the set of parameters are given by the biases and weights $\\boldsymbol{\\Theta}=\\{\\boldsymbol{a},\\boldsymbol{b},\\boldsymbol{W}\\}$.\n", "**Note the vector notation** instead of $x_i$ and $h_j$ for $f$. The vectors $\\boldsymbol{x}$ and $\\boldsymbol{h}$ represent a specific instance of stochastic variables $x_i$ and $h_j$. These arrangements of $\\boldsymbol{x}$ and $\\boldsymbol{h}$ lead to a specific energy configuration." @@ -3592,9 +3115,7 @@ { "cell_type": "markdown", "id": "15f0ce3a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## More compact notation\n", "\n", @@ -3604,9 +3125,7 @@ { "cell_type": "markdown", "id": "b2a97150", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(\\boldsymbol{x},\\boldsymbol{h};\\boldsymbol{\\Theta}) = \\frac{\\exp{(\\boldsymbol{a}^T\\boldsymbol{x}+\\boldsymbol{b}^T\\boldsymbol{h}+\\boldsymbol{x}^T\\boldsymbol{W}\\boldsymbol{h})}}{Z(\\boldsymbol{\\Theta})},\n", @@ -3616,9 +3135,7 @@ { "cell_type": "markdown", "id": "9c794d16", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where the biases $\\boldsymbol{a}$ and $\\boldsymbol{h}$ and the weights defined by the matrix $\\boldsymbol{W}$ are the parameters we need to optimize." ] @@ -3626,9 +3143,7 @@ { "cell_type": "markdown", "id": "a6986c04", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Examples of gradient expressions\n", "\n", @@ -3641,9 +3156,7 @@ { "cell_type": "markdown", "id": "de147884", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial E(\\boldsymbol{x}, \\boldsymbol{h};\\boldsymbol{\\Theta})}{\\partial w_{ij}}=-x_ih_j,\n", @@ -3653,9 +3166,7 @@ { "cell_type": "markdown", "id": "bd7c2f00", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and" ] @@ -3663,9 +3174,7 @@ { "cell_type": "markdown", "id": "5b2b56a2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial E(\\boldsymbol{x}, \\boldsymbol{h};\\boldsymbol{\\Theta})}{\\partial a_i}=-x_i,\n", @@ -3675,9 +3184,7 @@ { "cell_type": "markdown", "id": "937d72bc", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and" ] @@ -3685,9 +3192,7 @@ { "cell_type": "markdown", "id": "cbec00b4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial E(\\boldsymbol{x}, \\boldsymbol{h};\\boldsymbol{\\Theta})}{\\partial b_j}=-h_j.\n", @@ -3697,9 +3202,7 @@ { "cell_type": "markdown", "id": "757dfd72", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Network Elements, the energy function\n", "\n", @@ -3714,9 +3217,7 @@ { "cell_type": "markdown", "id": "3ee3e71c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Defining different types of RBMs\n", "\n", @@ -3730,9 +3231,7 @@ { "cell_type": "markdown", "id": "91c70e4b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{align*}\n", @@ -3744,9 +3243,7 @@ { "cell_type": "markdown", "id": "a7073293", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where the binary values taken on by the nodes are most commonly 0 and 1." ] @@ -3754,9 +3251,7 @@ { "cell_type": "markdown", "id": "772217ef", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Gaussian-binary RBM\n", "\n", @@ -3766,9 +3261,7 @@ { "cell_type": "markdown", "id": "a1488d82", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{align*}\n", @@ -3780,9 +3273,7 @@ { "cell_type": "markdown", "id": "78aa59cf", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "This type of RBMs are useful when we model continuous data (i.e., we wish $\\boldsymbol{x}$ to be continuous). The paramater $\\sigma_i^2$ is meant to represent a variance and is foten just set to one." ] @@ -3790,9 +3281,7 @@ { "cell_type": "markdown", "id": "50b290c2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Code for RBMs using PyTorch" ] @@ -3801,11 +3290,129 @@ "cell_type": "code", "execution_count": 3, "id": "c085169e", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: 'dlopen(/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/torchvision/image.so, 0x0006): Symbol not found: __ZN3c1017RegisterOperatorsD1Ev\n", + " Referenced from: <2D1B8D5C-7891-3680-9CF9-F771AE880676> /Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/torchvision/image.so\n", + " Expected in: /Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/torch/lib/libtorch_cpu.dylib'If you don't plan on using image functionality from `torchvision.io`, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have `libjpeg` or `libpng` installed before building `torchvision` from source?\n", + " warn(\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz\n", + "Failed to download (trying next):\n", + "HTTP Error 403: Forbidden\n", + "\n", + "Downloading https://ossci-datasets.s3.amazonaws.com/mnist/train-images-idx3-ubyte.gz\n", + "Downloading https://ossci-datasets.s3.amazonaws.com/mnist/train-images-idx3-ubyte.gz to ./data/MNIST/raw/train-images-idx3-ubyte.gz\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "100%|██████████████████████████████████████████████████████████████████████████████████████████| 9912422/9912422 [00:02<00:00, 4337161.71it/s]\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Extracting ./data/MNIST/raw/train-images-idx3-ubyte.gz to ./data/MNIST/raw\n", + "\n", + "Downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz\n", + "Failed to download (trying next):\n", + "HTTP Error 403: Forbidden\n", + "\n", + "Downloading https://ossci-datasets.s3.amazonaws.com/mnist/train-labels-idx1-ubyte.gz\n", + "Downloading https://ossci-datasets.s3.amazonaws.com/mnist/train-labels-idx1-ubyte.gz to ./data/MNIST/raw/train-labels-idx1-ubyte.gz\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "100%|███████████████████████████████████████████████████████████████████████████████████████████████| 28881/28881 [00:00<00:00, 131135.28it/s]\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Extracting ./data/MNIST/raw/train-labels-idx1-ubyte.gz to ./data/MNIST/raw\n", + "\n", + "Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz\n", + "Failed to download (trying next):\n", + "HTTP Error 403: Forbidden\n", + "\n", + "Downloading https://ossci-datasets.s3.amazonaws.com/mnist/t10k-images-idx3-ubyte.gz\n", + "Downloading https://ossci-datasets.s3.amazonaws.com/mnist/t10k-images-idx3-ubyte.gz to ./data/MNIST/raw/t10k-images-idx3-ubyte.gz\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "100%|██████████████████████████████████████████████████████████████████████████████████████████| 1648877/1648877 [00:01<00:00, 1168573.50it/s]\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Extracting ./data/MNIST/raw/t10k-images-idx3-ubyte.gz to ./data/MNIST/raw\n", + "\n", + "Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz\n", + "Failed to download (trying next):\n", + "HTTP Error 403: Forbidden\n", + "\n", + "Downloading https://ossci-datasets.s3.amazonaws.com/mnist/t10k-labels-idx1-ubyte.gz\n", + "Downloading https://ossci-datasets.s3.amazonaws.com/mnist/t10k-labels-idx1-ubyte.gz to ./data/MNIST/raw/t10k-labels-idx1-ubyte.gz\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "100%|████████████████████████████████████████████████████████████████████████████████████████████████| 4542/4542 [00:00<00:00, 1005995.08it/s]\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Extracting ./data/MNIST/raw/t10k-labels-idx1-ubyte.gz to ./data/MNIST/raw\n", + "\n", + "Training loss for 0 epoch: -8.471399307250977\n", + "Training loss for 1 epoch: -6.6942901611328125\n", + "Training loss for 2 epoch: -4.579244136810303\n", + "Training loss for 3 epoch: -3.2101542949676514\n", + "Training loss for 4 epoch: -2.2072556018829346\n", + "Training loss for 5 epoch: -1.5162527561187744\n", + "Training loss for 6 epoch: -1.025678277015686\n", + "Training loss for 7 epoch: -0.7097488045692444\n", + "Training loss for 8 epoch: -0.43329933285713196\n", + "Training loss for 9 epoch: -0.2937687039375305\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAigAAAEqCAYAAAA/LasTAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAADMwklEQVR4nOydd3hU1daHf2f6THrvvSe00JEmRawUlasoqNjFgohI9V7EK1VBFERsqFgQK2AFLPQeIIX0Num9TMr0Wd8fuXM+JjNpkGQm4bzPsx7ImTMza5999pm11157LYaICBwcHBwcHBwcNgTP2gpwcHBwcHBwcLSGM1A4ODg4ODg4bA7OQOHg4ODg4OCwOTgDhYODg4ODg8Pm4AwUDg4ODg4ODpuDM1A4ODg4ODg4bA7OQOHg4ODg4OCwOTgDhYODg4ODg8Pm4AwUDg4ODg4ODpuDM1A4ODg4ODg4bA6rGig7duxASEgIJBIJhg0bhuPHj1tTHQ4ODg4ODg4bwWoGyt69e7Fo0SKsWrUKly5dwvjx43H77bejoKDAWipxcHBwcHBw2AiMtYoFjho1CkOHDsX777/PHouJicGsWbOwfv36dt9rMBhQUlICBwcHMAzT06pycHBwcHBwdANEhIaGBvj6+oLHa99HIuglnUzQaDRISEjA8uXLTY5PmzYNp06dMjtfrVZDrVazfxcXFyM2NrbH9eTg4ODg4ODofgoLC+Hv79/uOVZZ4qmqqoJer4eXl5fJcS8vL5SVlZmdv379ejg5ObHCGSccHBwcHBx9FwcHhw7PsWqQbOvlGSKyuGSzYsUK1NfXs1JYWNhbKnJwcHBwcHB0M50Jz7DKEo+7uzv4fL6Zt6SiosLMqwIAYrEYYrG4t9Tj4ODg4ODgsDJW8aCIRCIMGzYMhw8fNjl++PBh3HTTTdZQiYODg4ODg8OGsIoHBQAWL16Mhx56CMOHD8eYMWPw4YcfoqCgAM8884y1VOLg4ODg4OCwEaxmoNx///2orq7G66+/jtLSUgwYMAC//fYbgoKCrKUSBwcHBwcHh41gtTwo14NCoYCTk5O11eDg4ODg4OC4Burr6+Ho6NjuOVbzoHBw2Cp8Ph9CoRB6vR5ardba6nBwcFgRhmEgEonYpGJEBLVajT44t+9zcAYKB0crxo8fj6effhpnzpzBjh07OCOFg+MGJiAgAEuXLoWnpycAoLKyEps2bYJcLreyZv0fzkDh4GhFYGAgpk+fDp1OBz6fD51OB4Gg46FCRNDpdL2gYc/C5/M7TEF9NXq9HgaDoQc1sj4Mw4DP57eZu4GIoNfruVl1H4ZhGIvj3M3NDXfccQdCQkIAAHK5HB9++GG/N1B4PB74fD77tzXGOWegcHC04tSpU3j22Wchl8uh1WoxePBgPPXUU5BIJO2+r7y8HO+++y5KS0t7SdPuRyqVYsGCBRgwYECn37N//37s37+/B7WyPqGhoXj++efbjH2rra3Ftm3bkJ+f37uKcXQbMTExePbZZyGTyUyOu7i4wN3d3UpaWY/Zs2fjtttuY//+/fff8d133/WqDpyBYkMwDGMyQyMibkZmBXJycpCbm8v+HRQUhHnz5nWYmjkrKwtffvklysvL+5RH4WpviVQqxa233opp06Z16r1EhIKCAvz888/99n7l8Xjw8vLCnDlz4O3tbfGcoqIifPPNNygoKOhTfc/x//j6+uL++++Hi4sLeDzeDV+Idvjw4Xj00UfZv6uqqvDjjz/CYDD02jjnDBQbQSAQ4KGHHsLQoUMBtFRs/u6773DixAkra3bjMXLkSDzwwAOsezMkJKRTmYw9PDywYsUK5OTk4MMPP0RxcXFPq3rdODk54cknn2S394vFYkRHR3fpM26//Xa4ubnhzz//7HeelIiICDz++OMICwtr10B1dnbGkiVLkJOTg48//tjEwOXoG6Snp2PFihWIjY3FE0880alaMTcSU6ZMwTvvvIMjR47ghx9+6B0jhfog9fX1BKBfiVQqpb1795LBYCCDwUAajYaefvppq+t1I8r8+fNJrVazfdFaOiI/P5+GDBli9XZ0Rnx8fOjMmTMdtq0z12Ljxo3EMIzV29SdcvPNN1NVVVWb16E1ZWVlNG7cOKvrzcm1y7hx46isrKzNez4vL48GDx5sdT17UhiGoU2bNpnc58b/v/POO8Tj8a77O+rr6zt8lnIelG7G19cXDz/8MJydnU2O19fXY/fu3airq8PDDz9slpBOIBBg0KBBMBgM+OGHH3D+/HmcPXu2FzXvPHw+H/feey/r7ekMZ86cwf79+216CWD06NGYOXMmNBoN/v3vf2Pw4MH417/+haysLOzZswdBQUGYO3cupFJpu5/j7OyMF198EdnZ2di9e7dNF7dsaGjAe++9h4MHD2LevHkIDQ21eJ7BYMCPP/6ICxcusMfuuOMOTJw4kf174sSJ2LBhA06ePMku+fQ3SkpKsHv3bohEIjz88MPw8PAwed3e3h7PPPMMpkyZgq+++grZ2dlW0pTjWsnPz8cbb7xhFotipL6+vk/HmbUHwzCYMWMGxowZg8bGRixfvhxTp07F1KlTcfToUfz+++9ISEjovWXMDk0YG8SWPSjx8fEkl8vNdC4sLKRhw4aRp6cnHT9+nIjIbCZmMBhIrVbTww8/bPV2tCcikYg+//zzLvXZjh07iM/nW1339mTBggWk0+lo165dJBQK6f7776fm5mbav38/2dvb09SpU6mmpqbTnpXi4mIaNWqU1dvVGXF3d6cjR4602TaNRkOPPfaYyXtaz7CMvPvuu90yw7IFae1BSUhIoICAAIqJiaGMjIw2+766upqmTJlidf054aQrwuPxaNu2baTX62nJkiUEgFavXk0Gg4E2bNjQrd/FeVB6ET8/P9x3332IiIiwmB3PwcEBTz75JOrr6xEQEAClUonvv/8excXFuPvuuxEaGor9+/cjOTkZSUlJVmhB24hEItxzzz2IiIgA0OJBGTRokJW16n4uXLiA//73v0hMTIRer8eVK1ewbt065OTkQKPRIDc3F5s2bTLzoHh6erLBdVfj4OCAxx57DOPHj8d3331n09sSm5ub8fnnn+Po0aMWX9fr9bh06ZLJsT///BNKpRITJ07EhAkT2KDCESNG4N///jcSEhLw22+/9cmg0fDwcNxzzz2Ijo626DGrqqrCe++9h7CwMMyZM4fNkdGXEQgEmDVrFmJiYjo898SJE/jnn396QSvrYxzfrb3ijY2N+Pbbb/tErFlnISL8/vvvqKmpgZ2dHf79738DAP773/9aJx6yS9NgG8EWPSijR4+mkpKSNmdUrWekxhmWg4MD7d+/n5qbm+m+++6zejssiYODA/38889djsW4mr7gQblWGThwIOXm5rbZ55WVlTRx4kSr69lTYpxhtW73Rx99RAKBwOr6XYvcdtttVFdXZ3avGz0oxvNCQ0MpKSnJrO/7ogdFKpXSt99+26YX7Wp54403rK5vb8nAgQMpJyfH7Br0JQ9pV+XqGJTVq1f3yHdwHpRepLS0FDt37ux05LdSqYRcLodGo8G+ffuQmZmJiIgILFq0CIcOHUJqamoPa9wxEokE06dPR0REBEJDQ9tNUnXkyBEkJCSwx0aPHo2xY8ciISEBR48exalTp/rkTLozVFZW4qOPPkJYWBhmzZoFNzc3ALhhtyn2xXbHxMTg1ltvZXduRUdHQywWd9iWuro6fP7554iIiMDMmTPh4OCAAwcOIDs72+Y8Zg4ODpg5cybs7e1x4MABlJSUAGjxkN5xxx2Ijo5GZGRkn+y/nqSyshIff/wxO76FQiH27duHnJycfheLwjAMJk2ahPj4eIwYMcLa6oDzoHSz1dkVufq9UqmUvvnmG1KpVDYTg+Lq6kqHDx8mvV7frsdEr9fTokWLTNq2atUqMhgMtGXLln4Tj9BR30dGRlJqaqrZ9bnRPChG+pIHZe7cuaRUKkmv17P3u6U2tfagGPs+ICCAzp8/z+7iscXdTP7+/nT+/HkqKSmhMWPGsMcdHBzowIEDHY7zq7mRPCjGPjaOb+MuPVvs4+sVHo9H7777rskY4Dwo/QS6jl0LOp0Ohw8fRmVlJTIzM7tRq65jTNYVGhoKf39/NpGXTqfD33//jZycHJPziQhJSUkm7U9ISMDOnTtx8uTJfus5uRqykKSsqakJBw8eRG5uLjtb7Y9cuHABO3fuZP8eOHAgxo4di5iYGDz11FPIyMjAP//8Y/P3AcMwZin+i4uLcejQIahUKgBAYWEhGhsbTc4hIrZtUqkUM2fOREREBA4dOmRz8QkMw8DOzg533303oqKicOjQITQ0NJi0Xa/X4+jRo8jIyDB778iRIzF06FAMHToUzzzzDBITE3H69GlrNKVXaT2+LY33vgzDMJg4cSJiYmIwcOBAMAyDM2fO4PLlyyY793qdTpnLNoatelCuV3g8HvH5fKtb5p6ennTs2DHSarUmMypjnAyfzzeT1jozDGMTbelNiYiIMPGglJSU0E033dRvY29a97VRFi5cSDqdjvR6PWm1Wtq9ezeJRCKr69mezJ07l1Qqldmz5p9//iFPT0+2bW15A/38/Oj8+fNkMBhIp9NRRUUFTZ482ertulr8/f3pwoULrI7l5eU0adIkNsbMiFqtpkceecTiOF+/fj0ZDAa2b998880bZowbx3d+fn6/y4MiEAjoww8/JK1Wy3pPli5d2qPPcM6Dch1ER0dj+PDh7Hpsbm4uTp8+3aOzQGvNMO3t7TFp0iQ2St3JyQne3t5s4SytVovjx48jNzcXeXl50Ov1HX4m/a94mpHIyEiMHDkSubm5OHPmjM3PprsDqVSKadOmISwsDACg0Whw/PjxfudNad3Xxr7l8Xis2Bqtx/dNN90EHo+HgoICnDhxAl5eXhg/fjxbALIz9zzw/0UFZTIZpk2bBk9PT5w4cQJFRUU92Zx2MY5vb29vnDlzBleuXAHQ4uErKyuDVqvF33//jZqaGgAtntLs7GyLbW7dtwMGDMBDDz2EjIwMnDt3rl95FVqjUCiwf/9+iEQi1NbWWludbofH47HPfPqfV7Cz932Pce1+DOvRGx6U5557jpqbm0mlUpFKpaJdu3bZ/CzwWiU0NJQuXrzItlWtVpNerze53rNmzSKRSHTN8SRPPvkkNTU10UcffURCodDqbe4Jae1BMea1MV7X8vJyuuWWW6yuZ0/L888/Tzqdjr0OX375pc2NndbjW6PRkMFgoJ9++omcnZ1p+vTpVF9fT3///Te5urp2+HlGD8rVfa/RaKiyspLuvPNOq7bVOL5zcnJo5MiRJBKJSCQSkVAoZGfHAoGAPd7eOF+7dq2JV1Wn05FKpaJt27b1e08hABIKhSbXrb+IQCCgjz/+2OT+NeZB6SnhPCjXAZ/Ph1gsZmd/oaGhmDlzJgoKCnDhwgXrW5bdCMMwEAqFbL0ZlUqFY8eOsTMqpVKJwsJCaDSaa/4OuVyO3377DYmJif3We9LY2IjDhw9DLpdj9OjREIvFOHv2LHsdm5ubUVlZaWUte46IiAgMGDAAgwcPtvmdIK3HtxGDwQCNRoOioiIcOHAAubm57d73MpkMY8aMQUhIiEmeDK1Wi7Nnz6KgoADl5eU91YxOYRzfQqEQOp3OYnt0Ol27nzFw4ECEh4cjKirK5Difz2flRkCr1VpbhR6FiJCYmIicnBxkZWVZWx1wHpQ2ZOHChSZeBK1WS42NjfT111+TVCq1usXbnRIWFkbJyclsW8vKymjSpEkkk8lIJpORVCq97tmRQCAgmUxmczPp7hSGYUgikVB0dDQlJiZSSUkJTZw40eQ69ucdTS+99BI1NDSwdYyM2KIHpfX4NvLDDz+QTCYjHo9HUqmUxGJxu58THBxM586do6amJhOvUU1NDd1xxx3dMnauV4zju6CggIYOHdrl9zMMQxs3bqTGxkbSaDQWn8n9Oc/RjSBGD4pOp6OXXnqJZDJZj+/A4zwo3YhAIIBAIDDLjeDq6oohQ4agqakJly5dglgsxtChQ6HT6XDp0iU0NzdbUev2sbe3R3x8vFmlVpFIhCFDhoBhGFy+fJn1ALRHdHS0WX0hoKWuRUZGBnQ6XYeztL4OEUGlUqG5uRkGg4G9jnw+H5cuXeqX69ZXIxQKYWdnZ9Pek5CQEERGRppVbC4pKUFKSgouXboEvV4Pg8EApVLZ4efxeDxIJBKzui1EBLVa3anP6CnaG9/x8fGws7Pr9PgWiUSws7MzO56fn4/09HSkpqbabPxJcHAwoqKiUFBQgLS0NLPXPTw8MHjwYNTV1eHy5cvsc8rR0RHx8fGQSCQAWjygFy9eRFNTU6/q3xVkMhmGDh0KHo+HS5cuoaGhocufodFobOd3q2N/he1hDQ+KEeMMy3jeuHHjKDc3lw4dOkSenp40aNAgSk5OpjNnzlBYWJjVLeP2JDo6mhISEkihUJjM/vR6PTU0NFBubi5NmDChw89hGIbWrVtHtbW1ZrJ27dp+t17bkQQGBtKlS5fY65iTk0Njx461ul49LUuXLrWYR8OWPCiLFi2i2tpaam5uNtH1m2++IQ8PD5Ox3Rmx5Uyyrce30YPi4eFBBw8e7NL4fvvtty0+i7dv307Ozs4kkUis3rdtyfPPP081NTX01ltvWfRg3nbbbVRUVEQ//fQTOTk5scfj4+MpNTWVfZZduHCBIiMjrd6e9iQ8PJzOnTtHiYmJFBcX1+n3Xe1Bee6553pFV86Dcg0EBAQgODiY3XlhpLy8HJmZmaioqMCYMWPYddzQ0FBkZGSgqqoKw4cPh4+PDzw8PGBvb49Ro0bB1dUVV65csR2L9Cr4fD4cHBzMst/yeDzY29tDrVazUd1XwzAMIiMj2fojPB4P4eHhZrUqgJaaJuPHj0dJSUm/rexqZ2eH2NhYdqbl7e0NOzs7aLVaZGVloaioyCx3Rn8iMDAQQUFBCAkJMTleWlqK7OxspKenW312HRQUhMDAQNjZ2SExMRF+fn4ICwtDaWkpsrKykJycjNra2k57+WQyGWJjYxEWFmbRs2ALqFQqpKenQ6fTIS4ujvXuBgQEwNfXF46OjhbHd2coKChAfn4+0tLSUF9fb/X+tYTxvoyJiYGzszNCQ0MxYcIEs/jBIUOGwMXFxcz7x+fz4ejoyD7XPD09MXLkSDg7O+PKlSs25Umxs7NDXFwcIiIi4OXlBYPB0Km4IIZhEBUVBW9vbygUCtvbZdihCWOD9KQHZeHChVReXk4NDQ0mM6xvv/2WfH196YEHHiC5XE4VFRVUUVFBv/zyC4WHh9PUqVMpNTWVampqSKfTkVarperqajp//jxFR0db3bK2JHFxcZSZmdnmda6qqrKYy0EoFNL27dvZa1BRUUFNTU0WP6OpqYkqKipo06ZN/Tb+IiYmhi5cuMBei6qqKtJqtVRaWkq33norubm59dudSwDolVdeoYqKCrMx88UXX5C3tzc5ODhYXcelS5dSRUUFrVu3jjw8PGjlypWk1+tp9+7d16RjZGQknTlzhqqrq0mr1Zrd97bgQeHz+eTi4kLDhw+ntLQ00ul0VFNTQ1VVVaTRaNoc363Fkgdly5Yt5O7uTnZ2dlbv27Zk8eLFJvdlc3OzyTPLKMaaS4cOHSJnZ2f2/cOHD6eioiK2zcZnekJCQpe8E70hsbGxdOHCBfZ+zMvLo0GDBnX4PpFIRDt37qSSkhJ66qmnyN3dvde8YZwHpQv4+vrC19cX9vb2yM/PN1tHT0tLQ2VlJUpLSyGXy+Hl5YWwsDB4eXkhKCgIrq6uKCkpgcFggEwmg1gshqurK2pra695ltLTKJVKJCcnQ6lUIiIiwmLVVqDFQ2Jcx87Ly0NDQwOqqqqQn5+PoKCgdiu5KhQKFBYWorq6uqeaYTVkMhkiIiIQExMDb29veHh4AADUajUyMjJQUFCAoqKiftl2APD394e3tzfCw8Ph7u7Ojpny8nIUFBQgPT0dlZWVVtnxFhgYaHJfRkREwN3dHb6+vggODoarqyuAljHQFR1lMhnCw8MRHR0NHx8f9nOMaLVaZGdno6SkBAqFovsadA0IhUL4+/vD398fQqEQfD7fpOK2QCBAVFQUamtrkZubi/r6erPPCAoKgpeXF7y8vEBEKC4uRmlpKTIzM1FVVdWbzTGBYRgEBwfDzc0N+fn5qKqqQkBAALy8vNhzjH1uvC+lUmmbzzjg/2NOjHEbsbGxEAqF7OsCgQCurq5oaGiAQCCAUChEeHg4xGIxcnJyrineo7sQCARwc3Mzux/bgsfjITQ0FJ6enggODoaHhwf8/f0RHByMgoICNnOy1enQhLFBesKDsnjxYiosLKS1a9eSv78/+fn5mYiLiwsBIIlEQr6+vjRv3jxSKBSkVCqpuLiYfvvtNxo0aBDdeuutJlZ3dnY2DRgwwOoWtiURCATk5eVFY8eOpfT0dLPrbJxh2dvb06effkqZmZlsTgcXFxcKCgqiL774ot2++uCDDyggIMBkZtJfZPDgwWz9latn0YWFhTRt2jTy8fGxmdiLnpBVq1ZRYWEh1dfXm3hOdu3aRQEBAeyY6W3h8Xj0xhtvUGFhISvG2Vp9fT0VFhZSbW0tGQwG+uCDD7q0+yQuLo7OnDlDpaWlFne0VFZW0qxZs8jX17fDHUA9LZGRkXTs2DEqKSmxqKter6eKigpKTU216O0xZo4tLCwkhUJBBoOB1q9fT35+fuTo6GjVtgmFQnrnnXcoLy+PHnzwQWIYhl577TWTPjd6RjqLUqmkoqIi9v1lZWUmsXlGjJlkPT09af/+/ZSYmEijR4+26vUYNGgQ5eXlsTp25EGRSqX00UcfUWFhITU1NZHBYKDq6mrKz8+nRx55pFd05jwoXUClUqG2thYVFRXtZn1UqVQoKSlBfn4+UlJSYG9vDwCorq5GcXEx+Hx+n8mRotPpUF5eDpFIhLS0NLP8CPX19WhsbAQRoampCfX19ew5tbW1qK+vR2ZmJlJSUuDt7Q13d3ez72hsbERRUZFNrlFfL0KhED4+PuysTa1WQy6XIz8/H3K5vE9VOuXz+WyMBtCSD6SwsLDdWaGTkxP8/PxQUVEBuVwODw8PeHl5QSwWw9nZGXq9vtd3Lvn5+cHNzQ1SqRS1tbXw8fExuS8dHR3h6Oh4zZ8vEong4+MDb29vk+NarRZyuRzFxcWQy+U2sY6v1+uhUChQVVXVrrejqampzfweLi4u8Pf3BwAQERQKhc3UFzI+k9zd3TFgwACEhYWxul5NVVWVxbHo5OQEf39/NDY2orCwEDKZDIGBgR3Gbhg9Jy4uLggODoavry+bQ8raGJ9BeXl5Fr0gPB4PgYGB8PDwQEhIiMn1cnV1hZOTE/ubZhN02ry0IXrCg+Li4kJhYWGdyhoJgGQyGYWEhFBYWBiFhYWRn58f8fl8io+PJ7lczupqyx4UowgEAgoICGDbYpTg4GCSSqXEMAx5e3tTSEiI2Zqzh4cHhYeH065duyz21ebNm/vtLp7Wa9S5ubk0bdo0CgwM7HOeExcXF/r6668pKyuLsrKyKCkpqcOst5s2bSKDwUDvvPMOhYWF0YYNG8hgMFBtbS1lZ2fTunXrejU3hkAgoA0bNlBGRga9+OKLFB4eTh999FG7z5KuelBaj28jJSUlNHPmTAoODraZHS1CoZACAwPNxnVrCQkJsbh7ic/n086dO9k2GgwGWrFihdXbZRRPT0+KiIigd999l7Kzs6m2ttZiH3/00UcUHh5u1u6FCxeSUqmkP//8kwYOHEgPP/ww1dXVtXu/EBFpNBoqKCigvLw8UiqVNlGt3OhBycnJoVtuuaXNZ5CDgwN9+umnlJOTQw0NDWZt43bx2Ci1tbVdmu01NzcjLy/P7LharUZ+fj6ICD4+PhAKhQgMDERjYyNKS0uhVqu7U+1uQafTobCwsN1zysrK2nyNYRizmJ2amhpUV1dbdZ26szg7O8Pd3R319fWdyvQqkUjg4+ODgIAAk9mWVqtFQUEBCgoKelLdLiEWi+Hj49NhHJSLiwvCwsIQHh4OoOX+Dg8Pt3iPAy19fnU8Q+t7gKxU7fXq7zXqRESoqamxmO+js5l9JRIJvL29ERAQYPFa6vV6FBUVIT8//7r0706M9+O14OnpCRcXl+vyNvU0xgrMnp6e7K5LIkJ1dbXJszwnJwfZ2dlwcnKCu7s7GhoaUFFRwR43eldaP8NUKhVKS0vB5/Ph4+MDIkJpaamJp7mwsBC1tbVWy3djHN+BgYEQCARoampq9xnE4/Hg4+OD0NDQXtb02uAMlG4mNzcXTzzxBKKjo/Huu+/C19cX77zzDuRyORYuXIjU1FRrq9ht8Hg8LFiwAPfdd5+Zy/u7777Dtm3bUFVVZfPLOzNnzsSSJUuwd+9erFu3rsNU/LGxsdi6dSt8fX3h5ubWS1peGxEREXj33XfZAN62EAgECAgIYP+WSCRYvnw5nn32WYvnMwzDLm3NmTMHkydPZpdSfv31V2zYsAE1NTW9utyp0+mwbds2fPXVV3j00Ufx008/sTru2bMHO3fuNHtPZ3WMiorC1q1bERAQYHEpsz8hEAiwaNEizJgxAz4+PtZWxyJCoRCLFi3C9OnTzXT84osv8Mknn7B/GydJt912G1auXIkDBw5gzZo1OH78OO677z6MGDECu3btgru7u8nyRnZ2NhYuXAgnJyds27YNOp0OL7zwgpnR3pkJXk9hHN8BAQHw9PRsc0LRV+EMlG5Gr9ejsbERTU1NbDbR8PBwSCSSdiPI+xqurq5wcHBAZGQk4uLiALTMXurr69HQ0ICcnBy2aqqt4eTkZJL7JSwsDLGxsYiIiEBAQAAUCgVqa2shlUrh6uoKtVqN6upqiEQiuLu7Izg4GDExMewPlUajYde5rZUtVyKRwM3NzWwWaNS1tQGp0+lQVVUFg8EANzc3szV041p1Z/D09DTZMaPRaKBQKKwyqywpKUFZWRkkEonJfVlRUXFN96NYLIabmxuCg4MRGxvLtlOr1aKqqort79Yz674MwzDw8/MzuX51dXVoaGiw+s4khmHYWInIyEjExsaitraWNRCICNnZ2Rb72tXVFXFxcUhPT0dAQAC0Wi0aGhrA5/MRExMDPp+P0tJSCIVCuLu7Q6lUIiMjA46OjpDL5dDpdEhPT7epfE4ymQyRkZHw8/MD0GJc+vj4sLFHer0e7u7uEIlEAFpisCQSCQwGA2pqaszGqPH3y1bodgNl/fr1+PHHH5Geng6pVIqbbroJGzduNCkyRURYs2YNPvzwQ9TW1mLUqFF477332AHRlwkNDcVbb73FBk/1RwQCARYuXIi77rrL7Efsq6++wqeffmrTAaLz5s3D/Pnz2b+9vLzAMAxuu+02xMXF4cCBA1i3bh3GjRuH1atX4+zZs3j11VcxcOBAbNiwAT4+PnBycmLfL5fL8fLLLyM3N7fdAOueZOTIkXjjjTfMjGA7OzuLWw8rKirw8ssvo66uDm+++SYGDBjQbbrceeedGDhwIPbv34/169f3maBxSwwYMAAbN26Er6+vyZJWYWEhlixZwrrSNRoNcnJyrKVmj0JE2LVrF/bs2WP1AFmRSISlS5diypQpCA4Ohl6vx/vvv4+ffvqJPacjHSdPnozvvvuO9ew6OztDIpHgwoULWL58OcLCwrBp0yb2fLlcjmeeeQZEZLXx3Vn8/Pywbds25ObmYsmSJaitrcWmTZvY31Y+n4/Q0FCo1Wps2LABR44cMfsMW1qi7nYD5ejRo3juuecwYsQI6HQ6rFq1CtOmTUNqaiq7Q2DTpk3YsmULPvvsM0RGRuKNN97ALbfcgoyMDLOspn0FgUDA7moYMmQIGx2t1+tRX19vMtvqyzg6OsLe3h4xMTEYNmyY2evFxcVISEiwgmYdY29vz844hg8fbva60ROQmZnJ5gcYNmwYGhsb4eXlhcDAQMTHx5tlzFUqlUhJSbGqe1Umk8Hf35+NGzBW6dVqtaivrwePx4OTkxPr5SotLUVSUhKqq6tRVFTEegYYhoGTkxM74+oKzc3N7OwrMDCw0zkZugNHR0fWC8Tn86/ZWykSieDo6AitVguFQgEHBwcMGTKEXcozHi8pKUFiYiJyc3O7rQ3WgmEYODo6sn0uFAohkUhARGhsbERzczMyMzNtYlwb83cYnz06nQ5yubxd3aRSKezt7dnfFnd3d4vLdAqFApcvX4ZSqURZWRkaGhrg4uICpVKJ1NTUPlGF3ehB0Wq18PLygkQiwcCBAzF06FCT85qampCTk2MTfdoe3W6g/PHHHyZ/f/rpp/D09ERCQgImTJgAIsLWrVuxatUq3HPPPQCAzz//HF5eXvj666/x9NNPd7dKvUJUVBTeeOMNBAUFmdz85eXlWLFiBdLS0vr8DEskEuHll1/G5MmTERERYW11ugTDMHjssccwe/ZsBAcHt3vu5MmTsXfvXri7u0MoFGL48OH46quv4ODgYFtb8K7i3LlzmDt3LsaMGYPXXnuNfRhnZWVh1apVcHZ2xrp166DVarFixQpkZWVBLpdDq9Vi+fLl7Pn29vZYs2YNRo4c2WUdfvrpJ3zwwQfszLSsrKxXvCcikQgvvfQSpkyZAqClr681CHD06NF49dVXcfnyZbz22mtmr+fl5WHlypXIy8uzia3E3YFUKsWKFStw0003AWi5fhERETAYDNi5cycOHDjQpw2xGTNm4Nlnn4Wvry9ruLdHWloaHnvsMYSEhGDjxo2orKzEypUrbdorbKSkpAQrVqxAQ0MDXnjhBQQFBfW5Z/XV9HgMijE7oXE2lZeXh7KyMkybNo09RywWY+LEiTh16pRFA0WtVpvsfumOdVCpVGpxlqhWq032j4tEIkilUmi1Wov1dPh8Puzs7ODj44MxY8aYZDIEWmbXFy9eREpKynXrbE3s7Ozg4OCAQYMGYdy4cSav0f+q+BrF2shkMpMMkEqlElqtFj4+Phg4cCAAoK6uDmKxGBKJxCxuw5g504ibmxv78AZaZm1NTU3g8Xiws7NjaxoZl30MBgMbg9RbKBQKpKamwtvb2+R7lUol0tPT4ezsjOrqamg0GqSnpyMnJ4et0H31zhMnJyc2901zc3Ob+TGAlrgXY/0hACgqKsLx48d7pH3twTAMoqOjMX78+Gv+DKFQyObBGDt2LICWfre3twfDMGyfl5WV4ezZszbv6u8MDMNAJpPBxcUFgwcPZq+fse9ra2uRmpqKEydOWFnT9pFKpXBycoJSqYRGo4FEIjGJqQoPD8e4cePMjJPWVXuN971CocDZs2eh0WgQEhICZ2dnk+eJLaHT6aBQKNjViYqKCpw/fx5qtRpLly5FdHS0iTeRrspp1d7YthV61EAhIixevBjjxo1j17iN21Vb/5B7eXlBLpdb/Jz169djzZo13aYXj8fD448/jrvuusvstW+//Ra7du1i/54yZQpeeOEFHDlyBFu2bDFbpomLi8O///1vBAQEWCyW1x+QSqVYunQpbrrpJouxCkSEzz//HD/99BOysrKsoOH/IxaL2XsOaDEWtm3bht9//x2fffYZ/vnnH/bcBx98EA8//HCXvyM9PR2vv/46vLy8sHr1agQGBmLbtm1swFlFRQVee+21Xp11jhgxAsuWLYO/vz9kMhl7PDw8HO+//z6KioqwZs0a8Pl8vPLKKybxFFcjFAoxaNAgqNVqvPXWWzh16lSb3/nEE0/gX//6V7e3xRqMHz8eixcvhp+fH8RiMYYMGYJPP/2UXdLMzMzE66+/Drlc3ie2zncGe3t7rFq1CsOHD8egQYPY4zqdDu+99x7+/PNPpKWlWVHDjuHxeHjqqadw++23Y+fOndi/fz8efPBB3Hfffew5wcHBZpMQADh16hQ2b97MBjdXV1ebFADMycnBggULoNVqO70dvbfJzs7GggULWIOsqakJxcXFMBgMWLp0KYKDg/Gf//wH0dHR7Ovr1q3DuXPnkJycbE3VO0WPGijPP/88kpKSLFrgrW8YIrJ4EwHAihUrsHjxYvZvhUJhsiXyWggNDcW4ceMgEolMrOOUlBTY2dmxburQ0FDccsstqKurg4ODg1keE19fX0yaNAnOzs5Qq9VmFS6VSmWfWLtsD4FAgKFDh2Lq1Kkmx4kIGo0GGo0GKSkpOHTokJU0/H94PB4GDx6MW2+9FUDLw/bXX3+FTCZDYWGhyXZA40y5q9TW1uLIkSMICAhAbW0t/P39WYNIpVJBLpezlVHFYjF4PB5UKlWP3gceHh6YMmWK2RKUk5MTbr75ZiQnJ2PdunUQiUQYOnQoIiMj2/28pqYmXL58ud0+nTBhgsn9bmszMuP9qdVq29RNIBBAJBIhODgYU6dOZR/0xuup1+uhUqlQVlaGI0eOoLy83OLnGPuaYRio1eo+MeaNy5fGpTEiglqthlKpxOXLl3H48GEra2gZ43NWLBZDIBAgLi4OMTEx+Ouvv9jK4sbxD6BN77dcLseff/7ZptdXoVDg2LFjPdaO7kChUODo0aMWXzt9+jTkcrlJjSWdToeEhAT89ddfvaXiddFjBsoLL7yAAwcO4NixYybpdI3bHcvKykz2r1dUVJh5VYyIxeJuTSVsMBjw2Wef4dixY5g/fz5mzpzJvjZ9+nSEhYWxBkpwcDB4PB7GjRuHTz/91OzB4+HhAQcHB2RnZ2PDhg2oq6szeb2pqclqe+R7GoPBgF27duHgwYM2u4Rl9JYZH8JXc/XOsmshNzcXL7zwAqKjo7FixQro9XqsX78emZmZKCgogLOzM5YtWwYfHx9s2rTJqtuuAwMD8fbbb7OJmrqDPXv24MKFC+zf1vaeWWLPnj3Yv39/m56ASZMm4emnn0ZQUJDFBGypqal48803IZfLzcb21bi6umL58uVwd3fHxo0bkZ6e3l1N6DU0Gg22bduGEydO4NKlS9ZWxyIajQbvvvsu9u3bh+eeew4333wzgBYD8aGHHsLYsWMRGxtr8p7ffvsNn332mdlnFRUV9Zut4f2WDnPNdhGDwUDPPfcc+fr6UmZmpsXXvb29aePGjewxtVpNTk5OJmmV26O7Ut0zDEMbNmwglUplsWS6JXQ6HalUKtJoNGQwGEiv15NKpaIjR46Ql5eXVdMd94QIhUJyd3enX3/91aQPNRoNNTY20uOPP251HY0iEAjI2dmZfvzxx071pRGtVmvSp6372Pi3Uf766y9yd3dnv3fAgAGUlpZGycnJFBUVxR739vamY8eOUVFREY0fP75H237XXXdRVVUVq6NarTa5P41/t8ZgMJBarTZpn0qlourqapo+fbrV+7QzIhaLac+ePWZt0+v1tHTpUhKJRG3KggULLI5943U7ePBgp8pf+Pn50cmTJ6mgoIDGjBlj9WvSGXF1daU///yTbXNTUxPNnj3b6np1RkQiEe3atYtUKhXpdDoyGAzsODaKsV/ffPPNfltuoyPx9fWlM2fOsOO8rKyMpk6danW9ACulun/uuefw9ddfY//+/XBwcGBjTpycnCCVSsEwDBYtWoR169YhIiICERERWLduHWQyGR588MHuVqddiAjffvstUlJScM8992DWrFltLjMZuXjxInbu3Ino6Gg899xzyM/Px7Zt25CXl2exXHlfxs7ODi+88AIGDx5sskat1+vxySef4Pjx4zh//rwVNfx/pFIpnnvuOQwdOtTi9uf2OHToEPbs2YMpU6bg4YcfxsmTJ7Fr1y4MHz4cTz31FFJSUrBjxw7WFVxZWWlSRK+oqAgrVqyAwWAwifSvr6/HunXrYG9vj4yMjO5paBtcvnwZzz33HOsFiI6OxsKFC1FZWYmtW7fCwcEBL730kllG2bq6OmzdutVsh5ler7fZWXRnYRgG9913HxsYbYnw8HCLOzuSk5Px3nvvIS8vr1OJq2pra/HGG29AJpPZVCKv/opOp8PHH3+MI0eOYP78+Zg4cSL27NmDgwcPsufMmjUL9957rxW1tB0aGhrw7rvvIjk52WYTaFqkCxPNToE2rKVPP/2UPcdgMNDq1avJ29ubxGIxTZgwgZKTkzv9HT1RLHD16tWk1WrbFKOV/sMPP5BMJqNbb72Vqqur6fDhw50uMNgXhM/ns+Lh4UF//fWXSb/pdDpqamqiefPmWV3Xq8XR0ZF+/vln0mq1pNfrze4Zo+6W+nbjxo3EMAw988wzpFKp6OOPPyaBQED33XcfKRQK+umnn8je3t7qbeyKjB8/nkpKSujUqVMUEBBA8fHxlJ2dbdZ2uVxOI0aMsLq+1yNteVC6gl6vN7kuP//8Mzk6Olq9bT0lPB7PbHz3JQ+KUfh8Pr3//vukVqvp+eefN3lt5cqVpNFo2PFtbV2tIUYPSkVFRY97cbsqVvGgUCfqrjAMg9dee81ingFr8csvv7S7z33UqFEmOz5SU1Px8ssvo6qqyiwwtq/i4uKCBQsWsNlhxWKxSZyGVqvFZ599hvPnz+Ps2bPWUtMiSqUSO3bswMGDB/Hoo4+aJSYiInz//ff4+++/zd6blJQEIsKxY8fwwgsvIDMzEwaDAQkJCXjppZdQVFRkE9unu0JWVhaWL18ONzc3LF26FDqdDlu3bjVbc29ubrapAnfW4vz589i9eze7S6+wsNBqBeB6GgcHBzzzzDOIjY297jgsa2MwGPD111/j4sWLOH36tMlrf/zxB6qqqtjxzdEHua5ph5XoCQ9KRzJ37lxSKpX0/fffWyxN3teFYRgKDAykhISENq97X5hhSaVS+u6779j4C6NotVp64YUXrK5fb8uIESOosLCQLl26REFBQVbXpydELBbT119/bdLfXZXPP//cYnn6/iYMw5C3tzedPHmSHdfGsdLY2Gjz45uTromPjw+dPn2aysrKOA9KfyYhIQHLli1DXl5ev4v8dnNzw2OPPYaIiAi26FRfR6/XY+/evWwqZyKy+YRTPUFhYSFef/11KJVKkxL0/QmdTocvv/zyuuKh0tLS+kUpivZwdnbGo48+isjISJNsymq1Gl9++SWSkpKQmJhoPQU5uh2FQoF3330XTk5OfTMbcHd4NHoba3hQ+rOEhYVRSkoKe30NBoNF6UseFJVKRXPnzrW6PpxwYisSEBBAFy5cMBvn9fX1dOedd1pdP05uLOE8KBzt4unpiQcffBBhYWFssTigJXfLnj17zKpaarVam48A12q1+Oabb3Dx4kVuNsjBgZYcLQ8++CDCw8NNKqwrlUrs3bsXaWlpPb7LjIPjmugRF0cPw3lQukfi4uIoMzPTLD9GaWkp3XTTTVbXjxNOOLl+CQsLo+TkZLNxXlVVRZMnT7a6fpzcmMJ5UDjapbKyEh9++CFbSt5IQ0NDvyiGxsHB8f90lOOJg8PW4AyUG5iKigq89dZb1laDg4ODg4PDDM5A4eDg4OhHeHh44M4772QLR7q7u5t4SZubm/Hrr78iJyen39YJ4+gfcAYKBwcHRz/C398fq1evNqn4fnU6f4VCgXfeeQenT5/uE1WXOW5cOAOFg4ODox9RXV2N77//Hi4uLhZfVygUKCsr44wTDpuHIep7OYAVCgWcnJysrQYHBweHTSIQCNoMiiUi6PV6Lv07h1Wpr6+Ho6Nju+dwHhQODg6OfkZ/z4rLcWNgXmecg4ODg4ODg8PKcAYKBwcHBwcHh83BGSgcHBwcHBwcNgdnoHBwcHBwcHDYHJyBwsHBwcHBwWFzcAYKBwcHBwcHh83BGSgcHBwcHBwcNgeXB6WHEIvFCAkJgVgsBgBotVrk5eVBqVRaWTMODg6OzuHv7w83NzcUFxejqqrK2upw3GBwHpQeIiAgADt27MC+ffuwb98+fPzxxwgPD7e2WhwcHBydgs/nY8GCBfjxxx8xffp0a6vDcQPCeVC6CYlEAj8/PwgELZc0LCwMISEhCA4OBtCSelokEoHP58PPzw8ikQglJSVobm62otYcHByWsLOzg6+vr0mRPUtotVoUFRVBo9H0kmY9B8Mw8PHxgYODA4CWAoN2dnZQq9XQ6/VW1o7jRoQzULqJqKgobN++He7u7gAAkUgEX19fs/OcnJywfv16hIWFYdGiRThz5kxvq8rBwdEBQ4cOxZYtW2Bvb9/ueYWFhXjuueeQlZXVS5r1HCKRCK+88gpuu+02AIDBYMAHH3yAu+++GxUVFVbWjuNGhDNQugmpVIqwsDD4+Pi0ex7DMBCJRBCLxR3Ozjg4OKyDnZ0dIiIiIBKJUFlZCaFQCA8PD9ZDakQmkyEkJARKpRIVFRV92pPC4/Hg5+eH6OhoAC31fOzs7KBSqW5ID4pIJIKnpyf4fL7Jcb1ej/Lycmi1WitpduPAGSi9TH19PVasWAGJRIL8/Hxrq8PBwdEOKSkpeOWVVxAYGIgtW7awHlIjXl5eeOeddyCXy/HSSy8hLS3NSpp2P8YYlNmzZ2Pz5s34+uuvra1SrxIeHo63334bnp6eJsdLS0uxaNEiZGZmWkmzGwfOQLlORCIRnJ2d4ebmZmZpAy2zkLq6OlRUVECr1UKn0yE7O9sKmnJ0FqlUCkdHxzbL1bdGq9Wirq7uhpxlAoC9vT3s7e3R2NiIxsZGa6tzXYjFYjg7O8POzg6VlZUoKChAUlIS1Gq1yYxZp9OhtrYWOp0OTk5O8PDwgFAotKLmPYOjoyP4fD7s7OysrUqvIRQK4eLiAm9vb3h5ecHPzw/Ozs4AgNraWmi12n7Z17YIZ6BcJwMGDMDatWtNbuKrKS8vx7Jly5CWloacnJzeV5Cjy0yePBnLli0zc+e3RU5ODpYuXYrS0tIe1sw2uf/++/H444/jyy+/xI4dO6ytznUxevRorF69Gnl5eXjyySdRUVGBhoYGs/MKCwuxbNkyFBYWAgA0Gk2/G98GgwE7d+7Evn37bihvb2xsLNatW4eGhgYsW7YM7u7u2LRpE3Q6HZYtW4aMjAzk5eVZW80bAs5A6SJCoRD29vbs7Nrf3x8jR46Eq6srgJaZVWNjIzubLi8vx8WLF/uV6/dqZDIZJBKJ2XGlUtlncr5IJBLIZDKoVCo0NzfDw8MDo0aNgkgk6tT7HR0d4eXlBbVaDaBljfrqe6C/IpVKIZVKERERgdGjR+P8+fNwdXVlr2NfxN7eHhEREaisrERCQoKZcWIc36WlpUhISEB+fj7s7e0hEAggFoshFArR2NgInU5npRZ0HoZhYG9vz3oDJBIJRCIRiAhNTU1obm5GWlraDRPILxAIYG9vDz8/P4wcORIFBQUoKiqCQqFAZWUlNBoNEhIS+kVAtCXs7OzYvF2WsMYznTNQusioUaOwYsUKtiNdXFzYbXkAkJeXh//85z9s1LtSqURBQYFVdO1peDwennrqKdx1111mr3311Vf49NNPraBV15k+fTqefvppHDhwANu3b+/y+4OCgvDee++xg7eiogL//ve/+92MujUPPfQQ7r//fnYr/cyZMzFgwAD89NNPeO+990BE1lXwGjh37hzmzZuHuro6i0aWcXzn5eWhtLQUXl5e+O9//8teg7q6OqxevRpXrlzpZc27jr29PVavXo0hQ4YAaIk5iY2NhU6nw/bt23Ho0CFkZGRYV8leZNCgQfjPf/4Df39/ODo6Ijw8HDt37kRRURHWrFmD0tJSFBcXW1vNHsEYb2TcwWWJvXv34qOPPupFrTgDpdMIBAJIJBIEBgbi5ptvhkwmM3ldr9dDqVSivLwcJ06cQFFRkZU0vX4YhoFEIrEYU3M1AoEAAwcOxJQpU8xeu3jxIhwcHKDRaFjPgq0SGBiISZMmITc3Fw4ODpBIJBbjT3Q6XZsziIEDB4JhGEilUhQXF8PLywvl5eUAACLqlzshwsPDMXnyZPZvHx8fuLq6IiEhAQzD9EkDpbKyEkePHjU7rtfr0dTUZDa+fXx8MHLkSMTFxUGpVKKsrMxkwmLLCAQCREVFYeTIkZBKpeDz+VCpVFAoFEhMTMQ///xjbRV7BT6fD6lUCn9/f0ycOJFdqufz+RgyZAj4fD4uXrwIuVxuXUW7CWNfX41QKGzzWW7kwoULPa2aGT1uoKxfvx4rV67Eiy++iK1btwJoeWCvWbMGH374IWprazFq1Ci89957iIuL62l1rplx48Zh4cKFCAgIsOj6z8rKwtq1ayGXy/t8SmgnJyesXLkSMTEx7Z7HMAwGDBhg8bVZs2YhJiYG+/btwyeffNITanY7t9xyC7788kv4+/tbNM6Sk5OxcePGNgNB/fz88Oqrr8Ld3R3r16+HQqEAACgUCqxfvx4pKSk9qr+1+fnnn/HZZ58hLy8PBoPB2up0K1lZWViwYAHUarXF8V1TU4O1a9fiypUrfcbr0NjYiLVr12L37t1YsmQJBg8ejHfeeQfHjx9HYmKitdXrNYYNG4YlS5YgICDAJBg4JycHa9euRX5+fr/JA2NnZ4elS5di6NChJscZhsHAgQOtpFU7UA9y7tw5Cg4OpkGDBtGLL77IHt+wYQM5ODjQDz/8QMnJyXT//feTj48PKRSKTn1ufX09AegV4fP5JJFI6NFHHyWVSsXqoNPpqLm5mVQqFRkMBjp16hR5e3v3ml49JSKRiIKCgujUqVPX1fcajYaam5tpw4YNxDCM1dtlSQQCAUkkElq2bBnp9fo222Ls659//pmcnJza/LyIiAhKTU01e39lZSVNnDjR6u3tzusmlUppy5YtJn39xhtvWF23nhKGYUgsFpNEIjGRmJgYSklJocLCQho2bJjV9eyqiEQi8vDwoF9//ZWamppo9uzZVtept+WOO+6giooKam5uNpHjx4+Tr6+v1fXrzr728vKiw4cPX9Mz/a233iKpVEoCgaBb9Kmvr+/wO3vMg9LY2Ii5c+fio48+whtvvMEeJyJs3boVq1atwj333AMA+Pzzz+Hl5YWvv/4aTz/9dE+pdE1MnDgRjz32GMLCwkx2dSQnJ+Odd95BSEgIXnrpJStq2H24ubnh5ZdfRkxMDCIiIq7rs3755Rfs3bsXGRkZNuvqnz59Ou677z5ERUW1u6X48uXL2LZtG/Lz8/ts8Gd3wTAMHnroIUybNg1DhgwBEWH//v34/vvv+20gONCynLV48WKznXrGoMq+uL3azs4OL730EoYMGYLBgwdbWx2rcfnyZSxYsMBs115NTQ1qa2utpFX34uDggJdeegmDBg265pWKO+64A35+fjh48CA+//zzXnmu95iB8txzz+HOO+/E1KlTTQyUvLw8lJWVYdq0aewxsViMiRMn4tSpUxYNFLVabRLHYHSd9wZhYWG4//772ZvXYDBAr9ejsLAQP/74I+Lj4/HUU09Bp9PZ7A9xZ5FKpZg2bRqGDRtmcpyIoNfru+S2T01NxY8//mjTcRfR0dGskdxeVsiCggJ8//33aGpqMjnOMAy7FNQXdm1cL3w+HwKBAMOHD8ecOXMAtNwbKSkp2Lt3r5W16xmMfezl5YVZs2bB29vb4nl9zUAx5jaZPHkybr75Zuj1ejQ3N/e7pbnOUFJSgh9++MHaavQoxt/Y8ePHtxtbSETsb5wRPp8PPp+PmJgYxMTEoLKyErt37+67Bso333yDixcv4vz582avlZWVAWjJwHg1Xl5ebQYhrV+/HmvWrOl+Ra+BS5cu4eOPP0ZOTg6USiUyMjLw8ssvQ6FQ9Krh1JuoVCp8+OGHSEpK6vR7/P398f777+Pw4cM2++P122+/oaysrMOEbAUFBRYDfYOCgvD888+jqakJ27Zt6yk1bQIej4eHHnoI48ePx+jRo62tTq8RGRmJZ599FiEhIXB0dLS2Ot2Co6MjnnvuOcTGxiIqKood3xcvXsTFixetrR5HD9DQ0IDNmzfjxx9/xDPPPNNm7CAR4ZtvvsHff//NHpsxYwZmzJjR6cSV3Um3GyiFhYV48cUXcejQIYv5MYy0biwRtXkBVqxYgcWLF7N/KxQKBAQEdI/CV+lj6ftbH5PL5di9ezfr6i8rK+tyCmjjd9nibKW19Qy0eLAOHz6MX3/9tcP3G9u2fPlyzJ8/H3V1dfj2229t0ruUmJh4zcGAPB4Pnp6euO+++1BTU4PPP//crLYSEbHeJ1tsf1fg8XgYN24cHnvsMWur0mNYqo3l5+eHuXPnws3NzeJ7LM04bRmGYWBnZ4e77roLY8aMARFBoVB0enz3Zdp6xhsxjtf+iFqtxm+//QYnJyfcddddbW6AMBgMOHXqFHbt2sUe8/f3x1133QWGYcDj8dh/e+N6dbuBkpCQgIqKCpNlAr1ej2PHjmH79u1shHtZWZlJYb2Kigozr4oRsVjcbgKZ7uD222+3uAc8Li6uW4v62dvb44knnoC3tzd27dplU/Uc6urqsHnzZnh4eJgc12g0nc7rcOedd2LatGmoqanBokWLcPny5X436ENDQ/H4449DKBRiy5Yt4PP5eO655+Dh4WFyDysUCnz88cdIT0/v9zlR+jrDhw/Hgw8+aBaH4OfnZ5ZS4Gqqq6vx0UcfITMz0+bzHTk7O+PJJ59EVFQUQkJC0NzcjF27diElJaVP5G25XsaOHYvZs2e3+TxPT0/Hp59+2mcSTF4LSqUS77//fpvGKBHh+PHjJsf++OMP1NTUYOrUqZgxYwbGjx+Pt99+G6dOncLevXt7dqJ9TeG87aBQKCg5OdlEhg8fTvPmzaPk5GQyGAzk7e1NGzduZN+jVqvJycmJdu7c2anv6O5dPAzD0Jo1a8hgMLT5nQaDgQwGA33//fckk8mu+bs8PT3p+PHjVFVVRVOmTLF6ZHd3X8d169aRwWDo1zs6xo8fT+Xl5XTu3Dny8/OjQYMGUW5uLnuPGKW4uJhGjRpldX27Q4RCIX3yyScWx8Tq1autrt/1yty5c0126XX0HDBKdnY2DRgwwOr6d0YCAgLowoULbFuqqqpo8uTJVtert2TBggWk0+na7NODBw+Ss7Oz1fW0VVm6dKnJb+RHH310XTt6rLKLx8HBwWx9y87ODm5ubuzxRYsWYd26dYiIiEBERATWrVsHmUyGBx98sLvV6ZA77rgD48ePx7hx49o97/Lly/jhhx+Qmpp6XSXVGxsbsXPnTri7u/erooF33nknxo0bh4kTJ1plrbInGDRoEGbPnm1WGMzf3x92dnbw9/fH8uXLIZVK4eLigurqauzevZvNmdDY2Gjzs+qO4PP5+Ne//oX4+HgTrygR4ffff8exY8dw4sQJK2rYu5SUlOCLL75AXV0dgBavozGuzlZxdXXFww8/jPDwcPj5+VlbHZsjPT0d33zzDTIyMvq196QvYpVMskuXLoVSqcSzzz7LJmo7dOhQr2dgZBgGkyZNwpIlS8xeo1bLEleuXMGbb755XcYJADQ3N+Orr766rs+wNRiGwdSpU7Fo0SJrq9KtxMTEYMmSJZBKpRZft7Ozw/PPPw+g5X7JysrCRx99hPT09N5Us0cRCASYMWMGHnjgAfaYcWz8888/eOutt6ylWo9A7cTCERHKy8uxY8cOtkhgX8DFxQWPP/444uLi+s3k4Xpo/WzPysrCli1b+txOrBuBXjFQjhw5YvI3wzB47bXX8Nprr/XG118T58+fx6FDh9ibOTExsUuBcO7u7rjvvvvYIoJtodFo8NNPP7EFqHg8Hu666y4MGDAAf/zxBxdVbwUGDx6MO+64A0OGDOlURePy8nJ8++23yMnJ6fNZhDvCYDDg4MGDuHDhAk6fPm1tdbqNlJQUrF+/HnFxcZg5c6ZZtuiioiJ89913yMrK6nO79Wpra/HBBx8gPDwc9913n0nsX39m3LhxuPnmm9m/R44cCYZhkJiYiN9++419nmdkZFz3xLM/M2HCBEyYMKHDVYYeocNFIBuku2JQGIahN9980+J3bN26lXg83jV/dkxMDGVkZJitWbeW+vp6mj59Ovs+oVBIn376KWm1WnrmmWesvu7Y2ev49ttvm1y/vhyDMn/+fFKr1WwfdURiYiKFhIRYXe+eELFYTF9//TXbVp1OR88995zV9eopufvuu6mxsdGsj/tDpugbLQbl1VdfNXveEhHt2rWLhEKh1fXrK7J69WqLz8I+GYPSFzAu7QwZMgTDhw83ee3cuXM4ceIEjh8/3qndJ/7+/pg+fbrZlmovLy+4uLigvr4eBw4cgEqlwsyZM812KrV2uRpnqFVVVTZfD4NhGEyePBmDBw9GfHy8tdW5bgYPHoxJkyZh1KhR4PP5N7Q7XCAQ4Pbbb0d0dDSioqKsrU6vkZ2djW3btpl5UORyuVmivr5GZ55nbTFx4kST+i1nz57FqVOnukOtHqer49jX1xfTp09vd/fW1SiVSvzyyy99ukBsZ+gXeVD6AgzD4J577sGCBQvMLvo///yDlStXdnrrVHh4ONasWWMxTwLDMMjLy8Nbb72Furo6DB06tM2t1Eb0ej327t2LvXv32vz2XB6Ph9mzZ+Opp57qFz/mN910EzZt2nTDGycAIBKJMH/+fMyaNeuGuhbJyclYuXKlxddsfTz2FAzDYObMmXjxxRfZY+vWreszBkpXCQ0NxZo1a8zSLbRFVVUVUlNT+72BYg1uSAMFAJtspjX0v8RLHREQEIApU6Zg4MCBkEqlbe6tp/8ls2loaMCPP/6I1NRUTJs2rc2U2cb39BWuvo5EhPPnz+PSpUs4d+6clTW7Nng8Htue7OxsHDlyhF2r9vf3x9SpU81y8ri6uuKBBx5g408aGxtx8OBBVFdX967y3YBIJMLUqVMRFhaG0NBQ9loYDAYcO3YMaWlp/T5nRl8af92BUCjElClTEBgYaPF1hmEwaNAgk2ecLRutI0eONPGOJyYm4syZM+zrp06dsviM9/Pzw9SpUxEXFweZTNbp/FcymQwzZsxAaGgo/vzzT5SVlWHSpEkIDAzE0aNH++xuzVGjRmHw4MFmpU9SUlJw6tQpHD9+vOeTjXa4CGSDXG8MCo/Ho/fee8/iZ69fv75Tn3HrrbdSdXU16XS6duMUrs6TwOfzycfHh06cOMG+rlAoTGJQ+pLw+XyT3DUGg4FWrVpFfD7fZisYtyet8yR88803ZG9vT3w+n/h8Pt16661UU1Nj1scGg4G0Wi0r2dnZNHjwYKu351rEycmJfvvtN9JqtSYVnjUaDT3xxBN9tm85Afn7+1uMQXFwcKB9+/aZ3MOtpXW1b1uOMVu7dq2Jzps3byaBQMCO47ZiC6dMmUIVFRUdPtMtjX+dTkdVVVU0bdo0kslk9O2331JjYyM98MADVr8e1yobNmyw2Pfbtm0joVB4XTGaABeD0iZEhHPnzsHR0RFDhw5FTEwMkpKSkJycjEuXLrX73oCAAIwdOxZDhw6FRCJps/BSXV0djh49ipycHDZngl6vR1NTEw4ePIji4mKMHz8ezs7OuPnmm2FnZ4fTp0+3WY/IliEiJCQkIC0tDUlJSX0m7XdrMjMz8dVXX7EzpzNnzkCtVrPtKSoqwrfffgs7OzuT9xn70N7eHgDg5OSEO++8E2FhYTh69Gif86TweDyLu5f6Ukp3jo4Ri8WYOnUqQkNDERgY2Kkda7bM8OHDER0djYEDB5q0xXjfUgeesbKyMvz4448IDg7GxIkT2y3VcjVqtRrHjh1Dfn4+SktLodPpcOLECTQ2NiI/P/96mmRV2noO0DUUj71mOm0m2hDdsYtHIBCQRCKht99+mwwGA61Zs4ZEIhHx+fx23zdz5kyqqakhjUbTrpWdmppKcXFxJBKJzGacQqGQvLy86K+//iKDwUAajYZqa2vp3nvvtbrV3BUxelD0ej29/PLLnbp+tiw8Ho9EIhErrSPUGYYxed0ow4YNo9zcXLbvDQYDqdVqysvLoxEjRli9XV0RJycn+uOPP8zuZ41GQ4899pjV9ePk2qW1B8V4n6pUKosZVtvD1jwoDMPQW2+9RSqVirRarYmub775Zqe8fsbxPWnSJKqoqOj0taisrKSpU6eaPOsFAgGJRKLr9jJYUzZt2mSxve+++263tIvzoLRDbGwsQkND0dTUhH379uHKlSsme+FDQ0MxcOBAyOVyJCYmws/PD0OHDsWYMWMglUrNsovW1NTg7NmzkEgkGDVqFIgIGo3GbH+9WCzGyJEjERQUBDc3N+h0OiQkJEAul6O0tLRX2n69MAyD+Ph4BAcHIzg4GEBLUrM77rgDqampNlVfqCsYDIZ28yEY+7Q1Go3GZHamVCpx5swZ5Ofns94zW0ckEmHkyJEIDAyEp6cne1yv1+PixYuQy+U2693j8XgYOnQo/P39AbT0U1JSEvLy8nr8u2UyGUaPHs1WOlar1Th79ixqamp6/Ls7i52dHUaNGoWQkBA4OzuzxxmGMdutZAkiQkpKCrKzs9kKyLaIQCAwiQ9LT09HWloaUlJSOhVX5O7ujhEjRsDf3x9//vknfHx8MGrUKDZRY2NjI86cOQO9Xo9Ro0ZBIpHg7Nmz7LP76meDTqfr/gZehZubG0aOHInm5macPXsWKpWqU+8TCAQYOXIkXF1dkZCQwP7mMAyDIUOGICgoiD03IiLC5L2ZmZlITU1FUlJS78VpddpMtCG6IwZly5YtpFAoaMmSJSSTycz2xS9YsIAUCgVt27aN+Hw+zZkzh6qrq0mpVFr0nJw5c4ZCQ0NpzJgxVFBQQFeuXKGIiAiz7/bw8KDDhw9TY2Mj6XQ6UigUNHv2bJLJZH3G+yAUCunDDz+kxsZG1pOkUqmooaGBlixZYnX9elsGDhxIOTk57L0gl8tp9OjRJJVK+8wMys3Njf744w9qbGw0mYEqlUqaP38+yWSy68p50JMiFovps88+o8bGRmpsbKT6+np68skne+W7g4OD6ezZs+x35+bm0ujRo61+Ta6WkJAQOnfuHDU1NXXZU0JEpNfraenSpWRnZ0ebNm2yyVpbDMPQ1q1bWZ0NBgOtX7+e7OzsSCQSdeozJk2aREVFRfTnn39SYGAgjR8/noqLi9nPzMrKovj4eIqOjqbExEQqKSmhiRMnkkwm6/VxPnbsWMrPz6cjR450KT+Pk5MT7du3j8rLy+muu+5ijwsEAtqxYwd7Hxuf7VezefPmLl3PjoTzoLSCYRjExsYiICAA4eHhsLe3R0REBMaNG4fs7Gzk5uay5wqFQtjZ2SE0NBS33HILhgwZAgcHBzPPiRGDwYDm5mZUVFTg6NGjUCqVaG5uZl8Xi8Wsherj42MSx6BWq03OtVV4PB4GDBgAf39/hISEQCaTIS0tzSTtd19KAd5dNDQ04Pjx4ygpKcGQIUPYvhaLxbh8+TLq6+utrWKHMAwDqVRqFl8DACqVyqbuT4FAgMGDB7Nb+4VCIYKCgiCVSpGamorCwsIe90ba2dlhyJAhCAsLg4eHB/h8Pi5fvgy5XG4zmWbt7OwQHx/P6tg6r4dWq8Xly5c75e3h8XgYO3YsVCoVDh06BL1ej2nTpiE/P98qHlOGYRAXFwdfX1/2b6VSiYMHD7LnpKWldSl3DZ/Ph0wmg5eXF0aNGgUvLy8TD5PBYIBSqURNTQ1OnjwJZ2dnlJeXW2Vs8Hg8VtdJkyaZxbkplUpcunQJWq0WQ4YMgZOTEwCwNcQcHR0xfPhw1uvD5/MREhJicfwb8ff3x9ixY1FQUNB75Ty6bE7bANfqQREIBLR9+3aqra0llUpFBoOBmpubqaamhpYuXWpy7sKFC0mv15NKpaLa2lpqbGxsN+bEmGmSz+eTg4MDOTg4mFjVvr6+9Pfff1NdXZ3JDLUv7eKRSCT06aefUm1tLanVatJqtbRo0SJycnJiRSwWW13P3hYej0cODg40dOhQSk9PJ71eTwqFgtLS0mjYsGFW168z4u7uTkePHjW7r5VKJc2ZM8fq+l0tTk5O9OOPP1JtbS0rarWaNBoNPfvss+Tk5NRts7y2JCoqii5cuED19fWk0+moqKiIbr75ZnJ0dLQZT2h0dDQlJCSwOramurqapk+fbjJ+LYmLiwtt3ryZampqaOXKleTk5ESrVq2impoaeuONN6yyq0skEtHOnTvZ/q+urjZ7Fkkkki595tSpU6mmpoa0Wi3V1dWRQqEw2cGSkZFB0dHRxDAM2dvbW7Wvx48fTxUVFayuV4+F2tpaSkxMpLi4OPL29qY///yTPV5XV8d6vRsbG03e01E1b6VSSbW1tbRx40YuBqUnYBgGMpnMZB1WKpVCIpEgMjLSpNaARCLBiRMn4OPjg/DwcHbff1VVFTIyMswimI0xLBKJBNHR0dDr9UhNTQXQEu9i9JwYLVmtVov09HSUlJT0mV0eDMPAzs6OvX56vR6+vr4YOHAgcnNzUVJSYl0FrYREIkFUVBQiIiIgFouh0WiQlZWFgoICm6+OKhKJEBMTg4CAADaOwlYRCoWIiYmBn58f/P39TcYx0BIL1Nzc3CseKx6Px3pUjZ6T0tJSm/GeAC2zYgcHB5P4mNTUVKjVasTGxoKI0NTU1O71Cg8Ph4+PD3u9g4KCMHDgQPB4PKSkpFg1bu7qZ5HBYIDBYOhS37u7uyMyMpLdtWfc/SMQCODk5ISGhgacP3+e9TIUFhZCqVSCiHq8sKCXlxfCw8NRVVWFzMxMuLi4ICoqit01OmjQIAiFQlbX1iiVSowYMQJ1dXXw8fExGysA2vWWGCEiFBQUmFRlLykp4WJQ2uNaPShCoZB27dpl9nlGa7K8vJyVzZs3k5eXFy1evNjE4/HLL79QYGAgeXh4mIizszPxeDwaMGAAXbx4kY4fP06hoaEUFBRER48epcrKSpM1PeOuHXd39x6f7XWXSKVS+vbbb02um0KhoLKyMnrqqaesrp+1JCYmhs6fP09VVVWk1WqpoKCAJk2aRG5ubjYbt2EULy8v+v3336myspLUarXZ2LAlD4qbmxvt37+fKioqLOqqVqtp/vz5vdbnGRkZlJeXR+PHj7fJvo6Li6PMzEz2+hQVFdGUKVMoNjaWzp4922EtHj6fT2+++SaVl5dTU1MTGQwGamhooPLycvrvf/9LHh4eZGdnZ5W2iUQi+uKLL9i26fV6WrhwYZc+Y9asWSSXy9lnfm1trYnH5OLFizRo0CD2Ge/q6tprHpMHHniAiouLaefOnSQSiei2226j3NxcVteamhqz/CRXo9PpqLq62ux3p6sYDAbatGmTyW+dvb19t7SR86C0goiQn5+PhIQEBAQEsLsVjJ6Bqy1KHx8fBAYGstWIq6urkZ+fj5KSEvj5+bW52yMqKgo+Pj5obm7G4MGDWS+Du7u7yXkGgwF1dXV9ovotj8dDaGgoPD094eLiAoPBALlcznp+6H+Zcm0BhmEQEhICFxcXAGBnAD1xnWUyGcLDwxETEwMfHx82JsJgMKCmpsamPWMikQjh4eEIDAyEv7+/2f2p1+uRl5eHiooK1NbWWklLUwwGAyoqKtjdRMb70tLssKeQSqUIDw9HVFQUJBIJtFotamtrbaqvZTIZwsLCEB0dbZLLg8/nw9vbG3w+H2KxGAKBAJGRkazXwWAwsDvPgoKC4OnpyY57oGUs1dXVoaysDCUlJaisrLRK+4y65OXlISEhAYGBgRZLjfj4+MDX1xfl5eUoKiqCh4cHAgICWG94TEwMvLy8oFQqkZuba+YVT01NRWlpqVXa2dTUhIKCAhgMBsTHxyM2NhZeXl6drg/E5/PZ3y5LGMd3Z3YZZmVlWa+vr9m0siLXs4vHxcWFgoKC6PPPP2/3OxQKBRUUFFB1dTUZDAb64YcfKDQ0lObNm0eZmZlUUFBgUUpLS0mr1ZJGo6GSkhIqLi62ONurrq6mKVOmWGX20VWxt7enXbt2UWFhITU1NZFaraYXX3yR/P39Wekuq/p6RSKR0M6dO9n+yM3N7bFsjoMGDaJz586xfW4kPz/f5jPJ+vv70x9//EHFxcUW156bmpro8ccfJ39/f5LJZFbXF2iJ9fH09GTvuaioKDp48CCrc294UOLi4uj06dNUUlJCGo3GJFO0rciAAQPozJkzrI5GtFotlZaWUlFREZv7pLy8nB0rGRkZdMcdd5BQKKS3336bCgoKqKGhgX2/wWCgDRs2kL+/Pzk5OVm9na6urhQcHExfffWVRQ/KwoULSS6X06uvvkoMw9BDDz1EeXl5bHsrKyvJYDDQoUOHKCoqyuR55u/vT56enlaLMbGzsyN/f3969NFHKTs7myorK9v1mHSVq8d3R+Lo6NgjbeQ8KBaora1FfX09srKycOXKFXh5eZnNHgHAwcEBDg4O7N9CoRCOjo5wdXWFv78/uze+NUZrnMfjsdZ6QUGB2T51hULRZ6qjMgwDDw8PNs+ERqNBTU2NTRbHYhgGbm5uCAgIANCSj8CY4bW7kMlkCAgIQHR0NAIDA9kCkGq1GgUFBcjPz+90XoLeRiQSITAwECEhIQgODmZ3QRjR6/UoLCxEZWUl8vLybKqPjR4U49hydHTsdPZTDw8PeHp6orKyEhUVFXB1dYW3tzdqa2tRWloKJycn+Pr6dlh/JTo6GkFBQXBxcUF+fj5ycnJsrq9FIhF8fX3h4+NjclwgEJjVALs6541KpUJUVBSKi4sRGhrKjiEiQllZGaqrq5GdnW0T94Rx15mjoyO7s9Lb2xtxcXHsOeHh4QgICEBYWBhiY2MRERGBgIAAKBQKlJSUoL6+HmVlZcjIyEBhYaFN7VRrampCU1MTlEol/Pz8Op3VtjVardZi25RKpc2Nb0vccAYK0PKg++CDD7B3714sW7YMjz/+eIfvmThxIr777jvY2dmZFYu7mtzcXCxYsAB2dnbYsWMH+Hw+XnzxRbOteHq9HmVlZdfdFo7eJyYmBu+99x58fX1NXMslJSV44YUXkJ6ebrNJ93x9fbF9+3Z2KbI1TU1NWLNmDY4dO4by8nIraNgx9vb2eO211zB+/PgOq4MbeeCBB/D888/jww8/xFtvvYW77roLq1atwnfffYfVq1dj4sSJWL9+fZtpBIyIxWK4u7tDLpfj2WefRVZWls32dVcRiUR45ZVX8Oyzz5oYLkSEXbt24bPPPrOZpSyBQIBFixZh1qxZ8PDwAMMweOKJJzB79mz2HOMSx8yZMzF27Fg4OTmBx+Ph77//xquvvsomUzMaAv2RmpoavPLKK0hKSjI5bjAYbHZ8X80NaaAAYKsMUxvRyHV1daiuroaDgwM8PDzg6OjYqV0OarUa+fn5EIvFyMnJAZ/PR3Z2dp+qaGlcqzZa7XZ2dpDJZOwMtra21mZiTrqCm5sbnJ2dUV1d3am1V3t7e3h4eJjNqsPDwxEeHs4aJ2q1GmVlZcjJyUF2drbNZlwFWn6EgoKC2AzARowGc0VFhVlOIFuEYRh2Zs8wDDw9PSGRSODp6YmwsDCz8419FhYWhrCwsDb/biuzqkqlQllZGTQaDeRyOTumbbGvjc8gImKN0KsznRo9Ka1n5TweDz4+PiAiVFRUoKKiAkDLs9IWn2HGWJLKykpUVVXBzc3NJPtpTU0NcnJy2HMVCgUUCgXblp7O9todNDY2Ijc3Fy4uLvDy8jJ7FhnvS+Mz22hga7ValJWVobCw0Cb7rtN026JWL9IdmWRXrVpFSUlJVFlZafE7vvjiCxo4cCCtXr3arLZDeyQkJFBAQACJxWKKiIigyMjIPpcbxM3NjXbv3k1JSUmUlJREV65cofr6elIqlbR48WKKi4sjZ2dnq+tpSaRSKX333Xdsf2i1WnryySeJYRhatmwZJSUldbqmzG233Ubnz59nr4NRcnJyTO6JjIwMuvXWWykiIsLm+zoyMpLS0tLM7tvq6mp65JFHKCYmxmbiidoSHo9HwcHBNGDAABowYAANHTqUfvvtNzIYDFRYWGjWX0lJSVReXk5ERBUVFZSUlETFxcVkMBiourqakpOTqaCgoN01/suXL9PNN9/Mfmd4eLjN7r6TSCQUFRVFM2fOJLlcTnK5nGbOnMnqPnHiRLp06VKbbdVoNLRixQr2/Li4OHJzc7N6u64WhmHI39+f1XHgwIEmu3oMBgPt3LmTff1q8fPzs7r+nRUnJyeKjY2lF154gZqamsz6KikpiSZNmkT33HMPFRUVsceLioronnvuoaioKJJKpVZvhyXhYlDaQSqVwsXFBSqVyuI6XE5ODlJSUjBw4EAUFRV1eq27oqICer0earUaWVlZ3a12j8Ln8+Hm5gZfX1/ExMQgLi4O1dXVUKvVUCgUUCqVyMjIwJUrV6ytqhkMw8DV1RUuLi5mke7Ozs7w9/eHh4cHnJ2dzWaO9vb2FnMJhISEYMCAAR2u/6pUKmRmZvZK7ZeeQq/XIzc3F2lpadZWpUOMu02MiMVi5OXlobi4GADg4uICZ2dni7FHHh4e8PDwYP92dXWFq6srmpqa2s3jk5+fjytXrlh150pnUalUyMjIABFBLpeDYRikpaWxy8zu7u7Iz8+3GHsHtMSYZWZmIiUlpTfV7hL0vxwu9L+dRc3NzcjJyTF5lmdnZ9t0G9pDJpPBxcUFSqUSqamp8Pf3R2FhoVnukvz8fKSmpsLV1RUFBQXsikBhYSHS0tKQkZFhDfW7DYaotzKudB8KhcLiD0pXaF0UrTXl5eUoLCyEu7u7mTu8PZqbm5Gdnd1u0TlbxdPTE2+99RYGDhyI8PBw6PV6rFixAufPnwfQ8sOQl5dnM9tOr8bOzg5r167FuHHjEBISwq4/Gx/SFRUV+P777/HPP/+gqKjIJP5n7ty5WLhwIesyNuLq6oqQkJAOAyeTkpIwa9asPmGgREZGYv/+/YiOjjY5XllZiXvvvRfHjx+3kmbXjnFrubHP+Xw+lixZYhKP0BG///47/vvf/0Kr1Vp8vampCdnZ2W2+botIJBJ2ySMrK4sN5hUIBGypD0vQ/9Ix2Eq8iSWEQiFeffVVTJs2DRs3bsS+ffsQEBBgEpNUWlrKGq19jenTp2PFihX466+/8Prrr7NlV1o/i4z3pUAgQEREBLtEaUwWacuxNfX19R2GTdywHpTW2fHaoqqqqk/kKrkeeDwenJyc4OPjg0GDBmHw4MEAWnK/ZGRk4MKFC1bWsGN4PB68vLwQGBgIvV6PyspKODg4QCKRIDg4GIGBgdi/fz8787569hgWFobhw4d3aIgY0Wq1UCgUbN6E2tpa6PX6bm8TR+cgIuTm5rJxMzweD7m5uV3yduTl5eHChQt9ygDpCJVKheTkZLPjOp2u92qp9CDu7u4ICgpiDa3CwsJ+UwvMwcEBwcHBbABwXV0dLl682Ob5Wq3WLBC2P3DDGigc/4+3tzfWrVuH6OhohIaGWluda6K5uRlr167Fjh07ALTMsJYtW4Zp06YBaJllP/roo7j11lvN3uvn52fmPWmPtLQ0/Pvf/2Y9SU1NTWxAIYf1MRgM+Oijj/DLL790+j0VFRV9ImiSowWdTod3330X33zzTZ9bSu8Mf/31F+677z5UVVX1K6O5q3AGCgekUimGDRuGAQMGAPj/yswKhaLPPLT1er3JerNIJML999/P7tZhGIatYt0RGo2GrblhieLiYpw6dapPetYMBgMaGhrMdjEpFIp+5QXq0zsXODqEiJCRkdHnYyzaory8vE9sA+5pOAOFw4z6+nr897//RWJiYp8NMtNqtdi+fTt+/PFHAC0GytNPP41Zs2Z1+N4jR47g3XffbdM4q62ttamicF2hpKQEixcvNgu202g0fSJAloOD48aBM1A4YDAYoFKp0NzcDLFYDLVajTNnzuD06dPWVu2aISIkJyeza/AMw2Dy5Mmdyt6bl5eHgwcP9hnvUVdobm7GiRMnrK0GBwcHR4dwBgoHKioqsHz5coSGhmLZsmWdKsPd1yAifPnll50yuuRyeb9a7uDg4ODoi3AGCgeamprw119/ISMjA4899hiEQmGb8Rd9maSkpH4Z6c7BwcHRH+EMFA6WmpoavPHGG5BKpWyKaA4ODg4ODmtwwyZq4+Dg4ODg4LAOnUnU1rnMVF2kuLgY8+bNg5ubG2QyGYYMGYKEhAT2dSLCa6+9Bl9fX0ilUtx88802mT6dg4ODg4ODwzp0u4FSW1uLsWPHQigU4vfff0dqaio2b94MZ2dn9pxNmzZhy5Yt2L59O86fPw9vb2/ccsstfbJCLgcHBwcHB0cP0GE5wS6ybNkyGjduXJuvGwwG8vb2pg0bNrDHVCoVOTk50c6dOzv1HddbzZgTTjjhhBNOOLGedKaacbd7UA4cOIDhw4fjX//6Fzw9PREfH4+PPvqIfT0vLw9lZWVsCnKgpRrpxIkTcerUKYufaayme7VwcHBwcHBw9F+63UDJzc3F+++/j4iICBw8eBDPPPMMFi5ciN27dwMAW0X26qqTxr+vrjB7NevXr4eTkxMrAQEB3a02BwcHBwcHhw3R7QaKwWDA0KFDsW7dOsTHx+Ppp5/Gk08+iffff9/kvNbF2YiozYJtK1asQH19PSv9pWIlBwcHBwcHh2W63UDx8fFBbGysybGYmBgUFBQAaKmcC8DMW1JRUWHmVTEiFovh6OhoIhwcHBwcHBz9l243UMaOHWtWYTIzMxNBQUEAgJCQEHh7e+Pw4cPs6xqNBkePHsVNN93U3epwcHBwcHBw9EU6tW2mC5w7d44EAgGtXbuWsrKy6KuvviKZTEZffvkle86GDRvIycmJfvzxR0pOTqYHHniAfHx8SKFQdOo7uF08nHDCCSeccNJ3pTO7eLrdQCEi+vnnn2nAgAEkFospOjqaPvzwQ5PXDQYDrV69mry9vUksFtOECRMoOTm505/PGSiccMIJJ5xw0nelMwYKl+qeg4ODg4ODo1exWqp7Dg4ODg4ODo7rgatmzMFxnTg5OeGmm26CnZ2dyXGVSoVTp06hpqbGSppxcHBwXB+xsbGIjY1FRkYGkpOTe/W7OQOFg+M68ff3x+bNm9mdakbKysowZ84czkDh4ODos8yaNQurVq3Cli1bOAOlvyMSiTBs2DDY29vj4sWLqKurw5AhQ+Du7o6kpCSUlpZaW8VOwzAM4uLiEBgY2OG5BQUFSElJ6QWteg8nJycMGzYM0dHRcHFxARHhwoULaGxsBADU1NTcEGUZIiIiEBERwf6dn5+P1NRUK2rEwcFxrcTFxZlMtmJjYyGVSiEUCntfma7tz7EN+vIuHg8PDzp48CDl5+fTxIkTyd7enr755hsqLy+nu+++2+r6dUX4fD5t3bqV6uvrO5QtW7YQj8ezus7dKfHx8ZSWlkYNDQ2k1+spKyuLhg8fTg4ODuTg4ED29vb9rs2WZPny5SZ9vWnTphui3Zxw0t+Ex+PRm2++aTKeVSoVEbWkB+nO7+rMLh7Og9JLiMViDBgwAIGBgfDx8YGDgwMEAgH0ej2ys7Ph5OSE2tpaa6vZLjweDzExMfDw8AAA8Pl8hIaGdiqzb1hYGG6++WaUlJQgIyMD1Pc2j7E4OTlhwIABGDhwINzc3GBvbw+gpcxDY2MjGhoarKxh98EwDGJiYuDp6dnmOVFRUXBwcGBLVUgkkt5Sj6OXCA8Ph7+/P3Jzc1FQUICgoCCEhIS0eX5FRQXS09NhMBh6UUuOjuDz+YiLi4OTkxPS0tJQXV1tMr55PB7CwsIsPtMDAwMxadIks2d3eXk50tPTe+aZ3qOujh6iL3pQfHx86M8//6Tq6mrSaDRUXV1NU6ZMIQDk4OBArq6uJBKJrK5neyKRSOiTTz6hqqoqVpRKZaf6TKlUUlVVFW3fvp2EQqHV23I9Mnr0aEpPT6fa2lrS6XRsGzMyMig6Otrq+nWniEQi+uCDD0z6vLU0NTWZ9PW7777LeVD6kTAMQ2+88QZVVlbSokWLCAC99NJLVFlZ2eY98fHHH5NEIrG67pyYiqOjI3333Xckl8vpjjvusDi+23qmNzc3W+zrnTt3XtNvF+dBsQHEYjEiIyMRHBwMX19fuLq6mp3TV2bcDMPAwcEBbm5uXX6vRCKBRCIx2+nSFxEIBHBxcYGzszMAoLm5GRkZGcjOzoZSqbSuct3M9fS5ETc3N4SGhqKurg45OTncrNpGYRgG4eHhZs8ohmEQFhYGNzc3REZGYtSoUYiMjISbm5tZgdfKykrk5uairKysz3lJHR0dERERAYHA8s+iSqVCRkYGVCpVL2vWfRgMBhQXFyM/Px++vr4YNWoUgoOD2fFNRMjPz0d5ebnZe319fREQEGDW50YPck/AGSg9jJeXF7Zu3YrY2FiLxglH30cul2PBggXIy8vjduxYYMKECdi8eTP+/vtvvPDCC/3OiOsviMViLFmyBHfeeafJcYZhWJf/3LlzMWPGDDg4OFj8jH/++QevvPIKGhoaoFare1zn7mTw4MH44IMP2lyyzsvLw6OPPors7Oxe1qz7aGpqwuuvvw5nZ2f897//xZo1a9iJFtBiwHz88cf47LPPzN777LPPYuXKlb2nLDgDpccQi8UIDg5GWFgYAgIC2CrOWq0WcrkcpaWl7G4PW4fH4yEwMBCenp6dzuBbXl6OsrIyeHh4wMfHh7W6XV1dMWjQIFRWVqKwsLBPzLIcHR0RGBgIHq8lr2F4eLjJLEur1aKiogIVFRXWUrHb4fF4CAgIgIeHh8kDrDVEhPLycpMZl1qtxsCBA9m+jYqKgp+fH1xdXc1mX30dkUiE4OBg8Pl85Ofns8YXn89HUFAQZDIZCgoKbHI3l1FH4wxYIpEgODgYfn5+JucREUpKSpCTk8Meq6qqsviZlZWVcHd3h7OzMwICAqBQKFBQUGDTXjPj+I6JiYGfn1+bBkpTU1Ob3pW+AhGhpqYGKpUKUqkU3t7eKCwsRF1dHYAWAyUnJwclJSVm783OzkZiYiI8PDzg6+vbK2O5b19tGyYgIAA7duxAREQEvLy82OP19fVYsWIFzp0712d+0KRSKVatWoWpU6eyAbId8f3332Pz5s147LHHsHLlSvZmnjRpEgYOHIhff/0VS5Ys6ROzrBEjRmDr1q2QyWQAWh7k/b3UglgsxooVKzBt2rQO+3zv3r1455132L9nzZqFH374ge1ze3t762xR7AW8vLzwzjvvwNHREU8//TS7ld7e3h5r1qzB4MGD8dJLL+Gvv/6ysqbm2Nvb4/XXX8eYMWMAtBillvqaiLBr1y58+umnHX7mtGnTsGfPHvaH/O+//8aiRYvQ1NTUvcp3I8bx7ebm1i+WoLuCVqvFli1b8PPPPwNo6evq6mqL5/700084cuQIHnnkEfznP//hDJS+iFgshq+vLyIjIxEaGoqAgAAALTdCSUkJiouL2Uh4W4fH48HX1xdubm6QSCTQaDTtzoSICFVVVaipqUF2djby8vKQk5ODzMxMuLi4wNPTEw4ODnBwcICHhwfrkbBVHBwc4O3tjYiICISEhFjMFGtcz9VqtVbSsmdgGAaenp7sTg2jp6S+vt7s3NraWgiFQtTU1KCqqgparRbBwcFoaGhARUUF9Hp9n/CUtYdEIoGvry/7w6vRaFBcXAyhUAg/Pz94eHggMjISGo0GQMtOr9DQUAQHB7OGra3A5/Ph6+sLb29vhIaGIjQ01OR1IkJZWRnr9SEi5OTkIC8vz+yzXFxcTIwaZ2dnhISEQKVSobS0tE/0vcFggFarhUKhaNfTlZ+fz/ZvX8foFcvMzGyzb1tTX1+P+vp65OTkICMjAy4uLiaT756AM1C6mdDQUGzfvh1BQUHw9fVlj1dXV2PJkiVITExEUVGRFTXsPA4ODnj99dcxePBgvP3229i8eTPWr1+P2267rc33fPnll/jwww9ZK/zAgQM4c+YM5syZg1dffRV8Pr+31L9uxo0bh3Xr1sHNzQ1SqdTs9aysLLzwwgsoKCiwGFTWn9Dr9di+fTt++OEHs9fuv/9+7Nu3D5988gk2b97MHv/zzz+xZs0a3HbbbVi7dm1vqtvtREVF4d1332V/jHNycvD888+zr7u5ueGtt95iAyj5fD78/PxscmnD2dkZGzZswPDhw82Wc4CWydQ777yD/fv3s8faur9nzJiBpUuXsrNpJycnCAQCnDhxAitXrkRVVZXNB5WeP38ec+bM6XDCpNFo+syzuyNUKhXWr1+Pbdu2dTk56K+//ooLFy5g9uzZWL16dQ9p2AJnoHQzEokE4eHhZtlVdTod8vLykJWVZSXNrg2JRAKxWIySkhKkp6cjNzcXcrncJP8HEaG2thYKhQLZ2dlIT09n36/VatHc3NynZh52dnbszpPo6GizvB4qlQqVlZXIy8tDenp6nzZO+Hw+PDw8IBKJTI7LZDKTmT8RQaPRoLm5GbW1tSY7z+rr600yTTY0NLBGm/F8uVyOyspKm5tN29vbmwSvK5VKVFVVQSwWw8PDg/3hDQkJQXR0NJsvQiAQIDg4GHq9HiKRCEKh0GJeEON1YhgGbm5ukMlkqK6utuqSB8MwkEgkkEql4PF4MBgMqKqqYuNnNBoNsrKyTMZxW7i6uiI6Opr9cW9sbERBQQFyc3ORnp5u88YJ0KJzZmamtdXoVYgIxcXFXXqPo6Ojyc7FmpoayOXyNuORuoVOJbGwMWw5D0p8fDzJ5XIznQsLC2nYsGFW168rwufzKTw8nAYNGkSOjo7EMAwFBwfTiBEj6Oeff2bbptfr6c0336QhQ4aQl5eXyWfMnTuXLly4QHK5nAwGA/ueb775hqRSqdXbaEmmT59OZ8+epdzcXNLr9WZ9efHiRZo8eTJFRkb2+ZwuHh4etHv3bkpISDCRy5cvU21tLdtmg8FA+fn5dP78ebrvvvtMPsPb25vi4+PJ19eXAJC7uzsNGTKEFi5cSOfPn6cdO3bQiBEjKCQkxObyo9x999107tw5tt1vvfUWSSQSuummm+jYsWPs8fT0dNJoNOz1aG5upitXrlBKSgo1Nze3+axSKBQ0ffp0EolEtGHDBjp79izdddddVm2zQCCgiIgImjhxIp09e5YaGhpo4cKFFB8fT/Hx8TR48GBycXHp1Ge99NJLJmPkwIEDNGLECAoODiaGYazev5x0nzzyyCMmz4itW7fSsGHDKCgo6Jr6msuD0osIhUI4OzvD1dXVxFWo0+lQV1eHysrKPhenYMxyezX5+fkoLi5GXl4eysrKAABEhMzMTFy+fNnsM+zt7eHr69ungs/c3NwwZMgQ1qugUqlQX1/Pzv4LCgqQmJjYZjBZX4LH48Hd3R3+/v5wdnY286RcjbOzM4RCoVlflpWVsfcC0LLDo6qqCsOGDcPgwYMhl8vZmAZbW/Jwd3dHfHw8G1tSWloKb29vBAUFYfDgwW3u6BAKhfD09AQRdWpnB8MwcHFxgY+PT6/HpPD5fDg7O4NhGNTV1UGn0yErKwuVlZUoKCiAq6srUlNTcenSpQ4/y97e3qT/jduNm5ub0dDQgLy8PFy6dAk6na7H2tMRrXU0YtTxWjFeR2N/6/V61NfX97nnelexs7ODvb09wsLCEB8fD6VSCYVCAb1ej8uXL0Ov1/fYd3MGSjcRGRmJ9evXIzAw0CRorLy8HMuXL0daWprJNr2+jE6nw7vvvos9e/awx/Lz8y2ee+DAASQlJeHuu+/G4sWL+1QMipELFy7gtddeQ3NzM4AWt72lYNG+SE1NDZYtWwZ/f3+sW7cOQ4YMsXieXq/He++9h19++aVTAXVXM2HCBOzduxdHjx7FmjVrbNrtP3r0aOzZsweOjo7tGhKFhYVYvnw5DAYDNmzYgLCwsHY/V6PRYPPmzfj00097/Tng7e2NDRs2wM7ODsuXL2eXMxobG7F69WrY29t3eul57ty5ePjhh9m/jSkEDh48iC1btqCsrKxHf7A6w7x58/DQQw+ZHf/xxx/x9ttvX7OR7OPjg40bNyI4OBhAiyG+YsWKfl8Y8/7778djjz3Gxiv9+uuveOedd3qlrzkD5ToRCASwt7eHn58fRo4cyUY16/V6NDY2ory8HAkJCUhLS7Oypt0HESE7O7vdhEXGNW6VSoX09HSTGbatIhaLIZPJYGdnZ7KFrqamBmfPnu0zeWs6A4/Hg729Pfh8PoqKiqBUKlkDDGjp46amJjZ2SKfTITU1FadPn+7yd3l4eMDDwwNlZWU9tnPLzs6uXe9PW7Q2Qtzc3Eyy5mq1WjQ2NprFzlRWVrI/6pa2yhsMBjQ1NaG2thYajYb1MloDiUSCIUOGwMnJySTrp7FPO4NUKoVEIkFUVBTGjBnDjg+VSoXa2lrk5ubi1KlTNuEhCwoKwk033WR23JKH1xICgQB2dnZsbS1j30ulUsTHxyMmJgYAUFJS0mbCuv6Asc8jIyNx0003sX1eVlaG06dP90pfcwbKdRIXF4f//Oc/CAgIMEloVVRUhP/85z/Izs7uE1uKu5t77rkHjz76KPu3v7+/zW8rnjZtGp5//nn4+/v3SU9PV/Dw8MDrr7/ObjEViUTsgxdo+eF58803cfLkSQAtBoutGtlCoRCLFi3CzTff3OX3+vn5tXtfXrlyBa+//rrZ9lN3d3esWLECvr6+ZgHxwP9n7Dx//jyuXLnSZb1sCYZhMH/+fNxzzz0ICwszMd5/+eUXfPDBBygsLLQJ46Q7iI6OxurVq1FeXo7Vq1f3i6Xca2HevHm47777zLah9yacgXKN8Pl8NjfCxIkTzWqVNDU14fTp031u1053ERwcjClTppgl89FqtVCr1VCpVDa3o8PLywujRo0Cj8cz2WVxrboad0vweDyoVCqru76NOkmlUri5uWHMmDGIi4uDSqWCwWCAVCoFEUGlUkGhUODixYvXlGBMKBRCLBb3WlVjHo+HAQMGYOrUqRZfp//tQNJoNBCJRBCJROx9CKDdHTUlJSU4evSoWQmDmJgYvP7664iMjDQ5bjAYoFKpUFdXhwsXLuDo0aPX2brrx2AwoLm5ma2efi1ERkZiypQp0Gg0JnEcWVlZ+Ouvv2xuLF+NRqOBWq0283QJBAKIxWKzZ5SPjw8mTpyI/Px8iMVi8Hg8SCQSyGQydteTSqVCU1OTTYzpniIiIqLNMdVbcAbKNTJ48GC88sorCAwM7Nduvu7m2LFjeO+991BQUGBzW4///PNPzJs3z+yBVVZWdk1xE87Ozli5ciW8vb2xceNGNsuoNfHx8cGrr76KiIgIBAYGora2FuvWrWNjpYKDg/Hmm2/i/PnznQqatMS0adPw5JNPIigoyGY8UT/88AO++eYb3HvvvXj44Ydx8OBBfPzxxx3+sFZVVXUpsFKhUGD9+vVITEy0if4GWuLgli1bBoFA0GasWGfZu3cvvv/+e/bvnJwcmzZOgJaYic8++wx5eXkmXp7Ro0fjxRdfNFsadHNzMwmODggIwKpVqxAWFgZfX19UVVVh7dq1SEtL69N1efoCnIHSRXg8HoRCIfz9/XHHHXe0GeV/oyIQCMDn89lId51OB51Oxx4rLCzEL7/8YhOR71frCbQsy13LA5xhGAiFQjPDxsnJCRMmTEBoaCg+/vjj61X3ujDq6ObmhsmTJyMqKgoAUFxcjBMnTiAnJwfz58+Hu7s7Tp48eU2eE+P1jIyMxF133dWrxolWq4VKpYJAILC4qyY9PR0///wzIiMjoVKpkJmZiZ9//rnTP64Mw5j8kBln3sYspMbPUSgUOH78+DXF6vQUSqUSp06dAsMwHU4KhEKh2ZIXj8djPWspKSlsWvS+Qk5ODn7++WfweDyIxWL2eHBwMO68807Wo3Y1RAStVguxWAwXFxdMmDABQUFBAFp2eh05cgRJSUm92o7ewjiGWo8j47O8N5/dnIHSRYYNG4Znn30WwcHBFrOL3ujce++9mDFjBuLi4gAAhw4dwpdffokpU6Zg/vz51lWuFdOnT8e//vUv1rA4fPgwPvvssy7PCP39/fHyyy+zSbyMSCQSq67fXk1AQABefvllhIaGwsfHx+z1xsZGbNiwAc7OztccM3H77bfjgQceQGRkZK/GG2m1WuzcuROHDh3CE088gYkTJ7Z57m+//Ybi4mJkZmZ2qZ/DwsKwePFiNs7M0dERXl5eqKysxObNm1FYWAigJWDW1pZ1PTw88PLLL0Mmk2Hz5s1tGuFCoRALFizA6NGjTY4TEa5cuYLHHnusT/8oT5o0CfPnz2fvzcDAQIhEIly+fBnbt28385I6Ozvj1VdfhVarxaZNm9gg8ubm5n4bV8gwDObNm4dp06aZ7ej7/fffsWfPHmRmZvZavBFnoHQShmHY6p//+te/2H32BoMBer2efR34f0vT1l2fV8Pn88EwzHXnLxg0aBAeeOAB9kc/MzMT33//PZydnTF37lybCKTj8Xjg8XhmutbW1uLzzz/vsN+u7mugJWByxowZFjOJAm1Xfu1NnJ2dMWPGDHaLZGvUavV1F7SLiorC7NmzzTwnxjHSU+v1BoMBp06dwrlz5zBx4kSLOziM911aWlqXgn2Nfe3t7Y27776brUpORNDr9SgtLcVvv/1m04GwDg4OuOOOO+Dk5IRPP/0U+fn57Hi/GolEgvHjx2P27NkAWtpoMBig0+lw5swZfPPNN9ZQv8vo9XqzWb5AIEB0dDTmzJljdn8WFxfj+++/N9ulN2zYMCxbtgw1NTXYsGHDdS+P2TrG1YERI0Zgzpw5rHfQSEpKiklqid6AM1A6yahRo/DII48gPDzcxNWbmZmJnTt3wsvLC88++yzq6+uxfft25Obm9plqxR4eHnj++echkUjY+JDuYsqUKdixYwfKy8vxwgsvIDMz0+qBZbNmzcLtt9/eZs6PjggLC8Ozzz7Lxh65urrC3d29GzXsmxw+fBh1dXWYPHky5syZw/4Anjt3Dp999hlycnJ6NO5Ir9dj9+7dOHPmjNlrFy9evKbPjIyMxDPPPIPQ0FCT5dzS0lK89957yM7Otlia3pZxdHTEs88+a5a7RSAQYOjQoezfBoMBX3/9NY4cOYKzZ8/2tprXzL59+0xy9fj4+GDHjh2IiYnpUgVeuVyOlStXQqVS2cQkoyfh8Xh46KGHMH78eLa69Y8//oiDBw+y51jFe9ZhrlkbxBqp7ufOnUsqlYrVwWAwkE6noz///JNcXV1p9OjRVFRURBcuXKCAgACrpyXuioSFhVFycjIVFBTQ8OHDicfjEY/Hs5i+mGGYdtOVr1271iSlvfE6vffee8Tn863aTqPuGzZsMLmfjDpu3769zbYZrwmPx6MJEyZQeXl5p+/XyspKmjhxotXay+PxaMiQIZSXl2emW1FREY0cObLbvpPH49HChQtJp9Ox1/WLL74gkUhk9fv8WtoyefJkqqqqIoPBQHq9nnQ6Hel0OkpOTqaIiAir69gZCQ0NpaSkJMrPz6f4+Hjy8vKikydPtnm/GtuqVqvpySeftLr+1yuLFy8mrVbL9l1r2bdvH1vKw9q6dqcwDNPpNgkEAvr4449N7oElS5b0qH5cqvseJC0tDZ9//jmys7PR3NyM/Px8rF69mi2O1hdxdnbG4sWLWc9Pc3MzPv30U3ZNXSgUYu7cuRg4cCC++eYbnD9/vsPPPHLkCPbv34+kpCSrLu8wDIO7774bEyZMwNixY01e++eff3DgwAEkJiZaXN6Ji4vDww8/zAbY+fr6miS8skXCwsLw6KOPsnq6u7ubFMXrCaZMmYLp06dj8ODB4PF4OHPmDL799lukpaVZ3WvWVWJiYliPqUwmQ35+Pnbt2sXmQ6mpqUFlZaWVtewc1dXV2LhxI0QiEYqKijqMD9Lr9fjmm29w9uxZmwr2vVb++ecfLF68uE3viUwmw5o1a5CZmYnPPvuMLZrYl3F1dcUTTzwBOzs7fPLJJ303ZqbT00AbwhoelHnz5pl4UH777TdydHS0upXcHRIWFkYpKSlm17mqqoomT57MnieVSunbb78ltVpNDz30kEWLfd26dSYelM2bN9vEzIRhGNq6datJ+wwGAxkMBnrzzTfb1XH69OmkUCi6fJ8aP7+ioqLXPSgTJkygiooKM11aS1FREY0YMaJbvnPp0qUmff/RRx+RQCCwet9fi9x+++1UV1fHtuXUqVPk7e1tdb26Q9ryoBjvCZVKZXF891cxju9Dhw6Rs7Oz1fXpDgkKCqLLly932kMqFArpk08+Ye8DvV7PeVD6AiNGjMCMGTMwaNAgm8np0FVCQ0Px4IMPorKyEl999RUcHR0xb948ODk5AQBcXFzMdqAALTOLxx9/HOPGjcPevXtZK5zP5+Nf//oXoqOjzd4zceLELq3zWgsiwj///IM///wTZ8+eNfGcDB06FHfffTc704yIiDDZnthZGhoa8NVXXyEjI6PL9WuulfDwcHYnzdUF08rKyvDll1+aefcaGhpQVFTU5ufx+XzMnj0bAwYMwL59+5CQkMC+dtttt2H8+PHs3613f/RFYmNjcd999yE6OrrXEs31Fk5OTpg3bx6bA6c1Op0O3333HRITE685Bw6HdfD29sa8efOg1+vxxRdftHmeRCLBnDlzEBERYXKcx+Nh2LBhMBgM+OWXX3D69GkcP368p9XumC5PC22A3vSgPPnkk6TVak1mhkR9y4NiXEc/efIkeXl50dChQ6mgoMBkJt0WBoOB6uvr6c4772Q9KMbjlqQ1tupBMRgM9Prrr1s8d/78+aRWqzt1fdqjuLiYRo0a1avtnDZtGtXW1prpfPnyZQoKCury54nFYvrqq69Io9HQY489ZvLapk2b2u3/vuhBufvuu6mxsdGsLf3Bg+Lv708XLlxo835uamqi2bNnW13P3pb+4EEZNGgQ5eXlUXp6OkVFRbXpQXFycqI//vijzWe3Tqej5557rld0tooHRafT4bXXXsNXX32FsrIy+Pj4YP78+Xj11VfZGSkRYc2aNfjwww9RW1uLUaNG4b333mNzZ9gCI0aMwNSpUzFy5EjweLw+4RXoCD8/P7z44otwdHSEg4NDp9pkTFB1//33Y/jw4Wy9ls5ej1GjRmHlypVISEjAwYMHe33rNcMwuP322xEfH48RI0a0e+7QoUNx6623YtiwYRa3Yebm5uKnn36Ch4cH7r33XrOS7jU1Nfj222/Z2h0NDQ0oLi7u3gZ1EqPuxi2U2dnZ11yBmWEY8Hg83Hnnnew2W4ZhMHr0aJNrdPr0afz999/s3xcuXLCJbeWWMI7v1n0cFxcHoVCInJwc7Nu3j02PXlBQ0G+KRfaHZ1l3EBsbi7vuugsDBgy4pkKTvUlERARmzZoFoVBo8XVvb284OTlBr9fjmWeegU6ng6enJ+zs7DB//nw2Zb0xN1OfuQc6Px/sHG+88Qa5ubnRL7/8Qnl5efTdd9+Rvb29yex1w4YN5ODgQD/88AMlJyfT/fffTz4+Pp1e5+8ND8rVOxEs0Rc9KB15O9riWt939Xt37NhhlV08fD6fPvjgAzPdLXlQFixYwHrLLLXTOMMaN26cxV08GRkZFBMTY9W+NnpQjJw9e5Z8fX2v+fPEYjF9/fXXnfKYbdy40Sa8ZZ2Rq8e3JelL47uzYvSgtMWN5kG5//77qbm5me1zW/agzJgxgxQKRZv3a1elLfq9B+X06dOYOXMm7rzzTgAt6YT37NmDCxcuAACICFu3bsWqVatwzz33AAA+//xzeHl54euvv8bTTz/d3Sp1itGjR2PUqFHs3xMmTADDMKylmZaWhsOHD5skfLK1WjKtCQ4Oxu233464uDhIJJJrtpqv5X3nzp3DqVOnEB8fjwkTJlzT93YnxjYQEU6ePIkLFy5Y3KFwdZ+3JjAwEAsWLIC3t7dJFuGqqiocOHAAOTk5Vqt8Ghoaittuuw0DBw40iZeha/RYCQQC3HbbbYiOjkZUVFSb14SIcPbsWZw5cwbHjx+32eSExvF94cIFtkIzwzBISUnB33//jaioKNxyyy3Iy8vDH3/8geTkZJsf353FyckJM2bMQFhYGDw9PdHU1IRffvkFtbW1uPPOOxEQEGBtFa1KXl4efv/9d6SkpJgVFLQ2xvtyyJAhEIlE7DgsKirCr7/+yu44cnd3x/Tp09m4wq6i1+vx999/Izk5GYmJid2m/3XToQnTRdavX09BQUGUkZFBRC1r356envT1118TEVFOTg4BoIsXL5q8b8aMGfTwww9b/EyVSkX19fWsFBYWdrs199prr5FerzeRqy3NL774giQSCbu3vC/MFG+//Xaqra01a0tvsHbtWuLxeLR48WLS6/VW96AYMRgMtHz5cov9t2DBAtLpdG22yRjd3vp6JiYmUmhoqFXviRkzZlB9fb2ZbmfOnLkmD4pMJqMffvihw3vHYDDQa6+9ZvNjwji+161bR0CLB0Wv19OHH35IQqGQHnroIVKpVPTDDz+Qvb29Tbelq2LMg2Lsy9LSUho7diy5ubnRX3/9xfbljepBOXDggM3mQZkzZw41NTWZjcOjR4+Sp6cnO+7aynPUWTQaDT3++OO9eg2s4kFZtmwZ6uvrER0dDT6fD71ej7Vr1+KBBx4A0LKbAGgpbX81Xl5ekMvlFj9z/fr1WLNmTXerCqAlRmLQoEEYOnSoxdlzWloaTp48idOnT5sUBbNlQkJCcPPNNyM+Pp4tF26JmpoaHD58GESEadOmdZgnQ6fT4ciRI5DL5Rg/fjzCw8Nx8uRJi6nDz507B4PBYJPXa8SIEXjiiSfYv5OSkjqVKbP1/VFZWYlDhw4hMzMT9fX1Vm+rMYX/tSCRSDB16lS2To9IJEJISEibn0dEOHfuHBITE3HhwgWrt70jEhIS8PHHH0Or1eLJJ59ks2VGRUXhsccew4gRI9hderZ633YVFxcX3HLLLQgPD4ebmxuam5tx6NAh5ObmorS0FEqlEr/++isKCgowadIkeHh4WFvlXiEyMhLjx4/HyJEjwefz2ZT+ttrnbY1rg8EAd3d33HLLLXB3d8cvv/wCT09PTJs2ja0Z1dTUhMOHD5tlNRcKhZg0aRICAgJw9OhRZGZmIj093fauwTWbXG2wZ88e8vf3pz179lBSUhLt3r2bXF1d6bPPPiMiopMnTxIAKikpMXnfE088QbfeeqvFz+wpDwrDMLRx40bSaDRtzpw/+ugjEovF7WZPtTW59957qb6+vt0YGiKilJQUioqKosjISIt5UFrT3NxM999/P8lkMvriiy9Iq9XSggULSCAQmInxer300ks250HR6XSk0WhYMcZOdORBac358+cpMDDQ6hlygRYPSkNDg5mOnfWguLu7019//WVyXfR6fZttNxgMtHTpUpO+tmXh8XgkEAho2bJl7Hg3esSu/vuHH34gmUxmdX27Q2JjYyktLY2NqyooKKARI0aY3K98Pp9cXFzol19+uWE8KA8//DA1NTWxfb5//36yt7e3ul6WZM6cOaRUKs3G39GjR8nd3Z1GjBhBcrmcEhISKCwsjGJjYyk9PZ09r7i4mMaMGWP2fHZ0dKR9+/aRUqmkuXPnkkAg6HUPklU8KK+88gqWL1+OOXPmAAAGDhwIuVyO9evX45FHHmF3ARh3+BipqKgw86oYEYvF15SHoiOICImJifj2228xaNAgDBgwAKmpqbh8+TJ7zpkzZ6DVaju9G8FYmlun0+HYsWNoaGjodr07gmEYCASCDvO20P8KnhFRu5azVqvFyZMnkZeXh7y8PKjVapw8eRJEhIyMjHYLDKanp+Prr7/G+fPnbWJHh7H429XXJi4uDg8++CBGjBjRqXibiooKHD9+HKmpqWhoaLDpLKnu7u64++67kZubi2PHjoHH42HixIkmdWWAloJyvr6+be4SMBgMOHfuHHJycgCArXB7vcUle4u4uDgMHDgQQ4YMgUAgYPvZuEMpLy8PZ86cwdmzZ/tMm1rj7u6OCRMmsPlb7O3tcezYMTZ3TXV1NSorK03uV71eD6VSiX/++QcVFRV9N+NoJ4iIiMDw4cNx0003QSQS2XReq9DQUIwcORJjx461qKenpyfuvfdeeHp6wsHBAUSEu+66CwKBAA4ODmhsbMSxY8eQm5uL8vJys3tarVbjyJEjqK2tRW5uru3e8x2aMF3E1dWVduzYYXJs3bp1FBERQUQtMy9vb2/auHEj+7parSYnJyfauXNnp76jO3fxCAQCEovF9MYbb5DBYKBNmzaRRCIhsVhMYrG4y3kcBg8eTJmZmXTx4kUKDw+3itU9e/Zsampq6vA6JicnU3h4OFuLp73rfffdd5t4kozXrSPvAZ/Pv6br2F3S2oNiCa1WS0qlkjQaTadidU6cOEH+/v4kFAqt0iZL0pYHRa/Xk0qlonPnzlFQUBBFRERQYmIiKZVKE1GpVO16TDQaDT311FPsuOhM39uS/Pvf/yalUmkxpxER0VdffUUODg421addldGjR1N+fj7bp5cuXaKYmBi2v0QiUZuzZKFQ2Oc8xV2Vp556ihobG83GuS16UB555BFqaGho85lkHNfGfE3Gv43jOC8vj4YPH27TfW4VD8r06dOxdu1aBAYGIi4uDpcuXcKWLVvw2GOPAWiZsSxatAjr1q1DREQEIiIisG7dOshkMjz44IPdrU6H6HQ66HQ6dlYRGhqKW2+9FTk5OUhJSeny5ykUChw9ehQqlQpNTU3drW6XKC8vx4ULF+Ds7IwRI0aY7fV3dHTElClT2P8rlUqcPXvWLF9Gc3MzCgsLTSLcjdetLSIiIhATE4OcnByrlqKn/3nJ9u/fzx6LjY01yaQoEAggEPz/UCgoKMDly5fb9CqlpqZCoVCYlXS3Rerr63Hu3DmUlJRg7NixcHJygqura4dZUnU6HS5evIjS0lIALTPtnJwcm9vl0FkEAgHEYnGbHjKDwQC1Wt0n+rQtGIaBWCxm+9bV1RUTJkyAr68vzp07Z+bNFQqFGDFiBFuJW6/X49KlS32uOnNHhIWFIS4uDvHx8ZBKpSgqKjIZ3+fOnbM5D0JRUREOHTqEwMBAxMfHm3lReDyeyaqCse8bGxtx4sQJ5Obmoqqqqt2daH3iXu/QhOkiCoWCXnzxRQoMDCSJREKhoaG0atUqUqvV7DkGg4FWr15N3t7eJBaLacKECe3O4FvTE3lQ1qxZQwaDgdRqNTU0NNCmTZuuaU2Ox+ORTCYjqVRqtahwowfl999/J29vb7rjjjuourra7Drq9Xpqampio8QLCwtp7NixZGdnZyIymazLs+VXXnmFGhoaaP369VaPjheJRGxb7O3t6a233mr3/tq9ezc5OzubXQejSCQSq7bHkrTlQTl//jyFh4fTiBEjKDU1le3rjmhsbKQ5c+aYtLuvZYW9Wozjuy36asXlq2XMmDFUWlpqNr4TExMt5ucxxp40NDRQQ0MDVVRU0KxZs6zeju6WF154gRQKBalUqv9r78zjo6yu//95Zs1MErInM5M9YUnCEiEFDLIJqIUiKj8riwtV2goWlRaogMpiFbBQxApUapVKLSCWRasCArKI7FkgJITs+zLZZrJOZju/P/Kdp04mKySZmXDfr9d5vTLPc2dy7nPmPnPuec49l8xms834dsTxLBKJyNXVlX71q1+1mYPSHhkZGTRy5EiSy+UOHw2zSwTF3d0d27Ztw7Zt29ptw3Ec1q1bh3Xr1vX0v+82gwYNQmhoKCIiIgC0rF6wyJ1gNpvR2NjYkyp2m/Lycnz//fdITk6GVquFTqcDEUGj0eD69evtetUVFRUoLy/vVuQnOjoagYGByMjIsHp+XVRUhAsXLiAvL+9uu3PX6PV6qz6npaXhxIkT7ba/fv26w+eWtKaiogLff/89goKCMHz4cD6XxM3NDXFxcfDw8ICXlxfkcnmHn2MwGJCSkoLi4mIUFBTYPQp4p0RFRSEoKIh/bRnfubm5yMrKsml/8+ZNh8iR6kkEAgHkcjnkcrnVKhCJRILY2FgEBwdb7cwtEAisIonOTnh4OB89cXNzQ1FREdLT051ifFsi1Hl5eTh16lSXf4+KiopQUVFh99+gHqPLrpkD0VMRFI7j6K233qLq6mpqbGy0+h/btm2z+8z/TkUsFpOHhwe5uroS8L9KspcuXaLBgweTp6dnmzJgwIBuRUqEQiFt3bqVKisradGiRVbnpFIpeXp6kkwms/v1aC0uLi7tXgNPT0+nXMVhsfns2bOtduE1GAyk0Wj4VV2dodFoaPbs2eTh4eG0+RgCgYA2b95M1dXVvFjG93vvvUdeXl79wuatpXUExUJmZiYNHTqUb+fn50fHjh2jmpoa0uv1fLv+topn+fLlVF1dTQ0NDWQ2m+kf//gH+fj4OJWtxWJxh/eqtu7hjh45sQjbzbgTiAjl5eVITU1FSEgIQkJCUFRUhLy8vD7bfbY3MBgMVnkkGo0Gly9fRkFBASoqKqDRaHrk/xARSkpKkJaWZlNBtbm52WHzFXQ6HXQ6nb3V6FEsNi8sLMTFixcRGBiIqKgo6HQ63L59u8tVUevr61FYWHjH+/bYE47jEBkZCaVSiYiICHh5edm00el0Njs693dkMhlGjhzJ18bw9vaGSqXiXxuNRmRkZKC0tBSVlZX2U7SHCA0NRVBQEAYPHgxPT0+UlJQgMTERaWlpqKmpcapImcFg6LH7tVPSqQvjgPRkDoqbmxv5+/vTli1byGw20/vvv0/+/v4Ol9V9NyIWi8nX15e8vLx63Lt2d3cnf39/h4yU3IsikUjI19eXZs2aRZWVlZSQkEAjRowgf3//Lomvr6/T5mKIRCLatm0blZWVtbuKbePGjXbXs7ekvQiKwWCgyspKKisro7KyMlKr1VY5gbW1tfT000+Tn58fSaVSu/fjbuX111+nsrIyqqur4/cBCwgIIHd3d7vrxuR/wiIoXaC+vh719fXIzMxEYmIiMjMzbaruOTsGg6HXZkZ1dXV2qfXCaBu9Xo/KykoUFhYiOTkZxcXFKCkp6Rcz4/bgOA5hYWHw8/NDeHi4TT0lIkJpaSlKS0vttrt0X9DQ0ICUlBTU1NQgMjKSz1sQiUTw8fGxaW80GpGTk4OysjLk5+ejoqKir1XuUYKCguDv74/IyEir70BDQwPUarXjVUlldM6dRjHsSW+s4vHw8KDg4GDy8PCwu2fJhMndilQqpcDAQAoICHCqeiV32tcPPviA8vPz21zJZDab6S9/+Uu/H98SiYQCAwNpxowZVFxc3Ol9tKamhubOnUtBQUEOuZKlO8JxHK1du5by8/OtcrCIiDZv3uy0+YT9WVgEpRtotVqnfO7OYLRFc3Nzv44WtMbX1xfBwcEoKSlBQUEBFAoFvLy8oFarUVlZiaysLBQWFtpbzV5Fr9ejuLgYHh4euHXrVqe5CxqNBrm5uSgqKuobBXsJpVIJb29vREZGIiQkhD9eUVGBiooKFj1xYpiDwmAw+gVGoxEffPABjhw5gjfffBPz58/H7t278cknn9gkcfdnsrOz8Zvf/KbTJcMmk4kvxOesCIVCvPTSS3jqqaesNjskIhw4cADvv//+PZcU3Z9gDgqDwXBqiAhqtRq5ubnQarUwmUyoqKhAdnY2srKykJmZaW8V+5Tm5manXoXYXQQCAYRCIaqrq1FdXc0fz87Ovuds39/gyAljX7W1tfDw8LC3GgwGwwHgOA5BQUHw8fHBwoULMX78eOzZswcnT57sN0tnGe0TGBgIb29vm+NqtRrl5eV20IjRFbRarc2mpa1hERQGg+HUEBEKCwtRWloKvV4Pb29vVFVVISUlxd6qMfqA4uLieyrf6l6CRVAYDEa/wLLc2NfXF/n5+f2uXACD0Z/oSgSFOSgMBoPBYDD6lK44KIIOzzIYDAaDwWDYAeagMBgMBoPBcDiYg8JgMBgMBsPhYA4Kg8FgMBgMh4M5KAwGg8FgMBwO5qAwGAwGg8FwOJiDwmAwGAwGw+FglWQZDAaDwfgJAoEAHMcBaKlUbDab+eMA+NeM3oU5KAwGg8Fg/B+RkZH4zW9+A3d3dwAtpfR37doFgUCARYsWQSKR4MMPP2Tl9fsA5qAw7AbHcXDCQsYMBqMfo1Qq8fzzz8Pf3x8AcOPGDezfvx8ikQjz5s2DXC7HwYMHUVJSwu5fvQxzUBh2YcSIEZgzZw6ysrKwd+9eNDc321slBoPBsEGpVOK1116DQCCAv78/RCIRli5diqysLOzZswcFBQX2VrH/Qk6IVqslAEycWH75y19SQ0MD/fe//yU3Nze768OECRMmAGj8+PFUXl5u87tjNputpLi4mMaOHWt3fZ1VtFptp7/1LILCsAtpaWl45513kJubC71eb291GAwAQFRUFGbPng2xWNxhu4qKCuzfvx/V1dV9pBnD3lRXV2Pfvn2orKwEANTV1aGoqMjOWvVvmIPCsAupqalITU21txoMhhXR0dFYvXo15HJ5h+3S0tJw4sQJ5qDcQ1RVVWHHjh1IT0+3tyr3DMxBYTB6mAEDBuDxxx+Hq6srjhw5gtLSUnur1GWkUikeffRRhIWFddr27NmzuHr1KsaNG4dx48bhypUrOHfuXO8r2QvExMTgkUceQWxsLMRiMb/EtD18fX2xcOFCZGVl4ciRI/ysur8zYcIEjB07tt3zqampOH78eL9ahltRUYHDhw8jOzubOaR9TW/kiPQ2LAeFiSNLSEgIJSYmUlFRkdM9o/bw8KCjR4+SyWTqUIxGIy1fvpwA0Jo1a8hkMtGmTZvsrv+dytNPP01NTU18fkFnWNplZmbSsGHD7K5/X8k777zT4fdi9+7dJBaL7a7n3UjrHJTr169TRESE3fXqb8JyUPoAlUqFRx55BC4uLm2er62txdGjR+9Zzzs2Nhbjxo3rtF1RURG+++67frGap76+Hl988QXc3d1RXl5ub3U6RC6XY/r06fySSplMhpCQEL4gldFoxOnTp5GdnY1JkyYhOjoaAEBEiI+Px+LFizFmzBhwHIeRI0fipZde4pde3rp1C2fPnnXopZjR0dGYNGkSxo4dC5FI1G7kpKysDMeOHeOvl6VGRmeRFmdEIBBg8uTJGDJkiM25uLg4cBzXbr+joqLw4osvIiMjA6dOnYLJZOptdXuc0tJS7N69G5GRkfj5z38OAA79Hb4bhEIhHnzwQQwaNMjm3MWLF5GcnNz3Sv2UOw5j2BFHiqBMmjSJysvLyWAwtCkZGRk0dOhQu+tpL1m6dCnp9fp2r49Fjh8/Tl5eXnbXt6dEIBCQQCCwux6diVKppAsXLvB2MBqNVhGEpqYmevrpp0kqldInn3zCHzebzWQymchgMJDJZCIi4l9bZNeuXSQSiezex45k4cKFpNPpyGQydRg5uXjxIqlUKoqLi6P8/Hz+eFZWVr+LoIjFYtq9e3eb49Ri6/awfAf27t1LMpnM7n25UxEIBDR27FgqKiqi69evU3h4uN116g2RSqX0r3/9y8bOer2eli1b1qv/u1ciKOfOncPmzZuRkJCA0tJSHD58GI8//jh/noiwfv16/P3vf0dNTQ3Gjh2LHTt2YOjQoXyb5uZmLF++HPv27UNTUxOmTp2KnTt3IigoqLvq2I3AwEBMnDgRI0aMgFwuh0jU9qX08PDAY489hujoaJw+fRpVVVV9rGnfMHr0aMTExNgcHzNmTJsz09zcXPz4449QKpWYOHEiP2PvD7i6umLKlClwcXHB6dOnHTo/geM4CAQCGI1GnDlzBmVlZVbniQgqlQrz5s1DeXk5/vnPf7b7WcOGDUNcXBzS09Nx5coVXLp0yeFzEbKysvDvf/8bgwYNwrhx41BUVIQffvgBRqPRql12djYaGhocvj93g1AoxPjx4xEZGYlBgwa1e09ri9u3b+Py5cv89bl69arNNXQ0wsLC8MADD9j0s7y8HGfOnIFarcYXX3yB+vp61NXV2UnLnkUsFmPSpEkIDAwEAIhEIgwcONDmGhARRo8ejeeeew7Xr1/H9evX7aEuuh1B+fbbb+n111+ngwcPEgA6fPiw1flNmzaRu7s7HTx4kFJSUmjOnDmkVCqptraWb7No0SIKDAykEydOUGJiIj344IMUGxtLRqOxSzo4QgTlkUceoYqKCtLr9R3OvMxmM+n1esrOzqZRo0bZXe/eEI7jaPPmzdTc3GwjBoOhzetz4MABGjBgAD3++ONUW1tLJ06c6DcRlJCQELpy5Qrl5eXRmDFj7K5PR6JSqejSpUukVqtpypQpJBaLrcTd3Z0+++wzqq+vp4ULF9qc/6msWrWKzGYzbd++nVxcXEgoFNq9f52JQCAgsVhMCxYsIJ1OR4cPHyZPT0+bvlkiQSNHjuy3ERQXFxfau3cvNTc3d/lebGHXrl3k4uLCXy9nsP0vf/lL0mg0NvesU6dOka+vL3EcZ2X7/iAeHh709ddfW/W3LVubzWYyGAyk0+nojTfe6BVdeiWCMn36dEyfPr3Nc0SEbdu24fXXX8fs2bMBAJ9++ikCAgKwd+9evPjii9Bqtfj444/xr3/9C9OmTQMAfPbZZwgODsbJkyfxyCOPdFelPiE4OBhxcXEQCoUAgJEjR0Iul7dbL6G2thYXL17kn9VLJJJ+97ya4zjcd999iIyMRFRUFMRiMVJTU5Geno6oqCgMHToUWVlZuHHjhs0z3IsXL0Kn06GwsBBffvklMjMznb4eipubG+Lj46FQKPgZR01NjZ216hpSqRTjxo2Dq6srLl++DLVazZ+7evUqRCIRcnNzYTAYbN4bHR2NmJgYPj8lPDwcM2fORE5ODpKSkhz6+b3ZbIbZbEZubi4OHz6MhIQENDY2ttlPANBoNPj2228RERGB+Ph4uLm5Ydq0aQgODsbFixeh0Wj6tgPdQCaTIT4+Hl5eXlbHDQYDrly5Aq1WC5FIBIlEAqDl2iQnJyMnJ6fTz7527Rr0er1TRJhCQ0MxatQoxMfHQyaT8f0tLy/H5cuXcePGDej1ehBRu98DZ0MikeD+++9HaGgoAgMD+T63B8dxEIlEEAqF/G9ea8LCwjBy5EgUFRUhISGhd2zfLTe5FYB1BCU7O5sAUGJiolW7WbNm0XPPPUdERKdOnSIAVF1dbdVmxIgRtGbNmjb/j06nI61Wy0thYWGfe55z586lmpoaamhooIaGBtLpdB1GTtLS0mj48OEUExNDN2/epMLCQoqLi7O7B92TIhQK6a9//Ss1NDTwkaQ1a9aQTCajtWvXktlspvfff59cXV1JJpNZiUQiIaBlBiuTyUgqldq9P3crkZGRlJiYSOnp6TR69GhycXFx+DwUSwTFbDZTU1MTlZSU0NSpU63aSCQSkslk7c6KV69eTQ0NDdTc3MxHDBsaGmjHjh1OM/sUCoVW38v2hOM4cnFxoVGjRlFmZiaZTCZqbGykjIwMGjlypN370ZEEBgbS+fPn+XuYRcrKyujnP/85yWQyOnDgAH8PMxgMtGTJEpux25Y408qdZ555hjQajc09/OTJk6RUKvvFvai1+Pj40PHjx6mhoaFb0TGz2Uxr165t8zMXLFhAWq2WPvnkk07HTVvS56t4LM+vAwICrI4HBAQgPz+fbyORSGy8+ICAAJvn3xY2btyI9evX96SqnRIaGsrPCAFg1KhRcHV1RVVVFa5fv96pt1hQUICqqiqYzWacO3cObm5u0Gq1va12n0JEyMjIwNmzZ/nX6enpaGpqQnp6Oo4ePYqbN2+ioaGh3c8wm81oamrqK5V7BTc3N8TFxWHgwIHw8fEBEUGv10On09lbtS7DcRxcXFwgk8lsZkydRbbEYjFkMhkfISwuLsatW7dw+/Zth46e/BSTydSl7yERQafToaKiAmfOnEFRURHi4uIgk8kcNo9KLpdj1KhRCA8PR0BAQJtF6MaOHQupVAqFQgGz2YyUlBQUFhYiOzvb6cdna0QikVXkxILlXtQfVhJakEqlGDVqFEJCQqBUKiEWi5GYmIiamhrExsbC19cXKSkp7dZqIiJkZWW1ec5yHcPCwvDII4/w0abKykokJyf3SA5Srywzbv0og4g6fbzRUZtVq1bhD3/4A/+6trYWwcHBd69oB0yfPh0bNmzgbzpisRgikQhXrlzBiy++iMbGxg7fb7nhERFWrFgBjuM6fY+zYTab8Y9//AN79uzhj1l+lA8fPoyjR4/2q8HeHiEhIfjrX/+K8PBwyOXye7789fHjx7Fy5UrodDqnXGbaFYqLi/H73/8eERER2LdvHwYMGGBvldolICAAW7ZsQUxMTJvOiUwmw4oVK2AymSCTyWA0GrFz507s37+/3zkn9xpeXl54++238bOf/QxyuRwNDQ3YsGEDLl26hI8++gjTpk3DBx98gP/85z/tfkZn9/AHHngAcXFx/GTk1KlTeP7551FbW3vX+veog6JQKAC0REmUSiV/XK1W81EVhUIBvV6PmpoaqyiKWq1ut16GVCqFVCrtSVXbJTg4GBEREYiJiYGHhwfvoJSXl+PatWu4fv06ampquvXD21EEwdnR6XRtRgr0er3T55S0xsvLCzExMTYZ7xEREfDz84NQKERCQgJyc3NRX19vJy3vDpFIhBEjRqCxsRG3bt3qcNVZREQEgoODbarO6vV61NbWOkU+Qndxc3PD0KFD+bpHSqWy3RpI9kYul2PYsGGIjIyEv78/X7ulNSaTCTk5ObytjUYj8vLyeuQHxhEICgpCZGQk/3rIkCFW0a7KykrcunUL169fd/iVR11FKpVi2LBhCA0NhVKp5B1oIkJjYyM0Gg2Sk5MhlUo7tXVoaGiblaUHDBiAH374AUqlEkOGDEF1dTVu3bqFrKysnpuYdPlhVBsA1jkoZrOZFAoFvfvuu/yx5uZm8vDwoA8//JCIiDQaDYnFYvr888/5NiUlJSQQCOjYsWNd+r+9uYrnlVdeIbVaTXV1dVbPJw8cOEBKpZLc3d3t/jyRiX1k0qRJlJmZSRUVFVZSXV1NRqORsrKyKD4+nry8vJxiFQPwvxwUCyaTiTQaDeXn59P06dM7fO/atWupoqKCGhoarMbnX//6V4fPvblTGT58OCUlJfG2r6qqIoPB4JA5ZkOGDKHLly/zOrZHbW0tzZs3j3x8fHjpT3kYL730EqnVat5mtbW1Vvf2o0ePUmhoKA0YMIA4jrO7vj0hwcHBdObMGaqqqiK9Xs/3taamhh5++GECQO7u7uTj49Np/siKFSts7nkVFRX0t7/9jQICAujFF18knU5H33zzDQUHB3f5OvZKDkp9fb3VM6nc3FwkJyfD29sbISEhWLp0KTZs2IBBgwZh0KBB2LBhA+RyOebPnw+gpS7IwoULsWzZMvj4+MDb2xvLly/H8OHD+VU99kSj0SAnJwcKhQKurq788ebmZlRVVXUaFZDL5Rg0aJBNxEev1yMjI8MpHvOIRCIMGjQIMpkMWVlZ/WYmdbeIxWL4+PjY5E81NjYiJSUFWVlZKC0tdZqVO0DL9zItLQ1isRgDBw6EXC5HaWkpysvL2/2uhoaGIiAggM+5KS4uRkpKCpRKZa8/eu1r3N3drWqCDB48GAqFAr6+vgBaIoi3bt1CXl6ew0VKRSIRvLy84O3t3WE7IkJtbW2/rdEkk8ng4+PDR00qKipw69Yt/pFEamoq1Gp1v3qcJRAI4OnpydveYDAgKysLJSUlfC5kZ7VdgoODoVQqMWjQIPj4+PApGGVlZSgoKEBhYSEqKyuRl5eHq1ev4ubNm6ioqOjZ3LsuBCysOH36dJve0IIFC4jof1m/CoWCpFIpTZw4kVJSUqw+o6mpiZYsWULe3t4kk8lo5syZVFBQ0GUdejOC4ubmRiqVit58802rqon/+te/upSpPHToULp8+TIVFRVZydWrV2nEiBF296y7Ir6+vnTkyBFKSUmhBx54wO76OIpMmzbNZvUZUcuKrfvvv58CAgKcZtWKRQQCAfn5+dHw4cPp4sWLpNFoaN68eaRUKsnFxaXN9m+//TYVFRXxM9EtW7ZQYGAgvf3222Q2m/tVBCU+Pp5SUlL4cWypGm0hLy+PpkyZQgqFwuFWsgwdOpQyMjK6dD/9xS9+YXd9e0uWLVtmdS/fv38/hYaGkkqlIpVKRd7e3v0mcmKR0NBQSk5O5vtcWVlJs2fP7tYqpddff52KiopIq9VaRZw+/vhjCgoKIk9PTwJAMpnsjq5jr0RQJk+e3GFmPsdxWLduHdatW9duGxcXF3zwwQf44IMPuvvve536+nrU19e3u+JmwIABCA4OtsnYb2xsRH5+PiQSCRQKBV+pzwIRtVszxdEQCoXw8/ODUqnss9wfR8bd3R0hISEIDw+3WuFisXlaWhoKCwsdft+dtjCbzaioqIDBYMCtW7cAtIxhb29vNDQ0oLm5GUFBQfD09ATQMjOLiIiASqVCWVkZcnNzkZWVheLiYmRnZ+PGjRsoKSlx+NU7Pj4+UKlUnbaLjo626n9TUxMyMzP5FQtFRUUoKChodwWiMyAQCBAWFoahQ4eiqKjI6VcbKhQK+Pn5Wb3mOA5VVVUoKSlBRkYGiouL+02+SUcYDAbk5+ejuLgYeXl53dpZ3cPDw+p3TK1Wo6ysDJmZmSguLubHeFNTU69Fn9hmgd3k/vvvx1/+8hfIZDKr40lJSVi8eLGdtGL0JqNHj8a2bdvg6+tr9dgvKysLixYtclrn5KfU1tbi9ddfh6+vL9544w1+5dzZs2fxhz/8AY8++iiAFufFx8cHQEsRxo8++oh/pHXo0CGcOXMGtbW1Du+gzJw5E2+88UanqwtlMplVcmlOTg4WLVqEkpISAC0/AM7snAAtj6XfeOMNvPTSS1i+fDmOHj1qb5XuimeffRYvvvgi/9riXB49ehTr16+HVqu9J5wTAKiursaKFSuQlJR01/eow4cP489//jO0Wm2fjW/moLSDRqNBZmYmfwNraGhAZGQklEoliMgmS7k/rFgQiURQKpVQKBSoqKiAWCy2ykPgOA4BAQFwc3NDeXm51TNMX19feHl5obKy0qlyMLqCq6srwsLC+B+qpqYmlJSUIDMzEzk5OU7vnAAt39/S0lI0NjbCw8MDERERGDhwIEpKSjBw4EB+FQQRoaKiAhkZGcjKyrKqMqrVap1m9m3pY1drlzQ1NaG0tBSZmZnIzs7u1kzUXjQ3NyMvLw8ikQgqlardaKhAIIBCoYCnpycGDhzI72xrsbWz2NSCt7e31aodC7W1tcjJyen2vVoikUClUtlEwI1GI0pLSx2q3pFYLIZKpUJ4eDikUimMRiNKSkr4OmRdwc/PD15eXvDy8gIRobq6GlVVVTbjvS9gDko7fP3117h69Sr/euLEidi3bx8SExPx3HPP2SwzbmpqQk1NjVMnCfr6+mLr1q1QqVTYvHkz0tLSrGp6SCQS/PGPf8SUKVOwZs0afPXVVwBaHJeFCxfi2Wefxfvvv4+PPvrIXl3oE9LS0vDqq6+ipKSk3yYWuri4YNWqVXj55ZetHoWYzWb8/e9/x969e63K4fd3bt++jVdffRWFhYWoqKiwtzpdorCwEIsXL8bAgQOxfft2DBw4sMP2lvG9aNEiAC3Lj//0pz/hiy++6At1HZbQ0FBs377dZjPbiooKvPLKK7hx44adNLMlMDAQ27dvx+DBgxEcHNzt+5NAIMBvf/tbzJ8/H/7+/gCA/fv3Y8eOHXa51zEHpR2qq6tRXV3Nv46KioKrqysMBgNu375t88zNknuiUCja3bvAURGJRPD19UVoaCiGDBnCP7NtaGiAyWQCx3Hw9fWFp6cnBg8ejOjoaL4GBtDypR40aBBiYmL41Q39AblcDm9vb/j5+UEgEKCpqQlVVVXIzc3FrVu3rL4fzo5AIICvry/8/Pz4qqg/dbYtM6na2lpkZmby+Sr3CiaTCQ0NDXzxRWegubkZ2dnZfF0TS+VUgUAAPz8/m4iKQCCw+hE2Go2IjIy0+h5otVqHXdXn6emJAQMG2BTNq62thUaj6XS8uri4WK32sRAeHo6oqCiEhIRYHS8pKeHHiq+vLyQSCSorK+0aUZFIJPxu1B1heVTbOlVBKBTy93Lgf1E0e4135qB0kbNnz+Kpp56CVqtts0jbkCFDsGXLFgQHB1slaDkDCoUCW7duRVRUFCIjIyEWi7Fx40aUlJTgj3/8I9LS0rB69WpMmjQJYWFhEIlEePXVV/HMM88AaPmyt04K7g+MHz8ea9euhb+/P1xcXHD16lWsXLkSpaWlDnuTvlMGDBiAt99+G6NHj0ZERITNeZPJhO3bt+PLL7+8JyvlDhkyBJ988glu3ryJZcuWOVXeSWlpKV555RX+x8jb2xtbtmxBbGxsh+8TCoVYtGgRnnzySf7Yjh07sHv37l7V907gOA6//vWvMXfuXJt70Zdffon3338farW6w8c7sbGx2LRpk01BO5lMZrN9y09xd3fHn/70JwwZMgSrVq3CxYsX764zfYBUKsXKlSsxefJkq+Mcx9lEiuwJc1C6SOuISmtcXV0xfPhwqwq6QMuNXaPRoLKy0mETs0QiEQICAvi9GiwzLKBlVmGZIahUKuj1elRUVEAmkyEoKAgDBgzgb3zOMrPsDBcXF7i7uyMsLAyjRo3iVwAUFBQgOTnZ6Z7Jt4WlToKlvoePjw+GDRuG++67r832RISamhoUFxeD4zj4+/ujvr7eKer6tEVjYyPKy8ttZstSqRQeHh7Q6/XQarUQi8Xw8PCAm5sbRowYAY7joFQq+fPOUMpfr9dbzYD9/PxQWFjIV/62wHEcBgwYwFfG5TgOoaGhCAkJ4W3dXjVae+Lu7g5XV1cMHjwYcXFxNufLy8uRmJjY7v1JIpHAw8MDISEhuO+++/ikWgtGoxEajQZ1dXVWY0YoFMLLywsBAQEIDAyESqWyW1VhkUgEDw8P+Pj4WEXwLTpaHtdYkMvliI6ObvN6AS3j3WJze9b3YQ5KL1NWVoZVq1bxJYAdkbKyMixduhRhYWHYuHEj/Pz8sHr1aiQnJ/Ob/23YsAEffvih1fsEAgGWLVuGxx57zE6a9w5TpkzBihUroFQqIZFIcOHCBaxfvx6lpaVOW8K+Nf7+/ti4cSOflyAWixEVFdVue6FQiN/97nf8bJqI8Le//Q379u3rE317mm+++QZpaWk2q3gmTpyINWvWICkpCevWrcPgwYPx9ttvw8PDA0BLuH/Xrl24ffs2Vq1a5ZTRJI1Gg9WrV9s8CpFKpXjjjTfw4IMPWh0nIuzevRsHDhzoVrJlXyAQCLB48WLMmjUL4eHhd/QZo0ePxtq1a6FUKuHm5mZzPj8/HytXroRQKMSmTZv4su9eXl7YtGkTysvL8dlnnyEjIwO3b9++m+7cMaGhodi0aRPCw8OtIkheXl549913bYqyCYVCDB48uMPP3LNnD/bt24eCgoJe0bkrMAflLhGJRHB1dYW7u3ubKwJ0Oh2SkpJw8+ZNO2jXNSw6qtVqfnVCUlISrly5wrdp6xmkUCjEE088gZqaGshkMkilUri4uMDT07PdPXqcAX9/f4wbN45/Zl9dXY1Lly71G+cEaPkxGjlypE2Y32w2o7GxEUajEXK53CpvwVIdGmj50bIkSTsjpaWlba7E8fLyQlVVFYqKinDx4kU0NjaisrISAoEArq6ucHNzw+jRo+Hq6mrz/N5ZMBgMSElJsTnu4uKC/Px81NTUQC6XW+WolJWVITU11eGqrXIch8jISDzwwAM255qamqDT6drVWSwWQy6XIzg4GPHx8bxzYjQaUV9fD4FAADc3NzQ0NCAhIQEcx6G8vJx3VoGWjUJdXV2Rk5ODy5cv904nu4CLiwsGDRqEsLAwK7tJpdI2H+URERoaGtpdcUn/tzP9jz/+2Gs6dwXmoNwlQ4cOxZtvvong4GCb0KCzUVlZiZUrV0IqlXZpJmAymfDRRx/h6NGjWLRoEZ544gnMmzcP8fHx2Lt3L/75z3/2vtKMHqWxsRHvvPMObt68iZUrV7Z54+/PXLlyBc8++yw0Gg0aGxuRmpqKhQsXIjY2FmvXru20bLwzo9frsXXrVhw4cAC///3v8dBDDwFocQIWLFiAyZMnY/fu3di/f7+dNe0cIsKhQ4fwz3/+E/n5+W0+3hk3bhyWL1+OoKAgq0czt27dwltvvYWAgACrgqNlZWX4/e9/b1ULCWhx+NLS0nqtL10hPz8fS5YswcCBA/HWW291upq0ubkZW7Zs6dAByczM7Gk1uw1zUO4SLy8vTJ48mS9eZTKZoNPp+AHR2NjoNDVSmpubuz0LuHXrFtLT0zFjxgxwHMfPsn8afWE4DhzHwcXFBa6urlYRPyKCTqeDVqvFlStXcOnSJcyfPx+xsbGQSqVOUwX5blGr1VbLp7VaLa5evQqBQNDvdudujdlsRkpKCtLT0/HEE0+gvr4eUqkUIpEIYWFhUKlU+O677+ytZocQEfR6PQwGA9LT03Hy5Ml22yoUCkydOpWPhBmNRjQ3N6O0tBRnzpxBSEgIqqur0dTUxEcWf5oAaxlLHMfZPVrc2NiIxMREfk8lb29vuLi42KwotYzz2tpaJCQkdHh9HAHmoPQw+fn52LBhA18robGxEYWFhXbWisFoQalUYvXq1Rg8eLDVssna2lq8++67SE5ORkpKCpqbm/Hee+/h4MGDeOWVVzBx4kQ7am0/YmJi8NprryEkJMTpI6RdxWg04m9/+xuOHj2KF198EQ899BA+++wzfPXVV06xvPzzzz/HwYMHkZGR0a333bx5E5s3b0Z+fj5f1O3ll1/mFwa0xtPTE6tWrYJCocC7776L1NTUnupCtwkJCcHq1ashlUqxdetWSKVSrF692iYvp7GxEZs3b8a1a9eQkJBgJ227DnNQegiz2QyDwQC1Wo3vvvvOIZ0SjuMgFov5xEAigsFg6JHVNwaDATqdDiKRiM9yd3ZMJhMMBkOPXSNHwM3NDVOnTuUTYi3fgbq6Ovzwww84f/483/bq1au4ceMGZs2ahTFjxvArvIxGI4xGo1OsYLlbfH19MX36dD5Cei9AREhJScHt27cxY8YMAEB6ejqOHz/ukCsRjUYjf+8RCoVIS0vrVn6UZZwXFxfj22+/5VdvNTU14ezZs3xUpjVSqRRjx45FZGSk3YtTDhgwAA899BCam5uxYcMG6HQ6LFmyhD9vGef19fX48ccfHT5yYqF//JI4ANnZ2di2bRtyc3MdtrrowIEDsXTpUj7Jq6ioCO+9995dl2onIuzfvx/Jycl46qmnMGvWrJ5Q1+5cuXIFu3btQl5eXpu1b/oDGo0G7733HtLS0trMOzIYDNi1axdOnjyJF154ARMnTsS///1vnDhxAklJSXbQmNHbCIVC/PrXv8aECRMwevRocByHuXPn4r777sOBAwccKjnabDbj008/xYULF/DMM8/gkUce6fZnXL9+HTt27EBeXh4aGhowePBgvPrqq/wqp8LCQmzdutUmiqLRaPDOO+/A3d3dbqt3ukpdXR3ef/993Lhxw6EXbLSGOShdhOO4NivEWmaU5eXlOHjwoEPvy+Lr64vZs2fz9Q9SU1Oxe/duVFZWtjsbFgqFnW6oBrSs+klMTER0dDRmzJjhNHk3P8ViY0uf8/Pz8fnnn9v9+fKdwHEcBAKBje1EIhE4juP3k6qrq8OxY8estnX4KWazGRcuXMCVK1cwYcIExMfH49q1azhw4IBDRFAEAoFVLo3JZOrRaBcR8REjy/g3mUwwGo39JqrWGoFAgDFjxuCXv/wlPxbi4uIwatQopKamOpSDQkS4du0akpOTMXbs2A4dlNb3cMvfJSUlOHToEL9KT6FQ4P/9v//H1w65ceMG/v73v9s4KDqdziFycoRCIR+15jiOj2JbIkNAy15y3333nVWE1BlgDkoXGTNmDBYsWGDjpNTV1WH9+vUoLi52uuqiSqUS69evR1ZWFnbs2GGz7FIul2PRokUYMmRIh59DRPjiiy9w6tQpfPXVVygoKHDK2fXEiRMxb948REVFOd12Ba0ZOHAgFi1aZFPXwdPTEwEBAaiqqsLOnTtx+/Zt5OXldfp5JpMJn332Ga5evYqQkBDs2LEDBw8etOsNWiqV8itsfqrjhQsXeux/ZGRkYMWKFYiJicHixYuh1Wqxc+dOZGdn99u9iEwmE/bs2YPLly9j/vz5Dp1/JBAIeB3j4+M7bGsZ3xan3VI1e/jw4di6dSv/Y65QKByyIF1bhIWF4aWXXkJERAR8fHxgMpmwZs0aqNVqfPrpp/wS66amJoetw9Uh5IRotVoC0Kfy9NNPU2NjI5lMJis5efIkeXt797k+dyLx8fFUUlJCJpOJzGYzfz0zMzNp6NChVm05jiMfHx86deqUzfU3m81W18BgMNCrr75q9/7djXAcRy+99BIZjUa+n/v27SMXFxe763YnMmHCBCorK7OxtYXc3FwaMWJEl6+NRUQiEX3wwQdkNBpp2bJldu2jm5sbHTlyhO9Tc3MzPf/888RxXI99Jywybtw4Ki0tpYSEBAoODra7fXtbOI4jsVhMu3btshr3q1atsrtuPxWhUGij48qVK61sZ5HW47s9Wt/fkpOTKSwszO59bUvGjBlDRUVFNjpnZ2dTbGxsj42F3hCtVtupLVgEpYskJCTgtddesynGVlRU5DTlvvPy8rB27VoMGTIEL7zwAry8vNps5+7ujueffx7R0dFtbjplMpnw+eefWz0WOHfuXK/p3dtMnjwZs2bNQmxsbJceZzkDOTk5WLNmDaKiovDCCy9YFZfqDiKRCPPmzcOoUaMAAEQEtVqNZcuWOVy4WCgUYu7cuRg2bBgOHTp0V0WmoqKisGDBAr4+hjPNqu8WoVCIOXPmYMyYMbj//vvtrU63+cUvftHm3jmWrQo6IycnB7t37+Yf+VRWVjr8xqAlJSX4+OOPeT1FIhHmz5+PJ554Ap988oldq8HeFZ26MA6IPSIo/UlGjRpF+fn5ZDabyWw220RQAgIC6Pz58/z51qLT6ejZZ5+1ez96SpYtW0Ymk4n/fln66cwRFIuMHj2an2H9VHJycroUQZFKpbR3717+2hiNRvrd735n934BthGUn+q4ePHiu/rs6dOnk0ajsflO3AsRFIlEQp9++qnNdXXkCIrFPl2hvfuaRc6dO0d+fn5271tXxDK+k5KSKDQ0lD8eGhpKSUlJVFRURKNHj7a7nm0Ji6Aw2qSkpASbN2/ms9QFAgHmzJnDR4eICMeOHcO3337b5vtNJhOSk5P7St0+JzExEV9++SVSUlIccllldyguLsaf//xnDBw4EM888wyfp5GVleVUO/J2B47jMGvWLKhUKhw7dgw//vgjpkyZgilTpnT5MyIjI62qixYWFmLv3r3IysrqF5tFtoVIJMKTTz6J2NjYdjeNdDTMZjO+/PJLFBQUYMaMGRg3blyn77GM7/aSvAsLC+26QV53sIzvxsZGaDQa/rhGo8GOHTsgl8tRXFxsPwXvli65nA4Gi6D0rERGRlJKSgo/gygtLaX4+Hi769VX0jqC8sknn5BYLLa7Xj0pI0aMoNzcXEpPT6chQ4Z0+X3OGEEh+l8egSVPZv369Z3OnFvLT7lw4QIpFAq797k3RSaT0YEDB9qNRDhiBMUiHMfRtm3buvT70R/HtzMKi6DcBT/72c8wbdo0XL9+HceOHeu3SwoBoKamBv/4xz/4ZXX19fXO7XV3kQceeAATJ07E+PHjwXEcEhIScOLECVy7ds0hltDeDcHBwZg9ezbkcjmAlhwKDw+PLq88EYvFmDVrFmJiYhATE9ObqvYKllyDKVOmQCKR8HsKdTfHKD8/H4cOHUJGRobDbRYZGRmJxx57zGpzOKBlz5j//Oc/NjvYuru748knn4S7uzsOHjzIj3GJRILHHnsM0dHRiIqK4q+R2WzG8ePHrVbk9eQKqZ6EiHDixIkuRT4SExP73fi2UFVVhf/85z8OnzPTZbrkcjoYfRFBeeWVV8hoNNKuXbtIKBTa3dtk0vPyxhtvWM2Wd+zY0W9sPWHCBCovL7eJCnQ1giKXy+nQoUM20QRniaBYaC8i0lW+//578vHxsXtf25IZM2aQRqOxsfG1a9coKCjIpn1QUBBdu3bNJkLq7u5OX331lc11MhgM9Nvf/tbu/WRiK3c7vh1BWATlDoiLi+Nn1r2xokMmk+EXv/gFvL29cfToUYcsiX+vkZSUhPPnz+OHH35wygJz7cFxnM132MvLC88++2ynkRSJRILIyEj+/SaTCadPn8bNmzdx/fr1XtO5O+j1evz3v/9tt44Lx3EYP348Ro4cicuXL3d7I0ygZUdXSy0JR6QtG/v7++OFF15AdnY2vv76awDAzJkzERkZCX9/f8jlcjz11FMYPXo0AMDFxQXh4eH9ZgXbvUJH4zsrKwtff/01Kisr7aRdD3FH0wo705sRlD/+8Y9kNBr5+hE9HUHx9/enc+fOUUVFBU2ZMsXuXuy9LJYIypYtW0goFDp0zYDuyoQJE0itVtuMHUtuhtFo7FR+OpvW6/W0cOFCEggEdu/bT4XjOBIIBG2KUCikLVu2kNlsprVr17bbriNx5O/EjBkz2pyFWmx848YNioyMpIiICLpx4wZ/T2vrO9BWhIlFUBxXOhvf2dnZXa5zZC9hEZRuMGrUKIwcORJjxoxps0R4T9HU1IRvvvkGiYmJNpVbGX1LUlISPv74Y1y+fNnpn0m3pqysDJ999plN7Q4PDw889NBD/M68DQ0NOHnyJOrq6jB16lQolcp2P5OIHC7CREQd5oddvnwZH3/8MRISEhxO97ulsLAQe/bs4VcbBQQEYOrUqZDL5eA4Dt7e3njqqadARPD29raq4dTW/c1oNOLs2bPIzc0F0JKDkp6e3jedYXQLy/iOiIjAtGnT4OrqCuB/UZXW9bqclk5dGAekNyIoa9asIb1ebzOb6I0cFKFQSCKRyKFnZ/eCCAQCEolEDhcV6AmxVH1tLTExMZSens5/v4uLiyk+Pp4UCgWdO3eu3TGn1+vphRdesHu/mI3bt7GlerAFs9lMBoOBDAZDl3JwGhsbac6cOVafye5RjikW28fHx1NxcbGNLbtTKdpewiIo3SAtLQ0HDx5EdHQ0RowYgYyMDCQlJeHatWs9voKnv83WnRWz2dzvZtUW6P82uWtNTU0NvvnmG35lRk1NDdRqNR9JaW/1lslkQk5OTq/q3BvcSzYuLy/HkSNH7rhqsF6vR15entPX/rkXsNherVbjyy+/tKkKXlFRYVUXxVnhqKd/ffuA2traOx6E7SESiSAUCrF8+XL86U9/ws6dO7FixQoYDAY2YBn9Bo7jIBaLrUL8er0eRASJRNLho02j0cicaweG4zhIJJK7+gyDwdBvHbr+SHs2JyIYDAaHLo+h1Wr5YqHtwSIo/4dlS/X09HT897//RXJyMnQ6nUMbmMHoLkQEvV7f5rn2jjOcAyJCc3OzvdVg9CH93eYsgtIKsVgMqVQKvV7PbtgMBoPBYPQCXYmgdDvV99y5c3j00UehUqnAcRyOHDnCnzMYDHjttdcwfPhwuLq6QqVS4bnnnkNJSYnVZzQ3N+Pll1+Gr68vXF1dMWvWLBQVFXVXlV7BYDCgvr6eOScMBoPBYNiRbjsoDQ0NiI2Nxfbt223ONTY2IjExEW+++SYSExP5EtGzZs2yard06VIcPnwY+/fvx/nz51FfX4+ZM2ey59sMBoPBYDBa6HSdTwcAoMOHD3fY5sqVKwSA8vPziYhIo9GQWCym/fv3822Ki4tJIBDQsWPHuvR/2WaBTJgwYcKEifNKV5YZ93o1F61WC47j+MJQCQkJMBgMePjhh/k2KpUKw4YNa3cjqubmZtTW1loJg8FgMBiM/kuvOig6nQ4rV67E/Pnz+WSYsrIySCQSm3XbAQEBKCsra/NzNm7cCA8PD16Cg4N7U20Gg8FgMBh2ptccFIPBgLlz58JsNmPnzp2dtieidmswrFq1Clqtlhe2wR6DwWAwGP2bXnFQDAYDnnrqKeTm5uLEiRNWS4kUCgX0ej1qamqs3qNWqxEQENDm50mlUgwYMICX1vuLMBgMBoPBcB6oCxVOetxBsTgnmZmZOHnyJHx8fKzOx8XFQSwW48SJE/yx0tJS3Lx5E+PGjevS/6irq+tRnRkMBoPBYPQdXfkd73Yl2fr6emRlZfGvc3NzkZycDG9vb6hUKjz55JNITEzE119/DZPJxOeVeHt7QyKRwMPDAwsXLsSyZcvg4+MDb29vLF++HMOHD8e0adO6pINKpUJaWhpiYmJQWFjYabEXRt9SW1uL4OBgZhsHg9nFcWG2cUyYXXoeIkJdXR1UKlWnbbtdSfbMmTN48MEHbY4vWLAA69atQ3h4eJvvO336NCZPngygJXl2xYoV2Lt3L5qamjB16lTs3LmzW8mvlmqyXalGx+hbmG0cE2YXx4XZxjFhdrEvTlnqHmBfHEeG2cYxYXZxXJhtHBNmF/vS63VQGAwGg8FgMLqL0zooUqkUa9euhVQqtbcqjFYw2zgmzC6OC7ONY8LsYl+c9hEPg8FgMBiM/ovTRlAYDAaDwWD0X5iDwmAwGAwGw+FgDgqDwWAwGAyHgzkoDAaDwWAwHA6ndVB27tyJ8PBwuLi4IC4uDj/88IO9VbqnWLduHTiOsxKFQsGfJyKsW7cOKpUKMpkMkydPRmpqqh017r+cO3cOjz76KFQqFTiOw5EjR6zOd8UWzc3NePnll+Hr6wtXV1fMmjULRUVFfdiL/kdndvnVr35lM4buv/9+qzbMLj3Pxo0bMXr0aLi7u8Pf3x+PP/44bt++bdWGjRnHwCkdlM8//xxLly7F66+/jqSkJEyYMAHTp09HQUGBvVW7pxg6dChKS0t5SUlJ4c/9+c9/xtatW7F9+3ZcvXoVCoUCDz30ENtHqRdoaGhAbGwstm/f3ub5rthi6dKlOHz4MPbv34/z58+jvr4eM2fOhMlk6qtu9Ds6swsA/PznP7caQ99++63VeWaXnufs2bP43e9+h0uXLuHEiRMwGo14+OGH0dDQwLdhY8ZBICdkzJgxtGjRIqtjUVFRtHLlSjtpdO+xdu1aio2NbfOc2WwmhUJBmzZt4o/pdDry8PCgDz/8sI80vDcBQIcPH+Zfd8UWGo2GxGIx7d+/n29TXFxMAoGAjh071me692da24WIaMGCBfTYY4+1+x5ml75BrVYTADp79iwRsTHjSDhdBEWv1yMhIQEPP/yw1fGHH34YFy5csJNW9yaZmZlQqVQIDw/H3LlzkZOTA6BlA8mysjIrG0mlUkyaNInZqI/pii0SEhJgMBis2qhUKgwbNozZq5c5c+YM/P39MXjwYPzmN7+BWq3mzzG79A1arRZAy4a2ABszjoTTOSiVlZUwmUwICAiwOh4QEMDvnMzofcaOHYs9e/bg+PHj+Oijj1BWVoZx48ahqqqKtwOzkf3pii3KysogkUjg5eXVbhtGzzN9+nT8+9//xvfff4+//OUvuHr1KqZMmYLm5mYAzC59ARHhD3/4A8aPH49hw4YBYGPGkRDZW4E7heM4q9dEZHOM0XtMnz6d/3v48OGIj49HZGQkPv30Uz7Rj9nIcbgTWzB79S5z5szh/x42bBh+9rOfITQ0FN988w1mz57d7vuYXXqOJUuW4MaNGzh//rzNOTZm7I/TRVB8fX0hFAptvFS1Wm3j8TL6DldXVwwfPhyZmZn8ah5mI/vTFVsoFAro9XrU1NS024bR+yiVSoSGhiIzMxMAs0tv8/LLL+Orr77C6dOnERQUxB9nY8ZxcDoHRSKRIC4uDidOnLA6fuLECYwbN85OWjGam5tx69YtKJVKhIeHQ6FQWNlIr9fj7NmzzEZ9TFdsERcXB7FYbNWmtLQUN2/eZPbqQ6qqqlBYWAilUgmA2aW3ICIsWbIEhw4dwvfff4/w8HCr82zMOBB2S8+9C/bv309isZg+/vhjSktLo6VLl5Krqyvl5eXZW7V7hmXLltGZM2coJyeHLl26RDNnziR3d3feBps2bSIPDw86dOgQpaSk0Lx580ipVFJtba2dNe9/1NXVUVJSEiUlJREA2rp1KyUlJVF+fj4Rdc0WixYtoqCgIDp58iQlJibSlClTKDY2loxGo7265fR0ZJe6ujpatmwZXbhwgXJzc+n06dMUHx9PgYGBzC69zOLFi8nDw4POnDlDpaWlvDQ2NvJt2JhxDJzSQSEi2rFjB4WGhpJEIqFRo0bxS8QYfcOcOXNIqVSSWCwmlUpFs2fPptTUVP682WymtWvXkkKhIKlUShMnTqSUlBQ7atx/OX36NAGwkQULFhBR12zR1NRES5YsIW9vb5LJZDRz5kwqKCiwQ2/6Dx3ZpbGxkR5++GHy8/MjsVhMISEhtGDBAptrzuzS87RlEwC0e/duvg0bM44BR0TU11EbBoPBYDAYjI5wuhwUBoPBYDAY/R/moDAYDAaDwXA4mIPCYDAYDAbD4WAOCoPBYDAYDIeDOSgMBoPBYDAcDuagMBgMBoPBcDiYg8JgMBgMBsPhYA4Kg8FgMBgMh4M5KAwGg8FgMBwO5qAwGAwGg8FwOJiDwmAwGAwGw+FgDgqDwWAwGAyH4/8DkI+c93AtUwsAAAAASUVORK5CYII=", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], "source": [ "import numpy as np\n", "import torch\n", @@ -3914,9 +3521,7 @@ { "cell_type": "markdown", "id": "00df5c90", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Energy-based models and Langevin sampling\n", "\n", @@ -3928,9 +3533,7 @@ { "cell_type": "markdown", "id": "8d9b63c1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Tensor-flow examples\n", "\n", @@ -3942,9 +3545,7 @@ { "cell_type": "markdown", "id": "e94b6669", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Kullback-Leibler divergence\n", "\n", @@ -3959,9 +3560,7 @@ { "cell_type": "markdown", "id": "5f3aedf1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "D_{KL}(p \\| q) = \\int_x p(x) \\log \\frac{p(x)}{q(x)} dx.\n", @@ -3971,9 +3570,7 @@ { "cell_type": "markdown", "id": "02b822f4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "The KL-divegernce $D_{KL}$ achieves the minimum zero when $p(x) == q(x)$ everywhere." ] @@ -3981,9 +3578,7 @@ { "cell_type": "markdown", "id": "768c52c0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## VAEs\n", "\n", @@ -4000,9 +3595,7 @@ { "cell_type": "markdown", "id": "760347a3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(\\boldsymbol{x}) = \\int p(\\boldsymbol{x}, \\boldsymbol{h})d\\boldsymbol{h}\n", @@ -4012,9 +3605,7 @@ { "cell_type": "markdown", "id": "79d2eb57", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "or, we could also appeal to the chain rule of probability" ] @@ -4022,9 +3613,7 @@ { "cell_type": "markdown", "id": "0a0ed231", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(\\boldsymbol{x}) = \\frac{p(\\boldsymbol{x}, \\boldsymbol{h})}{p(\\boldsymbol{h}|\\boldsymbol{x})}\n", @@ -4034,9 +3623,7 @@ { "cell_type": "markdown", "id": "fa3584c7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We suppress here the dependence\ton the optimization parameters $\\boldsymbol{\\Theta}$." ] @@ -4044,9 +3631,7 @@ { "cell_type": "markdown", "id": "21d47f8b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Introducing the encoder function\n", "\n", @@ -4065,9 +3650,7 @@ { "cell_type": "markdown", "id": "db2fa210", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## ELBO\n", "\n", @@ -4077,9 +3660,7 @@ { "cell_type": "markdown", "id": "db635fe9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{align*}\n", @@ -4099,9 +3680,7 @@ { "cell_type": "markdown", "id": "3c265a28", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The VAE\n", "\n", @@ -4117,9 +3696,7 @@ { "cell_type": "markdown", "id": "d52cb01f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Dissecting the equations\n", "To make\n", @@ -4129,9 +3706,7 @@ { "cell_type": "markdown", "id": "be24ba23", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{align*}\n", @@ -4146,9 +3721,7 @@ { "cell_type": "markdown", "id": "091563ed", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Bottlenecking distribution\n", "\n", @@ -4164,9 +3737,7 @@ { "cell_type": "markdown", "id": "758e86fd", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Decoder and encoder\n", "The two terms in the last equation each have intuitive descriptions: the first\n", @@ -4184,9 +3755,7 @@ { "cell_type": "markdown", "id": "88073363", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Defining feature of VAEs\n", "\n", @@ -4196,9 +3765,7 @@ { "cell_type": "markdown", "id": "c1223aab", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{align*}\n", @@ -4211,9 +3778,7 @@ { "cell_type": "markdown", "id": "9ee0fee9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Analytical evaluation\n", "\n", @@ -4223,9 +3788,7 @@ { "cell_type": "markdown", "id": "6bf8c57e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{align*}\n", @@ -4237,9 +3800,7 @@ { "cell_type": "markdown", "id": "c358063b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where latents $\\{\\boldsymbol{h}^{(l)}\\}_{l=1}^L$ are sampled from $q_{\\boldsymbol{\\phi}}(\\boldsymbol{h}|\\boldsymbol{x})$, for every observation $\\boldsymbol{x}$ in the dataset." ] @@ -4247,9 +3808,7 @@ { "cell_type": "markdown", "id": "6cbc958c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Reparameterization trick\n", "\n", @@ -4264,9 +3823,7 @@ { "cell_type": "markdown", "id": "bcbd0cae", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Actual implementation\n", "\n", @@ -4281,9 +3838,7 @@ { "cell_type": "markdown", "id": "27cf3c73", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{align*}\n", @@ -4295,9 +3850,7 @@ { "cell_type": "markdown", "id": "c6e6c809", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Interpretation\n", "An arbitrary Gaussian distributions can be interpreted as\n", @@ -4313,9 +3866,7 @@ { "cell_type": "markdown", "id": "160f179e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Deterministic function\n", "\n", @@ -4325,9 +3876,7 @@ { "cell_type": "markdown", "id": "438a8fe4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{align*}\n", @@ -4339,9 +3888,7 @@ { "cell_type": "markdown", "id": "602f562a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where $\\odot$ represents an element-wise product. Under this\n", "reparameterized version of $\\boldsymbol{h}$, gradients can then be computed\n", @@ -4355,9 +3902,7 @@ { "cell_type": "markdown", "id": "12c01bc2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## After training\n", "\n", @@ -4374,9 +3919,7 @@ { "cell_type": "markdown", "id": "8468374e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Diffusion models, basics\n", "\n", @@ -4391,9 +3934,7 @@ { "cell_type": "markdown", "id": "1ef22e88", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Problems with probabilistic models\n", "\n", @@ -4416,9 +3957,7 @@ { "cell_type": "markdown", "id": "426a8f94", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Diffusion models\n", "Diffusion models have several interesting features\n", @@ -4434,9 +3973,7 @@ { "cell_type": "markdown", "id": "ec419261", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Original idea\n", "\n", @@ -4455,9 +3992,7 @@ { "cell_type": "markdown", "id": "9c189aa8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Diffusion learning\n", "\n", @@ -4473,9 +4008,7 @@ { "cell_type": "markdown", "id": "ffb28301", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Mathematics of diffusion models\n", "\n", @@ -4487,9 +4020,7 @@ { "cell_type": "markdown", "id": "bb277cc9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Chains of VAEs\n", "\n", @@ -4507,9 +4038,7 @@ { "cell_type": "markdown", "id": "f7181b35", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Mathematical representation\n", "\n", @@ -4520,9 +4049,7 @@ { "cell_type": "markdown", "id": "7a49e79b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{align*}\n", @@ -4535,9 +4062,7 @@ { "cell_type": "markdown", "id": "92a0b26e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Diffusion models for hierarchical VAE, from \n", "\n", @@ -4556,9 +4081,7 @@ { "cell_type": "markdown", "id": "f082e462", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Equation for the Markovian hierarchical VAE\n", "\n", @@ -4568,9 +4091,7 @@ { "cell_type": "markdown", "id": "a086a9e0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{align*}\n", @@ -4583,9 +4104,7 @@ { "cell_type": "markdown", "id": "8819e64d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We will modify this equation when we discuss what are normally called Variational Diffusion Models." ] @@ -4593,9 +4112,7 @@ { "cell_type": "markdown", "id": "39b41989", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Variational Diffusion Models\n", "\n", @@ -4613,9 +4130,7 @@ { "cell_type": "markdown", "id": "0073275c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{align*}\n", @@ -4627,9 +4142,7 @@ { "cell_type": "markdown", "id": "794774c3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Second assumption\n", "\n", @@ -4643,9 +4156,7 @@ { "cell_type": "markdown", "id": "76389ef8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Parameterizing Gaussian encoder\n", "\n", @@ -4665,9 +4176,7 @@ { "cell_type": "markdown", "id": "20cd18dc", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Encoder transitions\n", "\n", @@ -4677,9 +4186,7 @@ { "cell_type": "markdown", "id": "b7bdd9ed", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "\n", "
\n", @@ -4694,9 +4201,7 @@ { "cell_type": "markdown", "id": "c2db9314", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Third assumption\n", "\n", @@ -4710,9 +4215,7 @@ { "cell_type": "markdown", "id": "a77ec743", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{align*}\n", @@ -4726,9 +4229,7 @@ { "cell_type": "markdown", "id": "4781cb88", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Noisification\n", "\n", @@ -4741,9 +4242,7 @@ { "cell_type": "markdown", "id": "aa03ad89", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Diffusion models, from \n", "\n", @@ -4757,9 +4256,7 @@ { "cell_type": "markdown", "id": "a1b5ed7c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Gaussian modeling\n", "\n", @@ -4778,9 +4275,7 @@ { "cell_type": "markdown", "id": "02f43f35", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Optimizing the variational diffusion model" ] @@ -4788,9 +4283,7 @@ { "cell_type": "markdown", "id": "7fa0dc3b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{align*}\n", @@ -4810,9 +4303,7 @@ { "cell_type": "markdown", "id": "f78bdd91", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Continues" ] @@ -4820,9 +4311,7 @@ { "cell_type": "markdown", "id": "71d11858", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{align*}\n", @@ -4838,9 +4327,7 @@ { "cell_type": "markdown", "id": "187c698c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Interpretations\n", "\n", @@ -4854,9 +4341,7 @@ { "cell_type": "markdown", "id": "2a8aaad9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The last term\n", "\n", @@ -4866,9 +4351,7 @@ { "cell_type": "markdown", "id": "b6beab9a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Diffusion models, part 2, from \n", "\n", @@ -4882,9 +4365,7 @@ { "cell_type": "markdown", "id": "e9ee7f5e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Optimization cost\n", "\n", @@ -4905,9 +4386,7 @@ { "cell_type": "markdown", "id": "4ed42219", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## More details\n", "\n", @@ -4917,7 +4396,25 @@ ] } ], - "metadata": {}, + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.18" + } + }, "nbformat": 4, "nbformat_minor": 5 } diff --git a/doc/pub/catania/ipynb/data/MNIST/raw/t10k-images-idx3-ubyte b/doc/pub/catania/ipynb/data/MNIST/raw/t10k-images-idx3-ubyte new file mode 100644 index 00000000..1170b2ca Binary files /dev/null and b/doc/pub/catania/ipynb/data/MNIST/raw/t10k-images-idx3-ubyte differ diff --git a/doc/pub/catania/ipynb/data/MNIST/raw/t10k-images-idx3-ubyte.gz b/doc/pub/catania/ipynb/data/MNIST/raw/t10k-images-idx3-ubyte.gz new file mode 100644 index 00000000..5ace8ea9 Binary files /dev/null and b/doc/pub/catania/ipynb/data/MNIST/raw/t10k-images-idx3-ubyte.gz differ diff --git a/doc/pub/catania/ipynb/data/MNIST/raw/t10k-labels-idx1-ubyte b/doc/pub/catania/ipynb/data/MNIST/raw/t10k-labels-idx1-ubyte new file mode 100644 index 00000000..d1c3a970 Binary files /dev/null and b/doc/pub/catania/ipynb/data/MNIST/raw/t10k-labels-idx1-ubyte differ diff --git a/doc/pub/catania/ipynb/data/MNIST/raw/t10k-labels-idx1-ubyte.gz b/doc/pub/catania/ipynb/data/MNIST/raw/t10k-labels-idx1-ubyte.gz new file mode 100644 index 00000000..a7e14154 Binary files /dev/null and b/doc/pub/catania/ipynb/data/MNIST/raw/t10k-labels-idx1-ubyte.gz differ diff --git a/doc/pub/catania/ipynb/data/MNIST/raw/train-images-idx3-ubyte b/doc/pub/catania/ipynb/data/MNIST/raw/train-images-idx3-ubyte new file mode 100644 index 00000000..bbce2765 Binary files /dev/null and b/doc/pub/catania/ipynb/data/MNIST/raw/train-images-idx3-ubyte differ diff --git a/doc/pub/catania/ipynb/data/MNIST/raw/train-images-idx3-ubyte.gz b/doc/pub/catania/ipynb/data/MNIST/raw/train-images-idx3-ubyte.gz new file mode 100644 index 00000000..b50e4b6b Binary files /dev/null and b/doc/pub/catania/ipynb/data/MNIST/raw/train-images-idx3-ubyte.gz differ diff --git a/doc/pub/catania/ipynb/data/MNIST/raw/train-labels-idx1-ubyte b/doc/pub/catania/ipynb/data/MNIST/raw/train-labels-idx1-ubyte new file mode 100644 index 00000000..d6b4c5db Binary files /dev/null and b/doc/pub/catania/ipynb/data/MNIST/raw/train-labels-idx1-ubyte differ diff --git a/doc/pub/catania/ipynb/data/MNIST/raw/train-labels-idx1-ubyte.gz b/doc/pub/catania/ipynb/data/MNIST/raw/train-labels-idx1-ubyte.gz new file mode 100644 index 00000000..707a576b Binary files /dev/null and b/doc/pub/catania/ipynb/data/MNIST/raw/train-labels-idx1-ubyte.gz differ diff --git a/doc/pub/catania/ipynb/generate.png b/doc/pub/catania/ipynb/generate.png new file mode 100644 index 00000000..89d5eca0 Binary files /dev/null and b/doc/pub/catania/ipynb/generate.png differ diff --git a/doc/pub/catania/ipynb/real.png b/doc/pub/catania/ipynb/real.png new file mode 100644 index 00000000..ca8a683d Binary files /dev/null and b/doc/pub/catania/ipynb/real.png differ