Sreyam_pytorch_B.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Mode Connectivity of Neural Networks (Part 2)\n",
    "\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "In this part of the project we will implement a different path algorithm based on property of a well trained network - **dropout-stability**. Notably enough, the approach is constructive, i.e., we get a closed form expression for piecewise linear path between minimas found by SGD with *no optimization required*.\n",
    "\n",
    "<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "# importing all packages necessary\n",
    "\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim\n",
    "\n",
    "from torchvision import datasets, transforms\n",
    "from torch.optim.lr_scheduler import StepLR\n",
    "\n",
    "from copy import deepcopy\n",
    "\n",
    "import numpy as np\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SGD Solutions for Large Network Width are Dropout Stable\n",
    "\n",
    "In this section, we will train a couple of two-layer networks with SGD to see that when the number of neurons $N$ is sufficiently large the loss of full network and the network with half of connections (suitably rescaled) are close through the whole training proceedure.\n",
    "\n",
    "#### Setup\n",
    "\n",
    "In order to be aligned with theory we consider a slightly different version of the network and training algorithm. In particular, the two-layer mean-field network\n",
    "\n",
    "$$\n",
    "y_{{\\theta}_N}({x}) = \\frac{1}{N} \\sum_{i=1}^N a_i \\sigma({w}_i^T {x} + b_i),\n",
    "$$\n",
    "\n",
    "where $x \\in \\mathbb{R}^d$ is network input, $\\theta_N = \\{(a_i \\in \\mathbb{R}, w_i \\in \\mathbb{R}^d, b_i \\in \\mathbb{R})\\}_{i=1}^N \\in \\mathbb{R}^{d+2}$ are network parameters and $\\sigma: \\mathbb{R} \\rightarrow \\mathbb{R}$ stands for the activation function. The initial values for parameters are drawn from normal distribution with variance independent from $N$, e.g., from normal distribution with unit variance. This ensures that the network output concentrates to the mean value as $N \\rightarrow \\infty$.\n",
    "\n",
    "**Remark.** Notice $\\frac{1}{N}$ factor in network definition.\n",
    "\n",
    "For optimization algorithm we consider SGD with rescaled by a factor of $N$ gradient (in order to make theory work):\n",
    "\n",
    "$$\n",
    "\\theta^{k+1}_N = \\theta^{k}_N - \\alpha \\nabla_{\\theta}L(y^k,y_{{\\theta_N^k}}({x}^k)) \\cdot N,\n",
    "$$\n",
    "\n",
    "where $L$ defines a loss function and $(x^k, y^k) \\sim \\mathcal{D}$ is current training example and $\\mathcal{D}$ si data distribution.\n",
    "\n",
    "#### Dropout stability\n",
    "\n",
    "First we define the dropout network with corresponding dropout pattern $A \\in \\{1,\\cdots,N\\}$\n",
    "\n",
    "$$\n",
    "y_{{\\theta}_N, A}(x) = \\frac{1}{|A|} \\sum_{i\\in A} a_i \\sigma({w}_i^T {x} + b_i),\n",
    "$$\n",
    "\n",
    "where $|A|$ stands for cardinality of set $A$.\n",
    "\n",
    "The set of network parameters $\\theta_N$ is called dropout stable if there exists a dropout pattern $A$ such that\n",
    "\n",
    "$$\n",
    "|\\mathbb{E}_{(x,y) \\sim \\mathcal{D}} L(y,y_{{\\theta_{N},A}}({x})) - \\mathbb{E}_{(x,y) \\sim \\mathcal{D}} L(y,y_{{\\theta_{N}}}({x}))| \\leq \\varepsilon_D,\n",
    "$$\n",
    "\n",
    "for some small value of $\\varepsilon_D$.\n",
    "\n",
    "Without loss of generality we can assume that the dropout pattern is always $A = \\{1,\\cdots,N/2\\}$ and define the shorthand notation\n",
    "\n",
    "$$\n",
    "y_{{\\theta}_N, A}(x) = y_{{\\theta}_{N/2}}(x).\n",
    "$$\n",
    "\n",
    "It can be shown that during SGD\n",
    "\n",
    "$$\n",
    "|\\mathbb{E}_{(x,y) \\sim \\mathcal{D}} L(y,y_{{\\theta^k_{N},N/2}}({x})) - \\mathbb{E}_{(x,y) \\sim \\mathcal{D}} L(y,y_{{\\theta^k_{N}}}({x}))| \\rightarrow 0\n",
    "$$\n",
    "\n",
    "for squared risk, i.e., $L(y,y') = (y-y')^2$ as long as $N \\rightarrow \\infty$ and $\\alpha \\rightarrow 0$. In particular, if $K$ is total number of iterations then $K\\alpha = \\textrm{const}$. For example, we can pick $\\alpha = \\frac{C_1}{N}$ and $K = CN$ for some constants $C_1$ and $C_2$.\n",
    "\n",
    "**Remark:** Aforementioned relation between number of iterations, learning rate and width of network is crucial. It should be preserved later when we train two networks of different sizes.\n",
    "\n",
    "In a nutshell, according to aforementioned we expect the difference between the *dropout* network and full one to vanish as width $N$ grows larger with step size $\\alpha \\rightarrow 0$.\n",
    "\n",
    "#### Multiple outputs \n",
    "\n",
    "We defined dropout network for the case when the number of outputs $M=1$. Now consider $M>1$. In this case, we have\n",
    "\n",
    "$$\n",
    "y^m_{{\\theta}_N}({x}) = \\frac{1}{N} \\sum_{i=1}^N a^m_i \\sigma({w}_i^T {x} + b_i), \\quad m \\in \\{1,\\cdots,M\\},\n",
    "$$\n",
    "\n",
    "thus, the dropout network will correspond to dropping the half of $a_i^m$ for current $m$ and rescaling remaining ones, i.e.,\n",
    "\n",
    "$$\n",
    "y^m_{{\\theta}_{N/2}}(x) = \\frac{2}{N} \\sum_{i=1}^{N/2} a^m_i \\sigma({w}_i^T {x} + b_i).\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### MNIST Dataset\n",
    "\n",
    "First we download the dataset and normalize each image for optimization stability. ``train_loader`` loader and ``test_loader`` are iterators for corresponding parts of the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 431,
   "metadata": {},
   "outputs": [],
   "source": [
    "# convert image to tensor for PyTorch, and normalize with mean and std for stability\n",
    "transform = transforms.Compose([\n",
    "        transforms.ToTensor(),\n",
    "        transforms.Normalize((0.1307,), (0.3081,))\n",
    "        ])\n",
    "\n",
    "# downloading training and test parts of MNIST dataset\n",
    "dataset1 = datasets.MNIST('../data', train=True, download=True,\n",
    "                   transform=transform)\n",
    "dataset2 = datasets.MNIST('../data', train=False,\n",
    "                   transform=transform)\n",
    "\n",
    "# get train and test samplers with B=100 for train and B=1000 for test\n",
    "train_loader = torch.utils.data.DataLoader(dataset1, batch_size=100, shuffle=True)\n",
    "test_loader = torch.utils.data.DataLoader(dataset2, batch_size=1000, shuffle=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Network Definition\n",
    "\n",
    "Define two layer network described above with $N$ neurons in hidden layer and 10 outputs for MNIST classification task. For activation use Rectified Linear Unit (ReLU). Network gets as an input flattened to a vector of size $28\\times28$ image.\n",
    "\n",
    "**Remark:** ``nn.Linear`` initializes ``.weight from`` distribution with variance that vanishes with $N \\rightarrow \\infty$. Thus, to define network defined above you need to reinit the weights $w$ and $a$ from distribution with constant variance. **Use the values provided for your group.** **Keep init for ``.bias`` of the first layer as is.**\n",
    "\n",
    "\n",
    "To disable bias in last linear layer, i.e., to get only $a$'s use flag: ``bias=False`` in ``nn.Linear(...)``. Weight for each ``nn.Linear`` object can be accessed via ``object.weight``. Reinit of weight should be done in init method of class.\n",
    "\n",
    "**Do not forget ot rescale output of network by $\\frac{1}{N}$**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 494,
   "metadata": {},
   "outputs": [],
   "source": [
    "class FCN(nn.Module):\n",
    "    def __init__(self, N):\n",
    "        super(FCN, self).__init__()\n",
    "        self.N = N\n",
    "        # => define layers and reinit a, w using layer.weight.data.normal_() or other dist depending on your group\n",
    "        self.fc1 = nn.Linear(28 * 28, N)\n",
    "        self.fc1.weight.data.normal_(mean=0, std=1.2)\n",
    "        self.act = nn.ReLU()\n",
    "        self.fc2 = nn.Linear(N, 10,bias=False)\n",
    "        self.fc2.weight.data.normal_(mean=0, std=1.2)\n",
    "        coin = np.random.randint(0,2)\n",
    "        if coin==0:\n",
    "            self.fc2.bias=torch.nn.Parameter(torch.Tensor(np.random.uniform(low=0.5, high=1.51, size=10)))\n",
    "        if coin==1:\n",
    "            self.fc2.bias=torch.nn.Parameter(torch.Tensor(np.random.uniform(low=-1.5, high=-0.51, size=10)))\n",
    "    def forward(self, x):\n",
    "        # => Implement a forward pass\n",
    "        output = self.fc1(x)\n",
    "        output = self.act(output)\n",
    "        output = self.fc2(output)\n",
    "        return output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training with different $N$'s\n",
    "\n",
    "Below we provide a function to compute test loss and test classification error to estimate $\\mathbb{E}_{(x,y) \\sim \\mathcal{D}} L(y,y_{{\\theta^k_{N},N/2}}({x}))$ and $\\mathbb{E}_{(x,y) \\sim \\mathcal{D}} L(y,y_{{\\theta^k_{N}}}({x}))$ during training, i.e., metrics for full and dropout network. **You can modify the function to print additional info if you need.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 433,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test(model, test_loader):\n",
    "    model.eval()\n",
    "    test_loss = 0\n",
    "    correct = 0\n",
    "    with torch.no_grad():\n",
    "        for data, target in test_loader:\n",
    "            data = data.view(-1,28 * 28)\n",
    "            output = model(data)\n",
    "            test_loss += F.cross_entropy(output, target, reduction='sum').item()\n",
    "            pred = output.argmax(dim=1, keepdim=True)\n",
    "            correct += pred.eq(target.view_as(pred)).sum().item()\n",
    "    test_loss /= len(test_loader.dataset)\n",
    "    #print('Accuracy of the network on the 1000 test images: {} %'.format(100 * correct / len(test_loader.dataset)))\n",
    "    \n",
    "    return test_loss, 1. - 1. * correct / len(test_loader.dataset), 100 * correct / len(test_loader.dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Implement a function which takes full network and returns its dropout version, i.e., where $\\hat{a}_j^k = 0$ for $j \\geq N/2,\\ k \\in \\{1,\\cdots,10\\}$ and  $\\hat{a}_j^k = 2 a_j^k$ for $j < N/2,\\ k \\in \\{1,\\cdots,10\\}$. In other words, we set half of the final layer weights to zero and rescale the remaining ones by factor of two."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 434,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_dropout_network(model,N):\n",
    "    model_dropout = deepcopy(model)\n",
    "    # => Drop and rescale a's \n",
    "    model_dropout.fc2.weight.data[:,int(N/2):]=0\n",
    "    model_dropout.fc2.weight.data[:,0:int(N/2)]*=2\n",
    "    return model_dropout"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you need to train two networks with sizes $N=N_1$ and $N=N_2$. **Use $N_1$ and $N_2$ values provided for your group.**\n",
    "\n",
    "As it was mentioned before pick learning rate $\\alpha$ and total number of iterations $K$ such that $\\alpha=C_1/N$ and $K = C_2 N$. In particular, if for $N=800$ the value of learning rate is $\\alpha$ and number of iterations is equal to $K$, then for $N=3200$ the learning rate should be set to $\\alpha / 4$ and number of iterations to $4K$.\n",
    "\n",
    "Through training we would like to track the **test loss** and **test classification error** for both full network and dropout network. To evaluate the metrics use ``test(...)`` function defined above. Save corresponding values to plot later.\n",
    "Define a uniform grid in **log** scale w.r.t. the total number of iterations and evaluate metrics at each point $k$ of grid.\n",
    "\n",
    "**Remark:** Do not forget to rescale gradients by a factor of $N$ during training in accrodance with theory mentioned before.\n",
    "\n",
    "Implement a function below which trains a network of size $N$ and returns \n",
    "\n",
    "$$\\text{(list of test losses and errors of full network, list of test losses and errors of dropout network, final_model)}$$\n",
    "\n",
    "The final full model should achieve reasonable accuracy, i.e. $\\approx 97-98$%.\n",
    "\n",
    "**Note:** ``population_`` prefix stands for loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 450,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train(model, train_loader, optimizer, epoch,N,epochs):\n",
    "    model.train()\n",
    "    for batch_idx, (data, target) in enumerate(train_loader): # sample subset of dataset - (x,y)\n",
    "        data = data.view(-1,28 * 28) # reshape 2d image to vector of size 28x28\n",
    "        target = target\n",
    "        # => Implement a computation of gradient for current batch of data\n",
    "        optimizer.zero_grad()\n",
    "        output=model(data)/N\n",
    "        loss = F.cross_entropy(output, target)  \n",
    "        loss.backward()\n",
    "        for p in model.parameters():\n",
    "            p.grad *= N  # or whatever other operation\n",
    "        optimizer.step()\n",
    "        # => Implement progress tracker\n",
    "        if batch_idx % 60 == 0:\n",
    "            print ('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}' \n",
    "                   .format(epoch, epochs, batch_idx, len(train_loader), loss.item()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 451,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_curves_and_model(N,learning_rate,epochs):\n",
    "    population_curve, error_curve = [], [] # list to store test losses and errors of full network\n",
    "    population_curve_dropout, error_curve_dropout = [], [] # list to store test losses and errors of dropout network\n",
    "    model = FCN(N)\n",
    "    #learning_rate = 0.001\n",
    "    optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate, momentum=0.9)\n",
    "    # => Implementation\n",
    "    for epoch in range(1, epochs + 1):\n",
    "        train(model,train_loader,optimizer,epoch,N,epochs)\n",
    "        population_curve.append(test(model,test_loader)[0])\n",
    "        error_curve.append(test(model,test_loader)[1])\n",
    "        print(test(model,test_loader)[2])\n",
    "        model_dropout=get_dropout_network(model,N)\n",
    "        population_curve_dropout.append(test(model_dropout,test_loader)[0])\n",
    "        error_curve_dropout.append(test(model_dropout,test_loader)[1])\n",
    "        print(test(model_dropout,test_loader)[2])\n",
    "    return population_curve, error_curve, population_curve_dropout, error_curve_dropout, model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get curves and network for $N=N_1$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 452,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch [1/30], Step [0/600], Loss: 6.1347\n",
      "Epoch [1/30], Step [60/600], Loss: 1.2693\n",
      "Epoch [1/30], Step [120/600], Loss: 1.1422\n",
      "Epoch [1/30], Step [180/600], Loss: 0.6575\n",
      "Epoch [1/30], Step [240/600], Loss: 0.7487\n",
      "Epoch [1/30], Step [300/600], Loss: 0.5734\n",
      "Epoch [1/30], Step [360/600], Loss: 0.5189\n",
      "Epoch [1/30], Step [420/600], Loss: 0.7014\n",
      "Epoch [1/30], Step [480/600], Loss: 0.6638\n",
      "Epoch [1/30], Step [540/600], Loss: 0.5113\n",
      "84.41\n",
      "48.33\n",
      "Epoch [2/30], Step [0/600], Loss: 0.3630\n",
      "Epoch [2/30], Step [60/600], Loss: 0.5434\n",
      "Epoch [2/30], Step [120/600], Loss: 0.5596\n",
      "Epoch [2/30], Step [180/600], Loss: 0.4860\n",
      "Epoch [2/30], Step [240/600], Loss: 0.4094\n",
      "Epoch [2/30], Step [300/600], Loss: 0.4330\n",
      "Epoch [2/30], Step [360/600], Loss: 0.6272\n",
      "Epoch [2/30], Step [420/600], Loss: 0.3740\n",
      "Epoch [2/30], Step [480/600], Loss: 0.4499\n",
      "Epoch [2/30], Step [540/600], Loss: 0.2674\n",
      "87.94\n",
      "50.39\n",
      "Epoch [3/30], Step [0/600], Loss: 0.2066\n",
      "Epoch [3/30], Step [60/600], Loss: 0.4666\n",
      "Epoch [3/30], Step [120/600], Loss: 0.5583\n",
      "Epoch [3/30], Step [180/600], Loss: 0.3036\n",
      "Epoch [3/30], Step [240/600], Loss: 0.5499\n",
      "Epoch [3/30], Step [300/600], Loss: 0.3002\n",
      "Epoch [3/30], Step [360/600], Loss: 0.4007\n",
      "Epoch [3/30], Step [420/600], Loss: 0.5065\n",
      "Epoch [3/30], Step [480/600], Loss: 0.3389\n",
      "Epoch [3/30], Step [540/600], Loss: 0.3043\n",
      "89.36\n",
      "51.33\n",
      "Epoch [4/30], Step [0/600], Loss: 0.5094\n",
      "Epoch [4/30], Step [60/600], Loss: 0.4211\n",
      "Epoch [4/30], Step [120/600], Loss: 0.4837\n",
      "Epoch [4/30], Step [180/600], Loss: 0.3144\n",
      "Epoch [4/30], Step [240/600], Loss: 0.2811\n",
      "Epoch [4/30], Step [300/600], Loss: 0.4349\n",
      "Epoch [4/30], Step [360/600], Loss: 0.4752\n",
      "Epoch [4/30], Step [420/600], Loss: 0.2854\n",
      "Epoch [4/30], Step [480/600], Loss: 0.4920\n",
      "Epoch [4/30], Step [540/600], Loss: 0.3027\n",
      "90.58\n",
      "52.34\n",
      "Epoch [5/30], Step [0/600], Loss: 0.2484\n",
      "Epoch [5/30], Step [60/600], Loss: 0.5058\n",
      "Epoch [5/30], Step [120/600], Loss: 0.5129\n",
      "Epoch [5/30], Step [180/600], Loss: 0.2947\n",
      "Epoch [5/30], Step [240/600], Loss: 0.2492\n",
      "Epoch [5/30], Step [300/600], Loss: 0.2997\n",
      "Epoch [5/30], Step [360/600], Loss: 0.2481\n",
      "Epoch [5/30], Step [420/600], Loss: 0.2252\n",
      "Epoch [5/30], Step [480/600], Loss: 0.2351\n",
      "Epoch [5/30], Step [540/600], Loss: 0.3601\n",
      "91.05\n",
      "52.66\n",
      "Epoch [6/30], Step [0/600], Loss: 0.2256\n",
      "Epoch [6/30], Step [60/600], Loss: 0.1900\n",
      "Epoch [6/30], Step [120/600], Loss: 0.3085\n",
      "Epoch [6/30], Step [180/600], Loss: 0.4645\n",
      "Epoch [6/30], Step [240/600], Loss: 0.2806\n",
      "Epoch [6/30], Step [300/600], Loss: 0.2968\n",
      "Epoch [6/30], Step [360/600], Loss: 0.3479\n",
      "Epoch [6/30], Step [420/600], Loss: 0.3096\n",
      "Epoch [6/30], Step [480/600], Loss: 0.3026\n",
      "Epoch [6/30], Step [540/600], Loss: 0.1684\n",
      "91.59\n",
      "53.98\n",
      "Epoch [7/30], Step [0/600], Loss: 0.2779\n",
      "Epoch [7/30], Step [60/600], Loss: 0.2214\n",
      "Epoch [7/30], Step [120/600], Loss: 0.2083\n",
      "Epoch [7/30], Step [180/600], Loss: 0.2730\n",
      "Epoch [7/30], Step [240/600], Loss: 0.3493\n",
      "Epoch [7/30], Step [300/600], Loss: 0.3458\n",
      "Epoch [7/30], Step [360/600], Loss: 0.3995\n",
      "Epoch [7/30], Step [420/600], Loss: 0.3705\n",
      "Epoch [7/30], Step [480/600], Loss: 0.1990\n",
      "Epoch [7/30], Step [540/600], Loss: 0.2754\n",
      "91.72\n",
      "52.62\n",
      "Epoch [8/30], Step [0/600], Loss: 0.3118\n",
      "Epoch [8/30], Step [60/600], Loss: 0.2468\n",
      "Epoch [8/30], Step [120/600], Loss: 0.3701\n",
      "Epoch [8/30], Step [180/600], Loss: 0.3993\n",
      "Epoch [8/30], Step [240/600], Loss: 0.3549\n",
      "Epoch [8/30], Step [300/600], Loss: 0.1756\n",
      "Epoch [8/30], Step [360/600], Loss: 0.3766\n",
      "Epoch [8/30], Step [420/600], Loss: 0.2725\n",
      "Epoch [8/30], Step [480/600], Loss: 0.2232\n",
      "Epoch [8/30], Step [540/600], Loss: 0.2485\n",
      "92.31\n",
      "53.89\n",
      "Epoch [9/30], Step [0/600], Loss: 0.1815\n",
      "Epoch [9/30], Step [60/600], Loss: 0.3690\n",
      "Epoch [9/30], Step [120/600], Loss: 0.2328\n",
      "Epoch [9/30], Step [180/600], Loss: 0.1760\n",
      "Epoch [9/30], Step [240/600], Loss: 0.1720\n",
      "Epoch [9/30], Step [300/600], Loss: 0.2637\n",
      "Epoch [9/30], Step [360/600], Loss: 0.2891\n",
      "Epoch [9/30], Step [420/600], Loss: 0.2955\n",
      "Epoch [9/30], Step [480/600], Loss: 0.2821\n",
      "Epoch [9/30], Step [540/600], Loss: 0.2373\n",
      "92.74\n",
      "54.24\n",
      "Epoch [10/30], Step [0/600], Loss: 0.1578\n",
      "Epoch [10/30], Step [60/600], Loss: 0.3928\n",
      "Epoch [10/30], Step [120/600], Loss: 0.2444\n",
      "Epoch [10/30], Step [180/600], Loss: 0.1830\n",
      "Epoch [10/30], Step [240/600], Loss: 0.2190\n",
      "Epoch [10/30], Step [300/600], Loss: 0.1300\n",
      "Epoch [10/30], Step [360/600], Loss: 0.2836\n",
      "Epoch [10/30], Step [420/600], Loss: 0.2308\n",
      "Epoch [10/30], Step [480/600], Loss: 0.1809\n",
      "Epoch [10/30], Step [540/600], Loss: 0.2204\n",
      "92.95\n",
      "53.73\n",
      "Epoch [11/30], Step [0/600], Loss: 0.2706\n",
      "Epoch [11/30], Step [60/600], Loss: 0.2102\n",
      "Epoch [11/30], Step [120/600], Loss: 0.2386\n",
      "Epoch [11/30], Step [180/600], Loss: 0.2409\n",
      "Epoch [11/30], Step [240/600], Loss: 0.2543\n",
      "Epoch [11/30], Step [300/600], Loss: 0.2620\n",
      "Epoch [11/30], Step [360/600], Loss: 0.2040\n",
      "Epoch [11/30], Step [420/600], Loss: 0.3299\n",
      "Epoch [11/30], Step [480/600], Loss: 0.3043\n",
      "Epoch [11/30], Step [540/600], Loss: 0.2805\n",
      "93.14\n",
      "54.06\n",
      "Epoch [12/30], Step [0/600], Loss: 0.1132\n",
      "Epoch [12/30], Step [60/600], Loss: 0.3222\n",
      "Epoch [12/30], Step [120/600], Loss: 0.1379\n",
      "Epoch [12/30], Step [180/600], Loss: 0.2760\n",
      "Epoch [12/30], Step [240/600], Loss: 0.1020\n",
      "Epoch [12/30], Step [300/600], Loss: 0.1416\n",
      "Epoch [12/30], Step [360/600], Loss: 0.1448\n",
      "Epoch [12/30], Step [420/600], Loss: 0.2355\n",
      "Epoch [12/30], Step [480/600], Loss: 0.1761\n",
      "Epoch [12/30], Step [540/600], Loss: 0.2673\n",
      "93.34\n",
      "54.7\n",
      "Epoch [13/30], Step [0/600], Loss: 0.2790\n",
      "Epoch [13/30], Step [60/600], Loss: 0.2811\n",
      "Epoch [13/30], Step [120/600], Loss: 0.3518\n",
      "Epoch [13/30], Step [180/600], Loss: 0.1370\n",
      "Epoch [13/30], Step [240/600], Loss: 0.1395\n",
      "Epoch [13/30], Step [300/600], Loss: 0.2103\n",
      "Epoch [13/30], Step [360/600], Loss: 0.1741\n",
      "Epoch [13/30], Step [420/600], Loss: 0.1505\n",
      "Epoch [13/30], Step [480/600], Loss: 0.1253\n",
      "Epoch [13/30], Step [540/600], Loss: 0.2978\n",
      "93.59\n",
      "54.57\n",
      "Epoch [14/30], Step [0/600], Loss: 0.2091\n",
      "Epoch [14/30], Step [60/600], Loss: 0.2345\n",
      "Epoch [14/30], Step [120/600], Loss: 0.1402\n",
      "Epoch [14/30], Step [180/600], Loss: 0.2703\n",
      "Epoch [14/30], Step [240/600], Loss: 0.1626\n",
      "Epoch [14/30], Step [300/600], Loss: 0.2130\n",
      "Epoch [14/30], Step [360/600], Loss: 0.2126\n",
      "Epoch [14/30], Step [420/600], Loss: 0.3099\n",
      "Epoch [14/30], Step [480/600], Loss: 0.0867\n",
      "Epoch [14/30], Step [540/600], Loss: 0.2111\n",
      "93.69\n",
      "54.52\n",
      "Epoch [15/30], Step [0/600], Loss: 0.2642\n",
      "Epoch [15/30], Step [60/600], Loss: 0.2190\n",
      "Epoch [15/30], Step [120/600], Loss: 0.1992\n",
      "Epoch [15/30], Step [180/600], Loss: 0.1268\n",
      "Epoch [15/30], Step [240/600], Loss: 0.1676\n",
      "Epoch [15/30], Step [300/600], Loss: 0.2062\n",
      "Epoch [15/30], Step [360/600], Loss: 0.2987\n",
      "Epoch [15/30], Step [420/600], Loss: 0.2546\n",
      "Epoch [15/30], Step [480/600], Loss: 0.2866\n",
      "Epoch [15/30], Step [540/600], Loss: 0.3040\n",
      "93.76\n",
      "54.31\n",
      "Epoch [16/30], Step [0/600], Loss: 0.2973\n",
      "Epoch [16/30], Step [60/600], Loss: 0.2330\n",
      "Epoch [16/30], Step [120/600], Loss: 0.1000\n",
      "Epoch [16/30], Step [180/600], Loss: 0.2182\n",
      "Epoch [16/30], Step [240/600], Loss: 0.1198\n",
      "Epoch [16/30], Step [300/600], Loss: 0.2820\n",
      "Epoch [16/30], Step [360/600], Loss: 0.1873\n",
      "Epoch [16/30], Step [420/600], Loss: 0.2954\n",
      "Epoch [16/30], Step [480/600], Loss: 0.3070\n",
      "Epoch [16/30], Step [540/600], Loss: 0.1958\n",
      "93.92\n",
      "55.27\n",
      "Epoch [17/30], Step [0/600], Loss: 0.1277\n",
      "Epoch [17/30], Step [60/600], Loss: 0.1702\n",
      "Epoch [17/30], Step [120/600], Loss: 0.2340\n",
      "Epoch [17/30], Step [180/600], Loss: 0.1507\n",
      "Epoch [17/30], Step [240/600], Loss: 0.2190\n",
      "Epoch [17/30], Step [300/600], Loss: 0.0917\n",
      "Epoch [17/30], Step [360/600], Loss: 0.2139\n",
      "Epoch [17/30], Step [420/600], Loss: 0.1529\n",
      "Epoch [17/30], Step [480/600], Loss: 0.3412\n",
      "Epoch [17/30], Step [540/600], Loss: 0.4031\n",
      "93.97\n",
      "55.27\n",
      "Epoch [18/30], Step [0/600], Loss: 0.2461\n",
      "Epoch [18/30], Step [60/600], Loss: 0.2577\n",
      "Epoch [18/30], Step [120/600], Loss: 0.1326\n",
      "Epoch [18/30], Step [180/600], Loss: 0.1299\n",
      "Epoch [18/30], Step [240/600], Loss: 0.2050\n",
      "Epoch [18/30], Step [300/600], Loss: 0.3727\n",
      "Epoch [18/30], Step [360/600], Loss: 0.1616\n",
      "Epoch [18/30], Step [420/600], Loss: 0.0987\n",
      "Epoch [18/30], Step [480/600], Loss: 0.2348\n",
      "Epoch [18/30], Step [540/600], Loss: 0.1808\n",
      "94.22\n",
      "55.39\n",
      "Epoch [19/30], Step [0/600], Loss: 0.2818\n",
      "Epoch [19/30], Step [60/600], Loss: 0.4335\n",
      "Epoch [19/30], Step [120/600], Loss: 0.2814\n",
      "Epoch [19/30], Step [180/600], Loss: 0.1924\n",
      "Epoch [19/30], Step [240/600], Loss: 0.0476\n",
      "Epoch [19/30], Step [300/600], Loss: 0.2376\n",
      "Epoch [19/30], Step [360/600], Loss: 0.1555\n",
      "Epoch [19/30], Step [420/600], Loss: 0.2366\n",
      "Epoch [19/30], Step [480/600], Loss: 0.1941\n",
      "Epoch [19/30], Step [540/600], Loss: 0.1234\n",
      "94.35\n",
      "54.8\n",
      "Epoch [20/30], Step [0/600], Loss: 0.2869\n",
      "Epoch [20/30], Step [60/600], Loss: 0.1203\n",
      "Epoch [20/30], Step [120/600], Loss: 0.1178\n",
      "Epoch [20/30], Step [180/600], Loss: 0.2151\n",
      "Epoch [20/30], Step [240/600], Loss: 0.0686\n",
      "Epoch [20/30], Step [300/600], Loss: 0.1744\n",
      "Epoch [20/30], Step [360/600], Loss: 0.2873\n",
      "Epoch [20/30], Step [420/600], Loss: 0.2843\n",
      "Epoch [20/30], Step [480/600], Loss: 0.2065\n",
      "Epoch [20/30], Step [540/600], Loss: 0.1928\n",
      "94.46\n",
      "55.87\n",
      "Epoch [21/30], Step [0/600], Loss: 0.2442\n",
      "Epoch [21/30], Step [60/600], Loss: 0.0950\n",
      "Epoch [21/30], Step [120/600], Loss: 0.1225\n",
      "Epoch [21/30], Step [180/600], Loss: 0.1033\n",
      "Epoch [21/30], Step [240/600], Loss: 0.1233\n",
      "Epoch [21/30], Step [300/600], Loss: 0.2154\n",
      "Epoch [21/30], Step [360/600], Loss: 0.1604\n",
      "Epoch [21/30], Step [420/600], Loss: 0.3157\n",
      "Epoch [21/30], Step [480/600], Loss: 0.1160\n",
      "Epoch [21/30], Step [540/600], Loss: 0.2989\n",
      "94.54\n",
      "55.22\n",
      "Epoch [22/30], Step [0/600], Loss: 0.2117\n",
      "Epoch [22/30], Step [60/600], Loss: 0.2324\n",
      "Epoch [22/30], Step [120/600], Loss: 0.2062\n",
      "Epoch [22/30], Step [180/600], Loss: 0.2276\n",
      "Epoch [22/30], Step [240/600], Loss: 0.1655\n",
      "Epoch [22/30], Step [300/600], Loss: 0.2231\n",
      "Epoch [22/30], Step [360/600], Loss: 0.0791\n",
      "Epoch [22/30], Step [420/600], Loss: 0.1875\n",
      "Epoch [22/30], Step [480/600], Loss: 0.1454\n",
      "Epoch [22/30], Step [540/600], Loss: 0.1564\n",
      "94.62\n",
      "55.24\n",
      "Epoch [23/30], Step [0/600], Loss: 0.2836\n",
      "Epoch [23/30], Step [60/600], Loss: 0.1146\n",
      "Epoch [23/30], Step [120/600], Loss: 0.0902\n",
      "Epoch [23/30], Step [180/600], Loss: 0.2146\n",
      "Epoch [23/30], Step [240/600], Loss: 0.1661\n",
      "Epoch [23/30], Step [300/600], Loss: 0.2175\n",
      "Epoch [23/30], Step [360/600], Loss: 0.0883\n",
      "Epoch [23/30], Step [420/600], Loss: 0.1194\n",
      "Epoch [23/30], Step [480/600], Loss: 0.1729\n",
      "Epoch [23/30], Step [540/600], Loss: 0.1055\n",
      "94.72\n",
      "55.64\n",
      "Epoch [24/30], Step [0/600], Loss: 0.1612\n",
      "Epoch [24/30], Step [60/600], Loss: 0.1473\n",
      "Epoch [24/30], Step [120/600], Loss: 0.1868\n",
      "Epoch [24/30], Step [180/600], Loss: 0.0856\n",
      "Epoch [24/30], Step [240/600], Loss: 0.1615\n",
      "Epoch [24/30], Step [300/600], Loss: 0.0951\n",
      "Epoch [24/30], Step [360/600], Loss: 0.1608\n",
      "Epoch [24/30], Step [420/600], Loss: 0.1072\n",
      "Epoch [24/30], Step [480/600], Loss: 0.1007\n",
      "Epoch [24/30], Step [540/600], Loss: 0.1681\n",
      "94.68\n",
      "54.9\n",
      "Epoch [25/30], Step [0/600], Loss: 0.1401\n",
      "Epoch [25/30], Step [60/600], Loss: 0.3026\n",
      "Epoch [25/30], Step [120/600], Loss: 0.1192\n",
      "Epoch [25/30], Step [180/600], Loss: 0.2557\n",
      "Epoch [25/30], Step [240/600], Loss: 0.0881\n",
      "Epoch [25/30], Step [300/600], Loss: 0.2175\n",
      "Epoch [25/30], Step [360/600], Loss: 0.1205\n",
      "Epoch [25/30], Step [420/600], Loss: 0.0949\n",
      "Epoch [25/30], Step [480/600], Loss: 0.0706\n",
      "Epoch [25/30], Step [540/600], Loss: 0.1427\n",
      "94.91\n",
      "55.63\n",
      "Epoch [26/30], Step [0/600], Loss: 0.2144\n",
      "Epoch [26/30], Step [60/600], Loss: 0.2157\n",
      "Epoch [26/30], Step [120/600], Loss: 0.0644\n",
      "Epoch [26/30], Step [180/600], Loss: 0.1968\n",
      "Epoch [26/30], Step [240/600], Loss: 0.1754\n",
      "Epoch [26/30], Step [300/600], Loss: 0.1480\n",
      "Epoch [26/30], Step [360/600], Loss: 0.1071\n",
      "Epoch [26/30], Step [420/600], Loss: 0.1448\n",
      "Epoch [26/30], Step [480/600], Loss: 0.0958\n",
      "Epoch [26/30], Step [540/600], Loss: 0.1073\n",
      "94.87\n",
      "55.44\n",
      "Epoch [27/30], Step [0/600], Loss: 0.2398\n",
      "Epoch [27/30], Step [60/600], Loss: 0.0544\n",
      "Epoch [27/30], Step [120/600], Loss: 0.1996\n",
      "Epoch [27/30], Step [180/600], Loss: 0.1729\n",
      "Epoch [27/30], Step [240/600], Loss: 0.1275\n",
      "Epoch [27/30], Step [300/600], Loss: 0.1321\n",
      "Epoch [27/30], Step [360/600], Loss: 0.1222\n",
      "Epoch [27/30], Step [420/600], Loss: 0.2708\n",
      "Epoch [27/30], Step [480/600], Loss: 0.2402\n",
      "Epoch [27/30], Step [540/600], Loss: 0.1210\n",
      "94.97\n",
      "55.29\n",
      "Epoch [28/30], Step [0/600], Loss: 0.3254\n",
      "Epoch [28/30], Step [60/600], Loss: 0.2019\n",
      "Epoch [28/30], Step [120/600], Loss: 0.1995\n",
      "Epoch [28/30], Step [180/600], Loss: 0.1803\n",
      "Epoch [28/30], Step [240/600], Loss: 0.1284\n",
      "Epoch [28/30], Step [300/600], Loss: 0.1160\n",
      "Epoch [28/30], Step [360/600], Loss: 0.1866\n",
      "Epoch [28/30], Step [420/600], Loss: 0.0796\n",
      "Epoch [28/30], Step [480/600], Loss: 0.1051\n",
      "Epoch [28/30], Step [540/600], Loss: 0.1057\n",
      "94.97\n",
      "55.2\n",
      "Epoch [29/30], Step [0/600], Loss: 0.1363\n",
      "Epoch [29/30], Step [60/600], Loss: 0.1164\n",
      "Epoch [29/30], Step [120/600], Loss: 0.1577\n",
      "Epoch [29/30], Step [180/600], Loss: 0.1541\n",
      "Epoch [29/30], Step [240/600], Loss: 0.1046\n",
      "Epoch [29/30], Step [300/600], Loss: 0.1865\n",
      "Epoch [29/30], Step [360/600], Loss: 0.4640\n",
      "Epoch [29/30], Step [420/600], Loss: 0.0698\n",
      "Epoch [29/30], Step [480/600], Loss: 0.0956\n",
      "Epoch [29/30], Step [540/600], Loss: 0.2584\n",
      "94.94\n",
      "54.75\n",
      "Epoch [30/30], Step [0/600], Loss: 0.1329\n",
      "Epoch [30/30], Step [60/600], Loss: 0.1809\n",
      "Epoch [30/30], Step [120/600], Loss: 0.1239\n",
      "Epoch [30/30], Step [180/600], Loss: 0.1689\n",
      "Epoch [30/30], Step [240/600], Loss: 0.2681\n",
      "Epoch [30/30], Step [300/600], Loss: 0.1144\n",
      "Epoch [30/30], Step [360/600], Loss: 0.1119\n",
      "Epoch [30/30], Step [420/600], Loss: 0.1328\n",
      "Epoch [30/30], Step [480/600], Loss: 0.2031\n",
      "Epoch [30/30], Step [540/600], Loss: 0.1698\n",
      "95.08\n",
      "54.69\n"
     ]
    }
   ],
   "source": [
    "N_1 = 50# paste your group N_1 value\n",
    "learning_rate = 0.001\n",
    "epochs=30\n",
    "population_curve_N_1, error_curve_N_1, \\\n",
    "population_curve_dropout_N_1, error_curve_dropout_N_1, model_N_1 = get_curves_and_model(N_1,learning_rate,epochs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot two figures: \n",
    "\n",
    "1) test risks of full and dropout network during training\n",
    "\n",
    "2) test errors of full and dropout network during training\n",
    "\n",
    "Use log scale for $x$-axis (iterations)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 563,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEGCAYAAACKB4k+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3de3xcdZ3/8ddnZnJrk9JbWgptaUurFgq9GAsUhFqwC1gEXFAQEZBdREXg5y7C6u4iXhERUXFVFAQVscilIKsoAgUrLtDScmtBoBca6CUtTdu0uc58fn98T04mJW2TNMlkkvfz8ZjHOXPmzDnf00nnPd/v+Z7vMXdHREQEIJHrAoiISO+hUBARkZhCQUREYgoFERGJKRRERCSWynUB9sXw4cN93LhxuS6GiEheWbJkySZ3L2/rtbwOhXHjxrF48eJcF0NEJK+Y2ZrdvabmIxERiSkUREQkplAQEZGYQkFERGIKBRERiSkUREQkplAQEZFYXl+nICLSJ7jDqidg7VOQKoaCEigYsJtpNF8yOMx3MYWCiEgurXkSHv0GrFnUsfedfD3M/NcuL45CQUQkFyoXw6Nfh5WPQelIOOk6mH4u4NBYC407W6YNO9+5bOxR3VIshYKISE9a9xw89k34x0MwYBjM/TpUXAiFA1rWKRyYs+IpFEREesLGFSEMVjwAxYPh+P+GmZ+GotJcl6wVhYKICMDGl+H//gcKS2HkITDiECh/T+tf8B2VSUPVK/DX78KL94RtH3cVHPVZKN6v68rehRQKe1NbHabF+4FZbssiIl1v2zpY+E1Y+mtIlYBnoKk2etFg6IQoJA6FkdFjyDhIJCHdCNvehOo3oHptmG6NptVvhNcyTaG30DGXw6xLYcDQXB7tXikUdlW3LfQGWPVEeGx4ISxPFsLAEVBavst0BAwsD49hE2G/A3NbfhFpn7pt8OQP4Mmbwhf3ERfD+/89dPXcsho2vAgblsPGl8J0xYOAh/emSqBkCNSsDyESMyjbHwaPhdHvg8H/HObfMy98Z+QBhULDztA3uDkE3loKnoZkEYw9Aj7w5dAXuGYj7KgK0+1vhZNFO6rCus0SKfinb8LMi1SrEOmtmhpgyW3w+Ldh5yaYcgbM+U8YOr5lnWEHh8chp7Ysa9gJVS/Dhpdg4/LQirDf6PClP3gM7DcmPE8V9fghdaX+GQoblsOK34cQqHwa0g3hC/3A98L7vwDjj4XRM6GgeM/byWSgdgvs2BjC4v9+DH/8Irz5LMz73r61RYpI13KH5ffDI9fA2yth3Pvhg1+FA2e07/2FA8K67V0/T/XPUFj5GCz8Fow6HI74NIw/DsYeCUVlHdtOIgEDh4XHiMnhj+yv14ceBhtego/9qvWvDxHJjTVPwp//C95cDOWT4eO/g0kfVI2+DebuuS5Dp1VUVHinbsdZuyX8auiuEz6vPgz3XAgY/PMtMOmE7tmPiLQtkw4Xh732F3jt4dAsXHYAzPkyTD07nCTux8xsibtXtPlavwyFnvD2Sph/bqgxzPlPOOYLoWYhIt1j+wZ4/ZHwo+z1R6GuGiwRmoInnwIVn1KTbmRPodA/m496wtAJcOHD8PtL4dGvhV8qp/0YigflumQi+27rm2GIhreehYNmwYQPhHNxJYN7rgzpJqh8JtQEXn0Y1j8flpeOhPd8CCaeAAd/IPQSknZTTaG7ucNTP4U/fSmcX/jYHTDiPbkvk9pSpTMadsDffgB/+37oijn2SHhzCTTUhF/lB8wIX8QHzwldMpMF7d+2e9Rxowp2bIKdm0PvoJ2bYcfm6HnzsrfDek11YEkYMzOEwKQPwsjDVCvfi5w0H5nZrcA8YKO7T4mWDQXmA+OA1cBH3X2LmRnwfeBkYCdwvrs/u7d95EUoNFv9N/jd+WEgq9P+p3VXt570yNfghbvg3AWhy53su3QTJHt5pTuTgYbtnb+KNpOB5+eHnjvb18Ghp8MJ18CQg8IFXJWLQweO1x8NIeGZcPXuuGNCLWLCcYCF99ZsCNPt66Np1vN0fdv7LywN4wQ1PwYOD9PRFWH7PVlD6QNyFQrHAjXAL7NC4TrgbXe/1syuAoa4+5VmdjLweUIoHAF8392P2Ns+8ioUALa9BXd9MlR5R06BwQeFPs5Domnzo7suf//7j0KNxZJhnxf+JfScks7ZsiY0D761NAxqNv3c3lkDW/c8LPhsuBDzgBnw7pPCY+SU9pV3zd/hT/8RjvOAGXDit0INYXdqq2H1X+H1KCS2rGp7vaJBoamnbH8oGwVlI8O0dGTrABgwbO/dw6VDcnai2czGAQ9mhcIrwGx3X2dmo4CF7v5uM/tpNH/nruvtaft5FwoATfWh6v3mknAZ/JY10Lij9TrF+0UBcRAcMA1mXQapwn3b7wt3hx5Rk0+BIz8Hvzw1bPuT93fLjTr6tEwGFt8CD18dmkyGTwpt6xNmwynfD0Mg9AZNDfDEd2DRDVAyFKZ9HFYvCn97eLjY6l0nhoAYd8w7L7rashoe/u/Qt7/sADjhK3DYmR1vmtmyOnQJTRZGX/77hy/+XjYQXH/Sm0Kh2t0HZ72+xd2HmNmDwLXuviha/ghwpbu/4xvfzC4CLgIYO3bse9esWdNt5e8R7qF9tHpNy3gpzfNb1sCmV8KXzUd/1fmT1K8/BnecGdpdP3Fv+NX10oLQnHXIqXDGL9QG216bX4cHPg9r/gYHHx9CYNCBsOTWEBKegeOvDjc/yWW3x7eWwoLPhSEaDj8r/Lpv7oK9fQO8+id45aHwS76pNjTPTDwe3nUSHHQULL41XIyZSMHRl8Osz6vnTh+SD6Hwv8C3dgmFL7r7kj1tPy9rCh219Nfw+8vCaI3n/A4GHdCx9697Dn5xcqh5XPDH1m2vT/4Q/vyf4T/83K93bbn7mkw6fEk++vXwi/fEb8K0c1o3v1SvhQcvD33jxxwBH/4hlL+7Z8vZVB+Gb1h0YxiXa96N8O4Td79+Y224sv+VP4SQqFnf8trUs8Pwzh39m5Nerzd1Sd1gZqOymo82RssrgTFZ640G3urhsvVO0z8Rqtx3fRJ+fkIIhpGHtu+9b6+CX58Rxm7/xD3vPBl31CWhNvLkD0NTVTfc2q9PqHoF7v9cOBf0rhPDECZtfVEOHgPn3B1OyD50FfzkGDjuSjj6so71wumsyiVw/2fD+DzTzoF/+sbeu2MWlMC7/ik8PpSBdctCE9O4Y/r8cA7Stp5uM3gAOC+aPw+4P2v5Jy04Eti6t/MJ/crE48Ov/Ewabj0RVj6+9/fUVMGvPxLGdTr33ra/xMzgpG+HJoM/fjH8UuxOjXWw9I7weOP/QrfD3twlOt0Ef70BfvJ+2PwafORncPZv9/zL2QymngWfezq01T/6NfjZB0KNrbs01oW2/1tOgPrtIZhO+5+O989PJEIQHH2pAqEf687eR3cCs4HhwAbgamABcBcwFngDONPd3466pN4EnEjoknpBW+cTdtUvmo+yVa8N5wY2vwan/gimfqzt9epr4PZTwp2ePnl/GO11Txp2hCamTf+AC/4AB0zv2nJn0vDcnfDYt2BbZevXiveDoQeHYceHTWwZnXLowbm70K9hZxgF83//LfxynnwKnPzd0Dumo5Y/AH/49xCAR18KY2dB7duhP37tlnA+qXk+e3lTfTg5PGBo+HIfMDTr+dCoV87Q8G/7l6vDZzfjk6EZsJfevEV6Dw1z0ZfUVsP8T4Quf8f/dxg+I7tdO90Iv/lY6DN+1m/Cr9X22L4hNE+l6+Ff/hLOQewr99BW/chXQ5PGAdPDSdjBY8MJ282vtTzeXhluTpJt6AQ48rOhq2dXdEnMpEOTWtxXfn1oQ2/uJ1+zIczXbw3rDxgOH7o+9MnfF7Vb4E9fhmV37PKChS/w5i/+kiHhC79kSOhtVrsFdm4JF2zVvt0SINnDtUPoRfThH4QLxkTaQaHQ1zTVhzbuF34H770ATr4+XDzlDvddDM//Fk75Abz3vL1vK9vGl+GWuTBoFHzqT/t2QdDqv8FfvhKGJh82Eeb8V+jptKd+8Q07Q5/25sB45Q+hHb90ZDj/UfGpznVj3L4elv4KlvwStr7R+rVkUev+8c3dJctGhfMHXXkdx4aXwond5gAo3q/jPZTcoW5rFBJboH5buICroyP8Sr+mUOiLMhl49Kuw6Hvhy+uMW+Hx6+BvN4YbAx33xc5td9UT8KuPhIuTPnFvx6+PWP8C/OWaMB5N2SiYfRVM+0Tnrvh1D+X563dh1ePhi/SIz4QT4nsb4TaTCbWlJb+AV/4Y7qw1/thwQ5XBY1sulioe3DsvOBPpRgqFvuyZn8Mfrgh95beuDb+mP3TDvn3RPTcf7rso9G8//Sft29bbq8J9JF74XTgXcMwXwh3ouqpv+9pnQjj844+hT/37Lgy1h9IRrder2Ri68T57e7hoasCwcNHWjPNh+MSuKYtInlMo9HWv/BHu/lTopXTm7V1z0dTj34HHvg6p4jAsRiIZwsES0SOZNZ8IbfOJAjjy4tAFs7tGplz/YrhC96X7wvUC088NJ3DfXgmLfwEvPxhqBQcdAxUXhJPEeX57RJGuplDoD+q2hXblrmoKcQ+/tje/Hq7SbeuRSUfzHtrej/hMOB/REza/HsLhud+GEIAQRNPOgRnnQfm7eqYcInlIoSB9V/Xa0Ktn6ASY/GENnCbSDr3pimaRrjV4TDiZLSJdQqOgiYhITKEgIiIxhYKIiMQUCiIiElMoiIhITKEgIiIxhYKIiMQUCiIiElMoiIhITKEgIiIxhYKIiMQUCiIiElMoiIhITKEgIiIxhYKIiMQUCiIiElMoiIhITKEgIiIxhYKIiMQUCiIiElMoiIhITKEgIiKxnISCmf0/M3vJzF40szvNrNjMxpvZU2b2qpnNN7PCXJRNRKQ/6/FQMLMDgUuBCnefAiSBs4BvA99z90nAFuDCni6biEh/l6vmoxRQYmYpYACwDpgD3B29fjtwWo7KJiLSb/V4KLj7m8D1wBuEMNgKLAGq3b0pWq0SOLCnyyYi0t/lovloCHAqMB44ABgInNTGqr6b919kZovNbHFVVVX3FVREpB/KRfPRCcAqd69y90bgXmAWMDhqTgIYDbzV1pvd/WZ3r3D3ivLy8p4psYhIP5GLUHgDONLMBpiZAccDy4HHgDOidc4D7s9B2URE+rVcnFN4inBC+VnghagMNwNXAl8ws9eAYcAtPV02EZH+LrX3Vbqeu18NXL3L4pXAzBwUR0REIrqiWUREYgoFERGJKRRERCSmUBARkZhCQUREYgoFERGJKRRERCSmUBARkZhCQUREYgoFERGJKRRERCSmUBARkZhCQUREYgoFERGJKRRERCSmUBARkZhCQUREYgoFERGJKRRERCSmUBARkZhCQUREYgoFERGJpXJdABGRjmhsbKSyspK6urpcF6XXKy4uZvTo0RQUFLT7PQoFEckrlZWVlJWVMW7cOMws18XptdydzZs3U1lZyfjx49v9PjUfiUheqaurY9iwYQqEvTAzhg0b1uEalUJBRPKOAqF9OvPvpFAQEemgZDLJtGnT4sfq1av3uP64cePYtGkTAKWlpT1Qws7TOQURkQ4qKSlh2bJluS5Gt1BNQUSkC9x2221ccskl8fN58+axcOHC3BWok1RTEJG8dc3vX2L5W9u6dJuHHDCIq085dI/r1NbWMm3aNADGjx/Pfffd16VlyKV2hYKZvdfdl+yy7BR3/31ndmpmg4GfA1MABz4FvALMB8YBq4GPuvuWzmxfRKQ79eXmo/bWFH5mZue5+wsAZnY2cDnQqVAAvg885O5nmFkhMAD4EvCIu19rZlcBVwFXdnL7ItIP7O0XfU9KpVJkMpn4eb5eXNfecwpnALeb2WQz+1fgs8DczuzQzAYBxwK3ALh7g7tXA6cCt0er3Q6c1pnti4jkwrhx41i2bBmZTIa1a9fy9NNP57pIndKumoK7rzSzs4AFwFpgrrvXdnKfE4Aq4BdmNhVYAlwGjHT3ddH+1pnZiLbebGYXARcBjB07tpNFEBHpWkcffTTjx4/nsMMOY8qUKcyYMSPXReqUPYaCmb1AaPNvNhRIAk+ZGe5+eCf3OQP4vLs/ZWbfJzQVtYu73wzcDFBRUeF7WV1EpMvV1NS8Y5mZcccdd7S5fvZ1DG29tzfZW01hXjfssxKodPenoud3E0Jhg5mNimoJo4CN3bBvERHZgz2eU3D3Ne6+hhAe66P58YT2/62d2aG7rwfWmtm7o0XHA8uBB4DzomXnAfd3ZvsiItJ57e19dA9QYWYTCSeIHwB+A5zcyf1+Hrgj6nm0EriAEFB3mdmFwBvAmZ3ctoiIdFJ7QyHj7k1m9hHgRnf/oZkt7exO3X0ZUNHGS8d3dpsiIrLv2tsltTG6NuGTwIPRsvbftUFERPJCe0PhAuAo4BvuvsrMxgO/7r5iiYhILrQrFNx9ubtf6u53Rs9Xufu13Vs0EZHeqXno7EMPPZSpU6dyww03tLqauactWLCA5cuXd8m29nadwl3u/tE2rlcwwDt5nYKISF7LHvto48aNfPzjH2fr1q1cc801rdZramoiler+cUcXLFjAvHnzOOSQQ/Z5W3urKVwWTecBp2Q9mp+LiPRrI0aM4Oabb+amm27C3bnttts488wzOeWUU5g7dy7uzhVXXMGUKVM47LDDmD9/PgALFy7k2GOP5fTTT+eQQw7h4osvjmsbd955Z3xl9JVXtgwBl32Dnrvvvpvzzz+fJ598kgceeIArrriCadOm8frrr+/T8ewxwrKGnViTvdzMksBZwJq23ici0iP+eBWsf6Frt7n/YXBSx1rHJ0yYQCaTYePGcM3t3//+d55//nmGDh3KPffcw7Jly3juuefYtGkT73vf+zj22GMBePrpp1m+fDkHHXQQJ554Ivfeey+zZs3iyiuvZMmSJQwZMoS5c+eyYMECTjut7eHgZs2axYc//GHmzZvHGWecsW/Hzl5qCmY2yMz+w8xuMrO5FnyecG3BR/d57yIifYR7Swv7Bz/4QYYOHQrAokWLOPvss0kmk4wcOZLjjjuOZ555BoCZM2cyYcIEkskkZ599NosWLeKZZ55h9uzZlJeXk0qlOOecc3jiiSd67Dj21tj1K2AL8HfgX4ArgELg1OhaAxGR3OngL/rusnLlSpLJJCNGhHE8Bw4cGL+WHRa7MrN3PG/v+t01NPfezilMcPfz3f2nwNmEC87mKRBERIKqqiouvvhiLrnkknd8yQMce+yxzJ8/n3Q6TVVVFU888QQzZ84EQvPRqlWryGQyzJ8/n2OOOYYjjjiCxx9/nE2bNpFOp7nzzjs57rjjABg5ciQrVqwgk8m0uttbWVkZ27dv75Lj2VsoNDbPuHsaWOXuXbNnEZE81Xw7zkMPPZQTTjiBuXPncvXVV7e57umnn87hhx/O1KlTmTNnDtdddx37778/AEcddRRXXXUVU6ZMYfz48Zx++umMGjWKb33rW3zgAx9g6tSpzJgxg1NPPRWAa6+9lnnz5jFnzhxGjRoV7+Oss87iO9/5DtOnT9/nE822l6pKGtjR/BQoAXbS0iV10D7tfR9VVFT44sWLc1kEEelhK1asYPLkybkuxj5buHAh119/PQ8++ODeV94Hbf17mdkSd29rqKG99j5KdmHZRESkl+v+qypEROQdZs+ezezZs3NdjHdo79hHIiLSDygURCTv7OlcqLTozL+TQkFE8kpxcTGbN29WMOyFu7N582aKi4s79D6dUxCRvDJ69GgqKyupqqrKdVF6veLiYkaPHt2h9ygURCSvFBQUMH78+FwXo89S85GIiMQUCiIiElMoiIhITKEgIiIxhYKIiMQUCiIiElMoiIhITKEgIiIxhYKIiMQUCiIiElMoiIhILGehYGZJM1tqZg9Gz8eb2VNm9qqZzTezwlyVTUSkv8plTeEyYEXW828D33P3ScAW4MKclEpEpB/LSSiY2WjgQ8DPo+cGzAHujla5HTgtF2UTEenPclVTuBH4IpCJng8Dqt29KXpeCRzY1hvN7CIzW2xmizWeuohI1+rxUDCzecBGd1+SvbiNVdu8rZK73+zuFe5eUV5e3i1lFBHpr3Jxk52jgQ+b2clAMTCIUHMYbGapqLYwGngrB2UTEenXerym4O7/4e6j3X0ccBbwqLufAzwGnBGtdh5wf0+XTUSkv+tN1ylcCXzBzF4jnGO4JcflERHpd3J6j2Z3XwgsjOZXAjNzWR4Rkf6uN9UUREQkxxQKIiISUyiIiEhMoSAiIjGFgoiIxBQKIiISUyiIiEhMoSAiIjGFgoiIxBQKIiISUyiIiEhMoSAiIjGFgoiIxBQKIiISUyiIiEhMoSAiIjGFgoiIxBQKIiISUyiIiEhMoSAiIjGFgoiIxBQKIiISUyiIiEhMoSAiIjGFgoiIxBQKIiISUyiIiEhMoSAiIjGFgoiIxHo8FMxsjJk9ZmYrzOwlM7ssWj7UzB42s1ej6ZCeLpuISH+Xi5pCE/Bv7j4ZOBL4nJkdAlwFPOLuk4BHouciItKDejwU3H2duz8bzW8HVgAHAqcCt0er3Q6c1tNlExHp73J6TsHMxgHTgaeAke6+DkJwACN2856LzGyxmS2uqqrqqaKKiPQLOQsFMysF7gEud/dt7X2fu9/s7hXuXlFeXt59BRQR6YdyEgpmVkAIhDvc/d5o8QYzGxW9PgrYmIuyiYj0Z7nofWTALcAKd78h66UHgPOi+fOA+3u6bCIi/V0qB/s8GjgXeMHMlkXLvgRcC9xlZhcCbwBn5qBsIiL9Wo+HgrsvAmw3Lx/fk2UREZHWdEWziIjEFAoiIhLrl6FQvbOBZWurc10MEZFep1+Gwi2LVnHaj/7G5+54ltWbduS6OCIivUYueh/l3KePOxgz42dPrORPL63n40eM5dLjJzG8tCjXRRMRySlz91yXodMqKip88eLFnX7/xm113PjIq8x/Zi3FqQSfPu5g/uX94xlQ2C+zUkT6CTNb4u4Vbb7Wn0Oh2etVNVz30Mv86aUNlJcVcfkJk/hYxRhSyX7ZuiYifdyeQkHfesDB5aX89NwK7vnMURw0dABfvu9F5t74BA+9uJ58Dk0RkY5STWEX7s7Dyzfw7Yde5vWqHbz3oCGc9b4xzDhoCBOGDySM0iEikr/UfNQJTekMdy2u5PuP/IMN2+oB2K+kgOljBzN9zBCmjx3MtLGDGVRc0C37FxHpLnsKBZ1R3Y1UMsHHjxjLWe8bw2tVNSx9YwvPrqlm6dotPP6PKtzBDCaWlzJjbEtIHFxeSoHORYhInlJNoRO21TXy3Npqlr5RzdI3trB0bTXVOxsBKEgaE0eUMXlUGZP3H8R7RpUxedQgdXcVkV5DNYUuNqi4gPdPKuf9k8JNftydVZt28HzlVlas38bL67az6NVN3Pvsm/F7hpcWhaAYNYj37F/GuOEDGT2khPLSIp2nEJFeQ6HQBcyMCeWlTCgv5TQOjJdvrqnnlfXbWb5uGy+v387L67dx25OraWjKxOsUphKMHlzCgUNKGD2khAMHlzB6yAAOjOZHDiommVBoiEjPUCh0o2GlRcyaWMSsicPjZU3pDKs27eCNt3fyZnUtlVtqeXNLLZVbdvLwum1sqmlotY2EwdCBRQwvLWR4aRHDSgsZNjBMy5uflxYxbGAhZcUpiguSFKUSqn2ISKcoFHpYKplg0sgyJo0sa/P12oY0b1bXRoGxk3XVdWzeUc+mmgY219Szdu1ONm2vZ0dDeo/7KS5IUFyQpDiVpKQwBEVxQZKSgiQDi5KUlxVRXlbMiLIiysuKGFFWxIhBxQwvLaQoleyOQxeRPKBQ6GVKCpNMHFHKxBGle1yvtiHN5h31bK5pCKGxvYEdDU3UNWaobUxT35imrjFNbWOausZMPF/fmKFySy3L1lazeUcDbfUzGDygIA6LIQMKGVRSwH57eAwqKaCsKEVCzVwieU+hkKdKCpOMLhzA6CEDOr2NpnSGzTsa2LitnqqaOjZuq2fj9no2bq+januYf6t6G9tqG9la20hTZs891YpSCUoKW9dOSgpD7aS5llIc1VQGFqUojR5hPklpUQEDi5JheXGKAYUpilIJNYeJ9CCFQj+WSiYYOaiYkYOKgf32uK67s7MhzdYoILIf22ob2VbX1Kp2UhvVTpof2+oaQy2mIc3OhiZq6ptoTLe/O3RzOBRF50zCI0lRQZgvTCWjaYKiZDSNnhemEhQmw7qFyQQF0ToFKaMgueuyRLwskQDDMAv3jzXbZZ5wrUoyYa1CTx0DJJ8pFKRdzIyB0a/6AwaXdMk265vS1NQ1saM+TU19CIod9U1sj6Y76puob8pEj9D0Fc83ZahvDNO6xjRbdzZQ35ShIZ2hoSk86qNpQzpDei+1nK5UkAwh0RIUCUoKkhQVJClMJkgmjFTCSCWNVCJBKmlZyxJhmgihVZhMtJoviB6pZMvzZKLl/YnmqVm8PHt/RankOwI2lTDVxCSmUJCcKUolKSpNMmzPp0+6RFM6KzDSGRrTTkNThsbsZfFrYVnGwR0cj6ahxtRqmUM649Q1RbWkhgx1TWlqG9LUR9O6xpZltY1pmtIZmjJOOuM0RoHVlHGa0tE0k6EpHV5rSGfaPO/TlRJGXOsqTiUpTIWgMYNEVCNKNNeSmp9HtahEwiiIAqegOdCSCQqyAq8gmiZ2qWkB0fPs5c37MhLRfMJaamnZzxMWwi6ZMAqSRjKRiMMvLEu0Csvs2Ns1BLOftd4HJBKt99lSrtahm2zetzUvS5CMnicShGnWv2NvpVCQfiGVTJBKJhhQmOuSdFxzeDRGYdYcWs3PmzIZMhlIu5POZEhniJc1ZTJk3ElniN/XUtMKtaz6XZbVN6VJe0sAZrKntF6e9hC4TWmnpqkpDrMQci3la0p71vujbQBkh230WvZ+Mln770viYEm0hEyyOfgSzfNGMtESQIlE6/UuP+FdnDL1gC4vm0JBpJcLv0ZDc1R/1VJDCyGRcY+DsK2aV2PaoxpYCKR0Vqq8M2Bav+ZAJuNRTbElmLLDMePEAdwcxk3xPsO0+dGYCbW95m2m3XEPrzXvo3k+PrYoyN+xXnN5Ms7gAd0zGKdCQUR6vebmI4AkvbfppaJItk0AAAXVSURBVC/QcJ4iIhJTKIiISEyhICIiMYWCiIjEFAoiIhJTKIiISEyhICIiMYWCiIjEzPP4+nEzqwLW7LJ4OLApB8XpLn3teKDvHVNfOx7oe8fU144H9u2YDnL38rZeyOtQaIuZLXb3ilyXo6v0teOBvndMfe14oO8dU187Hui+Y1LzkYiIxBQKIiIS64uhcHOuC9DF+trxQN87pr52PND3jqmvHQ900zH1uXMKIiLSeX2xpiAiIp2kUBARkVifCQUzO9HMXjGz18zsqlyXpyuY2Woze8HMlpnZ4lyXpzPM7FYz22hmL2YtG2pmD5vZq9F0SC7L2BG7OZ6vmNmb0ee0zMxOzmUZO8LMxpjZY2a2wsxeMrPLouX5/Bnt7pjy8nMys2Ize9rMnouO55po+Xgzeyr6jOabWZfcbLZPnFMwsyTwD+CDQCXwDHC2uy/PacH2kZmtBircPW8vujGzY4Ea4JfuPiVadh3wtrtfGwX4EHe/MpflbK/dHM9XgBp3vz6XZesMMxsFjHL3Z82sDFgCnAacT/5+Rrs7po+Sh5+TmRkw0N1rzKwAWARcBnwBuNfdf2tmPwGec/cf7+v++kpNYSbwmruvdPcG4LfAqTkukwDu/gTw9i6LTwVuj+ZvJ/yHzQu7OZ685e7r3P3ZaH47sAI4kPz+jHZ3THnJg5roaUH0cGAOcHe0vMs+o74SCgcCa7OeV5LHfwRZHPizmS0xs4tyXZguNNLd10H4DwyMyHF5usIlZvZ81LyUN00t2cxsHDAdeIo+8hntckyQp5+TmSXNbBmwEXgYeB2odvemaJUu+87rK6HQ1p28879dDI529xnAScDnoqYL6X1+DBwMTAPWAd/NbXE6zsxKgXuAy919W67L0xXaOKa8/ZzcPe3u04DRhJaRyW2t1hX76iuhUAmMyXo+GngrR2XpMu7+VjTdCNxH+GPoCzZE7b7N7b8bc1yefeLuG6L/tBngZ+TZ5xS1U98D3OHu90aL8/ozauuY8v1zAnD3amAhcCQw2MxS0Utd9p3XV0LhGWBSdDa+EDgLeCDHZdonZjYwOkmGmQ0E5gIv7vldeeMB4Lxo/jzg/hyWZZ81f3lGTiePPqfoJOYtwAp3vyHrpbz9jHZ3TPn6OZlZuZkNjuZLgBMI50keA86IVuuyz6hP9D4CiLqX3QgkgVvd/Rs5LtI+MbMJhNoBQAr4TT4ek5ndCcwmDPO7AbgaWADcBYwF3gDOdPe8OHm7m+OZTWiScGA18Onm9vjezsyOAf4KvABkosVfIrTB5+tntLtjOps8/JzM7HDCieQk4Yf8Xe7+1eg74rfAUGAp8Al3r9/n/fWVUBARkX3XV5qPRESkCygUREQkplAQEZGYQkFERGIKBRERiSkURNpgZums0TSXdeXIu2Y2LnuUVZHeJLX3VUT6pdpoWAGRfkU1BZEOiO5x8e1ofPunzWxitPwgM3skGmztETMbGy0faWb3RWPhP2dms6JNJc3sZ9H4+H+OrlTFzC41s+XRdn6bo8OUfkyhINK2kl2ajz6W9do2d58J3ES4ip5o/pfufjhwB/CDaPkPgMfdfSowA3gpWj4J+JG7HwpUA/8cLb8KmB5t5+LuOjiR3dEVzSJtMLMady9tY/lqYI67r4wGXVvv7sPMbBPhxi6N0fJ17j7czKqA0dnDD0TDOT/s7pOi51cCBe7+dTN7iHATnwXAgqxx9EV6hGoKIh3nu5nf3TptyR6jJk3L+b0PAT8C3gssyRoFU6RHKBREOu5jWdO/R/NPEkbnBTiHcMtEgEeAz0B8o5RBu9uomSWAMe7+GPBFYDDwjtqKSHfSrxCRtpVEd7pq9pC7N3dLLTKzpwg/qs6Oll0K3GpmVwBVwAXR8suAm83sQkKN4DOEG7y0JQn82sz2I9w46nvR+PkiPUbnFEQ6IDqnUOHum3JdFpHuoOYjERGJqaYgIiIx1RRERCSmUBARkZhCQUREYgoFERGJKRRERCT2/wGQ6YEgMg5DLAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3de3xcdZ3/8ddnZnJrkl7Spmmhl6TSgqVXCAW5lFKBhV0uIrBS1AV1f/zwt/0J6rKw6/5U2IuIrO66oC4qggoFFqFWZcEbpaJIW6BQaSmUttD0mqSXXNpcZub7++N7kkzLpEmbTCaT834+HudxLnMy8z0dOO/5fr/nfI855xARkfCKZLsAIiKSXQoCEZGQUxCIiIScgkBEJOQUBCIiIRfLdgGO1pgxY1xlZWW2iyEiklNeeumlOudcebrXci4IKisrWb16dbaLISKSU8zsne5eU9OQiEjIKQhEREJOQSAiEnIKAhGRkFMQiIiEnIJARCTkFAQiIiEXniCofxt+fTsk4tkuiYjIoBKeIHjjF/D81+Hhv4SD+7JdGhGRQSM8QXDWZ+DSb8Lm5+B75/sagoiIhCgIAE69Dv7qp3CgHr67EDYtz3aJRESyLlxBAFB5NtzwLJSOhx99GFZ+N9slEhHJqowGgZldZGYbzGyjmd2W5vXrzazWzNYE019nsjydRlXCp34JJ5wPT/0t/OLzkGgfkI8WERlsMhYEZhYF7gUuBqYDi8xseppdH3XOzQmm72WqPO9ROBwWLYEzPwOrvgc/vhIO7BmwjxcRGSwyWSOYB2x0zm1yzrUBjwCXZ/Dzjl4kChf+E1z+LXj3BfjeB6H2zWyXSkRkQGUyCI4Htqas1wTbDnelmb1mZo+b2cR0b2RmN5jZajNbXVtb2/8lnftRuO5n0NLgryja+Ov+/wwRkUEqk0Fgaba5w9Z/BlQ652YBvwYeTPdGzrn7nHPVzrnq8vK0D9jpu0ln+E7kkRPhoavh55+FbS+DO7zIIiJDSyaDoAZI/YU/AdieuoNzrt451xqsfhc4NYPl6dnISfDJZ2DOR2HNw/Dd8+DbZ8If7oGmDNREREQGgUwGwSpgqplVmVk+cA2wLHUHMxufsnoZsD6D5emdghK4/B74/Ab4i69DXhH88gvw9ZNgybX+DmVdYSQiQ0jGnlnsnIub2WLgGSAK3O+ce93M7gBWO+eWAZ8xs8uAOLAHuD5T5TlqRSPhtE/5afcbsOYheO1R2PALKC6HWR+BOddCxcnZLqmISJ+Yy7E28Orqape1h9cn4r4jec2PYcPTkGyH4+bCqZ+AmVdBfnF2yiUi0gMze8k5V532NQXBMWquh7WPwUsPQu16KBgOs6/xoVCR7nYJEZHsURBkknOw9UVY9X1YtxQSbTDpA1D9SZh+OcQKsl1CEREFwYBprvd9Cavvh72bYdhofwXSqdfD6Pdlu3QiEmIKgoGWTPrhrld/H954ClwCppwHVfNh1GQYWennw0aDpbvdQkSkfx0pCDJ21VCoRSLwvvP81LADXvmRnzY9e+h++SX+3oWRk4OACOZjp0NZVf+Vp3En1G+ESWf6somIpFCNYCC1NsK+d2HvO7DvnffO25q69p15NZz3D1A2pW+f9/tvwgv3QPsBHzAL/h7ef6lqIiIhoxrBYFFQ6u87SHfvgXN+9NN9W2D9z+GP34bXn/T9C/P/Dkorev85iXZ4+UFYfic018KMK2HKAh8Kj30cxs2C874A0/5MgSAiqhEMWo074bm7/Ak9mg9nfNoPmV00svu/cQ7e+Dn8+su+KWjyWXDBP8GEYOSORNxf8vrcV2HvFji+2tc63rdQgSAyxKmzOJfVvw3P/iv86XEoHAnnfA7m3eCHvki1dSX88v/B1j/CmBPhgju6/8WfaPdXNz33NWio8X0HC7/gn94mIkOSgmAo2PEa/OYO2PgrKD0OFtzmL03d946vAaxfBiUV/hf+nI9BtBetfvFWePmHsOJuaNoJVef6JqNJp/e9vG3NsHOtH8F151rIH+afDNcxjZzsHw4kIgNCQTCUbHnen/hrVsGISdC4HaIFcNZNcObiYxvmov2gv/fh+W/4PoWSCt9JXTbFX71UNgXK3ueXC0e89+/jbbDrT7D9Zdj+Cmx7xd9t7ZL+9ZIKHzot+w79u2GjDw2HUZUwZhocdwrE8o/+OESkWwqCocY52PAUvPAtKD/R1w5Kxvb9fdua4ZWHYOersGcz7NkEjTsO3WfYmK6QyCuCHWtg1+v+jmqAojI4/hR/Mj9url8uHedfO7jXXyG1d7Pvo0id9m3191sA5A3zz4eomu+n8XP80+SyLZnwx9BcB9E83SQoOUVBIMeurbkrFA6ZNkNbo78CqeOEf9wp/r6IY+l4TsR9f8XOtbD5d7B5ha9VABSMgMqzuoKh/P093w/hHMRboLUJ2pt9v0ii3Q8UmIgH8/aueaLdh1nLPn+ib649dH6gDg7Ud9VyAI4/FeZ+3F+VpWYuGeQUBJKbmnb7QNi8Arb8zgcQ+FpJ5VmQX+rDqLXRn/DbmoLlYOqoYRyLwhF+uPFhY6B4jF9OnTfuhFd+DLvX+RrM9A/BKR/340wNliuw2lsgfhCKRmW7JJLOgT2+SXV/jf8Rkmj3Tagdy4m2906zroGqc47p43QfgeSmkrF+eO+ZV/n1fVt9IGxeAe/83jfV5Jf4hwkVlPr9C4b79fxgW0GpP1FH8/wU6ZjHDl3vWC4a6U/+vemjOOP/+M7wV34Ia38Crz4Mo0+AuR+D2dce3b0f/eHgPj8A4jt/gHdf8P01yYS/gfCMT8PE0wdPSB2t/TXw5jP+uy+bAtMuggnVg6PJsCeJdn85984/+RP/rtf9/PBm18NZ1F86Hs33/z1G86Hy2EKgJ6oRiPSHtmZ4fakfSuTdF/z/xNP+zIfC6BN69x4W9Z39+cU+yHpq/mrY3nXSf/eP/gSD8yF33FxfO3FJX6aW/b6v5fQbYcaHj21U3ETch0v9W4CBRYLJ/Am5c71jisKICb4v5Wg/L5n0Fx9s+B8fALvW+u2lx0HTLl/bKyqDE873/84nfDCzNR/n/N35rU1+Hm/xF1nEW/x6e8uhy21NUPeWP+HXvtHVhxbJg/KT/E2l42b4+ahKiBV2nfSj+cEPk/4NOTUNiQykurf8yXfNEmjefezvkzcsJRhKu5Zjhf4Es++dYL9imDgPJp/pT/7Hn+ov1+3Q1uyfrvfif/mTUnG5Hya9+pNdHfnpOAe71/sBFDc952thrQ1HfxwW9Se78hP9NOZEKJ/mrxArKO3ar6XBj8f15jPw1i99/4xF/YUD0y7y05ipPtTe/q3fb+OvfN+NRX2NZ9qFfr/yk95b+3HOn6AP1AfTXj8/uMe/Z8t+X4aWff44O9f3+/Vk/OiOu6QiGElghp/GzYDRU7N2RZyCQCQbEu2+KePwy2a7k0z4E1Vbs59aG7uW25pSXjvgf2V3nPjHzerdfSPOwabl8OJ3/Ek0EoOTr/C1hI67z/e+03Xi37yiK8hGVcGUc/29JuNn+5Osc77G0d2UaPfvV7cBaoNpz9uHnlCHT/Ch4JKw5fe+875wBJxwAZx4sb/rfVjZkf/Ntr0Mbz4Nbz3jLzYAf2n1+Fn+JH6g3rfHH6j379+d/FLf6V84wjcxHrI8wq/nl/iAziv081ihv3ouVti1PVbktxWU9PydDCAFgYgcqv5tWPU93+Hd2uCbjVr2+0t7AYrHdp34p5zrrwbrD4l2f8VZ3QZfO6l90y8n2n3zzrSL/S/73gRbOg3bfW3izWf8MQ4r8/erFI3y82Gju7YNG+2bl4aV+RN9LvQ39IGCQETSa230TVivLvHNRB0n/nRNK5LTdNWQiKRXUAqn3+AnCS09pUREJOQUBCIiIacgEBEJOQWBiEjIKQhEREJOQSAiEnIKAhGRkFMQiIiEnIJARCTkFAQiIiGnIBARCTkFgYhIyGU0CMzsIjPbYGYbzey2I+x3lZk5M0s7Mp6IiGROxoLAzKLAvcDFwHRgkZlNT7NfKfAZ4MVMlUVERLqXyRrBPGCjc26Tc64NeAS4PM1+/wTcBbRksCwiItKNTAbB8cDWlPWaYFsnM5sLTHTO/TyD5RARkSPIZBCke7xR5+PQzCwCfAP4fI9vZHaDma02s9W1tbX9WEQREclkENQAE1PWJwDbU9ZLgRnAcjPbApwBLEvXYeycu885V+2cqy4vL89gkUVEwieTQbAKmGpmVWaWD1wDLOt40Tm33zk3xjlX6ZyrBP4IXOac0wOJRUQGUMaCwDkXBxYDzwDrgcecc6+b2R1mdlmmPldERI5ORh9e75x7CnjqsG1f7GbfBZksi4iIpKc7i0VEQk5BICIScgoCEZGQUxCIiIScgkBEJOQUBCIiIacgEBEJOQWBiEjIKQhEREJOQSAiEnIKAhGRkFMQiIiEnIJARCTkFAQiIiGnIBARCTkFgYhIyCkIRERCTkEgIhJyCgIRkZBTEIiIhJyCQEQk5BQEIiIhpyAQEQk5BYGISMgpCEREQk5BICIScgoCEZGQUxCIiIRcLNsFEBHpSXt7OzU1NbS0tGS7KINeYWEhEyZMIC8vr9d/oyAQkUGvpqaG0tJSKisrMbNsF2fQcs5RX19PTU0NVVVVvf47NQ2JyKDX0tLC6NGjFQI9MDNGjx591DUnBYGI5ASFQO8cy7+TgkBEpAfRaJQ5c+Z0Tlu2bDni/pWVldTV1QFQUlIyACXsG/URiIj0oKioiDVr1mS7GBnTY43AzKJm9rVjeXMzu8jMNpjZRjO7Lc3rN5rZWjNbY2bPm9n0Y/kcEZGB9sADD7B48eLO9UsuuYTly5dnr0B90GONwDmXMLNTzcycc663b2xmUeBe4AKgBlhlZsucc+tSdnvYOfedYP/LgK8DFx3VEYhIqNz+s9dZt72hX99z+nHD+dKlJ3f7+sGDB5kzZw4AVVVVPPnkk/36+dnW26ahV4Cfmtl/A80dG51zTxzhb+YBG51zmwDM7BHgcqAzCJxzqd9mMdDroBERGShDvWmot0FQBtQDC1O2OeBIQXA8sDVlvQY4/fCdzOxvgM8B+Ye9f+o+NwA3AEyaNKmXRRaRoehIv9wHUiwWI5lMdq7n8s1uvQoC59wnjuG9013D9J5f/M65e4F7zexa4B+B69Lscx9wH0B1dbVqDSKSdZWVlXzrW98imUyybds2Vq5cme0iHbNeBYGZTQD+EzgLfzJ/HrjJOVdzhD+rASamrE8Ath9h/0eAb/emPCIi2XbWWWdRVVXFzJkzmTFjBqecckq2i3TMets09APgYeDqYP1jwbYLjvA3q4CpZlYFbAOuAa5N3cHMpjrn3gpW/wJ4CxGRQaapqek928yMhx56KO3+qfcZpPvbwaa3QVDunPtByvoDZnbzkf7AORc3s8XAM0AUuN8597qZ3QGsds4tAxab2flAO7CXNM1CIiKSWb0Ngjoz+xiwJFhfhO88PiLn3FPAU4dt+2LK8k29/HwREcmQ3g4x8UngL4GdwA7gqmCbiIjkuB5rBMGNYVc65y4bgPKIiMgA67FG4JxL4G8EExGRIai3fQS/N7N7gEc59M7ilzNSKhERGTC9DYIzg/kdKdsc3dwJLCIylESjUWbOnEl7ezuxWIzrrruOm2++mUgkOyP5L126lGnTpjF9ev+M09mbPoII8G3n3GP98okiIjkmdayh3bt3c+2117J//35uv/32Q/aLx+PEYpkf3X/p0qVccskl/RYEvekjSAKLe9pPRCQMxo4dy3333cc999yDc44HHniAq6++mksvvZQLL7wQ5xy33HILM2bMYObMmTz66KMALF++nPnz53PFFVcwffp0brzxxs6xipYsWdJ5h/Ktt97a+VmpD7V5/PHHuf766/nDH/7AsmXLuOWWW5gzZw5vv/12n4+pt9H1KzP7W97bR7CnzyUQETka/3Mb7Fzbv+85biZcfGevd58yZQrJZJLdu3cD8MILL/Daa69RVlbGT37yE9asWcOrr75KXV0dp512GvPnzwdg5cqVrFu3jsmTJ3PRRRfxxBNPcOaZZ3Lrrbfy0ksvMWrUKC688EKWLl3Khz70obSffeaZZ3LZZZdxySWXcNVVV/X92Ol9EHTcM/A3KdscMKVfSiEikmNSH89ywQUXUFZWBsDzzz/PokWLiEajVFRUcO6557Jq1SqGDx/OvHnzmDLFnzYXLVrE888/T15eHgsWLKC8vByAj370o6xYsaLbIMiE3o4+WpXpgoiI9MpR/HLPlE2bNhGNRhk7diwAxcXFna8d6fldhz9Y3sx6vX8mh7k+Yh+Bmf1dyvLVh732r5kqlIjIYFVbW8uNN97I4sWL33NiB5g/fz6PPvooiUSC2tpaVqxYwbx58wDfNLR582aSySSPPvooZ599NqeffjrPPfccdXV1JBIJlixZwrnnngtARUUF69evJ5lMHvJUtNLSUhobG/vtmHrqLL4mZfnvD3tNj5QUkVDoeFTlySefzPnnn8+FF17Il770pbT7XnHFFcyaNYvZs2ezcOFC7rrrLsaNGwfABz7wAW677TZmzJhBVVUVV1xxBePHj+crX/kK5513HrNnz+aUU07h8sv9Pbx33nknl1xyCQsXLmT8+PGdn3HNNdfwta99jblz5/ZLZ7H1UC15xTk39/DldOsDpbq62q1evXqgP1ZEsmj9+vW8//3vz3Yx+mT58uXcfffd/PznP8/4Z6X79zKzl5xz1en276lG4LpZTrcuIiI5qKfO4tlm1oB/7GRRsEywXpjRkomIDCELFixgwYIF2S5GWkcMAudcdKAKIiIi2ZGdgTJERI7Skfozpcux/DspCERk0CssLKS+vl5h0APnHPX19RQWHl3LfeZHRxIR6aMJEyZQU1NDbW1ttosy6BUWFjJhwoSj+hsFgYgMenl5eVRVaYCDTFHTkIhIyCkIRERCTkEgIhJyCgIRkZBTEIiIhJyCQEQk5BQEIiIhpyAQEQk5BYGISMgpCEREQk5BICIScgoCEZGQUxCIiIRcRoPAzC4ysw1mttHMbkvz+ufMbJ2ZvWZmvzGzyZksj4iIvFfGgsDMosC9wMXAdGCRmU0/bLdXgGrn3CzgceCuTJVHRETSy2SNYB6w0Tm3yTnXBjwCXJ66g3PuWefcgWD1j8DRPU1BRET6LJNBcDywNWW9JtjWnU8B/5PuBTO7wcxWm9lqPaFIRKR/ZTIILM22tA8cNbOPAdXA19K97py7zzlX7ZyrLi8v78ciiohIJh9VWQNMTFmfAGw/fCczOx/4AnCuc641g+UREZE0MlkjWAVMNbMqM8sHrgGWpe5gZnOB/wIuc87tzmBZRESkGxkLAudcHFgMPAOsBx5zzr1uZneY2WXBbl8DSoD/NrM1Zrasm7cTEZEMyWTTEM65p4CnDtv2xZTl8zP5+SIi0jPdWSwiEnIKAhGRkFMQiIiEnIJARCTkFAQiIiGnIBARCTkFgYhIyCkIRERCTkEgIhJyCgIRkZBTEIiIhFyogmD/gfZsF0FEZNAJTRB8//nNfPDry6lt1CMPRERShSYIzpk6hoaWOLf+5DWcS/ugNBGRUApNEEyrKOXvLz6J376xm4defDfbxRERGTRCEwQA132gknOmjuGff7GOt2ubsl0cEZFBIVRBEIkYd189m8K8KJ99dA3tiWS2iyQiknWhCgKAiuGF3PnhmbxWs59v/uatbBdHRCTrQhcEABfNGM/Vp07g3mc3snrLnmwXR0Qkq0IZBABfuuxkJowaxmcfW0Nji+4vEJHwCm0QlBTE+MZHZrNt70Fu/9m6bBdHRCRrQhsEAKdOLmPxeSfw+Es1PLV2R7aLIyKSFaEOAoD/+8GpzJ4wgn94ci0797dkuzgiIgMu9EGQF43wjY/MobU9yS2Pv0oyqbuORSRcQh8EAFPKS/h/l0znd2/V8cAftmS7OCIiA0pBEFg0byLnv38sdz79Bht2Nma7OCIiA0ZBEDAz7rxyFsMLY9z0yCu0xhPZLpKIyIBQEKQYU1LAXVfN4o2djVx3/0re2NmQ7SKJiGScguAwC0+q4Csfnsn6HY38+X/8jn9cupb6Jj3DQESGLgVBGovmTWL53y7grz5QyZKVW1lw93K+97tNtMU1SJ2IDD0Kgm6MKs7ny5edzDM3n8Mpk0bxz79Yz5/9+wp+vW6XHmwjIkOKgqAHJ4wt5cFPzuMHnziNiMFf/3A1f3X/Sl1ZJCJDhoKgl847cSxP3zyfL106nVe37uPi/1jBPy5dy57mtmwXTUSkTzIaBGZ2kZltMLONZnZbmtfnm9nLZhY3s6syWZb+kBeN8ImzqnjulvP4+BmTWbJyK+fe9Sz/9ssN7FUgiEiOylgQmFkUuBe4GJgOLDKz6Yft9i5wPfBwpsqRCaOK87n98hk8fdM5nD11DP/5242c/dXf8tWn39AVRiKSc2IZfO95wEbn3CYAM3sEuBzoHPPZObcleC0nL8eZWlHKtz92Kht2NvKfv32L7zz3Ng/8fgsf/8Bk/tc5UygvLch2EUVEepTJpqHjga0p6zXBtqNmZjeY2WozW11bW9svhetPJ44r5Z5rT+FXn53PRTPG8b3fbeLsr/6WO362jl0NGtFURAa3TAaBpdl2TNddOufuc85VO+eqy8vL+1iszDlhbCnf+MgcfvP5BVw6+zgefGEL59z1LF/86Z/Yvu9gtosnIpJWJpuGaoCJKesTgO0Z/LxBo2pMMXdfPZvPLJzKt5Zv5OEX32XJync5rbKM6eOHM/244Zx83AjeV15MLKoLt0QkuzIZBKuAqWZWBWwDrgGuzeDnDTqTRg/jzitnsXjhCfzg91tYvWUPP/rjO7QGdyjnxyKcNK6U6eOHc/JxPiBOGjec4oJMfi0iIoeyTN4la2Z/Dvw7EAXud879i5ndAax2zi0zs9OAJ4FRQAuw0zl38pHes7q62q1evTpjZc60eCLJprpm1m1v4PXt+1m3o4HXtzew70A7AGa+RjFn4kjmThrF3IkjOWlcqWoOItInZvaSc6467Wu5NlxCrgdBOs45duxv4fXtDazb3sDabftYs3UfdU3+3oSivCgzjx/B3EkjmTtpJHMmjmLciMIsl1pEcomCIAc556jZe5CX393Lmq37eOXdfazb3kBbwjcrjR9RyJyJIzluZBHFBTFKCqKUFORRXBCltDBGcX6MksIYJQUxigtiDC/MIz+mWoVIWB0pCNQYPUiZGRPLhjGxbBiXz/FX3bbGE6zb3tAZDK/W7GPFm7U0t/X8EJ28qDFrwkhOqyzjtMpRVE8uY8SwvEwfhojkANUIhoBk0tHcFqe5NUFTa5ym1jjNwbypJU5zW5xtew+yasse1m7bT3vCYQYnVpT6YKgqY15lmZqbRIYw1QiGuEjEKC3Mo7Sw51/4Le0J1mzdx6rNe1i5ZQ9PvFzDj/74DgATy4o4bXIZM44fwUnjSpk2rpQxJbo7WmSoUxCETGFelDOmjOaMKaMBfxXT+h2NrNyyh9Vb9rDirTqeeGVb5/5jSvKZVlHKtIrSznCYVlFKiS5xFRky1DQkh3DOUdfUxoadjWzY1cibOxt5Y1cjb+1q5EBKX8SEUUVMKS9hTEk+Y0oKGF2cT1lxsFySz+hgW2FeNItHIyId1DQkvWZmlJcWUF5awNlTx3RuTyYd2/Yd5I2djby5q5ENOxvZUt/M27ubqGtq7bxJ7nDF+VHKSvIpKy5g1LA8yoblM3JYPmXFecE8n5HD8igrzu98TVc3iQwsBYH0SiTSdRXTBdMrDnnNOceBtgT1TW3UN7d2zZvb/HJTK3sOtLOnuY2Nu5vYd6CdptZ4t59VWhCjrCSfUcPyO2saHdOo4q5tHYFVEFOtQ6QvFATSZ2ZGcXC/wqTRw3r1N63xBPsOtLP3QBt7mtvY29y1vKe5rXN5x/4W1u1ooL65jbZuah1lxfmMLS1g7PBCKkoLqBheSMXwAspL/XxMSQFF+VGK8qIU5kWJRtKNhygSXgoCyYqCWJSK4VEqhvfuktWOWsee5jbqm9vY09xKbWMruxpa2dXQwu7GVnY3tPDmzkZqm1pJJLvv+yqIRTqDoXMeLBfEIhTEouTHIuRHI+THIhTE/LxjKoj5/YcX+Rv1RhTlMbzIz0sLY+RpOBDJMQoCyQmptY6JZUeudSSSjj3NbUFAtFDf1EZLe4IDbQkOtgdTWzC1d80bW+LUxZO0xRO0JZK0xZO0xv28LZ4kfoRwSVWcH+0MhuGFeRTkRYhFjFg0Ql7UiEUi5HUsd64bRfkxxqbUaCqGFzK6OF/jTEnGKQhkyIlGujq8YUS/vW8i6TpD4UB7nMaWOPsPttNwsD1lHqehpf2Q7U2tceIJR3siSXvCB0rHejwZzBOOlniCwy/iixiMLinwwVBayNjhhYwtLWBEUd4hQ4iUdEyFMUryYxQXRBUg0msKApFeikbMNyXlRxlBHuP7L2MAf09HXVNbZ1PXroYWdje0sKuhld2NLezY38KrNV2DEfakMC9CSUGMovwoxfl+Piw/yrD8WDDvWu5oIiuIBc1jeb4JrDCYd7dNfS5Dg4JAZJCIRSOMG1HY41Af7Ykkza2+RuKHFgmWWxM0tbbT1JroGmKkNc7BtgQH2uIcaEsE/SwHO9cPtiVobou/pyZyVOWOWBAUUQqDeep6R9AU5vnw6eqTiVGU5/tr8qIR4klHIun8PKgtdazHE45EMknCOYYX5nXW+MaU+HnZsHwiCqRjpiAQyTF50Qgjg3su+oNzjtZ4ktb2JC3xBK3tSVrjCVqCeWv80PWW9iSt7R3bk7S0d+3j38O/3hK8Vt/cxoG9PnRS+2qORTRiaS8EiEbMX1IcBEN5aQElBTGSriNYgnmyK2DaO8MF8qM+kIblvTe0ClO250cjWC/yxjlIOEcy6Ug4/3lJ50gk/T058WB7Mun8EDEFMUqDpj4/XEzX+kA08SkIRELOzCgMLq0dwcCMSNsRPgfbEhxoT9AeT3Z2nEcjRixiRKN+Hov4zvZIxHDO0dQap66pjdrG1mBq6VpvaqWuqZU3dzXS2BIP3tOC94wQi1rX+6e8b3s82XnhwIG2OC3tyc4h37OtKC9KSRAMN58/jctmH9fvn6EgEJEBl9IfvVUAAAY0SURBVBo+o47y7zoGWKwaU5yx8oHvs+m4yqylzV8g0N29LOlEzIdONGJdy+YDLmpGJAJR8zWcpqB5z8/bU5YP3TYqQ0PHKwhERNKIRSOURiO9GtW3r8Zm/BOOTNeXiYiEnIJARCTkFAQiIiGnIBARCTkFgYhIyCkIRERCTkEgIhJyCgIRkZDLuYfXm1kt8M5hm8cAdVkoTqYMteOBoXdMQ+14YOgd01A7HujbMU12zpWneyHngiAdM1vtnKvOdjn6y1A7Hhh6xzTUjgeG3jENteOBzB2TmoZEREJOQSAiEnJDJQjuy3YB+tlQOx4Yesc01I4Hht4xDbXjgQwd05DoIxARkWM3VGoEIiJyjBQEIiIhl9NBYGYXmdkGM9toZrdluzz9wcy2mNlaM1tjZquzXZ5jYWb3m9luM/tTyrYyM/uVmb0VzI/mwVRZ1c3xfNnMtgXf0xoz+/NslvFomNlEM3vWzNab2etmdlOwPZe/o+6OKSe/JzMrNLOVZvZqcDy3B9urzOzF4Dt61Mz65cHVOdtHYGZR4E3gAqAGWAUscs6ty2rB+sjMtgDVzrmcvRHGzOYDTcAPnXMzgm13AXucc3cGoT3KOXdrNsvZW90cz5eBJufc3dks27Ews/HAeOfcy2ZWCrwEfAi4ntz9jro7pr8kB78nMzOg2DnXZGZ5wPPATcDngCecc4+Y2XeAV51z3+7r5+VyjWAesNE5t8k51wY8Alye5TIJ4JxbAew5bPPlwIPB8oP4/0lzQjfHk7Occzuccy8Hy43AeuB4cvs76u6YcpLzmoLVvGBywELg8WB7v31HuRwExwNbU9ZryOEvPoUDfmlmL5nZDdkuTD+qcM7tAP8/Ldl/TGt/WGxmrwVNRznTjJLKzCqBucCLDJHv6LBjghz9nswsamZrgN3Ar4C3gX3OuXiwS7+d83I5CCzNttxs5zrUWc65U4CLgb8JmiVk8Pk28D5gDrAD+LfsFufomVkJ8BPgZudcQ7bL0x/SHFPOfk/OuYRzbg4wAd8C8v50u/XHZ+VyENQAE1PWJwDbs1SWfuOc2x7MdwNP4v8DGAp2Be24He25u7Ncnj5xzu0K/kdNAt8lx76noN35J8BDzrkngs05/R2lO6Zc/54AnHP7gOXAGcBIM4sFL/XbOS+Xg2AVMDXoRc8HrgGWZblMfWJmxUFHF2ZWDFwI/OnIf5UzlgHXBcvXAT/NYln6rOOEGbiCHPqego7I7wPrnXNfT3kpZ7+j7o4pV78nMys3s5HBchFwPr7f41ngqmC3fvuOcvaqIYDgUrB/B6LA/c65f8lykfrEzKbgawEAMeDhXDwmM1sCLMAPmbsL+BKwFHgMmAS8C1ztnMuJDthujmcBvrnBAVuA/93Rvj7YmdnZwO+AtUAy2PwP+Db1XP2OujumReTg92Rms/CdwVH8D/bHnHN3BOeIR4Ay4BXgY8651j5/Xi4HgYiI9F0uNw2JiEg/UBCIiIScgkBEJOQUBCIiIacgEBEJOQWBSMDMEimjVK7pzxFtzawydfRSkcEk1vMuIqFxMLilXyRUVCMQ6UHwjIivBuPDrzSzE4Ltk83sN8GAZr8xs0nB9gozezIYS/5VMzszeKuomX03GF/+l8Edo5jZZ8xsXfA+j2TpMCXEFAQiXYoOaxr6SMprDc65ecA9+LvZCZZ/6JybBTwEfDPY/k3gOefcbOAU4PVg+1TgXufcycA+4Mpg+23A3OB9bszUwYl0R3cWiwTMrMk5V5Jm+xZgoXNuUzCw2U7n3Ggzq8M/DKU92L7DOTfGzGqBCam3/gdDI//KOTc1WL8VyHPO/bOZPY1/8M1SYGnKOPQiA0I1ApHecd0sd7dPOqljwiTo6qP7C+Be4FTgpZTRJUUGhIJApHc+kjJ/IVj+A37UW4CP4h8nCPAb4NPQ+XCR4d29qZlFgInOuWeBvwNGAu+plYhkkn55iHQpCp4I1eFp51zHJaQFZvYi/sfTomDbZ4D7zewWoBb4RLD9JuA+M/sU/pf/p/EPRUknCvzYzEbgH7b0jWD8eZEBoz4CkR4EfQTVzrm6bJdFJBPUNCQiEnKqEYiIhJxqBCIiIacgEBEJOQWBiEjIKQhEREJOQSAiEnL/H5ecbws+0kKaAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# => Plot \n",
    "plt.figure()\n",
    "plt.plot(np.arange(1,epochs + 1,1),population_curve_N_1,label=\"Full\")\n",
    "plt.plot(np.arange(1,epochs + 1,1),population_curve_dropout_N_1,label=\"Dropout\")\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Risk')\n",
    "plt.legend(loc='best')\n",
    "plt.savefig(\"N50_risk.jpg\")\n",
    "plt.figure()\n",
    "plt.plot(np.arange(1,epochs + 1,1),error_curve_N_1,label=\"Full\")\n",
    "plt.plot(np.arange(1,epochs + 1,1),error_curve_dropout_N_1,label=\"Dropout\")\n",
    "plt.legend(loc='best')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Error')\n",
    "plt.savefig(\"N50_error.jpg\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get curves and network for $N=N_2$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 454,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch [1/30], Step [0/600], Loss: 2.4642\n",
      "Epoch [1/30], Step [60/600], Loss: 0.7141\n",
      "Epoch [1/30], Step [120/600], Loss: 0.5026\n",
      "Epoch [1/30], Step [180/600], Loss: 0.4143\n",
      "Epoch [1/30], Step [240/600], Loss: 0.3485\n",
      "Epoch [1/30], Step [300/600], Loss: 0.3523\n",
      "Epoch [1/30], Step [360/600], Loss: 0.2535\n",
      "Epoch [1/30], Step [420/600], Loss: 0.2748\n",
      "Epoch [1/30], Step [480/600], Loss: 0.2884\n",
      "Epoch [1/30], Step [540/600], Loss: 0.4073\n",
      "92.54\n",
      "89.6\n",
      "Epoch [2/30], Step [0/600], Loss: 0.2130\n",
      "Epoch [2/30], Step [60/600], Loss: 0.3484\n",
      "Epoch [2/30], Step [120/600], Loss: 0.2476\n",
      "Epoch [2/30], Step [180/600], Loss: 0.3082\n",
      "Epoch [2/30], Step [240/600], Loss: 0.2398\n",
      "Epoch [2/30], Step [300/600], Loss: 0.3164\n",
      "Epoch [2/30], Step [360/600], Loss: 0.1602\n",
      "Epoch [2/30], Step [420/600], Loss: 0.1459\n",
      "Epoch [2/30], Step [480/600], Loss: 0.3009\n",
      "Epoch [2/30], Step [540/600], Loss: 0.3006\n",
      "93.77\n",
      "91.51\n",
      "Epoch [3/30], Step [0/600], Loss: 0.2624\n",
      "Epoch [3/30], Step [60/600], Loss: 0.2843\n",
      "Epoch [3/30], Step [120/600], Loss: 0.2376\n",
      "Epoch [3/30], Step [180/600], Loss: 0.2475\n",
      "Epoch [3/30], Step [240/600], Loss: 0.1860\n",
      "Epoch [3/30], Step [300/600], Loss: 0.1720\n",
      "Epoch [3/30], Step [360/600], Loss: 0.2640\n",
      "Epoch [3/30], Step [420/600], Loss: 0.1421\n",
      "Epoch [3/30], Step [480/600], Loss: 0.2289\n",
      "Epoch [3/30], Step [540/600], Loss: 0.2296\n",
      "94.57\n",
      "92.76\n",
      "Epoch [4/30], Step [0/600], Loss: 0.1960\n",
      "Epoch [4/30], Step [60/600], Loss: 0.2703\n",
      "Epoch [4/30], Step [120/600], Loss: 0.1841\n",
      "Epoch [4/30], Step [180/600], Loss: 0.1085\n",
      "Epoch [4/30], Step [240/600], Loss: 0.2096\n",
      "Epoch [4/30], Step [300/600], Loss: 0.2492\n",
      "Epoch [4/30], Step [360/600], Loss: 0.2110\n",
      "Epoch [4/30], Step [420/600], Loss: 0.2233\n",
      "Epoch [4/30], Step [480/600], Loss: 0.2424\n",
      "Epoch [4/30], Step [540/600], Loss: 0.0502\n",
      "94.97\n",
      "93.42\n",
      "Epoch [5/30], Step [0/600], Loss: 0.3056\n",
      "Epoch [5/30], Step [60/600], Loss: 0.2613\n",
      "Epoch [5/30], Step [120/600], Loss: 0.1150\n",
      "Epoch [5/30], Step [180/600], Loss: 0.2169\n",
      "Epoch [5/30], Step [240/600], Loss: 0.2879\n",
      "Epoch [5/30], Step [300/600], Loss: 0.2834\n",
      "Epoch [5/30], Step [360/600], Loss: 0.1607\n",
      "Epoch [5/30], Step [420/600], Loss: 0.1843\n",
      "Epoch [5/30], Step [480/600], Loss: 0.1546\n",
      "Epoch [5/30], Step [540/600], Loss: 0.2135\n",
      "95.43\n",
      "93.57\n",
      "Epoch [6/30], Step [0/600], Loss: 0.1922\n",
      "Epoch [6/30], Step [60/600], Loss: 0.1113\n",
      "Epoch [6/30], Step [120/600], Loss: 0.1187\n",
      "Epoch [6/30], Step [180/600], Loss: 0.2001\n",
      "Epoch [6/30], Step [240/600], Loss: 0.1371\n",
      "Epoch [6/30], Step [300/600], Loss: 0.2044\n",
      "Epoch [6/30], Step [360/600], Loss: 0.2236\n",
      "Epoch [6/30], Step [420/600], Loss: 0.1314\n",
      "Epoch [6/30], Step [480/600], Loss: 0.2259\n",
      "Epoch [6/30], Step [540/600], Loss: 0.0892\n",
      "95.72\n",
      "94.08\n",
      "Epoch [7/30], Step [0/600], Loss: 0.1733\n",
      "Epoch [7/30], Step [60/600], Loss: 0.0897\n",
      "Epoch [7/30], Step [120/600], Loss: 0.1888\n",
      "Epoch [7/30], Step [180/600], Loss: 0.1372\n",
      "Epoch [7/30], Step [240/600], Loss: 0.2107\n",
      "Epoch [7/30], Step [300/600], Loss: 0.0988\n",
      "Epoch [7/30], Step [360/600], Loss: 0.1454\n",
      "Epoch [7/30], Step [420/600], Loss: 0.1335\n",
      "Epoch [7/30], Step [480/600], Loss: 0.1746\n",
      "Epoch [7/30], Step [540/600], Loss: 0.0635\n",
      "95.99\n",
      "94.56\n",
      "Epoch [8/30], Step [0/600], Loss: 0.1035\n",
      "Epoch [8/30], Step [60/600], Loss: 0.1846\n",
      "Epoch [8/30], Step [120/600], Loss: 0.0960\n",
      "Epoch [8/30], Step [180/600], Loss: 0.0945\n",
      "Epoch [8/30], Step [240/600], Loss: 0.1637\n",
      "Epoch [8/30], Step [300/600], Loss: 0.1642\n",
      "Epoch [8/30], Step [360/600], Loss: 0.1364\n",
      "Epoch [8/30], Step [420/600], Loss: 0.0799\n",
      "Epoch [8/30], Step [480/600], Loss: 0.0702\n",
      "Epoch [8/30], Step [540/600], Loss: 0.2091\n",
      "96.08\n",
      "94.9\n",
      "Epoch [9/30], Step [0/600], Loss: 0.0524\n",
      "Epoch [9/30], Step [60/600], Loss: 0.0749\n",
      "Epoch [9/30], Step [120/600], Loss: 0.1034\n",
      "Epoch [9/30], Step [180/600], Loss: 0.1161\n",
      "Epoch [9/30], Step [240/600], Loss: 0.1151\n",
      "Epoch [9/30], Step [300/600], Loss: 0.1499\n",
      "Epoch [9/30], Step [360/600], Loss: 0.0773\n",
      "Epoch [9/30], Step [420/600], Loss: 0.1199\n",
      "Epoch [9/30], Step [480/600], Loss: 0.0500\n",
      "Epoch [9/30], Step [540/600], Loss: 0.1340\n",
      "96.31\n",
      "94.89\n",
      "Epoch [10/30], Step [0/600], Loss: 0.1251\n",
      "Epoch [10/30], Step [60/600], Loss: 0.1101\n",
      "Epoch [10/30], Step [120/600], Loss: 0.1263\n",
      "Epoch [10/30], Step [180/600], Loss: 0.1315\n",
      "Epoch [10/30], Step [240/600], Loss: 0.0919\n",
      "Epoch [10/30], Step [300/600], Loss: 0.1383\n",
      "Epoch [10/30], Step [360/600], Loss: 0.1028\n",
      "Epoch [10/30], Step [420/600], Loss: 0.1693\n",
      "Epoch [10/30], Step [480/600], Loss: 0.0767\n",
      "Epoch [10/30], Step [540/600], Loss: 0.0455\n",
      "96.67\n",
      "95.27\n",
      "Epoch [11/30], Step [0/600], Loss: 0.0538\n",
      "Epoch [11/30], Step [60/600], Loss: 0.1175\n",
      "Epoch [11/30], Step [120/600], Loss: 0.0972\n",
      "Epoch [11/30], Step [180/600], Loss: 0.0811\n",
      "Epoch [11/30], Step [240/600], Loss: 0.0988\n",
      "Epoch [11/30], Step [300/600], Loss: 0.1054\n",
      "Epoch [11/30], Step [360/600], Loss: 0.0751\n",
      "Epoch [11/30], Step [420/600], Loss: 0.1259\n",
      "Epoch [11/30], Step [480/600], Loss: 0.0829\n",
      "Epoch [11/30], Step [540/600], Loss: 0.1520\n",
      "96.69\n",
      "95.23\n",
      "Epoch [12/30], Step [0/600], Loss: 0.0730\n",
      "Epoch [12/30], Step [60/600], Loss: 0.0504\n",
      "Epoch [12/30], Step [120/600], Loss: 0.0736\n",
      "Epoch [12/30], Step [180/600], Loss: 0.0296\n",
      "Epoch [12/30], Step [240/600], Loss: 0.1670\n",
      "Epoch [12/30], Step [300/600], Loss: 0.0907\n",
      "Epoch [12/30], Step [360/600], Loss: 0.0851\n",
      "Epoch [12/30], Step [420/600], Loss: 0.1176\n",
      "Epoch [12/30], Step [480/600], Loss: 0.0512\n",
      "Epoch [12/30], Step [540/600], Loss: 0.1047\n",
      "96.81\n",
      "95.46\n",
      "Epoch [13/30], Step [0/600], Loss: 0.1224\n",
      "Epoch [13/30], Step [60/600], Loss: 0.1453\n",
      "Epoch [13/30], Step [120/600], Loss: 0.1069\n",
      "Epoch [13/30], Step [180/600], Loss: 0.1123\n",
      "Epoch [13/30], Step [240/600], Loss: 0.1354\n",
      "Epoch [13/30], Step [300/600], Loss: 0.0371\n",
      "Epoch [13/30], Step [360/600], Loss: 0.0451\n",
      "Epoch [13/30], Step [420/600], Loss: 0.1346\n",
      "Epoch [13/30], Step [480/600], Loss: 0.0587\n",
      "Epoch [13/30], Step [540/600], Loss: 0.1201\n",
      "96.87\n",
      "95.33\n",
      "Epoch [14/30], Step [0/600], Loss: 0.0973\n",
      "Epoch [14/30], Step [60/600], Loss: 0.0784\n",
      "Epoch [14/30], Step [120/600], Loss: 0.0895\n",
      "Epoch [14/30], Step [180/600], Loss: 0.0571\n",
      "Epoch [14/30], Step [240/600], Loss: 0.0852\n",
      "Epoch [14/30], Step [300/600], Loss: 0.0665\n",
      "Epoch [14/30], Step [360/600], Loss: 0.1166\n",
      "Epoch [14/30], Step [420/600], Loss: 0.0932\n",
      "Epoch [14/30], Step [480/600], Loss: 0.0282\n",
      "Epoch [14/30], Step [540/600], Loss: 0.1010\n",
      "97.0\n",
      "95.44\n",
      "Epoch [15/30], Step [0/600], Loss: 0.1981\n",
      "Epoch [15/30], Step [60/600], Loss: 0.0655\n",
      "Epoch [15/30], Step [120/600], Loss: 0.0482\n",
      "Epoch [15/30], Step [180/600], Loss: 0.1075\n",
      "Epoch [15/30], Step [240/600], Loss: 0.0731\n",
      "Epoch [15/30], Step [300/600], Loss: 0.1385\n",
      "Epoch [15/30], Step [360/600], Loss: 0.0561\n",
      "Epoch [15/30], Step [420/600], Loss: 0.0738\n",
      "Epoch [15/30], Step [480/600], Loss: 0.0543\n",
      "Epoch [15/30], Step [540/600], Loss: 0.0821\n",
      "97.12\n",
      "95.82\n",
      "Epoch [16/30], Step [0/600], Loss: 0.1149\n",
      "Epoch [16/30], Step [60/600], Loss: 0.0587\n",
      "Epoch [16/30], Step [120/600], Loss: 0.0330\n",
      "Epoch [16/30], Step [180/600], Loss: 0.1206\n",
      "Epoch [16/30], Step [240/600], Loss: 0.0349\n",
      "Epoch [16/30], Step [300/600], Loss: 0.0733\n",
      "Epoch [16/30], Step [360/600], Loss: 0.1139\n",
      "Epoch [16/30], Step [420/600], Loss: 0.0847\n",
      "Epoch [16/30], Step [480/600], Loss: 0.0371\n",
      "Epoch [16/30], Step [540/600], Loss: 0.1001\n",
      "97.22\n",
      "95.81\n",
      "Epoch [17/30], Step [0/600], Loss: 0.0345\n",
      "Epoch [17/30], Step [60/600], Loss: 0.1393\n",
      "Epoch [17/30], Step [120/600], Loss: 0.0627\n",
      "Epoch [17/30], Step [180/600], Loss: 0.1087\n",
      "Epoch [17/30], Step [240/600], Loss: 0.0821\n",
      "Epoch [17/30], Step [300/600], Loss: 0.0866\n",
      "Epoch [17/30], Step [360/600], Loss: 0.0888\n",
      "Epoch [17/30], Step [420/600], Loss: 0.0263\n",
      "Epoch [17/30], Step [480/600], Loss: 0.0465\n",
      "Epoch [17/30], Step [540/600], Loss: 0.1063\n",
      "97.23\n",
      "95.69\n",
      "Epoch [18/30], Step [0/600], Loss: 0.0573\n",
      "Epoch [18/30], Step [60/600], Loss: 0.0449\n",
      "Epoch [18/30], Step [120/600], Loss: 0.0845\n",
      "Epoch [18/30], Step [180/600], Loss: 0.0456\n",
      "Epoch [18/30], Step [240/600], Loss: 0.0979\n",
      "Epoch [18/30], Step [300/600], Loss: 0.1114\n",
      "Epoch [18/30], Step [360/600], Loss: 0.0893\n",
      "Epoch [18/30], Step [420/600], Loss: 0.0459\n",
      "Epoch [18/30], Step [480/600], Loss: 0.0577\n",
      "Epoch [18/30], Step [540/600], Loss: 0.0301\n",
      "97.32\n",
      "95.82\n",
      "Epoch [19/30], Step [0/600], Loss: 0.1217\n",
      "Epoch [19/30], Step [60/600], Loss: 0.0452\n",
      "Epoch [19/30], Step [120/600], Loss: 0.0447\n",
      "Epoch [19/30], Step [180/600], Loss: 0.0887\n",
      "Epoch [19/30], Step [240/600], Loss: 0.0554\n",
      "Epoch [19/30], Step [300/600], Loss: 0.0658\n",
      "Epoch [19/30], Step [360/600], Loss: 0.0298\n",
      "Epoch [19/30], Step [420/600], Loss: 0.1798\n",
      "Epoch [19/30], Step [480/600], Loss: 0.0266\n",
      "Epoch [19/30], Step [540/600], Loss: 0.0428\n",
      "97.47\n",
      "96.0\n",
      "Epoch [20/30], Step [0/600], Loss: 0.0393\n",
      "Epoch [20/30], Step [60/600], Loss: 0.0629\n",
      "Epoch [20/30], Step [120/600], Loss: 0.0285\n",
      "Epoch [20/30], Step [180/600], Loss: 0.0423\n",
      "Epoch [20/30], Step [240/600], Loss: 0.0387\n",
      "Epoch [20/30], Step [300/600], Loss: 0.0135\n",
      "Epoch [20/30], Step [360/600], Loss: 0.0532\n",
      "Epoch [20/30], Step [420/600], Loss: 0.0915\n",
      "Epoch [20/30], Step [480/600], Loss: 0.0512\n",
      "Epoch [20/30], Step [540/600], Loss: 0.1037\n",
      "97.43\n",
      "95.94\n",
      "Epoch [21/30], Step [0/600], Loss: 0.1145\n",
      "Epoch [21/30], Step [60/600], Loss: 0.0188\n",
      "Epoch [21/30], Step [120/600], Loss: 0.0472\n",
      "Epoch [21/30], Step [180/600], Loss: 0.0286\n",
      "Epoch [21/30], Step [240/600], Loss: 0.0502\n",
      "Epoch [21/30], Step [300/600], Loss: 0.0872\n",
      "Epoch [21/30], Step [360/600], Loss: 0.0778\n",
      "Epoch [21/30], Step [420/600], Loss: 0.0489\n",
      "Epoch [21/30], Step [480/600], Loss: 0.0886\n",
      "Epoch [21/30], Step [540/600], Loss: 0.0534\n",
      "97.5\n",
      "96.14\n",
      "Epoch [22/30], Step [0/600], Loss: 0.0285\n",
      "Epoch [22/30], Step [60/600], Loss: 0.0631\n",
      "Epoch [22/30], Step [120/600], Loss: 0.0276\n",
      "Epoch [22/30], Step [180/600], Loss: 0.0510\n",
      "Epoch [22/30], Step [240/600], Loss: 0.0534\n",
      "Epoch [22/30], Step [300/600], Loss: 0.0440\n",
      "Epoch [22/30], Step [360/600], Loss: 0.0492\n",
      "Epoch [22/30], Step [420/600], Loss: 0.0377\n",
      "Epoch [22/30], Step [480/600], Loss: 0.0758\n",
      "Epoch [22/30], Step [540/600], Loss: 0.0419\n",
      "97.51\n",
      "96.18\n",
      "Epoch [23/30], Step [0/600], Loss: 0.0599\n",
      "Epoch [23/30], Step [60/600], Loss: 0.0613\n",
      "Epoch [23/30], Step [120/600], Loss: 0.0785\n",
      "Epoch [23/30], Step [180/600], Loss: 0.0292\n",
      "Epoch [23/30], Step [240/600], Loss: 0.0577\n",
      "Epoch [23/30], Step [300/600], Loss: 0.0323\n",
      "Epoch [23/30], Step [360/600], Loss: 0.0285\n",
      "Epoch [23/30], Step [420/600], Loss: 0.0605\n",
      "Epoch [23/30], Step [480/600], Loss: 0.0591\n",
      "Epoch [23/30], Step [540/600], Loss: 0.0518\n",
      "97.55\n",
      "96.11\n",
      "Epoch [24/30], Step [0/600], Loss: 0.0273\n",
      "Epoch [24/30], Step [60/600], Loss: 0.0525\n",
      "Epoch [24/30], Step [120/600], Loss: 0.0689\n",
      "Epoch [24/30], Step [180/600], Loss: 0.0438\n",
      "Epoch [24/30], Step [240/600], Loss: 0.0555\n",
      "Epoch [24/30], Step [300/600], Loss: 0.0573\n",
      "Epoch [24/30], Step [360/600], Loss: 0.0535\n",
      "Epoch [24/30], Step [420/600], Loss: 0.0472\n",
      "Epoch [24/30], Step [480/600], Loss: 0.0363\n",
      "Epoch [24/30], Step [540/600], Loss: 0.0608\n",
      "97.64\n",
      "96.25\n",
      "Epoch [25/30], Step [0/600], Loss: 0.0431\n",
      "Epoch [25/30], Step [60/600], Loss: 0.0345\n",
      "Epoch [25/30], Step [120/600], Loss: 0.0454\n",
      "Epoch [25/30], Step [180/600], Loss: 0.0593\n",
      "Epoch [25/30], Step [240/600], Loss: 0.0892\n",
      "Epoch [25/30], Step [300/600], Loss: 0.0366\n",
      "Epoch [25/30], Step [360/600], Loss: 0.0538\n",
      "Epoch [25/30], Step [420/600], Loss: 0.1160\n",
      "Epoch [25/30], Step [480/600], Loss: 0.0451\n",
      "Epoch [25/30], Step [540/600], Loss: 0.0388\n",
      "97.53\n",
      "96.26\n",
      "Epoch [26/30], Step [0/600], Loss: 0.0494\n",
      "Epoch [26/30], Step [60/600], Loss: 0.0596\n",
      "Epoch [26/30], Step [120/600], Loss: 0.0163\n",
      "Epoch [26/30], Step [180/600], Loss: 0.0455\n",
      "Epoch [26/30], Step [240/600], Loss: 0.0722\n",
      "Epoch [26/30], Step [300/600], Loss: 0.0478\n",
      "Epoch [26/30], Step [360/600], Loss: 0.0234\n",
      "Epoch [26/30], Step [420/600], Loss: 0.0260\n",
      "Epoch [26/30], Step [480/600], Loss: 0.0332\n",
      "Epoch [26/30], Step [540/600], Loss: 0.0255\n",
      "97.71\n",
      "96.22\n",
      "Epoch [27/30], Step [0/600], Loss: 0.0521\n",
      "Epoch [27/30], Step [60/600], Loss: 0.0631\n",
      "Epoch [27/30], Step [120/600], Loss: 0.0404\n",
      "Epoch [27/30], Step [180/600], Loss: 0.0372\n",
      "Epoch [27/30], Step [240/600], Loss: 0.0378\n",
      "Epoch [27/30], Step [300/600], Loss: 0.0820\n",
      "Epoch [27/30], Step [360/600], Loss: 0.0456\n",
      "Epoch [27/30], Step [420/600], Loss: 0.0512\n",
      "Epoch [27/30], Step [480/600], Loss: 0.0305\n",
      "Epoch [27/30], Step [540/600], Loss: 0.0758\n",
      "97.69\n",
      "96.33\n",
      "Epoch [28/30], Step [0/600], Loss: 0.0453\n",
      "Epoch [28/30], Step [60/600], Loss: 0.0442\n",
      "Epoch [28/30], Step [120/600], Loss: 0.0744\n",
      "Epoch [28/30], Step [180/600], Loss: 0.0810\n",
      "Epoch [28/30], Step [240/600], Loss: 0.0688\n",
      "Epoch [28/30], Step [300/600], Loss: 0.0344\n",
      "Epoch [28/30], Step [360/600], Loss: 0.0329\n",
      "Epoch [28/30], Step [420/600], Loss: 0.1005\n",
      "Epoch [28/30], Step [480/600], Loss: 0.0293\n",
      "Epoch [28/30], Step [540/600], Loss: 0.0396\n",
      "97.81\n",
      "96.26\n",
      "Epoch [29/30], Step [0/600], Loss: 0.0372\n",
      "Epoch [29/30], Step [60/600], Loss: 0.0464\n",
      "Epoch [29/30], Step [120/600], Loss: 0.0296\n",
      "Epoch [29/30], Step [180/600], Loss: 0.0524\n",
      "Epoch [29/30], Step [240/600], Loss: 0.0271\n",
      "Epoch [29/30], Step [300/600], Loss: 0.0275\n",
      "Epoch [29/30], Step [360/600], Loss: 0.1160\n",
      "Epoch [29/30], Step [420/600], Loss: 0.0393\n",
      "Epoch [29/30], Step [480/600], Loss: 0.0514\n",
      "Epoch [29/30], Step [540/600], Loss: 0.0196\n",
      "97.82\n",
      "96.38\n",
      "Epoch [30/30], Step [0/600], Loss: 0.0612\n",
      "Epoch [30/30], Step [60/600], Loss: 0.0314\n",
      "Epoch [30/30], Step [120/600], Loss: 0.0454\n",
      "Epoch [30/30], Step [180/600], Loss: 0.0382\n",
      "Epoch [30/30], Step [240/600], Loss: 0.0262\n",
      "Epoch [30/30], Step [300/600], Loss: 0.0760\n",
      "Epoch [30/30], Step [360/600], Loss: 0.0200\n",
      "Epoch [30/30], Step [420/600], Loss: 0.0308\n",
      "Epoch [30/30], Step [480/600], Loss: 0.0340\n",
      "Epoch [30/30], Step [540/600], Loss: 0.0287\n",
      "97.73\n",
      "96.41\n"
     ]
    }
   ],
   "source": [
    "N_2 = 2000 # paste your group N_2 value\n",
    "learning_rate=0.001\n",
    "epochs=30\n",
    "population_curve_N_2, error_curve_N_2, \\\n",
    "population_curve_dropout_N_2, error_curve_dropout_N_2, model_N_2 = get_curves_and_model(N_2,learning_rate,epochs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the same figures but for $N=N_2$. **Did full network and dropout network become closer?**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 564,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEGCAYAAACKB4k+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3deXgUZbrw/++dTsi+k4RAEgJhCyCbCAiKgMK4oIA76ii+LqOjo45zHGd7j3p+s3g8Hp1xHJ3BV8cdxR3XcQUHQRCQRYgoe0KAhBCykIQs/fz+eCoxQCBbdyqd3J/rqqurq6ur76K17zy7GGNQSimlAILcDkAppVTnoUlBKaVUA00KSimlGmhSUEop1UCTglJKqQbBbgfQHj179jSZmZluh6GUUgFl9erV+40xSU29FtBJITMzk1WrVrkdhlJKBRQR2Xm817T6SCmlVANNCkoppRpoUlBKKdUgoNsUlFLdT01NDXl5eVRVVbkdSqcXFhZGWloaISEhLX6PJgWlVEDJy8sjOjqazMxMRMTtcDotYwxFRUXk5eXRr1+/Fr9Pq4+UUgGlqqqKxMRETQjNEBESExNbXaLSpKCUCjiaEFqmLf9O3TMpFOTAv34LNVonqZRSjXXPpHBwFyx/FHK/dDsSpVQA8ng8jBo1qmHbsWPHCc/PzMxk//79AERFRXVAhG3XPRua+06CoBDY+hn0n+J2NEqpABMeHs7atWvdDsMvumdJITQK0sfBts/cjkQp1UU8/fTT3HrrrQ3PZ86cyeLFi90LqI26Z0kBoP9U+OwPcKgIIhPdjkYp1Qb3vb2RTfmlPr3m0N4x3HP+sBOeU1lZyahRowDo168fb7zxhk9jcFP3TQpZ0+Cz38P2xTD8IrejUUoFkK5cfdR9k0LvURAWB1s/1aSgVIBq7i/6jhQcHIzX6214HqgjrrtnmwJAkAf6TYati8EYt6NRSgW4zMxM1q5di9frJTc3l5UrV7odUpt035ICQNZUyFkERVug50C3o1FKBbBJkybRr18/TjrpJIYPH86YMWPcDqlNundS6D/VPm79TJOCUqrFysvLjzkmIrzwwgtNnt94HENT7+1Mum/1EUBCP4jP1K6pSinl6N5JAWxpYfu/oa7G7UiUUsp1mhSypkJ1Gexe7XYkSinlOk0K/SaDBNl2BaWU6uY0KYTHQ+/R2q6glFJoUrD6T4W8VVBV4nYkSinlKk0KYNsVTB3sWOp2JEqpAFA/dfawYcMYOXIkDz300BGjmTvam2++yaZNm3xyLU0KAGnjICRS2xWUUi1SP/fRxo0b+eijj3jvvfe47777jjmvtra2Q+LRpOBrwT0gc5K2KyilWi05OZn58+fz6KOPYozh6aef5pJLLuH8889nxowZGGO46667GD58OCeddBIvv/wyAIsXL2by5MnMmTOHoUOHctNNNzWUNhYsWNAwMvruu+9u+KzGC/S8+uqrzJs3j2XLlrFo0SLuuusuRo0axdatW9t1P917RHNj/afC9x/aVdniMtyORinVEu//CvZu8O01e50E59zfqrf0798fr9dLQUEBAMuXL2f9+vUkJCTw2muvsXbtWtatW8f+/fs55ZRTmDx5MgArV65k06ZN9O3bl7PPPpvXX3+diRMncvfdd7N69Wri4+OZMWMGb775JrNnz27ysydOnMgFF1zAzJkzufjii9t372hJ4QdZjaa8UEqpVjKNJtacPn06CQkJACxdupS5c+fi8XhISUnhjDPO4KuvvgJg3Lhx9O/fH4/Hw9y5c1m6dClfffUVU6ZMISkpieDgYK688ko+//zzDrsPLSnUSxoC0am2Cunka9yORinVEq38i95ftm3bhsfjITk5GYDIyMiG18wJZmEWkWOet/R8f03NrSWFeiK2CmnbEnCxF4FSKrAUFhZy0003ceuttx7zIw8wefJkXn75Zerq6igsLOTzzz9n3LhxgK0+2r59O16vl5dffpnTTjuN8ePHs2TJEvbv309dXR0LFizgjDPOACAlJYWcnBy8Xu8Rq71FR0dTVlbmk/vRpNBY1lSoPAB717kdiVKqE6tfjnPYsGGcddZZzJgxg3vuuafJc+fMmcOIESMYOXIk06ZN44EHHqBXr14AnHrqqfzqV79i+PDh9OvXjzlz5pCamsqf/vQnpk6dysiRIxkzZgyzZs0C4P7772fmzJlMmzaN1NTUhs+4/PLL+Z//+R9Gjx7d7oZmOVFRpbMbO3asWbVqle8uWF4ADw6EM++B0+/03XWVUj6Tk5NDdna222G02+LFi3nwwQd55513/Po5Tf17ichqY8zYps7XkkJjUcmQMly7piqlui2/JQURCRORlSKyTkQ2ish9zvF+IrJCRL4XkZdFpIdzPNR5vsV5PdNfsZ1Q/ymw60uornDl45VS3cOUKVP8XkpoC3+WFA4D04wxI4FRwNkiMgH4b+BhY8xAoBi4zjn/OqDYGDMAeNg5r+NlTYW6ati1zJWPV0o1L5CrvTtSW/6d/JYUjFW/7lyIsxlgGvCqc/wZoH5ExiznOc7rZ0pTTfn+ljERPD10vIJSnVRYWBhFRUWaGJphjKGoqIiwsLBWvc+v4xRExAOsBgYAfwO2AgeNMfUTguQBfZz9PkAugDGmVkRKgERg/1HXvBG4ESAjww8jj3tEQMYE2LbY99dWSrVbWloaeXl5FBYWuh1KpxcWFkZaWlqr3uPXpGCMqQNGiUgc8AbQVJeB+nTfVKngmD8FjDHzgflgex/5KNQj9Z8Kn9xneyNFJfvlI5RSbRMSEkK/fv3cDqPL6pDeR8aYg8BiYAIQJyL1ySgNyHf284B0AOf1WOBAR8R3jPopL7S0oJTqZvzZ+yjJKSEgIuHAWUAO8BlQP2vTNcBbzv4i5znO658atyoNe42E8ARtV1BKdTv+rD5KBZ5x2hWCgIXGmHdEZBPwkoj8HvgaeNI5/0ngORHZgi0hXO7H2E4sKAj6nwFbPwVj7BQYSinVDfgtKRhj1gOjmzi+DRjXxPEq4BJ/xdNq/afCxjeg8FtIDvzRk0op1RI6ovl4dCptpVQ3pEnheOIyICFLp7xQSnUrmhROJGsa7PgCaqvdjkQppTqEJoUTyZoKNYcgb6XbkSilVIfQpHAimaeBeLRdQSnVbWhSOJGwWEgfD9994HYkSinVITQpNCf7fNj3DRS1bzUjpZQKBJoUmpM90z7mLHI3DqWU6gCaFJoTlwG9R8MmTQpKqa5Pk0JLZF8A+WvgYK7bkSillF9pUmiJobPs47edb+k8pZTyJU0KLZGYBcnDtApJKdXlaVJoqaEXwK7lULbP7UiUUspvNCm0VPYFgNEqJKVUl6ZJoaWSsyFxgHZNVUp1aZoUWkrElha2/xsq3FklVCml/E2TQmtknw+mDja/53YkSinlF5oUWqP3aIjN0F5ISqkuS5NCa4jY0sK2z6Cq1O1olFLK5zQptNbQC6CuGr7/0O1IlFLK5zQptFbaOIjqBZvecjsSpZTyOU0KrRUUZGdO3fIxVFe4HY1SSvmUJoW2yL4AaipsYlBKqS5Ek0Jb9J0E4Qk6kE0p1eVoUmgLTzAMOQ+++xfUHnY7GqWU8hlNCm2VfQEcLoVti92ORCmlfEaTQlv1PwNCY3Qgm1KqS9Gk0FbBoTDobNj8LtTVuB2NUkr5hCaF9hh6AVQWw46lbkeilFI+4bekICLpIvKZiOSIyEYRud05fq+I7BaRtc52bqP3/FpEtojIZhH5kb9i85msMyEkAnLedjsSpZTyCX+WFGqBXxhjsoEJwC0iMtR57WFjzChnew/Aee1yYBhwNvCYiHj8GF/79YiAgdPtwjter9vRKKVUu/ktKRhj9hhj1jj7ZUAO0OcEb5kFvGSMOWyM2Q5sAcb5Kz6fyb4AyvdB7gq3I1FKqXbrkDYFEckERgP1v5y3ish6EXlKROKdY32A3EZvy6OJJCIiN4rIKhFZVVhY6MeoW2jQj8ATqgPZlFJdgt+TgohEAa8BdxhjSoHHgSxgFLAH+N/6U5t4uznmgDHzjTFjjTFjk5KS/BR1K4RGQ9Y0265gjglXKaUCil+TgoiEYBPCC8aY1wGMMfuMMXXGGC/wBD9UEeUB6Y3engbk+zM+n8k+H0pyIX+N25EopVS7+LP3kQBPAjnGmIcaHU9tdNoc4BtnfxFwuYiEikg/YCCw0l/x+dTgcyAoWAeyKaUCXrAfrz0J+DGwQUTWOsd+A8wVkVHYqqEdwE8AjDEbRWQhsAnbc+kWY0ydH+PznYgEyDzdtiucda9doU0ppQKQ35KCMWYpTbcTHHfVe2PMH4A/+Csmvxo6C965w06nPXC629EopVSb6IhmXxk5F5KGwKLboPKg29EopVSbdMuk4PUaVmwr8u1FQ8Jg9uN2zMIHv/bttZVSqoN0y6TwyupcLpv/JetyffwXfZ8xcPqdsO5F+Pa4tWRKKdVpdcukcO5JqUSFBvP0sh2+v/jkX0LKSfD27VBxwPfXV0opP+qWSSE6LISLT07jnfX5FJRV+fbiwT1gzuN29tR3f+HbayullJ91y6QAMG9iJrVew4srdvn+4r1OgjPuho2vw8Y3fH99pZTyk26bFDJ7RjJ1cDLPf7mLw7V+GA5x2s+h92h4504oL/D99ZVSyg+6bVIAW1rYX36Y9zbs8f3FPcEw++9QfQje+bnOi6SUCgjdOimcPrAnWUmR/POLHRh//GgnD4Fpv7PrLaxf6PvrK6WUj3XrpCAizJuYyfq8Er72dffUeqfeAukT4P27oDQw5vdTSnVf3TopAFw4Jo3osGCe/mKHfz4gyAOzH4PaajvaWauRlFKdWLdPCpGhwVw2Np33Nuxhb4mPu6fWS8yC6ffBlo/g6+f88xlKKeUD3T4pAFx9aiZ1xvDCip3++5BTbrAzqX7wGzjoh26wSinlA5oUgIzECM4cksKLK3ZRVeOn2bqDgmDW3wADb90KXq9/PkcppdpBk4Lj2kmZFB2q5p31fuieWi++L8z4PWxfAp/cC9UV/vsspZRqA00KjolZiQxKieKfX2z3T/fUeifPgxGXwRd/gUdGwYr5UHvYf5+nlFKtoEnBYbun9mNjfimrdhb784Pgwvlw7QeQONB2VX1kDKx+Bupq/Pe5SinVApoUGpk9ujex4SH+657aWN9TYd478OM3IToF3r4N/jbODnLzBsYqpEqprkeTQiMRPYK5/JR0Pti4l/yDlf7/QBHImgrXfwJzX4aQSHj9Bnh8Imx6S8c0KKU6XIuSgoic3MSx830fjvt+fGpfjDE8/6Ufu6ceTQQGnw0/+RwueRqMFxZeDf+YDN9/3HFxKKW6vZaWFJ4QkZPqn4jIXOB3/gnJXWnxEcwY2osFK/3YPfV4goJg2Bz46Zd2Mr2qEnjhIp1+WynVYVqaFC4GnhGRbBG5AfgpMMN/Yblr3qRMiitqeGvtbncCCPLAqLlw6ypIO8VOj3FguzuxKKW6lRYlBWPMNuBy4DVsgphhjCnxZ2BuGt8vgSG9ov03e2pLBfeAi5+y1UuvXqtdV5VSfnfCpCAiG0RkvYisB14FEoBMYIVzrEsSEa6dlMm3e8tYsd3ldZbjMmDWY5D/NXx8r7uxKKW6vOBmXp/ZIVF0QrNG9eH+97/l6S92MKF/orvBZM+E8TfBl4/Z+ZOGnOtuPEqpLuuEJQVjzE5jzE5s8tjr7PcDZgFdtvoIICzEw9xxGXy4aS+5BzrBdBTT/wtSR8KbN8PBXLejUUp1US1taH4NqBORAcCT2MTwot+i6iSumtAXEeHZ5TvcDgWCQ+Hif9qBba9dp6OflVJ+0dKk4DXG1AIXAn82xvwcSPVfWJ1D77hwZo5I5YUVuygq7wSNvIlZcMFfIHcFfPp7t6NRSnVBLU0KNc7YhKuBd5xjIf4JqXP52bSBVNXUMf/zbW6HYg2/CE6+Fr74sw5sU0r5XEuTwrXAqcAfjDHbRaQf8PyJ3iAi6SLymYjkiMhGEbndOZ4gIh+JyPfOY7xzXETkERHZ4vR4GtOeG/OVAclRzBrVh2eW76CwrBOUFgDO/hMkD4M3boRSP071rZTqdlo6TmGTMeY2Y8wC5/l2Y8z9zbytFviFMSYbmADcIiJDgV8BnxhjBgKfOM8BzgEGOtuNwOOtvhs/+dm0AVTXevnHkq1uh2KFhNvpMGoq4bXrdQI9pZTPNDdOYaHz2DBewdk2NDdOwRizxxizxtkvA3KAPtieS884pz0DzHb2ZwHPGutLIE5EOkW7Rf+kKGaP7sPzK3ZSUOandZxbK2kQnPcQ7FwKSx5wOxqlVBfRXEnhdudxJnB+o63+eYuISCYwGlgBpBhj9oBNHECyc1ofoHFfyzzn2NHXulFEVonIqsLCwpaG0G63TRtITZ3h74s7SdsC2KkwRl4BS/4bti1xOxqlVBfQ3DiF+h/vnY037A/2aS35ABGJwnZpvcMYU3qiU5sKoYmY5htjxhpjxiYlJbUkBJ/I7BnJhU5pYV9pJyktAJz7P9BzoJ1yu7zA7WiUUgGuueqjGBH5tYg8KiIznMbgnwHbgEubu7iIhGATwgvGmNedw/vqq4Wcx/pfsjwgvdHb04D81t2Of/1s2kC8XsPjiztJ2wJAaJRtX6gqgZeugKoT5V2llDqx5qqPngMGAxuA64EPsRPizTLGzDrRG0VEsAPdcowxDzV6aRFwjbN/DfBWo+NXO4lnAlBSX1LpLDISI7hoTBovrtzFnpIOWISnpVKGwUX/z86P9PyFNkEopVQbNJcU+htj5hlj/gHMBcYCM40xa1tw7UnAj4FpIrLW2c4F7gemi8j3wHTnOcB72BLIFuAJ7PTcnc6t0wbg9Roe+6wTlRYAss+3JYb8r+E5TQxKqbZpbkK8hrkUjDF1IrLd6UnULGPMUppuJwA4s4nzDXBLS67tpvSECC4Zm87LX+Vy85QseseFux3SD7LPh0ufhYXXwHNz4KrXITzO7aiUUgGkuZLCSBEpdbYyYET9voh028rrW6cNwGD422db3A7lWEPOs4lhz3qbGCoPuh2RUiqANNf7yGOMiXG2aGNMcKP9mI4KsrPpExfOpWPTWbgql7ziTjCD6tGGnAuXPQd7N8Bzs6Gy2O2IlFIBoqXTXKij3DJ1AIJ0ztICwOBz4LLnYd9GeHYWVLi8WJBSKiBoUmij3nHhXD4unVdW5XWO9RaaMvhsmxgKcjQxKKVaRJNCO/x0ygCCgoRHP+2kpQWAQT+Cy1+Ews3w7AWaGJRSJ6RJoR16xYZxxbgMXl2Tx86iQ26Hc3wDpzuJ4Tt45gI4VOR2REqpTkqTQjvdPCWL4CDhr525tAAw8CyY+yLs/86WGLTxWSnVBE0K7ZQSE8YV4zN44+vd7NjfiUsLAAPOgrkLbFXSS1dCbSdZH0Ip1WloUvCBm6dkEeIRHvn0e7dDad6AM2H247DzC3jjJ+D1uh2RUqoT0aTgA8nRYVw1vi9vfr2bD77pVNM1NW3EJXDWfbDxDfjo/7odjVKqE9Gk4CM/O3MgI9PjuPmFNTyzbIfb4TRv0u0w7kZY/ih82WkWuVNKuUyTgo/Ehofw4vUTOHNICvcs2sif3s/B6z1mOYjOQwTOvh+GzIQPfg2b3mr+PUqpLk+Tgg+F9/Dw96vGcOX4DP6xZBt3LlxLdW0nrrMP8tgpt9NOgddugJ3L3Y5IKeUyTQo+FuwJ4vezh3PXjwbz5tp8rn16JWVVNc2/0S0h4TD3JYhLhwWX27EMSqluS5OCH4gIt0wdwIOXjGTFtgNc8vflnWsJz6NFJsKVr4InBJ6/CMr2uh2RUsolmhT86OKT03hq3inkHqjgwseW8f2+Fi1F4Y6EfnDFQqgoghcugcOdOFallN9oUvCzyYOSePknp1Jd5+Wix5excnsnnnuozxi7etu+jXahnrpOXO2llPILTQodYHifWF6/eSI9o0O56skVvL+hE49lGDQDzv8zbP0E3r4DzFE9qIyxI6Eri6F0DxzYZpPIwV3uxKuU8ikxR/9PH0DGjh1rVq1a5XYYLVZ8qJrrnvmKr3MP8utzhnDD6f0ROd6KpS777E+w5H6I6wveOqithBpn4zj/zcx+HEZd0aFhKqVaT0RWG2PGNvVac2s0Kx+Kj+zBizdM4Ocvr+WP733Lmp0HeeCSEcSEhbgd2rGm/AqCQ+3qbSHhEBxmHxv2IyDEeQwOg1VPwVu3QGi0XStaKRWQtKTgAmMMTy7dzp/e/5b0+HD+duUYhvWOdTus9jlcbpf+3LPONlhnTXU7IqXUcZyopKBtCi4QEa4/vT8v3TiBypo6LnxsGQu/ynU7rPYJjYIrX4HEgXYG1tyVbkeklGoDTQouOiUzgXdvO52xmfH88rX1/Mcr66isrnM7rLYLj4cfvwFRyfDCxbD3G7cjUkq1kiYFl/WMCuXZ/zOe26YN4LU1ecx57Au2FZa7HVbbRafA1W9BSCQ8NweKtrodkVKqFTQpdAKeIOHOGYP557xT2FtaxQWPfsF7nbnbanPi+8LVb4K3Fp6dDSW7W/f+qhL44i/wyrW2oduXArgNTamOoA3Nnczug5Xc8sIa1uYe5NpJmfz6nGx6BAdo7s7/Gp4+H2JS4dr3IbLnic8vybPTeK9+BqrLbGmjrhqm/sZO9R3kaXssZfvgX7+xs8EmDYH0UyB9vJ0MMKG/nTVWqW7iRA3NmhQ6oepaL398L4enl+1gTEYc/7x2HLHhnbDbakvs+AKevxCSBsM1b0NYE72s9n4Dy/4K37xq/5IfNhsm/gxiM+Ddn9sf8vTxMOfv9ge8NbxeWP0UfPxfdqzFyMtt8slbBYdL7TkRPW1ySB9nt95joEdE++9dqU5Kk0KAentdPncuXMuE/ok8Ne8UQjwBWmL47l/w0hX2h/2q1+xYB2Ng22JY9ghs/dSWCsZcDRNuttVP9YyBDa/Au/9hq6N+9Ac4eV7L/rLfu8GOyt69CvpNhvMehp4D7GveOrtWde4KyPvK9pYqcpZTFQ/0Gg6T7oDhF/r6X0Mp12lSCGALV+Xyy1fXc8X4DP4we3jnHQHdnA2vwmvXw8DpcNIlNhns3QBRKTD+J3DytRCRcPz3l+TBmz+F7Utg4Ay44K8Q3avpcw+Xw+I/2aqo8Hj40R9hxKXNJ5KKAz8kiM3vQ2EOXPIMDL2g7fetVCekSSHAPfDBtzy2eCu/Oy+b609vZfVJZ7LqKXjn53a/5yBbRTTiMjtyuiW8XvjqCfjoP21pY+bDMGzOked8+y6890sozbMlirPutYmhtXQwnurCXEkKIvIUMBMoMMYMd47dC9wAFDqn/cYY857z2q+B64A64DZjzL+a+4zukhS8XsOtC9bw/jd7mf/jsUwfmuJ2SG23aZFNAgOmQ1Abq8MKv4M3fgL5a+CkS+HcB+yP+Pt3w+Z3IXkozPwzZIxvX6yVxfDP86B4h+1NlT6ufddTqpNwKylMBsqBZ49KCuXGmAePOncosAAYB/QGPgYGGWNOOJKruyQFgMrqOi6fv5zv9pXzyk2nMrxPgE+L0V51NfDvh2DJf0Nkkl3/wXjtnE2n3mIXDPKFsn3wz7PtOhPz3rNtDUoFOFemuTDGfA60dPGAWcBLxpjDxpjtwBZsglCO8B4enrhmLAmRPbjuma/YW9KJV3LrCJ4QmHI3XP+xTQr9z4BbVsBpd/guIYAdjPfjN3Uwnuo23OjOcquIrBeRp0SkvrK3D9B48p8859gxRORGEVklIqsKCwubOqXLSo4O48l5Yzl0uI7rnvmKQ4dr3Q7JfX3GwM1LYe6CI3st+VL9YDxT17bBeEoFkI5OCo8DWcAoYA/wv87xprqFNFmvZYyZb4wZa4wZm5SU5J8oO7EhvWL46xWjydlTyu0vraXOG7gdBQJK0mDbnbay2JYYDhW5HZFSftGh6ykYY/bV74vIE8A7ztM8IL3RqWlAfgeGFlCmDk7mnvOHcc+ijdz/fg6/PW+o2yF1D71HwxUvwfMX2QF517wNYTH+/cy6WijbY7vkluTZXlUljbbS3ZA6CibfBZmT/BuL6hY6NCmISKoxpn5SnzlA/TSai4AXReQhbEPzQEDnXj6BayZmsn3/IZ7493Yye0Zy5Xg/VZ2oI2WeBpc+awfjLZgLV71qu8f6yoHtdrDe1k/hYC6U5dsG9MbC4yE2DeIyoM/JdkzF0+dCxkSY/B+QNU2n7VBt5rekICILgClATxHJA+4BpojIKGzV0A7gJwDGmI0ishDYBNQCtzTX80jB787LZmfRIf7zrY1kJERw+sDuV53mikE/gjn/sIPxFl4Dl7/QvsbtigOw8XVYv9COsAboMxb6nW5//Bu2dIjpY9euaKymEtY8B1/82ZZgeo+xJYfB52hyUK2mg9cCXPnhWi5+fBm7iytZeNOpZKf6uTpD/eCrJ+HdO2HITDsIL6G/3Voyb1JNpf0Lf/1C2PKRncIjKRtGXgbDL4a49OavcbTaali3AJY+ZMdWpAy3JYfsC1o+mWB9dVVEos7/1IXpiOYubvfBSmb/7QsOVlQzb2Imt04bGLgT6AWaL/4CH997ZBVPdCokZEFCP0jMcpJFlu3FtHu1TQSbFtmZYKNT7bQfIy61P+K++Mu+rtZOLvjv/4X939nR46f/wiYbb63THrHLVk+V5MLBRvul+baXVVCI7dnVdyL0nWTnrfJ3+4nqMJoUuoGC0ioe/HAzr6zOIy48hDvOGsQV4zMCdxK9QFJVCge2OdtW2y5QtNXuH2qi23RojP3rfcSlto2iPVOCn4i3DnIWwecPwr5v7FiLmkNHniNBtkoqNt22UcQ5VVQHd9oZbvPX2EQiQdBrhE0QmZMg49QTz1XVlLpau1ZGZbGzHWi0X2yr0SqLnVLTEDtQMGUYxGW2ffS7apImhW5kY34Jf3g3h2Vbi+ifFMlvz81m2pDkwJ1IL9A1ThjF222pYdDZvm2cbo4xtqpqy8d2EsHYdPvjH5cB0b3Bc4KmxeoKO0ngzi9g5zK7X+sMnEweahOF8dppyWuq7Gu1VUftV9rH6hOtKCh2WvXweFtaOrCdhl7pPaJsckgZZktTKcMhZSiERvvqX8j36mptIs5daUuHUUnQ9zTImADhcW5Hp0mhuzHG8ElOAX98L4dt+w8xaUAivz13KH6Z57IAABI+SURBVEN7a/FftVPtYdi95ockUfitbWQPDrdzWoWEQ3CY3ULCjjweGmN/9CMS7GPjLSz2yBJTdQUU5Ngf1n3f2DU39m2EwyU/nBOfad/b8BvmPBrTaN95SQRCImw7SUgE9Ii0W/1+48fInjZ5RvWy+y0pyVUedGbYXeFMx776h1JZZDJUHbQLRiGQOsImiLaWuHxAk0I3VVPn5YUvd/LnT76npLKGS09O5xczBpEcE+Z2aEq1njG23WPfRpskCjZCdX11mFMSbigRy5H7pg5qKmyyqT5kf7CrK+yxmorjf6Z47PTu0Sm2/ac+WUT3slVq9VOtF+Y45wdBr5NsG0z9FptmS0p5q2DHUptQc1dC3WEbW8pwmyD6ToLUkTbW2sM/bHWHbSeC2ipn39lShtsVBNtAk0I3V1JRw6Offc/Ty3YQ4gni5jOyuGFyf8JC/FSXrVQg8TrVX9UVtoqrosj2wCrb+8NWXr+/x75eLzTWWdp1gp1Ft8/Jx3YZbkpNla1W2vmFTRS5K20MrTHpdpj+X617j0OTggJgZ9Eh7n//W97/Zi/pCeHcM3MYZwXyNNxKuaG2Gsr32b/WE/r7phG8tto26hduBk8PW+XWsIWBJ/TIY55Q26bSkgTUBE0K6gjLtuznPxdtZEtBOdOGJHPP+UPpmxjpdlhKqQ7iytTZqvOaOKAn799+Or89N5sV24qY/vDnPPThZiqrdRC5Ut2dJoVuKsQTxA2T+/Ppf0zh7GG9eOTTLUx/eAkfbtxLIJcelVLto0mhm0uJCeORuaNZcMMEInp4uPG51Vz79Ffs2H+o+TcrpbocTQoKgFOzEnn3ttP53XnZrNpRzIyHP+fBf22moloX8lGqO9GGZnWMgtIq/vheDm+uzSc2PITLTknnqvF9yUjUCdKU6gq095Fqk9U7i3lq6XY+2LgXrzFMHZzM1af2ZfLAJIKCdNoMpQLViZJChy6yowLLyX3jOblvPHtLqnhxxU5eXJnLvH9+RWZiBFdN6MslY9N1NlaluhgtKagWq6718v43e3hu+U5W7SwmPMTD7NF9uPrUvrqOg1IBRKuPlM9tzC/hueU7eXPtbqpqvIzrl8Bvzs1mVLr7M0AqpU5Mk4Lym5KKGl5Zncv8z7dRWH6Yy8am88uzh5AQ2cPt0JRSx6EjmpXfxEaEcP3p/fnkF2dww+n9eXV1HlMfXMxzy3dQ5w3cPziU6q40KSifiA4L4TfnZvPBHaczvE8M//etjZz/16Ws3nnA7dCUUq2gSUH51IDkaJ6/bjyPXTmGgxXVXPT4cu5cuJaCsiq3Q1NKtYAmBeVzIsK5J6Xy8S/O4KdTsnh7XT5nPriEJ5dup6bO2/wFlFKu0YZm5XfbCsu57+1NLPmukMEp0VwzMZPhfWIYlBKtC/0o5QLtfaRcZ4zho037+P/e3UTuAbvClCdIGJAUxdDeMQzrHcPQ1BiG9o4hLkJ7LinlTzqiWblORJgxrBdnZaeQW1zBpvxSNuaXsmlPKcu27ueNr3c3nNsnLpzs1BhGpsVyzkmpDEhu2+pSSqnW05KC6hT2lx9mk5MkNuaXsim/hG37D2EMDOsdw+xRfTh/ZG96xYa5HapSAU+rj1RAKiit4p31e3hr7W7W5ZUgAuP7JTB7VB/OGZ5KbITOu6RUW2hSUAFv+/5DLFqbz1trd7Nt/yFCPMKUwcnMHtWHM7OTtcFaqVbQpKC6DGMM3+wu5a21u1m0Lp+CssNEhQYzZXASZwyyW3KMVjEpdSKuJAUReQqYCRQYY4Y7xxKAl4FMYAdwqTGmWEQE+AtwLlABzDPGrGnuMzQpdG91XsOKbUUsWpfPp98WUFB2GIAhvaI5w0kSY/sm0CNYh+Mo1ZhbSWEyUA482ygpPAAcMMbcLyK/AuKNMXeLyLnAz7BJYTzwF2PM+OY+Q5OCqmeM4du9ZSz5rpAlmwtZtfMANXWGyB4eTs3qaZPEwCRdPU4pXKw+EpFM4J1GSWEzMMUYs0dEUoHFxpjBIvIPZ3/B0eed6PqaFNTxlB+uZfnWIpZ8V8CS7wobxkb0TYxgeO9YslOjyU6NITs1htTYMGxhVanuoTONU0ip/6F3EkOyc7wPkNvovDzn2DFJQURuBG4EyMjI8G+0KmBFhQYzfWgK04emYIxhR1EFSzYXsHxbERt2l/Duhh/+04oNDzkiSQxNjWFAcpQ2XqtuqbMMXmvqz7QmizDGmPnAfLAlBX8GpboGEaFfz0j69ezHvEn9ACirqmHz3jJy9pSyaY99XLByF1U1dm4mT5CQHh9OekIEGY22dGfTZUhVV9XRSWGfiKQ2qj4qcI7nAemNzksD8js4NtWNRIeFMDYzgbGZCQ3H6ryGHUWHyNlTSs6eUnYUVZB7oIJ3N+zhYEXNEe+PDQ/5IVkkRjAoJYpBKdEMSI4iNFhLGCpwdXRSWARcA9zvPL7V6PitIvIStqG5pLn2BKV8zRMkZCVFkZUUxcwRvY94raSyhtwDFeQVV7DrQP1WyaY9pXy4aS81dabhGpmJEQzuFc3glBgG97LJom9iJJ4gbbdQnZ/fkoKILACmAD1FJA+4B5sMForIdcAu4BLn9PewPY+2YLukXuuvuJRqi9jwEGL7xDK8T+wxr9XUedmx/xDf7i3ju31lbN5bxqb8Ut7/Zi/1/ThCg4MYmBLF8N6xjM6IY0xGPFlJUQRpolCdjA5eU8pPKqpr2VJQzmYnWXy7t4z1eSWUVNqqqOiwYEalxzE6I57RGXGMTo/TGWJVh+hMvY+U6jYiegQzIi2OEWlxDce8XsP2okOs2VnM17kH+XrXQR799Hvql7PunxTJmIx4hqbGEBnqoUdwED089jE0OMg+Dw6ih8c+Dw32kBwTqj2llM9oSUEpl5UfrmV9nk0QX+8qZs2ugxw4VN3i9zdel2Joql2bIjs1hvhILXWopmlJQalOLCo0mIlZPZmY1ROwo7MPHKrmcK2Xw7Vequu3urqG5/WPVTV17DpQwcb8UpZvLTpiXYresWENiWJo7xiG9IqhT3w4IR6d9kMdnyYFpToZESExKrRN7y0qP0zOnjI27Slx1qUo5dNvCxqqpzxBQmpsGOnx9eMuwhvGXqTHR9AzqoeO7u7mNCko1YUkRoVy2sBQThvYs+FYVU0dm/faXlG5xXbsxa4DFXy6uYBCZxLBeuEhHtITwkmLjyA1NozeceH0iQund1w4qbFh9IoNa7akUV3rpejQYYrKqykst49F5YeJj+jBgJQoBiZHER2mg/86K00KSnVxYSEeRqbHMTI97pjXKqvrGsZe5B6oILe4kl0HKsg/WMnXu4opPmrQngikRIfRO84mjLiIEIoP1VBYfpj9TgKo7111IqmxYQxMiWZgchSDUqIYkBzNwJQoYjRZuE6TglLdWHgPj/1xTolu8vWK6lryD1aRf7CSPSWV7Hb28w9W8s1u2702PrIHPaNCGdIrmp5RofSMCiUxqoezbx8TIntQVF7N9wXlfLevjC0F5XxfUMYLK4oaphYB6BUTxqBe0YxKj2NMhu2uq1OKdCztfaSUck2d17C7uJLvC8r4bp9NFJvyS/luX1lDO8iA5CjGOAP+xvSNZ8AJBv3Vl3xsNVllQwmotKqGtPgI+jrTkvRNjCQzMaLbjgvRldeUUgGl/HAt63MPssbportmV3HD/FONB/0ZYxraSHKLK49pIwkLCSIt3k5gmFdcwb7SI1+PCQumb2KkTRQJEfRNjCAqNIQgsQ3+jR+DRJBGz0ODPfSJD6dXTFjATWGiSUEpFdCMMWzff6ghQazZWcx3+8oASI0N/6EnVfwPM9mmJ4STFBV6RG+qymrbhXdn0SHnsYKdzvPdxZXUelv/exgcJKTGhZEWF0FavG2kt4/hpCVEtClpVFbXsd9pp9nvNNTX79cfP39kb64c37fV8YKOU1BKBTgRoX9SFP2Torj45DTA/nAGe6RV4y7Ce3jsZIW9jm1Dqa3zsqekisqaOrzG4PWCwWAM9rnzaIxNUhXVdew+WElecQV5xZXkFVfy+feFx5RGgoOE8BAPQUG2hOEJEoLEbp4gISjIlkI8ItR6DUXlhzlUXddk/NGhwfSMDiUxsgdBfuo6rElBKRWQwnv4dmqPYE8Q6QntX661qqaOPSVVjZJFBRXVdXi9NrHUGYPXa6jz/pBo6ryGOmPwiDQ00ic1brB3EkFHTGeiSUEppXwoLMTjLOoU6XYobaLj3ZVSSjXQpKCUUqqBJgWllFINNCkopZRqoElBKaVUA00KSimlGmhSUEop1UCTglJKqQYBPfeRiBQCO4863BPY70I4/tLV7ge63j11tfuBrndPXe1+oH331NcYk9TUCwGdFJoiIquON9FTIOpq9wNd75662v1A17unrnY/4L970uojpZRSDTQpKKWUatAVk8J8twPwsa52P9D17qmr3Q90vXvqavcDfrqnLtemoJRSqu26YklBKaVUG2lSUEop1aDLJAUROVtENovIFhH5ldvx+IKI7BCRDSKyVkQCcjFqEXlKRApE5JtGxxJE5CMR+d55jHczxtY4zv3cKyK7ne9prYic62aMrSEi6SLymYjkiMhGEbndOR7I39Hx7ikgvycRCRORlSKyzrmf+5zj/URkhfMdvSwiPXzyeV2hTUFEPMB3wHQgD/gKmGuM2eRqYO0kIjuAscaYgB10IyKTgXLgWWPMcOfYA8ABY8z9TgKPN8bc7WacLXWc+7kXKDfGPOhmbG0hIqlAqjFmjYhEA6uB2cA8Avc7Ot49XUoAfk8iIkCkMaZcREKApcDtwJ3A68aYl0Tk78A6Y8zj7f28rlJSGAdsMcZsM8ZUAy8Bs1yOSQHGmM+BA0cdngU84+w/g/0fNiAc534CljFmjzFmjbNfBuQAfQjs7+h49xSQjFXuPA1xNgNMA151jvvsO+oqSaEPkNvoeR4B/B9BIwb4UERWi8iNbgfjQynGmD1g/wcGkl2OxxduFZH1TvVSwFS1NCYimcBoYAVd5Ds66p4gQL8nEfGIyFqgAPgI2AocNMbUOqf47DevqyQFaeJY4NeLwSRjzBjgHOAWp+pCdT6PA1nAKGAP8L/uhtN6IhIFvAbcYYwpdTseX2jingL2ezLG1BljRgFp2JqR7KZO88VndZWkkAekN3qeBuS7FIvPGGPynccC4A3sfwxdwT6n3re+/rfA5XjaxRizz/mf1gs8QYB9T0499WvAC8aY153DAf0dNXVPgf49ARhjDgKLgQlAnIgEOy/57DevqySFr4CBTmt8D+ByYJHLMbWLiEQ6jWSISCQwA/jmxO8KGIuAa5z9a4C3XIyl3ep/PB1zCKDvyWnEfBLIMcY81OilgP2OjndPgfo9iUiSiMQ5++HAWdh2ks+Ai53TfPYddYneRwBO97I/Ax7gKWPMH1wOqV1EpD+2dAAQDLwYiPckIguAKdhpfvcB9wBvAguBDGAXcIkxJiAab49zP1OwVRIG2AH8pL4+vrMTkdOAfwMbAK9z+DfYOvhA/Y6Od09zCcDvSURGYBuSPdg/5BcaY/7L+Y14CUgAvgauMsYcbvfndZWkoJRSqv26SvWRUkopH9CkoJRSqoEmBaWUUg00KSillGqgSUEppVQDTQpKNUFE6hrNprnWlzPvikhm41lWlepMgps/RaluqdKZVkCpbkVLCkq1grPGxX8789uvFJEBzvG+IvKJM9naJyKS4RxPEZE3nLnw14nIROdSHhF5wpkf/0NnpCoicpuIbHKu85JLt6m6MU0KSjUt/Kjqo8savVZqjBkHPIodRY+z/6wxZgTwAvCIc/wRYIkxZiQwBtjoHB8I/M0YMww4CFzkHP8VMNq5zk3+ujmljkdHNCvVBBEpN8ZENXF8BzDNGLPNmXRtrzEmUUT2Yxd2qXGO7zHG9BSRQiCt8fQDznTOHxljBjrP7wZCjDG/F5EPsIv4vAm82WgefaU6hJYUlGo9c5z9453TlMZz1NTxQ/veecDfgJOB1Y1mwVSqQ2hSUKr1Lmv0uNzZX4adnRfgSuySiQCfADdDw0IpMce7qIgEAenGmM+AXwJxwDGlFaX8Sf8KUapp4c5KV/U+MMbUd0sNFZEV2D+q5jrHbgOeEpG7gELgWuf47cB8EbkOWyK4GbvAS1M8wPMiEotdOOphZ/58pTqMtiko1QpOm8JYY8x+t2NRyh+0+kgppVQDLSkopZRqoCUFpZRSDTQpKKWUaqBJQSmlVANNCkoppRpoUlBKKdXg/weLQLdysSb3EQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3deXhU9dnw8e+dyb6QPSxJIGGJbGENi4iAiIitglZUcCm2Wmurba1Prfr0bWvt01fb+rTV6mtd64JVrAviXndEkU3ZdzBA2LIACQlZZ37vH+cEQpyQhMzkzEzuz3Wda86cOct9GJ0757eKMQallFKquTCnA1BKKRWYNEEopZTyShOEUkoprzRBKKWU8koThFJKKa/CnQ7AV9LS0kxOTo7TYSilVFBZtWpVqTEm3dtnIZMgcnJyWLlypdNhKKVUUBGRXS19pkVMSimlvNIEoZRSyitNEEoppbwKmToIpVTXU19fT1FRETU1NU6HEvCio6PJysoiIiKizcdoglBKBa2ioiISEhLIyclBRJwOJ2AZYygrK6OoqIjc3Nw2H6dFTEqpoFVTU0Nqaqomh1aICKmpqe1+0tIEoZQKapoc2uZ0/p00QRzZAx/cbb0qpZQ6ThNE7VH49H+hcInTkSilgpDL5WLEiBHHl8LCwlPun5OTQ2lpKQDx8fGdEOHp00rq9IEQlQh7voARc52ORikVZGJiYli9erXTYfiFPkGEhUH2GNiz3OlIlFIh4qmnnuLmm28+/v7CCy/k448/di6g06RPEADZ4+GjP0D1EYhJcjoapdRp+N3rG9i4r8Kn5xzcqxu/vWjIKfeprq5mxIgRAOTm5vLqq6/6NAYnaYIAyB4LGChaCQOmOR2NUiqIhHIRkyYIgMzRIC6rHkIThFJBqbW/9DtTeHg4Ho/n+Ptg7emtdRAAUfHQYyjsWeZ0JEqpEJCTk8Pq1avxeDzs2bOH5cuDs47TrwlCRGaIyBYR2S4id3j5fJKIfCkiDSIyu9ln80Rkm73M82ecgFUPUbQK3A1+v5RSKrSdddZZ5Obmkp+fzy9+8QtGjRrldEinxW9FTCLiAh4CzgOKgBUissgYs7HJbruBa4FfNDs2BfgtUAAYYJV97GF/xUv2WFj+CBxcD71G+O0ySqnQUllZ+Y1tIsJzzz3ndf+m/SS8HRtI/PkEMRbYbozZaYypA14AZjXdwRhTaIxZC3iaHXs+8J4x5pCdFN4DZvgxVug93nrVYiallAL8myAygabjVxTZ23x2rIjcICIrRWRlSUnJaQcKQGIWdMuE3V907DxKKRUi/JkgvI0MZXx5rDHmUWNMgTGmID3d65zb7ZM9TjvMKaWUzZ8JogjIbvI+C9jXCceevuxxUFEE5UV+v5RSSgU6fyaIFcAAEckVkUhgDrCojce+C0wXkWQRSQam29v8q/c461XrIZRSyn8JwhjTANyM9cO+CXjRGLNBRO4WkZkAIjJGRIqAy4BHRGSDfewh4PdYSWYFcLe9zb+6D4WIWNitCUIppfzak9oY8xbwVrNtv2myvgKr+MjbsU8CT/ozvm9wRVi9qvUJQinVRi6Xi/z8fOrr6wkPD2fevHnccssthIU50w954cKF5OXlMXjw4A6fS3tSN5c9Dg6sg9rAbp+slAoMjWMxbdiwgffee4+33nqL3/3ud9/Yr6GhczrhLly4kI0bN7a+Yxtogmiu93gwbtj3pdORKKWCTEZGBo8++igPPvggxhieeuopLrvsMi666CKmT5+OMYbbbruNoUOHkp+fz4IFCwD4+OOPmTRpEpdccgmDBw/mxhtvPD6W0/PPP09+fj5Dhw7l9ttvP36tppMNvfTSS1x77bV8/vnnLFq0iNtuu40RI0awY8eODt2PDtbXXFaB9bp7GeROcjYWpVTbvX2H9fTvSz3y4YJ723VI37598Xg8FBcXA7B06VLWrl1LSkoKL7/8MqtXr2bNmjWUlpYyZswYJk2yfmeWL1/Oxo0b6dOnDzNmzOCVV15hwoQJ3H777axatYrk5GSmT5/OwoULufjii71ee8KECcycOZMLL7yQ2bNne92nPfQJormYZEgfpPUQSqnTZsyJblvnnXceKSkpACxZsoS5c+ficrno3r07kydPZsWKFQCMHTuWvn374nK5mDt3LkuWLGHFihVMmTKF9PR0wsPDueqqq1i8eHGn3Yc+QXiTPRY2LgSPx5pxTikV+Nr5l76/7Ny5E5fLRUZGBgBxcXHHP2uaOJoTkW+8b+v+/hpOXH/9vOk9HmrKoXSL05EopYJISUkJN954IzfffPM3fvABJk2axIIFC3C73ZSUlLB48WLGjh0LWEVMX3/9NR6PhwULFjBx4kTGjRvHJ598QmlpKW63m+eff57JkycD0L17dzZt2oTH4zlpFruEhASOHj3qk/vRBOFNtt1hTsdlUkq1onHK0SFDhjBt2jSmT5/Ob3/7W6/7XnLJJQwbNozhw4czdepU/vSnP9GjRw8AzjzzTO644w6GDh1Kbm4ul1xyCT179uSee+7hnHPOYfjw4YwaNYpZs6wxT++9914uvPBCpk6dSs+ePY9fY86cOfz5z39m5MiRHa6kllM9wgSTgoICs3LlSt+czBj4c38YMB0uedg351RK+dymTZsYNGiQ02F02Mcff8x9993HG2+84dfrePv3EpFVxpgCb/vrE4Q3IvbAffoEoZTqujRBtKT3ODi0Eyo7OIy4Ukq1YsqUKX5/ejgdmiBakq0D9ykVDEKlmNzfTuffSRNES3qOAFekJgilAlh0dDRlZWWaJFphjKGsrIzo6Oh2Haf9IFoSEW0lCU0QSgWsrKwsioqK6PCMkl1AdHQ0WVlex0ZtkSaIU+k9DpY9Ag21EB7ldDRKqWYiIiLIzc11OoyQpUVMp5I9Dtx1sG+105EopVSn0wRxKlpRrZTqwjRBnEp8BiTnaoJQSnVJmiBa03u8lSC0lYRSqovRBNGa7LFQVWJ1mlNKqS5EE0Rrssdbr3uWOxuHUkp1Mk0QrUkfCFGJOi6TUqrL0QTRmrAwyB6jTxBKqS5HE0RbZI+D4k1QfcTpSJRSqtNogmiL7HGAgSIfzTehlFJBQBNEW2SOBnFpPYRSqkvRBNEWUfHQY6h2mFNKdSmaINoqexwUrQJ3g9ORKKVUp9AE0VbZ46C+Cg6udzoSpZTqFJog2koH7lNKdTGaINoqKRu6ZcJurahWSnUNmiDaI2ci7PwIGuqcjkQppfxOE0R7DL0Uqg/Djg+djkQppfxOE0R79D0HYpJh/UtOR6KUUn6nCaI9wiNh8CzY/BbUVTkdjVJK+ZUmiPbKv8xq7rrlbacjUUopv9IE0V69J0BCL1j/stORKKWUX2mCaK+wMBj6Hdj2Hhw75HQ0SinlN5ogTkf+bPDUw6bXnY5EKaX8RhPE6eg5AlL6aWsmpVRI82uCEJEZIrJFRLaLyB1ePo8SkQX258tEJMfeHiEiT4vIOhHZJCJ3+jPOdhOxniK+/hQq9jsdjVJK+YXfEoSIuICHgAuAwcBcERncbLfrgMPGmP7AX4E/2tsvA6KMMfnAaOCHjckjYAydDRjY8KrTkSillF/48wliLLDdGLPTGFMHvADMarbPLOBpe/0l4FwREcAAcSISDsQAdUCFH2Ntv/Q86DFMi5mUUiHLnwkiE9jT5H2Rvc3rPsaYBqAcSMVKFlXAfmA3cJ8x5htNhkTkBhFZKSIrS0pKfH8HrcmfDXtXQdmOzr+2Ukr5mT8ThHjZZtq4z1jADfQCcoH/EpG+39jRmEeNMQXGmIL09PSOxtt+Qy+1Xte/0vnXVkopP/NngigCspu8zwL2tbSPXZyUCBwCrgTeMcbUG2OKgc+AAj/GenoSs6yOc+v+DaZ57lNKqeDmzwSxAhggIrkiEgnMARY122cRMM9enw18aIwxWMVKU8USB4wHNvsx1tOXPxtKt+hMc0qpkOO3BGHXKdwMvAtsAl40xmwQkbtFZKa92xNAqohsB24FGpvCPgTEA+uxEs0/jTFr/RVrhwy+GMLCYZ1WViulQouYECkaKSgoMCtXrnTm4vNnQ8lm+NlaaygOpZQKEiKyyhjjtQhff818If8yKN8DRcudjkQppXxGE4QvDPwWhEdrMZNSKqRogvCFqATIm2H1qnY3OB2NUkr5hCYIX8m/DI6VwtcfOx2JUkr5hCYIXxlwHkQlajGTUipkaIIAjDF0uDVXeBQMugg2vQH11b4JTCmlHNTlE8S6onKm/eUTNh842vGT5c+GuqOw7T8dP5dSSjmsyyeIzOQYdpUdY+HqvR0/We4kiMuwht5QSqkg1+UTREpcJJPy0nl99T48ng4WM4W5YMglsPU/UFPumwCVUsohXT5BAMwa0Yt95TUsL/zGiOLtl38ZuGth85sdP5dSSjlIEwRw3uDuxEa6eM0XxUxZBZDUR4uZlFJBTxMEEBsZzvTB3Xlz7X5qG9wdO1njfNU7P4HKYt8EqJRSDtAEYZs1MpOKmgY+2eKDmemGzQEMfHB3x8+llFIO0QRhO7t/Gqlxkby2uvmcRqchPQ8m/AS+eha2f9Dx8ymllAM0QdjCXWFcOKwn7286yNGa+o6fcMp/Q1oevP4zqKno+PmUUqqTaYJoYtbITGobPLyz/kDHTxYRDbMegvIieO83HT+fUkp1Mk0QTYzMTqJ3SqxvipkAssfCmTfBqn9aldZKKRVENEE0ISLMGtGLz3eUUlxR45uTnvMrSOkHi26G2krfnFMppTqBJohmZo3IxGNg0RofPUVExlpFTUf2wAe/8805lVKqE2iCaKZ/RjxDM7v5LkEA9DkTxv0Qlj8KhUt8d16llPIjTRBeXDwik7VF5ews8WGR0Lm/geQceO1mqDvmu/MqpZSfaILw4qLhvRCBhb6qrAaIjIOZD8Lhr+HD3/vuvEop5SeaILzo3i2aCf1SeW313o5PJNRU7tkw5nr44mHY/YXvzquUUn6gCaIFs4ZnsqvsGKv3HPHtiaf9DhKz4bWbdOY5pVRA0wTRghn5PYgMD/Ndn4hGUfEw6+9Qth0++oNvz62UUj7UaoIQEZeI/Lkzggkk3aIjOHdgBm+s3UeD2+Pbk/edAqOvhaUPwZ4Vvj23Ukr5SKsJwhjjBkaLiHRCPAFl1ohMSivr+GxHme9Pft7vIaEXvPZjqPdRpzyllPKhthYxfQW8JiLXiMh3Ghd/BhYIzhmYTrfocF77ygcTCTUX3Q1m3g+lW60kcWAd+LJCXCmlOii8jfulAGXA1CbbDPCKzyMKIFHhLr6V35PX1+yjus5NTKTLtxfoPw3OugU+/zusf9ka/XXopdaSNsC311JKqXYSnzbjdFBBQYFZuXKlz8/7+Y5SrnxsGQ/MHcnM4b18fn4Aqkph42uw/hXY9RlgoEe+lSiGfAeS+/jnukqpLk9EVhljCrx91qYiJhHJEpFXRaRYRA6KyMsikuXbMAPT+NxUenSL9k8xU6O4NBhzHXzvTbh1E8y4F8Kj4f274P5h8Pg0q+9ExX7/xaCUUs20tQ7in8AioBeQCbxubwt5YWHCzBG9+GRrCYeq6vx/wW49YfyP4Pr34WdrYNpd0FAD79wBfx0CG171fwxKKUXbE0S6MeafxpgGe3kKSPdjXAFl1oheNHgMb63r5L/gk3Ng4s/hxiVw0wrIKoBXfqhNY5VSnaKtCaJURK62+0S4RORqrErrLmFwz24MyIjntdV+LGZqTXoezHkeuvWC5+fA4ULnYlFKdQltTRDfBy4HDgD7gdn2ti5BRLh4ZCYrCg9TdNjBkVjjUuGqf4OnAZ67HKp9PAyIUko10aae1MClxpiZxph0Y0yGMeZiY8yuTogvYDS2YPL50BvtlTYArpgPh3bCi98Fd72z8SilQlZbe1LP6oRYAlp2SiwFfZJ5YcVu6hp8PPRGe+WeDTMfgK8/gTd+rh3slFJ+0dYips9E5EEROVtERjUufo0sAN00tT97DlXzr2UB8PA04ko4+xfw1bPw2d+cjkYpFYLa2pN6gv16d5NthpN7Voe8KXnpnNk3lQc+3M6lo7NIiI5wNqBzfmUVNb1/FyTnwpCLnY1HKRVS2lIHEQY8bIw5p9nSanIQkRkiskVEtovIHV4+jxKRBfbny0Qkp8lnw0RkqYhsEJF1IhLdznvzORHhzm8N5FBVHY98stPpcCAsDC5+GLLGwqs/hCLf9yRXSnVdbamD8AA3t/fEduX2Q8AFwGBgrogMbrbbdcBhY0x/4K/AH+1jw4H5wI3GmCHAFCAgamOHZSUxc3gvHl+ykwPlATAKa0Q0zH0eEnrYzV8DoPhLKRUS2loH8Z6I/EJEskUkpXFp5ZixwHZjzE5jTB3wAt+s7J4FPG2vvwScaw8rPh1Ya4xZA2CMKbMrywPCbeefgdtj+Ot7W50OxRKXBlf+G9x18K/Loabc6YiUUiGgPf0gbgIWA6vspbXyjExgT5P3RfY2r/sYYxqAciAVyAOMiLwrIl+KyC+9XUBEbhCRlSKysqSkpI230nHZKbFcMz6Hf6/aw9aDRzvtuqeUngeXP2vNVPfiPG3+qpTqsDYlCGNMrpelbyuHeZtgqHl7zJb2CQcmAlfZr5eIyLle4nrUGFNgjClIT+/ckT9+MrU/cVHh/PHtzZ163VPqOxkuuh92fgRv3qrNX5VSHXLKBNH0L3cRuazZZ/+3lXMXAdlN3mcBzXuZHd/HrndIBA7Z2z8xxpQaY44BbwEB1aw2OS6SH03pxwebi/liZwCNOjLyaqv565fPwEetfUVKKdWy1p4g5jRZv7PZZzNaOXYFMEBEckUk0j7Xomb7LALm2euzgQ+NNUHFu8AwEYm1E8dkYGMr1+t03z8rl56J0dzz1iYCal6Nqf8HRl4Di/8Eyx51OhqlVJBqLUFIC+ve3p/ErlO4GevHfhPwojFmg4jcLSIz7d2eAFJFZDtwK3CHfexh4C9YSWY18KUx5s023E+nio5wcet5eawpKufNzh7p9VRE4MK/wRnfhrd/aU1EpJRS7XTKGeVE5EtjzKjm697eO81fM8q1xu0xfPuBTzlW5+b9WycTGd7Wev9OUF8Nz34HilZYg/z1O8fpiJRSAaYjM8oNF5EKETmKVeRT0eR9vs8jDUKuMOH2Cway+9CxwBiCo6mIGKuPRFoeLLga9n3ldERKqSByygRhjHEZY7oZYxKMMeH2euN7h8eZCBxT8tKZ0M8aguNoTYA1L41JgqtfhpgUmD8bynY4HZFSKkgEUHlI8BIR7rxgUOAMwdFct55wzauAgWcvhqMHnI5IKRUENEH4SH5WYmANwdFcWn+rHqKqDOZfqr2tlVKt0gThQwE3BEdzmaNhznwo2QLPXwn1AZjIlFIBQxOEDwXkEBzN9ZsKl/wDdi2Bl68DT8AMcaWUCjCaIHwsIIfgaC5/Nsz4I2x+Q2ekU0q1SBOEjyXHRfLjKf35YHMxL67Y0/oBThl/I5z9X/Dl0/Dva6E2QJ94lFKO0QThB987K4ezB6Txy5fX8sgnAdysdOqvYdrvYNMieOxcq25CKaVsmiD8IDrCxRPzxnDhsJ7c8/bmwBurqZEITLwFrlkIx8rgsamwYaHTUSmlAoQmCD+JDA/j/jkjuWZ8Hx5ZvJPbXlpLg9vjdFje9Z0MP1wMGYPg3/Pg3V+Bu8HpqJRSDtME4UeuMOHuWUO4ZdoAXlpVxI3zv6SmPkBbDSVmwrVvwZgfwNIH4ZmZcPSg01EppRykCcLPRIRbpuXx+1lD+GDzQb77xHLKqwNsOI5G4ZHw7fvgO4/B3i/hkUmwa6nTUSmlHKIJopNcc2YOD8wZyVd7DjPn0S8oPhrAndSGXQ7Xvw+RsfD0hfDFw9oUVqku6JTDfQcTp4b7bq/FW0u4cf4q0uKjePa6sfRJjXM6pJZVH4GFP4Ytb8LQS+H8e6yKbXedvTQ0Wa8/sR4eBb0nQJj+/aFUoDvVcN+aIBywes8RvvfP5bjCwnj6+2MY0ivR6ZBa5vHAZ3+FD/8HTDsq2QfPgksesYYcV0oFLE0QAWh78VGueWI5lTUNPD6vgHF9U50O6dT2rLAmHgqPBFfjEnFiPSz8xPquJfDB3ZA9DuY8D3EBfm9KdWGaIALUviPVXPPEMvaX1/DsdeMY3SfZ6ZB8Z8NCeOUGq3XUVS9Baj+nI1JKedGRGeWUH/VKiuH5G8bTvVs01/5zOev3htAQ3EMuhnmvW/UYj0+D3cucjkgp1U6aIByWkRDN/OvH0S06gu8+uZztxSE0JlLvcVZrqJgkePoi7aWtVJDRBBEAMpNimH/9OMJEuOrxZewuO+Z0SL6T2g+uex96DrcGBfz879pkVqkgoQkiQOSmxfHc9eOobfBw1RNfBOasdKcrLhXmLYLBM+E//wfeuk3noVAqCGiCCCBn9Ejgme+P5XBVPVc9/gWllbVOh+Q7ETEw+ymY8BNY8Ri8cBXUVbXt2PpqOHbIr+Eppb5JWzEFoOVfH+K7Ty6jb1o8z98wnsSYCKdD8q3lj8Hbv7SKnc6/x5ofu/IgVBVDZZOl8X1thXXcGd+yhijvPtjZ+JUKIdrMNQh9srWE659ewdDMROZfN464qHCnQ/KtLW/DS9+H+mb1LdGJEN8d4jIg3l7i0q39lj9mTWw07HKYciek5DoTu1IhRBNEkHpn/X5u+tdXjMtN4clrxxAd4XI6JN869DWUbLaTgJ0MwqNa3v/YIfjsb7DsEasOY/Q8mHQbJPRo+zWNgdJtsPUd2LsKug+FnImQOdrqBKhUF6MJIoi98mURt764hnMHZvCPa0YT4dJqIyr2w+I/wZfPQFiENX3qWT+DmBY6GjbUwe7PYeu7VmI4tNPa3i0TKvZa6xGxVs/vnImQOwl6jbR6iisV4jRBBLlnv9jFrxeu59vDenL/FSMI1yRhKdsBH98D616C6G5Wkhh3I0TGQVUpbHvPSgg7PrTqMVxR1uRIeefDgPMhKdt6KilcYi+fQvFG69wRcdB7POSeDTlnQ88R4AqxYj6l0AQREh75ZAf3vL2Z/MxE/nzZMAb26OZ0SIHjwHr48PdWMojvDkl9rHGjMBDfw0oIeTOs5BDZyui5VaUnJ4ySzdb2HsPgygXQrZffb0epzqQJIkS8tW4/v164noqaen46dQA3TumnRU5N7f4CPr7XahWVd7619BjesWHHK4th23/g7dutCvSr/g3dh/guZqUcpgkihJRV1vLbRRt4Y+1+hvTqxn2XDWdQT32a8LsD6+C5y61WVFc8A/2mOh2RUj6hg/WFkNT4KB68chQPXzWKgxU1zHxwCfe/v416dzvmalDt1yPfGlcquQ88dxl8+azTESnld5oggtQF+T35z88nc8HQnvz1/a3MevAzNu6rcDqs0JaYCd9726q0XnSzPYlSaDyBK+WNJogglhIXyQNzR/KPq0dTfLSWmQ8u4W/vb6WuQZ8m/Ca6m1UPMfJqWPxnePWHVjNaf3I3+Pf8SrVA2+2FgBlDezAuN4W7Xt/A397fxrsbDvLn2cMYmhnAU5kGM1cEzHwQknLgo/+Bin1wxbMt98Nor/pq2L3Uap67/UMo2QS9RlktsfLOt4q7RHxzLaVOQSupQ8y7Gw7wq1fXc/hYHd+bkMPPz8sLvWE6AsmaBfDaTZDS13qySO7T/nMYA8WbrISw40PY9Rk01FjTt/YebzWx3b3U6vkNVge/xqa7uZN03m/VIdqKqYspP1bPH9/dzL+W7aZnYjR3zRzC9MHdEf2r0z++/hQWXGV1xLtyAWSOsrYbA+56cNfZS7P1g+tPJIWj+61j0vKg37lWK6mcs07ut3H0IGxv7Pz3EdRVQniM3fnPfrrQfhqqnTRBdFGrdh3mV6+uY/OBo0wblMFdM4eQlRzrdFihqXiz1bqpYq81npS7DjxtqDuIToK+U6D/udD3HKt3d1s01Fqd+ba+C1vfhiO7re3Z42DM9TB41qnHtVLKpgmiC6t3e3jqs0L+8t5WAG6ZNoDvT8zVDnb+cPQgLPuHlRxckfYSYS+RTV4jISwcknOsMZ/COjgIozFQsgW2vAVfzYdDOyA2FUZeAwXfs66jVAscSxAiMgO4H3ABjxtj7m32eRTwDDAaKAOuMMYUNvm8N7ARuMsYc9+prqUJ4tT2Hqnmt69t4P1NBxnYI4E/XDKU0X1SnA5L+ZrHA19/AiufgM1vgfHAgPOg4Drrtb3JyOMGCdNK8RDmSIIQERewFTgPKAJWAHONMRub7PNjYJgx5kYRmQNcYoy5osnnLwMeYJkmCN/4z4YD3LVoA/vKa5g7NpvbZwwkKVaHuQ5J5Xvhy6dh1dNQeQASe0PBtTDyuxCffmK/+mo4XGgNv37465Nfj+yCxGw48yYYcRVEahFlqHEqQZyJ9Zf/+fb7OwGMMfc02edde5+lIhIOHADSjTFGRC4GzgKqgEpNEL5TVdvA/R9s44klX9MtOpyz+qcxPCuJYVmJDM1M1FZPocZdD5vfhBWPWwMQhkVYleB1lVYSOLrv5P2jEiElB5JzrVZZhUusFlSxqTD2BhjzA2uecRUSnEoQs4EZxpjr7ffXAOOMMTc32We9vU+R/X4HMA6oBt7Hevr4BS0kCBG5AbgBoHfv3qN37drll3sJVZv2V/DgR9tZvfsIe49UA1ZJQv/0eIZlJTE8O5FhWUkM7JEQepMVdVUlW2Dlk9YAhPHdrSSQkmu/9rXWY5JPLlIyBnZ9Dp8/YLWgCo+xOgqeeZPO6hcCnEoQlwHnN0sQY40xP2myzwZ7n6YJYixwJ7DcGPOiiNyFPkH4XWllLeuKyllTdIS1ReWsLTpCaaXVQzjCJZzRI4GpA7tz3cTc0JsjW7Vd8Wb4/O+wdgEYNwyaCWf91JqRTwWloCtiAhYDje39krDqIX5jjHmwpetpgvAtYwz7y2tYW3SENUXlrN59hKU7y0iMieDGyf24dkIOMZH6VNFlVey3WmytfNKajCnnbGuypoxB1tSxkfHtq9huqIWy7dYTTunWE6+VxdakTYMugv7nQVS8/+6pi3IqQYRjVVKfC+zFqqS+0hizock+NwH5TSqpv2OMubzZee5CnyACwoZ95dz37hY+2lJCRkIUPzl3AFcUZBMZrk1mu6yaCqsi/IuHT0zfClYxVLw9z/oEg6AAABIHSURBVHh8d4hLt17j0635x2uPQukWKNlqvR4utFpcASBW3UfaGRCTZHUkrCqxOiL2P9dKFnkzIFZb4fmCk81cvwX8DauZ65PGmD+IyN3ASmPMIhGJBp4FRgKHgDnGmJ3NznEXmiACyorCQ/zpnc2sKDxM75RYbj0vj5nDexEWpk0hu6zGeb+PHoDKg9Zf/pXFUFV8Yv1YGdDk9yYsAlL7Q3qelQzS7SW1/8nDh3jc1mRQm163looiENeJJ4uBF0JCD9/ch7vBirPyoBV7XDp0H9rxvioBTDvKKZ8zxvDxlhL+9O4WNu2vYGCPBH4x/QzOHZShQ3oo79wNcKzU+vGNiLM68LV3nm9jYN9XdrJYZBVLIZA91hqzqnmHRG8dFZETcVQWn5zMqko5KYmBNZNgn4mQM9FKShlD2j9LocdtNRku3W49+fiig6SPaIJQfuPxGN5ct5+/vLeVr0urGNU7idvOH8iZ/bQZpPKzxh7km16Hza/DkT0nj3fV/Ie+ufBoqwgsLuNEcdhJ6+nWOQs/tZbDhdZxMcnQ5yxroMSciZA+6ETCaKiFsh0nis9KNlt1KWXbrQEYG0UnWWNoNY671dYhVvxAE4Tyu3q3h5dWFXH/+9s4UFHDD87O5Y4LBuHSYiflBGOsv9rddeCpPzlxGA/EpkFUQvsq0o/ssfqEFC6BwsUnxr+KTbWKocqL7LoUt32AQFJvq9gsLe/Ea8XeE0O5N/ZBSR1g1a/0m2oln06sjNcEoTpNTb2be97axNNLdzE5L52/XzmSbtHaLFaFoMO7TiSM4o12Mhh4IhGk9j91z/PGJ6AdH1gJo/AzaKi26mZ6j7dahiV0t55YYpKtp47G9cg4nw1/oglCdbp/LdvNb15bT5/UWB6fN4bctLjWD1KqK6uvOTFR1I4PreHgWxIWfnLSGHAeTP7laV1WE4RyxBc7y/jR/FV4DDx05SgmDkhzOiSlgkddFVQfhuoj9uthqDnifVv2eDjnztO6jCYI5Zg9h45x/dMr2V5Sya+/PYh5E3K0lZNSAeRUCUJ7OCm/yk6J5eUfT+CcMzK46/WN/Per66hr8LR+oFLKcZoglN/FR4Xz6DWj+fGUfjy/fA9XP7GMsspap8NSSrVCE4TqFGFhwi9nDOT+OSNYvecIsx76jM0HKpwOSyl1Cjrwv+pUs0ZkkpMaxw+eWcl3/t/n/ODsvqTERRIT6SIuMpzYKBexES7iosKJjbReGz/TPhVKdS5NEKrTDc9O4vWfTORH81dx/wfb2nRMeJgwsncSZw9IZ1JeOvmZiZowlPIzbcWkHGOMobbBQ1VtA8fq3Byrc1NV10B1nfv4tsb3JUdr+XxHGev3lWMMJMZEMLF/GpPy0jh7QDq9kmJav6BS6htO1YpJnyCUY0SE6AgX0REu2jpyU1llLUu2l/LptlI+3VbCm+v2A9AvPY5JeelMGpDOqD7JOqmRUj6gTxAqaBlj2Hqwkk+3lbB4WynLdpZRazehTYqNoE9KLL1T48hJjaV3Six9UuPokxpLRkKU9sVQyqYd5VSXUFPvZkXhITbtr6Cw7Bi7y46x61AVew9X42nyn3lMhIveKbHkpsVxVv9Upg7qTqYWUakuShOE6tLq3R72Hq5m16Fj7C6rorDsGLvKjrHlYAV7DlUDMLBHAucOyuDcQd0ZnpWkFeCqy9AEoZQXxhh2llbx4aZi3t90kJW7DuP2GFLjIplyRgbTBmUwcUAaCToarQphmiCUaoPyY/V8sq2EDzcd5KMtJZRX1xPhEsblpjLljHQm56XTPyNe6y9USNEEoVQ7Nbg9fLn7CB9sPsgHm4rZXlwJQM/EaM4eYDWtndg/jeS4SIcjVapjNEEo1UFFh4+xZJvVvHbJ9lLKq+sRgWGZicc7743snUSES0evUcFFE4RSPuT2GNYWHWHxVqsvxld7juD2GOKjwhnfN4UxOSkU5CQzNDORqPDAmJheqZZoglDKjypq6vl8exmfbivhs+2lFJYdAyDSFUZ+ViIFfZIZbS+p8VEOR6vUyTRBKNWJSo7WsmrXYb7cfZiVhYdYv7eCOrfVgS83LY5RvZMpyEmmT2oska4wIo4vYr2GW+uNn0WGh2nRlfIbTRBKOaim3s36veWs3HWYlYVW4jhUVdfm48ME+mfEk5+ZxPDsRIZlJTGoZ4IWXymf0AShVAAxxvB1aRUHK2qpd3to8HioazDUuz3Hlzq3ob7B+qyiuoEN+8pZW1ROmZ1YIlzCwB7dyM9KZHiWlTQGZMQTrk8aqp10sD6lAoiI0Dc9nr7p8e06zhjD3iPVrCsqZ01ROWuLjvD66n38a9luAKIjwhibm8pFw3oyfUgPHbBQdZg+QSgVxDweQ2FZFWuLylm9x+q3sedQNZGuMCafkc5Fw3sxbVAGsZH6t6DyTouYlOoijDGsKSrn9TX7eGPtPg5W1BIT4eLcQRlcNLwXk/PSiY5ove7C7TEcPlZHydFaYiNdZCXH6vhUIUoThFJdkMdjWFF4iNfX7uOtdQc4VFVHQlQ404f0YNqgDOrcHkqO1lJSWUvJ0VpKK+vs11rKKmtPGgE3MjyMvmlx9M+IP2nJTYvTyvIgpwlCqS6uwe3h8x1lvL5mH+9sOMDRmobjn0W6wkiLjyQ9IYq0+KhvvFbVNrC9pJJtB4+yvaSSosPVNP5shAn0Tomlf0Y8OalxxEa6mjTVbdJ013Vy09346PDj5+8WHa7jWzlIE4RS6rjaBjcb91WQEB1BekL7f6Br6t3sKKlke3ElO4or2W6v7yo7dnzCpvaIDA8jPT6KtIQo0pslqoyEaIb06kZWckyHkkhxRQ1Ld5bxxc4yvi6t4vwhPfjOqCytyEcThFKqkxhjaPAYGtyGuibNdusbrPdWk16r6W7p8aKt2m8UdZVV1dL0pyklLpL8zBNNeodlJ5KREN1iHCVHa/nCTghLd5axs6QKgISocHokRrOtuJKYCBcXj+zFVeP6MDQz0d//NAFLm7kqpTqFiNjFShDD6ddNuD2GQ1V1VrPeveWs3XOEtUXlfLqt5HjdSI9u0QzLSmR4dhLDshI5WtPA0h1WQmgcfTc+KpwxOcnMGZPN+L6pDOmViCtMWL+3nPlf7OLVr/by/PI9jOydxNXj+vDtYT3bVInfVegThFIqaByra2DDvgrW2Alj3d5yvi6tOv55bKSLMTkpnNkvlfF9Uxnaq9spOw+WV9fz8qoi5i/bxc6SKpJjI7isIJurxvWmT2pcZ9yS47SISSkVssqP1bN+XzkxkS7yMxNPa9wqYwxLd5Qxf9ku3t1wELfHMCkvnRFZiVTVuTlW5+ZYXQNVtfZrnZvqJu8NUNAnmUl56Zw9IJ2c1NhOq3g3xlBd7z7tvi6aIJRSqo0OVtTwwvI9PL98NweP1hAXGU5MpIu4SBexkeHERbmIiQw/6X1dg9VKbPchayTf7JQYa56QAWlM6J9GNx9NW2uM4UBFDWv2lLNur/UUtbaonPMGd+e+y4af1jk1QSilVDs1/ja250lgV1kVi7eWsHhbKUt3lFFZ24ArTBiRncSkAemcnZdGv/R4u7mv4AqTU57/UFUda4qOsNZOCGuKyik5WguAK0w4o3sCw7MTmZyXzoyhPU/rPjVBKKVUJ6t3e/hq9xE+3VbC4q0lrN1bTvOfWxGICLP7i9h9RxqTR12Dh33lNcf365sWx/Asq0J+WHYSg3t280mFuiYIpZRy2OGqOj7bUXp8FN/6hiYj9x5vEnxiXYBBPa0Re/MzE0nwUTFVc441cxWRGcD9gAt43Bhzb7PPo4BngNFAGXCFMaZQRM4D7gUigTrgNmPMh/6MVSml/Ck5LpILh/VyOox28dvg8SLiAh4CLgAGA3NFZHCz3a4DDhtj+gN/Bf5oby8FLjLG5APzgGf9FadSSinv/Dm7yFhguzFmpzGmDngBmNVsn1nA0/b6S8C5IiLGmK+MMfvs7RuAaPtpQymlVCfxZ4LIBPY0eV9kb/O6jzGmASgHUpvtcynwlTGmtvkFROQGEVkpIitLSkp8FrhSSin/Jghvbbea14ifch8RGYJV7PRDbxcwxjxqjCkwxhSkp6efdqBKKaW+yZ8JogjIbvI+C9jX0j4iEg4kAofs91nAq8B3jTE7/BinUkopL/yZIFYAA0QkV0QigTnAomb7LMKqhAaYDXxojDEikgS8CdxpjPnMjzEqpZRqgd8ShF2ncDPwLrAJeNEYs0FE7haRmfZuTwCpIrIduBW4w95+M9Af+LWIrLaXDH/FqpRS6pu0o5xSSnVhXaIntYiUALuabU7D6lMRSkLtnvR+Al+o3VOo3Q907J76GGO8tvIJmQThjYisbCkzBqtQuye9n8AXavcUavcD/rsnf1ZSK6WUCmKaIJRSSnkV6gniUacD8INQuye9n8AXavcUavcDfrqnkK6DUEopdfpC/QlCKaXUadIEoZRSyquQTRAiMkNEtojIdhG5o/UjApuIFIrIOrtXeVD2CBSRJ0WkWETWN9mWIiLvicg2+zXZyRjbo4X7uUtE9jYZAeBbTsbYHiKSLSIficgmEdkgIj+ztwfzd9TSPQXl9yQi0SKyXETW2PfzO3t7rogss7+jBfbwRh2/XijWQdiTFW0FzsMaEHAFMNcYs9HRwDpARAqBAmNM0HbwEZFJQCXwjDFmqL3tT8AhY8y9diJPNsbc7mScbdXC/dwFVBpj7nMyttMhIj2BnsaYL0UkAVgFXAxcS/B+Ry3d0+UE4fckIgLEGWMqRSQCWAL8DGuooleMMS+IyD+ANcaYhzt6vVB9gmjLZEWqkxljFmOP1ttE00mjnsb6nzcotHA/QcsYs98Y86W9fhRrDLVMgvs7aumegpKxVNpvI+zFAFOxJl0DH35HoZog2jJZUbAxwH9EZJWI3OB0MD7U3RizH6z/mYFQGJTxZhFZaxdBBU1xTFMikgOMBJYRIt9Rs3uCIP2eRMQlIquBYuA9YAdwxB4gFXz4exeqCaItkxUFm7OMMaOw5vi+yS7eUIHnYaAfMALYD/yvs+G0n4jEAy8DtxhjKpyOxxe83FPQfk/GGLcxZgTWHDtjgUHedvPFtUI1QbRlsqKg0jhHtzGmGGsipbHORuQzB+1y4sby4mKH4+kQY8xB+39gD/AYQfY92eXaLwPPGWNesTcH9Xfk7Z6C/XsCMMYcAT4GxgNJ9qRr4MPfu1BNEG2ZrChoiEicXcGGiMQB04H1pz4qaDSdNGoe8JqDsXRY4w+p7RKC6HuyK0CfADYZY/7S5KOg/Y5auqdg/Z5EJN2eUA0RiQGmYdWrfIQ16Rr48DsKyVZMAHaztb8BLuBJY8wfHA7ptIlIX6ynBoBw4F/BeD8i8jwwBWto4oPAb4GFwItAb2A3cJkxJigqflu4nylYxRYGKAR+2Fh+H+hEZCLwKbAO8Nib/xurzD5Yv6OW7mkuQfg9icgwrEpoF9Yf+C8aY+62fyNeAFKAr4CrjTG1Hb5eqCYIpZRSHROqRUxKKaU6SBOEUkoprzRBKKWU8koThFJKKa80QSillPJKE4RSrRARd5NRP1f7cnRgEclpOhqsUoEkvPVdlOryqu2hDZTqUvQJQqnTZM/R8Ud7fP7lItLf3t5HRD6wB4L7QER629u7i8ir9lj+a0Rkgn0ql4g8Zo/v/x+7hywi8lMR2Wif5wWHblN1YZoglGpdTLMipiuafFZhjBkLPIjVcx97/RljzDDgOeABe/sDwCfGmOHAKGCDvX0A8JAxZghwBLjU3n4HMNI+z43+ujmlWqI9qZVqhYhUGmPivWwvBKYaY3baA8IdMMakikgp1iQ19fb2/caYNBEpAbKaDoFgD0H9njFmgP3+diDCGPM/IvIO1oREC4GFTeYBUKpT6BOEUh1jWlhvaR9vmo6Z4+ZE3eC3gYeA0cCqJqN1KtUpNEEo1TFXNHldaq9/jjWCMMBVWNNCAnwA/AiOT/rSraWTikgYkG2M+Qj4JZAEfOMpRil/0r9IlGpdjD2DV6N3jDGNTV2jRGQZ1h9bc+1tPwWeFJHbgBLge/b2nwGPish1WE8KP8KarMYbFzBfRBKxJsD6qz3+v1KdRusglDpNdh1EgTGm1OlYlPIHLWJSSinllT5BKKWU8kqfIJRSSnmlCUIppZRXmiCUUkp5pQlCKaWUV5oglFJKefX/Ab63CPxBGgGgAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# => Plot \n",
    "plt.figure()\n",
    "plt.plot(np.arange(1,epochs + 1,1),population_curve_N_2,label=\"Full\")\n",
    "plt.plot(np.arange(1,epochs + 1,1),population_curve_dropout_N_2,label=\"Dropout\")\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Risk')\n",
    "plt.legend(loc='best')\n",
    "plt.savefig(\"N2000_risk.jpg\")\n",
    "plt.figure()\n",
    "plt.plot(np.arange(1,epochs + 1,1),error_curve_N_2,label=\"Full\")\n",
    "plt.plot(np.arange(1,epochs + 1,1),error_curve_dropout_N_2,label=\"Dropout\")\n",
    "plt.legend(loc='best')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Error')\n",
    "plt.savefig(\"N2000_error.jpg\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Constructing a path\n",
    "\n",
    "As it was mentioned previously in introduction dropout stability of solution allows to obtain a closed form of piece-wise linear path which connects two solutions. Before going straight to the point we need to train another network with $N=N_2$ neurons (for second solution)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 458,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch [1/30], Step [0/600], Loss: 2.5285\n",
      "Epoch [1/30], Step [60/600], Loss: 0.7036\n",
      "Epoch [1/30], Step [120/600], Loss: 0.3526\n",
      "Epoch [1/30], Step [180/600], Loss: 0.3785\n",
      "Epoch [1/30], Step [240/600], Loss: 0.4282\n",
      "Epoch [1/30], Step [300/600], Loss: 0.2914\n",
      "Epoch [1/30], Step [360/600], Loss: 0.3841\n",
      "Epoch [1/30], Step [420/600], Loss: 0.2747\n",
      "Epoch [1/30], Step [480/600], Loss: 0.2641\n",
      "Epoch [1/30], Step [540/600], Loss: 0.3198\n",
      "92.13\n",
      "88.95\n",
      "Epoch [2/30], Step [0/600], Loss: 0.1665\n",
      "Epoch [2/30], Step [60/600], Loss: 0.3452\n",
      "Epoch [2/30], Step [120/600], Loss: 0.2262\n",
      "Epoch [2/30], Step [180/600], Loss: 0.3186\n",
      "Epoch [2/30], Step [240/600], Loss: 0.1560\n",
      "Epoch [2/30], Step [300/600], Loss: 0.2354\n",
      "Epoch [2/30], Step [360/600], Loss: 0.2255\n",
      "Epoch [2/30], Step [420/600], Loss: 0.2714\n",
      "Epoch [2/30], Step [480/600], Loss: 0.1822\n",
      "Epoch [2/30], Step [540/600], Loss: 0.1550\n",
      "93.65\n",
      "91.61\n",
      "Epoch [3/30], Step [0/600], Loss: 0.2414\n",
      "Epoch [3/30], Step [60/600], Loss: 0.2158\n",
      "Epoch [3/30], Step [120/600], Loss: 0.2326\n",
      "Epoch [3/30], Step [180/600], Loss: 0.2207\n",
      "Epoch [3/30], Step [240/600], Loss: 0.1625\n",
      "Epoch [3/30], Step [300/600], Loss: 0.3090\n",
      "Epoch [3/30], Step [360/600], Loss: 0.1420\n",
      "Epoch [3/30], Step [420/600], Loss: 0.3084\n",
      "Epoch [3/30], Step [480/600], Loss: 0.1522\n",
      "Epoch [3/30], Step [540/600], Loss: 0.2435\n",
      "94.51\n",
      "92.09\n",
      "Epoch [4/30], Step [0/600], Loss: 0.1627\n",
      "Epoch [4/30], Step [60/600], Loss: 0.2794\n",
      "Epoch [4/30], Step [120/600], Loss: 0.1656\n",
      "Epoch [4/30], Step [180/600], Loss: 0.1718\n",
      "Epoch [4/30], Step [240/600], Loss: 0.2810\n",
      "Epoch [4/30], Step [300/600], Loss: 0.1106\n",
      "Epoch [4/30], Step [360/600], Loss: 0.1684\n",
      "Epoch [4/30], Step [420/600], Loss: 0.1187\n",
      "Epoch [4/30], Step [480/600], Loss: 0.1770\n",
      "Epoch [4/30], Step [540/600], Loss: 0.2100\n",
      "94.92\n",
      "92.77\n",
      "Epoch [5/30], Step [0/600], Loss: 0.1539\n",
      "Epoch [5/30], Step [60/600], Loss: 0.2143\n",
      "Epoch [5/30], Step [120/600], Loss: 0.1493\n",
      "Epoch [5/30], Step [180/600], Loss: 0.1367\n",
      "Epoch [5/30], Step [240/600], Loss: 0.0667\n",
      "Epoch [5/30], Step [300/600], Loss: 0.2768\n",
      "Epoch [5/30], Step [360/600], Loss: 0.2035\n",
      "Epoch [5/30], Step [420/600], Loss: 0.1836\n",
      "Epoch [5/30], Step [480/600], Loss: 0.0701\n",
      "Epoch [5/30], Step [540/600], Loss: 0.1361\n",
      "95.5\n",
      "93.34\n",
      "Epoch [6/30], Step [0/600], Loss: 0.1878\n",
      "Epoch [6/30], Step [60/600], Loss: 0.1625\n",
      "Epoch [6/30], Step [120/600], Loss: 0.1333\n",
      "Epoch [6/30], Step [180/600], Loss: 0.0752\n",
      "Epoch [6/30], Step [240/600], Loss: 0.1142\n",
      "Epoch [6/30], Step [300/600], Loss: 0.1222\n",
      "Epoch [6/30], Step [360/600], Loss: 0.0805\n",
      "Epoch [6/30], Step [420/600], Loss: 0.1448\n",
      "Epoch [6/30], Step [480/600], Loss: 0.0767\n",
      "Epoch [6/30], Step [540/600], Loss: 0.2427\n",
      "95.64\n",
      "93.72\n",
      "Epoch [7/30], Step [0/600], Loss: 0.1862\n",
      "Epoch [7/30], Step [60/600], Loss: 0.0865\n",
      "Epoch [7/30], Step [120/600], Loss: 0.1367\n",
      "Epoch [7/30], Step [180/600], Loss: 0.1511\n",
      "Epoch [7/30], Step [240/600], Loss: 0.1999\n",
      "Epoch [7/30], Step [300/600], Loss: 0.1286\n",
      "Epoch [7/30], Step [360/600], Loss: 0.1803\n",
      "Epoch [7/30], Step [420/600], Loss: 0.1399\n",
      "Epoch [7/30], Step [480/600], Loss: 0.1304\n",
      "Epoch [7/30], Step [540/600], Loss: 0.2093\n",
      "95.93\n",
      "94.27\n",
      "Epoch [8/30], Step [0/600], Loss: 0.1189\n",
      "Epoch [8/30], Step [60/600], Loss: 0.0971\n",
      "Epoch [8/30], Step [120/600], Loss: 0.1022\n",
      "Epoch [8/30], Step [180/600], Loss: 0.0889\n",
      "Epoch [8/30], Step [240/600], Loss: 0.1048\n",
      "Epoch [8/30], Step [300/600], Loss: 0.1121\n",
      "Epoch [8/30], Step [360/600], Loss: 0.1637\n",
      "Epoch [8/30], Step [420/600], Loss: 0.0936\n",
      "Epoch [8/30], Step [480/600], Loss: 0.0849\n",
      "Epoch [8/30], Step [540/600], Loss: 0.1575\n",
      "96.16\n",
      "94.26\n",
      "Epoch [9/30], Step [0/600], Loss: 0.0815\n",
      "Epoch [9/30], Step [60/600], Loss: 0.1825\n",
      "Epoch [9/30], Step [120/600], Loss: 0.0556\n",
      "Epoch [9/30], Step [180/600], Loss: 0.1048\n",
      "Epoch [9/30], Step [240/600], Loss: 0.0654\n",
      "Epoch [9/30], Step [300/600], Loss: 0.0603\n",
      "Epoch [9/30], Step [360/600], Loss: 0.1113\n",
      "Epoch [9/30], Step [420/600], Loss: 0.0929\n",
      "Epoch [9/30], Step [480/600], Loss: 0.1801\n",
      "Epoch [9/30], Step [540/600], Loss: 0.1234\n",
      "96.34\n",
      "94.47\n",
      "Epoch [10/30], Step [0/600], Loss: 0.1579\n",
      "Epoch [10/30], Step [60/600], Loss: 0.1055\n",
      "Epoch [10/30], Step [120/600], Loss: 0.0876\n",
      "Epoch [10/30], Step [180/600], Loss: 0.1295\n",
      "Epoch [10/30], Step [240/600], Loss: 0.2224\n",
      "Epoch [10/30], Step [300/600], Loss: 0.1327\n",
      "Epoch [10/30], Step [360/600], Loss: 0.1208\n",
      "Epoch [10/30], Step [420/600], Loss: 0.1049\n",
      "Epoch [10/30], Step [480/600], Loss: 0.0766\n",
      "Epoch [10/30], Step [540/600], Loss: 0.0805\n",
      "96.52\n",
      "94.94\n",
      "Epoch [11/30], Step [0/600], Loss: 0.0892\n",
      "Epoch [11/30], Step [60/600], Loss: 0.1856\n",
      "Epoch [11/30], Step [120/600], Loss: 0.1112\n",
      "Epoch [11/30], Step [180/600], Loss: 0.0824\n",
      "Epoch [11/30], Step [240/600], Loss: 0.0527\n",
      "Epoch [11/30], Step [300/600], Loss: 0.0410\n",
      "Epoch [11/30], Step [360/600], Loss: 0.0642\n",
      "Epoch [11/30], Step [420/600], Loss: 0.2445\n",
      "Epoch [11/30], Step [480/600], Loss: 0.0723\n",
      "Epoch [11/30], Step [540/600], Loss: 0.1420\n",
      "96.69\n",
      "95.06\n",
      "Epoch [12/30], Step [0/600], Loss: 0.1169\n",
      "Epoch [12/30], Step [60/600], Loss: 0.0726\n",
      "Epoch [12/30], Step [120/600], Loss: 0.1880\n",
      "Epoch [12/30], Step [180/600], Loss: 0.0988\n",
      "Epoch [12/30], Step [240/600], Loss: 0.0495\n",
      "Epoch [12/30], Step [300/600], Loss: 0.0935\n",
      "Epoch [12/30], Step [360/600], Loss: 0.1184\n",
      "Epoch [12/30], Step [420/600], Loss: 0.0504\n",
      "Epoch [12/30], Step [480/600], Loss: 0.0858\n",
      "Epoch [12/30], Step [540/600], Loss: 0.1458\n",
      "96.81\n",
      "95.1\n",
      "Epoch [13/30], Step [0/600], Loss: 0.0361\n",
      "Epoch [13/30], Step [60/600], Loss: 0.1842\n",
      "Epoch [13/30], Step [120/600], Loss: 0.0912\n",
      "Epoch [13/30], Step [180/600], Loss: 0.1204\n",
      "Epoch [13/30], Step [240/600], Loss: 0.0707\n",
      "Epoch [13/30], Step [300/600], Loss: 0.1044\n",
      "Epoch [13/30], Step [360/600], Loss: 0.1127\n",
      "Epoch [13/30], Step [420/600], Loss: 0.1727\n",
      "Epoch [13/30], Step [480/600], Loss: 0.0636\n",
      "Epoch [13/30], Step [540/600], Loss: 0.0484\n",
      "96.84\n",
      "95.09\n",
      "Epoch [14/30], Step [0/600], Loss: 0.1680\n",
      "Epoch [14/30], Step [60/600], Loss: 0.0617\n",
      "Epoch [14/30], Step [120/600], Loss: 0.0654\n",
      "Epoch [14/30], Step [180/600], Loss: 0.1071\n",
      "Epoch [14/30], Step [240/600], Loss: 0.0384\n",
      "Epoch [14/30], Step [300/600], Loss: 0.1986\n",
      "Epoch [14/30], Step [360/600], Loss: 0.0820\n",
      "Epoch [14/30], Step [420/600], Loss: 0.0997\n",
      "Epoch [14/30], Step [480/600], Loss: 0.1456\n",
      "Epoch [14/30], Step [540/600], Loss: 0.0834\n",
      "97.07\n",
      "95.11\n",
      "Epoch [15/30], Step [0/600], Loss: 0.0828\n",
      "Epoch [15/30], Step [60/600], Loss: 0.1933\n",
      "Epoch [15/30], Step [120/600], Loss: 0.0232\n",
      "Epoch [15/30], Step [180/600], Loss: 0.0832\n",
      "Epoch [15/30], Step [240/600], Loss: 0.0833\n",
      "Epoch [15/30], Step [300/600], Loss: 0.0930\n",
      "Epoch [15/30], Step [360/600], Loss: 0.0491\n",
      "Epoch [15/30], Step [420/600], Loss: 0.0408\n",
      "Epoch [15/30], Step [480/600], Loss: 0.0542\n",
      "Epoch [15/30], Step [540/600], Loss: 0.0430\n",
      "97.07\n",
      "95.23\n",
      "Epoch [16/30], Step [0/600], Loss: 0.0601\n",
      "Epoch [16/30], Step [60/600], Loss: 0.1228\n",
      "Epoch [16/30], Step [120/600], Loss: 0.1489\n",
      "Epoch [16/30], Step [180/600], Loss: 0.0895\n",
      "Epoch [16/30], Step [240/600], Loss: 0.0478\n",
      "Epoch [16/30], Step [300/600], Loss: 0.0596\n",
      "Epoch [16/30], Step [360/600], Loss: 0.0586\n",
      "Epoch [16/30], Step [420/600], Loss: 0.0894\n",
      "Epoch [16/30], Step [480/600], Loss: 0.0204\n",
      "Epoch [16/30], Step [540/600], Loss: 0.0983\n",
      "97.12\n",
      "95.32\n",
      "Epoch [17/30], Step [0/600], Loss: 0.1290\n",
      "Epoch [17/30], Step [60/600], Loss: 0.0611\n",
      "Epoch [17/30], Step [120/600], Loss: 0.0769\n",
      "Epoch [17/30], Step [180/600], Loss: 0.0555\n",
      "Epoch [17/30], Step [240/600], Loss: 0.0848\n",
      "Epoch [17/30], Step [300/600], Loss: 0.0461\n",
      "Epoch [17/30], Step [360/600], Loss: 0.0468\n",
      "Epoch [17/30], Step [420/600], Loss: 0.0334\n",
      "Epoch [17/30], Step [480/600], Loss: 0.0323\n",
      "Epoch [17/30], Step [540/600], Loss: 0.0912\n",
      "97.21\n",
      "95.46\n",
      "Epoch [18/30], Step [0/600], Loss: 0.0748\n",
      "Epoch [18/30], Step [60/600], Loss: 0.1194\n",
      "Epoch [18/30], Step [120/600], Loss: 0.0676\n",
      "Epoch [18/30], Step [180/600], Loss: 0.0632\n",
      "Epoch [18/30], Step [240/600], Loss: 0.0522\n",
      "Epoch [18/30], Step [300/600], Loss: 0.0515\n",
      "Epoch [18/30], Step [360/600], Loss: 0.0776\n",
      "Epoch [18/30], Step [420/600], Loss: 0.0751\n",
      "Epoch [18/30], Step [480/600], Loss: 0.0466\n",
      "Epoch [18/30], Step [540/600], Loss: 0.0754\n",
      "97.31\n",
      "95.39\n",
      "Epoch [19/30], Step [0/600], Loss: 0.0549\n",
      "Epoch [19/30], Step [60/600], Loss: 0.0531\n",
      "Epoch [19/30], Step [120/600], Loss: 0.0203\n",
      "Epoch [19/30], Step [180/600], Loss: 0.0755\n",
      "Epoch [19/30], Step [240/600], Loss: 0.0750\n",
      "Epoch [19/30], Step [300/600], Loss: 0.0506\n",
      "Epoch [19/30], Step [360/600], Loss: 0.0505\n",
      "Epoch [19/30], Step [420/600], Loss: 0.1569\n",
      "Epoch [19/30], Step [480/600], Loss: 0.0373\n",
      "Epoch [19/30], Step [540/600], Loss: 0.0719\n",
      "97.29\n",
      "95.43\n",
      "Epoch [20/30], Step [0/600], Loss: 0.0539\n",
      "Epoch [20/30], Step [60/600], Loss: 0.0443\n",
      "Epoch [20/30], Step [120/600], Loss: 0.0427\n",
      "Epoch [20/30], Step [180/600], Loss: 0.0293\n",
      "Epoch [20/30], Step [240/600], Loss: 0.0822\n",
      "Epoch [20/30], Step [300/600], Loss: 0.0904\n",
      "Epoch [20/30], Step [360/600], Loss: 0.0582\n",
      "Epoch [20/30], Step [420/600], Loss: 0.1193\n",
      "Epoch [20/30], Step [480/600], Loss: 0.0383\n",
      "Epoch [20/30], Step [540/600], Loss: 0.0509\n",
      "97.42\n",
      "95.56\n",
      "Epoch [21/30], Step [0/600], Loss: 0.0595\n",
      "Epoch [21/30], Step [60/600], Loss: 0.0422\n",
      "Epoch [21/30], Step [120/600], Loss: 0.0453\n",
      "Epoch [21/30], Step [180/600], Loss: 0.0441\n",
      "Epoch [21/30], Step [240/600], Loss: 0.1020\n",
      "Epoch [21/30], Step [300/600], Loss: 0.0390\n",
      "Epoch [21/30], Step [360/600], Loss: 0.1327\n",
      "Epoch [21/30], Step [420/600], Loss: 0.0384\n",
      "Epoch [21/30], Step [480/600], Loss: 0.0589\n",
      "Epoch [21/30], Step [540/600], Loss: 0.0524\n",
      "97.51\n",
      "95.45\n",
      "Epoch [22/30], Step [0/600], Loss: 0.0828\n",
      "Epoch [22/30], Step [60/600], Loss: 0.1446\n",
      "Epoch [22/30], Step [120/600], Loss: 0.0758\n",
      "Epoch [22/30], Step [180/600], Loss: 0.0984\n",
      "Epoch [22/30], Step [240/600], Loss: 0.0569\n",
      "Epoch [22/30], Step [300/600], Loss: 0.0590\n",
      "Epoch [22/30], Step [360/600], Loss: 0.0542\n",
      "Epoch [22/30], Step [420/600], Loss: 0.0473\n",
      "Epoch [22/30], Step [480/600], Loss: 0.0858\n",
      "Epoch [22/30], Step [540/600], Loss: 0.0584\n",
      "97.54\n",
      "95.65\n",
      "Epoch [23/30], Step [0/600], Loss: 0.0489\n",
      "Epoch [23/30], Step [60/600], Loss: 0.0526\n",
      "Epoch [23/30], Step [120/600], Loss: 0.0235\n",
      "Epoch [23/30], Step [180/600], Loss: 0.0743\n",
      "Epoch [23/30], Step [240/600], Loss: 0.0511\n",
      "Epoch [23/30], Step [300/600], Loss: 0.0570\n",
      "Epoch [23/30], Step [360/600], Loss: 0.0290\n",
      "Epoch [23/30], Step [420/600], Loss: 0.0422\n",
      "Epoch [23/30], Step [480/600], Loss: 0.0274\n",
      "Epoch [23/30], Step [540/600], Loss: 0.0448\n",
      "97.59\n",
      "95.65\n",
      "Epoch [24/30], Step [0/600], Loss: 0.1333\n",
      "Epoch [24/30], Step [60/600], Loss: 0.0693\n",
      "Epoch [24/30], Step [120/600], Loss: 0.0454\n",
      "Epoch [24/30], Step [180/600], Loss: 0.0507\n",
      "Epoch [24/30], Step [240/600], Loss: 0.0419\n",
      "Epoch [24/30], Step [300/600], Loss: 0.0870\n",
      "Epoch [24/30], Step [360/600], Loss: 0.0807\n",
      "Epoch [24/30], Step [420/600], Loss: 0.0482\n",
      "Epoch [24/30], Step [480/600], Loss: 0.0648\n",
      "Epoch [24/30], Step [540/600], Loss: 0.0417\n",
      "97.6\n",
      "95.79\n",
      "Epoch [25/30], Step [0/600], Loss: 0.0538\n",
      "Epoch [25/30], Step [60/600], Loss: 0.0504\n",
      "Epoch [25/30], Step [120/600], Loss: 0.0301\n",
      "Epoch [25/30], Step [180/600], Loss: 0.0776\n",
      "Epoch [25/30], Step [240/600], Loss: 0.0224\n",
      "Epoch [25/30], Step [300/600], Loss: 0.0452\n",
      "Epoch [25/30], Step [360/600], Loss: 0.0417\n",
      "Epoch [25/30], Step [420/600], Loss: 0.0416\n",
      "Epoch [25/30], Step [480/600], Loss: 0.0977\n",
      "Epoch [25/30], Step [540/600], Loss: 0.0203\n",
      "97.65\n",
      "95.51\n",
      "Epoch [26/30], Step [0/600], Loss: 0.0157\n",
      "Epoch [26/30], Step [60/600], Loss: 0.0641\n",
      "Epoch [26/30], Step [120/600], Loss: 0.0291\n",
      "Epoch [26/30], Step [180/600], Loss: 0.0538\n",
      "Epoch [26/30], Step [240/600], Loss: 0.1204\n",
      "Epoch [26/30], Step [300/600], Loss: 0.0515\n",
      "Epoch [26/30], Step [360/600], Loss: 0.0798\n",
      "Epoch [26/30], Step [420/600], Loss: 0.0465\n",
      "Epoch [26/30], Step [480/600], Loss: 0.0407\n",
      "Epoch [26/30], Step [540/600], Loss: 0.0583\n",
      "97.71\n",
      "95.74\n",
      "Epoch [27/30], Step [0/600], Loss: 0.0209\n",
      "Epoch [27/30], Step [60/600], Loss: 0.0954\n",
      "Epoch [27/30], Step [120/600], Loss: 0.0515\n",
      "Epoch [27/30], Step [180/600], Loss: 0.0567\n",
      "Epoch [27/30], Step [240/600], Loss: 0.0209\n",
      "Epoch [27/30], Step [300/600], Loss: 0.0478\n",
      "Epoch [27/30], Step [360/600], Loss: 0.0470\n",
      "Epoch [27/30], Step [420/600], Loss: 0.0417\n",
      "Epoch [27/30], Step [480/600], Loss: 0.1018\n",
      "Epoch [27/30], Step [540/600], Loss: 0.0218\n",
      "97.81\n",
      "95.9\n",
      "Epoch [28/30], Step [0/600], Loss: 0.0709\n",
      "Epoch [28/30], Step [60/600], Loss: 0.0366\n",
      "Epoch [28/30], Step [120/600], Loss: 0.0408\n",
      "Epoch [28/30], Step [180/600], Loss: 0.0325\n",
      "Epoch [28/30], Step [240/600], Loss: 0.0610\n",
      "Epoch [28/30], Step [300/600], Loss: 0.0344\n",
      "Epoch [28/30], Step [360/600], Loss: 0.0908\n",
      "Epoch [28/30], Step [420/600], Loss: 0.0444\n",
      "Epoch [28/30], Step [480/600], Loss: 0.0224\n",
      "Epoch [28/30], Step [540/600], Loss: 0.0406\n",
      "97.76\n",
      "95.82\n",
      "Epoch [29/30], Step [0/600], Loss: 0.0258\n",
      "Epoch [29/30], Step [60/600], Loss: 0.0214\n",
      "Epoch [29/30], Step [120/600], Loss: 0.0507\n",
      "Epoch [29/30], Step [180/600], Loss: 0.0264\n",
      "Epoch [29/30], Step [240/600], Loss: 0.0357\n",
      "Epoch [29/30], Step [300/600], Loss: 0.0229\n",
      "Epoch [29/30], Step [360/600], Loss: 0.0281\n",
      "Epoch [29/30], Step [420/600], Loss: 0.0292\n",
      "Epoch [29/30], Step [480/600], Loss: 0.0364\n",
      "Epoch [29/30], Step [540/600], Loss: 0.0209\n",
      "97.78\n",
      "96.06\n",
      "Epoch [30/30], Step [0/600], Loss: 0.0177\n",
      "Epoch [30/30], Step [60/600], Loss: 0.0524\n",
      "Epoch [30/30], Step [120/600], Loss: 0.0392\n",
      "Epoch [30/30], Step [180/600], Loss: 0.0376\n",
      "Epoch [30/30], Step [240/600], Loss: 0.0551\n",
      "Epoch [30/30], Step [300/600], Loss: 0.0602\n",
      "Epoch [30/30], Step [360/600], Loss: 0.0233\n",
      "Epoch [30/30], Step [420/600], Loss: 0.0341\n",
      "Epoch [30/30], Step [480/600], Loss: 0.0332\n",
      "Epoch [30/30], Step [540/600], Loss: 0.0330\n",
      "97.84\n",
      "95.98\n"
     ]
    }
   ],
   "source": [
    "population_curve_N_2_2, error_curve_N_2_2, \\\n",
    "population_curve_dropout_N_2_2, error_curve_dropout_N_2_2, model_N_2_2 = get_curves_and_model(N_2,learning_rate,epochs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 514,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_1 = deepcopy(model_N_2)\n",
    "model_2 = deepcopy(model_N_2_2)\n",
    "#list(model_1.parameters())\n",
    "\n",
    "#model_1.fc2.bias"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Piece-wise Linear Path\n",
    "\n",
    "Assume that we have two dropout stable sets of parameters $\\theta$ and $\\theta'$, i.e.,\n",
    "\n",
    "$$\n",
    "|\\mathbb{E}_{(x,y) \\sim \\mathcal{D}} L(y,y_{{\\theta,N/2}}({x})) - \\mathbb{E}_{(x,y) \\sim \\mathcal{D}} L(y,y_{{\\theta}}({x}))| \\leq \\varepsilon, \\quad |\\mathbb{E}_{(x,y) \\sim \\mathcal{D}} L(y,y_{{\\theta',N/2}}({x})) - \\mathbb{E}_{(x,y) \\sim \\mathcal{D}} L(y,y_{{\\theta'}}({x}))| \\leq \\varepsilon\n",
    "$$\n",
    "\n",
    "For some loss $L$ which is convex in parameters of the last layer, i.e., $a$'s.\n",
    "\n",
    "Now consider the case where there is no bias in the first layer, or simply add $1$ to the feature vector $x$ and increase dimension of $w$ by 1. **Provide a proof for lemma below**.\n",
    "\n",
    "**Lemma.** Consider a two-layer mean-field network with one output and some activation $\\sigma$. Let $\\theta=((a_1, w_1), (a_2, w_2), \\ldots, (a_N, w_N))$ and $\\theta'=((a_1', w_1'), (a_2', w_2'), \\ldots, (a_N', w_N'))$. Assume that $N$ is even. Consider the piecewise linear path in parameter space that connects $\\theta$ to $\\theta'$ via the following intermediate points:\n",
    "\n",
    "\\begin{equation}\n",
    "\\begin{split}\n",
    "\\theta_1&=((2a_1, w_1), (2a_2, w_2), \\ldots, (2a_{N/2}, w_{N/2}), (0, w_{N/2+1}), (0, w_{N/2+2}), \\ldots, (0, w_N)),\\\\\n",
    "\\theta_2&=((2a_1, w_1), (2a_2, w_2), \\ldots, (2a_{N/2}, w_{N/2}), (0, w_{1}'), (0, w_{2}'), \\ldots, (0, w_{N/2}')),\\\\\n",
    "\\theta_3&=((0, w_1), (0, w_2), \\ldots, (0, w_{N/2}), (2a_1', w_1'), (2a_2', w_2'), \\ldots, (2a_{N/2}', w_{N/2}')),\\\\\n",
    "\\theta_4&=((0, w_1'), (0,w_2'), \\ldots, (0,w_{N/2}'), (2a_1',w_1'), (2a_2', w_2'), \\ldots, (2a_{N/2}', w_{N/2}')),\\\\\n",
    "\\theta_5&=((2a_1', w_1'), (2a_2', w_2'), \\ldots, (2a_{N/2}', w_{N/2}'), (0, w_1'), (0, w_2'), \\ldots, (0, w_{N/2}')),\\\\\n",
    "\\theta_6&=((2a_1', w_1'), (2a_2', w_2'), \\ldots, (2a_{N/2}', w_{N/2}'), (0, w_{N/2+1}'), (0, w_{N/2+2}'), \\ldots, (0, w_N')).\n",
    "\\end{split}\n",
    "\\end{equation}\n",
    "\n",
    "The loss along this path is upper bounded by $\\max(\\mathbb{E}_{(x,y) \\sim \\mathcal{D}} L(y,y_{{\\theta}}({x})), \\mathbb{E}_{(x,y) \\sim \\mathcal{D}} L(y,y_{{\\theta'}}({x})))+\\varepsilon$. \n",
    "\n",
    "**Remark:** The path consists of linear segments, i.e., \n",
    "\n",
    "$$\n",
    "\\theta(t) = (1-t)\\theta_{k-1} + t\\theta_{k}, \\quad t \\in [0,1].\n",
    "$$\n",
    "\n",
    "**Proof of Lemma:** *your_proof_goes_here :)*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Implementation\n",
    "\n",
    "In this task you would need to implement a path defined above and evaluate the test loss for 10 points in each segment and then plot the resulting curve of loss vs position on path.\n",
    "\n",
    "**Note:** The path for multiple outputs looks essentially the same. The only difference is that you need to do the aforementioned for each class $m$ weight $a_i^m$.\n",
    "\n",
    "Below we provide a couple of helper functions.\n",
    "\n",
    "**get_partition** Partitions each layer parameters on two portions - top half and bottom half (which is exactly needed for the path above). In particular, last layer parameters, i.e., $a$'s, are ensambled into $A_1$ which is top half and $B_1$ which bottom, $A_2, B_2$ are corresponding parts of first layer weights and $a_2,b_2$ is first layer bias partition.\n",
    "\n",
    "\n",
    "**insert_params** Inserts provided set of parameters into a dummy network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 662,
   "metadata": {},
   "outputs": [],
   "source": [
    "h = int(N_2 / 2)\n",
    "\n",
    "def get_partition(model):\n",
    "    W, bias, a_ = deepcopy(list(model.parameters())[0:3])\n",
    "    #W, bias, a_ = deepcopy(list(model.parameters())[0:1]+list(model.parameters())[3:4]+list(model.parameters())[2:3])\n",
    "    #W, bias, a_ = deepcopy(list(model.parameters())[0].append(list(model.parameters())[2]).append(list(model.parameters())[3]))\n",
    "    #W, bias, a_ = deepcopy(torch.Tensor(np.array(list(model.named_parameters()))[[0,2,3]]))\n",
    "    #W, bias, a_ = deepcopy(list(model.parameters()))\n",
    "    W, bias, a_ = W.data, bias.data, a_.data\n",
    "    A_1, B_1, A_2, B_2, a_2, b_2 = a_[:,:h], a_[:,h:], W[:h,:], W[h:,:], bias[:h], bias[h:]\n",
    "    return A_1, B_1, A_2, B_2, a_2, b_2\n",
    "\n",
    "def insert_params(model, A_1, B_1, A_2, B_2, a_2, b_2):\n",
    "    for i, param in enumerate(model.parameters()):\n",
    "        if i == 0:\n",
    "            param.data[:h,:] = A_2\n",
    "            param.data[h:,:] = B_2\n",
    "        if i == 1:\n",
    "            param.data[:h] = a_2\n",
    "            param.data[h:] = b_2\n",
    "        if i == 2:\n",
    "            param.data[:,:h] = A_1\n",
    "            param.data[:,h:] = B_1\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 661,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "W, bias, a_ = deepcopy(list(model.parameters())[0:1]+list(model.parameters())[3:4]+list(model.parameters())[2:3])\n",
    "W, bias, a_ = W.data, bias.data, a_.data\n",
    "A_1, B_1, A_2, B_2, a_2, b_2 = a_[:,:h], a_[:,h:], W[:h,:], W[h:,:], bias[:h], bias[h:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 651,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2000])"
      ]
     },
     "execution_count": 651,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.shape(bias)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Partition parameters for later usage. Parameters of second network, i.e., $\\theta'$ have additional underscore after the name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 663,
   "metadata": {},
   "outputs": [],
   "source": [
    "A_1, B_1, A_2, B_2, a_2, b_2 = get_partition(model_1)\n",
    "A_1_, B_1_, A_2_, B_2_, a_2_, b_2_ = get_partition(model_2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get a dummy model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 664,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = FCN(N=N_2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below we provide an example for the first line segement between $\\theta$ and $\\theta_1$. **Implement** the remaining pieces. All segments should be functions with ``_i`` postfix done in the same fashion as ``segment_1``."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 665,
   "metadata": {},
   "outputs": [],
   "source": [
    "segment_1 = lambda t: insert_params(model, \n",
    "    A_1 * (1-t) + 2 * A_1 * t, B_1 * (1 - t), A_2, B_2, a_2, b_2)\n",
    "# => Implement remaining 6 segments\n",
    "segment_2 = lambda t: insert_params(model, \n",
    "    2*A_1, 0, A_2, (1-t)*B_2 + t*A_2_, a_2, b_2)\n",
    "\n",
    "segment_3 = lambda t: insert_params(model, \n",
    "    (1-t)*2*A_1, t*2*A_1_, A_2, A_2_, a_2, b_2)\n",
    "\n",
    "segment_4 = lambda t: insert_params(model, \n",
    "    0, 2*A_1_, (1-t)*A_2 + t*A_2_, A_2_, a_2, b_2)\n",
    "\n",
    "segment_5 = lambda t: insert_params(model, \n",
    "    t*2*A_1_, (1-t)*2*A_1_, A_2_, A_2_, a_2, b_2)\n",
    "\n",
    "segment_6 = lambda t: insert_params(model, \n",
    "    2*A_1_, 0, A_2_, (1-t)*A_2_ + t*B_2_, a_2, b_2)\n",
    "\n",
    "segment_7 = lambda t: insert_params(model, \n",
    "    (1-t)*2*A_1_ + t*A_1_, t*B_1_, A_2_, B_2_, a_2, b_2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Getting test loss for 10 points in each linear segment for plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 666,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "  0%|          | 0/7 [00:00<?, ?it/s]\n",
      "\n",
      "\n",
      " 14%|█▍        | 1/7 [00:31<03:08, 31.48s/it]\n",
      "\n",
      "\n",
      " 29%|██▊       | 2/7 [01:02<02:36, 31.24s/it]\n",
      "\n",
      "\n",
      " 43%|████▎     | 3/7 [01:33<02:04, 31.15s/it]\n",
      "\n",
      "\n",
      " 57%|█████▋    | 4/7 [02:04<01:33, 31.17s/it]\n",
      "\n",
      "\n",
      " 71%|███████▏  | 5/7 [02:36<01:02, 31.33s/it]\n",
      "\n",
      "\n",
      " 86%|████████▌ | 6/7 [03:07<00:31, 31.29s/it]\n",
      "\n",
      "\n",
      "100%|██████████| 7/7 [03:40<00:00, 31.94s/it]"
     ]
    }
   ],
   "source": [
    "from tqdm import trange\n",
    "\n",
    "loss = []\n",
    "for i in trange(1,8):\n",
    "    name = f'segment_{i}'\n",
    "    line = globals()[name]\n",
    "    \n",
    "    for t in np.linspace(0,1,10):\n",
    "        loss.append(test(line(t), test_loader)[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Implement linear interpolation between $\\theta$ and $\\theta'$ to compare. ``segment_linear`` should be implemented in the same fashion as previously ``segment_i``. Linear interpolation means\n",
    "\n",
    "$$\n",
    "\\theta(t) = (1-t) \\theta + t \\theta', \\quad t \\in [0,1].\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 667,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "  0%|          | 0/70 [00:00<?, ?it/s]\n",
      "\n",
      "\n",
      "  1%|▏         | 1/70 [00:03<03:51,  3.35s/it]\n",
      "\n",
      "\n",
      "  3%|▎         | 2/70 [00:06<03:45,  3.32s/it]\n",
      "\n",
      "\n",
      "  4%|▍         | 3/70 [00:09<03:40,  3.28s/it]\n",
      "\n",
      "\n",
      "  6%|▌         | 4/70 [00:13<03:42,  3.37s/it]\n",
      "\n",
      "\n",
      "  7%|▋         | 5/70 [00:16<03:32,  3.27s/it]\n",
      "\n",
      "\n",
      "  9%|▊         | 6/70 [00:19<03:25,  3.21s/it]\n",
      "\n",
      "\n",
      " 10%|█         | 7/70 [00:22<03:18,  3.15s/it]\n",
      "\n",
      "\n",
      " 11%|█▏        | 8/70 [00:25<03:12,  3.10s/it]\n",
      "\n",
      "\n",
      " 13%|█▎        | 9/70 [00:28<03:12,  3.15s/it]\n",
      "\n",
      "\n",
      " 14%|█▍        | 10/70 [00:31<03:09,  3.15s/it]\n",
      "\n",
      "\n",
      " 16%|█▌        | 11/70 [00:34<03:04,  3.13s/it]\n",
      "\n",
      "\n",
      " 17%|█▋        | 12/70 [00:38<03:04,  3.17s/it]\n",
      "\n",
      "\n",
      " 19%|█▊        | 13/70 [00:41<02:59,  3.15s/it]\n",
      "\n",
      "\n",
      " 20%|██        | 14/70 [00:44<02:57,  3.16s/it]\n",
      "\n",
      "\n",
      " 21%|██▏       | 15/70 [00:47<02:51,  3.12s/it]\n",
      "\n",
      "\n",
      " 23%|██▎       | 16/70 [00:50<02:51,  3.17s/it]\n",
      "\n",
      "\n",
      " 24%|██▍       | 17/70 [00:53<02:46,  3.14s/it]\n",
      "\n",
      "\n",
      " 26%|██▌       | 18/70 [00:56<02:38,  3.04s/it]\n",
      "\n",
      "\n",
      " 27%|██▋       | 19/70 [00:59<02:29,  2.93s/it]\n",
      "\n",
      "\n",
      " 29%|██▊       | 20/70 [01:02<02:24,  2.89s/it]\n",
      "\n",
      "\n",
      " 30%|███       | 21/70 [01:06<02:44,  3.36s/it]\n",
      "\n",
      "\n",
      " 31%|███▏      | 22/70 [01:09<02:29,  3.11s/it]\n",
      "\n",
      "\n",
      " 33%|███▎      | 23/70 [01:11<02:18,  2.94s/it]\n",
      "\n",
      "\n",
      " 34%|███▍      | 24/70 [01:14<02:09,  2.82s/it]\n",
      "\n",
      "\n",
      " 36%|███▌      | 25/70 [01:16<02:05,  2.78s/it]\n",
      "\n",
      "\n",
      " 37%|███▋      | 26/70 [01:19<01:58,  2.70s/it]\n",
      "\n",
      "\n",
      " 39%|███▊      | 27/70 [01:21<01:54,  2.65s/it]\n",
      "\n",
      "\n",
      " 40%|████      | 28/70 [01:24<01:52,  2.68s/it]\n",
      "\n",
      "\n",
      " 41%|████▏     | 29/70 [01:27<01:51,  2.73s/it]\n",
      "\n",
      "\n",
      " 43%|████▎     | 30/70 [01:30<01:54,  2.86s/it]\n",
      "\n",
      "\n",
      " 44%|████▍     | 31/70 [01:33<01:52,  2.88s/it]\n",
      "\n",
      "\n",
      " 46%|████▌     | 32/70 [01:36<01:48,  2.86s/it]\n",
      "\n",
      "\n",
      " 47%|████▋     | 33/70 [01:39<01:42,  2.77s/it]\n",
      "\n",
      "\n",
      " 49%|████▊     | 34/70 [01:41<01:38,  2.73s/it]\n",
      "\n",
      "\n",
      " 50%|█████     | 35/70 [01:44<01:34,  2.70s/it]\n",
      "\n",
      "\n",
      " 51%|█████▏    | 36/70 [01:47<01:36,  2.83s/it]\n",
      "\n",
      "\n",
      " 53%|█████▎    | 37/70 [01:49<01:30,  2.74s/it]\n",
      "\n",
      "\n",
      " 54%|█████▍    | 38/70 [01:52<01:23,  2.62s/it]\n",
      "\n",
      "\n",
      " 56%|█████▌    | 39/70 [01:54<01:17,  2.50s/it]\n",
      "\n",
      "\n",
      " 57%|█████▋    | 40/70 [01:56<01:13,  2.44s/it]\n",
      "\n",
      "\n",
      " 59%|█████▊    | 41/70 [01:59<01:09,  2.41s/it]\n",
      "\n",
      "\n",
      " 60%|██████    | 42/70 [02:01<01:06,  2.37s/it]\n",
      "\n",
      "\n",
      " 61%|██████▏   | 43/70 [02:03<01:04,  2.39s/it]\n",
      "\n",
      "\n",
      " 63%|██████▎   | 44/70 [02:06<01:06,  2.56s/it]\n",
      "\n",
      "\n",
      " 64%|██████▍   | 45/70 [02:09<01:04,  2.56s/it]\n",
      "\n",
      "\n",
      " 66%|██████▌   | 46/70 [02:11<01:00,  2.53s/it]\n",
      "\n",
      "\n",
      " 67%|██████▋   | 47/70 [02:14<00:57,  2.49s/it]\n",
      "\n",
      "\n",
      " 69%|██████▊   | 48/70 [02:16<00:55,  2.54s/it]\n",
      "\n",
      "\n",
      " 70%|███████   | 49/70 [02:19<00:56,  2.69s/it]\n",
      "\n",
      "\n",
      " 71%|███████▏  | 50/70 [02:22<00:55,  2.76s/it]\n",
      "\n",
      "\n",
      " 73%|███████▎  | 51/70 [02:25<00:52,  2.78s/it]\n",
      "\n",
      "\n",
      " 74%|███████▍  | 52/70 [02:28<00:51,  2.88s/it]\n",
      "\n",
      "\n",
      " 76%|███████▌  | 53/70 [02:32<00:51,  3.02s/it]\n",
      "\n",
      "\n",
      " 77%|███████▋  | 54/70 [02:34<00:44,  2.80s/it]\n",
      "\n",
      "\n",
      " 79%|███████▊  | 55/70 [02:36<00:40,  2.69s/it]\n",
      "\n",
      "\n",
      " 80%|████████  | 56/70 [02:39<00:35,  2.57s/it]\n",
      "\n",
      "\n",
      " 81%|████████▏ | 57/70 [02:41<00:32,  2.47s/it]\n",
      "\n",
      "\n",
      " 83%|████████▎ | 58/70 [02:43<00:28,  2.40s/it]\n",
      "\n",
      "\n",
      " 84%|████████▍ | 59/70 [02:45<00:25,  2.34s/it]\n",
      "\n",
      "\n",
      " 86%|████████▌ | 60/70 [02:48<00:23,  2.31s/it]\n",
      "\n",
      "\n",
      " 87%|████████▋ | 61/70 [02:50<00:20,  2.31s/it]\n",
      "\n",
      "\n",
      " 89%|████████▊ | 62/70 [02:52<00:18,  2.32s/it]\n",
      "\n",
      "\n",
      " 90%|█████████ | 63/70 [02:54<00:16,  2.29s/it]\n",
      "\n",
      "\n",
      " 91%|█████████▏| 64/70 [02:57<00:13,  2.28s/it]\n",
      "\n",
      "\n",
      " 93%|█████████▎| 65/70 [02:59<00:11,  2.29s/it]\n",
      "\n",
      "\n",
      " 94%|█████████▍| 66/70 [03:01<00:09,  2.27s/it]\n",
      "\n",
      "\n",
      " 96%|█████████▌| 67/70 [03:04<00:06,  2.26s/it]\n",
      "\n",
      "\n",
      " 97%|█████████▋| 68/70 [03:06<00:04,  2.28s/it]\n",
      "\n",
      "\n",
      " 99%|█████████▊| 69/70 [03:08<00:02,  2.32s/it]\n",
      "\n",
      "\n",
      "100%|██████████| 70/70 [03:11<00:00,  2.32s/it]"
     ]
    }
   ],
   "source": [
    "from tqdm import trange, tqdm\n",
    "\n",
    "# => Implement linear interpolation\n",
    "segment_linear = lambda t: insert_params(model, \n",
    "    (1-t)*A_1 + t*A_1_, (1-t)*B_1 + t*B_1_, (1-t)*A_2 + t*A_2_, (1-t)*B_2 + t*B_2_, a_2, b_2)\n",
    "\n",
    "loss_linear = []\n",
    "for t in tqdm(np.linspace(0,1,70)):\n",
    "    loss_linear.append(test(segment_linear(t), test_loader)[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot test risk on linear path and piece-wise linear which uses dropout stability. Use ``loss_linear`` and ``loss`` list for corresponding values. **Is more \"intelligent\" path better?**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 673,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEGCAYAAACKB4k+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3dd3zV5dn48c+VTfaEBBJ2ABkynSi4UNxWcdbWVbHVatUuW3+11sc+jtr2aZ+naqmD2rpt3QsHiqKiYJAhssIKGSRA9jzJ/fvjPgkpRjhJzjnfc77ner9eeYWceX0P55zre6/rFmMMSimlFECU0wEopZQKHZoUlFJKddGkoJRSqosmBaWUUl00KSillOoS43QA/ZGdnW2GDx/udBhKKRVWVqxYUWWMyenpurBOCsOHD2f58uVOh6GUUmFFRLZ903XafaSUUqqLJgWllFJdNCkopZTqEtZjCj1pa2ujpKSE5uZmp0NxvYSEBPLz84mNjXU6FKWUn7guKZSUlJCSksLw4cMREafDcS1jDLt376akpIQRI0Y4HY5Syk9c133U3NxMVlaWJoQAExGysrK0RaaUy7guKQCaEIJEX2el3Md13UdKHUxzWztLN1XR1h7OZeMNxoAB7+/uf9t/d5j9LrN3s5d779fRdfl+l3W7H/vdtvN6uj1X98s7Hy/czRieyawxPa7vcjVNChFm69atfPTRR1xyySW9ut/ll1/OGWecwbx58wIUWfD8+/Od/PL51U6H4Wrh3og0BoZlJfL+T493OpSg06QQYbZu3coTTzzR66TgJtv3NBIbLbxw3UyE8P32EvH+IN7ftktPBKJEvH/vu77zPlH73Yb9HqPzerFXfOPj7f/cXfcL94wA3PX6Oh79cCvGGFccT29oUgiAhoYGLrjgAkpKSmhvb+dXv/oVo0eP5uabb6a+vp7s7GwWLlxIXl4en332GVdddRVJSUkcc8wxvP7666xZs4aFCxfywgsv0N7ezpo1a/jxj39Ma2sr//jHP4iPj+e1114jMzOTzZs3c91111FZWUliYiJ/+9vfGDduHJdffjmpqaksX76c8vJy7r33XubNm8ctt9zCunXrmDJlCpdddhk33HADt9xyC++99x4tLS1cd911XHPNNRhjuP7663n33XcZMWIEbtqhr6K2mYEpCUwYnOZ0KCpE5aYm0NrewZ6GVrKS450OJ6hcnRR+8/Javiyt9etjjh+cyq/PnHDA27zxxhsMHjyYV199FYCamhpOPfVUXnzxRXJycnj66ae59dZbeeSRR7jiiitYsGABRx99NLfccst/PM6aNWsoKiqiubmZ0aNHc88991BUVMRNN93EY489xo033sj8+fN58MEHKSwsZNmyZVx77bW8++67AJSVlfHhhx/y1VdfcdZZZzFv3jzuvvtu7rvvPl555RUAFixYQFpaGp999hktLS3MnDmTk08+maKiItavX8/q1aupqKhg/PjxXHnllX59LZ1SXtPMoNTI+qCr3slLSwCgrKZZk4Lqv0mTJvGTn/yEn//855xxxhlkZGSwZs0a5syZA0B7ezt5eXlUV1dTV1fH0UcfDcAll1zS9WUNcPzxx5OSkkJKSgppaWmceeaZXY+/atUq6uvr+eijjzj//PO77tPS0tL173POOYeoqCjGjx9PRUVFj7EuWrSIVatW8dxzzwE2gW3cuJElS5Zw8cUXEx0dzeDBgznhhBP8+yI5qKKumXG5KU6HoUJYbtoAwLYqJw6JrBalq5PCwc7oA2XMmDGsWLGC1157jV/84hfMmTOHCRMm8PHHH//H7fbu3XvAx4mP33eGEhUV1fV3VFQUHo+Hjo4O0tPTWbly5UHv/03dP8YY/vd//5dTTjnlPy5/7bXXXNuXWlHTzOwInFWifNe9pRBpXLlOwWmlpaUkJiZy6aWX8pOf/IRly5ZRWVnZlRTa2tpYu3YtGRkZpKSk8MknnwDw1FNP9ep5UlNTGTFiBM8++yxgv+C/+OKLA94nJSWFurq6rr9POeUUHnjgAdra2gDYsGEDDQ0NzJo1i6eeeor29nbKyspYvHhxr2ILVXXNbTS0tpObmuB0KCqEZSfHEx0llEdgUnB1S8Epq1ev5qc//SlRUVHExsbywAMPEBMTww033EBNTQ0ej4cbb7yRCRMm8PDDD3P11VeTlJTEcccdR1pa75qqjz/+OD/4wQ+48847aWtr46KLLmLy5MnfePtDDz2UmJgYJk+ezOWXX86PfvQjtm7dyrRp0zDGkJOTwwsvvMC3vvUt3n33XSZNmsSYMWOYPXt2f1+WkFBRaz/kuWmaFNQ3i44SBqXER2RLQcJ5VsmMGTPM/pvsrFu3jkMOOcShiHqvvr6e5ORkAO6++27Kysr405/+5HBUvgu31/vDjVVc+vAynpp/JEeOzHI6HBXCzr1/KQPionn8e0c6HYrficgKY8yMnq7TloLDXn31Ve666y48Hg/Dhg1j4cKFTofkauWdLQXtPlIHkZuWwFfldQe/octoUnDYhRdeyIUXXuh0GBFDu4+Ur3JTB/De+sqIW8CmA80qopTXNJM2IJaE2GinQ1EhLi8tgcbWdupaPE6HElSaFFREKa9t1q4j5ZPO1mSkzUDSpKAiSkVtMwN1NbPyQaSuVdCkoCJKhbYUlI/2tRSaHI4kuDQpBEDnFNPS0lJXlJp2C097B5V1LTrIrHwyMCUBEW0pKD8aPHhwV02hQPF4ImsQrD+q6lvpMDBIWwrKB3ExUWQlxeuYgvKfrVu3MnHiRAAWLlzIueeey9y5cyksLORnP/tZ1+0WLVrEUUcdxbRp0zj//POpr68H4I477uCwww5j4sSJzJ8/v6t+0XHHHccvf/lLZs+eHVYL3ZymaxRUb+WlJURcS8Hd6xRevwXK/bzDVu4kOPXuPt115cqVFBUVER8fz9ixY7n++usZMGAAd955J2+//TZJSUncc889/OEPf+C2227jhz/8IbfddhsA3/nOd3jllVe6KqVWV1fz/vvv++2wIkHnGZ92Hylf5aYlsGNPo9NhBJW7k0KIOfHEE7tqG40fP55t27ZRXV3Nl19+ycyZMwFobW3lqKOOAmDx4sXce++9NDY2smfPHiZMmNCVFHTBW+91LlzT7iPlq7y0BD7dssfpMILK3Umhj2f0gdK9lHV0dDQejwdjDHPmzOHJJ5/8j9s2Nzdz7bXXsnz5cgoKCrj99ttpbt7XjE1KSgpa3G5RXttMbLSQlRTndCgqTOSmJVDT1EZjq4fEOHd/XXbSMQWHHXnkkSxdupRNmzYB0NjYyIYNG7oSQHZ2NvX19QEfsI4EFTV2G86oqMgpWaD6Jy8CF7BFRuoLYTk5OSxcuJCLL764a9e0O++8kzFjxnD11VczadIkhg8fzmGHHeZwpOGvvFa34VS9k5tqd2Arr2lmZE6yw9EEh5bOVv0STq/3Cb9/j3G5Kdz/7elOh6LCxJaqBo6/7z1+f/5kzpue73Q4fnOg0tnafaQixq7aFgam6CCz8l3n9OXO6cyRQJOCigj1LR7qWzw6HVX1yoC4aNITYyNqTMGVSSGcu8TCSTi9zl1rFHQ6quql3NTIWsAWsKQgIo+IyC4RWdPDdT8RESMi2d6/RUT+LCKbRGSViEzr6/MmJCSwe/fusPrCCkfGGHbv3k1CQnh8yeoaBdVXeWkJlNdGTlG8QM4+Wgj8H/BY9wtFpACYA2zvdvGpQKH35wjgAe/vXsvPz6ekpITKysq+3F31QkJCAvn54TH4pquZVV/lpg1g9c4ap8MImoAlBWPMEhEZ3sNVfwR+BrzY7bKzgceMPb3/RETSRSTPGFPW2+eNjY1lxIgRfQlZuZjWPVJ9lZeWQFV9Ky2eduJj3L9jX1DHFETkLGCnMeaL/a4aAuzo9neJ97KeHmO+iCwXkeXaGlC+qqhtJjUhhgFx7v9QK//qPJHYVdvicCTBEbSkICKJwK3AbT1d3cNlPQ4KGGMWGGNmGGNm5OTk+DNE5WLlNc3adaT6JDfCdmAL5ormUcAI4AsRAcgHPheRw7Etg4Jut80HSoMYm3K5itpmHWRWfdJV6iJC1ioEraVgjFltjBlojBlujBmOTQTTjDHlwEvAd72zkI4EavoynqDUNynXbThVH0XatpyBnJL6JPAxMFZESkTkqgPc/DWgGNgE/A24NlBxqcjT3mGorGvRloLqk5SEWJLjY7T7qL+MMRcf5Prh3f5tgOsCFYuKbFX1LXYbTh1TUH2Um5YQMauaXbmiWanudDWz6q9I2pZTk4JyPV2joPorN1VbCkq5RleJizTdS0H1TW5aArvqmvG0dzgdSsBpUlCuV17TTEyUkJ2kSUH1TW5aAh0GqupbnQ4l4HTntRD08helLPxoq9Nh9EtMlHDH2RMZm5vidCiU1zYzMCVet+FUfda5VqG0psn1iyA1KYSgZ5bvYGNFHYfmpzsdSp8YDEs37ebtdRUhkRQqapt15pHql/yMRAB27Glk2tAMh6MJLE0KIai4soHjxw3kTxdNdTqUPjvqrnfYvKve6TAAKKtuZlye88lJha+hmTYpbN/d6HAkgadjCiGmua2d0pomRmaH9ybho3KS2VzV4HQYeNo72L6nkeFZSU6HosJYQmw0g1Lj2bZHk4IKsi1VDRgDI3PC+0tsZE4SxbvqHd/saMfeJjwdhhHZ4f16KucNy0zSloIKvuJKe3Yd7l9io3KSqWvxUFnnbLnhLVW2Cyvck6xy3tCsRLbtcb71G2iaFEJMcaU7vsRG5djur82Vzn6IOpNsuHfHKecNy0ykoraF5rZ2p0MJKE0KIWZLVQN5aQkkxoX3HIDOpLa50tnB5i1VDaQnxpKRFOdoHCr8Dc3yDja7fFxBk0KI2VzVEPatBLBlARLjokMiKYR7V5wKDcO8kxW2uXxcQZNCCDHGUFxZ74qujqgoYWROkuPdR5oUlL8M805L3bbb3eMKmhRCSFV9K3XNHle0FMCOKxQ72FJobPVQVtPMSE0Kyg/SE2NJSYjR7iMVPJ1foG45sx2ZnczO6iaaWp0ZmNtS1TmTK/xbXsp5IsKwrETtPlLBU+z9EuucuRPuRg1Mwph9X87Bti8puCPJKucNy0zSloIKni1VDcTFRDE4fYDTofjFvmmpznQhbXHJmg8VOoZmJVKyt5H2DmcXZQaSJoUQUlxZz4isJKJdUs1zRHYSIvvWCgTblqoGBqclMCAu2pHnV+4zLDORtnZDaXWT06EEjCaFEFJc6Y7pqJ0SYqMZkj7AsZZCcVUDI1z0eirnRcJaBU0KIaLNW7jNTUkBvIXxHEgKndN7tetI+VMkrFXQpBAitu9p9BZuc8cgc6eROUkUVzbQEeQ+2D0NrdQ2e1z3eipn5aYmEBstrq6BpEkhRHTV6HFhS6GprZ3y2uBuet4580jXKCh/io4SCjISXV0tVZNCiOis5jnKZWe2Ts1AKtbpqCpAhmYl6piCCrziygaykuJIS4x1OhS/GjXQWxgvyLuwbalqIDZayM9wx/ReFTqGZdqWgtN7hQSKJoUQ4baZR51ykuNJiY/pOnMPli2VDQzNTCQmWt/iyr+GZiVR1+Jhb2Ob06EEhH5iQkRxlTsK4e1PRBg5MPgzkGwhPPe9nsp5bi+Mp0khBNQ0tVFV3+raOfWjcpLYvCt4H6D2DsOW3e5seSnnDXP5WgVNCiHA7TNlRuUkU17bTH2LJyjPV1rdRKunQweZVUAUdLUUNCmoANm3Bac7uztGec/YtwSp3IUWwlOBlBAbTW5qgiYFFTjFlQ1ERwlDvWcgbhPsaaldLS/tPlIBYqel6piCCpDiqnqGZiYSF+PO/46hWYlESXCTQnJ8DDnJ8UF5PhV5hmW6d18Fd34LhZniygbXjicAxMdEMzQzMWhJodi7BaeIO6rNqtAzLCuRXXUtjm0gFUiaFBzW0WEiYh/h0QNT+Kq8LijPpYXwVKAN9RbGc+MMpIAlBRF5RER2iciabpf9TkS+EpFVIvK8iKR3u+4XIrJJRNaLyCmBiivU7NjbSIung9ED3TnI3GlKQRrFlQ1UN7YG9Hma29rZWd2kSUEFlJvXKgSypbAQmLvfZW8BE40xhwIbgF8AiMh44CJggvc+94tIROyM0nn2PDY3xeFIAmvasAwAinZUB/R5tu9pxBgdZFaB5ea1CgFLCsaYJcCe/S5bZIzpnKz+CZDv/ffZwFPGmBZjzBZgE3B4oGILJeu9SWHMIHcnhcn56UQJfL5tb0CfZ21pDeD+11M5Kz0xjtSEGFcONjs5pnAl8Lr330OAHd2uK/Fe9jUiMl9ElovI8srKygCHGHjry+sYmplIUnyM06EEVFJ8DONyU/l8e2CTQtH2apLiojUpqIAbkZNMcZUzuwoGkiNJQURuBTzA450X9XCzHksQGmMWGGNmGGNm5OTkBCrEoPmqvNb1XUedpg/LYOX26oBuel60vZrJBemu2edaha7CgclsrNCk0G8ichlwBvBts6/2bAlQ0O1m+UBpsGMLtua2drbubmRchCSFacPSaWht7+oy87em1nbWldUydWj6wW+sVD8VDkxmV10LNS6rlhrUpCAic4GfA2cZY7p3xr0EXCQi8SIyAigEPg1mbE7YtKue9g4TOS2FoZkAAetCWlNag6fDMLUgIyCPr1R3nV2UG3cFZ6p1sARySuqTwMfAWBEpEZGrgP8DUoC3RGSliDwIYIxZCzwDfAm8AVxnjHHfqpD9dJ4xR0pLoSBzANnJcQEbbC7yJpsp2lJQQdA5jXxjkDeQCrSAjW4aYy7u4eKHD3D73wK/DVQ8oWh9RR1xMVEMz4qM6ZMiwrShGQFrKXy+rZqhmYlka3kLFQRD0gcwIDbadeMKuqLZQV+V1zE6JzmidgebNiyDrbsbqapv8evjGmP4fPteHU9QQRMVJRQOStbuI+U/68trI6brqNP0zkVs2/27iK2spplddS1MLdCkoIJntAtnIGlScEh1YysVtS0RM8jcadKQNGKihBV+HlfoTDJTh+ogswqeMYNSKK9tpqbJPTOQNCk4JFLKW+wvITaaCUPS/D6uULR9L/ExURySl+rXx1XqQAq9g82bXDTYrEnBIftmHkXel9i0oemsKqmmrb3Db49ZtKOaSUPSXLsnhQpNhQPtSd0mF40r6CfIIV+V15E2IJZBqZE3U2b6sAya2zpYV1brl8dr9XSwemeNDjKroMvPGEBCbBQbXDSuoEnBIeu95S0icSOYad5+f3+tV1hXVkurp0PHE1TQRUWJHWyOtO4jEfmRiKSK9bCIfC4iJwc6OLcyxrChoj7iZh51Gpw+gNzUBFb4aQZS56I1bSkoJxQOTGFjReR1H11pjKkFTgZygCuAuwMWlcuV7G2ivsUTcYPM3U0fluG3lkLRjmpyUxPISxvgl8dTqjcKByVTVtNMXbM7ZiD5mhQ6+zhOAx41xnxBz5VNlQ8irbxFT6YOTWdndRPlNc39fixdtKactG+w2R1dSL4mhRUisgibFN4UkRTAf1NHIsz6isjYWOdAjhyZBcD7G3b163Eq61rYsadJk4JyzJhB3hpILhls9jUpXAXcAhzmrW4ah+1CUn3wVXkdQ9IHkJIQ63QojpkwOJWCzAG8vqa8X4+zcocuWlPOys9IJD4myjXlLnxNCgYYD9zg/TsJSAhIRBEgEstb7E9EOHViHks3VfVrNWjR9r3ERAmThqT5MTqlfBcdJYzKcc8MJF+Twv3AUUBn5dM64C8BicjlWj0dFFc2RPQgc6e5E3Npaze8s66iT/c3xrDoywqmFKSTEBvt5+iU8t2YQe6pgeRrUjjCGHMd0AxgjNmL7UJSvbS5sh5PBG2scyBT8tPJTU3ocxfSyh3VbNpVz7zp+X6OTKneKRyUws5qO6sw3PmaFNpEJBrvvskikoMONPdJJJe32F9UlDB3Yi5LNlTS0IcP0zPLS0iIjeL0Q/MCEJ1SvnNTDSRfk8KfgeeBgSLyW+BD4L8DFpWLrdlZQ1xMFCNzImNjnYM5dWIuLZ4OFq/v3SykptZ2XvmilNMm5UX0gL0KDYWdW3O6YBGbTzuvGWMeF5EVwInY9QnnGGPWBTQyl1rpLdwWG0Eb6xzIjOGZZCfH8frqcs44dLDP93tjbRl1LR7On14QwOiU8s3QzETiYqIip6UgIqOALcaYvwBrgDkiohPDe6mzcNsU3QimS3SUcPKEXBav30Vzm+/bcj+7vIShmYkcMSIzgNEp5ZvOGUgbXNBS8PV09V9Au4iMBh4CRgBPBCwql/qqvJYWT4cutNrPqRNzaWxt5/0NlT7dfseeRj7avJt50/OJitKF9So0FLqkMJ6vSaHDGOMBzgX+ZIy5CdDRvV7ShVY9O3JkFmkDYnnDx1lIz60oQQTO01lHKoQUDkzuqmsWznoz++hi4LvAK97LdHSvl4q2V5OTEs/gNF33111sdBQnjx/E2+sqaPEcuAupo8Pw3IoSjhmdzZB0LYCnQsfEfLuAcnVJjcOR9I+vSeEK7OK13xpjtojICOCfgQvLnVbuqGZqQXpE7qFwMKdOyqWu2cNHm3Yf8HafFO9mZ3UT58/QAWYVWqbk227hzh6BcOVTUjDGfGmMucEY86SIZAApxhgtnd0Lexta2VLVwBQdT+jRzNHZpMTH8MD7mw844PzM8h2kJsRw8vhBQYxOqYPLSIpjWFYiK3f4d//xYPN19tF73k12MoEvgEdF5A+BDc1dVpZ4xxMKdDyhJ/Ex0fzm7Al8umUPNzxZhKeH/ZuXbKjk9TXlnDVlsJa1UCFpSkF6ZLQUgDTvJjvnYvdTmA6cFLiw3KdoezVRAofma+G2b3LutHxuP3M8i76s4GfPraKjwwC2xtFDHxRz+aOfMiI7ietPKHQ4UqV6NqUgnYraFspqmpwOpc98WrwGxIhIHnABcGsA43Gtou17GTMohaR4X1/yyHT5zBHUNXv4/VsbSE6I4ZenHcL/e2ENz60oYe6EXH5/wWR9DVXI6lyDtHJ7NXmTwnMihK+frjuAN4GlxpjPRGQksDFwYblLR4fhix3VnN6LFbuR7IcnjKauxcOCJcUsWltBeW0zN55UyA0nFOq6BBXSxg9OJS46ipU7qjl1UnjO2ve1zMWzwLPd/i4GzgtUUG5TXNVAbbOHqbqS2Sciwi9OHUdjq4fnP9/JA9+eFrYfMBVZ4mOiOWRwKkVhPK7g60Bzvog8LyK7RKRCRP4lIrpyyEf7Fq1pUvCViHDnOZMouu1kTQgqrEwtSGd1SU2PkyXCga8DzY8CLwGDgSHAy97LlA+Ktu8lJT6GUTnJTocSduJitHCgCi9TCtJpamtnQ5huuuPrJy7HGPOoMcbj/VkI5AQwLldZuaOayQXp2h+uVAToGmwO0y4kX5NClYhcKiLR3p9LgQMvPVUANLZ6+Kq8TiujKhUhhmUlkpEYG7aL2HxNCldip6OWA2XAPGzpC3UQq0tqaO8wOp6gVIQQESaH8SI2X8tcbDfGnGWMyTHGDDTGnINdyPaNROQR78D0mm6XZYrIWyKy0fs7w3u5iMifRWSTiKwSkWn9OqoQ0vnG0JaCUpFjSkE6G3fVU9fc5nQovdafUbybD3L9QmDufpfdArxjjCkE3vH+DXAqUOj9mQ880I+4QkrR9mqGZiaSlRzvdChKqSCZUpCOMeFZMbU/SeGAo6bGmCXAnv0uPhv4u/fffwfO6Xb5Y8b6BEj3rqAOa8YYVmzfq11HSkWYyd6KqeG4XqE/ScH04T6DjDFlAN7fA72XDwF2dLtdifeyrxGR+SKyXESWV1b6tlOXU9aV1VFZ18LM0dlOh6KUCqKMpDiGZyWG5bjCAVc0i0gdPX/5C+DPwh49tTp6TDrGmAXAAoAZM2b0JTEFzZKNNmnNHqOzd5WKNFMK0lm6eTfGmLDaQ+WALQVjTIoxJrWHnxRjTF+qklV0dgt5f+/yXl4CdN81JR8o7cPjh5T311cyLjeFQam605pSkWZKQTqVdS2U1jQ7HUqvBHu56EvAZd5/Xwa82O3y73pnIR0J1HR2M4WrhhYPy7ft0VaCUhFqincv9pXbw6sLKWBJQUSeBD4GxopIiYhcBdwNzBGRjcAc798ArwHFwCbgb8C1gYorWD4p3k1bu2GWJgWlItIheSkkxEbx6ZbwWucbsML0xpiLv+GqE3u4rQGuC1QsTnh/QyUDYqOZMVx3WlMqEsXHRHPEiCw+2FjldCi9otXGAmTJhkqOGpVFfIxuG6lUpJo1JofiqgZ27Gl0OhSfaVIIgG27G9i6u5FZhToVValI1vkdEE6tBU0KAbBkg3cq6tiBB7mlUsrNRg9MJi8toes7IRxoUgiA9zdUUZA5gOFZiU6HopRykIgwqzCHpZurwmbTHU0Kftbq6eDjzVXMHpMTVgtWlFKBMWtMDnXNnrBZ3axJwc9WbNtLQ2s7swp1KqpSCmaOziJKYEmYjCtoUvCzJRsriYkSjtZ6R0opID0xjkPz08NmXEGTgp+9v76S6cMySI4P2BIQpVSYmTUmh1Ul1VQ3tjodykFpUvCjyroWviyr1VXMSqn/MHtMNh0Glm4K/dXNmhT86O11FQAcN1aTglJqn8n56aQkxIRFF5ImBT96oWgno3KSGJ+X6nQoSqkQEhMdxTGjs1mysRJb1Sd0aVLwk9LqJj7duoezpwzRqahKqa85tjCHsppmNu2qdzqUA9Kk4CevrCrFGDhr8mCnQ1FKhaBZY+yMxFCfmqpJwU9eXFnK5IJ0hmcnOR2KUioE5WckMjInKeTHFTQp+MGmXXWsLa3lbG0lKKUO4PixA/l4825qGtucDuUbaVLwg5dWlhIlcMbkPKdDUUqFsHOmDKG1vYNXV4fuxpKaFPrJGMMLK0uZOTqbgSm6F7NS6ptNHJLKqJwkni8qcTqUb6RJoZ9W7qhm+55GHWBWSh2UiHDutHw+27o3ZDfe0aTQTy+uLCUuJopTJuY6HYpSKgycPcWeQL5QtNPhSHqmSaEfPO0dvLKqjBPHDSQ1IdbpcJRSYSA/I5EjRmTyfNHOkFzIpkmhHz4u3k1VfUtX5ldKKV+cO20IxVUNfFFS43QoX6NJoR+e/3wnKQkxHKfbbiqleuHUSXnExUSFZBeSJoU+Kqtp4uVVpU4LMbAAABaJSURBVJw7dQgJsdFOh6OUCiOpCbHMOWQQL39RSluIbdOpSaGPHl26lQ4D3zt2pNOhKKXC0LemDmF3Q2vIrXDWpNAHNU1tPLFsO6dPyqMgM9HpcJRSYWj22Bwyk+L4d4h1IWlS6IMnlm2nvsXD/FnaSlBK9U1sdBRnHprHW19WUNscOmUvNCn0UounnUeWbuHYwmwmDklzOhylVBg7b3o+rZ4Onvp0u9OhdNGk0EsvFO2ksq6Fa2aNcjoUpVSYOzQ/ndljcrj/vc0h01rQpNALHR2Gvy4pZsLgVGaOznI6HKWUC/z0lLFUN7bx0JJip0MBNCn0ytvrKiiubOCa2aN0dzWllF9MHJLG6Yfm8dCHW6iqb3E6HE0KvjLGthLyMwZwmtY5Ukr50c1zxtDi6eD+xZudDkWTgq+e/mwHK7bt5ZpZI4mJ1pdNKeU/o3KSmTctn39+so2d1U2OxqLfbj5Ys7OG215ay7GF2VxyxDCnw1FKudCPTioEgT+9vcHRODQpHERNYxs/eHwFWUlx/M+FU4iO0rEEpZT/DU4fwHeOHMZzK0rYtKvesTgcSQoicpOIrBWRNSLypIgkiMgIEVkmIhtF5GkRiXMitu46Ogw3P7OS8ppm/vLtaWQlxzsdklLKxa49bhQDYqP5zctrae9wpqx20JOCiAwBbgBmGGMmAtHARcA9wB+NMYXAXuCqYMe2vwfe38w7X+3i1tMOYdrQDKfDUUq5XFZyPLeePp4PNlbxuzfXOxKDU91HMcAAEYkBEoEy4ATgOe/1fwfOcSg2u+9y0U5+v2g9Z04ezGVHD3cqFKVUhLnkiKFceuRQHnx/syOltWOC/YTGmJ0ich+wHWgCFgErgGpjjMd7sxJgSE/3F5H5wHyAoUOH+j2+TbvquP2lL/lwUxWT89O469xJuiZBKRVUvz5zApt21fOzf61iRHYSkwvSg/bcTnQfZQBnAyOAwUAScGoPN+2xQ80Ys8AYM8MYMyMnJ6dPMRRv2cLnD17NUx9tZPH6Xawvr2NXbTN3vbaOuf/zAatKqrnj7An86wdHkxwf9LyplIpwsdFR3P/t6QxMiWf+P5ZTUdsctOd24hvvJGCLMaYSQET+DRwNpItIjLe1kA+UBiqAvesWM738Gfbu3Mj8tpto6/YyXDAjn5/NHUe2DiorpRyUmRTH3747g/Me+IirH1vOA5dOZ0j6gIA/rwR742gROQJ4BDgM2320EFgOzAL+ZYx5SkQeBFYZY+4/0GPNmDHDLF++vE9xtH/6MNGv3czeoSfz4dT7qKhvZ8bwTKYEsZmmwtzerVCx1v7eu83+rivja43cpBzIGL7vJ3ss5IwF7ZZUPli0tpzrnywCYP6skXx/9iiS+tmDISIrjDEzerwu2EkBQER+A1wIeIAi4HvYMYSngEzvZZcaYw5YCKQ/SQGAZQvg9Z/C+HPgvIchWruK1AE07IYt70Pxe/anetu+6+JS7Bd+ah5EdXsfmQ6or7AJo2nvvsuTBsLI2TDyOPuTlh/4+FXY2lndxL1vfMWLK0sZmBLPb2YmcMrMw4iK7VuPRsglBX/pd1IA+Pgv8OYvYdL58K2/QpTut6y68bTA+teg6HHY/I79ko9PhRGz7Jf5kGmQMQIGZBz8zL+p2iaSslX7kkuDdyvG/MNh2ndgwrcgPiXAB6XC1Ypte3nu+We5Ze+vWTfwdI687qE+Pc6BkoKeGh91HbS3wtu327/PeQCiYx0NSYWAXV/B8odh9bP2DD91CMy8EcaeBoOn9q1VOSDd/uRNtgnAGNj1JWxcBCufgJeuh9d/bhPDjCshv8fPrIpg05s+Zlr9r6hPySPn5B8H5Dk0KQAcc5P9/fbt0FIH5y+E2MAP6KgQVFoES+6Dr16B6Hg45AyY8m3bKvB3K1IEBk2wPzNvhJLPoOgfsObfsPJxGDEbZv0Uhh+j4w8KPv8HvHwDMngqKZc8S0pSYPZ00e6j7j57GF79MQybCRc/CQmp/ntsFdq2fwJLfgeb3oaENDj8Gjji+xCgD94BtdTDikdh6Z+hYRcMPQpm/QRGnajJIRIZAx/+Ad65w74HLngM4pP79ZA6ptAbq5+D56+xZ2+X/huSsv37+Cq0VHwJ7/wGNrwBidm2O/Gw74XGCUFbkz07XPo/ULsThh8Lp/zWdj+pyNDugTdugc/+Zsc9z74fYvpfFk6TQm9tWATPfAfSCuDS5+ysEuUuNSWw+C744gk7c+jYm2zrIC7R6ci+ztMKKxbCe3fZ8Y3JF8OJv4LUwU5HpgKppQ6eu9KOOR19PZx0B0T5Z72xJoW+2PYxPHkRRMfBJU/bWSYq/LU22qb40j8DBg6fD8f+GBIznY7s4Jqq4YPfw7IHQaLhmBth5o90/MuNanbCExfaiQin/x5mXOHXh9ek0FeV6+HxedBQBfMehbFzA/dcKrCMgXUvwZu3Qs0O2xQ/8TZI93/9rIDbu9VOilj7PKQNhbn/DePO0PEGtyhdaU9IW+rhgr/D6BP9/hQHSgq6yc6B5IyFq96G7DHw1MV2IFqFn8oN8I9vwTPftWsMrngdznsoPBMC2O7M8xfCZa/YAcenL7XHV+lMqWXlR6ufg0fm2gWQVy0KSEI4GE0KB5MyCC5/FQpPhldvtmea7Z6D3085z9MK798LD86EnZ/DqffCNUtg2NFOR+YfI46Faz6wx1X6OTxwNLx1mz3DVOGl3WO/W/51lV0Hc/W7MGi8I6Fo95Gv2j125fOnf7XTwuY9bFexqtC0cwW8eD3sWmsXg516LyQPdDqqwGmogrd/DUX/tAvt5t4Fh5ylXUrhoGE3PHeFXeV++Hw45b8DvoBWu4/8IToGTrsXzvwzbFkCfzvRdkuo0NLaaM+4HjoJmvbARU/arhY3JwSwU6fP/gtcucierDzzXfjnebB7s9ORqQPZ+TksOM6ukzn7fjjtd45XVNCk0FvTL4PLXobmGnjoRNjwptMRqU7bPrJdRR//H0y7DK5bBuNOczqq4Bp6BMx/H+beDTs+tV1KS+6zXWkqdBgDy/4KD59s62ld8TpM/bbTUQGaFPpm2FEw/z3IGAZPXABv/NIWTlPOaG2wNYMePQ062m3SPvN/7MrkSBQdA0f+AH74GYw5Bd79L/jrLHs2qpzXVG3XQb3+Mxh9Enz/A8if7nRUXTQp9FV6AVz1Fhx2NXzyF293ks7+CLotH9iz4WUP2v7YH3xkK5gqW8b7gsfg4qehtR4eOQVeusH2YStnlKywCXr963Dyb205nRBbI6NJoT9iB8Dp98HFT9kyBH+dDcsftU1DFVjNNfDyjfD3MwCxM8ROu7ffNWFcaexcuPYTOOqHdiD6/6bb6dUd7U5HFjnaPfDePfDwnH3dRUf/MCQnAujsI3+pLYMXvm9r5I86Ac74o5bHCJQNb9qEUF8OR14Lx98amuUpQtGudfDaT2HrB5B7qF0tW3C401G5W9UmeH6+nRF36IV2JtwAZ3d41NlHwZCaB5c+D6fdZwf47j/KbuCjZ2P+U18J/7rajuMkpNmFhaf8VhNCbww8xI65zHvETmN9eA78e749qVH+1dEBnz0Efz3WzgKb9yicu8DxhHAw2lIIhJoSeOVm2PgmDJ4GZ/0Zcic5HVX46uiAosfgrV/bQeVjf2x//FAtMqK11MMH99mTl6hYOPZm28UUm+B0ZOGvahO8/CPY9iGMPB7OuT+kChhq7SMnGANr/w2v/cxWtjzi+3D8L3Srxd6qWAuv3AQ7lsGwY2y3XM4Yp6Nylz3FsOhXdmOh9GF28dS400OyvzvktbfBR3+24wcxCXDKnTD1OyH3WmpScFLjHrs5xoqFkJJrV5qOPyfk3iQhp7nGlqhY9qDtKjr5TlsyWl+3wNm8GN74BVSus2e3p95j638p3+z4zJ7AVKy2q8lP+539zIcgTQqhYMdntnZS+So7ED33Hj3j7UlHu92S8p3/gsbdMO27cNLtITdtz7XaPXZv6sW/tV11h18Dx/08ctd8+KKuHN7+jd2bIyXPJoNDznQ6qgPSpBAq2j124Gnxf0Nbg51XP/vnIT/wFDRbl9pdpspX2S0o594Ng6c4HVVkaqiCd++0LdzELDjuFph+ueMlGEKKpxWWPWBbtO2tdte+Y38cFl3EmhRCTX0lLL4TVvzdngGf8Ct7RuzvjeHDRdkXtmWw6S1IzYeT74AJ52pXUSgoXWlrSW37EDJH2VbbIWdG9v9NR4cdL1z8WzseM2auHYfJGuV0ZD7TpBCqyr6wfbjblto9G467BcZ/y29b7oW8qo32g7X2eUhIh2Nusq0nnWIaWoyxW0K+dRtUfgUFR8BJv7HlXiKJMXYv73f+y1bfHTgB5vwGCuc4HVmvaVIIZcbAupdtl1LlOvtGO/4X7t5Jq2ItfPhHWPMviBlgm91H/1D7rUNduwdWPm7fq/Xldo+RE34FeYc6HVlgtXtg/at2C9edyyFzpF0wOeHcsD2B06QQDjraYc2/7ebsezbDoEn2i3LCue6Zj7/9E/jgD3b9Rlyy7aOeeSMk5zgdmeqN1kb4dIFN7M3V9j16/C8hu9DpyPyrvhI+X2hL19TutFufzvoJTLkk7MdWNCmEk3YPrHoalv4Jqtbb2QyHz7dfoOE4A6ep2va/Fj1uz7IGZNoKnod9LzyPR+3TVG3LlH98P7Q12rUNR19vu5fCtZXb0WE3u1n5OHz5oh1AHnm8/QyOOcU1436aFMJRRwdsfgc++l/7Jo1NtOsbpl5qt5MM5Q+dp8VWL/3iCVj3CrS3QM4hNrFN+w7EJTkdofKn+kq7I+FnD9mFmvmH2ZXR404PnzPqPVtg5RPwxZNQs8OOcR16oT15ceHUcU0K4a58td2QY+3ztgRyxgiY8m2YdJ7t3wwFNSWw8S37U/yenXKbkA6Tzrebh+RNCe1EpvqvtcF+sX78F9i7BZJyYNIFtrsld6LT0X1dbSmsfcGObe1cDohdQzT1Uhh7mqvLfWhScIvWBjsoXfRPW+USIKvQDvgVzrEtiJj4wMdhDFRvg20f25lT2z+G3ZvsdWkF++IZdUJw4lGhpaPdzlZa+TisfwM62mxF1onn2ffGwEOcOUEwBqo2wKa34avX7HsX443tXJvA0oYEPy4HaFJwo73b7PS4jYtsV017i53JM3gq5M/w/hxmxyT68wH0tNq52BVrbIul83d9hb0+Ic0uNBs20yaCnHHaIlD7NOy2Z+IrH4eylfay1CF2x7HRJ9r3aKAKxXW0226h8lV2X/VN70DNdntdzjg7QD7xXPcNkPtAk4LbtTbYxFC8GEqW2w9Bu3dP3rgUSB+67yclFxJSIT7VzgCKS4S2ZmiptT/NtVBXZs/8d2+2/aumwz5WVKz9MOVOhCHTbTIYOD5sp+WpIKvZac/SN70Fxe/b9xtAci4MmWYrCmePhpTBNlGk5B54TMIY8DRDQyVU77Dv1ZodsHcrVHxp947wNNnbxiXDyONsIhp1ot1KN4JpUog0nhZ7Nr9zhT3Lr95uf/Zug9a6g98/LgWyRkLWaLuKNWs0DJpgF9i5ZXqsclZ7G5QWwc7PofRz+3v3xv1uJLYETHScPSGJjoGoGHsS01pnS3+bHvYrSRpou6gGTbTv20Hj7foffe92OVBSiAl2MCoIYuL3dSF1Z4ydOthSDy119oPV2mC3FY33th7iU+zf2gWkAik61u741n3Xt+Zae6ZfW2rXBdSWQWOVTSAdHtv67fDYbtL4ZHv2H59sazOlFdiWcFq+ff+qPnMkKYhIOvAQMBEwwJXAeuBpYDiwFbjAGLPXifhcS8ROB41LgpRBTkej1H9KSIWECfbsXjnGqc7gPwFvGGPGAZOBdcAtwDvGmELgHe/fSimlgijoSUFEUoFZwMMAxphWY0w1cDbwd+/N/g6cE+zYlFIq0jnRUhgJVAKPikiRiDwkIknAIGNMGYD398Ce7iwi80VkuYgsr6ysDF7USikVAZxICjHANOABY8xUoIFedBUZYxYYY2YYY2bk5GghNaWU8icnkkIJUGKMWeb9+zlskqgQkTwA7+9dDsSmlFIRLehJwRhTDuwQkc4dwU8EvgReAi7zXnYZ8GKwY1NKqUjn1DqF64HHRSQOKAauwCaoZ0TkKmA7cL5DsSmlVMRyJCkYY1YCPa2mOzHYsSillNonrMtciEglsK2Pd88GqvwYTjjQY44MesyRoT/HPMwY0+NMnbBOCv0hIsu/qfaHW+kxRwY95sgQqGPW8pZKKaW6aFJQSinVJZKTwgKnA3CAHnNk0GOODAE55ogdU1BKKfV1kdxSUEoptR9NCkoppbq4PimIyFwRWS8im0Tka4X3RCReRJ72Xr9MRIYHP0r/8uGYbxaRL0VklYi8IyJhv2HtwY652+3miYgRkbCfvujLMYvIBd7/67Ui8kSwY/Q3H97bQ0VksbcC8yoROc2JOP1FRB4RkV0isuYbrhcR+bP39VglItP6/aTGGNf+ANHAZmy57jjgC2D8fre5FnjQ+++LgKedjjsIx3w8kOj99w8i4Zi9t0sBlgCfADOcjjsI/8+FQBGQ4f17oNNxB+GYFwA/8P57PLDV6bj7ecyzsAVD13zD9acBrwMCHAks6+9zur2lcDiwyRhTbIxpBZ7CbubTXffNfZ4DThQJ6w2KD3rMxpjFxphG75+fAPlBjtHffPl/Bvgv4F6gOZjBBYgvx3w18Bfj3dbWGBPulYd9OWYDpHr/nQaUBjE+vzPGLAH2HOAmZwOPGesTIL2z2nRfuT0pDAF2dPu7xHtZj7cxxniAGiArKNEFhi/H3N1V2DONcHbQYxaRqUCBMeaVYAYWQL78P48BxojIUhH5RETmBi26wPDlmG8HLhWREuA1bPFNN+vt5/2gnKqSGiw9nfHvPwfXl9uEE5+PR0QuxRYmnB3QiALvgMcsIlHAH4HLgxVQEPjy/xyD7UI6Dtsa/EBEJhq7/W048uWYLwYWGmN+LyJHAf/wHnNH4MNzhN+/v9zeUigBCrr9nc/Xm5NdtxGRGGyT80DNtVDnyzEjIicBtwJnGWNaghRboBzsmFOAicB7IrIV2/f6UpgPNvv63n7RGNNmjNkCrMcmiXDlyzFfBTwDYIz5GEjAFo5zK58+773h9qTwGVAoIiO8ezdchN3Mp7vum/vMA9413hGcMHXQY/Z2pfwVmxDCvZ8ZDnLMxpgaY0y2MWa4MWY4dhzlLGPMcmfC9Qtf3tsvYCcVICLZ2O6k4qBG6V++HPN2vCX4ReQQbFJw82buLwHf9c5COhKoMd697vvK1d1HxhiPiPwQeBM7c+ERY8xaEbkDWG6MeQl4GNvE3IRtIVzkXMT95+Mx/w5IBp71jqlvN8ac5VjQ/eTjMbuKj8f8JnCyiHwJtAM/Ncbsdi7q/vHxmH8M/E1EbsJ2o1wezid5IvIktvsv2ztO8msgFsAY8yB23OQ0YBPQiN2wrH/PGcavl1JKKT9ze/eRUkqpXtCkoJRSqosmBaWUUl00KSillOqiSUEppVQXTQpK+ZmIpIvItU7HoVRfaFJQyv/SsdV3lQo7mhSU8r+7gVEislJEfud0MEr1hi5eU8rPvBs1vWKMmehwKEr1mrYUlFJKddGkoJRSqosmBaX8rw5brlupsKNJQSk/81YiXSoia3SgWYUbHWhWSinVRVsKSimlumhSUEop1UWTglJKqS6aFJRSSnXRpKCUUqqLJgWllFJdNCkopZTq8v8B5Qf3aMwGQKAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# => Plot risk on two paths\n",
    "plt.plot(np.linspace(0,1,70),loss,label=\"segmented\")\n",
    "plt.plot(np.linspace(0,1,70),loss_linear,label=\"linear\")\n",
    "plt.legend(loc='best')\n",
    "plt.xlabel('t')\n",
    "plt.ylabel('Losses')\n",
    "plt.savefig(\"linear_vs_piecewise_linear.jpg\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 565,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(70,)"
      ]
     },
     "execution_count": 565,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.shape(loss_linear)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}