Navigation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Navigation\n",
    "\n",
    "---\n",
    "\n",
    "In this notebook, you will learn how to use the Unity ML-Agents environment for the first project of the [Deep Reinforcement Learning Nanodegree](https://www.udacity.com/course/deep-reinforcement-learning-nanodegree--nd893).\n",
    "\n",
    "### 1. Start the Environment\n",
    "\n",
    "We begin by importing some necessary packages.  If the code cell below returns an error, please revisit the project instructions to double-check that you have installed [Unity ML-Agents](https://github.com/Unity-Technologies/ml-agents/blob/master/docs/Installation.md) and [NumPy](http://www.numpy.org/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from unityagents import UnityEnvironment\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "import torch\n",
    "import numpy as np\n",
    "from collections import deque\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will start the environment!  **_Before running the code cell below_**, change the `file_name` parameter to match the location of the Unity environment that you downloaded.\n",
    "\n",
    "- **Mac**: `\"path/to/Banana.app\"`\n",
    "- **Windows** (x86): `\"path/to/Banana_Windows_x86/Banana.exe\"`\n",
    "- **Windows** (x86_64): `\"path/to/Banana_Windows_x86_64/Banana.exe\"`\n",
    "- **Linux** (x86): `\"path/to/Banana_Linux/Banana.x86\"`\n",
    "- **Linux** (x86_64): `\"path/to/Banana_Linux/Banana.x86_64\"`\n",
    "- **Linux** (x86, headless): `\"path/to/Banana_Linux_NoVis/Banana.x86\"`\n",
    "- **Linux** (x86_64, headless): `\"path/to/Banana_Linux_NoVis/Banana.x86_64\"`\n",
    "\n",
    "For instance, if you are using a Mac, then you downloaded `Banana.app`.  If this file is in the same folder as the notebook, then the line below should appear as follows:\n",
    "```\n",
    "env = UnityEnvironment(file_name=\"Banana.app\")\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:unityagents:\n",
      "'Academy' started successfully!\n",
      "Unity Academy name: Academy\n",
      "        Number of Brains: 1\n",
      "        Number of External Brains : 1\n",
      "        Lesson number : 0\n",
      "        Reset Parameters :\n",
      "\t\t\n",
      "Unity brain name: BananaBrain\n",
      "        Number of Visual Observations (per agent): 0\n",
      "        Vector Observation space type: continuous\n",
      "        Vector Observation space size (per agent): 37\n",
      "        Number of stacked Vector Observation: 1\n",
      "        Vector Action space type: discrete\n",
      "        Vector Action space size (per agent): 4\n",
      "        Vector Action descriptions: , , , \n"
     ]
    }
   ],
   "source": [
    "env = UnityEnvironment(file_name=\"./Banana_Windows_x86_64/Banana.exe\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Environments contain **_brains_** which are responsible for deciding the actions of their associated agents. Here we check for the first brain available, and set it as the default brain we will be controlling from Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the default brain\n",
    "brain_name = env.brain_names[0]\n",
    "brain = env.brains[brain_name]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Examine the State and Action Spaces\n",
    "\n",
    "The simulation contains a single agent that navigates a large environment.  At each time step, it has four actions at its disposal:\n",
    "- `0` - walk forward \n",
    "- `1` - walk backward\n",
    "- `2` - turn left\n",
    "- `3` - turn right\n",
    "\n",
    "The state space has `37` dimensions and contains the agent's velocity, along with ray-based perception of objects around agent's forward direction.  A reward of `+1` is provided for collecting a yellow banana, and a reward of `-1` is provided for collecting a blue banana. \n",
    "\n",
    "Run the code cell below to print some information about the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"\\n# reset the environment\\nenv_info = env.reset(train_mode=True)[brain_name]\\n\\n# number of agents in the environment\\nprint('Number of agents:', len(env_info.agents))\\n\\n# number of actions\\naction_size = brain.vector_action_space_size\\nprint('Number of actions:', action_size)\\n\\n# examine the state space \\nstate = env_info.vector_observations[0]\\nprint('States look like:', state)\\nstate_size = len(state)\\nprint('States have length:', state_size)\\n\""
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"\"\"\n",
    "# reset the environment\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "\n",
    "# number of agents in the environment\n",
    "print('Number of agents:', len(env_info.agents))\n",
    "\n",
    "# number of actions\n",
    "action_size = brain.vector_action_space_size\n",
    "print('Number of actions:', action_size)\n",
    "\n",
    "# examine the state space \n",
    "state = env_info.vector_observations[0]\n",
    "print('States look like:', state)\n",
    "state_size = len(state)\n",
    "print('States have length:', state_size)\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 100\tAverage Score: 7.74\n",
      "Episode 151\tAverage Score: 13.06\n",
      "Environment solved in 51 episodes!\tAverage Score: 13.06\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEKCAYAAAASByJ7AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzsvXmYI1d97/39SSrtvanVs3fP1mO8YLwwNmNmDGYxBkJY3tz7Akkc7g3BkJuNm5CEJM9NyAL3vjckvgkPWcwSIOEaSEjC5hBs42DPGC/jfRnPTM+Mp3v2Vmu6W2tJKp33j6pTqiqVtm5t0/p9nmeeaUmlqtPV0vme33pICAGGYRiG8fR6AAzDMEx/wILAMAzDAGBBYBiGYQxYEBiGYRgALAgMwzCMAQsCwzAMA4AFgWEYhjFgQWAYhmEAsCAwDMMwBr5eD6AV4vG42LZtW6+HwTAMc0nxxBNPJIQQE42Ou6QEYdu2bTh48GCvh8EwDHNJQUQnmzmOXUYMwzAMABYEhmEYxoAFgWEYhgHAgsAwDMMYsCAwDMMwALogCEQ0SUQPENEhInqBiH7NeD5GRPcS0VHj/7FOj4VhGIapTTcshBKA3xBCXAFgD4BfIqIrAXwcwP1CiF0A7jceMwzDMD2i44IghDgrhHjS+DkF4BCAzQDeBeDLxmFfBvDuTo+FYS41vvPMGSxmC70eBjMgdDWGQETbAFwH4FEA64UQZwFdNACsq/GeO4joIBEdnJ+f79ZQGabnJNIqfuXup/CdZ870eijMgNA1QSCiKIBvAvioEGK52fcJIe4SQuwWQuyemGhYec0wa4blXBEAkCloPR4JMyh0RRCISIEuBl8VQvyz8fR5ItpovL4RwIVujIVhLhXSagkAoBbLPR4JMyh0I8uIAHwBwCEhxJ9bXvo2gA8YP38AwLc6PRaGuZRI53VByJfYQmC6Qzea2+0FcDuA54joaeO53wXwvwB8g4g+CGAWwH/uwlgY5pKBLQSm23RcEIQQ+wFQjZff1OnrM8ylihQEthCYbsGVygzTp7CFwHQbFgSG6VO6YSF89oEZ/PkPDnfs/P3Ar3/jafzrU6d7PYxLAhYEhulTZFBZLXZOEO598Tx+eHjtJvgJIfCdZ87gwEyi10O5JLikdkxjmEHCdBmVOucyWs4Va0f41gBLuSKKmjDvJVMfFgSG6VNMl1EHLYSlXBF+39p1FCTSKgCwIDQJCwLD9Cmmy6hDFoIQAku5IsJ+b0fO3w8k0nofKBaE5li7SwOGucTptIWQLWgolQVyHbRAeo1pIeRZEJqBBYFh+pRMh2MIS0avpKImUOhgnKKXJFK6IGTYQmgKFgSG6VNSHbYQpCAAQK7HDfSEEPjxsQXc9+J5HJhJQCuLFZ/r5UQG2YJ+76TLKGURhNOLuZ62FC9qZRw9n+rZ9evBgsAwfUqnYwhWQcgWe7uCfu70Et7/uUfwC185iJ/5/KN4+NjK00Tf9dkD+JsfHQcALGQqFoIQush88EuP40++d2j1g14h9zx3Fm/9i4dwfjnfszHUggWBYfqUTIcthMVsRRAyam8thIvGWH7j1ssAABeW1RWdp6iVsZQrmivw+ZRuCZQFzFjJmcVcT1fo8ykVWlng2Hy6Z2OoBQsCw/QhWlkgU9BABOSLZXN1206W+8hlJK9//VZ9a3Wr9dIKUjxnk1kAlaAyoFtc5bJASi2Zr/eCrPG7zvVwDLVgQWCYPiRj+MDHwn4AQEFrv9vI5jIq9NZlJCfy9cMBACsXBGkFzC5kIYRAIq1C8eqVdym1hFS+BCF0i2Q5v7JrrBYpCL0UpVqwIDBMHyLdReMRXRDyHWhwZxeE3loIUhCiAQXRgG/FgiAbAabUEhazRSTSKibHwgD0e2o9b69W6DlDfGeTuZ5cvx4sCAzTh8iA8nhUFwS1Aw3u+kkQ5Mo+qHgwElJs7qyVnAcADp1bRr5YxrZ4BIB+T/tBEDJ9bCFwpTLD9CEyTXI8qrtQOtECeylXhOIlFDVhuqh6RUUQvBgJKSt3GVmE7anZRQDA1nHdQkipJVizWU8u9MpCkG6tTE+uXw+2EBimD5Euo3hkZRbCF/efwP6j9VM3l3JFrB8OAlh9UPnFM8v4n/92aMXB77wRQA/4PKsTBIuF8OTJiwCAbeO6heB0GckV+tcem8U9z51d0fUA4ImTSXz+oeNNHy/FV8Yxzi/n8Qffer4vigNZEBimD5Euo7hhIbQSQ8ioJXzqnkP42uOzdY9byhWxcUQXhNW6jP7uwAn87Y+O4/AK0znzpTJCihdEtCpBsKboPjWnWwimy0gtYTGnp6GuGwpgNpmFVhb45D2H8OlV7AnxL0+dxp/+e/Pvt97ruWQW/3hwDl/+8UkcvdD7YjUWBIbpQ6pcRi1YCI+eWECpLBpOqsu5ItYNB0G0uiwjIQT2G/sNNLJKapEraAgqepO9dghCxO9FMqNP/tsNCyFliSFcvXkEc8ksnj21iFS+hOPzGZxZXFmQN1coQy2Vm64XyRU0rBvS/65zySweMu5Zr1N/ARYEhulLzCyjaOtZRvuPLgBAw8DsUq6IkZCCsOJdlYVwPJHB2SW96nalG9HkihpCUhDCq3cZ7Vo/ZD63cTQIxUumy8jv9WDX+iGcupjDg0cq492/wrFLIWg2EJ4plPCKDfr4XjqXwpOzumur14F9gAWBYfqSisuo9RiCnJTrTaqy9fVISEE44FvVZCSv9/rLJvDoieSKfOG5ooagok9HIyGlpRW37TwF/dqXrY8CAEbDChSvB9GAD2m1hOVcEcMhBVvHwyiVBb755ClcuXEY8ah/VWIGNF87kSto2DgSxGhYwT8/eRpFTY+7sCAwDONKulBCwOdBJKAnAjZrIVxI5XH4fAo+D9WdoGTr65GQgrDfuyqX0f6jCUzGQvjp10whW9DwlLHibYV8QUPI2JdhOKQAWFlxmpycLzMsBBmDiQR8ZtrpSMiHqZieeTSbzOLmy+LYOx3HgZnEioLi+RYFIaOWEPbrY7Cmnva6OBBgQWCYviSdL2Eo6EPQp0+Sza6W5Sp33644lvOlmhOcnLx0QVi5hVDSyvjxsQXsm47jpp3j8NDKXC/5kmb+riOrEAR5n6RLRhb2SQtBWkVSEABg33Qc+6bjSKQLeOlc64Hdli2Eoi5+k8YYdq3TrRm2EBimAwgh2tr7ZzXna9TGudZ502oJkYAPAcON0mzH0/1HFzAWVrBnxzi0cu29hO2C0JyF4DbWZ08vIaWWsHc6juGggmsmR2sKwmK2gAupPFIuLSNyFgthtYLgIWDamGTjRvB2KGgXhI0jQfg8BL/Pgxu2xbB3Og7APQbS6O8vg8HNjLdQKqOoCUT8XlOU3nLVegBsITBMR/jQV57AH3z7hbad76uPzuLm//1Ay6Lw6PEFXPn738fZJffslTOLOVz+P76P508vVb2WUUuIBlq3EB49sYCbdo5jLFx/UpWdTiuCUP/8n31gBu/4zP6q5x87kQQAvHZn3Ph/HM/MLVaNd//RBK79o3tx4yfvx/V/fG/VPckVy2aW0agUhGz12LOFEq75wx/g/kPnXceZK+jB6fVDQUT8Xmw06iwiFgthNOyHz+vB1HgYN26LIah4sWk0hB3xCB45nqw65ye+/QJ+4csHa96bVlxGUjxCfh92GOmwb7lyg/G79d5C4EplZs3x8kLGdRW6Uo6cT+HUxRzyxbK5im2Gf3v+HNRSGScXstg4EqoeZyIDtVTG8UQGr9w8YnstlV+ZhTCfUrFlLGxbZW8Zqz7OaSHMp+q3m565kMZL51IolwU8HjKfX0irCClexAzXzIaREMpCH7+c4AHgREJv9fy+GybxtcfncOpiznZP8tYsozoWQjJTwFKuiOPzGbzpiupx6sFpLzwewv/90B5sHtOvEQ34MJvMYilbNM//2Z++HtFAZQrcPBbCfLr6Phw+nzI32nFDxneaEQS570TY78U7r92E9cNBXDM5iqDi4bRThukEaklbcdqiG/JcrZ5zf4NsH/l8zsVVkFZLGAr4EGjBQsgXNailMkZCSsPA7LIjhtCodUVaLUErCyw6zpdWNUSDlUk16PO4jleO4z/v3qI/dqz+844so1pjl5NvrdV03mJpXDM5agaVowEflnNFpNSSeW+u2Dhs+vEBYDTsd00dXcqVUKzTbbaVGILcdyLs9yLg8+J1l00Yjxv/DboBCwKz5iiUyitujubGSgTh3FIeMxfSdd8nn3fbnCajlhAN+uD1EBQvNWUhWCd5OanWug+mhRBWEPJ7G65OZRpswrGClsIlkRZUzkUQQorXnKCd98Rah1BPzGT6bS1/e76ouVpx0YAPiXQBQlQEx8lIyL3L6nKuiGKd+9+KIMj7HPbbnTOhVdaCtAsWBGbNoZbKHbEQWtmH1xqcbDQpOydPoBJUBoCgz9uUhWB1A40a+yjUEyMPAVG/D5EmYggyOJ1wuJbS+aI5TgDmpO5mIViFqkoQChqCxkTu9RCGarTAbmQhWIXFitWKqS0IekGcM1a0lCvW3I9CKwuz7qKZRYgUsohDtCIBL7I93rUO6IIgENEXiegCET1vee4TRHSaiJ42/r290+NgBge1WEamoNU181tBujdaEZn9MwnEIn54qLGF4LbaTeUrK++A4m3KQlh0sRDqXXs4pMDjIYSMtNNynYwoWTnt9LFnVM3mh5eTsdPiWDR890NBxTZWACiXBVSjl5FkuEYL7IqFUEMQChXXkxXrGOsJgjMzq6iVkVZLNYvtrMLXVAzBDCrbBSHk9yHboa1SW6EbFsKXALzV5fk7hRDXGv/u6cI4mAFBrubaZSW06jKSvX32TscxHFJsexdbkZOi02VU1PTeOHISC/g8zVkIlsyhiN8Lb53iNLliByqr1XydamjZW2nBEVxNGa4tSUCp7TIaCSn66j/os032UuysQeha/YxU00Ko4TIqabbzSJoVBDlWiRynrCauut4KBcHpMgorXtdYUrfpuCAIIR4EUJ3LxaxptLLAPz1xqmEefrspaWXzmu0QBNnioZXzHb2QxnxKxb7p8bqN2ipBZfvkKVfjpstI8TS1H4LVZSS7htYSI6sghA1BcItlSGrHEIquFoKby0jGBpz3RIpHqBlBaMJCWI3LSI7VOm4ANmvzRCKDR44v2Mbu81DNe53MFPD95/X22jJwHHZxGcn7ny9q+NyDx/EX9x3FXQ8e68jmSLXoZQzhl4noWcOl5JIYp0NEdxDRQSI6OD8/383xMavg4MtJfOwfnzHz1LuF1dfbDkGQLR6A5puXyY6fe6fjdQVBns/pKpDHDwWlheBtalKwCoL8v3YdQsE8LmSsVmsFlrWyMCe+KkHIl+yCIK0Nh4AtWwRoNYIgz1trrLWCypEmLAS3YLb8uVQWpkvtrx6YwW984xnzeoDeTrvWvf7GwTl85B+exGK2YAkqV7uM5H348fEFfPKeQ7jzviP41D0v4d9fcK+56AS9EoS/BrATwLUAzgL4s1oHCiHuEkLsFkLsnpiY6Nb4mFUiV0KZGpWyncK6km6HILhNDo3YP5PA9njErAdoGENw3COZ8y6rbIOKp6leRvJ8cmIbrnPtRLqACdnnx5icZI68E6tP3ZmPn1E116ByLZcR4CIIxiQZ9DcjCPqxtVI0c8VKCwwrQy24jJZr/M3lYiNb0MxjZDO9dcPBmg35LhptuOdTak2XUcRSLS6Pv+/XX4fhoA8HVthSfCX0RBCEEOeFEJoQogzgcwBu7MU4mM4hvyjNFlS1C+v12pF62qogFLUyHjm+gL3T4wBqB0et53O6P+QqXE7YrVgIQwE9VRVAzb2JhRBIpFVTcEINXEZ2QahYCGpJQ0Erm5YMADOga13BFzU9yF9LEOQkKmsYAL1LqbPmQb9mIwvBvXhQuoz8Po9r0FmOC6j9N5duo4JWRrqg94mScZcNRkW0ex2D/lwiXUC2UAIRqsYQ8leyjOTx45EAbto5jv0rbLq3EnoiCES00fLwPQCer3Usc2kiV4graWG8GqzZIL2wEJ6eW0S2oGGf0RunKQuh4LQQ9ElX5uw3ayHI1s6SWtdOqyWopbLZWluu8GtNstLKCyoeW1BZxhWsLqOgi4VQqY/wuY5LfkasE/lwSEHBZcXdlIXgEkOI+CvXJqKq1+VrgHtQGagElgulMoTQhVzesw3GznNu97siCLqFEDZ2hrMS9nuRLWq2mNVwSMG+XRM4vZjDy13a/7njrSuI6G4AtwCIE9EpAH8A4BYiuhaAAPAygA93ehxMd5Ff3O5bCJasjxpBvlaQX86xJjdt2X80AQ8BN+2wC4IQwjYJlMuiEkNwWggpfdKV7SBasRBGwxVBGK0hCNLtMx4xLARjAq2VuZMyJv5t4xEcT2TM30VaFFaXUcDnARGgumTfjIRbiyHI91onePl5cgsqy5oAt6CytGJquYsAXdicmVmuFoIxhrRaMscu96ZuLAglhAPV027Y74NmpN9aLT25sJBuyE7TcUEQQrzf5ekvdPq6TG/J98hCUDtkIUzFwq4uDCf7ZxK4esuobfIrlQWyBbuvPV0oQSZgubmMRkIK/IYLpZUYwojDQpAtsK1iZFoghstIBjhrZe5IC2HbeAQvnUvp7R+CClKqfj+sFgIRIaR4bRaCM9htXf0HlUqVtDPtVL5XTrZA5fOUK2hVv5fpenJxCcl7X08Q3PZztsUQShWXEaALpbzmhhH3Cmzrc6aF4OLSks/lCpotI2vbeBibR0M4cDSB2/dsrTn2dsGVykxHkF/yXsYQ2iEIchU/GQs3jEmk8kU8PbeIfUb8AKjdl0daL0HFU7UyX8iopjsHaM1CcAqCWwvsBdMlZXcZ1RIE+f6tcb3vj6xWli4jawwBQENBcN6TXLG+IFiRf99SWVRVD+dcXE8SxetBwOepKwjyutb0UbegshSGjFoyP+fNWAgL6QKyNdJipUsrW9RsDfiIdCvh4WOJrqRwsyAwHaFXMQSby6hNFgKR3gnTra2BlcdOJKGVBfZNV7LhagqC8XjjSMjVZSTjB0DFQtDKAh/80uN46Kh7+rWbIMjn//A7L+DzDx0HAMwbLiMZtA75q11G//DISXzCaCEuJ365Wf2CkQUj/fgRhwtEX/VXC7Mcz6ijNbfMDAv5qwVBZtxIrJ8nZ8wj7yIsVoaCvoaC4MzMcnMZyf/TqsVCaMFl5LxfgOVvYNmzQbLX2OzoOZc26e2GBYHpCNLF0W0LQa7eFG/9LSSbZSlXxHBQwWjIj6ImXPsOSY6c15vZvXLzsPlcLUGQ1sYGI13RuvqzZgAB+gSnljScW87j/pcumHUObmO1TiTS7XBuKY9/eOQk7nlOL46SK3wZowgr1S6je547i+88cwaAxUIwBEG+P+USVNbH67FVPVeCyn7Xe+IWQ9gxEYHXQ3jm1KLt3NbPk1NI8y7nsfL7P3kV/uveba6vSZyZWTZBKFWCyoD+++eKlbRT5/GAHteQ92nesBDquYyyhsvIJgg7x/GuazfB7+38dM2CwHSE3lkIxhd0KNg2QWimNxCg788bi/jNfj1AExbCqD6RWFfn82nVXL0DsnVFGScXMubrTmTra2eWEQDc++J5FDWB2aS+KU0irWIsrMBnTDA+rwd+n8c2wc4ms0hmCygZvXwAYJt0GRnXl89XuYz8XuQLTbiMsrUFYSio4NrJUeyfWaj6PSVOV5u0SmpZCO+8ZhNetWXU9TVJdQyhZMZypMtItbqMjPGEFa9rQz6ruCRSKrKqu8tI1iVkCtUWwng0gL9433W4ctNw1fvaDQsC0xF6lWUkV28TQ4G21CHIpmzNCMJcMmvbqxdoxmWkC0Il5qIhlS/ZYwjGBHLMaKft7CdkPZ+by+h70jIwXBYLabtLCoBtG82iVsaZxRyEAJLZAtJqCUHFg4loAEQVl5OzxYbELYYQUrzmxCrHJYP08ncP+OzT0d7pOJ47tWjLFqtnIbgJS6s4W2Av54qmODuDytJlFFQ88HjItRBQPo5H/VjIqMgW3V1GzqDySLi+a6tTsCAwHcGc4HpkIUwMBZrKCmqETOU0/d51Ulln3QQh7L4vgTWGAAAZ437JyX7cYSEAeo8koLp9hPV8NkEwrn3qYs4sVptL5nSXlEMQIkbHU0Df2lN6sBIpXRCiAd2iiIX9FQshrxdZhR0TcNAhCIvZ2rENQG9IF/B5bDuxAcDNu+IoC72Vg8RqITgL6Sr1DCuf1qyZWXKMUpzd0k6dO705Px/yd9wxEUW+WEYiVajRWkN/7mK2aG5y1AtYEJiOIH3IvbIQ1g0FkG1DC2xZ7NXIQihpZZxezFUJQtTvc22BvZQrwuchc7KRq3NnURpQcYEcOZ+yHeM8H2AXhFHLz7deoW/kfnIhUxWjAGDbJGc2WSmCWsioRr8irzkumaWUUkv67+eYyIOK15Ym63SBSJeaKQgF9/5D106OIuL3Yv9MJYiuFsuW9hgOl1GDoHIzWDOzZOtr+bdwrUMoVArh3AoB5eOdE1FzjM69EIBKP6lzxl7TLAjMmkJOLr3KMlo3VDvroxWajSGcWcxDK4sqQajnSpDbVwKV+5VwpIQCFQthxuIycu5dIFem1sK0sN8LnzFZv/81UwD0yT6RLtjOL4+VWUMnLVWxibSqWwhGnGA86jcL2zJqjYwZxb6hj1MQnC2wa21qo3g9eM2OcRywxBHUkmYGw1sNKjfDaKiysZAc38SQXRDMLKN8yTb2+oJQKSoL+V1cRsY5zizlzXP1AhYEpiOYhWltaN07l8w2vQG51WUENCcIS7kizhorMyuyjUCtPYoLpTKOzeuTtFxVTzoEAajObQd0/7nc4B6ouIxklbKbhZBIF+AhPQd/Oa+f7+RCBlkjECmvJZGFViMhBfum4xgK+HD0fNq26pWELbumzSWzkIv+isvIZ47LGlSOBt0FIecIKg87Jjjr5Jmz7IPsZO90HCcSGZy6qN/ffLGMsYh+LucOY24Fbq1i/TtX/P/6vVJLZZTLwux+mzFjCM1bCEB1p1MACBsW2NlFthCYNYg035vp49+Id3xmP76w/3hTx1pdRkBzgvCp7x3Cz37+0arnZevrkZCCoYAP5HD9fOPgHG6780GcW8qbgjA17i4IbtknwzYLwXAZZapdRtZg6+Ub9EyTRFpFuSzwjs/sx5//4IirIADAllgYb7p8HbwewtR4GE/NXTTOb7cQRkKKmU46m8xiWzwCv9ejWwiWFtcTQwFcWFYhhJ5O6Uw5BfS0U2cvI+e4rPfEOqk62bMjBgB4clZPP1VLGsbCdjebpB0WwoirIMgYgr0YLqWWbM30RiO68FutNzdBcHMZ+b0eeD2Es2whMGsR6UNerYWglfVVutWvXQ+1pMHrIYxF6u8pbOXIhRROJDJV2yRaJ1mPhzActE/sR8+nUCoLPHR0HrPJLPxej1mgZKXWytFqIWQtFkLE77X51K2T5XVTetrkfKqA86k8UvkSHjh8wTy/NeUVAL7y8zfik++5GoDefkMGpp0Wwqu3juF4IoPzy7q4bY2FEY/6MS9dRsbEPzkWQq6oYSFTQEatIQj+6iyjRoIQqtGBVGb4LJvHlisuI4c7MudS4NYq1pRYUxAsLiNrTMzpMtoyFkZBK+OCZd/p5VwRfp/HTC/Wx1d9z4gIYcXLgsCsTdplIchJ2tmHv97x1hYFzaSezi5kURbA6UW728gtf946sZ80ROrATAJzySy2jIXMbB4rbi2wnYKQscQQnAFfq4Vw/dSYeZz09R+bz+DI+RSGgr6q64+EFHOCnIqFIQutnYKwVzZRO5rA7IKeLRUfCiCRNiZ+wzUkLaDZZNYmFFZCihcFw73ibH1tHZfpMqoRVAYqbatlzYNa0jAcVEDk4jIquqevtsJIuLbLqKiVbYuGTEEGlfXryfiRdfEi/86K14OxsH2HOifhgLempdctWBCYjiALk1ZrIVQEoTqzxg21VIbfIgiNLIS0WjJbMTitkEaCII/fP7OAk8mMa/zA7X3y3CMhxex8mbNkGTkna6uFcP1WXRAW0qptvA8cvtBwErGOzyk6V2wYxnjEj+89dxYptYTJWNjMKEpZgsdy0ptLZnVXUo0YAqD/7Z2tr93uSa1NbeS5PFRpn5EvlhFUPLY0WYmsCajV3roZrJ8bZ1C5UCrbXEbpvD2GsLWOIAAVYakpCBbLwRlz6RYsCExHaJeFoGr2/PyGxxftFkKjFthzli9vK4JQLgucSuYwMaQHWV84s1yVYSSxtsCW75V+dTl5ypx6XRDs/n1r0HJrLAyvh5BIFzCXzMLrIcQifuSLjXPXreMbj9iv4fEQXjsdxwOHL5jHxqN+nFvKo1AqmzuObRnTz3FyoY6F4CiyAlBVaFUVQ6gxSRIRIgEf0qpeG6CW9Ak4ZCmkk+RrZCu1QsTvNVtgOy2EglZG0VigBHwepFS7y2jTaAgeqhYEmf47bvxdnbulSaRQWDc56jYsCEzbKWplMxNjtWmnclOS+bTa1K5RBU23EPw+D0KKt6GFYP3yzrUgCOdTeRS0Mv7f3VsAAEKgriDIFthApfX1aFiB10MI+CpBWLcqYukCmYqF4TEEIGFYCJtGg7bNeOohxzcU8LkGcfdNj5supanxMMajAdN6khN/UPFiw3CwriDI1X6uqNV0gVhbYOeL7nsYSIYMQShqAmWh34+IJStKkqvRSbQVrC2wl3JFBBWP+TsWS5Wg8njEb2YZSQH0+zzYOBKyfY5asxD053tlHQAsCEwHkCLg89CqC9PMdgGlMlJN7M+sljQEfLXTAJ3IL+/EUMDsFSSRLgPrHsXyuVnDf79nxzh2GBuXuGUYAZUCMTkWabXI80YCPmQLJZS0MpLZgq1KGahYCPL84xZBmIqFmxaEzWP6CtbpLpLIOAIgLYTKcdZ6g6lYGEfOp1AWcHUZydV+vliuKQhW10ytOgTrtdP5klljolsI1S6jXB1LoxWsgjAS0kXbQ/YYQizqR7agIaPaM6SmYmHb56gVQQhZdnXrFSwITNuRq93RsAK1VF7VfrDWIF4i1TiOUCiVza6QTkH47rNncMdXDuKOrxzE3z9yEoBuIQwHfbh684jZ/E2ylNNbXw9ZNleRrh8zzTQWxr5dcfNnN5zxDOcYqeIwAAAgAElEQVQkGVL0/XST2QKEACYcLiOrhQDAcFMVzN5Je3c1JwiK14NNo6Eql5Rky1gY2+MRxKMBhP0+23HWBnaTsTAOG1XTtQrTAH1hUEsQZAHdYrZoC8y6EQ36kCmUzMy1gM9j9l4SQuDPfnAYh8+ldNdTjVhEKwyHFDxyPIkHjyTMQjXF67FlGcnU14JWrhIE6+fIWoMxYW5I5O4ykumooz3qYwR0Ycc0ZvDIG10nR0IKEukC1FLtwqNGWAVhIVPAjok6B0MPKgeMyWUsouBithJ7+NKBl3Ho7DJCfi9+fGwB779hEicXspgaD2MqFsZjJ5K2XbgOnU1hcixstmbYOh5GURM4Np/BrFG8tWk0hPfeMIkLy6ot19yKTAWVbZDNLqHGZBoJ6O6Pc0bK4XpH6upISMF7rtuM267aAEBfab5wZhnJTAGTMX1Hrdv3bMUtr2hwcwD87J6tVd1JrXz4dTtwblkfh7XjajRQmaSmYmHz7zJURxByRc3cz2A0bBchKW4vnVtGvlTfQogGfLbdyQKKF2G/F6l8CRdSKj7zwxlcWFZtNQGr4R1Xb8Q3nzwFAPiJq/Xt3/1eD9RSxUKwxmCsY58aD5tNBAM+fYxSDG95xQQOn0vVFGQ59l5aCCwITNupWAh+ABmodSpRG1HQKm6BZiwEGVQG9AZxL55Zrrw/reKNV6zHbVetxy//36fw7OklzCWzuHzjECZjYaTVEi5mi4hF/ChpZTxyfAE/ec0m8/17d+or8QMzCcN/H4Li9eCqTSP4m9tfXXNMzg1o5P8ywyjk11fA0uqQ+w5IPB7Cne+91nwcj/qRNCZaObH+8btf2fDeAMBHXr+z7uvvu3HK/Hnc5jKyTnoh8+dahWmA7tOX1dVjDkG4atMIRkIKfvjSBQiBuq6eaMCHs0t5c3UuLQRZMwHoW5euHw7UXH23wodetwMfet0O23N+n24hyLYVYzZBqFg3k2YWVs4sjpQT/FWbRvCX77+u5nUj7DJi1iJyJSd9581s/1gLawyimdRTVSvDb7gNJqIBm4gk0gWMR/x47c44iIAHj8zj1MUcpmKRqpTBZ04tIq2WcPMui199PIzJWAj7DUGo5SJyYm1tDFSK0OTzYaPVg6wrmIyFXM5Swerbb3YMK6GWy8h6TTeXUdDiMkqkVcQigaqsGa+H8Nqd43jgJT2rqZGFIAO48vxhI4YgYzmnF3OYuZBeVduKekiXkRlDsAic9ZrWz1GrNQXhPrAQWBCYtmO3ECpVyyvB6jKabyL1VC1qpoUQj/qN9gIa8kUNabWEiaEAYhE/rto0jH956jQKWhlTsbAZsJUBwf1HF0AE3LRj3Hb+fdMTeOTYAl5OZLC1RhDZiblfbg1BkC6juWTW9N/Xw7py3xqL1DlydYyF/eZEbncZVa7p5n4y004NQajlItk7rW8NCdTvPxQNyqByZQOcsNGd1Zoltpwv1Y1FrAbFR7bWFTHL72R1U0mxPLmQaVkQQpxlxKxFrEFlYHXFaTLtFKhsDl8PmXYKVFbSC5lCVRfRvdNxc0U+FQtjcqxScAXobqFXbhqxuQYAYN90HCnDtVSrEM1JlcvIiCHIiV/PmCkZVkd968D6OwwHfR3dSEWmuAJ2l1E86jdX9LUqlQHdQphPF8xgqpN9lqymRhZCumCJIRguo0yhhLlkFhtHgtg8Gmp4ntWgeD0oaO4xBKuYjYb1vldzVguhyb8Ru4yYNYmsUjZdRm2wEPRirNZiCFIQEinVbH0hn9vnSLEM+b2YGAqYLRmenL1oZg9ZuWnnOGQhbLPuGjmZmhZC0WEhGDn1zbqh5O9QK821nciJL2KxWojIHGc9l1GuoGHBpfJasnU8jC1jIdt73IgGfBACZoKAdBnli2W8vJCxpd62I6jsht/r0SuVHVlGgF2EiAiTsfCKXEb9EFRmQWDajrQIRo3JZDUWggwqbxgOmpO6WtInmmSmUJXSWtDKZh2CrAxNpFUzliAnpxu2xeD36R0mZeOxqVgYx+cz+OFLF1AqC5toSKS7SR7fDDIVUvYryqoaPFRJJw0ZGTNnXDbYcUOuuDsZP7BeKxqo3gRHWkeuLiMzy6iMRFqtqoqWEFFTE7msdZDV6tJCAIAj59PYOl5Jve2UhSCDyqosTIu6WwiA/nd5eSFrtlTnGAKz5rjvxfO4/o/vrWoX4Ibc7HwlFkJRK2PPp+7Hd589A6BiIWwcCSJhVCu/9f88hFf/yX24/o/vxZ33HbW93x5DMCyEtGpaF/KLHFS8uHFbDJNjeqYQAGwbj+DgyYv41bufQlDx4NVG3yAnN++aAFHzE7LHQ8YeATLLSEPY7zPTW8NGd9CyAKbGG8cEYhE/FC9hWxPHrpYNw0HXvPgdExEEFY9rIzn53EJaTwWtVQgH6PcScBcWiXRLSZehjCEAegrvVCyMvTvH4aH651kNMqhcbGAhAMC2eAQnEhl86p6X4KHmJ3h5n2tZVN2A006ZpjhyIYVkpoALyyq2xet/bKpiCC20r0jnSzi3nMfxeT24KwVh02gIh84u4/B5vVX1e3dP4r5D56uqi9VSxWUkV9LWTqnWL9un3nO1udEMAPz3W3fh2skRCADT66I13Ri/eMtO7JuOV+XW10MGjgF960drtao1iNyMyCheD776C3uwa5173UM7+fW3XGZu2mPlw6/bgduuWu/aSM7jIQQVD+aMTW3qTXBve+UGfO7nduO6ydGax0hBmLdZCPZCufFoAHd/aA8uWz/U3C/WIoqXbK0rwn4fgorHqH2wi+IH923H5rEQhBCYHAs3nfn0ul0T+Lv/cgOu2NiZ36EZWBCYpsgYgdBm9heopJ3qE2Yr7SukmMhzqBZByBQ03PfieQDAr755F545tWhrXyCEsAWVg4oX0YDPsCyqe/g4ffBbxsK4/aZtDcc4HFRsbR6aIWTpvZNRNYcgVGepNOLG7bGWrr9SNo6EsHGkOtA9Hg1UtdiwElS8OHVRd5nUyjICdPG49cr1dcdQz0IAKvfsNY6MsHaieD1IFUvmAkXxEqIBBfmiWjXhTwwFcPuerS1fw+f14A2Xr2vLeFcKu4yYppDth5sVBKv53oqFkHMIgcwy2mT4+b/19BnsiEeweTRkph5KipqAEPZ++HFjD2C3fQa6SVjxWQrTNNsmKXJy8/s8ZjHTpU5I8ZoZW6t1gcjAtWy0F/B5zKI+oLqQrxMEfJWgsof0yVt+vjsVt+gFLAhMU6RasBBk10m5cmrFQsg7LIRKDEFfpR69kDazf8JGha9EmvMBSz+bcaM4rV5wsxuELS6jbKFk20ZRuj8mx0JVwdtLlZDiNWsMVisIcuKVcSBrUDka8Jkbz3QSszDNYoHK7LFOFcP1go4LAhF9kYguENHzludiRHQvER01/neP3jF9Qysuo5zRElgWCbViIThdRQVN3xLTunKW7hqnhaAa7/U7LISFjOraVrqbWDexzzp2CJOTWzeyhrqFdZIcr+MyaoaKy6gAn4fg83rMVflkLLyqDXGaxVqpLJsnWluCrxW6YSF8CcBbHc99HMD9QohdAO43HjN9TLpFQQj4vOZKvaUYgpGhZLUQ/F6P6e7xkN5yGoBZnCSx9rqRxKMBi8uohxaCpV1zrqDZ8vqlhdAN10e3MDeeDytmFtdKsW6jKSffyg5ujQv52oEuCDJGJa0TxUxdXit0XBCEEA8CSDqefheALxs/fxnAuzs9DmZ1yBiCc2/gfFHDvz512lYPoBpdJ+XEvJoYQsHYElO6e66ZHDXT+MIBn81CkO4lv0MQLmYLuJgt9oGFoN/DTMGZZVRZ7a4VpHXYjnse8HmheMl2XnnPuiWifh+Z3U7l5zoa8K6p+AHQuxjCeiHEWQAw/q8ZWieiO4joIBEdnJ+f79oAGTvSQlh0bEl5z3Nn8dGvP40fH18wn5Mbnng8ZLYNbpaqGILhsw0qXlwzOYp3X7vZPDas2HfNqlgIljYLQ4GaG8t3E6vLyLmp/GQsjA3DQdywbe14TuVEWS/DqBWke0b+bUfDCnbEI3hNl7Kt/BaXkRSnq7eM4po66bKXIn2fdiqEuAvAXQCwe/fule+0wqyKWi6jl41+QPuPJvBaoz20dSvDgOJZpYUgTJ/tt35pr+1YOcmWywIeD7lbCJZAcrsmp5UQ9vvMHkaZQsnW8iEW8eOR331Tr4bWEaRrp15qaitEgz5czBbNvS4CPi9++LFb2nLuZrDFEIzP1wf3bccH923v2hi6Qa8shPNEtBEAjP8v9GgcTJPUSju1NoOT5Iqa7Yu7EgtBdVgIbsjUQ9kaQ7bZtsUQLMHonlsIRQ1aWTTcQ3gtIH+/iTbdcxlzCbRhR7SVoPiqs4zWIr36zb4N4APGzx8A8K0ejYNpgnJZmH14nIIg2w8/e3rJ3Cs4b9kjN6h4zMm9GWRMoGIhaKaF4ET6kbOO9ziDym4/d5uQ3wshgEWjQVutfXXXCsE2u4xk6mmn2ls3wgwqW7KM1iLdSDu9G8CPAbyCiE4R0QcB/C8AtxLRUQC3Go+ZPsWayeMmCDsnIhACePiYbiXkixUfecDnaam5ndw7wZZlVMtCkPsMqPaaBWfaqflzD4u+5ApXttEIu3QJXUvIv3+7RLgSQ+jNZOw34gaZQokthNUghHi/EGKjEEIRQmwRQnxBCLEghHiTEGKX8b8zC4npI2T8IKh4bFlG2UIJ8ykVP3nNJkT8Xuw33EY5m4Xgbam5XVUMoZ7LSFoIxZLxHukyqqy+owEf/D69CVukh6tyOUHK4qrwGncZtTuGEOlxzr/8DKbzJTPtdC2ydqWOaRuyKG3zaAgptQStrMf255J6r5odE1Hs2TFuxhFyBc384rZuIVRXKtcy0eUkm1HtImIVECLCRDSAeDTQlQKmWlQsBF0QrBvOrEXanWUkXUa9shBkLUVGLZnWwlqEBWFA+dbTp/Hxbz7b1LEpI6C8ydiVSloJ5qbwsTD27Yrj5YUs5pJZ5ItlUxBathAKUhCkhSBqWghyknXGHZyTRjzq76m7CKhYM9JlFGrDZvD9TKiNdQhA76uCpSCk1bXtMlrbn0qmJj86Mo9vPX0Gf/LuV8LXIEgmXUZyd6ulXBFjEb8pCFOxsDnhPXh0HgWtbHMZWVtMN8KaMSSEMPK+GwWVpcvIEARH4PGjb74M6PGirspltMaDym995UbkS2XzM7NaIj2PIRgWQqF2ksNagAVhQMkYrp8zi/mGWzFaXUZAJbA8l8xiKODDaFjBaFjB+uEA7j+kZxDLHvEBn2dFFkJZwMjq0GpOAs4sIxlUDnjtk22vWwoDFpdRajAEYcNIEB95/c62na/XFoK0CrRybYt1LbB2fzOmLnLVL1f59XC6jJYsLiPZXIyIsHc6bgaWrRZCKzGEXNFaeaw1CCobWUYFRx1Cj1IT61FtIfBarBX6JYYAgAUBAIhoHxH9V+PnCSJaWyV6A4YsNGtGENI1LISTxgbnkn3T8coq3RJUbsVCsFY154vlpoLK0mVkpp32oUkvg8gyhtDLjKdLkV5nGSmWQLLfu3b/dk19c4joDwD8NoDfMZ5SAPxDpwbFdJ5WLATpMrJaCOWywNzFnM3dZN1FzGYhrKB1BWBYCHXrEKoL0xQv9eWeAmHFvutXvU3lmWp67TJSfGwhWHkPgHcCyACAEOIMgN5t/MmsGikIc824jIzMCrlH8VKuiAspFYVS2dahc/1wEJet1/f5DVkthJZaV5RtPxfrZBkpXg/8Xo8thtCP1gFgdRnJSmV2GbVCr11G1s8Vp50CBaH3NxYAQERrp3H7gOJ0GWXUEh46Oo8Hj8zj6PlU1bFyP2K/z4OlXNGWcmpFWglmpbKi9zKytsd28nIig5Kx21muoGHIWA1KC6FeP319J7JKYVqgTwu+/D4PFC+hoOntk9dSD/1uYGYZ9Tio7Px5rdHsb/YNIvpbAKNE9CEA9wH4XOeGxXQSa28iObHfee8R3P6Fx/BzX3wM7/jMftNNBOhiIb+QoyEFS9kijl7QRWObox/9m6/QN0yXO5zJFV0tK+FipoBb7/wRvvfcWQB6DGHE2BIxX6wfVAbsLbDVYrlnK8hmkFbTWs8w6gQT0QAUL2F9j+pJOKhsQQjxaQD/BOCbAF4B4PeFEJ/p5MCYziF7E60bCmApV8RStoj/ODKPG7aN4U/e/UqopTIePVHZ3yCtlkwf7khIwVKuiIePLWDDcBCTjh2r9k7H8fDH34hd63WPormvco3A8kKmgKImcHpRr3rOFTWMGoIgs5vqTfIhyzaa/d6JUrqJ2F3UOuPRAB76rTeaC45uYw8q9+9nbLU0/M2IyEtE9wkh7hVC/KYQ4mNCiHu7MTimM8j4wVWbhgEAj72cxMyFNG69cj3+06u3IODzYP/RiiCk8iVzG8ORkIKL2QIenklg73TctR2EDD4Dlcm8VuqpdZ8FIQTyRQ1jYb95XaD+FzAS8JkC1+8WQjjAFsJq2DAS7FnCgC2GMMi9jIQQGoAsEY10YTxMF5DuoCs26oLwtcdmAQD7picQVLy4cXvMtr9BpmC3EJ45tYiL2SL27RpveK1GFoK5z0K2iIJWRlnA3CJTVjjXW/WHLC6j/rcQWBAuVdhlZCcP4Dki+gIR/aX818mBMZ1DrrylIDxw+ALGI35cvkF38+ydjuPw+RQuLOcB6JO2VRBkJpA1zbQWrVgI+YJ+3ioLoc4XMGLZV1ktaT3bQKUZ2GV06WL9DCprOMuo2U/m94x/zBpATsIbRoKIRfxIZgrYOx03zfF9xkR/4FgC77luix5DMFxGw8bq/RXrh7BuKNjwWg0tBIsgyBqESgxBtxDqZRmF/N5Lx2XEFsIli/Uz2M+fsdXSlCAIIb5MRH4AlxlPHRZCNN+xjOkrpMsoGvBhMhZGMlMwRQAArtw4jLGwgv1HFyqCYLEQgOasA6AJC8GY9JdyRbOATV6jGQshrNiDylK4+hEpBFyUdunhZ5dRBSK6BcBRAJ8F8FcAjhDR6zo4LqaDyIk2GvCZrSf27qpM8B4P4bXTcRyYSaColZEvlqsE4eZdzQlCIwvBujWntBCky0i22W4YVFYvFQtBv4cRdhldcii+wWhd0ewn888AvEUIcRgAiOgyAHcDeHWnBsZ0jrTFQnjHqzYiGvCZfYok+6bj+N6zZ/HsqUXzWAC4aec43nLleuzZ0TigbH1frRbYUpysglAJKjeZdmq8L5kt4JpQ/+Y+sIVw6TIoQeVmBUGRYgAAQogjRKR0aExMh5Er6kjAh9uu2oDbrtpQdYx0IX3/+XMAKhP7FRuHcdfP7W76WnLHLNnDx0la1YUilS+ZeyNHgz74PGTGEOoGlf1eFDWBpVwR8ynV1myv3whxDOGSxechEAFCsCAAwEEi+gKAvzce/wyAJzozJKbTpNQSAj5P3Q/2ZCyMreNhfP8FQxBW6JuPRfwgAuaNHj5O5PaXAHAhpWc1hRQvAj5PUzEEufPYEaPdxtR4/3ZVka4iWfXNXDoQERSvx2ilsnazjJqVul8E8AKAXwXwawBeBPCRTg2K6SzWNNJ67J2Om/smN3O8Gz6vB2Nhv7kPgBM56QPAOSPNNeT3Iqh4zdfq9jIyVtsvnTMEoY8tBNNl1Kf9lpj6yFhWP8epVkuzv5kPwF8IIf4fIcR7APwlAP5UX6JkLGmk9bjZkkm0mlVtPOpv6DICgPNLdguhmaCyKQhnlwH0uyBIC4G/Opci0jJYy0HlZgXhfgDWqGMIeoM75hLEmkZaj5t2jkN2phhaRTpnPBow2z47yaiaeW5pIQQUD4KKF+lCE2mnxiT70rkUogEfxsL9G9qqBJXZZXQpIj+HazmG0OxvFhRCpOUD4+f+XYoNMOWywFJWb1inld1bTqfypaZW/KNhP67erGftrNRlBOiNyWq5jNJqycxwOrdsbB5jtNmWHbPrmehy57Ej51Lmdp79ihlUZpfRJYl0XbIgABkiul4+IKLdAHKdGdJg85Y7f4SvPnpyxe//9W88jWv+6Ae45o9+gA//vXvcP1MomXsONOLmXXEQrdZC8JubyztJ5SuCIF1GQcVr2xmrflBZPy6lljDl6Lzab4wa6bQjfWzFMLXxD4AgNPst/yiAfySiM9A3ydkE4L0dG9WAopUFjpxP45DhD18JT88t4lVbRuD3evD03KLrMel8CTvizf3pP/L6nbhx+ziGgiufxOLRADIFDbmCVpWDn1FL2DymT+TzaRWKV8/msFoF9WMIld+jn+MHAHDDthjuuv3V2L11rNdDYVaAaSEMavtrIrqBiDYIIR4HcDmArwMoAfg+gBNdGN9AIXf+WsqVGhzpjlYWOHUxh33Tcbzh8nVIpFXznFbSTQaVAWAoqOD1l02saDySiai+qYnTbVTSysgVNYxHAvD7PNDKAkFfZS9miVI3hlA5rp9TTgG9AvwtV23oa7cWUxtZrTzIaad/C0BGA28C8LvQ21dcBHBXB8c1kOQsbRxWwtmlHEplgalY2NzrWKaNWkmrzbuM2sG4UZzmFISMpRBNVicH5dabTVsIFkHocwuBubTxG3t4r2VBbyQIXiFE0vj5vQDuEkJ8UwjxPwBMd3Zog0dmlYIwu6BvhzkVC5uTo9wiUyJ7E3WzOCpuWgj2TCOZRRQNeE1BkDn6NguhzorM+nuwIDCdRPHWL+ZcCzSaFbxE5BNClAC8CcAdLby3IUT0MoAUAA1ASQjRfE+ENYh07yyvVBCMyX8yFjazgk4uZGzHWDuddou4sQ+usxYhbTbZU8yAa0UQKgG8eiuygM9jpsY6+zExTDvxN6juXws0mhXuBvAjIkpAzyp6CACIaBrAUpvG8AYhRKLxYWuf1bqMZpNZ+DyEjSNBeD2EoYAPcw4LwWxs18U20eMRd5eRLEqzuYwUWQ1quI4aBPCICGHFi9Gwf81/WZneohguo7VM3VlBCPFJIrofwEYAPxBCZobDA+BXOj24QcPqMhJCtOyrnE1msWUsBJ/xoZ2MhU2r4dh8GhczBVMIumkhBBUvhoI+JNIFCCHw3WfP4tYr1yMtYwgWl1HQxUJoRDjgw2Sfp5wylz6Kl9b8oqOZPZUfEUL8ixAiY3nuiBDiyTZcXwD4ARE9QUR3uB1ARHcQ0UEiOjg/P9+GS/YvOcNlpJWFKQ6tMJvMmsFkQPepS0H4/W89jw995SCWc913GQF6HGE+reKxE0n8yt1P4XvPnrW5jORObCEzqKz/X6+PkeTqzSNNt+NmmJVyxcZhXGlsO7tW6XUN/V4hxBkiWgfgXiJ6SQjxoPUAIcRdMDKadu/e7V56u0awdv5cyhVbnrRnk1n8xNUbzcdT42H88PAFZAslPP7yRRRKZTx2YgFAd11GQKWf0f4Z3Tt4ciFj1h9YXUZuMYRGfPG/3NCJITOMjY+++bLGB13i9NT+EUKcMf6/AOBfANzYy/H0mmzRIgjZ1uIIS7kiFrNFW6bNVCyMQqmM7z17FoWSvmOZ2c66BxZCIl2oCEIyW3EZ+X1VLiNpIax1E51h+omefduIKEJEQ/JnAG8B8HyvxtMP5CxFZIs592ZwtZDBY6cgAMDdj81C8RK2jYfx/Gm9CroXgnD6Yg7PGNXTs8ms6TKK1IshrPEgHsP0E738tq0HsJ+IngHwGIDvCSG+38Px9Byry6jV1NM5S8qpRArCk7OLuH5qDG+8fL35WrddRuNRP3JFDWUB7JiIYC6ZRVotIqR44fN6qlxGbCEwTPfp2bdNCHFcCHGN8e8qIcQnezWWfiFXtMcQWuGktBDGK4KwaTQEj5GodPOuOG7eZdnfoMstmGVxWkjx4l3XbEYiXcCFlGoWlsmGbyG/kXbaQgyBYZj2wN+2PiJjbG0JtC4Is8ksxsIKhi1N6Pw+DzaO6IHbvdNx3Lg9BsVLCPu98Hq6W34vBeE1O2LYuU7vOXTo7LLZRbWmhcAuI4bpGr3OMmIs5Aoa4tEAzi3nGwrCH3/3Rdz74nnz8XxKxa710arjto6HsZwv4lVbRuH1EK6bGsOJRKbquE4zMaQXp+2bjpuurGPzGTONb3QVdQgMw7QHFoQ+IlvQEAl4MRz0NRSEf3/hHLwewvVTlVbKt121oeq4/3bLNBYyqmkRfOwtr6jqb9QNXrVlFL/0hp34qeu3mK0mtLIwt5OcGArgt996Od5mpM2yhcAw3YcFoY/IFEoI+30YDYuGLbCXckX81PVb8Il3XlX3uH2WuAEA3Lg9hhu3x1Y91lZRvB785m2XAwCEEBgK+pDKlxAN6JYBEeEXb9lpHs8WAsN0H/629RG5goaw34vhkFLXQtDKAql8yfS7X2oQEbYawe9aO7FxlhHDdB/+tvUR2YKGsFGkVU8QUnn9tUtVEIBKSqx0GTlhC4Fhug9/2/qIbKGEsF8v0qpXh7CYvfQFQdZLSJeRExlc5hgCw3QPjiH0EVnDZeTzUl0LQb52KQuCtBBqu4zYQmCYbsPftjbhTOU8Pp9GpVt4NXPJLIpa2fac02VU6/2mIIQvfUGI+Gu5jNhCYJhuw9+2NvD86SW84dP/gcdf1ncbPTafxhv/7Ef4t+fPuR6/nC/izX/+I3z98TnzOSGEzWWklYW5mY2TtWAhTK+L6pv51NjlTO6hMGHstsYwTOdhQWgDMq//8LkUAODo+TQA4IcvXXA9/mQiC7VUxpHzKfM5tVRGWQBhS6O3Wm6jtSAIG0dC+NFvvQG3XrHe9XWvh3D/r78e779xqssjY5jBhQWhDcitIaUwzCZ199GBmYSr2+ek8frJhUqBWNbYECesDIYgAPoeyJ46LTTWDQc5hsAwXYS/bW0gkTIEYUEKgv7/2aU8jru0iZCvW/c7zhqtr8N+n7l7WC1BWM4VEfB5TD87wzBMO2BBaAPzaX3vgoqFkEM8qvfuOWBsCGNFCsGpizloZd2CMC0Ei8uoVurpUq54yVsHDMP0HywIbWDBcBnNJbMQQmAumcVrdoxjMhbCQ0erBUIx+vkAABM+SURBVEEKR0Er4/xyHoBFEPzNuYxYEBiGaTcsCG1AxhBSagkLmQJOXcxiKhbGvuk4Hjm2gJIjvXQ2mTUtCCkOVpfRaFh/jQWBYZhuMpCCcOjsMv7uwIm2nS+RLphbUj5+IomiJgxBmEBKLeGZU0vmsUWtjDOLeeyd1pvOybhDVq1YCBFjvwIWBIZhuslACsK/PnUaf/idF6tW7islkVZx7eQoAJibyE/Fwnj1Vr019YtnKoJwZlGPG7xm+zi8HqpYCMWKIBAR4lE/zi+rrtdjQWAYphMMpCDkjck3la/fYroZsoUSsgUN101VC8K6oQACPo9t/wH5846JCDaNBiuCoFZcRvL9czX2LVjKFs1MJIZhmHYxkIKglnTLoNVtKt1YMDKMJmNhxKMBnFzI6hW4I0F4PISpWNhVELaOh7E1FrHEECoWgjyfmyBoZYGUeum2vmYYpn8ZSEGQFkI7BGHeCCjHo36zx//msRB8Rg8eXRBy5vGzySz8Xg/WDwVtk37OGFPIEISpWBhnl/NQS5rtestrpCiNYZj+Y0AFoX0WgixKi0cDZsM2+T9QWenLiuW5ZBZbYiHTeljIFJBWS8ioJfg8ZDZzm4qFIYReq2BlrVQpMwzTfwykIMhVd1sEwXAZxaMBs8f/pEUQpmJhpNUSkhn9uJML2SrhmF3IIlvQEDICyrbXHG4jFgSGYTrFQApCOy0EWZQ2HvW7WgjWiV0IgVk3QUhmkStoiPgrewNMGe4nZxxhLbS+ZhimPxlIQWjVQnh4JoFXfeLfcW4pX/VaIq1iOOhDwOfF9rg+iW8bj5ivy4l9NpnFQqaAlFqqEoQTiQwyRutryUQ0gKDiMesUJHLMo2whMAzTZgZSEKSFUG+bSivffe4slvMl/Mfh6nbWiXQB8ajes//6qTH81c9cjzdfsc58fXKsstJ/9Li+X8J1U3p9wkhYwY6JCB49sYCc4TKSEFVnKAHsMmIYpnMMpCC0aiHIBnX7XRrVzadVUxCICG+/eqOZYQToWUPrhgKYTWaxfyaBoYAP12wZMV/fNx3Ho8eTWMwVbS4jAHUFgesQGIZpNwMpCNJCkJvV12MumcXJhSyCigcPH1tAuWzf3yCRVhEf8tc9h5zYD8wksGfnuE0w9k3HkStqePbUos1CAPTg9KwlQwng1tcMw3SOgRSEVgrTpFXw83u3I5kp4MWzy7bXFywuo1pMxcJ4Zm4Js8ks9hk9jCR7do7DQ0BRE4gEvFXvyxY0LBgZSnLM7C5iGKYT9FQQiOitRHSYiGaI6OPduq7aQmHa/pkE1g8H8IHXbgNg39+gUCpjKVdsKAiTsbBZeLbXIQjDQQXXGH2QQkq1ywiwp56yIDAM0yl6JghE5AXwWQBvA3AlgPcT0ZXduHazFkK5LPDwTAJ7p+NYPxzEZeujtjjCQqZSlFYPWcG8YTiInRORqtel1RB2uIy2uqSesiAwDNMpemkh3AhgRghxXAhRAPA1AO/q9EW1skBBc88yWsoVcd+L583HL55dxsVsETfv0ifsvdNxPHYiaba+SKR0V854tHEMAQD27YqbhWdWTEFwuIy2jFUK1ySLWRYEhmE6Qy8FYTOAOcvjU8ZzNojoDiI6SEQH5+fnV33RgmEdBBUPUmrJ3MISAD734HH8wlcO4oSxD7K0Bvbu1Cfsm3fFoZbKePLkRQDA8URa/0VGQ3WvuWvdEMYjfvzEqza6vn7d1Bi2jYdx2boh2/NBxYv1wwGctFgICxkVY5H6AsQwDLMSeikI1UtlQFQ9IcRdQojdQojdExMTq76oXN2vHw4CsFsJDznSSw/MJHDZ+ijWGcfeuH0cPg+Zxz08s4DhoA9XbByue82RsIIn/seteMMr1rm+7vd58B+/+Qb81Ku3VL1mTT3NFzWcX1ZtldAMwzDtopeCcArApOXxFgBnOn1RGT9YN6T7/WUcYSlbxHOnFgEAB44mkC9qeOxEEvumKyIUDfhw3dQoDswkIITA/pkEXrszDq/HTdvag7Ujqmx0x4LAMEwn6KUgPA5gFxFtJyI/gPcB+HanLyotBLnql4Lw4+MLKAtg17ooHj6WwGMnklBLZezbNW57/97pOJ47vYSn5hZxejGHfbvsWUPtZioWxrnlPPJFzRSGSRYEhmE6QM8EQQhRAvDLAP4dwCEA3xBCvNDp6+aNKmWnhbB/Zh4Rvxe/eMtOLOdL+Ov/OAafh3Djdrsg3LwrDiGAP/3+YQCoqitoN7IN9unFnOk6YguBYZhO4Gt8SOcQQtwD4J5uXlM1qpTXOyyEAzML2LNjHK+7THcR/fj4Am7cFkM0YL9Fr9oyimjAhx8fX8Dm0ZCZGtoprLUIJxeyCPu9iDfIamIYhlkJA1epbLqMLBbCqYtZnEhksHc6jng0YAaJnUVkAKB4PdizIwZAtw7c0kjbidktdSGL2aTeOrvT12QYZjAZOEGQQWWrhSCrj2U8QNYdOOMHEukm6nT8ALC0wU5mMZfMcvyAYZiO0VOXUS+QFsJISEHA58FSroiXzqWwbiiAXeuiAIDb92yF3+vBtZNjrud493WbcWYpjzdd4Z5G2k5kG+yThoXQDRFiGGYwGThBUC2FaSMhBYvZAh6eSeD1l02YrpjJWBgfu+0VNc8xGvbjd99+RVfGC+hxhKfnLiJX1DigzDBMxxg4l5G0EAI+L0ZCCh49kcRCpuAaL+gXJmNhc+9mFgSGYTrFwAmCtBAChoVw0ugT1M+uGKsIcAyBYZhOMXCC4LQQAL0YTQaZ+xEpCETAlrH6fZMYhmFWysAJgjOGALinl/YT1vbZvFMawzCdYiAEYXYhi8df1je4V4saiAC/12PuS9zpauPVIttgs7uIYZhOMhCCcNdDx3DHVw4C0C2EgM8DIsKWsRDCfi9eYxSa9StBxYvt8Qgu3zDU+GCGYZgVMhBppyMhBcv5EoQQyBc1BHy62+X2m7bi7VdvxFCw/zec+fqH9yDiH4g/F8MwPWIgZpiRkAKtLJBWS1BLZQQV3TAK+LzY1GBzm35h3VD/Br0ZhlkbDITLSAaPl3JF5IsaB2YZhmFcGDhBkDEEhmEYxs5AzIzDbCEwDMM0ZCAEQVoIy7ki8kW2EBiGYdwYiJnR7jJiC4FhGMaNgRMEthAYhmHcGYiZMRrwwesh00IIsIXAMAxTxUAIAhEZex+whcAwDFOLgZkZR0KKmXbKMQSGYZhqBkYQhqUgFDUEfSwIDMMwTgZGEEZCCpZlYZoyML82wzBM0wzMzDgSUnAxW0RBK7OFwDAM48IACYIPF1J5AGALgWEYxoWBmRlHQgryRWO3NM4yYhiGqWJgZkZZnAaA6xAYhmFcGEhBCLLLiGEYpoqBmRltFgIHlRmGYaroiSAQ0SeI6DQRPW38e3unrznMFgLDMExdermF5p1CiE9362JsITAMw9RnYJbKHENgGIapTy9nxl8momeJ6ItENFbrICK6g4gOEtHB+fn5FV9sNOw3f2YLgWEYppqOCQIR3UdEz7v8exeAvwawE8C1AM4C+LNa5xFC3CWE2C2E2D0xMbHi8UT8Xng9BIAtBIZhGDc6FkMQQry5meOI6HMAvtupcViug5GQgmSmwBYCwzCMC73KMtpoefgeAM9347oyjsCtKxiGYarpVZbR/yaiawEIAC8D+HA3LipTT3k/BIZhmGp6IghCiNt7cV3TQuBeRgzDMFUM1Mw4ElJABPi9A/VrMwzDNMVAzYwjIR8CPg+IqNdDYRiG6Tt6Wancdd67ewq71g31ehgMwzB9yUAJwtVbRnD1lpFeD4NhGKYvGSiXEcMwDFMbFgSGYRgGAAsCwzAMY8CCwDAMwwBgQWAYhmEMWBAYhmEYACwIDMMwjAELAsMwDAMAICFEr8fQNEQ0D+DkCt8eB5Bo43A6Qb+Psd/HB/T/GPt9fACPsR302/i2CiEa7jB2SQnCaiCig0KI3b0eRz36fYz9Pj6g/8fY7+MDeIztoN/HVwt2GTEMwzAAWBAYhmEYg0EShLt6PYAm6Pcx9vv4gP4fY7+PD+AxtoN+H58rAxNDYBiGYeozSBYCwzAMU4eBEAQieisRHSaiGSL6eB+MZ5KIHiCiQ0T0AhH9mvF8jIjuJaKjxv9jfTBWLxE9RUTfNR5vJ6JHjTF+nYj8PRzbKBH9ExG9ZNzLm/rtHhLRfzf+xs8T0d1EFOz1PSSiLxLRBSJ63vKc630jnb80vjvPEtH1PRrfnxp/52eJ6F+IaNTy2u8Y4ztMRLd1eny1xmh57WNEJIgobjzu+j1cKWteEIjIC+CzAN4G4EoA7yeiK3s7KpQA/IYQ4goAewD8kjGmjwO4XwixC8D9xuNe82sADlke/38A7jTGeBHAB3syKp2/APB9IcTlAK6BPs6+uYdEtBnArwLYLYR4JQAvgPeh9/fwSwDe6niu1n17G4Bdxr87APx1j8Z3L4BXCiFeBeAIgN8BAON78z4AVxnv+SvjO9+LMYKIJgHcCmDW8nQv7uGKWPOCAOBGADNCiONCiAKArwF4Vy8HJIQ4K4R40vg5BX0i22yM68vGYV8G8O7ejFCHiLYA+AkAnzceE4A3Avgn45CejZGIhgG8DsAXAEAIURBCLKLP7iH0XQlDROQDEAZwFj2+h0KIBwEkHU/Xum/vAvAVofMIgFEi2tjt8QkhfiCEKBkPHwGwxTK+rwkhVCHECQAz0L/zHaXGPQSAOwH8FgBrcLbr93ClDIIgbAYwZ3l8yniuLyCibQCuA/AogPVCiLOALhoA1vVuZACA/wP9w102Ho8DWLR8MXt5L3cAmAfwd4ZL6/NEFEEf3UMhxGkAn4a+WjwLYAnAE+ife2il1n3rx+/PzwP4N+PnvhkfEb0TwGkhxDOOl/pmjI0YBEEgl+f6IrWKiKIAvgngo0KI5V6PxwoRvQPABSHEE9anXQ7t1b30AbgewF8LIa4DkEF/uNhMDD/8uwBsB7AJQAS6+8BJX3wea9BPf3MQ0e9Bd7l+VT7lcljXx0dEYQC/B+D33V52ea4v/+aDIAinAExaHm8BcKZHYzEhIgW6GHxVCPHPxtPnpSlp/H+hV+MDsBfAO4noZehutjdCtxhGDfcH0Nt7eQrAKSHEo8bjf4IuEP10D98M4IQQYl4IUQTwzwBei/65h1Zq3be++f4Q0QcAvAPAz4hKvny/jG8ndOF/xvjObAHwJBFtQP+MsSGDIAiPA9hlZHb4oQegvt3LARm++C8AOCSE+HPLS98G8AHj5w8A+Fa3xyYRQvyOEGKLEGIb9Hv2QyHEzwB4AMB/Mg7r2RiFEOcAzBHRK4yn3gTgRfTRPYTuKtpDRGHjby7H2Bf30EGt+/ZtAD9nZMrsAbAkXUvdhIjeCuC3AbxTCJG1vPRtAO8jogARbYceuH2s2+MTQjwnhFgnhNhmfGdOAbje+Jz2xT1sCiHEmv8H4O3QMxOOAfi9PhjPPugm47MAnjb+vR26j/5+AEeN/2O9Hqsx3lsAfNf4eQf0L9wMgH8EEOjhuK4FcNC4j/8KYKzf7iGAPwTwEoDnAfw9gECv7yGAu6HHNIrQJ64P1rpv0N0dnzW+O89Bz5jqxfhmoPvh5fflbyzH/54xvsMA3tare+h4/WUA8V7dw5X+40plhmEYBsBguIwYhmGYJmBBYBiGYQCwIDAMwzAGLAgMwzAMABYEhmEYxoAFgRkIiEgjoqct/+pWNRPRR4jo59pw3Zdl18sW33cbEX2CiMaI6J7VjoNhmsHX+BCGWRPkhBDXNnuwEOJvOjmYJrgZegHb6wAc6PFYmAGBBYEZaIw2A18H8AbjqZ8WQswQ0ScApIUQnyaiXwXwEeg9dF4UQryPiGIAvgi9yCwL4A4hxLNENA69aGkCevEZWa71s9DbYfuhNzP8b0IIzTGe9+L/b+9+XmSO4ziOP18ojRQhmwNKOUjKsnHYi3KS2gNqk39A5I5Ie3Xg4i6izU1qSw7kR0mENu3dfYv2Kr0c3p+pMY25aHYP83pcpj6f+fGdOcx7Pu/vd16finbeR+UgTQArko7bnhnFZxDRlZZRjItOX8totmduxfYx4B6V19TvKjDpyuK/2MbmgC9t7DrwsI3fAt65AveeAXsAJB0AZoHptlL5DVzofyHbT6hMpm+2D1H/cJ5MMYjVkBVCjIthLaP5ntu7A+YXgceSnlIRGVDxI2cBbL+UtF3SFqrFc6aNL0j60e5/EjgKfKxYIzr8O3hvPxVzALDJtWdGxMilIET8HUU8KMvlNPVFPwPclHSQ4ZHGg55DwAPb14YdiKRPwA5gg6QlYJekr8AV22+Hv42I/5OWUUS1crq373snJK0Ddtt+RW0WtBXYDLyhtXwknQCWXXta9I6fogL3oALjzkna2ea2SdrbfyC2p4AF6vzBbSqM8XCKQayGrBBiXHTaL+2u57a7l55ulPSB+oF0vu9x64FHrR0kai/kn+2k831Ji9RJ5W509BwwL+kz8Jq2t67tJUk3gBetyPwCLgPfBxzrEerk8yXgzoD5iJFI2mmMtXaV0ZTt5bU+loi1lpZRREQAWSFERESTFUJERAApCBER0aQgREQEkIIQERFNCkJERAApCBER0fwBkdjPBddsqKEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from dqn_agent import Agent\n",
    "\n",
    "agent = Agent(state_size=37, action_size=4, seed=0)\n",
    "\n",
    "\n",
    "def dqn(brain_name, n_episodes=20000, max_t=10000, eps_start=1.0, eps_end=0.01, eps_decay=0.995):\n",
    "    \"\"\"Deep Q-Learning.\n",
    "    \n",
    "    Params\n",
    "    ======\n",
    "        n_episodes (int): maximum number of training episodes\n",
    "        max_t (int): maximum number of timesteps per episode\n",
    "        eps_start (float): starting value of epsilon, for epsilon-greedy action selection\n",
    "        eps_end (float): minimum value of epsilon\n",
    "        eps_decay (float): multiplicative factor (per episode) for decreasing epsilon\n",
    "    \"\"\"\n",
    "\n",
    "\n",
    "    scores = []                        # list containing scores from each episode\n",
    "    scores_window = deque(maxlen=100)  # last 100 scores\n",
    "    eps = eps_start                    # initialize epsilon\n",
    "    for i_episode in range(1, n_episodes+1):\n",
    "        # reset the environment\n",
    "        env_info = env.reset(train_mode=True)[brain_name]\n",
    "        state = env_info.vector_observations[0]\n",
    "        score = 0\n",
    "        for t in range(max_t):\n",
    "            action = agent.act(state, 0)\n",
    "            \n",
    "            env_info = env.step(action)[brain_name]\n",
    "            next_state = env_info.vector_observations[0]\n",
    "            reward = env_info.rewards[0]\n",
    "            done = env_info.local_done[0]\n",
    "            \n",
    "            #next_state, reward, done, _ = env.step(action)\n",
    "            agent.step(state, action, reward, next_state, done)\n",
    "            state = next_state\n",
    "            score += reward\n",
    "            if done:\n",
    "                break \n",
    "        scores_window.append(score)       # save most recent score\n",
    "        scores.append(score)              # save most recent score\n",
    "        eps = max(eps_end, eps_decay*eps) # decrease epsilon\n",
    "        print('\\rEpisode {}\\tAverage Score: {:.2f}'.format(i_episode, np.mean(scores_window)), end=\"\")\n",
    "        if i_episode % 100 == 0:\n",
    "            print('\\rEpisode {}\\tAverage Score: {:.2f}'.format(i_episode, np.mean(scores_window)))\n",
    "        if np.mean(scores_window)>=13.0:\n",
    "            print('\\nEnvironment solved in {:d} episodes!\\tAverage Score: {:.2f}'.format(i_episode-100, np.mean(scores_window)))\n",
    "            torch.save(agent.qnetwork_local.state_dict(), 'checkpoint.pth')\n",
    "            break\n",
    "    return scores\n",
    "\n",
    "scores = dqn(brain_name)\n",
    "\n",
    "# plot the scores\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "plt.plot(np.arange(len(scores)), scores)\n",
    "plt.ylabel('Score')\n",
    "plt.xlabel('Episode #')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.save(agent.qnetwork_local.state_dict(), 'qnetwork.pth')\n",
    "torch.save(agent.policy_network.state_dict(), 'policy\n",
    "           .pth')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([3.3815, 0.4096], dtype=torch.float64)\n",
      "tensor([1.4774, 0.3433], dtype=torch.float64)\n"
     ]
    }
   ],
   "source": [
    "def logrelu(x):\n",
    "    neg = torch.lt(x, 0).double() * -1 * torch.log(-x+1)\n",
    "    neg[torch.isnan(neg)] = 0\n",
    "\n",
    "    pos = torch.gt(x, 0).double() * torch.log(x+1)\n",
    "    pos[torch.isnan(pos)] = 0\n",
    "\n",
    "    zero = torch.eq(x, 0).double() * x\n",
    "\n",
    "    return neg + zero + pos\n",
    "\n",
    "x = torch.tensor(np.random.normal((2, 1)))\n",
    "print(x)\n",
    "print(logrelu(x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Take Random Actions in the Environment\n",
    "\n",
    "In the next code cell, you will learn how to use the Python API to control the agent and receive feedback from the environment.\n",
    "\n",
    "Once this cell is executed, you will watch the agent's performance, if it selects an action (uniformly) at random with each time step.  A window should pop up that allows you to observe the agent, as it moves through the environment.  \n",
    "\n",
    "Of course, as part of the project, you'll have to change the code so that the agent is able to use its experience to gradually choose better actions when interacting with the environment!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Score: 11.0\n"
     ]
    }
   ],
   "source": [
    "env_info = env.reset(train_mode=False)[brain_name] # reset the environment\n",
    "state = env_info.vector_observations[0]            # get the current state\n",
    "score = 0                                          # initialize the score\n",
    "while True:\n",
    "    #action = np.random.randint(action_size)        # select an action\n",
    "    action = agent.act(state, 0)\n",
    "    env_info = env.step(action)[brain_name]        # send the action to the environment\n",
    "    next_state = env_info.vector_observations[0]   # get the next state\n",
    "    reward = env_info.rewards[0]                   # get the reward\n",
    "    done = env_info.local_done[0]                  # see if episode has finished\n",
    "    score += reward                                # update the score\n",
    "    state = next_state                             # roll over the state to next time step\n",
    "    if done:                                       # exit loop if episode finished\n",
    "        break\n",
    "    \n",
    "print(\"Score: {}\".format(score))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When finished, you can close the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "#env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. It's Your Turn!\n",
    "\n",
    "Now it's your turn to train your own agent to solve the environment!  When training the environment, set `train_mode=True`, so that the line for resetting the environment looks like the following:\n",
    "```python\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}