examples/quickstart/rec/rec_synthetic_data_collection.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Quickstart Example with Synthetic Recommender Simulation and Dataset (Data Collection)\n",
    "This notebook provides an example of visualizing the logged dataset collected on an recommender environment.\n",
    "\n",
    "This example on consists of the following case:\n",
    "1. Discrete Action Case\n",
    "2. Collecting Logged Datasets with Multiple Behavior Policies and Random Seeds\n",
    "\n",
    "\\* This library uses [d3rlpy](https://github.com/takuseno/d3rlpy)'s algorithm implementations.  \n",
    "\\* Also, our data collection module is highly inspired by [Open Bandit Pipeline](https://github.com/st-tech/zr-obp)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import SCOPE-RL modules\n",
    "import scope_rl\n",
    "from scope_rl.dataset import SyntheticDataset\n",
    "from scope_rl.policy import OnlineHead\n",
    "from scope_rl.policy import EpsilonGreedyHead\n",
    "from scope_rl.ope.online import (\n",
    "    calc_on_policy_policy_value,\n",
    "    visualize_on_policy_policy_value,\n",
    ")\n",
    "\n",
    "# import recgym\n",
    "from recgym import RECEnv\n",
    "\n",
    "# import d3rlpy algorithms\n",
    "from d3rlpy.algos import DiscreteRandomPolicy\n",
    "\n",
    "# import from other libraries\n",
    "import gym\n",
    "import torch\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.utils import check_random_state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "from dataclasses import dataclass\n",
    "from typing import Tuple, Union, Optional\n",
    "from glob import glob\n",
    "from tqdm import tqdm\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.0.0\n"
     ]
    }
   ],
   "source": [
    "# version\n",
    "print(scope_rl.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set random state\n",
    "random_state = 12345\n",
    "random_ = check_random_state(random_state)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ignore warnings\n",
    "import warnings\n",
    "warnings.simplefilter('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 0. Description of Synthetic Recommender Simulation Environment\n",
    "\n",
    "To begin with, we briefly describe the basic usage of the environment.\n",
    "\n",
    "#### RL setup for Recommendation\n",
    "In recommendation, the objective of the RL agent is to maximize reward\n",
    "\n",
    "We often formulate this recommendation problem as the following (Partially Observable) Markov Decision Process ((PO)MDP):\n",
    "- `state`: \n",
    "   - A vector representing user preference.  The preference changes over time in an episode by the actions presented by the RL agent.\n",
    "   - When the true state is unobservable, the agent uses observation instead of state.\n",
    "- `action`: Indicating which item to present to the user.\n",
    "- `reward`: User engagement signal. Either binary or continuous.\n",
    "\n",
    "Let's see how it works!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# setup environment\n",
    "env = gym.make('RECEnv-v0')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "agent = OnlineHead(\n",
    "    DiscreteRandomPolicy(),\n",
    "    name=\"random\",\n",
    ")\n",
    "agent.build_with_env(env)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# interact agent with the environment\n",
    "# only 6 lines are needed for RL interaction\n",
    "for episode in range(10):\n",
    "    obs, info = env.reset()\n",
    "    done = False\n",
    "\n",
    "    while not done:\n",
    "        action = agent.sample_action_online(obs)\n",
    "        obs, reward, done, truncated, info = env.step(action)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(5,)\n",
      "[ 0.13726447  0.0194582   0.10000844 -0.04762143  0.09446818]\n"
     ]
    }
   ],
   "source": [
    "# state \n",
    "print(obs.shape)\n",
    "print(obs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkIAAAGwCAYAAABFFQqPAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABh50lEQVR4nO3dd3yUVdr/8c+kJ6QTkhAIRaQ3KYIoKAorqIu6dhcVFd3ywIqi/sSKZRXXFcujri423FUWu49tUUQQC0pXQLpCApiEEEiFtJnfH4d7kkhLmZl7yvf9euWVw2Tmvq+h5co517mOw+VyuRAREREJQWF2ByAiIiJiFyVCIiIiErKUCImIiEjIUiIkIiIiIUuJkIiIiIQsJUIiIiISspQIiYiISMiKsDsAf+d0Otm1axcJCQk4HA67wxEREZFGcLlclJaWkpWVRVjYked9lAgdw65du8jOzrY7DBEREWmG3Nxc2rdvf8SvKxE6hoSEBMD8RiYmJtocjYiIiDRGSUkJ2dnZ7u/jR6JE6Bis5bDExEQlQiIiIgHmWGUtKpYWERGRkKVESEREREKWEiEREREJWaoR8pDa2lqqq6vtDiPkREZGEh4ebncYIiISoJQItZDL5SIvL499+/bZHUrISk5OJjMzU32eRESkyZQItZCVBKWnpxMXF6dvxj7kcrmoqKigoKAAgLZt29ockYiIBBolQi1QW1vrToJat25tdzghKTY2FoCCggLS09O1TCYiIk2iYukWsGqC4uLibI4ktFm//6rREhGRplIi5AFaDrOXfv9FRKS5lAiJiIhIyFIiJCIiIiFLiZD4tUWLFuFwONSeQEREvEKJkIiIBK6aSrsjkACnREioqqqyOwS/iEFEAsyqV+GvGfDDm3ZHIgFMiZAnuVxQVW7Ph8vV6DBHjhzJ5MmTufHGG0lLS2PMmDGsXbuWs846i/j4eDIyMrjyyispLCwE4MMPPyQ5OZna2loAVq9ejcPhYNq0ae5rXnfddVxxxRUA7Nmzh8svv5x27doRFxdH3759+c9//nPMGAA+/vhjunXrRmxsLKeffjrbtm1ryZ+IiASr6v2w4H7ABRs+sDsaCWBqqOhJ1RXwUJY9975jF0S1avTTX3nlFf785z/z9ddfs2/fPs444wyuu+46Hn/8cfbv389tt93GJZdcwueff86IESMoLS1l1apVDB48mC+++IK0tDQWLVrkvt4XX3zBbbfdBsCBAwcYNGgQt912G4mJiXz00UdceeWVdOnShSFDhhw2BoDc3FwuuOACJk2axB/+8AeWL1/OzTff7JnfHxEJLqtehbJ8M85fZ28sEtCUCIWorl278sgjjwDw17/+lQEDBvDQQw+5v/7SSy+RnZ3Npk2b6NatGyeccAKLFi1i8ODBLFq0iJtuuon77ruPsrIyiouL2bJlC6eddhoA7dq145ZbbnFf6y9/+QuffPIJb7zxRoNEqH4MAHfccQddunRh5syZAHTv3p01a9bwt7/9zau/FyISYGqq4Ksn6n69ZytUVUCUmttK0ykR8qTIODMzY9e9m2DQoEHu8ffff8/ChQuJj48/5Hlbt26lW7dunHbaaSxatIibb76ZL7/8khkzZvDGG2/w1VdfUVRURFZWFl27dgXM0SMPPfQQb7zxBjt37qSqqorKyspDOnDXjwFg/fr1DB06tMFjw4YNa9L7EpEQ8P1/oGQHxGeCsxoq9sDu9dBu0LFfK/IrSoQ8yeFo0vKUnVq1qouzrKyMcePGHXbmxTrIdOTIkbz00kt8//33REZG0qNHD0aOHMmiRYvYu3evezYI4O9//ztPPvkkTzzxBH379qVVq1bceOONhxRE149BRKRRamvgq8fM+JQbYPOn8NMiszymREiaQYmQMHDgQN5++206depERMTh/0pYdUKPP/64O+kZOXIkDz/8MHv37m1Qy/P1119z3nnnuYunnU4nmzZtolevXkeNo2fPnrz//vsNHvv2229b8tZEJNisfRv2boO41jDoaijZZRKhvLU2ByaBSrvGhEmTJlFUVMTll1/OsmXL2Lp1K5988gnXXHONe6dYSkoK/fr147XXXmPkyJEAnHrqqaxcuZJNmzY1mBHq2rUr8+fP55tvvmH9+vX88Y9/JD8//5hx/OlPf2Lz5s3ceuutbNy4kTlz5jB79mxvvGURCUROJ3z5qBkPm2Rm4DN6m1+rYFqaSYmQkJWVxddff01tbS1nnnkmffv25cYbbyQ5OZmwsLq/Iqeddhq1tbXuRCg1NZVevXqRmZlJ9+7d3c+76667GDhwIGPGjGHkyJFkZmZy/vnnHzOODh068Pbbb/Pee+/Rv39/nnvuuQYF3CIS4ta/D4WbICYJTrzePOZOhNY2qY2IiMXhculvztGUlJSQlJREcXExiYmJDb524MABfv75Zzp37kxMTIxNEYr+HERCgMsFz42A/DVw2m1w+h3m8ZpKeLAtuGrhph8hqZ29cYrfONr37/o0IyQiIv5v0ycmCYqKh6F/qns8IhrSupmxlsekGZQIiYiIf3O5YPHfzfjEiRCX2vDr7uWxNb6NS4KCEiEREfFvPy2CncshIgaGTT706yqYlhZQIuQBKrOyl37/pdlcLhXYBoLFB3eKDboa4tMP/XpGH/NZiZA0gxKhFoiMjASgoqLC5khCm/X7b/15iDRKaR7M7A7v/8XuSORotn8D27+CsEg4+YbDPyfzYCJUuBmqD/guNgkKaqjYAuHh4SQnJ1NQUABAXFwcDofD5qhCh8vloqKigoKCApKTkwkPD7c7JAkkWxaYQztX/Rv6XQqdR9gdkRyONRs0YPyRd4QltIXYFNi/Fwo3Qtv+votPAp4SoRbKzMwEcCdD4nvJycnuPweRRiv4sW786V1w/UII0yS5X9m5ArYuAEc4nHLjkZ/ncJjlsW1fmuUxJULSBEqEWsjhcNC2bVvS09Oprq62O5yQExkZqZkgaZ76idAvq83RDf0uti0cOYzFM83nfpdAauejPzejt0mEdNSGNJESIQ8JDw/XN2SRQJJ/MBHqNhY2zYMF90PPcRCpppx+IW8tbPwIcMDwqcd+fv0O0yJNoHlgEQk9FUVQlmfG5z5lakyKc2DpLHvjkjpfHpwN6v07aNPt2M/XURvSTEqERCT0WNuskzuY7dhn3GV+/eWjJkkSexVuhnXvmvGImxv3mjY9wREGFXugTDWb0nhKhEQk9Fj1QekHZxH6X27GB4rrdimJfb58DHBB93PqtsYfS1QcpHYxYy2PSRMoERKR0GMlQhm9zOewcDjzfjNeOguKfrYnLoG92+CH18341EbOBlnUYVqaQYmQiIQeq1A6vVfdY8ePhuNOB2c1LLjPnrgEvnrCnCTfZRS0G9S017o7TGtGSBpPiZCIhBaXCwrWm7E1g2A58wHAYepTdiz3eWghr3gnrH7NjE+9temv14yQNIMSIREJLftyoKrUHNnQ+viGX8vsa+qFwDRZ1O4j3/rmKaitgo7DoeOwpr/eSoR2b4SaKs/GJkFLiZCIhBarPiitG4Qf5ny6M+4yp5znLIENH/k2tlBWVgArZpvxqbc07xrJHSA60Sxv7tnssdAkuCkREpHQYi2bZPQ6/NeT2sFJ/2PGn02HWnWM94klz0DNfmg3GI4b2bxrOBxaHpMmUyIkIqHFqg9KP0IiBDD8JohLgz1b6mYpxHsqimDZC2Z86q0moWkudZiWJlIiJCKhxb11vveRnxOTCCOnmfGih+FAiffjCmXf/ROqyiCjL3Qb07JrWX+uOnNMGkmJkIiEjpoqKNxkxuk9j/7cQVebYuqKQvj6Sa+HFrIOlMB3z5rxqbe0bDYI6m2h19KYNI4SIREJHXs2g7PGFNQmZR/9ueGRMPpeM17yDJTs8np4IWnZC6ajd1p36Hluy69nLXmW5UF5YcuvJ0FPiZCIhA53I8WejZt56PFbyD7JFPF+/qB3YwtFVeWw5GkzHnEzhHngW1J0PKR0NmPNCkkjKBESkdBRcJiO0kfjcMCZfzXj1a9B3hrvxBWqVsw2h6SmdII+F3ruuto5Jk2gREhEQkdjCqV/LftE6HU+4IL593gjqtBUfQC+/l8zHj4VwiM8d20dtSFNoERIRELH4c4Ya4zR000n6q2fw5YFno8rFK1+1dTxJLar6+btKdpCL02gREhEQsOBEijOMeNj7Rj7tdTj4MTrzHj+PeCs9Wxsoaa22hyuCnDKjRAR5dnrW4lQwQaorfHstSXoKBESkdBgNVJMaAtxqU1//Wn/D6KTzCzD93M9G1uo+eF1KM6FVukw8ErPXz+lM0S2gtpKKNrq+etLUFEiJCKhoeBg4WxTl8Uscalw6s1m/PlfoarCM3GFGmctfDnTjE/+C0TGev4eYWF1R6hoeUyOQYmQiIQGa0boSGeMNcaQP0JSByjdBd/+wzNxhZp170LRTxCbAoOv9d59tHNMGkmJkIiEBnehdBN2jP1aZAyMutuMv3oCyna3OKyQ4nTC4kfN+KRJpuePt1g7x3TUhhyDEiERCX4uV72lsSYWSv9an4ugbX+oKoUv/tby2ELJxo9g93rT2XvI9d69l2aEpJGUCIlI8CvNg/17wREGbbq37FphYXVNFle8DIWbWx5fKHC5YPHfzXjIHyA22bv3s2rBSnaYP3uRI1AiJCLBz5oNSu3imeLczqdC1zHm3LLP7m359ULBls/gl+8hMg5O+h/v3y822dRzQd2yqMhhKBESkeBnfSNsSaH0r/3mfjPDtOFD2P6N564bjFwu+OIRMx58LbRq7Zv7anlMGkGJkIgEP2vHWEsKpX8tvQcMONgD59O7zTd7ObxtX8KOpRAebbbM+4o6TEsjBFwi9Mwzz9CpUydiYmIYOnQoS5cubdTr5s6di8Ph4Pzzz/dugCLif6ylMU/OCAGcfodp3LdzudkWLodn1QYNvAoSMn13XyVC0ggBlQi9/vrrTJ06lenTp7Ny5Ur69+/PmDFjKCgoOOrrtm3bxi233MKIESN8FKmI+A1nLezeaMbNbaZ4JAmZcMoNZrzgPqip9Oz1g0HOd/DzYgiLgFOm+Pbe1hb6gvU6FkWOKKASoccee4zrr7+ea665hl69evHcc88RFxfHSy+9dMTX1NbWMn78eO677z6OO+44H0YrIn6h6CeoOQARsZDSyfPXHzYZ4jNg7zZY9qLnrx/ovjzYN6j/5ZCc7dt7t+4CETFQXWH+fEQOI2ASoaqqKlasWMHo0aPdj4WFhTF69GiWLFlyxNfdf//9pKenM3HixEbdp7KykpKSkgYfIhLArELZ9B4QFu7560fHmyUygMWPwP59nr9HoNq1GjZ/aorKR0z1/f3Dwuv6Rml5TI4gYBKhwsJCamtrycjIaPB4RkYGeXl5h33NV199xYsvvsjzzz/f6PvMmDGDpKQk90d2to9/ghERz/JGofSvnXAFtOlh+tVY52hJ3WxQ34sh1aYZee0ck2MImESoqUpLS7nyyit5/vnnSUtLa/Trbr/9doqLi90fubm5XoxSRLzOW4XS9YVHmO30AN/9E/bleO9egaJgPaz/AHDAcBtmgyw6akOOIcLuABorLS2N8PBw8vPzGzyen59PZuahuxC2bt3Ktm3bGDdunPsxp9MJQEREBBs3bqRLly6HvC46Opro6GgPRy8itnGfMdbCozWOpeuZ0GmE2Sq+4AG4sPEz0UHJmhnrda5ZlrSLdo7JMQTMjFBUVBSDBg1iwYIF7secTicLFixg2LBhhzy/R48erFmzhtWrV7s/zj33XE4//XRWr16tJS+RUFBVYYqlwbtLYwAOB5z5gBmveQN2rvTu/fzZnq2w9m0zHnGLvbFYM0L7tsMB1XzKoQJmRghg6tSpTJgwgcGDBzNkyBCeeOIJysvLueaaawC46qqraNeuHTNmzCAmJoY+ffo0eH1ycjLAIY+LSJDavQFwQVxriE/3/v2yBkDfS0wiNP8emPCBSZBCzVePgcsJ3cZC2372xhKXCglZULrLLNd1GGpvPOJ3AioRuvTSS9m9ezf33HMPeXl5nHDCCcybN89dQJ2Tk0NYWMBMcomItxVYy2K9fJeQjLobfvw/s0S26RPoPtY39/UX+3Lg+7lmbPdskCWjt0mE8tcqEZJDBFQiBDB58mQmT5582K8tWrToqK+dPXu25wMSEf9l7RjL8PKyWH3JHeCkP8HXT5pZoeNHm2LqUPH1k+Yw2uNGQvaJdkdjZPSGLfO1c0wOS9MnIhK83D2EvLhj7HCGT4XYVCjcCKv+7dt726nkF1h58P2eequ9sdRn1QmpYFoOQ4mQiASv+ktjvhSbDKf9PzNe+BBUlvn2/nZZ8jTUVkKHYdDxFLujqePeOfYjHNw9LGJRIiQiwal8D5QdbLdhx/btwRMhpTOUF8A3T/n+/r5WXgjLDx53dOot/lUkntYVwqOgqhSK1eNJGlIiJCLByWqkmNwRohN8f/+IKBg93Yy/+V8oPXwH/KCx5BlzplfWAOgyyu5oGgqPhDbdzVh1QvIrSoREJDjZUSj9a73Oh/YnmgRh4UP2xeFt+/fC0oMNJE+91b9mgyzuOiElQtKQEiERCU52FUrX53DAmX8141X/rkvOgs13s8yyU3pv6HaW3dEcnjpMyxEoERKR4GQVSnvzjLHG6HAS9PitaTA4/x57Y/GGylL49h9mfOrN4K+93KxESGeOya/46d9YEZEWcDrrnTpvcyIEMPo+CIuAzZ/CT1/YHY1nLXsRDuyD1sebpUB/ZS2NFf0EVeX2xiJ+RYmQiASf4hyoKoOwSPMN2m5px8Pga83407uCZwt3VYXZMg8w4mYIC7c3nqOJT4dW6YALCjbYHY34ESVCIhJ8rBPn23Q3O4b8wWm3QXQi5P0Aa960OxrPWPkvKN9tumn3vdjuaI5NdUJyGEqERCT42NVI8WhapcHwG8348weg+oCt4bRYTaU5TgNg+E3+k3AejTsR0s4xqaNESESCj78USv/aSf8Die2gOBe+e87uaFpm9RxzkGlCWzhhvN3RNI620MthKBESkeCT74czQgCRsXDGXWb85WNQUWRvPM1VWw1fPWbGp0yBiGh742ks94zQGnC57I1FjMWPwrw76v7N2kCJkIgEl5oq2LPZjP0tEQLodylk9IXKYvjiEbujaZ41b8G+HIhLg4ET7I6m8dp0N7v3DhRDyU67oxEwM4vfPmNmSW2iREhEgkvhJnDWQHQSJLW3O5pDhYXDmfeb8bIXzHbuQOKshS9nmvHJkyEqzt54miIiGtK6mbGWx+xXthuKtppx9hDbwlAiJCLBxd0/qKd/HvUA0OUMcx6Xsxo+u8/uaJrmx/8zM24xyeZg2UCjnWP+I/c787lNT4hNsS0MJUIiElysw1b9rVD6135zP+CAH9+D3KV2R9M4Tqep6QA46c8Qk2hvPM2hnWP+I2eJ+dxhqK1hKBESkeDir4XSv5bZBwYc3G316V2BUby7aZ5JNKMSYMgf7I6meaydYzpqw37WjFCHYbaGoURIRIKLP/YQOpLT74SIWPMNYf0HdkdzdC4XLP67GQ+5DuJS7Y2nuawZoT2bA7+XUyCr3g+7VptxtmaEREQ840Bx3e4Tf18aA0jMMgXHAJ/da7al+6utn8OulSZxO2mS3dE0X0JbU4/icsJuHbVhm50rTY1cfCakdLI1FCVCIhI8rELphCxbiy+b5JQp0KqN2T2z/GW7ozkyqzZo8DUQ38beWFrC4VBjRX+Q+6353GGo7ZsalAiJSPDw147SRxOdACOnmfEXD5tZLX+z7WvI+QbCo+Dkv9gdTcspEbJfzsFEKPske+NAiZCIBJNAKZT+tYEToHVXqNgDXz1hdzSHsmqDBlxplvMCnbbQ28vprFcorURIRMRzAqlQur7wSPjNwX5C3/4DinfYG099O5bDTwtNR+ZTptgdjWfUT4QCYbdesNm9wcx8RsZBZl+7o1EiJCJBwuWqW+oIpKUxS/ezocPJUHMAPn/Q7mjqWLVB/S6DlI72xuIp6T3BEWZm4Mry7Y4m9Fj1Qe0GmR8CbKZESESCQ+kvcGAfOMIhrbvd0TSdwwFn/tWMv/8P/PKDvfGAiWHTf03SMPwmu6PxnMhYaH28GWt5zPdy/KN/kEWJkIgEB2tZrHUXiIyxN5bmaj8I+lwIuGD+3fYv21hnivW+ANKOtzcWT1OHafv4SUdpixIhEQkOgVoo/Wuj7jG7s35aBFsW2BfH7o3mXDGAETfbF4e3KBGyR2ke7NtuZhnb23fQan1KhEQkOLi3zve2N46WSulUd3zF/LvNae92+PIxwAU9fhuYNVfHoi309rC2zaf39puz6pQIiUhwsL6hpfe0Nw5PGHEzxCSZ5G71HN/fv+gnWPOmGZ96i+/v7wtWwrx7I9RU2RtLKMmp10jRTygREpHAV1tjvqFB4C+NgTnH69RbzXjhg1BV7tv7f/UEuGrh+N9A1gDf3ttXkrIhOskc81C4ye5oQoe7o7R/FEqDEiERCQZFP0FtpelLktLZ7mg8Y8gfILmD2Q235B++u+++3LpZKCsZC0YOh+qEfK2yrG43pM0HrdanREhEAp9VH9SmB4QFyX9rEdEwaroZf/0ElBX45r7f/K+ZJek0wq+WL7xCHaZ9a+cKM9OY2B6Ss+2Oxi1I/scQkZAWiGeMNUbvC8zSVFUZLHrY+/crzYcVr5hxMM8GWTQj5FvuYzX8K8FWIiQigc9dKB1kiVBYWF2TxRWzYbeXa1mWPGWWGNsPgc6nevde/sC9c0wzQj5h9Q/yg4NW61MiJCKBL1DPGGuMTsPN8RuuWvjsXu/dp3wPLHvJjE+91dTQBDtrh2FZPpTttjeWYOeshdxlZuwHB63Wp0RIRAJbVTkU/WzGgd5D6EhG32eODtn4EWz72jv3+O5ZqC6HzH7Q9TfeuYe/iY6vK64v0PKYV+Wvg6pSiErwu3+nSoREJLDt3gC4IC4N4tPtjsY72nSDQRPM+NO7wOn07PX374Pv/mnGoTIbZMlUY0WfsOqDsk+EsHB7Y/kVJUIiEtgK1pvPwVYo/Wsjb4eoeNi1Eta949lrL3seKkvMrrsev/Xstf2dOkz7htVI0c/qg0CJkIgEOvcZY/413e5x8elwyhQzXnAf1FR65rqVZXV9ikbcEjztBxpLW+h9ww87SltC7G+8iASdgiA6WuNYhk2C+EzYlwNLn/fMNVe8DPuLIPU46P07z1wzkFiJUMEG06FcPG9fLpTsMHVu7QbbHc0hlAiJSGDLD5LDVhsjqhWccacZL/477N/bsutV74dvnjLj4VMhPKJl1wtEyZ3MkmNtJezZYnc0wcmqD8rsawrU/YwSIREJXOWFUH6w43KbHvbG4isnjDdtAg7sg8WPtuxaq141W8eTsqHfpR4JL+CEhdW1XdDymHfk+N/5YvUpERKRwGX1D0rp5Jc/aXpFWDj85n4zXjoL9m5r3nVqqszhqmBqjyKiPBFdYFKHae/K9d/6IFAiJCKBLFQKpX/t+NHQ+TSorYIFDzTvGj/MNXUb8Rkw4ErPxhdolAh5z4GSut9XP9wxBkqERCSQhVKhdH0Ox8GjNxyw9i1zmGVT1NbAl4+Z8ck3QGSMx0MMKNpC7z07loHLCckdIbGt3dEclhIhEQlc+UF62GpjtO0H/S8z40/vBper8a9d9w7s/RliU2HwNd6JL5BYf39KdkBFkb2xBBt3fZB/zgaBEiERCVROZ10zxVBbGrOccRdExMD2r2Hjfxv3Gqezrsh62CSzEy3UxSRBUgczturOxDNylQiJiHjHvu3mbKzwKGjdxe5o7JHUHk76sxl/Nr1xfXA2fACFGyE6CYZc7934AomO2vC82mrYcXDZ1k/rg0CJkIgEKms2KK07hEfaG4udht8Eca2hcBOsfOXoz3W5TP8hgKF/NDMhYqjDtOflrTE/rMQk+XV7CyVCIhKYrELpUKwPqi8mCU67zYwXzYDK0iM/d/On5ptTVHzdTJIY2jnmee6DVof69dEt/huZiMjRuLfOh9iOscMZdI05IqN8N3z9v4d/jssFXzxixidOhLhU38UXCKydYwXrwVlrbyzBImeJ+Zztn/2DLEqERCQwFYRoD6HDiYiC0fea8ZKnoeSXQ5/z8xewc7kprh422afhBYTU48zvTXUFFP1sdzSBz+WCnIMzQn7aUdqiREhEAk9NJRRuNuNQXxqz9DzX/ORdXQELHzz069ZOsUFXm5PspaGw8LrZRdUJtdy+7VCWB2GR0G6g3dEclRIhEQk8hZvBVWt2PiW2szsa/+Busgisfq1u6RBg+xLY9qX5pnTyDfbEFwjUWNFzrP5BbftDZKy9sRxDwCVCzzzzDJ06dSImJoahQ4eydOnSIz73+eefZ8SIEaSkpJCSksLo0aOP+nwRCRAF9RopOhz2xuJPsodAr/NMJ9/599Q9/uXB2aAB4yFJieMRKRHynABopGgJqETo9ddfZ+rUqUyfPp2VK1fSv39/xowZQ0FBwWGfv2jRIi6//HIWLlzIkiVLyM7O5swzz2Tnzp0+jlxEPCo/RI/WaIxR083Mz5b5sHUh7FwJWz4DRziccqPd0fk3baH3HGvHmBIhz3rssce4/vrrueaaa+jVqxfPPfcccXFxvPTSS4d9/muvvcb//M//cMIJJ9CjRw9eeOEFnE4nCxYsOOI9KisrKSkpafAhIn7GXSit+qBDtO5idoUBzL+7rm9Qv0sgtbN9cQUCKxHat90cFirNs39v3b9RP98xBgGUCFVVVbFixQpGjx7tfiwsLIzRo0ezZMmSRl2joqKC6upqUlOPvG10xowZJCUluT+ys7NbHLuIeJj7jDHtGDusU/8fRCeankEbPwYcMHyq3VH5v7hUSMgyYx210Xy5y8zn1C4BUZgfMIlQYWEhtbW1ZGRkNHg8IyODvLy8Rl3jtttuIysrq0Ey9Wu33347xcXF7o/c3NwWxS0iHrZ/nzkcE7Q0diStWsOIeolP7/OhTTfbwgko7qM2tDzWbFb/oABYFoMASoRa6uGHH2bu3Lm8++67xMTEHPF50dHRJCYmNvgQET+ye4P5nNgOYlPsjcWfDf0TpHQy9UIjbrE7msChDtMtF0D1QQARdgfQWGlpaYSHh5Ofn9/g8fz8fDIzM4/62kcffZSHH36Yzz77jH79+nkzTBHxNnehtOqDjioyFq5bAAeKQ/dQ2ubQzrGWqamCnf5/0Gp9ATMjFBUVxaBBgxoUOluFz8OGHblr5SOPPMIDDzzAvHnzGDx4sC9CFRFvKtDRGo3WKk1JUFO5Z4R+BKfT3lgC0S/fQ80BiE2FtK52R9MoATMjBDB16lQmTJjA4MGDGTJkCE888QTl5eVcc801AFx11VW0a9eOGTNmAPC3v/2Ne+65hzlz5tCpUyd3LVF8fDzx8fG2vQ8RaQEVSos3tT4ewqOgqtTsHtNOu6bJrdc/KEB6fAVUInTppZeye/du7rnnHvLy8jjhhBOYN2+eu4A6JyeHsHon3D777LNUVVVx0UUXNbjO9OnTuffee30Zuoh4gstVd+q8lsbEG8IjoU13s+Muf50SoaayGikGwLZ5S0AlQgCTJ09m8uTDHxi4aNGiBr/etm2b9wMSEd8p/cXUvDjCzTcrEW/I6FuXCPX8rd3RBA6Xq15Haf8+aLW+gKkREhFxL4u1Ph4iou2NRYKXOkw3T9FPUFEI4dGQdYLd0TSaEiERCRwFOlpDfEBb6JvH6h+UNSCgflBRIiQigUOF0uIL1hb6op+gqtzeWAJJAB20Wp8SIREJHCqUFl+IbwOt0gEXFKy3O5rAoURIRMSLamtg9yYzzlAiJF6mOqGmKd8DezabcQDtGAMlQiISKIp+gtpKiGwFyZ3sjkaCXaY6TDeJdaxGWndzeG0AUSIk3lGyC966FjbOszsSCRbuZbEeEKb/usTLdNRG07gPWg2s2SAIwD5CEgBqq+GNCbBjKeR8B11/A2HhdkclgS5fR2uID9VfGnO5AqZLsm3cB60GTv8gi36sEs/77F6TBAGU7IAtC476dJFGcZ8xph1j4gNp3SAswjTwLN5hdzT+rfoA7FplxgFWHwRKhMTT1n8IS54243aDzOeVr9gXjwQPa4lChdLiCxHRJhkCLY8dy65VUFtldtqlHmd3NE2mREg8p+hneO9/zHjYZDjvH2a88b9QmmdfXBL4qsph7zYz1oyQ+Iq7Tkg7x47KfdDq0IBcQlQiJJ5RfQDenACVxdB+CIy+1xS1Zp8ErlpY9ardEUog270BcEGrNqbHi4gvqMN047gPWg2s/kEWJULiGZ/eCb98D7GpcPHL5gRngEETzOeV/wKn0774JLCpUFrsoJ1jx+Z0BnShNCgREk9Y8xYse8GML3gektrXfa3X+RCdBPu2w8+L7IhOgoEKpcUO1ozQns1m1lsOVbgJ9u+FiFho28/uaJpFiZC0TOFm+GCKGY+4BbqObvj1qDjod4kZr1DRtDSTCqXFDgmZZpbb5YTdOmrjsKz6oPaD61YCAowSIWm+qgp44yqoKoNOI2Dk7Yd/nrU8tuEjKNvtu/gkeGhGSOzgcKhO6FhyDi6LBeC2eYsSIWm+j28136BapcOFL0L4EfpzZvaFrIHgrIbv5/g2Rgl8ZbuhfDfgMAX4Ir6U2dd8ViJ0eO6O0oFZKA1KhKS5Vr0Kq18FRxhc9CIkZBz9+YOuNp9XvGK6tIo0ljUblNIJolrZGoqEIB2+emRlBbD3Z8AB7U+0O5pmUyIkTZe/Dj66xYxPvwM6n3rs1/S5EKLioWgrbPvKu/FJcHEvi6k+SGxgJUJ5a/VD3K9Z2+bTe0Fssq2htIQSIWmaylJTF1SzH7qMguE3N+510fEmGQJ1mpamUaG02KlNDzPzvb8IyvLtjsa/WIlQAC+LgRIhaQqXy+wQ27MFEtuZrfJNOQXcWh778X2oKPJKiBKENCMkdoqMhdbHm3GelscayFUiJKFm+Yuw9m1zEOFFL0Or1k17fdYAU3hYWwnfz/VOjBJcnE4o2GDGGdoxJjZRndChqipME10I6B1joERIGmvXKph3cHv86PvMmTJN5XDUzQqtVNG0NMK+7VBdDuFRkNrF7mgkVKnD9KF2rgBnDSRkQXIHu6NpESVCcmz798IbE8zpwt3PgWGTmn+tvhdDZJw5Oyp3qedilOBkLYuldT9yewYRb1MidKgAP2i1PiVCcnQuF7w3yfxkntwRzn+mZX/pY5Kg9+/MeMVsj4QoQcw6Y0yF0mIna2mscCPUVNkbi78I8INW61MiJEe35BnY+JFZmrjkFYhNafk1reWxde/C/n0tv54Er4KDP4GrUFrslNTenJnorDFna4U6Zy3kLjPjAC+UBmj0XHNJSUmjL5qYmNisYMTP5HwHn00347EzTLGzJ7Q/Edr0NGf3rHkThlzvmetK8HHPCKlQWmxkHbWR840pmM7sY3dE9ipYD5XFENmqbtkwgDU6EUpOTsbRyCWR2traZgckfqJ8D7x1jfkJqM+FMHii567tcJjzx+ZNM52mT7wu4NeYxQtqKk2rBtCMkNgvs09dIhTqrPqg7BODonav0e9g4cKF7vG2bduYNm0aV199NcOGDQNgyZIlvPLKK8yYMcPzUYpvOZ3wzvVQstP0zxj3pOcTlX6XwvzpkL8Gdq2EdoM8e30JfIWbwFVr6soSs+yORkKdDl+t4z5oNfCXxaAJidBpp53mHt9///089thjXH755e7Hzj33XPr27cusWbOYMGGCZ6MU3/pqJmxdABExcMm/IDrB8/eIS4Ve58GaN8yskBIh+bX8eo0UNWModtPOsTo59XaMBYFmFUsvWbKEwYMHH/L44MGDWbpUW6ID2s+LYeFDZnzOTO/WZgw6mDCvecsc3SFSnwqlxZ+06QE4zDEbZbvtjsY+xTuhOMccOxLAB63W16xEKDs7m+eff/6Qx1944QWys7NbHJTYpDQf3poILieccAUMuMK79+t4ill6qy43HatF6tPWefEn0fGQ2tmMQ7lOyKoPyujjndUCGzSryunxxx/nwgsv5L///S9Dh5qpsaVLl7J582beflvf0AKSsxbengjlBeYn8LP/7v17OhwwcALMv9ssj1nb6kWg3hlj2jEmfiKjNxT9ZJbHupxudzT2sOqDOgyzNw4PataM0Nlnn83mzZs599xzKSoqoqioiHHjxrFp0ybOPvtsT8covrDwIdj2JUTFm7qgqDjf3PeE30NYpCmY/uUH39xT/N/+faZYHyC9p62hiLhl9DWfQ7lOKDe46oOgGTNC1dXVjB07lueee44HH3zQGzGJr22eD18+asbjnoS0rr67d6s06Plb01xx5SumLkmkYL35nNgeYpNtDUXELdQPX60shbw1ZhwkO8agGTNCkZGR/PCDfnIPGsU74J0/mPHgidD3It/HMPBg0fQPb5gTjUXchdKaDRI/YiVCuzdAbY29sdhhx3JTQ5rUAZLa2R2NxzRraeyKK67gxRdf9HQs4mu11fDmNbC/CNqeYLpH26HzaeYcs8oSMzMkokJp8UfJHU35QG1VXbPPUBJk2+YtzSqWrqmp4aWXXuKzzz5j0KBBtGrVqsHXH3vsMY8EJ1722b2wY6k5Q+fi2RARbU8cYWFmK/2C+83y2IDx9sQh/kOF0uKPwsLMZpIdS83yWHoPuyPyLXd9UPAsi0EzE6G1a9cycOBAADZtangAXWOP4RCbrf8Aljxtxuf/o25bqF1OGA+fPwi535n6EC2JhC6Xqy4R0oyQ+JuM3nWJkB2lBHaprTFLYxBU9UHQzESo/nEbEoCKfob3JpnxsMmmWNluCZnQ/SzY8KHZSn/Ww3ZHJHYp2QUHisERDmnd7I5GpKHMEO0wnb8WqsogOjHoflBtVo2QBLDqA/DmBHNycPshMPpeuyOqY/UR+mGuiVNCkzUb1Pp4+5ZrRY4kVI/ayLXOFxsCYeH2xuJhzT42dvny5bzxxhvk5ORQVVXV4GvvvPNOiwMTL/n0Tvjle4hNhYtfhvBIuyOq0+UMSMqG4lyzdNfvYrsjEjtY32C0LCb+yJoNKdkJFUXm3MRQkLPEfA6yZTFo5ozQ3LlzOfnkk1m/fj3vvvsu1dXVrFu3js8//5ykpCRPxyiesuYtWPaCGV/wPCS1tzeeXwsLrzvWY8VsW0MRG6lQWvxZTBIkdzBj6+9qsHO56nWUViIEwEMPPcTjjz/OBx98QFRUFE8++SQbNmzgkksuoUOHDp6OUTxh9yZ4/wYzHnELdB1tbzxHMuAKc5jf9q+gMAS3p4q2zov/s5bH8kKksWJxLpTugrAIaDfI7mg8rlmJ0NatWznnnHMAiIqKory8HIfDwU033cSsWbM8GqB4QFWFqQuqLodOI2Dk7XZHdGRJ7eH435jxylfsjUV8r7YGCjeasU6dF3/lrhMKkUTI6h+U2c93xy/5ULMSoZSUFEpLSwFo164da9eavwz79u2jokKdgf3Ox7eaKdxW6XDhCxDe7NIw3xh0sNP06jlQU3X050pwKdpqmtVFtjLN60T8kfuojRApmHY3Ugyeg1bra1YidOqppzJ//nwALr74YqZMmcL111/P5ZdfzqhRozwaoLTQqldh9atmuemiF802dX/XdQzEZ0JFIWz8yO5oxJesbyzpPUzzOhF/ZM0IFawHZ629sfiCtWMsyDpKW5o1NfD0009z4IDZ3nznnXcSGRnJN998w4UXXshdd93l0QClBfLXwUe3mPHpd0DnU+2Np7HCI0yt0JePmp5CvX9nd0TiK+5CaS2LiR9L7QwRsVCz3/RlSzve7oi8Z/++uh9QgnDHGDQzEUpNrdsuGBYWxrRp0zwWkHhIZSm8cZX5h9plFAy/2e6ImmbglSYR+mmh+Y/G7s7X4hvuQmntGBM/FhZuttHvWgn5a4I7EdqxHHBBSmdIyLA7Gq9o1tzzVVddxcsvv8zWrVs9HY94gssFH0wxhwImZJmt8oG2zJDSyfQVAlj1b1tDER/SjJAEilCpE7L6BwXhtnlLs747RkVFMWPGDLp27Up2djZXXHEFL7zwAps3b/Z0fNIcy1+EtW+brY4Xz4ZWre2OqHkGHiyaXvUq1FbbG4t4X1U57N1mxpoREn+X2dd8DvZEKDd4+wdZmpUIvfDCC2zatInc3FweeeQR4uPjmTlzJj169KB9ez9r0hdqdq2CeQe3x4++L7CL27qfDXFpUJYPmz6xOxrxtoINgAtatYFWaXZHI3J07hmhIN5CX1sdtAet1tei9ZKUlBRat25NSkoKycnJRERE0KZNG0/FJk21fy+8McFsP+5+DgybZHdELRMRBQPGm7F6CgW/AmvHmJbFJABYf0/35ZhDgoPRLz+YOtOY5KA+ALlZidAdd9zBySefTOvWrZk2bRoHDhxg2rRp5OXlsWrVKk/HKI3hcpkT5fdtN/1Xzn8GHA67o2o5a3ls83zYl2tvLOJdKpSWQBKXContzLhgvb2xeEuu1T/opMCrM22CZu0ae/jhh2nTpg3Tp0/nggsuoFu34M0UA8aSZ0zPnfAoUxcUm2J3RJ7Ruovphr3tS1MrdLofd8WWltGMkASajN7m8NW8NcFZQ+M+aDWASywaoVkp3qpVq7jzzjtZunQpp5xyCu3ateP3v/89s2bNYtOmTZ6OsYFnnnmGTp06ERMTw9ChQ1m6dOlRn//mm2/So0cPYmJi6Nu3Lx9//LFX47NFznfw2XQzHvMQtBtobzyeNuhq83nVv0OjeVmosn6q1hljEiiCeedYg4NWg7OjtKVZiVD//v254YYbeOedd9i9ezcff/wxUVFRTJo0iZ49e3o6RrfXX3+dqVOnMn36dFauXEn//v0ZM2YMBQUFh33+N998w+WXX87EiRNZtWoV559/Pueff777SJCgUL4H3roGnDXQ50I48Tq7I/K8Hr81M1wlO2HLZ3ZHI95QthvKdwMOaNPD7mhEGsd95lgQJkJ7f4byArPKkDXA7mi8qlmJkMvlYuXKlTz22GOce+65nH766bz66qv07duXG264wdMxuj322GNcf/31XHPNNfTq1YvnnnuOuLg4XnrppcM+/8knn2Ts2LHceuut9OzZkwceeICBAwfy9NNPH/EelZWVlJSUNPjwW04nvHO9SRBaHw/jngyOuqBfi4yB/r834xUqmg5K1rJYSieIamVrKCKN5j5q40fz/3Ewsc4Xa3uC+T84iDUrEUpNTWXo0KHMmTOHrl278sorr1BYWMjKlSt5/PHHPR0jAFVVVaxYsYLRo0e7HwsLC2P06NEsWbLksK9ZsmRJg+cDjBkz5ojPB5gxYwZJSUnuj+zsbM+8AW/4aiZsXQARMXDJvyA6we6IvMc6iHXTPCj5xd5YxPNUKC2BqPXxZsakqsxsVAkmOfUKpYNcsxKhV199lT179rB8+XJmzpzJuHHjSE5O9nBoDRUWFlJbW0tGRsMW3xkZGeTl5R32NXl5eU16PsDtt99OcXGx+yM31093Kv28GBY+ZMbnzAz+byBtups+Fq5ac4isBBcVSksgCo+oW8oNtuUxJUJHd84555CYmMiWLVv45JNP2L9/P2CWzAJddHQ0iYmJDT78Tmk+vDURXE44Ybw5oDQUWEXTK/8dfNPQoc49I6RESAKMu04oiGpPK4qgcKMZB/mOMWhmIrRnzx5GjRpFt27dOPvss/nlF7NUMXHiRG6+2TuHe6alpREeHk5+fn6Dx/Pz88nMzDzsazIzM5v0/IBQWwNvTzRFbOm94OxH7Y7Id3qdB9FJZgr650V2RyOe4nTC7g1mnB7kM5sSfDKDMBHKPbgbu3XXkOjy3qxE6KabbiIyMpKcnBzi4uLcj1966aXMmzfPY8HVFxUVxaBBg1iwYIH7MafTyYIFCxg27PBb+4YNG9bg+QDz588/4vMDwqIZpqdOVLypC4qKO/ZrgkVUHPS7xIxXzLY1FPGgfdugugLCoyH1OLujEWmaYNxC7z5oNfhng6CZDRU//fRTPvnkk0POFevatSvbt3uvYGzq1KlMmDCBwYMHM2TIEJ544gnKy8u55pprALjqqqto164dM2bMAGDKlCmcdtppzJw5k3POOYe5c+eyfPlyZs2a5bUYvWrzfPjy4AzQuCchrau98dhh0ARY9jxs+NhsuY7XkS4Bz1oWa9PN1FyIBBJraazoZ6gsg+h4e+PxhNzQ6B9kadaMUHl5eYOZIEtRURHR0dEtDupILr30Uh599FHuueceTjjhBFavXs28efPcBdE5OTnuZTqAk08+mTlz5jBr1iz69+/PW2+9xXvvvUefPn28FqPXFO+Ad/5gxoMnQt+L7I3HLpl9od0gcFbD93PsjkY8oeBgIqRlMQlErdIgPgNw1S3xBrKaSti50oyD+KDV+pqVCI0YMYJ//etf7l87HA6cTiePPPIIp59+useCO5zJkyezfft2Kisr+e677xg6tG7qbtGiRcyePbvB8y+++GI2btxIZWUla9eu5eyzz/ZqfF5RWw1vXgP7i6Btf9M9OpRZ54+teMV0P5XAZi0pqFBaAlUwnUS/azXUVkJcmjniKAQ0ax7673//O2eccQbLly+nqqqK//f//h/r1q2jqKiIr7/+2tMxymf3wo6lplD44leCvrnVMfW5ED65A4q2wravoPMIuyOSlrCO1tCMkASqjN6w9XPIC4JEqP5Bq8HYoPcwmjwjVF1dzQ033MAHH3zA8OHDOe+88ygvL+eCCy5g1apVdOkSGhmkz6z/AJYc7IR9/j8gtbO98fiD6Pi6pcGV6jQd0GoqYc8WM9aMkASqjL7mczAUTFv9g0Jg27ylyTNCkZGR/PDDD6SkpHDnnXd6IyaxFP0M700y42GToedv7Y3HnwycYHaO/fh/cNYjEJdqd0TSHLs3miaZMUmQ0NbuaESap/7OMZcrcGdSXK56jRRDo1AamlkjdMUVV/Diiy96Ohapr/oAvDkBKouh/RAYfa/dEfmXrAGQ2Q9qq+D7uXZHI81Vv1A6UL95iKR1g7AI8/918Q67o2m+ws2mFjUixtSjhohm1QjV1NTw0ksv8dlnnzFo0CBatWp4SOJjjz3mkeBC2qd3wi/fQ2wqXPwyhEfaHZF/cTjMVvqPbjbLYyf9Wd9IA5EKpSUYRERBWndzVEz+Okj24zMqj8aqD2o3yLynENGsRGjt2rUMHDgQgE2bNjX4mkPfjFpuzVuw7AUzvuB5SGp/9OeHqr4Xw6d3my2rud+FxJk4Qcc9I6RESAJcRu+DidAa6D7W7miaJ+dg/6AQqg+CZiZCCxcu9HQcYtm9Cd6/wYxH3AJdR9sbjz+LSYLeF5hDWFe8okQoEFk7xoL90GAJfhm9YQ2BXTDt7igdWv+XNqtGSLykqsLUBVWXQ6cRMPJ2uyPyf4MO9hRa9y7s32drKNJE+/dCyU4ztk7wFglU7jPHAjQRKtttWpIAZA+xNxYfUyLkTz6+1SwVtEqHC1/QcQON0f5EaNMTavbDmjftjkaawpoNSmwPscm2hiLSYtZRG3u2QPV+e2NpDutYjTY9ITbF3lh8TImQv1j1qlnicYTBRS9CQqbdEQUGhwMGXW3G6jQdWFQoLcEkPgPiWoPLGZhHbYToshgoEfIP+evgo1vM+PQ7oPOp9sYTaPpdYk4uz18Du1baHY00lgqlJZg4HIF9Er37oFUlQuJrlaXwxlVmaafLKBh+s90RBZ64VOh1nhmvmG1rKNIEKpSWYGMtjwXaURvV+80ZYxByO8ZAiZC9XC74YIpZU07IMlvlw/RH0izW8tiat01yKf7N5YJ8zQhJkLESoUA7fHXnSnBWQ3wmpHSyOxqf03ddOy1/Eda+bTqSXjwbWrW2O6LA1fFkaN3V7Lhb+7bd0cixlOw0XXgd4ZDW1e5oRDzj10dtBAr3QatDQ7IxrRIhu+xcCfMObo8ffa/5CyjN53DAwKvMWMtj/s+aDUrrChHR9sYi4iltepgNL/uLoDTP7mgaLwTPF6tPiZAdaqvhrWvNOVndzzEHqkrLnfB7CIuEXavglx/sjkaOpuBgMamWxSSYRMaYmWkInIJpp7OuUDoE64NAiZA9wiPhnJnmPJfznwnJqUivaJUGPX9rxitfsTcWOTprRkhb5yXYuJfHAqROaPcGOFAMkXGQ2dfuaGyhRMgux4+C6xaEXOMqrxt4sNP0D29AVbm9sciRWTvG0rVjTIJMoCVCVn1Q+8Ehe7i3EiE7aSbI8zqfZnY9VJbAuvfsjkYOp7YaCjeacXpPe2MR8TRrViVQlsbcB62GXv8gixIhCS5hYSqa9nd7tpr6uMhWkNzR7mhEPMuaESrcBDWV9sbSGO6O0qFZHwRKhCQYnXCFaUmwY2ndEoz4D3ehdE/1zZLgk9gOYpLAWWOSIX9W8gvs2252urUPrYNW69P/QhJ8EjKg21gzXqGiab+jQmkJZg5HvcaKfr48ZtUHpfeGmER7Y7GREiEJTlan6e//A9UHbA1FfkWF0hLsrOWxvDX2xnEsOaF7vlh9SoQkOHU5A5Ky4cA+WP++3dFIfQU6dV6CXKAcvuruKK1ESCT4hIXDgCvNWMtj/qOyDPZuM2M1U5RglREAO8cqy+oaz4ZoI0WLEiEJXgOuMEWA27+Cws12RyNgmrcBtEo3DTBFglF6D8AB5QVQVmB3NIe3cwW4aiGxPSRn2x2NrZQISfBKagddzzRjdZr2D/laFpMQENUKUo8zY3+dFcqpd9BqiFMiJMHN6jS9ek5g9PQIdgUHd4ypUFqCnb/XCeWG9kGr9SkRkuDW9UxIaAsVe2DDR3ZHIwXaOi8hwr2F3g+P2nDWQu4yMw7x+iBQIiTBLjzC1AqBlsf8gdVDSEdrSLDL9ONEKH8dVJVCVELdzFUIUyIkwW/AlYADfloERT/bHU3oKiuAikLAAW2UCEmQsxKM3RvN+Xr+JNc6X+xEs8M2xCkRkuCX0hG6nG7GK/9lbyyhzKqVSO0MUXH2xiLibUkdzIxLbRXs2WJ3NA1Z54uF8EGr9SkRktBgdZpe/Zr//XQWKtyF0qoPkhAQFlZXC+dvBdPqKN2AEiEJDd3OglZtoCwfNn1idzShyV0orZoECRHunWN+VCe0LxdKdoAjHNoPtjsav6BESEJDRBSc8HszXjHb1lBCVr5mhCTEuM8c86NEyKoPyuxr+h2JEiEJIVZPoS2fmZ+KxHeczrqu0kqEJFT441EbOeof9GtKhCR0tO4CnUYALlj1qt3RhJa9P0N1BYRH13XcFQl2VpuI0l1QUWRvLBZ1lD6EEiEJLVbR9Kp/m6Zi4htWfVCb7qa3k0goiEmE5I5m7A+zQgdKoOBgHNox5qZESEJLj99CbCqU7DRLZOIb+SqUlhDl7jDtB4nQjmXgcprkLLGt3dH4DSVCEloiY6D/5Wa8Qp2mfUZb5yVU+dPOMfeymGaD6lMiJKFn0MGi6U3zoOQXe2MJFUqEJFT5UyKUq0TocJQISehp093smHDVwmoVTXtd9QHYs9WMddiqhJrMgzvHCtbbW5dYWw07Vpix6oMaUCIkocnaSr/yX2Zrt3hP4UaTdMYkQ4LqEiTEpHSCyDioOQBFP9kXR94aqC6HmCRo08O+OPyQEiEJTb3Og+gk2JcDPy20O5rgVr9Q2uGwNxYRXwsLr9tGb+fymPug1aHm+A9x0++GhKaoOOh/qRmvVNG0V1nbdVUfJKHKXSdk484x90Gr6h/0a0qEJHRZy2MbPoKyAntjCWYF681n1QdJqLK20Nt11IbLVe+gVXWU/jUlQhK6MvtAu0HgrIHVc+yOJnjpjDEJdXb3Etq3HcryICwS2g20JwY/pkRIQpvVaXrlv8xPTeJZ+/ea4wWgrk5CJNRYs6HFOXCg2Pf3t/oHZZ0AkbG+v7+fUyIkoa33BRAVD0VbYdtXdkcTfKzZoKRss1tFJBTFpkBiezO2/k34kpUIqT7osJQISWiLjoe+F5nxitm2hhKU1EhRxLCzsaI6Sh+VEiERa3ls/fv+c0J0sLASIRVKS6iza+fY/r2w++CGBTVSPCwlQiJZAyCzH9RWwfdz7Y4muKhQWsSwa0Yod5n5nNoF4tv49t4BQomQCNSdP7ZitoqmPcXlqts6r0RIQp111Eb+j77tZm/1D9Ky2BEpERIB6HuxaYNfuLGuA6u0TPEOqCyGsAhI62Z3NCL2Su0C4dHmmIt923x3X+v/MyVCRxQwiVBRURHjx48nMTGR5ORkJk6cSFlZ2VGf/5e//IXu3bsTGxtLhw4duOGGGygutmHrovi/mCSzgwxghTpNe4RVH9S6K0RE2RuLiN3CIyD94BlfvqoTqqmCnTpo9VgCJhEaP34869atY/78+Xz44YcsXryYP/zhD0d8/q5du9i1axePPvooa9euZfbs2cybN4+JEyf6MGoJKFbR9Lp3Yf8+OyMJDtZ/9iqUFjF83Vjxl+/NYa+xqZDW1Tf3DEARdgfQGOvXr2fevHksW7aMwYMHA/DUU09x9tln8+ijj5KVlXXIa/r06cPbb7/t/nWXLl148MEHueKKK6ipqSEiIiDeuvhS+8GmlqXgR1jzJgy53u6IApvqg0Qa8nXBdG69bfM68PiIAmJGaMmSJSQnJ7uTIIDRo0cTFhbGd981vp6juLiYxMTEoyZBlZWVlJSUNPiQEOFw1J0/pqLpllMPIZGGrETIV2eOqZFiowREIpSXl0d6enqDxyIiIkhNTSUvL69R1ygsLOSBBx446nIawIwZM0hKSnJ/ZGdnNztuCUD9LoGIGPMT286VdkcTuGqrYfdGM9bSmIhhLY3t/Rkqj1zj6hEuV71Gijpo9WhsTYSmTZuGw+E46seGDRtafJ+SkhLOOeccevXqxb333nvU595+++0UFxe7P3Jzc1t8fwkgcanQ6zwzXjnb1lAC2p4t4Kw2x5ckdbA7GhH/0CoN4jPN2Fo69pY9W6Gi0OxUyzrBu/cKcLYWytx8881cffXVR33OcccdR2ZmJgUFBQ0er6mpoaioiMzMzKO+vrS0lLFjx5KQkMC7775LZGTkUZ8fHR1NdHR0o+KXIDVwAvzwOqx5G8Y8BNEJdkcUeKxi0PSeEBYQE88ivpHR25wEn78Wsk/03n2s+qB2AyFC39OOxtZEqE2bNrRpc+xOl8OGDWPfvn2sWLGCQYMGAfD555/jdDoZOvTIa58lJSWMGTOG6Oho3n//fWJiYjwWuwSxjiebLd97NsOat2DwNXZHFHhUKC1yeBm9YesC7+8cU31QowXEj2o9e/Zk7NixXH/99SxdupSvv/6ayZMnc9lll7l3jO3cuZMePXqwdOlSwCRBZ555JuXl5bz44ouUlJSQl5dHXl4etbW1dr4d8XcOR12n6ZXqKdQs7jPGetsbh4i/cW+h93LBtA5abbSASIQAXnvtNXr06MGoUaM4++yzGT58OLNmzXJ/vbq6mo0bN1JRUQHAypUr+e6771izZg3HH388bdu2dX+o7keOqf/lEBYJu1aZXhzSNPWXxkSkTma9XkLe2plaXmhmtEEzQo0QMM10UlNTmTNnzhG/3qlTJ1z1/lKNHDmywa9FmqRVGvQcB+veMZ2mf/uY3REFjspS2LfdjNM1IyTSQOuu5oesyhIozoVkL2wmsI7VSOtuNoDIUQXMjJCIz1nLY2vehKpye2MJJAUHd3rGZ0Cr1vbGIuJvIqKgTXcz9ladkJbFmkSJkMiRdDoVUjqZn9zWvWt3NIGjwFoWU6G0yGF5u8O0DlptEiVCIkcSFlav07SKphvN2jGmQmmRw3MnQl6YEao+YGobQfVBjaRESORoThgPYRGwYynk/2h3NIFBhdIiR+fNozZ2rYLaKmiVDqnHef76QUiJkMjRJGRAt7FmrK30x+Zy6YwxkWPJ6Gs+F22FqgrPXtt90OpQHbTaSEqERI5l0MGGit/PNdPOcmRlBVCxB3BAmx52RyPin+LTIS4NXE7Y3fJjpBrQ+WJNpkRI5Fi6nA5J2XBgH6x/3+5o/JtVKJ16HETF2RuLiL9yOLxTJ+R01hVKZ6tQurGUCIkcS1g4DLjSjFfMtjUUv+culNaymMhRZdRrrOgphZtg/16IiIW2/Tx33SCnREikMQZcAY4w2P41FG62Oxr/ZRWUq5GiyNF5Ywu9VR/UfjCEH/2AcamjREikMZLaQdczzVhF00dWoB1jIo1SPxHy1CkIOdaymLbNN4USIZHGsnoKrZ4DNZX2xuKPnLV1XaXVQ0jk6Nr0AEe4Wcoq/cUz18xZYj6rULpJlAiJNFbXMyGhrdkVteEju6PxP3u3Qc1+iIhR/xKRY4mMgbSuZuyJOqHSfNj7M+CA7BNbfr0QokRIpLHCI0ytEGh57HCs/8zbdDcF5iJydJ6sE7Lqg9J7QUxSy68XQpQIiTTFgCsBB/y0CIp+sjsa/2LtGFOhtEjjeHILfY7OF2suJUIiTZHSEbqcYcYr/21vLP5GhdIiTePJLfS5OnG+uZQIiTTVIKto+jWorbY3Fn9ibZ1XDyGRxrESocJNLduAUVUBv3xvxkqEmkyJkEhTdTsLWrWBsnzYNM/uaPxD9X5zbhJoaUyksRKzICYZnDWwe2Pzr7NzhblGQpbpgi9NokRIpKkiosyp9AArVDQNmP/EXU6ITYGETLujEQkMDodnlsdydNBqSygREmmOgVeZz1s+01Z6qHfifG/9RyzSFJ7YOZarg1ZbQomQSHO07gKDJwIueGsi7Fhud0T2KlB9kEiztHTnmLMWcpeZsTpKN4sSIZHmOusROP43pongnEtDezu9+4wx7RgTaRL30lgzZ4QK1kNlMUS2qruWNIkSIZHmCo+Ai2dD2/5QUQivXgTle+yOyh4FOmxVpFnSewIOKN8NZQVNf721LJZ9ovk/SZpMiZBIS0THw+/fgKQOZtfUfy4zO6hCSUVR3VlJmhESaZqoOLPUDs2bFbIKpbO1bb65lAiJtFRCJox/07S137EU3rnerNuHCms2KKkDxCTaG4tIIGpJnZA6SreYEiERT0jvAZfNgfAoWP8BfHqX3RH5jnW0hgqlRZqnuVvoi3dCcQ44wqD9YM/HFSKUCIl4SqfhcP6zZvztP2DJP+yNx1fydbSGSIs0dwu9VR+U0QeiEzwbUwhRIiTiSX0vgtH3mvEnd8CP/2drOD6hQmmRlrESod0bm3Zsj3tZTP2DWkKJkIinnXJjXY+hd/5Q959VMHK5tDQm0lLJHSEqAWqroHBz41+Xs8R87qD+QS2hREjE0xwO02Oo21ioOWB2khVusTsq7yjOhcoSCIuA1l3tjkYkMDkcTS+YriytW0rTjrEWUSIk4g3hEXDRS5A1APYXwWsXQtluu6PyPKuRYlo3cwabiDRPU+uEdiw35/sldYCkdt6LKwQoERLxlqhWpsdQckfYuw3+cylUVdgdlWe564O0LCbSIk2dEap/0Kq0iBIhEW+KT4cr3jansu9cAW9fF1w9hgp0tIaIRzR1C737oFUti7WUEiERb0vrCpf9B8KjYeNHMG+aKTIOBtbSWIZ2jIm0iLXZoHSX6dZ+NLU1dQc9qz6oxZQIifhCx2FwwT8BByydBUuetjuilquthsJNZqylMZGWiU6AlE5mfKw6ofy1UFUG0UmajfUAJUIivtL7d3DmX83407tg7Tv2xtNShZvBWW22/SZ3sDsakcDX2OWx3IMtObJPhLBw78YUApQIifjSsEkw5I9m/O4fYfs39sbTEvXrgxwOe2MRCQaN3Tlm9Q/SsphHKBES8SWHA8bOgB6/Nc3T/nM57N5kd1TNo0JpEc9qzM4xl0sHrXqYEiERXwsLhwueh/YnwoF9B3sMFdgdVdOpUFrEs6ylsYL1piD6cIpzTUF1WAS0G+S72IKYEiERO0TFweVzIaUz7MuBOZdAVbndUTVNgXXYqgqlRTwipTNExpmO9EU/Hf45Vv+gtv3N/yPSYkqEROzSKs30GIprDbtWwVvXHvmnQH9TWWoSONCMkIinhIXV/WBxpDohKxFSfZDHKBESsVPrLnD56xARA5vmwX9vDYweQ9ZBq/GZEJdqbywiweRYdULqKO1xSoRE7JZ9Ilz4AuCA5S/BV4/bHdGxWYXSOnFexLOOtoV+/766f3uaEfIYJUIi/qDnOBj7sBkvuA9+eNPeeI4lX2eMiXjF0WaEdiwHXKaWKCHDp2EFMyVCIv7ipD/BsMlm/N6f4ecv7Y3naHTYqoh3WLOsxTlmBqg+q39Qh2E+DSnYKRES8Se/eQB6nWc6Ns8dX1eL409crrqfVrU0JuJZsSmQlG3G1g8cFqujtOqDPEqJkIg/CQuD380y6/+VxfDaxVDyi91RNVSWD/uLwBEGbXrYHY1I8Dnc8lhttQ5a9RIlQiL+JjIGLv8PtD7eNE+bc4nZru4vrJ9SU4+DyFh7YxEJRoc7auOXH6Bmv5kxSutmT1xBSomQiD+KS4Xxb0FcGuT9AG9ebX4i9Af5OlpDxKsONyOUa/UPGmpmjsVj9Lsp4q9SO8Pv34CIWNjyGXw01T96DLkLpdVIUcQr3FvofwSn04zdhdJaFvM0JUIi/qz9ILjoJVOPs/JfsPhRuyNSobSIt6V2gfBoqC6HvT83PGhV9UEep0RIxN/1OBvOesSMF/4VVv/HvlictbB7gxlrRkjEO8Ij6pae89eZc8fKCyA8CrIG2BtbEFIiJBIIhlwPp0wx4/cnw0+L7Imj6GdzIGRErFm6ExHvqN9h2to2nzXAbKYQj1IiJBIoRt0LfS4EZw28fuWRzyLyJqs+qE13CAv3/f1FQkX9nWM59QqlxeOUCIkEirAwOP9Z6HgKVJaYHkPFO30bgzpKi/hG/Z1j7oNWVR/kDUqERAJJRDRc9hqkdYeSnabH0IES391fhdIivmEtje39GQo3mrFmhLwiYBKhoqIixo8fT2JiIsnJyUycOJGysrJGvdblcnHWWWfhcDh47733vBuoiLfFpsD4NyE+w0ybv3GV73oMaUZIxDdatYaEtnW/bt0VWqXZF08QC5hEaPz48axbt4758+fz4YcfsnjxYv7whz806rVPPPEEDofDyxGK+FBKR/j96xDZCn5aCB9M8X6Poer9ZvcK1E3bi4j31P93pmUxrwmIRGj9+vXMmzePF154gaFDhzJ8+HCeeuop5s6dy65du4762tWrVzNz5kxeeuklH0Ur4iNZA+Di2eAIh9WvwaKHvXu/3RvB5YTYVDMbJSLepUTIJwIiEVqyZAnJyckMHjzY/djo0aMJCwvju+++O+LrKioq+P3vf88zzzxDZmZmo+5VWVlJSUlJgw8Rv9XtTDhnphl/8TCsetV796q/LKYZVhHvs+qEQI0UvSggEqG8vDzS09MbPBYREUFqaip5eXlHfN1NN93EySefzHnnndfoe82YMYOkpCT3R3Z2drPjFvGJwdfAiJvN+IMpsGWBd+6jQmkR32o3yHSVT8qG1l3sjiZo2ZoITZs2DYfDcdSPDRs2NOva77//Pp9//jlPPPFEk153++23U1xc7P7Izc1t1v1FfOqMu6HfpabH0BtXmZOqPU2F0iK+1boLTPgArnhHs7BeFGHnzW+++Wauvvrqoz7nuOOOIzMzk4KCggaP19TUUFRUdMQlr88//5ytW7eSnJzc4PELL7yQESNGsGjRosO+Ljo6mujo6Ma+BRH/4HDAuU9DyS7Y9qXZVn/dZ5DU3nP3sE6dV6G0iO90Gm53BEHP4XL5w3HWR7d+/Xp69erF8uXLGTRoEACffvopY8eOZceOHWRlZR3ymry8PAoLCxs81rdvX5588knGjRtH586NOx6gpKSEpKQkiouLSUxMbPmbEfGm/fvgpbGwez206QnXzoPY5JZft6IIHjn4b+b2HRCd0PJrioh4UWO/fwdEjVDPnj0ZO3Ys119/PUuXLuXrr79m8uTJXHbZZe4kaOfOnfTo0YOlS5cCkJmZSZ8+fRp8AHTo0KHRSZBIwIlNhiveMv1Hdq+H16+AmqqWX9daFkvuoCRIRIJKQCRCAK+99ho9evRg1KhRnH322QwfPpxZs2a5v15dXc3GjRupqKiwMUoRP5DUHn7/BkTFm2Wy9ye3vMdQvuqDRCQ42Voj1BSpqanMmTPniF/v1KkTx1rlC4BVQBHPaNsPLvmXOY/sh9fNrpNRdzf/egUHd4wpERKRIBMwM0Ii0kTHj4Jz/9eMv3wUlr/c/GupUFpEgpQSIZFgNuAKOG2aGX90M2z6tOnXcLmgYL0Za0ZIRIKMEiGRYDdyGpwwHly18ObVsGtV016/LweqSiEsElof75UQRUTsokRIJNg5HDDuSTjudKguhzmXwt7tjX+9NRuU1hUiorwTo4iITZQIiYSC8EhTPJ3RB8ryTRH1/r2Ne60KpUUkiCkREgkVMYkw/k1IbAeFG2HueKipPPbr3IXSSoREJPgoERIJJYlZJhmKToTtX8N7fwan8+ivcZ8xph1jIhJ8lAiJhJqM3nDpv03x89q3YcF9R35uTRUUbjr4Os0IiUjwUSIkEoqOGwnnPW3GXz8By144/PP2bDEn2kclmKaMIiJBRomQSKjqfxmcfpcZf3wrbPzvoc9xL4v1NLvPRESCjBIhkVB26i0w8CpwOeGta2HnioZfzz+4Y0zLYiISpJQIiYQyhwPOeQyOHw3VFabHUNHPdV9XobSIBDklQiKhLjwSLp4Nmf2gfDe8dhFUFJmvaeu8iAQ5JUIiAtEJZlt9UrYpkP7P5VC2G4pzzNfVTFFEgpQSIRExEjJh/FsQkwS538K/f2cej8+EuFR7YxMR8RIlQiJSJ70HXDYHwqMgf415TMtiIhLElAiJSEOdhsP5z9b9WstiIhLEIuwOQET8UN+DBdNLnoY+F9gdjYiI1ygREpHDG/oH8yEiEsS0NCYiIiIhS4mQiIiIhCwlQiIiIhKylAiJiIhIyFIiJCIiIiFLiZCIiIiELCVCIiIiErKUCImIiEjIUiIkIiIiIUuJkIiIiIQsJUIiIiISspQIiYiISMhSIiQiIiIhS4mQiIiIhKwIuwPwdy6XC4CSkhKbIxEREZHGsr5vW9/Hj0SJ0DGUlpYCkJ2dbXMkIiIi0lSlpaUkJSUd8esO17FSpRDndDrZtWsXCQkJOBwOj123pKSE7OxscnNzSUxM9Nh1/Umwv0e9v8AX7O8x2N8fBP971PtrPpfLRWlpKVlZWYSFHbkSSDNCxxAWFkb79u29dv3ExMSg/MtdX7C/R72/wBfs7zHY3x8E/3vU+2ueo80EWVQsLSIiIiFLiZCIiIiELCVCNomOjmb69OlER0fbHYrXBPt71PsLfMH+HoP9/UHwv0e9P+9TsbSIiIiELM0IiYiISMhSIiQiIiIhS4mQiIiIhCwlQiIiIhKylAjZ5JlnnqFTp07ExMQwdOhQli5dandIHrN48WLGjRtHVlYWDoeD9957z+6QPGrGjBmceOKJJCQkkJ6ezvnnn8/GjRvtDstjnn32Wfr16+ducDZs2DD++9//2h2W1zz88MM4HA5uvPFGu0PxmHvvvReHw9Hgo0ePHnaH5VE7d+7kiiuuoHXr1sTGxtK3b1+WL19ud1ge06lTp0P+DB0OB5MmTbI7NI+ora3l7rvvpnPnzsTGxtKlSxceeOCBY54L5g1KhGzw+uuvM3XqVKZPn87KlSvp378/Y8aMoaCgwO7QPKK8vJz+/fvzzDPP2B2KV3zxxRdMmjSJb7/9lvnz51NdXc2ZZ55JeXm53aF5RPv27Xn44YdZsWIFy5cv54wzzuC8885j3bp1dofmccuWLeOf//wn/fr1szsUj+vduze//PKL++Orr76yOySP2bt3L6eccgqRkZH897//5ccff2TmzJmkpKTYHZrHLFu2rMGf3/z58wG4+OKLbY7MM/72t7/x7LPP8vTTT7N+/Xr+9re/8cgjj/DUU0/5PhiX+NyQIUNckyZNcv+6trbWlZWV5ZoxY4aNUXkH4Hr33XftDsOrCgoKXIDriy++sDsUr0lJSXG98MILdofhUaWlpa6uXbu65s+f7zrttNNcU6ZMsTskj5k+fbqrf//+dofhNbfddptr+PDhdofhU1OmTHF16dLF5XQ67Q7FI8455xzXtdde2+CxCy64wDV+/Hifx6IZIR+rqqpixYoVjB492v1YWFgYo0ePZsmSJTZGJs1VXFwMQGpqqs2ReF5tbS1z586lvLycYcOG2R2OR02aNIlzzjmnwb/FYLJ582aysrI47rjjGD9+PDk5OXaH5DHvv/8+gwcP5uKLLyY9PZ0BAwbw/PPP2x2W11RVVfHqq69y7bXXevTwbzudfPLJLFiwgE2bNgHw/fff89VXX3HWWWf5PBYduupjhYWF1NbWkpGR0eDxjIwMNmzYYFNU0lxOp5Mbb7yRU045hT59+tgdjsesWbOGYcOGceDAAeLj43n33Xfp1auX3WF5zNy5c1m5ciXLli2zOxSvGDp0KLNnz6Z79+788ssv3HfffYwYMYK1a9eSkJBgd3gt9tNPP/Hss88ydepU7rjjDpYtW8YNN9xAVFQUEyZMsDs8j3vvvffYt28fV199td2heMy0adMoKSmhR48ehIeHU1tby4MPPsj48eN9HosSIZEWmDRpEmvXrg2q+guA7t27s3r1aoqLi3nrrbeYMGECX3zxRVAkQ7m5uUyZMoX58+cTExNjdzheUf+n6n79+jF06FA6duzIG2+8wcSJE22MzDOcTieDBw/moYceAmDAgAGsXbuW5557LigToRdffJGzzjqLrKwsu0PxmDfeeIPXXnuNOXPm0Lt3b1avXs2NN95IVlaWz/8MlQj5WFpaGuHh4eTn5zd4PD8/n8zMTJuikuaYPHkyH374IYsXL6Z9+/Z2h+NRUVFRHH/88QAMGjSIZcuW8eSTT/LPf/7T5shabsWKFRQUFDBw4ED3Y7W1tSxevJinn36ayspKwsPDbYzQ85KTk+nWrRtbtmyxOxSPaNu27SFJec+ePXn77bdtish7tm/fzmeffcY777xjdygedeuttzJt2jQuu+wyAPr27cv27duZMWOGzxMh1Qj5WFRUFIMGDWLBggXux5xOJwsWLAi6Goxg5XK5mDx5Mu+++y6ff/45nTt3tjskr3M6nVRWVtodhkeMGjWKNWvWsHr1avfH4MGDGT9+PKtXrw66JAigrKyMrVu30rZtW7tD8YhTTjnlkJYVmzZtomPHjjZF5D0vv/wy6enpnHPOOXaH4lEVFRWEhTVMQcLDw3E6nT6PRTNCNpg6dSoTJkxg8ODBDBkyhCeeeILy8nKuueYau0PziLKysgY/ef7888+sXr2a1NRUOnToYGNknjFp0iTmzJnD//3f/5GQkEBeXh4ASUlJxMbG2hxdy91+++2cddZZdOjQgdLSUubMmcOiRYv45JNP7A7NIxISEg6p52rVqhWtW7cOmjqvW265hXHjxtGxY0d27drF9OnTCQ8P5/LLL7c7NI+46aabOPnkk3nooYe45JJLWLp0KbNmzWLWrFl2h+ZRTqeTl19+mQkTJhAREVzfrseNG8eDDz5Ihw4d6N27N6tWreKxxx7j2muv9X0wPt+nJi6Xy+V66qmnXB06dHBFRUW5hgwZ4vr222/tDsljFi5c6AIO+ZgwYYLdoXnE4d4b4Hr55ZftDs0jrr32WlfHjh1dUVFRrjZt2rhGjRrl+vTTT+0Oy6uCbfv8pZde6mrbtq0rKirK1a5dO9ell17q2rJli91hedQHH3zg6tOnjys6OtrVo0cP16xZs+wOyeM++eQTF+DauHGj3aF4XElJiWvKlCmuDh06uGJiYlzHHXec684773RVVlb6PBaHy2VDG0cRERERP6AaIREREQlZSoREREQkZCkREhERkZClREhERERClhIhERERCVlKhERERCRkKRESERGRkKVESEREREKWEiER8UuLFi3C4XCwb98+u0MRkSCmREhE/MLIkSO58cYb3b8++eST+eWXX0hKSrItJiVjIsEvuE5xE5GgERUVRWZmpt1hiEiQ04yQiNju6quv5osvvuDJJ5/E4XDgcDiYPXt2g9mY2bNnk5yczIcffkj37t2Ji4vjoosuoqKigldeeYVOnTqRkpLCDTfcQG1trfvalZWV3HLLLbRr145WrVoxdOhQFi1a5P769u3bGTduHCkpKbRq1YrevXvz8ccfs23bNk4//XQAUlJScDgcXH311YA5FXzGjBl07tyZ2NhY+vfvz1tvveW+pjWT9NFHH9GvXz9iYmI46aSTWLt2rdd/L0WkaTQjJCK2e/LJJ9m0aRN9+vTh/vvvB2DdunWHPK+iooL//d//Ze7cuZSWlnLBBRfwu9/9juTkZD7++GN++uknLrzwQk455RQuvfRSACZPnsyPP/7I3LlzycrK4t1332Xs2LGsWbOGrl27MmnSJKqqqli8eDGtWrXixx9/JD4+nuzsbN5++20uvPBCNm7cSGJiIrGxsQDMmDGDV199leeee46uXbuyePFirrjiCtq0acNpp53mjvfWW2/lySefJDMzkzvuuINx48axadMmIiMjffC7KiKN4vPz7kVEDuO0005zTZkyxf3rhQsXugDX3r17XS6Xy/Xyyy+7ANeWLVvcz/njH//oiouLc5WWlrofGzNmjOuPf/yjy+VyubZv3+4KDw937dy5s8G9Ro0a5br99ttdLpfL1bdvX9e999572Jh+HYPL5XIdOHDAFRcX5/rmm28aPHfixImuyy+/vMHr5s6d6/76nj17XLGxsa7XX3+9kb8jIuILmhESkYARFxdHly5d3L/OyMigU6dOxMfHN3isoKAAgDVr1lBbW0u3bt0aXKeyspLWrVsDcMMNN/DnP/+ZTz/9lNGjR3PhhRfSr1+/I8awZcsWKioq+M1vftPg8aqqKgYMGNDgsWHDhrnHqampdO/enfXr1zfxXYuINykREpGA8eslJYfDcdjHnE4nAGVlZYSHh7NixQrCw8MbPM9Knq677jrGjBnDRx99xKeffsqMGTOYOXMmf/nLXw4bQ1lZGQAfffQR7dq1a/C16Ojo5r85EbGFEiER8QtRUVENipw9YcCAAdTW1lJQUMCIESOO+Lzs7Gz+9Kc/8ac//Ynbb7+d559/nr/85S9ERUUBNIirV69eREdHk5OT06Ae6HC+/fZbOnToAMDevXvZtGkTPXv29MA7ExFPUSIkIn6hU6dOfPfdd2zbto34+Hj3rE5LdOvWjfHjx3PVVVcxc+ZMBgwYwO7du1mwYAH9+vXjnHPO4cYbb+Sss86iW7du7N27l4ULF7qTlY4dO+JwOPjwww85++yziY2NJSEhgVtuuYWbbroJp9PJ8OHDKS4u5uuvvyYxMZEJEya473///ffTunVrMjIyuPPOO0lLS+P8889v8fsSEc/R9nkR8Qu33HIL4eHh9OrVizZt2pCTk+OR67788stcddVV3HzzzXTv3p3zzz+fZcuWuWdqamtrmTRpEj179mTs2LF069aNf/zjHwC0a9eO++67j2nTppGRkcHkyZMBeOCBB7j77ruZMWOG+3UfffQRnTt3bnDvhx9+mClTpjBo0CDy8vL44IMP3LNMIuIfHC6Xy2V3ECIiwWTRokWcfvrp7N27l+TkZLvDEZGj0IyQiIiIhCwlQiIiIhKytDQmIiIiIUszQiIiIhKylAiJiIhIyFIiJCIiIiFLiZCIiIiELCVCIiIiErKUCImIiEjIUiIkIiIiIUuJkIiIiISs/w8QSsjgPGUojgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# let's visualize the transition of reward\n",
    "obs, info = env.reset()\n",
    "done = False\n",
    "reward_list = []\n",
    "\n",
    "while not done:\n",
    "    action = agent.sample_action_online(obs)\n",
    "\n",
    "    obs, reward, done, truncated, info = env.step(action)\n",
    "    reward_list.append(reward)\n",
    "\n",
    "# plot\n",
    "fig = plt.figure()\n",
    "ax1 = fig.add_subplot(111)\n",
    "ax1.plot(reward_list[:-1], label='reward', color='tab:orange')\n",
    "ax1.set_xlabel('timestep')\n",
    "ax1.set_ylabel('reward')\n",
    "ax1.legend(loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For more about the environmental configuration , please refer to [examples/quickstart/rec/rec_synthetic_customize_env.ipynb](https://github.com/hakuhodo-technologies/scope-rl/blob/main/examples/quickstart/rec/rec_synthetic_customize_env.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Discrete Action Case\n",
    "Here, we present how to collect logged data by a behavior policy in the case of discrete action.\n",
    "\n",
    "The procedure requires two steps:\n",
    "\n",
    "1. Learn a base deterministic policy\n",
    "2. Convert the deterministic policy into a stochastic policy.\n",
    "\n",
    "Below, we first learn a deterministic policy using [d3rlpy](https://github.com/takuseno/d3rlpy)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# standardized environment for discrete action\n",
    "env = RECEnv(random_state=random_state)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# for api compatibility to d3rlpy\n",
    "from scope_rl.utils import OldGymAPIWrapper\n",
    "env_ = OldGymAPIWrapper(env)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# learn a base deterministic policy for data collection\n",
    "from d3rlpy.algos import DoubleDQN\n",
    "from d3rlpy.models.encoders import VectorEncoderFactory\n",
    "from d3rlpy.models.q_functions import MeanQFunctionFactory\n",
    "from d3rlpy.online.buffers import ReplayBuffer\n",
    "from d3rlpy.online.explorers import LinearDecayEpsilonGreedy\n",
    "\n",
    "# model\n",
    "ddqn = DoubleDQN(\n",
    "    encoder_factory=VectorEncoderFactory(hidden_units=[30, 30]),\n",
    "    q_func_factory=MeanQFunctionFactory(),\n",
    "    target_update_interval=100,\n",
    "    use_gpu=torch.cuda.is_available(),\n",
    ")\n",
    "# replay buffer\n",
    "buffer = ReplayBuffer(\n",
    "    maxlen=10000,\n",
    "    env=env_,\n",
    ")\n",
    "# explorers\n",
    "explorer = LinearDecayEpsilonGreedy(\n",
    "    start_epsilon=1.0,\n",
    "    end_epsilon=0.1,\n",
    "    duration=1000,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# start training\n",
    "# skip if there is a pre-trained model\n",
    "ddqn.fit_online(\n",
    "    env_,\n",
    "    buffer,\n",
    "    explorer=explorer,\n",
    "    eval_env=env_,\n",
    "    n_steps=100000,\n",
    "    n_steps_per_epoch=1000,\n",
    "    update_start_step=1000,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save model\n",
    "ddqn.save_model(\"d3rlpy_logs/ddqn.pt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2023-06-01 17:47:48 [warning  ] Parameters will be reinitialized.\n"
     ]
    }
   ],
   "source": [
    "# reload model\n",
    "ddqn.build_with_env(env)\n",
    "ddqn.load_model(\"d3rlpy_logs/ddqn.pt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1. Epsilon-Greedy behavior policy\n",
    "\n",
    "Let's now convert the deterministic policy (i.e., ddqn policy) into a stochastic behavior policy.\n",
    "\n",
    "We use epsilon-greedy policy to collect logged data using `EpsilonGreedyHead`, where the behavior policy greedily takes an action chosen by the deterministic policy with probability $1 - \\epsilon$ and takes an action randomly with probability $\\epsilon$.\n",
    "\n",
    "Note that, `SyntheticDataset` has the following arguments:\n",
    "- `env`: REC environment for RL defined in the previous section.\n",
    "- `max_episode_steps`: Maximum number of timesteps in an episode."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# convert the base ddqn policy into a stochastic data collection policy\n",
    "from scope_rl.policy import EpsilonGreedyHead\n",
    "\n",
    "behavior_policy = EpsilonGreedyHead(\n",
    "    ddqn, \n",
    "    n_actions=env.action_space.n,\n",
    "    epsilon=0.3,  # probability of taking random action\n",
    "    name=\"ddqn_epsilon_0.3\",\n",
    "    random_state=random_state,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f07a4e76becd4b54a6c0655a523c5ee5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "[obtain_trajectories]:   0%|          | 0/10000 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# collect data\n",
    "dataset = SyntheticDataset(\n",
    "    env=env,\n",
    "    max_episode_steps=env.step_per_episode,\n",
    ")\n",
    "logged_dataset = dataset.obtain_episodes(\n",
    "    behavior_policies=behavior_policy,\n",
    "    n_trajectories=10000, \n",
    "    obtain_info=False,\n",
    "    random_state=random_state,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'size': 100000,\n",
       " 'n_trajectories': 10000,\n",
       " 'step_per_trajectory': 10,\n",
       " 'action_type': 'discrete',\n",
       " 'n_actions': 5,\n",
       " 'action_dim': None,\n",
       " 'action_meaning': None,\n",
       " 'action_keys': None,\n",
       " 'state_dim': 5,\n",
       " 'state_keys': None,\n",
       " 'state': array([[ 0.04479014,  0.11492637, -0.05131316, -0.0280654 ,  0.0347718 ],\n",
       "        [ 0.32416393,  0.77002748, -0.45581875, -0.29004355,  0.10039637],\n",
       "        [ 0.35804392,  0.78301367, -0.37393551, -0.30058144,  0.16887029],\n",
       "        ...,\n",
       "        [-0.18222542, -0.25948905,  0.87223293,  0.22615153,  0.29584546],\n",
       "        [-0.15421975, -0.20402914,  0.90884118,  0.27645134,  0.17936207],\n",
       "        [-0.16927321, -0.20199777,  0.82939778,  0.37230377,  0.32254101]]),\n",
       " 'action': array([4, 1, 3, ..., 2, 4, 2]),\n",
       " 'reward': array([-0.04291988, -0.20643552,  0.26225801, ..., -0.17007192,\n",
       "         0.67274739, -0.20491774]),\n",
       " 'done': array([0., 0., 0., ..., 0., 0., 1.]),\n",
       " 'terminal': array([0., 0., 0., ..., 0., 0., 1.]),\n",
       " 'info': {},\n",
       " 'pscore': array([0.76, 0.06, 0.76, ..., 0.06, 0.76, 0.06]),\n",
       " 'behavior_policy': 'ddqn_epsilon_0.3',\n",
       " 'dataset_id': 0}"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "logged_dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2. Softmax behavior policy\n",
    "We can also use `SoftmaxHead` to derive a stochastic behavior policy.\n",
    "\n",
    "This algorithm uses Q function of the original algorithm, which estimates the value of a given context and action pair (i.e., $(s, a)$) as $Q(s, a)$. \\\n",
    "Specifically, the behavior policy chooses actions stochastically as $\\pi(a \\mid s) = \\frac{\\exp(Q(s, a) / \\tau)}{\\sum_{a' \\in A} \\exp(Q(s, a') / \\tau)}$, where $A$ indicates the set discrete actions and $\\tau$ is an inverse temperature parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# convert base ddqn policy into a stochastic data collection policy\n",
    "from scope_rl.policy import SoftmaxHead\n",
    "\n",
    "behavior_policy = SoftmaxHead(\n",
    "    ddqn, \n",
    "    n_actions=env.action_space.n,\n",
    "    tau=1.0,  # temperature parameter\n",
    "    name=\"ddqn_softmax_tau_1.0\",\n",
    "    random_state=random_state,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d5e07822c57e4e39b72ab818769de5c5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "[obtain_trajectories]:   0%|          | 0/10000 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# collect data\n",
    "dataset = SyntheticDataset(\n",
    "    env=env,\n",
    "    max_episode_steps=env.step_per_episode,\n",
    ")\n",
    "logged_dataset = dataset.obtain_episodes(\n",
    "    behavior_policies=behavior_policy, \n",
    "    n_trajectories=10000, \n",
    "    obtain_info=False,\n",
    "    random_state=random_state,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'size': 100000,\n",
       " 'n_trajectories': 10000,\n",
       " 'step_per_trajectory': 10,\n",
       " 'action_type': 'discrete',\n",
       " 'n_actions': 5,\n",
       " 'action_dim': None,\n",
       " 'action_meaning': None,\n",
       " 'action_keys': None,\n",
       " 'state_dim': 5,\n",
       " 'state_keys': None,\n",
       " 'state': array([[ 0.04479014,  0.11492637, -0.05131316, -0.0280654 ,  0.0347718 ],\n",
       "        [ 0.25961711,  0.85642393, -0.29858954, -0.12037568,  0.30901616],\n",
       "        [ 0.19436985,  0.88078279, -0.22501215, -0.03685871,  0.36667834],\n",
       "        ...,\n",
       "        [-0.18222542, -0.25948905,  0.87223293,  0.22615153,  0.29584546],\n",
       "        [-0.18952932, -0.20353088,  0.86896687,  0.24622942,  0.32698849],\n",
       "        [-0.19081698, -0.19901956,  0.78968207,  0.35141699,  0.42058106]]),\n",
       " 'action': array([3, 3, 3, ..., 3, 4, 1]),\n",
       " 'reward': array([ 0.0457314 ,  0.38802573,  0.4453292 , ...,  0.12156692,\n",
       "         0.71189389, -0.2585296 ]),\n",
       " 'done': array([0., 0., 0., ..., 0., 0., 1.]),\n",
       " 'terminal': array([0., 0., 0., ..., 0., 0., 1.]),\n",
       " 'info': {},\n",
       " 'pscore': array([0.19989392, 0.9621771 , 0.91416824, ..., 0.24196722, 0.71648014,\n",
       "        0.0302837 ]),\n",
       " 'behavior_policy': 'ddqn_softmax_tau_1.0',\n",
       " 'dataset_id': 0}"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "logged_dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Collecting Logged Datasets with Multiple Behavior Policies and Random Seeds\n",
    "Finally, we show how to collect logged data by several behavior policies and random seeds.\n",
    "\n",
    "<!-- We show the example in the discrete action case, but the continuous action case can also be handled in a simmilar manner. -->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# standardized environment for discrete action\n",
    "env = gym.make(\"RECEnv-v0\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define several behavior policies\n",
    "behavior_policy_01 = EpsilonGreedyHead(\n",
    "    ddqn, \n",
    "    n_actions=env.action_space.n,\n",
    "    epsilon=0.1,  # probability of taking random action\n",
    "    name=\"ddqn_eps_0.1\",\n",
    "    random_state=random_state,\n",
    ")\n",
    "behavior_policy_03 = EpsilonGreedyHead(\n",
    "    ddqn, \n",
    "    n_actions=env.action_space.n,\n",
    "    epsilon=0.3,  # probability of taking random action\n",
    "    name=\"ddqn_eps_0.3\",\n",
    "    random_state=random_state,\n",
    ")\n",
    "behavior_policy_05 = EpsilonGreedyHead(\n",
    "    ddqn, \n",
    "    n_actions=env.action_space.n,\n",
    "    epsilon=0.5,  # probability of taking random action\n",
    "    name=\"ddqn_eps_0.5\",\n",
    "    random_state=random_state,\n",
    ")\n",
    "behavior_policy_07 = EpsilonGreedyHead(\n",
    "    ddqn, \n",
    "    n_actions=env.action_space.n,\n",
    "    epsilon=0.7,  # probability of taking random action\n",
    "    name=\"ddqn_eps_0.7\",\n",
    "    random_state=random_state,\n",
    ")\n",
    "behavior_policies = [behavior_policy_01, behavior_policy_03, behavior_policy_05, behavior_policy_07]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# initialize the dataset class\n",
    "dataset = SyntheticDataset(\n",
    "    env=env,\n",
    "    max_episode_steps=env.step_per_episode,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "logged_datasets = dataset.obtain_episodes(\n",
    "    behavior_policies=behavior_policies,\n",
    "    n_datasets=2,  # number of random states\n",
    "    n_trajectories=10000, \n",
    "    random_state=random_state,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'size': 100000,\n",
       " 'n_trajectories': 10000,\n",
       " 'step_per_trajectory': 10,\n",
       " 'action_type': 'discrete',\n",
       " 'n_actions': 5,\n",
       " 'action_dim': None,\n",
       " 'action_meaning': None,\n",
       " 'action_keys': None,\n",
       " 'state_dim': 5,\n",
       " 'state_keys': None,\n",
       " 'state': array([[ 0.04479014,  0.11492637, -0.05131316, -0.0280654 ,  0.0347718 ],\n",
       "        [ 0.32416393,  0.77002748, -0.45581875, -0.29004355,  0.10039637],\n",
       "        [ 0.28630736,  0.82019181, -0.40842954, -0.22933135,  0.16095323],\n",
       "        ...,\n",
       "        [-0.19559496, -0.25001936,  0.80463241,  0.3296956 ,  0.37828617],\n",
       "        [-0.1926843 , -0.2263963 ,  0.73783144,  0.40405561,  0.45162076],\n",
       "        [-0.1891678 , -0.20919712,  0.68836794,  0.44872563,  0.49522419]]),\n",
       " 'action': array([4, 3, 3, ..., 4, 4, 4]),\n",
       " 'reward': array([-0.04291988,  0.23319521,  0.29308288, ...,  0.74563016,\n",
       "         0.77307787,  0.78427577]),\n",
       " 'done': array([0., 0., 0., ..., 0., 0., 1.]),\n",
       " 'terminal': array([0., 0., 0., ..., 0., 0., 1.]),\n",
       " 'info': {},\n",
       " 'pscore': array([0.92, 0.92, 0.92, ..., 0.92, 0.92, 0.92]),\n",
       " 'behavior_policy': 'ddqn_eps_0.1',\n",
       " 'dataset_id': 0}"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "logged_dataset = logged_datasets.get(behavior_policy_name=behavior_policies[0].name, dataset_id=0)\n",
    "logged_dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For offline RL and OPE procedures, please refer to [examples/quickstart/rec/rec_synthetic_discrete_basic.ipynb](https://github.com/hakuhodo-technologies/scope-rl/blob/main/examples/quickstart/rec/rec_synthetic_discrete_basic.ipynb).\n",
    "\n",
    "For more advanced topic in OPE and OPS, please refer to [examples/quickstart/rec/rec_synthetic_discrete_advanced.ipynb](https://github.com/hakuhodo-technologies/scope-rl/blob/main/examples/quickstart/rec/rec_synthetic_discrete_advanced.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reference\n",
    "- Sarah Dean and Jamie Morgenstern. \\\n",
    "\"Preference Dynamics Under Personalized Recommendations.\", 2022.\n",
    "\n",
    "- Yuta Saito, Shunsuke Aihara, Megumi Matsutani, and Yusuke Narita. \\\n",
    "\"Open Bandit Dataset and Pipeline: Towards Realistic and Reproducible Off-Policy Evaluation.\", 2021.\n",
    "\n",
    "- Takuma Seno and Michita Imai. \\\n",
    "\"d3rlpy: An Offline Deep Reinforcement Library.\", 2021.\n",
    "\n",
    "- Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. \\\n",
    "\"Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor.\" 2018.\n",
    "\n",
    "- Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. \\\n",
    "\"OpenAI Gym.\", 2016.\n",
    "\n",
    "- Hado van Hasselt, Arthur Guez, and David Silver. \\\n",
    "\"Deep Reinforcement Learning with Double Q-learning.\", 2015."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  },
  "vscode": {
   "interpreter": {
    "hash": "70404ee114725fce8ed9e697d67827f8546c678889944e6d695790702cbfe1f5"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}