fernazHW6v5.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Zachary-Fernandes/CSCI-6962/blob/main/fernazHW6v5.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HXZog3W-zdgW"
      },
      "source": [
        "# Homework 6\n",
        "#####Zachary Fernandes\n",
        "#####Projects in Machine Learning and AI"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Markov Decision Processes (MDP)\n",
        "\n",
        "As demonstrated by [this article](https://towardsdatascience.com/real-world-applications-of-markov-decision-process-mdp-a39685546026), we can formulate a game show as an MDP. One round in this game involves asking a participant a question. If they answer correctly, they earn money, but if they answer incorrectly, they lose all their earned money. Answering correctly also means the participant can either choose to continue to the next round or quit and keep all their money. As rounds increase in number, their rewards also grow, but they become more and more challenging. For example, our game could last fifteen rounds\n",
        "\n",
        "The state space would include each of the rounds along with an end game state.\n",
        "\n",
        "The action space would be the choices a player can make, those being playing the next round or quitting.\n",
        "\n",
        "The transition model would focus on what actions the player takes. If they play the next round, they may win or lose that round. Winning means moving to the next round with some probability _p_; the reward the player earns is how much is offered for the current round. If the player wins the last round, they earn the final prize and move to the end game state as no rounds remain. Losing means losing all their money and moving to the end game state with 1 - _p_ probability. If the player chooses to quit, they move to the end game state without any further reward and probability 1.\n",
        "\n",
        "The reward is how much money a player earns for a correct answer. This reward increases as the round number increases. Additionally, as the reward value and round number increase, the probability of answering correctly decreases."
      ],
      "metadata": {
        "id": "B670T7WxoolH"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Reinforcement Learning (RL) in Healthcare\n",
        "\n",
        "One issue with healthcare that can be more effectively solved by RL is creating and configuring dynamic treatment regimes (DTRs). A DTR is, as put by [CapeStart](https://www.capestart.com/resources/blog/reinforcement-learning-in-health-care-why-its-important-and-how-it-can-help/), a sequence of rules that determine healthcare decisions, such as types of treatment, dosages of drugs, and timing of appointments. These are tailored to a specific patient based on medical history and conditions. For an RL algorithm, clinical observations and patient assessments would be its input data, and treatment options would be its output. This is done to reach the patient's most desired environmental state.\n",
        "\n",
        "RL can help automate decision-making while treatment regimes proceed, design DTRs for chronic conditions, and improve critical care from intensive care data. One open-source approach to this problem is [pydtr](https://github.com/fullflu/pydtr). pydtr is a Python library that conducts DTRs, and it can select optimal treatments for specific patients with sklearn-based interfaces. It conducts these DTRs through a regression version of Iterative Q-Learning. It is possible to use sklearn-based models or statsmodels-based models - if one is using the sklearn model for a regression function and there are categorical independent variables, one must encode the categorical variables prior to using the model."
      ],
      "metadata": {
        "id": "0MpJA-hBopCC"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CfwU6Ho80N9i"
      },
      "source": [
        "### Tic-Tac-Toe\n",
        "\n",
        "This notebook can be seen as an extension of the previous notebook that features two RL agents training and testing against each other. This was an idea inspired by some articles I read while looking into this problem. Additionally, this notebook experiments with using punishments alongside rewards to discourage specific behaviors (draws and losses - the goal is to win).\n",
        "\n",
        "Online Resources that Helped:\n",
        "\n",
        "[Framing Tic-Tac-Toe as a Reinforcement Learning Problem](https://levelup.gitconnected.com/framing-tic-tac-toe-as-a-reinforcement-learning-problem-eb76b6ece4de)\n",
        "\n",
        "[Setting up Tic-Tac-Toe for Reinforcement Learning in Python](https://levelup.gitconnected.com/setting-up-tic-tac-toe-for-reinforcement-learning-in-python-43e2f42cfce8)\n",
        "\n",
        "[Tabular Q-Learning Agent vs. Irrational Agent in the Game of Tic-Tac-Toe](https://levelup.gitconnected.com/tabular-q-learning-agent-vs-irrational-agent-in-the-game-of-tic-tac-toe-6de6c85f0c42)\n",
        "\n",
        "[Reinforcement Learning — Implement TicTacToe](https://towardsdatascience.com/reinforcement-learning-implement-tictactoe-189582bea542)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MoRfsvI_CwMB",
        "outputId": "26fbe37c-56de-4fad-8d4e-401ca354ec60"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Requirement already satisfied: imbalanced-learn in /usr/local/lib/python3.7/dist-packages (0.8.1)\n",
            "Requirement already satisfied: numpy>=1.13.3 in /usr/local/lib/python3.7/dist-packages (from imbalanced-learn) (1.21.6)\n",
            "Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from imbalanced-learn) (1.2.0)\n",
            "Requirement already satisfied: scikit-learn>=0.24 in /usr/local/lib/python3.7/dist-packages (from imbalanced-learn) (1.0.2)\n",
            "Requirement already satisfied: scipy>=0.19.1 in /usr/local/lib/python3.7/dist-packages (from imbalanced-learn) (1.7.3)\n",
            "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn>=0.24->imbalanced-learn) (3.1.0)\n",
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Requirement already satisfied: graphviz in /usr/local/lib/python3.7/dist-packages (0.10.1)\n",
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Requirement already satisfied: tensorflow-probability in /usr/local/lib/python3.7/dist-packages (0.17.0)\n",
            "Requirement already satisfied: decorator in /usr/local/lib/python3.7/dist-packages (from tensorflow-probability) (4.4.2)\n",
            "Requirement already satisfied: dm-tree in /usr/local/lib/python3.7/dist-packages (from tensorflow-probability) (0.1.7)\n",
            "Requirement already satisfied: gast>=0.3.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow-probability) (0.4.0)\n",
            "Requirement already satisfied: numpy>=1.13.3 in /usr/local/lib/python3.7/dist-packages (from tensorflow-probability) (1.21.6)\n",
            "Requirement already satisfied: absl-py in /usr/local/lib/python3.7/dist-packages (from tensorflow-probability) (1.3.0)\n",
            "Requirement already satisfied: six>=1.10.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow-probability) (1.15.0)\n",
            "Requirement already satisfied: cloudpickle>=1.3 in /usr/local/lib/python3.7/dist-packages (from tensorflow-probability) (1.5.0)\n",
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Requirement already satisfied: imageio in /usr/local/lib/python3.7/dist-packages (2.9.0)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from imageio) (1.21.6)\n",
            "Requirement already satisfied: pillow in /usr/local/lib/python3.7/dist-packages (from imageio) (7.1.2)\n",
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting git+https://github.com/tensorflow/docs\n",
            "  Cloning https://github.com/tensorflow/docs to /tmp/pip-req-build-5_hja8rf\n",
            "  Running command git clone -q https://github.com/tensorflow/docs /tmp/pip-req-build-5_hja8rf\n",
            "Requirement already satisfied: astor in /usr/local/lib/python3.7/dist-packages (from tensorflow-docs==0.0.0.dev0) (0.8.1)\n",
            "Requirement already satisfied: absl-py in /usr/local/lib/python3.7/dist-packages (from tensorflow-docs==0.0.0.dev0) (1.3.0)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.7/dist-packages (from tensorflow-docs==0.0.0.dev0) (2.11.3)\n",
            "Requirement already satisfied: nbformat in /usr/local/lib/python3.7/dist-packages (from tensorflow-docs==0.0.0.dev0) (5.7.0)\n",
            "Requirement already satisfied: protobuf<3.20,>=3.12.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow-docs==0.0.0.dev0) (3.19.6)\n",
            "Requirement already satisfied: pyyaml in /usr/local/lib/python3.7/dist-packages (from tensorflow-docs==0.0.0.dev0) (6.0)\n",
            "Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.7/dist-packages (from jinja2->tensorflow-docs==0.0.0.dev0) (2.0.1)\n",
            "Requirement already satisfied: jupyter-core in /usr/local/lib/python3.7/dist-packages (from nbformat->tensorflow-docs==0.0.0.dev0) (4.11.2)\n",
            "Requirement already satisfied: importlib-metadata>=3.6 in /usr/local/lib/python3.7/dist-packages (from nbformat->tensorflow-docs==0.0.0.dev0) (4.13.0)\n",
            "Requirement already satisfied: jsonschema>=2.6 in /usr/local/lib/python3.7/dist-packages (from nbformat->tensorflow-docs==0.0.0.dev0) (4.3.3)\n",
            "Requirement already satisfied: traitlets>=5.1 in /usr/local/lib/python3.7/dist-packages (from nbformat->tensorflow-docs==0.0.0.dev0) (5.1.1)\n",
            "Requirement already satisfied: fastjsonschema in /usr/local/lib/python3.7/dist-packages (from nbformat->tensorflow-docs==0.0.0.dev0) (2.16.2)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=3.6->nbformat->tensorflow-docs==0.0.0.dev0) (3.10.0)\n",
            "Requirement already satisfied: typing-extensions>=3.6.4 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=3.6->nbformat->tensorflow-docs==0.0.0.dev0) (4.1.1)\n",
            "Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.7/dist-packages (from jsonschema>=2.6->nbformat->tensorflow-docs==0.0.0.dev0) (22.1.0)\n",
            "Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.7/dist-packages (from jsonschema>=2.6->nbformat->tensorflow-docs==0.0.0.dev0) (0.19.2)\n",
            "Requirement already satisfied: importlib-resources>=1.4.0 in /usr/local/lib/python3.7/dist-packages (from jsonschema>=2.6->nbformat->tensorflow-docs==0.0.0.dev0) (5.10.0)\n",
            "Building wheels for collected packages: tensorflow-docs\n",
            "  Building wheel for tensorflow-docs (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for tensorflow-docs: filename=tensorflow_docs-0.0.0.dev0-py3-none-any.whl size=184426 sha256=3b9a12bf2984081c53a75aabc134124ba09d48c5068e2abe16b3239784515424\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-uddi1iug/wheels/cc/c4/d8/5341e93b6376c5c929c49469fce21155eb69cef1a4da4ce32c\n",
            "Successfully built tensorflow-docs\n",
            "Installing collected packages: tensorflow-docs\n",
            "Successfully installed tensorflow-docs-0.0.0.dev0\n"
          ]
        }
      ],
      "source": [
        "!pip install imbalanced-learn\n",
        "!pip install graphviz\n",
        "!pip install tensorflow-probability\n",
        "\n",
        "# to generate gifs\n",
        "!pip install imageio\n",
        "!pip install git+https://github.com/tensorflow/docs"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "aZCA63injOt2"
      },
      "outputs": [],
      "source": [
        "# import required libraries\n",
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "import seaborn as sns\n",
        "import numpy as np\n",
        "import random\n",
        "import graphviz\n",
        "\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.model_selection import cross_val_score\n",
        "from sklearn.model_selection import KFold \n",
        "from sklearn.linear_model import LogisticRegression\n",
        "from sklearn.svm import SVC\n",
        "from sklearn import tree\n",
        "from sklearn.ensemble import GradientBoostingClassifier\n",
        "from sklearn.ensemble import AdaBoostClassifier\n",
        "from sklearn.ensemble import RandomForestClassifier\n",
        "from sklearn.metrics import precision_score\n",
        "from sklearn.metrics import recall_score\n",
        "from sklearn.metrics import accuracy_score\n",
        "from sklearn.metrics import f1_score\n",
        "from sklearn.metrics import log_loss\n",
        "from sklearn.preprocessing import Normalizer\n",
        "from sklearn.preprocessing import StandardScaler\n",
        "from sklearn import utils\n",
        "\n",
        "import tensorflow as tf\n",
        "from tensorflow import feature_column\n",
        "from tensorflow import keras\n",
        "import keras\n",
        "from tensorflow.keras import layers\n",
        "from tensorflow.keras.layers import Dense, BatchNormalization\n",
        "from tensorflow.keras.preprocessing.image import ImageDataGenerator\n",
        "\n",
        "import IPython\n",
        "import IPython.display\n",
        "\n",
        "from gensim.test.utils import datapath, get_tmpfile\n",
        "from gensim.models import KeyedVectors\n",
        "from gensim.scripts.glove2word2vec import glove2word2vec\n",
        "\n",
        "from imblearn.over_sampling import SMOTE\n",
        "import warnings\n",
        "\n",
        "warnings.filterwarnings(\"ignore\")\n",
        "\n",
        "import os\n",
        "import zipfile\n",
        "\n",
        "import glob\n",
        "import imageio\n",
        "import PIL\n",
        "import tensorflow_probability as tfp\n",
        "import time\n",
        "import tensorflow_docs.vis.embed as embed\n",
        "\n",
        "from scipy import ndimage, misc\n",
        "import cv2"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "def check_over(x, printFlag = 0):\n",
        "    \"\"\"\n",
        "    Takes in the current board state x,\n",
        "    and determines if the game is won by \n",
        "    any player or if there is a draw.    \n",
        "    Returns True if episode is over. \n",
        "    Returns False if episode is not over. \n",
        "    Also, returns a string '1' if player 1\n",
        "    has won, a string '2' if player 2 has\n",
        "    won, or a string '3' if there is a draw. \n",
        "    \"\"\"\n",
        "    # Check if Player 1 won\n",
        "    # First check each row\n",
        "    # Then check each column\n",
        "    # Finally check each diagonal\n",
        "    if np.prod(x[0,:]) == 1 or np.prod(x[1,:]) == 1 or np.prod(x[2,:]) == 1 or \\\n",
        "        np.prod(x[:,0]) == 1 or np.prod(x[:,1]) == 1 or np.prod(x[:,2]) == 1 or \\\n",
        "        np.prod(np.diag(x)) == 1 or np.prod(np.diag(np.rot90(x))) == 1:\n",
        "        if printFlag == 1:\n",
        "            # Printing occurs during some testing\n",
        "            # and human matches\n",
        "            print(\"Player 1 won!\")\n",
        "        done = True\n",
        "        player_win = '1'\n",
        "        return done, player_win\n",
        "\n",
        "    # Check if Player 2 won\n",
        "    # First check each row\n",
        "    # Then check each column\n",
        "    # Finally check each diagonal\n",
        "    if np.sum(x[0,:]) == 6 or np.sum(x[1,:]) == 6 or np.sum(x[2,:]) == 6 or \\\n",
        "        np.sum(x[:,0]) == 6 or np.sum(x[:,1]) == 6 or np.sum(x[:,2]) == 6 or \\\n",
        "        np.sum(np.diag(x)) == 6 or np.sum(np.diag(np.rot90(x))) == 6:\n",
        "        if printFlag == 1:\n",
        "            print(\"PLAYER 2 WON!\")\n",
        "        done = True\n",
        "        player_win = '2'\n",
        "        return done, player_win\n",
        "\n",
        "    # Check if a draw occurred \n",
        "    if np.all(x):\n",
        "        if printFlag == 1:\n",
        "            print(\"Draw!\")\n",
        "        done = True\n",
        "        player_win = 'd'\n",
        "        return done, player_win\n",
        "    \n",
        "    # Match is not over\n",
        "    return False, ' '"
      ],
      "metadata": {
        "id": "FhBZ9xrIXS8n"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Player symbols - P1 = X, P2 = O\n",
        "symbols = {1: 'X', 2: 'O'}\n",
        "\n",
        "# Max number of state values per cell\n",
        "max_vals = 3\n",
        "# Max number of controls (actions)\n",
        "max_controls = 9\n",
        "# Create two Q-matrices\n",
        "QvalsP1 = np.zeros((max_vals, max_vals, max_vals, max_vals, max_vals,\n",
        "                  max_vals, max_vals, max_vals, max_vals, max_controls))\n",
        "QvalsP2 = np.zeros((max_vals, max_vals, max_vals, max_vals, max_vals,\n",
        "                  max_vals, max_vals, max_vals, max_vals, max_controls))"
      ],
      "metadata": {
        "id": "O_-5VMDIl3Ug"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "alphaP1 = 0.9\n",
        "gammaP1 = 0.99\n",
        "epsilonP1 = 1.0\n",
        "\n",
        "alphaP2 = 0.9\n",
        "gammaP2 = 0.99\n",
        "epsilonP2 = 1.0\n",
        "\n",
        "epochs = 100000\n",
        "\n",
        "winP1 = np.zeros((epochs,))\n",
        "winP2 = np.zeros((epochs,))\n",
        "draw = np.zeros((epochs,))\n",
        "\n",
        "decayStages = 20\n",
        "stageLength = epochs // decayStages\n",
        "\n",
        "decayFactor = 0.7"
      ],
      "metadata": {
        "id": "P7xBl2I-axoL"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print('Q-table: ', QvalsP1.shape)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zB9TvPY1dLPS",
        "outputId": "3991a6fb-b479-4c7d-c462-057a9322cf91"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Q-table:  (3, 3, 3, 3, 3, 3, 3, 3, 3, 9)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "def control_train(x, epsilon, controls, Qvals):\n",
        "    if np.random.random() >= epsilon:\n",
        "        # Exploitation\n",
        "        u = np.argmax(Qvals[x])\n",
        "\n",
        "        if u not in controls:\n",
        "            new_control = np.random.choice(list(controls))\n",
        "            return new_control\n",
        "        else:\n",
        "            return u\n",
        "    else:\n",
        "        # Exploration\n",
        "        u = np.random.choice(list(controls))\n",
        "        return u"
      ],
      "metadata": {
        "id": "X_g4su5edi1x"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# This function is \n",
        "def control_test(x, controls, Qvals):\n",
        "    u = np.argmax(Qvals[x])\n",
        "    \n",
        "    if u not in controls:\n",
        "        new_control = np.random.choice(list(controls))\n",
        "        return new_control\n",
        "    else:\n",
        "        return u"
      ],
      "metadata": {
        "id": "Dy4bMKz5esiY"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def decay_epsilon(episode, epsilon, stageLength, decayFactor):\n",
        "    if episode % stageLength == 0 and episode != 0:\n",
        "        epsilon *= decayFactor\n",
        "        print(f'epsilon = {epsilon}')\n",
        "    return epsilon"
      ],
      "metadata": {
        "id": "DXs39u1wesf4"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def reset():\n",
        "    controls = set()\n",
        "    for i in range(9):\n",
        "        controls.add(i)\n",
        "    return controls"
      ],
      "metadata": {
        "id": "A6J98pVSfdU5"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def batch_update(statesRL, controlsRL, r, nextStatesRL, gamma, alpha, Qvals):\n",
        "    for i, boardVals in enumerate(statesRL):\n",
        "        u = controlsRL[i]\n",
        "        boardValsNext = nextStatesRL[i]\n",
        "        td_target = r + gamma * np.max(Qvals[boardValsNext])\n",
        "        td_error = td_target - Qvals[boardVals][u]\n",
        "        Qvals[boardVals][u] += alpha * td_error\n",
        "    return Qvals"
      ],
      "metadata": {
        "id": "ihjt1-K2frwq"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "reward = 10\n",
        "\n",
        "for epoch in range(epochs):\n",
        "    if epoch % 1000 == 0:\n",
        "        print(epoch)\n",
        "    statesRLP1 = []\n",
        "    controlsRLP1 = []\n",
        "    nextStatesRLP1 = []\n",
        "\n",
        "    statesRLP2 = []\n",
        "    controlsRLP2 = []\n",
        "    nextStatesRLP2 = []\n",
        "\n",
        "    done = False\n",
        "    controls = reset()\n",
        "\n",
        "    boardVals = np.zeros((3,3))\n",
        "    turn = 0\n",
        "\n",
        "    board = np.array([[' ', ' ', ' '],\n",
        "                      [' ', ' ', ' '],\n",
        "                      [' ', ' ', ' ']])\n",
        "    epsilonP1 = decay_epsilon(epoch, epsilonP1, stageLength, decayFactor)\n",
        "    epsilonP2 = decay_epsilon(epoch, epsilonP2, stageLength, decayFactor)\n",
        "\n",
        "    while done != True:\n",
        "        if turn % 2 == 0:\n",
        "            # P1 turn - RL agent 2\n",
        "            u = control_train(tuple(boardVals.reshape(-1,).astype(int)),\n",
        "                              epsilonP1, controls, QvalsP1)\n",
        "            statesRLP1.append(tuple(boardVals.reshape(-1,).astype(int)))\n",
        "            controlsRLP1.append(u)\n",
        "            controls.remove(u)\n",
        "\n",
        "            boardValsNext = boardVals.reshape(-1,)\n",
        "            boardValsNext[u] = 1\n",
        "            boardValsNext = boardValsNext.reshape(3,3)\n",
        "            nextStatesRLP1.append(tuple(boardValsNext.reshape(-1,).astype(int)))\n",
        "\n",
        "            board = board.reshape(-1,)\n",
        "            board[u] = symbols[1]\n",
        "            board = board.reshape(3,3)\n",
        "\n",
        "            done, winner = check_over(boardVals)\n",
        "            if done == True:\n",
        "                if winner == '1':\n",
        "                    Qvals = batch_update(statesRLP1,\n",
        "                                         controlsRLP1,\n",
        "                                         reward,\n",
        "                                         nextStatesRLP1,\n",
        "                                         gammaP1,\n",
        "                                         alphaP1,\n",
        "                                         QvalsP1)\n",
        "                    winP1[epoch] = 1\n",
        "                elif winner == 'd':\n",
        "                    Qvals = batch_update(statesRLP1,\n",
        "                                         controlsRLP1,\n",
        "                                         -5,\n",
        "                                         nextStatesRLP1,\n",
        "                                         gammaP1,\n",
        "                                         alphaP1,\n",
        "                                         QvalsP1)\n",
        "                    draw[epoch] = 1\n",
        "                else:\n",
        "                    Qvals = batch_update(statesRLP1,\n",
        "                                         controlsRLP1,\n",
        "                                         -50,\n",
        "                                         nextStatesRLP1,\n",
        "                                         gammaP1,\n",
        "                                         alphaP1,\n",
        "                                         QvalsP1)\n",
        "            boardVals = boardValsNext\n",
        "            turn += 1\n",
        "        else:\n",
        "            # P2 - RL agent 2\n",
        "            u = control_train(tuple(boardVals.reshape(-1,).astype(int)),\n",
        "                              epsilonP2, controls, QvalsP2)\n",
        "            statesRLP2.append(tuple(boardVals.reshape(-1,).astype(int)))\n",
        "            controlsRLP2.append(u)\n",
        "            controls.remove(u)\n",
        "\n",
        "            boardValsNext = boardVals.reshape(-1,)\n",
        "            boardValsNext[u] = 2\n",
        "            boardValsNext = boardValsNext.reshape(3,3)\n",
        "            nextStatesRLP2.append(tuple(boardValsNext.reshape(-1,).astype(int)))\n",
        "\n",
        "            board = board.reshape(-1,)\n",
        "            board[u] = symbols[2]\n",
        "            board = board.reshape(3,3)\n",
        "\n",
        "            done, winner = check_over(boardVals)\n",
        "            if done == True:\n",
        "                if winner == '2':\n",
        "                    Qvals = batch_update(statesRLP2,\n",
        "                                         controlsRLP2,\n",
        "                                         reward,\n",
        "                                         nextStatesRLP2,\n",
        "                                         gammaP2,\n",
        "                                         alphaP2,\n",
        "                                         QvalsP2)\n",
        "                    winP2[epoch] = 1\n",
        "                elif winner == 'd':\n",
        "                    Qvals = batch_update(statesRLP1,\n",
        "                                         controlsRLP1,\n",
        "                                         5,\n",
        "                                         nextStatesRLP1,\n",
        "                                         gammaP1,\n",
        "                                         alphaP1,\n",
        "                                         QvalsP1)\n",
        "                    draw[epoch] = 1\n",
        "                else:\n",
        "                    Qvals = batch_update(statesRLP1,\n",
        "                                         controlsRLP1,\n",
        "                                         -50,\n",
        "                                         nextStatesRLP1,\n",
        "                                         gammaP1,\n",
        "                                         alphaP1,\n",
        "                                         QvalsP1)\n",
        "            boardVals = boardValsNext\n",
        "            turn += 1"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "fgqlKDayhiIU",
        "outputId": "7616d27b-26a3-41b7-cccb-dda46ed75cb5"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0\n",
            "1000\n",
            "2000\n",
            "3000\n",
            "4000\n",
            "5000\n",
            "epsilon = 0.7\n",
            "epsilon = 0.7\n",
            "6000\n",
            "7000\n",
            "8000\n",
            "9000\n",
            "10000\n",
            "epsilon = 0.48999999999999994\n",
            "epsilon = 0.48999999999999994\n",
            "11000\n",
            "12000\n",
            "13000\n",
            "14000\n",
            "15000\n",
            "epsilon = 0.3429999999999999\n",
            "epsilon = 0.3429999999999999\n",
            "16000\n",
            "17000\n",
            "18000\n",
            "19000\n",
            "20000\n",
            "epsilon = 0.24009999999999992\n",
            "epsilon = 0.24009999999999992\n",
            "21000\n",
            "22000\n",
            "23000\n",
            "24000\n",
            "25000\n",
            "epsilon = 0.16806999999999994\n",
            "epsilon = 0.16806999999999994\n",
            "26000\n",
            "27000\n",
            "28000\n",
            "29000\n",
            "30000\n",
            "epsilon = 0.11764899999999995\n",
            "epsilon = 0.11764899999999995\n",
            "31000\n",
            "32000\n",
            "33000\n",
            "34000\n",
            "35000\n",
            "epsilon = 0.08235429999999996\n",
            "epsilon = 0.08235429999999996\n",
            "36000\n",
            "37000\n",
            "38000\n",
            "39000\n",
            "40000\n",
            "epsilon = 0.05764800999999997\n",
            "epsilon = 0.05764800999999997\n",
            "41000\n",
            "42000\n",
            "43000\n",
            "44000\n",
            "45000\n",
            "epsilon = 0.04035360699999998\n",
            "epsilon = 0.04035360699999998\n",
            "46000\n",
            "47000\n",
            "48000\n",
            "49000\n",
            "50000\n",
            "epsilon = 0.028247524899999984\n",
            "epsilon = 0.028247524899999984\n",
            "51000\n",
            "52000\n",
            "53000\n",
            "54000\n",
            "55000\n",
            "epsilon = 0.019773267429999988\n",
            "epsilon = 0.019773267429999988\n",
            "56000\n",
            "57000\n",
            "58000\n",
            "59000\n",
            "60000\n",
            "epsilon = 0.01384128720099999\n",
            "epsilon = 0.01384128720099999\n",
            "61000\n",
            "62000\n",
            "63000\n",
            "64000\n",
            "65000\n",
            "epsilon = 0.009688901040699992\n",
            "epsilon = 0.009688901040699992\n",
            "66000\n",
            "67000\n",
            "68000\n",
            "69000\n",
            "70000\n",
            "epsilon = 0.006782230728489994\n",
            "epsilon = 0.006782230728489994\n",
            "71000\n",
            "72000\n",
            "73000\n",
            "74000\n",
            "75000\n",
            "epsilon = 0.004747561509942996\n",
            "epsilon = 0.004747561509942996\n",
            "76000\n",
            "77000\n",
            "78000\n",
            "79000\n",
            "80000\n",
            "epsilon = 0.003323293056960097\n",
            "epsilon = 0.003323293056960097\n",
            "81000\n",
            "82000\n",
            "83000\n",
            "84000\n",
            "85000\n",
            "epsilon = 0.002326305139872068\n",
            "epsilon = 0.002326305139872068\n",
            "86000\n",
            "87000\n",
            "88000\n",
            "89000\n",
            "90000\n",
            "epsilon = 0.0016284135979104473\n",
            "epsilon = 0.0016284135979104473\n",
            "91000\n",
            "92000\n",
            "93000\n",
            "94000\n",
            "95000\n",
            "epsilon = 0.001139889518537313\n",
            "epsilon = 0.001139889518537313\n",
            "96000\n",
            "97000\n",
            "98000\n",
            "99000\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "winTotP1 = np.cumsum(winP1)\n",
        "winTotP2 = np.cumsum(winP2)\n",
        "drawTot = np.cumsum(draw)\n",
        "\n",
        "winRateP1 = np.zeros(epochs,)\n",
        "winRateP2 = np.zeros(epochs,)\n",
        "drawRate = np.zeros(epochs,)\n",
        "\n",
        "for e in range(epochs):\n",
        "    winRateP1[e] = winTotP1[e] / (e + 1)\n",
        "    winRateP2[e] = winTotP2[e] / (e + 1)\n",
        "    drawRate[e] = drawTot[e] / (e + 1)\n",
        "\n",
        "plt.figure()\n",
        "plt.plot(np.arange(epochs), winRateP1, 'g-.', label = 'P1 Win Rate')\n",
        "plt.plot(np.arange(epochs), winRateP2, 'r:.', label = 'P2 Win Rate')\n",
        "plt.plot(np.arange(epochs), drawRate, 'k-', label = 'Draw Rate')\n",
        "plt.xlabel('Epochs')\n",
        "plt.legend()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 296
        },
        "id": "U6TbyutUk_h3",
        "outputId": "3897a72e-5c3c-4dd0-fca9-f3086ef29df9"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<matplotlib.legend.Legend at 0x7f9a45a93f50>"
            ]
          },
          "metadata": {},
          "execution_count": 307
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEGCAYAAACevtWaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXgUVdrw4d/J2iGEJSFIIISwhCVhSwhIiEEREGQEF9wAZ8YZfHGdeQdxQz8VR+dVh0VwxAUdxAVxRtERFQUEREdACAoxhCUBAgRZEiAJIWQ/3x/VaZqQpRO606nOc19XXV11anuqq/vp6lNVp5TWGiGEEJ7Fy90BCCGEcD5J7kII4YEkuQshhAeS5C6EEB5IkrsQQnggH3etuF27djoyMtJdqxdCCFPatm1bjtY6tK7p3JbcIyMjSU5OdtfqhRDClJRSBx2ZTqplhBDCA0lyF0IIDyTJXQghPJAkdyGE8ECS3IUQwgPVmdyVUouVUieUUqk1jFdKqZeVUhlKqRSlVJzzwxRCCFEfjhy5LwHG1jL+WiDK2k0DXrv0sGqxaRM8/7zxKoQQolp1Xueutf5OKRVZyyTXA+9qo+3gzUqpNkqpMK31USfFeN6mTTByJJSUgJ8frF0LCQlOX40QQpidM+rcOwGH7YazrGUXUUpNU0olK6WSs7Oz67+mb7+F4mIoLzcS/LffNiBcIYTwfI16QlVrvUhrHa+1jg8NrfPu2YtddRX4WP9s+PkZw0IIIS7ijOYHjgCd7YbDrWXOl5BgHK1/+62R2KVKRgghquWM5L4CeEAp9SFwOZDnkvr2SgkJktSFEKIOjlwKuQzYBPRSSmUppaYqpe5RSt1jnWQlsB/IAN4E7nNZtADvvw9Dh8K5cy5djRBCmJkjV8tMqmO8Bu53WkR18feHNm1AqUZbpRBCmI357lC95Rb4+muwWNwdiRBCNFnmS+5CCCHqZL7k/s47EBcnde5CCFEL8yX3oCAID5c6dyGEqIXbHrPXYDfdZHRCCCFqZL4jdyGEEHUyX3J/+22IiYHCQndHIoQQTZb5kntICERHS527EELUwnx17hMmGJ0QQogame/IXQghRJ3Ml9zfeguiouDsWXdHIoQQTZb5kntYGAwZAt7e7o5ECCGaLPPVuf/mN0YnhBCiRuY7chdCCFEn8yX3N96AiAipcxdCiFqYL7l36QKjRkmduxBC1MJ8de5jxxqdEEKIGpnvyF0IIUSdzJfcX30VOnSQOnchhKiF+ZJ7VBTccIPUuQshRC3MV+c+erTRCSGEqJH5jtyFEELUyXzJ/ZVXIDgYCgrcHYkQQjRZ5kvuffrAHXeAr6+7IxFCiCbLfHXuI0canRBCiBqZ78hdCCFEncyX3OfPh8BAqXMXQohamK9aJjYW7rtP6tyFEE1Sha7g1zO/cijvEIfzDnM4/zCH8w6TdSaLw3mHSYpIYu6YuS6Pw3zJ/corjU4IIdzgTPEZDuUdwtfbl54hPSktL+UPn/2B8T3Hc1vf28jMzaT7y90vmCfIL4jOrTvTuVVnOrXq1Chxmi+5CyGEi+04toP9p/dzIPcAB04f4GDeQQ7lHeJQ3iFOF50GYHK/ySy9aSm+3r6kHE8hvmM8AOGtwnnjujfo3KozEa0jCG8VTmtL60bfBqW1bvSVAsTHx+vk5OT6z/jSS/DII3DqFAQFOT8wIYRHKyor4mDuQfKK8xjSaQgAD656EF8vX14c/SIAHeZ04PjZ44Bx1B3ZJpKI1hFEtI6gS+suRLSOIKZ9DP0v69/o8Sultmmt4+uazqEjd6XUWGAB4A28pbV+ocr4COAdoI11mse01ivrHbUjBg82krufn0sWL4Qwv4KSAvad2kfGqYzz3WnjNSs/C4AurbuQ+ZdMAApLC/H1On8e74OJH9DavzVd23alraUtSil3bMYlqfPIXSnlDewFRgNZwFZgktY6zW6aRcDPWuvXlFLRwEqtdWRty23wkbsQQmDUfbf0a4lSis/3fM43+79hwbULALjxXzfyn93/sU3bPrA9PYJ70CO4B93adKNb2250D+7OsM7D3BV+gznzyH0IkKG13m9d8IfA9UCa3TQaaGXtbw38Wr9w60Fro1PK6IQQHquorIiMUxnsPbmX9JPppJ9KN/pPpXOs4Bi/PvgrYUFh/HLiFz7Z/Ql/H/13/H38+dOQPzGl3xR6BPege9vuBPk3vypcR47cbwbGaq3vsg7/Frhca/2A3TRhwGqgLRAIjNJab6tmWdOAaQARERGDDh48WP+I586Fhx6C/HypcxfCQxSWFrLj2A76tu9LkH8QH6d9zGPfPMaB3ANU6ArbdJcFXkZUSBQ9g3sSFRLFXXF30a5FO7TWpqw6aQin1rk7YBKwRGs9VymVALynlOqrtd1eAbTWi4BFYFTLNGhNw4bBM89InbsQJpRTmENadho7T+xkZ/ZOJvWdRGJEIpuzNjPy3ZGs+e0aRnUbRbsW7YgLi2NKvyn0btebniFGMm/l36ra5TaXxF4fjiT3I0Bnu+Fwa5m9qcBYAK31JqWUBWgHnHBGkBdISDA6IUSTVVJewrZft5FyPIXUE6nszDaS+Ymz51NCkF8Qg8IGkRiRyKCwQay4fQVxYXEAXBV5FVdFXuWm6D2DI8l9KxCllOqKkdRvByZXmeYQMBJYopTqA1iAbGcGalNWZnT+/lLnLkQTobXm6W+fZsBlA5gYPZFT504xbLFxsrKlX0tiQmO4Luo6okOjiWkfQ0xoDOGtwm1H3K0trRnfa7w7N8Hj1JnctdZlSqkHgFUYlzku1lrvVEr9FUjWWq8AZgBvKqWmY5xcvVO76gL6BQukzl0IN8gtyiXleAo7ju1gx/EdpBxPoXtwd5ZNXIZSig9++YCCkgImRk/kssDL+HLyl0SHRtOldRepNnEDh+rcrdesr6xS9pRdfxqQ6NzQapCUBM8/L3XuQrjYt5nf8sOhH/j52M/8dPQnDuQesI0LCQhhQIcB9G9//iaePQ/swdvLeLaxUopxUeMaPWZxnvmaHxgyxOiEEE71ya5PWLt/LQt/sxCA2RtnszJ9JT2CexDfMZ5pg6Yx4LIBDOgwgLCWYRcdjVcmdtE0mC+5FxcbXVCQ1LkLUU9lFWWkHE9hc9ZmNmVtYsuRLWz840ZCWoSwO2c33xz4huKyYvx9/Hl13Ku0DWhb4xUqomkzX3J/5RWpcxfCQccLjrMpa5MtmSf/mkxhaSFgXDM+NHwo+cX5hLQIYeYVM3k86XHbvF3adHFX2MIJzJfcr7zSuJHJ39/dkQjRpJRVlLHj2A46BnUkLCiML/Z+wfhlxhUovl6+xIbFclfsXSR0TmBo+NCLTnTKSU/PYr7kHh9vdEI0c6fOnWLT4U20sbQhMSKRE2dPEP9mPC+NeYm/DP0LgzsOZs7oOSR0TiAuLA6Lj8XdIYtGZL7kXlgIZ89CSAh4me8pgUI0hNaazNxMvj/0Pf899F/+e+i/7MrZBcDN0TeTGJFIx6CO/Oe2/zA0fCgAl7W8jBnDZrgzbOFG5kvur70mde6i2Xhvx3us3r+aDZkbOJx/GMA4Uu+cyB3972BY52EM7jjYNv31va93V6iiiTFfcr/6auOkqtS5Cw+0KmMV3x/6nueufg6A9395nx3HdjC8y3Ae7fIow7sMJ6Z9DF5K/rWK2pkvucfGGp0QJnes4BjrDqxj3YF1vDTmJYL8g/jxyI+89dNbPJH0BAG+Afz75n/Tyr+VnOwU9Wa+x+ydOWNUyYSFSZ27MJWCkgI2ZG5g9b7VfHPgG9KyjUcitLG04ZvffsOgjoMoKivC39tfkrmoUWM3+dt43nwTZsyAvDxoJTdXiKbtxNkTvLntTVbtW8WmrE2UVZRh8bEwvMtwfj/g91zd9WpiO8Ta7u6UK1qEs5gvuY8eDYsWgUW+BKLpqdAVrN63miC/IBIjEikqK+LJ9U8SGxbLjIQZjO42msSIREniwuXMl9z79TM6IZqIo2eOsu/0Pq6IuAKFYtrn07g8/HISIxKJaB3B8YeOExoY6u4wRTNjvuSemwunTkFkpNS5C7fQWrPt6DZW7FnB53s/Z/ux7bQPbM/RGUfxUl58fcfXdGvbzTa9JHbhDuZL7osXS527aHSl5aV8d/A7PtvzGZ/t+YxDeYfwUl4kdk7k+ZHPM6b7GBTGSdDo0Gg3RyuEGZP72LHQrp3UuYtG83Hax0z7fBqni05j8bEwuttonrnqGcb3HE9IixB3hydEtcyX3KOjjU4IF/n1zK/cv/J+7h98P6O6jaJHcA/G9xrPjb1v5Jru19DCt4W7QxSiTuZL7qdOwYkTEBUF3vJwANFwWmu2/rqV5WnLWXtgLRP7TGRm0kyCA4LZnbObnMIcAAZ2GMg7N7zj5miFqB/zJfclS6TOXTSY1potR7bwUdpHLN+1nMzcTHy8fIgLiyP9VDpgXGuedl+a3EgkTM18yX3cOOjYEQIC3B2JMJHUE6m8u+Ndlu9azv7T+/H18mV099E8feXTXN/retoGtL1geknswuzMl9x79zY6IepwsvAkgX6BWHwsfJ3xNfM3z2dE1xE8OfxJbuh9A20sbdwdohAuY762ZbKz4ehRiImROndRo13Zu4h9I5ZF4xfxuwG/I784n/KK8ouO0IUwG0fbljHfXUDvvQcDBhgP7BAC64nRI1u5+/O7eXTNowD0btebRxMfZUinIQC08m8liV00K+arlhk/3rg7Vercm709OXt4P+V9lqUuY9/pfQT4BHDf4PsAo878mRHPuDlCIdzHfMk9KsroRLNUWFrI0pSlvPnTm2z9dSteyotR3UbxaOKj3BpzK60trd0dohBNgvmS+7FjcPgwxMVJnbsJaa15deurXBl5JX3b963XvB+mfsg9X9xDXnEefdv3Ze41c7m97+10DOroomiFMC/z1bl/8AEMGSJ17iZzMPcgYFSX/HD4Bwa+PpCHVj9EblFujfMUlxWzcMtCNmRuACC2QyzX976eDXduIOWeFB5MeFASuxA1MF1y/0fYYX4zGX7K3eXuUISD3t3xLpELItlxbAcAC8Yu4M6BdzJv0zz6vdaP1ftWU1RWxLs73uWGD29g8c+LAfD28uaJdU+w9sBaAHq168U7N7zD8C7D5Tp0IepgumqZPa1LWdkTZlLq7lBEDQpLC1n2yzJa+rXktr63cX2v65kzeg4dWnYAjCZw35rwFvfE38OUT6Yw5v0x+Hn7UVJeQqBvIKknUrlz4J34ePmQel8q4a3C3bxFQpiP6ZL7SL/e/HwI2vsHuzsUUcW2X7fx1k9v8UHqB+QX5zM1diq39b2N1pbWzBg246Lp4zvGs+OeHby29TXST6Vzc/TNjIgcwcG8g7ZpJLG7V2lpKVlZWRQVFbk7lGbHYrEQHh6Or69vg+Z3KLkrpcYCCwBv4C2t9QvVTHMrMAvQwA6t9eQGRVSHLqs288NiSP9/51yxeFGDCl3BE2ufoG1AWx5JfIQ9OXtYd2AdwzoP48v0L/ko7SO2H9tOC98W3ND7Bu4edDdJEUl1LtfiY2F6wvQLyiLbRLpoK0R9ZWVlERQURGRkpFSFNSKtNSdPniQrK4uuXbs2aBl1JnellDewEBgNZAFblVIrtNZpdtNEATOBRK31aaVU+wZF44AF4Vn8+lv4w+mdRHUdRPbZbNrPac+jiY/ywqiLfnNM52zJWYrLi/FW3vx45EfW7FvDnpN7OJB7gNQTRhXF3gf2EuB76df5a63558//5FDeIR5PehyLj4UzxWf4985/szJjJVHBUcSFxXFTn5vw8fLhxyM/khCeAMD3h77nvpX32ZY1pNMQXrn2Fab0nyK39XuQoqIiSexuoJQiJCSE7Ozshi+jruYHlFIJwCyt9Rjr8EwArfXzdtP8HdirtX7L0RU3tPkB9cz5D9nzI59n5tqZF4zffvd2erfrjeVvdT/MY9PUTVze6XK3f3DnbZrHjNUzWHTdIu5feT+lFTWfT1Aozj1xjse+eYz5P87n1wd/JSwoDDCOrn849APvpbwHwIjIEZRWlNK1TVcSOifg4+XDliNb+HzP5zx79bMAxL4Ry/Zj2+netjst/Vqy4/gO27p8vHwoqyhj7wN7iQqJokJXoFAopSirKGNz1mZ2ntjJqG6j6B7c3YXvkHCXXbt20adPH3eH0WxV9/472vyAI8n9ZmCs1vou6/Bvgcu11g/YTfMfYC+QiFF1M0tr/XU1y5oGTAOIiIgYdPDgwaqT1CliuqLHKfiuC5Q7+TL313/zOldGXkmvkF4XJfxT504R4BNQryNmrTV5xXn8v3X/z7hGO/4eLD4WFv+8mAO5BwDYPHUzIS1CiPpHFD/88QfOFJ9h7NKxAPRt35eF4xbaqjeUUpwtOUsL3xZcu/RaVu1bRemTpRzJP0LUP6Jq/VH485A/s+DaBfztu7/x/H+f59D0QwQHBJNfnM93B7/j7i/uJrRFKL+J+g1DOg3hup7XkZmbyZ6TexjbYyxeynQXVgknkOTuXk0huX8BlAK3AuHAd0A/rXWNFzE39Mh9+ljFS6ug9WOQbz04X33HaqJCoui64HzdlLfyplyX89b4txgXNY7swmzatWjHudJzpBxPoU9oH/osbNiHNtA3kLOl56+znz50Oo8nPc6hvEPEdojl0W8eZWyPsQT6BjL0n0NrXVbqvanEtI9pUByVfsz6kbFLx5JblMvjVzzO1LipeCkvVqavZGCHgXy590ti2scwud9kyivKKasow9/H/5LWKZqHppDcvb296devH2VlZfTp04d33nmHFi1a8Mc//pEvvviC9u3bk5qaetF8ubm5dO/enZycHJRSbNq0iWHDhnH48GHCw8PJy8uja9eu5OTkcN111/HBBx/Qpo1jVYqzZs3izTffJDQ0lJKSEp588kkmTZpU6zzz589n2rRptGjh+JO8LiW5o7WutQMSgFV2wzOBmVWmeR34g93wWmBwbcsdNGiQbohO09FJd6K9n0Qzy+gqnSw8qZmFDv17qMPLKykr0Xd8codmFnpD5gbNLHT8onjbsu277gu6V1tu36UeT7X1l1eU6+lfT9cBzwXoJ9c9qUe9O0r3+kcv/V3md7qkrKRB2y9EY0pLS3N3CDowMNDWP3nyZD137lyttdYbNmzQ27Zt0zExMTXOGxMTo3fu3Km11nrOnDk6NjZW/+tf/9Jaa/3111/rMWPGNCimp59+Ws+ePVtrrfXevXt1UFCQLimp/TvdpUsXnZ2dXa/1VPf+A8m6jryttXboapmtQJRSqitwBLgdqHolzH+AScDbSql2QE9gvwPLrrcjrY2uOsEBwein69eEsa+3L+/d+B7v3WjUU9dnfq01SW8n8cPhH2xlvdv1ZvrQ6baqjHlj5jFvzLx6xSSEqF5SUhIpKSkADB8+nMzMzFqnHzZsGBs3biQ6OpqNGzcyffp0Nm7cyK233srGjRtJTEwEIDIykuTkZAoKCrj22mu54oor2LhxI506deKzzz4joJaGCqOiomjRogWnT5+mffv23HvvvWzdupVz585x880388wzz/Dyyy/z66+/MmLECNq1a8f69etZvXo1Tz/9NMXFxXTv3p23336bli1bOu29qjO5a63LlFIPAKsw6tMXa613KqX+ivELssI67hqlVBpQDjystT7ptCjtdDkNvXNgbTco84ad9+10xWocopTiv3/870XlksyFp7pqyVV1TnNdz+t4aNhDtunvHHgndw68k5zCHG7+980XTPvtnd86vO6ysjK++uorxo4d6/A8iYmJbNiwgbvuuov9+/dzyy238MYbbwCwceNGHnvssYvmSU9PZ9myZbz55pvceuutLF++nDvuuKPGdfz0009ERUXRvr1xkeDf/vY3goODKS8vZ+TIkaSkpPDnP/+ZefPmsX79etq1a0dOTg7PPfcc33zzDYGBgbz44ovMmzePp556yuFtq4tD17lrrVcCK6uUPWXXr4EHrZ1L3bgbW5177jMVbr/SRQjhWufOnWPgwIGAceQ+depUh+cdNmwYzz//PAcOHCAyMhKLxYLWmoKCArZt28bll19+0Txdu3a1rW/QoEE1/jt46aWXePvtt9m7dy+ff/65rfzf//43ixYtoqysjKNHj5KWlkb//v0vmHfz5s2kpaXZ/jmUlJSQkJDg8HY5wnR3qH7YFzaFw1lfec6lEI2tPkfaVadv16JdvecHCAgIYPv27fWeD4wqk9zcXD7//HNb8hw0aBBvv/02kZGR1VaD+Pufv9jA29ubc+eqv2Fy+vTpPPTQQ6xYsYKpU6eyb98+jh49ypw5c9i6dStt27blzjvvrPbuXq01o0ePZtmyZQ3aLkeY7vq2Y0HwY2fnXwYphPBMQ4cOZcGCBbbknpCQwPz5821HzZdqwoQJxMfH884775Cfn09gYCCtW7fm+PHjfPXVV7bpgoKCOHPmjC2mH374gYyMDADOnj3L3r17nRJPJdMl92c738n43ZAUVvslhkIIzzZp0iQSEhLYs2cP4eHh/POf/6x2usTERA4fPkx8vHH1YEJCAvv372fYsGFOi+Wpp55i3rx59OvXj9jYWHr37s3kyZMv+AGZNm0aY8eOZcSIEYSGhrJkyRImTZpE//79SUhIYPfu3U6LB0z4gOx374zld+9s5653b+at337kgsiEEJWawnXuzdmlXOduuiP3h0O3EzcN3s742N2hCCFEk2W65H7F4Jv4uSM8mPiQu0MRQogmy3TJ/S9tx3HzTripx3h3hyKEEE2W6ZJ72eef8dFHcOCI+25eEkKIps50yf1PoVuJuQ825f7i7lCEEKLJMl1yHxgzkrT2MCTCuXdzCSGEJzFdcu90JJ/JKaBK5QHZQjQH3t7eDBw4kL59+3LLLbdQWFjI4cOHGTFiBNHR0cTExLBgwYKL5svNzSUkJKSypVo2bdqEUoqsrCwA8vLyCA4OpqKignHjxpGbW2ML5ReZNWsWnTp1YuDAgURHRzt0p+n8+fMpLCx0eB2XynTJ/bLvf2bpJ5CeleLuUIQQjaCy+YHU1FT8/Px4/fXX8fHxYe7cuaSlpbF582YWLlxIWlraBfO1adOGsLAwdu3aBRgNhcXGxrJx40bAaN9lyJAheHl5sXLlSofbcq80ffp0tm/fzmeffcbdd99NaR0HnJLc63A4PIg5CRCW6ZJGJ4UQl2rTJnj+eePVyZKSksjIyCAsLIy4uDjAuK2/T58+HDly5KLpK5v8BS5o8rdy2L7J35ycHDIzM+nTpw//8z//Q0xMDNdcc02NbctUsm/yF+Dee+8lPj6emJgYnn76aYALmvwdMWIEAKtXryYhIYG4uDhuueUWCgoKnPAOnWeu5L5pE3cvTOfcJpj2+Mcu+fAIIWpx1VWwZInRX1pqDL//vjFcWAixsUbZk0/CyJHG8CefGONzcoxxlS0oHjtWr1VXNvnbr1+/C8ozMzP5+eefq23hMTEx0ZbMK5v8rbwzfuPGjdU2QZCens7999/Pzp07adOmDcuXL681ruqa/E1OTiYlJYUNGzbYmvzt2LEj69evZ/369Rc0+fvTTz8RHx/PvHnObSrcXK1Cfvst15aUkQn8T3EJHb79FpzcTKYQ4hLk5UFZGVRUQEmJMXyJamvyt6CggIkTJzJ//nxatWp10bzS5K9ZXHUVlTVW2tfXOAoQQjSeb7893+/re+FwixawdKlxxF5SAn5+xnBl0mrX7sLpO3RwaJU1NflbWlrKxIkTmTJlCjfddFO180qTv2aRkEBJkPFw2ey3F8pRuxBNTUICrF0Lzz5rvLroO6q1ZurUqfTp04cHH6z9GUHNtclfcx25A96WADhTiH/CFe4ORQhRnYQElx94/fDDD7z33nv069fPVoXyf//3f4wbN+6iaRMTE1m5cqXLm/ydPHkyu3btsjX527lz52qb/K2se69s8re4uBiA5557jp49ezotJtM1+duhXTuOnzzJ0cxMOnTp4oLIhBCVpMlf92pWTf5SYvzKlZ117mVDQgjhSUyX3Iv8jJCPqca7GUAIIczGdMnd19sPgNYB9bubTAghmhPTJXefCuM1UJvuXLAQQjQa0yV3rGeWS86ecXMgQgjRdJkuuZ/zN0LO9rr4xgAhhBAG0yV3pZTR46XcG4gQolFUNvkbExPDgAEDmDt3LhUVFS5d51VXXUWvXr0YMGAAgwcPrvYOWXu5ubm8+uqrLo2pvkyX3L3Kyowea/WMEMKzVTY/sHPnTtasWcNXX33FM888c9F0ZZW5wUmWLl3Kjh07uO+++3j44YdrnVaSuxN4lZYbPUWS3IVobtq3b8+iRYt45ZVX0FqzZMkSJkyYwNVXX83IkSMpKChg5MiRxMXF0a9fPz777DMAZs+ezcsvvwwYbcJcffXVAKxbt44pU6bUus6EhARbc8I1Lf+xxx5j3759DBw40PZDMHv2bAYPHkz//v1tTf82JtNdclIW4A8F5yAoyN2hCNGs/OUvf6mzeqK+Bg4cyPz58+s1T7du3SgvL+fEiROA0eRuSkoKwcHBlJWV8emnn9KqVStycnIYOnQoEyZMICkpiblz5/LnP/+Z5ORkiouLKS0t5fvvv2f48OG1ru/rr7/mhhtuAMBisVS7/BdeeIHU1FTb+7N69WrS09PZsmULWmsmTJjAd999V+e6nMl0yZ3KqnYlde5CCBg9ejTBwcGA0aDY448/znfffYeXlxdHjhzh+PHjDBo0iG3btpGfn4+/vz9xcXEkJyfz/fff247oq5oyZQolJSUUFBTYknZNy69q9erVrF69mtjYWMA44k9PT5fkXhtvW7WMXC0jRGOq7xG2q+zfvx9vb2/bwzECAwNt45YuXUp2djbbtm3D19eXyMhIioqK8PX1pWvXrixZsoRhw4bRv39/1q9fT0ZGRo1t5yxdupRBgwbx8MMP86c//YlPPvmkxuVXpbVm5syZ3H333a55ExzgUJ27UmqsUmqPUipDKfVYLdNNVEpppVSdjdo0lHdp5QnVEletQgjRRGVnZ3PPPffwwAMPnL9yzk5eXh7t27fH19eX9evXc/DgQdu4pKQk5syZw/Dhw0lKSuL1118nNja22uVUUkrx7LPPsnnzZnbv3l3j8u2b8wUYM2YMixcvtj067xqWzHIAABYtSURBVMiRI7ZqpMZS55G7UsobWAiMBrKArUqpFVrrtCrTBQH/C/zoikArlbawQME5dDWN7AshPE/lk5hKS0vx8fHht7/9bY1tuE+ZMoXx48fTr18/4uPj6d27t21cUlISf/vb30hISCAwMBCLxUJSUlKd6w8ICGDGjBnMnj2bF198sdrlh4SEkJiYSN++fbn22muZPXs2u3btsrUh37JlS95//33bv43GUGeTv0qpBGCW1nqMdXgmgNb6+SrTzQfWAA8DD2mta23Pt6FN/oaFh3HsyDHS0tPo00OaIhXClaTJX/dydZO/nYDDdsNZ1jL7lcUBnbXWX9a2IKXUNKVUslIqOTs724FVX8y3zPgxCqzwbtD8QgjRHFzyde5KKS9gHjCjrmm11ou01vFa6/jQ0NAGrU9bT16UFEp77kIIURNHkvsRoLPdcLi1rFIQ0Bf4VimVCQwFVrjqpGphgHHEfsrXuXejCSGq566ntTV3l/q+O5LctwJRSqmuSik/4HZghV0AeVrrdlrrSK11JLAZmFBXnXtDWXwsALT2b+2KxQsh7FgsFk6ePCkJvpFprTl58iQWi6XBy6jzahmtdZlS6gFgFeANLNZa71RK/RVI1lqvqH0JzuVjvc7dv9S1DQcJISA8PJysrCwaeo5MNJzFYiE8PLzB8zt0E5PWeiWwskrZUzVMe1WDo3EkFuszVIsLpT13IVyt8uYfYT6mazjsrMUIOddP/iYKIURNTJfca7ubTNiJjDTa32nsTgjRJJivbZniUqPn3Dn3BtJUNLWEah+PnIQTwm1Ml9y9yqwNh5U047ZlmlpCr4kjid6RbZEfCSHqzXTVMqWBxqVBulUrN0fiBmau+qhadVOfbZFqHyHqzXTJ3a/QuEM1YPlnbo6knqQ++7xL2RZPfD+EcAFzJfdFi7DkngUg8K8vwKJFbg7IypMSs9bmqQYx4/srRCMxV3JfvhxVZditmlpiqUzMl9Jd6rLcRRK9EBcwV3KfONHWq6sMN6qmlkTcnVjtXUqid9Y2SKIXwmTJfdo0KkKMZyWWznoapk1r3PU3pYTRFI6W6+JojPbTOPvfgCR60UyZK7kD3v7G1TJ+zz4Hl1/uuhW5st7cmdUnZlFT3I5siyR6IerNdMmdE8aTxivKy2HLFucn+Ev98ntiYnamS30vXJHoJdkLD2S6m5iU9SYm29dxyxYnLfgSv+DNPWm7g/17fqn7r7r5ZZ8KEzNXcnfFpY+S1D2DMxN9dcuR/SxMxlzJ/fnzz+R2ylftUqtfRNPk6kRfdR1CNEHmqnM/ehSnfFUXLarfl17qzc3LVftM6utFE2euI/fiYlvvBV/VyEjIzKz9i6Z1/b+IksQ9S9X9KUf1woOZ68gdqj9yP3iw7i9qfb7IGzfKF7Q5cNU/MrkSRzQB5jpydzVJ6EJOzAoPYdrk7vSviHzpRFWuqMaRKhzRSEyX3F3yJ1e+YMIRjXEVTtX1CNFA5qpz9/Nz7vLkyhfRUK68ekrq7IUTmOvI3dvb1lvn10mStmhMrroSp7plyWdbOMBcyb2oyLFqGfnwC3dzRRVObcuTz7yowlzVMm3b2norKnseeQS6dDE+8F26yIdcND1Vq3C8XPC1M/PTv4RLmCu5V1RceOTu5wcvvmjcwFRRYbwK0dSVlzfOHc+S8Js1c1XLFBbaeuVjKjyKK+vs61qu/Nv1SOZK7nYq6p5ECPNqrGRf07Il4ZueuaplfHwuPGL3Me1vkxD109iN10mVjumZK7lbLLUPC9GcNIWEL0m/yXIouSulxiql9iilMpRSj1Uz/kGlVJpSKkUptVYp1cX5oXLxTUzOvqlJCLNzR/PUkvSbpDqTu1LKG1gIXAtEA5OUUtFVJvsZiNda9wc+Bv7u7EABsFhs1TLaOiyEqIO7nkcgSd+tHDlyHwJkaK33a61LgA+B6+0n0Fqv11pXXsqyGQh3bphWbdqcX2eVYSFEPbjzATQ1JX1J/E7lSHLvBBy2G86yltVkKvBVdSOUUtOUUslKqeTs7GzHo6zk50eOtfe4dVgI4SRN4YljkvidxqknVJVSdwDxwOzqxmutF2mt47XW8aGhofVfwdSpRlLHqCdi6tQGRiqEcEh1Cd9dl0lK4q8XR5L7EaCz3XC4tewCSqlRwBPABK11cdXxTjFtmq23tF/MBcNCiEbUlJI+1J74m2nydyS5bwWilFJdlVJ+wO3ACvsJlFKxwBsYif2E88O8WFl4bTVDQgi3aGpJv1IzTP513gWktS5TSj0ArAK8gcVa651Kqb8CyVrrFRjVMC2Bj5TxRh3SWk9wYdwueBSTEMJlakvwTSG51hVDU/iBqieHbvHUWq8EVlYpe8quf5ST43IkpsZepRDCFZp64gfH4mhiOcm09++vWbXG3SEIIVzNDIm/UhP7ATBtcvf19XV3CEIId6orUTa15A+N+lQt0yb30aNHuzsEIURTZobkr5TLErxpk/vKlSvrnkgIIWpihuR/CczVKmQVsbGx5OXluTsMIYQnqumyzqZ0iWctTJ3ct2/fThtpX0YI4S6O/AAEBdU+v4uYOrlXUkqhlGLt2rXuDkUIIS6Un++WI3+PSO6VRo0aZUv0X375pa183bp1nDx5Uq6NF0I0G6Y9oVqX6667rsZxK1asYPz48Y0YjRBCNC7THbm37dDW1l9RUcHBgwfRWqO1Zt++fWzZsqXOZUyYMAGlFK+//rorQxVCCLcxXXLvNaQXAB06dEApRUREhG1ct27dGDx4sC3ZV1RUUFJSYhsuKipixowZtunvvfdelFI88MAD5OfnN/q2CCGEq5guuQeHBQMw/9X5dU6rlLrgTlZ/f3/mzJmD1prk5GRb+cKFC2ndujUhISE06CEiQgjRxJguuXv7eANwxfArLmk5gwYNQmtNenq6rezUqVO0b98epRQ33XQTJSUll7QOIYRwF9Mld+3ktn579OiB1pry8nJuv/12W/mnn36Kv78/Sim6du3K+PHjWbNmDeXl5U5dvxBCuILpknslhXNvDfby8mLZsmVorTl9+jQTJ060jcvMzOSLL77gmmuuwcfHh+7du7Nv3z6nrl8IIZzJtMndldq0acPHH39sOxFbXl7OunXr6N69OwD79++nR48eJCUlMXv2bPbs2ePmiIUQ4kKS3B3g5eXFiBEjyMjIQGtNamoqN954I7t27eKRRx6hd+/eBAYG8swzz7BlyxapuhFCuJ0k9waIiYnhk08+4dixY6xZs4bo6GgKCwuZNWsWl19+OT4+PowZM4YXXniBdevWcfr0aXeHLIRoZkx3h+oVEVfwBV/Qyr+Vu0PBx8eHUaNGsXPnTgCOHTvGunXreOWVV1i9ejWrV6+2TduiRQtuv/12oqOjuemmm4iMjESZvElRIUTTZbrk7qWMPxtNMTF26NCByZMnM3nyZMBI9qtWrWLXrl2kpKSwePFiAB566CFat27NlVdeydChQxk2bBiDBw+mRYsW7gxfCOFBTJfc008a16WfKz1HIIFujqZ2HTp04Pe//71tuLy8nI0bN/LLL7+wfPlydu/ezYoVKwDjX0B0dDRxcXHExMTQq1cv+vTpQ9euXfH29nbXJgghTMp0yT2nMAeAsooyN0dSf97e3iQlJZGUlMR9990HGDdObd68mY0bN7Jt2zZWrlzJkiVLbPNYLBZ69uxJdHQ0MTExxMTE0LNnT7p160ZAQICbtkQI0dSZLrkndE7gUz4lyL+WBvBNJDg4mHHjxjFu3Dhb2alTp9i7dy9paWmkpqayZ88eNm3axIcffnjBvOHh4fTo0YMePXoQFRVF9+7diYyMJDIykuDg4CZZdSWEaBymS+7NQXBwMEOHDmXo0KEXlBcUFLBr1y4yMjLIyMggPT2dffv2sWLFCk6cOHHBtC1btiQyMpLOnTsTHh5OREQEnTt3tr2Gh4djsVgac7OEEI3IdMl9Z7ZxZUphaWGTr3N3tpYtWzJ48GAGDx580bj8/Hz27dvHwYMHyczMtHWHDh0iOTm52gbRQkNDCQsLq7br0KEDYWFhhIaGEhQUJP8ChDAZ0yX3nLNGnXt5hdwoZK9Vq1bExsYSGxtb7fiioiKysrI4dOgQhw8f5tChQ2RlZXH06FGOHj1Kamoqx48fp6zs4nMZfn5+tGvXjtDQUNurfX/btm1p27Ytbdq0uaDfvkVOIUTjMl1yFw1jsVhs9fM1qaio4OTJk7aEf/z4cbKzs21dTk4O2dnZtn8CeXl5ta4zMDDwosTfqlUrWrVqRVBQkEP9gYGBeHnJvXZC1Jckd2Hj5eVlOyrv379/ndOXlJRw8uRJTp8+TW5uLqdPn7Z11Q1nZmaSn5/PmTNnyMvLq/ZfQnUCAwOr7Vq2bOlwucViISAg4KLXgIAAfH19pdpJeBxJ7qLB/Pz8bHX09aW1pri4mDNnzpCfn29L+vavlf1nz57l7NmzFBQU2PrPnj3LyZMnLyqvqKiodyxKqWoTf22v/v7++Pn5ueRV/qkIZ5DkLtxCKYXFYsFisRAaGuqUZVb+YNj/ABQUFFBYWEhRURHnzp2r87W6sjNnznDixIkLyouLiykpKaG4uLhBPyi18fb2tiV7X19fW+fj41PrsCPTNHQeb29v26t9/6WWyQ+Z65g2ucvfaFGV/Q9GSEhIo623vLz8gmRfn9e6piktLbV1ZWVlNQ4XFRVdNL62eRytEmsMDflhqGl8Zefl5XXRa3Vl7hoXHx9f6/kvZ3AouSulxgILAG/gLa31C1XG+wPvAoOAk8BtWutM54YqRNPk7e1NixYtTNU2kNaasrKyWn8w7MvKy8spLy+vtr+plJWVlVFSUkJFRQXl5eXVvjpjnDO89tpr7k/uSilvYCEwGsgCtiqlVmit0+wmmwqc1lr3UErdDrwI3OaKgIUQl67y4fG+vr7SjEU9VU32DfnB6NChg8vjdOTIfQiQobXeD6CU+hC4HrBP7tcDs6z9HwOvKKWU1tq5DzwFQlo03t9tIYSoqrLKxcenaddqO3I2oxNw2G44y1pW7TRa6zIgD7goCyulpimlkpVSydXdMemIm664ictHX04ri/vbcxdCiKaqUU9Va60Xaa3jtdbxDb1C4vrrr2fz6s3yV1IIIWrhSHI/AnS2Gw63llU7jVLKB2iNcWJVCCGEGziS3LcCUUqprkopP+B2YEWVaVYAlU+luBlY54r6diGEEI6p84yA1rpMKfUAsArjUsjFWuudSqm/Asla6xXAP4H3lFIZwCmMHwAhhBBu4tDpXq31SmBllbKn7PqLgFucG5oQQoiGknt/hRDCA0lyF0IIDyTJXQghPJAkdyGE8EDKXVcsKqWygYMNnL0dkOPEcMxAtrl5kG1uHi5lm7toreu8C9Rtyf1SKKWStdbx7o6jMck2Nw+yzc1DY2yzVMsIIYQHkuQuhBAeyKzJfZG7A3AD2ebmQba5eXD5Npuyzl0IIUTtzHrkLoQQohaS3IUQwgOZLrkrpcYqpfYopTKUUo+5O576UEp1VkqtV0qlKaV2KqX+11oerJRao5RKt762tZYrpdTL1m1NUUrF2S3r99bp05VSv7crH6SU+sU6z8tKKdX4W3oxpZS3UupnpdQX1uGuSqkfrXH+y9qcNEopf+twhnV8pN0yZlrL9yilxtiVN7nPhFKqjVLqY6XUbqXULqVUgqfvZ6XUdOvnOlUptUwpZfG0/ayUWqyUOqGUSrUrc/l+rWkdtdJam6bDaHJ4H9AN8AN2ANHujqse8YcBcdb+IGAvEA38HXjMWv4Y8KK1fxzwFaCAocCP1vJgYL/1ta21v6113BbrtMo677Xu3m5rXA8CHwBfWIf/Ddxu7X8duNfafx/wurX/duBf1v5o6/72B7paPwfeTfUzAbwD3GXt9wPaePJ+xnjU5gEgwG7/3ulp+xkYDsQBqXZlLt+vNa2j1ljd/SWo5xubAKyyG54JzHR3XJewPZ8Bo4E9QJi1LAzYY+1/A5hkN/0e6/hJwBt25W9Yy8KA3XblF0znxu0MB9YCVwNfWD+4OYBP1f2K8dyABGu/j3U6VXVfV07XFD8TGE8iO4D1goWq+88T9zPnn6McbN1vXwBjPHE/A5FcmNxdvl9rWkdtndmqZRx5WLcpWP+GxgI/ApdprY9aRx0DLrP217S9tZVnVVPubvOBR4AK63AIkKuNh6nDhXHW9LD1+r4X7tQVyAbetlZFvaWUCsSD97PW+ggwBzgEHMXYb9vw7P1cqTH2a03rqJHZkrtHUEq1BJYDf9Fa59uP08ZPs8dcn6qUug44obXe5u5YGpEPxl/317TWscBZjL/SNh64n9sC12P8sHUEAoGxbg3KDRpjvzq6DrMld0ce1t2kKaV8MRL7Uq31J9bi40qpMOv4MOCEtbym7a2tPLyacndKBCYopTKBDzGqZhYAbZTxMHW4MM6aHrZe3/fCnbKALK31j9bhjzGSvSfv51HAAa11tta6FPgEY9978n6u1Bj7taZ11Mhsyd2Rh3U3WdYz3/8Edmmt59mNsn/A+O8x6uIry39nPes+FMiz/jVbBVyjlGprPWK6BqM+8iiQr5Qaal3X7+yW5RZa65la63CtdSTG/lqntZ4CrMd4mDpcvM3VPWx9BXC79SqLrkAUxsmnJveZ0FofAw4rpXpZi0YCaXjwfsaojhmqlGphjalymz12P9tpjP1a0zpq5s6TMA08mTEO4yqTfcAT7o6nnrFfgfF3KgXYbu3GYdQ1rgXSgW+AYOv0Clho3dZfgHi7Zf0RyLB2f7ArjwdSrfO8QpWTem7e/qs4f7VMN4wvbQbwEeBvLbdYhzOs47vZzf+Edbv2YHd1SFP8TAADgWTrvv4PxlURHr2fgWeA3da43sO44sWj9jOwDOOcQinGP7SpjbFfa1pHbZ00PyCEEB7IbNUyQgghHCDJXQghPJAkdyGE8ECS3IUQwgNJchdCCA8kyV14HKVUuVJqu13ntBYElVKR9i0CCtFU+dQ9iRCmc05rPdDdQQjhTnLkLpoNpVSmUurv1vaytyileljLI5VS66xtbq9VSkVYyy9TSn2qlNph7YZZF+WtlHpTGW2Xr1ZKBVin/7My2upPUUp96KbNFAKQ5C48U0CVapnb7Mblaa37Ydz9N99a9g/gHa11f2Ap8LK1/GVgg9Z6AEbbMDut5VHAQq11DJALTLSWPwbEWpdzj6s2TghHyB2qwuMopQq01i2rKc8ErtZa77c24HZMax2ilMrBaCu71Fp+VGvdTimVDYRrrYvtlhEJrNFaR1mHHwV8tdbPKaW+Bgowmhv4j9a6wMWbKkSN5MhdNDe6hv76KLbrL+f8uavfYLQlEgdstWsNUYhGJ8ldNDe32b1usvZvxGhlEGAK8L21fy1wL9ieAdu6poUqpbyAzlrr9cCjGE3YXvTvQYjGIkcWwhMFKKW22w1/rbWuvByyrVIqBePoe5K17E8YT016GOMJSn+wlv8vsEgpNRXjCP1ejBYBq+MNvG/9AVDAy1rrXKdtkRD1JHXuotmw1rnHa61z3B2LEK4m1TJCCOGB5MhdCCE8kBy5CyGEB5LkLoQQHkiSuxBCeCBJ7kII4YEkuQshhAf6/zqybxRaomsSAAAAAElFTkSuQmCC\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "winRateP1[-1]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KTAibnh6pruo",
        "outputId": "8ab62131-da36-4881-a69c-9276b64163b0"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.83005"
            ]
          },
          "metadata": {},
          "execution_count": 308
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "winRateP2[-1]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0XZNsR0cpwAF",
        "outputId": "2f6c5df8-c665-4825-8af0-4c4bc6c09a40"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.15728"
            ]
          },
          "metadata": {},
          "execution_count": 309
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "drawRate[-1]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "W4Ghus8SpxnZ",
        "outputId": "55086091-1d0e-45b6-fa0f-303034eb026e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.01267"
            ]
          },
          "metadata": {},
          "execution_count": 310
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "winRateP1[-1] + winRateP2[-1] + drawRate[-1]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5JlxuaPSslRa",
        "outputId": "6b0bdac0-4c40-46f3-a6e8-dcfa146e1a9b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.9999999999999999"
            ]
          },
          "metadata": {},
          "execution_count": 311
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "def test(Qvals, epochs):\n",
        "    winP1 = 0\n",
        "    winP2 = 0\n",
        "    draw = 0\n",
        "    symbols = {1: 'X', 2: 'O'}\n",
        "\n",
        "    for epoch in range(epochs):\n",
        "        done = False\n",
        "        controls = reset()\n",
        "\n",
        "        boardVals = np.zeros((3,3))\n",
        "        turn = 0\n",
        "\n",
        "        board = np.array([[' ', ' ', ' '],\n",
        "                          [' ', ' ', ' '],\n",
        "                          [' ', ' ', ' ']])\n",
        "\n",
        "        if epoch % (epochs // 5) == 0:\n",
        "            print(f'Episode {epoch}, turn = {turn}, \\n', board)\n",
        "\n",
        "        while done != True:\n",
        "            if turn % 2 == 0:\n",
        "                # P1 = RL agent 1\n",
        "                u = control_test(tuple(boardVals.reshape(-1,).astype(int)),\n",
        "                                 controls, Qvals)\n",
        "                controls.remove(u)\n",
        "\n",
        "                boardValsNext = boardVals.reshape(-1,)\n",
        "                boardValsNext[u] = 1\n",
        "                boardValsNext = boardValsNext.reshape(3,3)\n",
        "\n",
        "                board = board.reshape(-1,)\n",
        "                board[u] = symbols[1]\n",
        "                board = board.reshape(3,3)\n",
        "\n",
        "                boardVals = boardValsNext\n",
        "                turn += 1\n",
        "\n",
        "                if epoch % (epochs // 5) == 0:\n",
        "                    print(f'Episode {epoch}, turn = {turn}, \\n', board)\n",
        "                    done, winner = check_over(boardVals, 1)\n",
        "                else:\n",
        "                    done, winner = check_over(boardVals)\n",
        "                \n",
        "                if done:\n",
        "                    if winner == '1':\n",
        "                        winP1 += 1\n",
        "                    elif winner == '2':\n",
        "                        winP2 += 1\n",
        "                    elif winner == 'd':\n",
        "                        draw += 1\n",
        "                \n",
        "            else:\n",
        "                # P2 - RL agent 2\n",
        "                u = np.random.choice(list(controls))\n",
        "                controls.remove(u)\n",
        "\n",
        "                boardVals = boardVals.reshape(-1,)\n",
        "                boardVals[u] = 2\n",
        "                boardVals = boardVals.reshape(3,3)\n",
        "\n",
        "                board = board.reshape(-1,)\n",
        "                board[u] = symbols[2]\n",
        "                board = board.reshape(3,3)\n",
        "                turn += 1\n",
        "\n",
        "                if epoch % (epochs // 5) == 0:\n",
        "                    print(f'Episode {epoch}, turn = {turn}, \\n', board)\n",
        "                    done, winner = check_over(boardVals, 1)\n",
        "                else:\n",
        "                    done, winner = check_over(boardVals)\n",
        "\n",
        "                if done:\n",
        "                    if winner == '1':\n",
        "                        winP1 += 1\n",
        "                    elif winner == '2':\n",
        "                        winP2 += 1\n",
        "                    elif winner == 'd':\n",
        "                        draw += 1\n",
        "\n",
        "    winRateP1 = winP1 / epochs\n",
        "    winRateP2 = winP2 / epochs\n",
        "    drawRate = draw / epochs\n",
        "\n",
        "    return winRateP1, winRateP2, drawRate"
      ],
      "metadata": {
        "id": "AEJ07FFqs0iV"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "testEpochs = 10000\n",
        "winRateP1, winRateP2, drawRate = test(QvalsP1, testEpochs)"
      ],
      "metadata": {
        "id": "WI1Iv0oAAoL3",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "26d2c000-e505-42a8-8d66-a05af18459c6"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Episode 0, turn = 0, \n",
            " [[' ' ' ' ' ']\n",
            " [' ' ' ' ' ']\n",
            " [' ' ' ' ' ']]\n",
            "Episode 0, turn = 1, \n",
            " [[' ' ' ' ' ']\n",
            " [' ' 'X' ' ']\n",
            " [' ' ' ' ' ']]\n",
            "Episode 0, turn = 2, \n",
            " [[' ' ' ' ' ']\n",
            " [' ' 'X' 'O']\n",
            " [' ' ' ' ' ']]\n",
            "Episode 0, turn = 3, \n",
            " [[' ' ' ' ' ']\n",
            " [' ' 'X' 'O']\n",
            " ['X' ' ' ' ']]\n",
            "Episode 0, turn = 4, \n",
            " [[' ' ' ' ' ']\n",
            " [' ' 'X' 'O']\n",
            " ['X' 'O' ' ']]\n",
            "Episode 0, turn = 5, \n",
            " [[' ' ' ' 'X']\n",
            " [' ' 'X' 'O']\n",
            " ['X' 'O' ' ']]\n",
            "PLAYER 1 WON!\n",
            "Episode 2000, turn = 0, \n",
            " [[' ' ' ' ' ']\n",
            " [' ' ' ' ' ']\n",
            " [' ' ' ' ' ']]\n",
            "Episode 2000, turn = 1, \n",
            " [[' ' ' ' ' ']\n",
            " [' ' 'X' ' ']\n",
            " [' ' ' ' ' ']]\n",
            "Episode 2000, turn = 2, \n",
            " [[' ' ' ' ' ']\n",
            " ['O' 'X' ' ']\n",
            " [' ' ' ' ' ']]\n",
            "Episode 2000, turn = 3, \n",
            " [[' ' ' ' 'X']\n",
            " ['O' 'X' ' ']\n",
            " [' ' ' ' ' ']]\n",
            "Episode 2000, turn = 4, \n",
            " [['O' ' ' 'X']\n",
            " ['O' 'X' ' ']\n",
            " [' ' ' ' ' ']]\n",
            "Episode 2000, turn = 5, \n",
            " [['O' ' ' 'X']\n",
            " ['O' 'X' ' ']\n",
            " [' ' ' ' 'X']]\n",
            "Episode 2000, turn = 6, \n",
            " [['O' 'O' 'X']\n",
            " ['O' 'X' ' ']\n",
            " [' ' ' ' 'X']]\n",
            "Episode 2000, turn = 7, \n",
            " [['O' 'O' 'X']\n",
            " ['O' 'X' ' ']\n",
            " [' ' 'X' 'X']]\n",
            "Episode 2000, turn = 8, \n",
            " [['O' 'O' 'X']\n",
            " ['O' 'X' 'O']\n",
            " [' ' 'X' 'X']]\n",
            "Episode 2000, turn = 9, \n",
            " [['O' 'O' 'X']\n",
            " ['O' 'X' 'O']\n",
            " ['X' 'X' 'X']]\n",
            "PLAYER 1 WON!\n",
            "Episode 4000, turn = 0, \n",
            " [[' ' ' ' ' ']\n",
            " [' ' ' ' ' ']\n",
            " [' ' ' ' ' ']]\n",
            "Episode 4000, turn = 1, \n",
            " [[' ' ' ' ' ']\n",
            " [' ' 'X' ' ']\n",
            " [' ' ' ' ' ']]\n",
            "Episode 4000, turn = 2, \n",
            " [[' ' 'O' ' ']\n",
            " [' ' 'X' ' ']\n",
            " [' ' ' ' ' ']]\n",
            "Episode 4000, turn = 3, \n",
            " [[' ' 'O' ' ']\n",
            " [' ' 'X' ' ']\n",
            " [' ' ' ' 'X']]\n",
            "Episode 4000, turn = 4, \n",
            " [[' ' 'O' ' ']\n",
            " [' ' 'X' ' ']\n",
            " ['O' ' ' 'X']]\n",
            "Episode 4000, turn = 5, \n",
            " [['X' 'O' ' ']\n",
            " [' ' 'X' ' ']\n",
            " ['O' ' ' 'X']]\n",
            "PLAYER 1 WON!\n",
            "Episode 6000, turn = 0, \n",
            " [[' ' ' ' ' ']\n",
            " [' ' ' ' ' ']\n",
            " [' ' ' ' ' ']]\n",
            "Episode 6000, turn = 1, \n",
            " [[' ' ' ' ' ']\n",
            " [' ' 'X' ' ']\n",
            " [' ' ' ' ' ']]\n",
            "Episode 6000, turn = 2, \n",
            " [['O' ' ' ' ']\n",
            " [' ' 'X' ' ']\n",
            " [' ' ' ' ' ']]\n",
            "Episode 6000, turn = 3, \n",
            " [['O' 'X' ' ']\n",
            " [' ' 'X' ' ']\n",
            " [' ' ' ' ' ']]\n",
            "Episode 6000, turn = 4, \n",
            " [['O' 'X' 'O']\n",
            " [' ' 'X' ' ']\n",
            " [' ' ' ' ' ']]\n",
            "Episode 6000, turn = 5, \n",
            " [['O' 'X' 'O']\n",
            " [' ' 'X' ' ']\n",
            " [' ' 'X' ' ']]\n",
            "PLAYER 1 WON!\n",
            "Episode 8000, turn = 0, \n",
            " [[' ' ' ' ' ']\n",
            " [' ' ' ' ' ']\n",
            " [' ' ' ' ' ']]\n",
            "Episode 8000, turn = 1, \n",
            " [[' ' ' ' ' ']\n",
            " [' ' 'X' ' ']\n",
            " [' ' ' ' ' ']]\n",
            "Episode 8000, turn = 2, \n",
            " [[' ' ' ' ' ']\n",
            " [' ' 'X' ' ']\n",
            " [' ' 'O' ' ']]\n",
            "Episode 8000, turn = 3, \n",
            " [['X' ' ' ' ']\n",
            " [' ' 'X' ' ']\n",
            " [' ' 'O' ' ']]\n",
            "Episode 8000, turn = 4, \n",
            " [['X' ' ' ' ']\n",
            " [' ' 'X' 'O']\n",
            " [' ' 'O' ' ']]\n",
            "Episode 8000, turn = 5, \n",
            " [['X' ' ' ' ']\n",
            " [' ' 'X' 'O']\n",
            " [' ' 'O' 'X']]\n",
            "PLAYER 1 WON!\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "winRateP1"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Nz7-KqqvA4tL",
        "outputId": "1923ddf7-ddec-4137-ffdc-53673d2ca8c3"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.8986"
            ]
          },
          "metadata": {},
          "execution_count": 314
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "winRateP2"
      ],
      "metadata": {
        "id": "9m3pClSaA5xO"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "drawRate"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dPOPVwaRA6vk",
        "outputId": "ca3cc33b-62a6-4bc5-da63-62c3e088a59b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.0071"
            ]
          },
          "metadata": {},
          "execution_count": 315
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "winRateP1 + winRateP2 + drawRate"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "E0c1VGSXAx5S",
        "outputId": "22e4a54d-29b3-4459-8985-d5e18b736717"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.9999999999999999"
            ]
          },
          "metadata": {},
          "execution_count": 316
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "def testHuman(Qvals):\n",
        "    winP1 = 0\n",
        "    winP2 = 0\n",
        "    draw = 0\n",
        "    symbols = {1: 'X', 2: 'O'}\n",
        "    round = 0\n",
        "\n",
        "    while int(input('Play (input 1) or exit (input 0)? ')) != 0:\n",
        "        round += 1\n",
        "        print('Round', round)\n",
        "        done = False\n",
        "        controls = reset()\n",
        "\n",
        "        boardVals = np.zeros((3,3))\n",
        "        turn = 0\n",
        "\n",
        "        board = np.array([[' ', ' ', ' '],\n",
        "                          [' ', ' ', ' '],\n",
        "                          [' ', ' ', ' ']])\n",
        "\n",
        "        print(f'Turn = {turn}, \\n', board)\n",
        "\n",
        "        while done != True:\n",
        "            if turn % 2 == 0:\n",
        "                # P1 - RL agent 1\n",
        "                u = control_test(tuple(boardVals.reshape(-1,).astype(int)),\n",
        "                                 controls, Qvals)\n",
        "                controls.remove(u)\n",
        "\n",
        "                boardValsNext = boardVals.reshape(-1,)\n",
        "                boardValsNext[u] = 1\n",
        "                boardValsNext = boardValsNext.reshape(3,3)\n",
        "\n",
        "                board = board.reshape(-1,)\n",
        "                board[u] = symbols[1]\n",
        "                board = board.reshape(3,3)\n",
        "\n",
        "                boardVals = boardValsNext\n",
        "                turn += 1\n",
        "\n",
        "                print(f'Turn = {turn}, \\n', board)\n",
        "\n",
        "                done, winner = check_over(boardVals, 1)\n",
        "                if done:\n",
        "                    if winner == '1':\n",
        "                        winP1 += 1\n",
        "                    elif winner == '2':\n",
        "                        winP2 += 1\n",
        "                    elif winner == 'd':\n",
        "                        draw += 1\n",
        "\n",
        "            else:\n",
        "                # P2 - human\n",
        "                u = int(input('Enter a square to place O (1-9): ')) - 1\n",
        "                controls.remove(u)\n",
        "\n",
        "                boardVals = boardVals.reshape(-1,)\n",
        "                boardVals[u] = 2\n",
        "                boardVals = boardVals.reshape(3,3)\n",
        "\n",
        "                board = board.reshape(-1,)\n",
        "                board[u] = symbols[2]\n",
        "                board = board.reshape(3,3)\n",
        "                turn += 1\n",
        "\n",
        "                print(f'Turn = {turn}, \\n', board)\n",
        "\n",
        "                done, winner = check_over(boardVals, 1)\n",
        "                if done:\n",
        "                    if winner == '1':\n",
        "                        winP1 += 1\n",
        "                    elif winner == '2':\n",
        "                        winP2 += 1\n",
        "                    elif winner == 'd':\n",
        "                        draw += 1\n",
        "    print('Game Over')\n",
        "    return winP1, winP2, draw, round"
      ],
      "metadata": {
        "id": "S952QhfVsIuI"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "\n",
        "print('Welcome to Tic-Tac-Toe! AI is P1, you are P2.')\n",
        "winP1, winP2, draw, rounds = testHuman(QvalsP1)\n",
        "\n",
        "if rounds == 0:\n",
        "    print('No games played; no stats reported.')\n",
        "else:\n",
        "    winRateP1 = winP1 / rounds\n",
        "    winRateP2 = winP2 / rounds\n",
        "    drawRate = draw / rounds\n",
        "    print('Rounds:       ', rounds)\n",
        "    print('AI Wins:      ', winP1)\n",
        "    print('AI Win Rate:  ', winRateP1)\n",
        "    print('Your Wins:    ', winP2)\n",
        "    print('Your Win Rate:', winRateP2)\n",
        "    print('Draws:        ', draw)\n",
        "    print('Draw Rate:    ', drawRate)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lk5VhJKu18J0",
        "outputId": "c016106d-9744-4200-dccc-90e2b2990e93"
      },
      "execution_count": 321,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Welcome to Tic-Tac-Toe! AI is P1, you are P2.\n",
            "Play (input 1) or exit (input 0)? 1\n",
            "Round 1\n",
            "Turn = 0, \n",
            " [[' ' ' ' ' ']\n",
            " [' ' ' ' ' ']\n",
            " [' ' ' ' ' ']]\n",
            "Turn = 1, \n",
            " [[' ' ' ' ' ']\n",
            " [' ' 'X' ' ']\n",
            " [' ' ' ' ' ']]\n",
            "Enter a square to place O (1-9): 4\n",
            "Turn = 2, \n",
            " [[' ' ' ' ' ']\n",
            " ['O' 'X' ' ']\n",
            " [' ' ' ' ' ']]\n",
            "Turn = 3, \n",
            " [[' ' ' ' 'X']\n",
            " ['O' 'X' ' ']\n",
            " [' ' ' ' ' ']]\n",
            "Enter a square to place O (1-9): 7\n",
            "Turn = 4, \n",
            " [[' ' ' ' 'X']\n",
            " ['O' 'X' ' ']\n",
            " ['O' ' ' ' ']]\n",
            "Turn = 5, \n",
            " [['X' ' ' 'X']\n",
            " ['O' 'X' ' ']\n",
            " ['O' ' ' ' ']]\n",
            "Enter a square to place O (1-9): 9\n",
            "Turn = 6, \n",
            " [['X' ' ' 'X']\n",
            " ['O' 'X' ' ']\n",
            " ['O' ' ' 'O']]\n",
            "Turn = 7, \n",
            " [['X' 'X' 'X']\n",
            " ['O' 'X' ' ']\n",
            " ['O' ' ' 'O']]\n",
            "PLAYER 1 WON!\n",
            "Play (input 1) or exit (input 0)? 1\n",
            "Round 2\n",
            "Turn = 0, \n",
            " [[' ' ' ' ' ']\n",
            " [' ' ' ' ' ']\n",
            " [' ' ' ' ' ']]\n",
            "Turn = 1, \n",
            " [[' ' ' ' ' ']\n",
            " [' ' 'X' ' ']\n",
            " [' ' ' ' ' ']]\n",
            "Enter a square to place O (1-9): 3\n",
            "Turn = 2, \n",
            " [[' ' ' ' 'O']\n",
            " [' ' 'X' ' ']\n",
            " [' ' ' ' ' ']]\n",
            "Turn = 3, \n",
            " [[' ' ' ' 'O']\n",
            " [' ' 'X' ' ']\n",
            " [' ' 'X' ' ']]\n",
            "Enter a square to place O (1-9): 2\n",
            "Turn = 4, \n",
            " [[' ' 'O' 'O']\n",
            " [' ' 'X' ' ']\n",
            " [' ' 'X' ' ']]\n",
            "Turn = 5, \n",
            " [[' ' 'O' 'O']\n",
            " ['X' 'X' ' ']\n",
            " [' ' 'X' ' ']]\n",
            "Enter a square to place O (1-9): 1\n",
            "Turn = 6, \n",
            " [['O' 'O' 'O']\n",
            " ['X' 'X' ' ']\n",
            " [' ' 'X' ' ']]\n",
            "PLAYER 2 WON!\n",
            "Play (input 1) or exit (input 0)? 1\n",
            "Round 3\n",
            "Turn = 0, \n",
            " [[' ' ' ' ' ']\n",
            " [' ' ' ' ' ']\n",
            " [' ' ' ' ' ']]\n",
            "Turn = 1, \n",
            " [[' ' ' ' ' ']\n",
            " [' ' 'X' ' ']\n",
            " [' ' ' ' ' ']]\n",
            "Enter a square to place O (1-9): 6\n",
            "Turn = 2, \n",
            " [[' ' ' ' ' ']\n",
            " [' ' 'X' 'O']\n",
            " [' ' ' ' ' ']]\n",
            "Turn = 3, \n",
            " [[' ' ' ' ' ']\n",
            " [' ' 'X' 'O']\n",
            " ['X' ' ' ' ']]\n",
            "Enter a square to place O (1-9): 3\n",
            "Turn = 4, \n",
            " [[' ' ' ' 'O']\n",
            " [' ' 'X' 'O']\n",
            " ['X' ' ' ' ']]\n",
            "Turn = 5, \n",
            " [['X' ' ' 'O']\n",
            " [' ' 'X' 'O']\n",
            " ['X' ' ' ' ']]\n",
            "Enter a square to place O (1-9): 4\n",
            "Turn = 6, \n",
            " [['X' ' ' 'O']\n",
            " ['O' 'X' 'O']\n",
            " ['X' ' ' ' ']]\n",
            "Turn = 7, \n",
            " [['X' ' ' 'O']\n",
            " ['O' 'X' 'O']\n",
            " ['X' ' ' 'X']]\n",
            "PLAYER 1 WON!\n",
            "Play (input 1) or exit (input 0)? 1\n",
            "Round 4\n",
            "Turn = 0, \n",
            " [[' ' ' ' ' ']\n",
            " [' ' ' ' ' ']\n",
            " [' ' ' ' ' ']]\n",
            "Turn = 1, \n",
            " [[' ' ' ' ' ']\n",
            " [' ' 'X' ' ']\n",
            " [' ' ' ' ' ']]\n",
            "Enter a square to place O (1-9): 7\n",
            "Turn = 2, \n",
            " [[' ' ' ' ' ']\n",
            " [' ' 'X' ' ']\n",
            " ['O' ' ' ' ']]\n",
            "Turn = 3, \n",
            " [[' ' ' ' ' ']\n",
            " [' ' 'X' 'X']\n",
            " ['O' ' ' ' ']]\n",
            "Enter a square to place O (1-9): 4\n",
            "Turn = 4, \n",
            " [[' ' ' ' ' ']\n",
            " ['O' 'X' 'X']\n",
            " ['O' ' ' ' ']]\n",
            "Turn = 5, \n",
            " [[' ' ' ' ' ']\n",
            " ['O' 'X' 'X']\n",
            " ['O' ' ' 'X']]\n",
            "Enter a square to place O (1-9): 1\n",
            "Turn = 6, \n",
            " [['O' ' ' ' ']\n",
            " ['O' 'X' 'X']\n",
            " ['O' ' ' 'X']]\n",
            "PLAYER 2 WON!\n",
            "Play (input 1) or exit (input 0)? 1\n",
            "Round 5\n",
            "Turn = 0, \n",
            " [[' ' ' ' ' ']\n",
            " [' ' ' ' ' ']\n",
            " [' ' ' ' ' ']]\n",
            "Turn = 1, \n",
            " [[' ' ' ' ' ']\n",
            " [' ' 'X' ' ']\n",
            " [' ' ' ' ' ']]\n",
            "Enter a square to place O (1-9): 2\n",
            "Turn = 2, \n",
            " [[' ' 'O' ' ']\n",
            " [' ' 'X' ' ']\n",
            " [' ' ' ' ' ']]\n",
            "Turn = 3, \n",
            " [[' ' 'O' ' ']\n",
            " [' ' 'X' ' ']\n",
            " [' ' ' ' 'X']]\n",
            "Enter a square to place O (1-9): 1\n",
            "Turn = 4, \n",
            " [['O' 'O' ' ']\n",
            " [' ' 'X' ' ']\n",
            " [' ' ' ' 'X']]\n",
            "Turn = 5, \n",
            " [['O' 'O' ' ']\n",
            " [' ' 'X' 'X']\n",
            " [' ' ' ' 'X']]\n",
            "Enter a square to place O (1-9): 3\n",
            "Turn = 6, \n",
            " [['O' 'O' 'O']\n",
            " [' ' 'X' 'X']\n",
            " [' ' ' ' 'X']]\n",
            "PLAYER 2 WON!\n",
            "Play (input 1) or exit (input 0)? 1\n",
            "Round 6\n",
            "Turn = 0, \n",
            " [[' ' ' ' ' ']\n",
            " [' ' ' ' ' ']\n",
            " [' ' ' ' ' ']]\n",
            "Turn = 1, \n",
            " [[' ' ' ' ' ']\n",
            " [' ' 'X' ' ']\n",
            " [' ' ' ' ' ']]\n",
            "Enter a square to place O (1-9): 4\n",
            "Turn = 2, \n",
            " [[' ' ' ' ' ']\n",
            " ['O' 'X' ' ']\n",
            " [' ' ' ' ' ']]\n",
            "Turn = 3, \n",
            " [[' ' ' ' 'X']\n",
            " ['O' 'X' ' ']\n",
            " [' ' ' ' ' ']]\n",
            "Enter a square to place O (1-9): 7\n",
            "Turn = 4, \n",
            " [[' ' ' ' 'X']\n",
            " ['O' 'X' ' ']\n",
            " ['O' ' ' ' ']]\n",
            "Turn = 5, \n",
            " [['X' ' ' 'X']\n",
            " ['O' 'X' ' ']\n",
            " ['O' ' ' ' ']]\n",
            "Enter a square to place O (1-9): 2\n",
            "Turn = 6, \n",
            " [['X' 'O' 'X']\n",
            " ['O' 'X' ' ']\n",
            " ['O' ' ' ' ']]\n",
            "Turn = 7, \n",
            " [['X' 'O' 'X']\n",
            " ['O' 'X' ' ']\n",
            " ['O' ' ' 'X']]\n",
            "PLAYER 1 WON!\n",
            "Play (input 1) or exit (input 0)? 0\n",
            "Game Over\n",
            "Rounds:        6\n",
            "AI Wins:       3\n",
            "AI Win Rate:   0.5\n",
            "Your Wins:     3\n",
            "Your Win Rate: 0.5\n",
            "Draws:         0\n",
            "Draw Rate:     0.0\n"
          ]
        }
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": [],
      "collapsed_sections": [
        "B670T7WxoolH",
        "0MpJA-hBopCC"
      ],
      "include_colab_link": true
    },
    "gpuClass": "standard",
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.6"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}