Copy_of_unit3.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Ryukijano/Deep-Reinforcement-Learning-and-Hugging-Face/blob/master/Copy_of_unit3.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k7xBVPzoXxOg"
      },
      "source": [
        "# Unit 3: Deep Q-Learning with Atari Games 👾 using RL Baselines3 Zoo\n",
        "\n",
        "<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/unit4/thumbnail.jpg\" alt=\"Unit 3 Thumbnail\">\n",
        "\n",
        "In this notebook, **you'll train a Deep Q-Learning agent** playing Space Invaders using [RL Baselines3 Zoo](https://github.com/DLR-RM/rl-baselines3-zoo), a training framework based on [Stable-Baselines3](https://stable-baselines3.readthedocs.io/en/master/) that provides scripts for training, evaluating agents, tuning hyperparameters, plotting results and recording videos.\n",
        "\n",
        "We're using the [RL-Baselines-3 Zoo integration, a vanilla version of Deep Q-Learning](https://stable-baselines3.readthedocs.io/en/master/modules/dqn.html) with no extensions such as Double-DQN, Dueling-DQN, and Prioritized Experience Replay.\n",
        "\n",
        "⬇️ Here is an example of what **you will achieve** ⬇️"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "J9S713biXntc",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 231
        },
        "outputId": "79e7f19d-fa7d-474a-84af-4162ceb81648"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "<video controls autoplay><source src=\"https://huggingface.co/ThomasSimonini/ppo-SpaceInvadersNoFrameskip-v4/resolve/main/replay.mp4\" type=\"video/mp4\"></video>\n"
            ]
          },
          "metadata": {}
        }
      ],
      "source": [
        "%%html\n",
        "<video controls autoplay><source src=\"https://huggingface.co/ThomasSimonini/ppo-SpaceInvadersNoFrameskip-v4/resolve/main/replay.mp4\" type=\"video/mp4\"></video>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 🎮 Environments: \n",
        "\n",
        "- SpacesInvadersNoFrameskip-v4 \n",
        "\n",
        "### 📚 RL-Library: \n",
        "\n",
        "- [RL-Baselines3-Zoo](https://github.com/DLR-RM/rl-baselines3-zoo)"
      ],
      "metadata": {
        "id": "ykJiGevCMVc5"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wciHGjrFYz9m"
      },
      "source": [
        "## Objectives of this notebook 🏆\n",
        "At the end of the notebook, you will:\n",
        "- Be able to understand deeper **how RL Baselines3 Zoo works**.\n",
        "- Be able to **push your trained agent and the code to the Hub** with a nice video replay and an evaluation score 🔥.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## This notebook is from Deep Reinforcement Learning Course\n",
        "<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/deep-rl-course-illustration.jpg\" alt=\"Deep RL Course illustration\"/>"
      ],
      "metadata": {
        "id": "TsnP0rjxMn1e"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nw6fJHIAZd-J"
      },
      "source": [
        "In this free course, you will:\n",
        "\n",
        "- 📖 Study Deep Reinforcement Learning in **theory and practice**.\n",
        "- 🧑‍💻 Learn to **use famous Deep RL libraries** such as Stable Baselines3, RL Baselines3 Zoo, CleanRL and Sample Factory 2.0.\n",
        "- 🤖 Train **agents in unique environments** \n",
        "\n",
        "And more check 📚 the syllabus 👉 https://simoninithomas.github.io/deep-rl-course\n",
        "\n",
        "Don’t forget to **<a href=\"http://eepurl.com/ic5ZUD\">sign up to the course</a>** (we are collecting your email to be able to **send you the links when each Unit is published and give you information about the challenges and updates).**\n",
        "\n",
        "\n",
        "The best way to keep in touch is to join our discord server to exchange with the community and with us 👉🏻 https://discord.gg/ydHrjt3WP5"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0vgANIBBZg1p"
      },
      "source": [
        "## Prerequisites 🏗️\n",
        "Before diving into the notebook, you need to:\n",
        "\n",
        "🔲 📚 **[Study Deep Q-Learning by reading Unit 3](https://huggingface.co/deep-rl-course/unit3/introduction)**  🤗 "
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We're constantly trying to improve our tutorials, so **if you find some issues in this notebook**, please [open an issue on the Github Repo](https://github.com/huggingface/deep-rl-class/issues)."
      ],
      "metadata": {
        "id": "7kszpGFaRVhq"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QR0jZtYreSI5"
      },
      "source": [
        "# Let's train a Deep Q-Learning agent playing Atari' Space Invaders 👾 and upload it to the Hub.\n",
        "\n",
        "To validate this hands-on for the certification process, you need to push your trained model to the Hub and **get a result of >= 200**.\n",
        "\n",
        "To find your result, go to the leaderboard and find your model, **the result = mean_reward - std of reward**\n",
        "\n",
        "For more information about the certification process, check this section 👉 https://huggingface.co/deep-rl-course/en/unit0/introduction#certification-process"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## An advice 💡\n",
        "It's better to run this colab in a copy on your Google Drive, so that **if it timeouts** you still have the saved notebook on your Google Drive and do not need to fill everything from scratch.\n",
        "\n",
        "To do that you can either do `Ctrl + S` or `File > Save a copy in Google Drive.`\n",
        "\n",
        "Also, we're going to **train it for 90 minutes with 1M timesteps**. By typing `!nvidia-smi` will tell you what GPU you're using.\n",
        "\n",
        "And if you want to train more such 10 million steps, this will take about 9 hours, potentially resulting in Colab timing out. In that case, I recommend running this on your local computer (or somewhere else). Just click on: `File>Download`. "
      ],
      "metadata": {
        "id": "Nc8BnyVEc3Ys"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Set the GPU 💪\n",
        "- To **accelerate the agent's training, we'll use a GPU**. To do that, go to `Runtime > Change Runtime type`\n",
        "\n",
        "<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/gpu-step1.jpg\" alt=\"GPU Step 1\">"
      ],
      "metadata": {
        "id": "PU4FVzaoM6fC"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "- `Hardware Accelerator > GPU`\n",
        "\n",
        "<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/gpu-step2.jpg\" alt=\"GPU Step 2\">"
      ],
      "metadata": {
        "id": "KV0NyFdQM9ZG"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Create a virtual display 🔽\n",
        "\n",
        "During the notebook, we'll need to generate a replay video. To do so, with colab, **we need to have a virtual screen to be able to render the environment** (and thus record the frames). \n",
        "\n",
        "Hence the following cell will install the librairies and create and run a virtual screen 🖥"
      ],
      "metadata": {
        "id": "bTpYcVZVMzUI"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jV6wjQ7Be7p5"
      },
      "outputs": [],
      "source": [
        "%%capture\n",
        "!apt install python-opengl\n",
        "!apt install ffmpeg\n",
        "!apt install xvfb\n",
        "!pip3 install pyvirtualdisplay"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Additional dependencies for RL Baselines3 Zoo\n",
        "!apt-get install swig cmake freeglut3-dev "
      ],
      "metadata": {
        "id": "fWyKJCy_NJBX",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d7760ce7-a0da-4a83-ffd0-35f7426ad5fc"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Reading package lists... Done\n",
            "Building dependency tree       \n",
            "Reading state information... Done\n",
            "cmake is already the newest version (3.10.2-1ubuntu2.18.04.2).\n",
            "The following package was automatically installed and is no longer required:\n",
            "  libnvidia-common-460\n",
            "Use 'apt autoremove' to remove it.\n",
            "Suggested packages:\n",
            "  swig-doc swig-examples swig3.0-examples swig3.0-doc\n",
            "The following NEW packages will be installed:\n",
            "  freeglut3-dev swig swig3.0\n",
            "0 upgraded, 3 newly installed, 0 to remove and 21 not upgraded.\n",
            "Need to get 1,224 kB of archives.\n",
            "After this operation, 6,583 kB of additional disk space will be used.\n",
            "Get:1 http://archive.ubuntu.com/ubuntu bionic/universe amd64 freeglut3-dev amd64 2.8.1-3 [124 kB]\n",
            "Get:2 http://archive.ubuntu.com/ubuntu bionic/universe amd64 swig3.0 amd64 3.0.12-1 [1,094 kB]\n",
            "Get:3 http://archive.ubuntu.com/ubuntu bionic/universe amd64 swig amd64 3.0.12-1 [6,460 B]\n",
            "Fetched 1,224 kB in 2s (697 kB/s)\n",
            "Selecting previously unselected package freeglut3-dev:amd64.\n",
            "(Reading database ... 126383 files and directories currently installed.)\n",
            "Preparing to unpack .../freeglut3-dev_2.8.1-3_amd64.deb ...\n",
            "Unpacking freeglut3-dev:amd64 (2.8.1-3) ...\n",
            "Selecting previously unselected package swig3.0.\n",
            "Preparing to unpack .../swig3.0_3.0.12-1_amd64.deb ...\n",
            "Unpacking swig3.0 (3.0.12-1) ...\n",
            "Selecting previously unselected package swig.\n",
            "Preparing to unpack .../swig_3.0.12-1_amd64.deb ...\n",
            "Unpacking swig (3.0.12-1) ...\n",
            "Setting up swig3.0 (3.0.12-1) ...\n",
            "Setting up freeglut3-dev:amd64 (2.8.1-3) ...\n",
            "Setting up swig (3.0.12-1) ...\n",
            "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install pyglet==1.5.1"
      ],
      "metadata": {
        "id": "C5LwHrISW7Q5",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "dee08390-e362-4c57-c9a5-a1b286c10db0"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting pyglet==1.5.1\n",
            "  Downloading pyglet-1.5.1-py2.py3-none-any.whl (1.0 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.0/1.0 MB\u001b[0m \u001b[31m57.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hInstalling collected packages: pyglet\n",
            "Successfully installed pyglet-1.5.1\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Virtual display\n",
        "from pyvirtualdisplay import Display\n",
        "\n",
        "virtual_display = Display(visible=0, size=(1400, 900))\n",
        "virtual_display.start()"
      ],
      "metadata": {
        "id": "ww5PQH1gNLI4",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d22a053b-ee0b-4274-e74c-e9575994fdb3"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<pyvirtualdisplay.display.Display at 0x7f8998b202b0>"
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mYIMvl5X9NAu"
      },
      "source": [
        "## Clone RL-Baselines3 Zoo Repo 📚\n",
        "You can now directly install from python package `pip install rl_zoo3` but since we want **the full installation with extra environments and dependencies** we're going to clone `RL-Baselines3-Zoo` repository and install from source."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "eu5ZDPZ09VNQ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6e04d71c-66cc-4ba9-d46b-dba9c32ef3a0"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Cloning into 'rl-baselines3-zoo'...\n",
            "remote: Enumerating objects: 4823, done.\u001b[K\n",
            "remote: Counting objects: 100% (116/116), done.\u001b[K\n",
            "remote: Compressing objects: 100% (60/60), done.\u001b[K\n",
            "remote: Total 4823 (delta 67), reused 95 (delta 53), pack-reused 4707\u001b[K\n",
            "Receiving objects: 100% (4823/4823), 3.04 MiB | 10.89 MiB/s, done.\n",
            "Resolving deltas: 100% (3188/3188), done.\n"
          ]
        }
      ],
      "source": [
        "!git clone https://github.com/DLR-RM/rl-baselines3-zoo"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HCIoSbvbfAQh"
      },
      "source": [
        "## Install dependencies 🔽\n",
        "We can now install the dependencies RL-Baselines3 Zoo needs (this can take 5min ⏲)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "s2QsFAk29h-D",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "fcc5a510-c872-4685-f907-56a0b1fc7878"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/content/rl-baselines3-zoo\n"
          ]
        }
      ],
      "source": [
        "%cd /content/rl-baselines3-zoo/"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3QaOS7Xj9j1s",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "eda11cbc-32b4-44ce-c889-db7eefacfa2c"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting gym==0.21\n",
            "  Downloading gym-0.21.0.tar.gz (1.5 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.5/1.5 MB\u001b[0m \u001b[31m50.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting stable-baselines3[docs,extra,tests]>=1.7.0\n",
            "  Downloading stable_baselines3-1.7.0-py3-none-any.whl (171 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m171.8/171.8 KB\u001b[0m \u001b[31m22.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting sb3-contrib>=1.7.0\n",
            "  Downloading sb3_contrib-1.7.0-py3-none-any.whl (79 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m79.1/79.1 KB\u001b[0m \u001b[31m11.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting box2d-py==2.3.8\n",
            "  Downloading box2d-py-2.3.8.tar.gz (374 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m374.5/374.5 KB\u001b[0m \u001b[31m37.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting pybullet\n",
            "  Downloading pybullet-3.2.5-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl (91.7 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m91.7/91.7 MB\u001b[0m \u001b[31m10.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting gym-minigrid\n",
            "  Downloading gym_minigrid-1.2.2-py3-none-any.whl (70 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m70.5/70.5 KB\u001b[0m \u001b[31m8.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting scikit-optimize\n",
            "  Downloading scikit_optimize-0.9.0-py2.py3-none-any.whl (100 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m100.3/100.3 KB\u001b[0m \u001b[31m12.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting optuna\n",
            "  Downloading optuna-3.0.5-py3-none-any.whl (348 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m348.5/348.5 KB\u001b[0m \u001b[31m26.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting pytablewriter~=0.64\n",
            "  Downloading pytablewriter-0.64.2-py3-none-any.whl (106 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m106.6/106.6 KB\u001b[0m \u001b[31m15.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 10)) (6.0)\n",
            "Requirement already satisfied: cloudpickle>=1.5.0 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 11)) (2.2.0)\n",
            "Requirement already satisfied: plotly in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 12)) (5.5.0)\n",
            "Collecting panda-gym==1.1.1\n",
            "  Downloading panda_gym-1.1.1-py3-none-any.whl (21 kB)\n",
            "Collecting rliable>=1.0.5\n",
            "  Downloading rliable-1.0.8-py3-none-any.whl (19 kB)\n",
            "Collecting wandb\n",
            "  Downloading wandb-0.13.9-py2.py3-none-any.whl (2.0 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.0/2.0 MB\u001b[0m \u001b[31m82.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting huggingface_sb3<3.*,>=2.2.1\n",
            "  Downloading huggingface_sb3-2.2.4-py3-none-any.whl (9.4 kB)\n",
            "Requirement already satisfied: seaborn in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 17)) (0.11.2)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 18)) (4.64.1)\n",
            "Collecting rich\n",
            "  Downloading rich-13.1.0-py3-none-any.whl (238 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m238.4/238.4 KB\u001b[0m \u001b[31m21.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting importlib-metadata~=4.13\n",
            "  Downloading importlib_metadata-4.13.0-py3-none-any.whl (23 kB)\n",
            "Requirement already satisfied: numpy>=1.18.0 in /usr/local/lib/python3.8/dist-packages (from gym==0.21->-r requirements.txt (line 1)) (1.21.6)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.8/dist-packages (from stable-baselines3[docs,extra,tests]>=1.7.0->-r requirements.txt (line 2)) (1.3.5)\n",
            "Requirement already satisfied: matplotlib in /usr/local/lib/python3.8/dist-packages (from stable-baselines3[docs,extra,tests]>=1.7.0->-r requirements.txt (line 2)) (3.2.2)\n",
            "Requirement already satisfied: torch>=1.11 in /usr/local/lib/python3.8/dist-packages (from stable-baselines3[docs,extra,tests]>=1.7.0->-r requirements.txt (line 2)) (1.13.0+cu116)\n",
            "Requirement already satisfied: pillow in /usr/local/lib/python3.8/dist-packages (from stable-baselines3[docs,extra,tests]>=1.7.0->-r requirements.txt (line 2)) (7.1.2)\n",
            "Requirement already satisfied: tensorboard>=2.9.1 in /usr/local/lib/python3.8/dist-packages (from stable-baselines3[docs,extra,tests]>=1.7.0->-r requirements.txt (line 2)) (2.9.1)\n",
            "Requirement already satisfied: psutil in /usr/local/lib/python3.8/dist-packages (from stable-baselines3[docs,extra,tests]>=1.7.0->-r requirements.txt (line 2)) (5.4.8)\n",
            "Collecting autorom[accept-rom-license]~=0.4.2\n",
            "  Downloading AutoROM-0.4.2-py3-none-any.whl (16 kB)\n",
            "Collecting ale-py==0.7.4\n",
            "  Downloading ale_py-0.7.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.6 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.6/1.6 MB\u001b[0m \u001b[31m74.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: opencv-python in /usr/local/lib/python3.8/dist-packages (from stable-baselines3[docs,extra,tests]>=1.7.0->-r requirements.txt (line 2)) (4.6.0.66)\n",
            "Collecting flake8>=3.8\n",
            "  Downloading flake8-6.0.0-py2.py3-none-any.whl (57 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m57.8/57.8 KB\u001b[0m \u001b[31m8.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting pytest-cov\n",
            "  Downloading pytest_cov-4.0.0-py3-none-any.whl (21 kB)\n",
            "Collecting pytest-env\n",
            "  Downloading pytest_env-0.8.1-py3-none-any.whl (5.2 kB)\n",
            "Requirement already satisfied: scipy>=1.4.1 in /usr/local/lib/python3.8/dist-packages (from stable-baselines3[docs,extra,tests]>=1.7.0->-r requirements.txt (line 2)) (1.7.3)\n",
            "Collecting black\n",
            "  Downloading black-22.12.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.5 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.5/1.5 MB\u001b[0m \u001b[31m80.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: pytest in /usr/local/lib/python3.8/dist-packages (from stable-baselines3[docs,extra,tests]>=1.7.0->-r requirements.txt (line 2)) (3.6.4)\n",
            "Collecting flake8-bugbear\n",
            "  Downloading flake8_bugbear-23.1.14-py3-none-any.whl (25 kB)\n",
            "Collecting isort>=5.0\n",
            "  Downloading isort-5.11.4-py3-none-any.whl (104 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m104.1/104.1 KB\u001b[0m \u001b[31m16.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting pytest-xdist\n",
            "  Downloading pytest_xdist-3.1.0-py3-none-any.whl (36 kB)\n",
            "Collecting pytype\n",
            "  Downloading pytype-2023.1.10-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.0 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.0/3.0 MB\u001b[0m \u001b[31m94.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting mypy\n",
            "  Downloading mypy-0.991-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.9 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m17.9/17.9 MB\u001b[0m \u001b[31m65.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting sphinx-copybutton\n",
            "  Downloading sphinx_copybutton-0.5.1-py3-none-any.whl (13 kB)\n",
            "Collecting sphinxcontrib.spelling\n",
            "  Downloading sphinxcontrib_spelling-7.7.0-py3-none-any.whl (15 kB)\n",
            "Collecting sphinx-rtd-theme\n",
            "  Downloading sphinx_rtd_theme-1.1.1-py2.py3-none-any.whl (2.8 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.8/2.8 MB\u001b[0m \u001b[31m50.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: sphinx in /usr/local/lib/python3.8/dist-packages (from stable-baselines3[docs,extra,tests]>=1.7.0->-r requirements.txt (line 2)) (3.5.4)\n",
            "Collecting sphinx-autobuild\n",
            "  Downloading sphinx_autobuild-2021.3.14-py3-none-any.whl (9.9 kB)\n",
            "Collecting sphinx-autodoc-typehints\n",
            "  Downloading sphinx_autodoc_typehints-1.20.2-py3-none-any.whl (12 kB)\n",
            "Requirement already satisfied: importlib-resources in /usr/local/lib/python3.8/dist-packages (from ale-py==0.7.4->stable-baselines3[docs,extra,tests]>=1.7.0->-r requirements.txt (line 2)) (5.10.2)\n",
            "Collecting gym-minigrid\n",
            "  Downloading gym_minigrid-1.2.1-py3-none-any.whl (70 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m70.3/70.3 KB\u001b[0m \u001b[31m11.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading gym_minigrid-1.2.0-py3-none-any.whl (70 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m70.4/70.4 KB\u001b[0m \u001b[31m6.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading gym_minigrid-1.1.0-py3-none-any.whl (57 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m57.4/57.4 KB\u001b[0m \u001b[31m9.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading gym_minigrid-1.0.3-py3-none-any.whl (48 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m48.1/48.1 KB\u001b[0m \u001b[31m7.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.8/dist-packages (from scikit-optimize->-r requirements.txt (line 7)) (1.2.0)\n",
            "Collecting pyaml>=16.9\n",
            "  Downloading pyaml-21.10.1-py2.py3-none-any.whl (24 kB)\n",
            "Requirement already satisfied: scikit-learn>=0.20.0 in /usr/local/lib/python3.8/dist-packages (from scikit-optimize->-r requirements.txt (line 7)) (1.0.2)\n",
            "Collecting colorlog\n",
            "  Downloading colorlog-6.7.0-py2.py3-none-any.whl (11 kB)\n",
            "Requirement already satisfied: sqlalchemy>=1.3.0 in /usr/local/lib/python3.8/dist-packages (from optuna->-r requirements.txt (line 8)) (1.4.46)\n",
            "Collecting cliff\n",
            "  Downloading cliff-4.1.0-py3-none-any.whl (81 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m81.0/81.0 KB\u001b[0m \u001b[31m12.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.8/dist-packages (from optuna->-r requirements.txt (line 8)) (21.3)\n",
            "Collecting alembic>=1.5.0\n",
            "  Downloading alembic-1.9.1-py3-none-any.whl (210 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m210.4/210.4 KB\u001b[0m \u001b[31m22.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting cmaes>=0.8.2\n",
            "  Downloading cmaes-0.9.1-py3-none-any.whl (21 kB)\n",
            "Collecting DataProperty<2,>=0.55.0\n",
            "  Downloading DataProperty-0.55.0-py3-none-any.whl (26 kB)\n",
            "Collecting mbstrdecoder<2,>=1.0.0\n",
            "  Downloading mbstrdecoder-1.1.1-py3-none-any.whl (7.7 kB)\n",
            "Collecting typepy[datetime]<2,>=1.2.0\n",
            "  Downloading typepy-1.3.0-py3-none-any.whl (31 kB)\n",
            "Requirement already satisfied: setuptools>=38.3.0 in /usr/local/lib/python3.8/dist-packages (from pytablewriter~=0.64->-r requirements.txt (line 9)) (57.4.0)\n",
            "Collecting pathvalidate<3,>=2.3.0\n",
            "  Downloading pathvalidate-2.5.2-py3-none-any.whl (20 kB)\n",
            "Collecting tabledata<2,>=1.3.0\n",
            "  Downloading tabledata-1.3.0-py3-none-any.whl (11 kB)\n",
            "Collecting tcolorpy<1,>=0.0.5\n",
            "  Downloading tcolorpy-0.1.2-py3-none-any.whl (7.9 kB)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.8/dist-packages (from plotly->-r requirements.txt (line 12)) (1.15.0)\n",
            "Requirement already satisfied: tenacity>=6.2.0 in /usr/local/lib/python3.8/dist-packages (from plotly->-r requirements.txt (line 12)) (8.1.0)\n",
            "Collecting arch==5.3.0\n",
            "  Downloading arch-5.3.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (907 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m907.3/907.3 KB\u001b[0m \u001b[31m49.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: absl-py>=0.9.0 in /usr/local/lib/python3.8/dist-packages (from rliable>=1.0.5->-r requirements.txt (line 14)) (1.3.0)\n",
            "Collecting property-cached>=1.6.4\n",
            "  Downloading property_cached-1.6.4-py2.py3-none-any.whl (7.8 kB)\n",
            "Requirement already satisfied: statsmodels>=0.11 in /usr/local/lib/python3.8/dist-packages (from arch==5.3.0->rliable>=1.0.5->-r requirements.txt (line 14)) (0.12.2)\n",
            "Requirement already satisfied: appdirs>=1.4.3 in /usr/local/lib/python3.8/dist-packages (from wandb->-r requirements.txt (line 15)) (1.4.4)\n",
            "Requirement already satisfied: Click!=8.0.0,>=7.0 in /usr/local/lib/python3.8/dist-packages (from wandb->-r requirements.txt (line 15)) (7.1.2)\n",
            "Requirement already satisfied: requests<3,>=2.0.0 in /usr/local/lib/python3.8/dist-packages (from wandb->-r requirements.txt (line 15)) (2.25.1)\n",
            "Collecting sentry-sdk>=1.0.0\n",
            "  Downloading sentry_sdk-1.13.0-py2.py3-none-any.whl (177 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m177.4/177.4 KB\u001b[0m \u001b[31m24.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: protobuf!=4.21.0,<5,>=3.12.0 in /usr/local/lib/python3.8/dist-packages (from wandb->-r requirements.txt (line 15)) (3.19.6)\n",
            "Collecting pathtools\n",
            "  Downloading pathtools-0.1.2.tar.gz (11 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting GitPython>=1.0.0\n",
            "  Downloading GitPython-3.1.30-py3-none-any.whl (184 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m184.0/184.0 KB\u001b[0m \u001b[31m25.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting docker-pycreds>=0.4.0\n",
            "  Downloading docker_pycreds-0.4.0-py2.py3-none-any.whl (9.0 kB)\n",
            "Collecting setproctitle\n",
            "  Downloading setproctitle-1.3.2-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (31 kB)\n",
            "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.8/dist-packages (from wandb->-r requirements.txt (line 15)) (4.4.0)\n",
            "Requirement already satisfied: wasabi in /usr/local/lib/python3.8/dist-packages (from huggingface_sb3<3.*,>=2.2.1->-r requirements.txt (line 16)) (0.10.1)\n",
            "Collecting huggingface-hub~=0.8\n",
            "  Downloading huggingface_hub-0.11.1-py3-none-any.whl (182 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m182.4/182.4 KB\u001b[0m \u001b[31m23.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting commonmark<0.10.0,>=0.9.0\n",
            "  Downloading commonmark-0.9.1-py2.py3-none-any.whl (51 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m51.1/51.1 KB\u001b[0m \u001b[31m7.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: pygments<3.0.0,>=2.6.0 in /usr/local/lib/python3.8/dist-packages (from rich->-r requirements.txt (line 19)) (2.6.1)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.8/dist-packages (from importlib-metadata~=4.13->-r requirements.txt (line 20)) (3.11.0)\n",
            "Collecting Mako\n",
            "  Downloading Mako-1.2.4-py3-none-any.whl (78 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m78.7/78.7 KB\u001b[0m \u001b[31m11.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting AutoROM.accept-rom-license\n",
            "  Downloading AutoROM.accept-rom-license-0.5.1.tar.gz (12 kB)\n",
            "  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
            "  Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting pyflakes<3.1.0,>=3.0.0\n",
            "  Downloading pyflakes-3.0.1-py2.py3-none-any.whl (62 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m62.8/62.8 KB\u001b[0m \u001b[31m6.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting pycodestyle<2.11.0,>=2.10.0\n",
            "  Downloading pycodestyle-2.10.0-py2.py3-none-any.whl (41 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m41.3/41.3 KB\u001b[0m \u001b[31m4.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting mccabe<0.8.0,>=0.7.0\n",
            "  Downloading mccabe-0.7.0-py2.py3-none-any.whl (7.3 kB)\n",
            "Collecting gitdb<5,>=4.0.1\n",
            "  Downloading gitdb-4.0.10-py3-none-any.whl (62 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m62.7/62.7 KB\u001b[0m \u001b[31m8.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: filelock in /usr/local/lib/python3.8/dist-packages (from huggingface-hub~=0.8->huggingface_sb3<3.*,>=2.2.1->-r requirements.txt (line 16)) (3.9.0)\n",
            "Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.8/dist-packages (from matplotlib->stable-baselines3[docs,extra,tests]>=1.7.0->-r requirements.txt (line 2)) (2.8.2)\n",
            "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.8/dist-packages (from matplotlib->stable-baselines3[docs,extra,tests]>=1.7.0->-r requirements.txt (line 2)) (3.0.9)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.8/dist-packages (from matplotlib->stable-baselines3[docs,extra,tests]>=1.7.0->-r requirements.txt (line 2)) (0.11.0)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.8/dist-packages (from matplotlib->stable-baselines3[docs,extra,tests]>=1.7.0->-r requirements.txt (line 2)) (1.4.4)\n",
            "Requirement already satisfied: chardet<6,>=3.0.4 in /usr/local/lib/python3.8/dist-packages (from mbstrdecoder<2,>=1.0.0->pytablewriter~=0.64->-r requirements.txt (line 9)) (4.0.0)\n",
            "Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas->stable-baselines3[docs,extra,tests]>=1.7.0->-r requirements.txt (line 2)) (2022.7)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.8/dist-packages (from requests<3,>=2.0.0->wandb->-r requirements.txt (line 15)) (2022.12.7)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.8/dist-packages (from requests<3,>=2.0.0->wandb->-r requirements.txt (line 15)) (2.10)\n",
            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.8/dist-packages (from requests<3,>=2.0.0->wandb->-r requirements.txt (line 15)) (1.24.3)\n",
            "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.8/dist-packages (from scikit-learn>=0.20.0->scikit-optimize->-r requirements.txt (line 7)) (3.1.0)\n",
            "Collecting urllib3<1.27,>=1.21.1\n",
            "  Using cached urllib3-1.26.14-py2.py3-none-any.whl (140 kB)\n",
            "Requirement already satisfied: greenlet!=0.4.17 in /usr/local/lib/python3.8/dist-packages (from sqlalchemy>=1.3.0->optuna->-r requirements.txt (line 8)) (2.0.1)\n",
            "Requirement already satisfied: google-auth<3,>=1.6.3 in /usr/local/lib/python3.8/dist-packages (from tensorboard>=2.9.1->stable-baselines3[docs,extra,tests]>=1.7.0->-r requirements.txt (line 2)) (2.15.0)\n",
            "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.8/dist-packages (from tensorboard>=2.9.1->stable-baselines3[docs,extra,tests]>=1.7.0->-r requirements.txt (line 2)) (0.4.6)\n",
            "Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.8/dist-packages (from tensorboard>=2.9.1->stable-baselines3[docs,extra,tests]>=1.7.0->-r requirements.txt (line 2)) (1.8.1)\n",
            "Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from tensorboard>=2.9.1->stable-baselines3[docs,extra,tests]>=1.7.0->-r requirements.txt (line 2)) (0.6.1)\n",
            "Requirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.8/dist-packages (from tensorboard>=2.9.1->stable-baselines3[docs,extra,tests]>=1.7.0->-r requirements.txt (line 2)) (0.38.4)\n",
            "Requirement already satisfied: grpcio>=1.24.3 in /usr/local/lib/python3.8/dist-packages (from tensorboard>=2.9.1->stable-baselines3[docs,extra,tests]>=1.7.0->-r requirements.txt (line 2)) (1.51.1)\n",
            "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.8/dist-packages (from tensorboard>=2.9.1->stable-baselines3[docs,extra,tests]>=1.7.0->-r requirements.txt (line 2)) (3.4.1)\n",
            "Requirement already satisfied: werkzeug>=1.0.1 in /usr/local/lib/python3.8/dist-packages (from tensorboard>=2.9.1->stable-baselines3[docs,extra,tests]>=1.7.0->-r requirements.txt (line 2)) (1.0.1)\n",
            "Collecting Click!=8.0.0,>=7.0\n",
            "  Using cached click-8.1.3-py3-none-any.whl (96 kB)\n",
            "Collecting mypy-extensions>=0.4.3\n",
            "  Downloading mypy_extensions-0.4.3-py2.py3-none-any.whl (4.5 kB)\n",
            "Requirement already satisfied: platformdirs>=2 in /usr/local/lib/python3.8/dist-packages (from black->stable-baselines3[docs,extra,tests]>=1.7.0->-r requirements.txt (line 2)) (2.6.2)\n",
            "Collecting pathspec>=0.9.0\n",
            "  Downloading pathspec-0.10.3-py3-none-any.whl (29 kB)\n",
            "Requirement already satisfied: tomli>=1.1.0 in /usr/local/lib/python3.8/dist-packages (from black->stable-baselines3[docs,extra,tests]>=1.7.0->-r requirements.txt (line 2)) (2.0.1)\n",
            "Collecting autopage>=0.4.0\n",
            "  Downloading autopage-0.5.1-py3-none-any.whl (29 kB)\n",
            "Requirement already satisfied: PrettyTable>=0.7.2 in /usr/local/lib/python3.8/dist-packages (from cliff->optuna->-r requirements.txt (line 8)) (3.5.0)\n",
            "Collecting cmd2>=1.0.0\n",
            "  Downloading cmd2-2.4.2-py3-none-any.whl (147 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m147.1/147.1 KB\u001b[0m \u001b[31m19.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting stevedore>=2.0.1\n",
            "  Downloading stevedore-4.1.1-py3-none-any.whl (50 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m50.0/50.0 KB\u001b[0m \u001b[31m6.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: attrs>=19.2.0 in /usr/local/lib/python3.8/dist-packages (from flake8-bugbear->stable-baselines3[docs,extra,tests]>=1.7.0->-r requirements.txt (line 2)) (22.2.0)\n",
            "Requirement already satisfied: pluggy<0.8,>=0.5 in /usr/local/lib/python3.8/dist-packages (from pytest->stable-baselines3[docs,extra,tests]>=1.7.0->-r requirements.txt (line 2)) (0.7.1)\n",
            "Requirement already satisfied: more-itertools>=4.0.0 in /usr/local/lib/python3.8/dist-packages (from pytest->stable-baselines3[docs,extra,tests]>=1.7.0->-r requirements.txt (line 2)) (9.0.0)\n",
            "Requirement already satisfied: atomicwrites>=1.0 in /usr/local/lib/python3.8/dist-packages (from pytest->stable-baselines3[docs,extra,tests]>=1.7.0->-r requirements.txt (line 2)) (1.4.1)\n",
            "Requirement already satisfied: py>=1.5.0 in /usr/local/lib/python3.8/dist-packages (from pytest->stable-baselines3[docs,extra,tests]>=1.7.0->-r requirements.txt (line 2)) (1.11.0)\n",
            "Collecting coverage[toml]>=5.2.1\n",
            "  Downloading coverage-7.0.5-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (227 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m227.0/227.0 KB\u001b[0m \u001b[31m28.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting pytest\n",
            "  Downloading pytest-7.2.1-py3-none-any.whl (317 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m317.1/317.1 KB\u001b[0m \u001b[31m35.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting exceptiongroup>=1.0.0rc8\n",
            "  Downloading exceptiongroup-1.1.0-py3-none-any.whl (14 kB)\n",
            "Collecting pytest\n",
            "  Downloading pytest-7.2.0-py3-none-any.whl (316 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m316.8/316.8 KB\u001b[0m \u001b[31m36.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading pytest-7.1.3-py3-none-any.whl (298 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m298.2/298.2 KB\u001b[0m \u001b[31m37.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading pytest-7.1.2-py3-none-any.whl (297 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m297.0/297.0 KB\u001b[0m \u001b[31m34.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading pytest-7.1.1-py3-none-any.whl (297 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m297.0/297.0 KB\u001b[0m \u001b[31m34.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading pytest-7.1.0-py3-none-any.whl (297 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m297.0/297.0 KB\u001b[0m \u001b[31m34.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading pytest-7.0.1-py3-none-any.whl (296 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m297.0/297.0 KB\u001b[0m \u001b[31m30.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading pytest-7.0.0-py3-none-any.whl (296 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m296.5/296.5 KB\u001b[0m \u001b[31m33.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading pytest-6.2.5-py3-none-any.whl (280 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m280.7/280.7 KB\u001b[0m \u001b[31m36.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: toml in /usr/local/lib/python3.8/dist-packages (from pytest->stable-baselines3[docs,extra,tests]>=1.7.0->-r requirements.txt (line 2)) (0.10.2)\n",
            "  Downloading pytest-6.2.4-py3-none-any.whl (280 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m280.6/280.6 KB\u001b[0m \u001b[31m36.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading pytest-6.2.3-py3-none-any.whl (280 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m280.5/280.5 KB\u001b[0m \u001b[31m34.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading pytest-6.2.2-py3-none-any.whl (280 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m280.1/280.1 KB\u001b[0m \u001b[31m35.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading pytest-6.2.1-py3-none-any.whl (279 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m279.9/279.9 KB\u001b[0m \u001b[31m35.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading pytest-6.2.0-py3-none-any.whl (279 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m279.6/279.6 KB\u001b[0m \u001b[31m27.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading pytest-6.1.2-py3-none-any.whl (272 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m272.4/272.4 KB\u001b[0m \u001b[31m35.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading pytest-6.1.1-py3-none-any.whl (272 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m272.2/272.2 KB\u001b[0m \u001b[31m34.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading pytest-6.1.0-py3-none-any.whl (272 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m272.3/272.3 KB\u001b[0m \u001b[31m31.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading pytest-6.0.2-py3-none-any.whl (270 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m270.8/270.8 KB\u001b[0m \u001b[31m35.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading pytest-6.0.1-py3-none-any.whl (270 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m270.6/270.6 KB\u001b[0m \u001b[31m34.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading pytest-6.0.0-py3-none-any.whl (270 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m270.6/270.6 KB\u001b[0m \u001b[31m33.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading pytest-5.4.3-py3-none-any.whl (248 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m248.1/248.1 KB\u001b[0m \u001b[31m32.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: wcwidth in /usr/local/lib/python3.8/dist-packages (from pytest->stable-baselines3[docs,extra,tests]>=1.7.0->-r requirements.txt (line 2)) (0.2.5)\n",
            "  Downloading pytest-5.4.2-py3-none-any.whl (247 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m247.4/247.4 KB\u001b[0m \u001b[31m32.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading pytest-5.4.1-py3-none-any.whl (246 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m246.7/246.7 KB\u001b[0m \u001b[31m25.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading pytest-5.4.0-py3-none-any.whl (247 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m247.0/247.0 KB\u001b[0m \u001b[31m28.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading pytest-5.3.5-py3-none-any.whl (235 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m235.2/235.2 KB\u001b[0m \u001b[31m31.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading pytest-5.3.4-py3-none-any.whl (235 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m235.3/235.3 KB\u001b[0m \u001b[31m29.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading pytest-5.3.3-py3-none-any.whl (235 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m235.5/235.5 KB\u001b[0m \u001b[31m27.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading pytest-5.3.2-py3-none-any.whl (234 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m234.5/234.5 KB\u001b[0m \u001b[31m31.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading pytest-5.3.1-py3-none-any.whl (233 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m233.9/233.9 KB\u001b[0m \u001b[31m29.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading pytest-5.3.0-py3-none-any.whl (233 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m233.7/233.7 KB\u001b[0m \u001b[31m24.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading pytest-5.2.4-py3-none-any.whl (227 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m227.4/227.4 KB\u001b[0m \u001b[31m30.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading pytest-5.2.3-py3-none-any.whl (227 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m227.4/227.4 KB\u001b[0m \u001b[31m31.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading pytest-5.2.2-py3-none-any.whl (227 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m227.3/227.3 KB\u001b[0m \u001b[31m29.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading pytest-5.2.1-py3-none-any.whl (226 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m226.8/226.8 KB\u001b[0m \u001b[31m25.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading pytest-5.2.0-py3-none-any.whl (226 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m226.7/226.7 KB\u001b[0m \u001b[31m30.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading pytest-5.1.3-py3-none-any.whl (224 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m224.1/224.1 KB\u001b[0m \u001b[31m29.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading pytest-5.1.2-py3-none-any.whl (224 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m224.2/224.2 KB\u001b[0m \u001b[31m23.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading pytest-5.1.1-py3-none-any.whl (223 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m223.6/223.6 KB\u001b[0m \u001b[31m30.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading pytest-5.1.0-py3-none-any.whl (223 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m223.6/223.6 KB\u001b[0m \u001b[31m29.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading pytest-5.0.1-py3-none-any.whl (221 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m221.8/221.8 KB\u001b[0m \u001b[31m26.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading pytest-5.0.0-py3-none-any.whl (221 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m221.8/221.8 KB\u001b[0m \u001b[31m26.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading pytest-4.6.11-py2.py3-none-any.whl (231 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m231.6/231.6 KB\u001b[0m \u001b[31m30.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading pytest-4.6.10-py2.py3-none-any.whl (231 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m231.6/231.6 KB\u001b[0m \u001b[31m27.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading pytest-4.6.9-py2.py3-none-any.whl (231 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m231.1/231.1 KB\u001b[0m \u001b[31m28.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading pytest-4.6.8-py2.py3-none-any.whl (230 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m230.5/230.5 KB\u001b[0m \u001b[31m30.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading pytest-4.6.7-py2.py3-none-any.whl (230 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m230.4/230.4 KB\u001b[0m \u001b[31m30.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading pytest-4.6.6-py2.py3-none-any.whl (230 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m230.2/230.2 KB\u001b[0m \u001b[31m30.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading pytest-4.6.5-py2.py3-none-any.whl (230 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m230.1/230.1 KB\u001b[0m \u001b[31m29.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading pytest-4.6.4-py2.py3-none-any.whl (229 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m229.6/229.6 KB\u001b[0m \u001b[31m31.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading pytest-4.6.3-py2.py3-none-any.whl (229 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m229.6/229.6 KB\u001b[0m \u001b[31m30.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading pytest-4.6.2-py2.py3-none-any.whl (229 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m229.5/229.5 KB\u001b[0m \u001b[31m26.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading pytest-4.6.1-py2.py3-none-any.whl (229 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m229.9/229.9 KB\u001b[0m \u001b[31m28.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading pytest-4.6.0-py2.py3-none-any.whl (229 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m229.8/229.8 KB\u001b[0m \u001b[31m29.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hINFO: pip is looking at multiple versions of pytest-cov to determine which version is compatible with other requirements. This could take a while.\n",
            "Collecting pytest-cov\n",
            "  Downloading pytest_cov-3.0.0-py3-none-any.whl (20 kB)\n",
            "  Downloading pytest_cov-2.12.1-py2.py3-none-any.whl (20 kB)\n",
            "  Downloading pytest_cov-2.12.0-py2.py3-none-any.whl (20 kB)\n",
            "  Downloading pytest_cov-2.11.1-py2.py3-none-any.whl (20 kB)\n",
            "  Downloading pytest_cov-2.11.0-py2.py3-none-any.whl (20 kB)\n",
            "  Downloading pytest_cov-2.10.1-py2.py3-none-any.whl (19 kB)\n",
            "  Downloading pytest_cov-2.10.0-py2.py3-none-any.whl (19 kB)\n",
            "INFO: pip is looking at multiple versions of pytest-cov to determine which version is compatible with other requirements. This could take a while.\n",
            "  Downloading pytest_cov-2.9.0-py2.py3-none-any.whl (19 kB)\n",
            "INFO: pip is looking at multiple versions of pytest-env to determine which version is compatible with other requirements. This could take a while.\n",
            "Collecting pytest-env\n",
            "  Downloading pytest_env-0.8.0-py3-none-any.whl (5.2 kB)\n",
            "  Downloading pytest_env-0.7.0-py3-none-any.whl (4.9 kB)\n",
            "  Downloading pytest-env-0.6.2.tar.gz (1.7 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting execnet>=1.1\n",
            "  Downloading execnet-1.9.0-py2.py3-none-any.whl (39 kB)\n",
            "INFO: pip is looking at multiple versions of pytest-xdist to determine which version is compatible with other requirements. This could take a while.\n",
            "Collecting pytest-xdist\n",
            "  Downloading pytest_xdist-3.0.2-py3-none-any.whl (36 kB)\n",
            "  Downloading pytest_xdist-2.5.0-py3-none-any.whl (41 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m41.7/41.7 KB\u001b[0m \u001b[31m5.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting pytest-forked\n",
            "  Downloading pytest_forked-1.4.0-py3-none-any.whl (4.9 kB)\n",
            "Collecting pytest-xdist\n",
            "  Downloading pytest_xdist-2.4.0-py3-none-any.whl (38 kB)\n",
            "  Downloading pytest_xdist-2.3.0-py3-none-any.whl (37 kB)\n",
            "  Downloading pytest_xdist-2.2.1-py3-none-any.whl (37 kB)\n",
            "  Downloading pytest_xdist-2.2.0-py3-none-any.whl (36 kB)\n",
            "  Downloading pytest_xdist-2.1.0-py3-none-any.whl (36 kB)\n",
            "INFO: pip is looking at multiple versions of pytest-xdist to determine which version is compatible with other requirements. This could take a while.\n",
            "  Downloading pytest_xdist-2.0.0-py2.py3-none-any.whl (36 kB)\n",
            "  Downloading pytest_xdist-1.34.0-py2.py3-none-any.whl (36 kB)\n",
            "Collecting pytest\n",
            "  Downloading pytest-4.5.0-py2.py3-none-any.whl (227 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m227.5/227.5 KB\u001b[0m \u001b[31m26.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading pytest-4.4.2-py2.py3-none-any.whl (224 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m224.0/224.0 KB\u001b[0m \u001b[31m25.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading pytest-4.4.1-py2.py3-none-any.whl (223 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m223.9/223.9 KB\u001b[0m \u001b[31m23.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading pytest-4.4.0-py2.py3-none-any.whl (223 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m223.6/223.6 KB\u001b[0m \u001b[31m25.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting pytest-xdist\n",
            "  Downloading pytest_xdist-1.33.0-py2.py3-none-any.whl (36 kB)\n",
            "  Downloading pytest_xdist-1.32.0-py2.py3-none-any.whl (36 kB)\n",
            "  Downloading pytest_xdist-1.31.0-py2.py3-none-any.whl (36 kB)\n",
            "INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.\n",
            "  Downloading pytest_xdist-1.30.0-py2.py3-none-any.whl (35 kB)\n",
            "  Downloading pytest_xdist-1.29.0-py2.py3-none-any.whl (35 kB)\n",
            "  Downloading pytest_xdist-1.28.0-py2.py3-none-any.whl (34 kB)\n",
            "  Downloading pytest_xdist-1.27.0-py2.py3-none-any.whl (35 kB)\n",
            "Requirement already satisfied: tabulate>=0.8.10 in /usr/local/lib/python3.8/dist-packages (from pytype->stable-baselines3[docs,extra,tests]>=1.7.0->-r requirements.txt (line 2)) (0.8.10)\n",
            "Collecting libcst>=0.4.5\n",
            "  Downloading libcst-0.4.9-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.8 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.8/2.8 MB\u001b[0m \u001b[31m85.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting jinja2>=3.1.2\n",
            "  Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m133.1/133.1 KB\u001b[0m \u001b[31m17.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting pydot>=1.4.2\n",
            "  Downloading pydot-1.4.2-py2.py3-none-any.whl (21 kB)\n",
            "Collecting ninja>=1.10.0.post2\n",
            "  Downloading ninja-1.11.1-py2.py3-none-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (145 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m146.0/146.0 KB\u001b[0m \u001b[31m20.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting networkx<2.8.4\n",
            "  Downloading networkx-2.8.3-py3-none-any.whl (2.0 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.0/2.0 MB\u001b[0m \u001b[31m82.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting importlab>=0.8\n",
            "  Downloading importlab-0.8-py2.py3-none-any.whl (21 kB)\n",
            "Requirement already satisfied: sphinxcontrib-applehelp in /usr/local/lib/python3.8/dist-packages (from sphinx->stable-baselines3[docs,extra,tests]>=1.7.0->-r requirements.txt (line 2)) (1.0.2)\n",
            "Requirement already satisfied: sphinxcontrib-qthelp in /usr/local/lib/python3.8/dist-packages (from sphinx->stable-baselines3[docs,extra,tests]>=1.7.0->-r requirements.txt (line 2)) (1.0.3)\n",
            "Requirement already satisfied: sphinxcontrib-htmlhelp in /usr/local/lib/python3.8/dist-packages (from sphinx->stable-baselines3[docs,extra,tests]>=1.7.0->-r requirements.txt (line 2)) (2.0.0)\n",
            "Requirement already satisfied: snowballstemmer>=1.1 in /usr/local/lib/python3.8/dist-packages (from sphinx->stable-baselines3[docs,extra,tests]>=1.7.0->-r requirements.txt (line 2)) (2.2.0)\n",
            "Requirement already satisfied: sphinxcontrib-jsmath in /usr/local/lib/python3.8/dist-packages (from sphinx->stable-baselines3[docs,extra,tests]>=1.7.0->-r requirements.txt (line 2)) (1.0.1)\n",
            "Requirement already satisfied: babel>=1.3 in /usr/local/lib/python3.8/dist-packages (from sphinx->stable-baselines3[docs,extra,tests]>=1.7.0->-r requirements.txt (line 2)) (2.11.0)\n",
            "Requirement already satisfied: sphinxcontrib-devhelp in /usr/local/lib/python3.8/dist-packages (from sphinx->stable-baselines3[docs,extra,tests]>=1.7.0->-r requirements.txt (line 2)) (1.0.2)\n",
            "Requirement already satisfied: alabaster<0.8,>=0.7 in /usr/local/lib/python3.8/dist-packages (from sphinx->stable-baselines3[docs,extra,tests]>=1.7.0->-r requirements.txt (line 2)) (0.7.12)\n",
            "Requirement already satisfied: sphinxcontrib-serializinghtml in /usr/local/lib/python3.8/dist-packages (from sphinx->stable-baselines3[docs,extra,tests]>=1.7.0->-r requirements.txt (line 2)) (1.1.5)\n",
            "Requirement already satisfied: imagesize in /usr/local/lib/python3.8/dist-packages (from sphinx->stable-baselines3[docs,extra,tests]>=1.7.0->-r requirements.txt (line 2)) (1.4.1)\n",
            "Requirement already satisfied: docutils<0.17,>=0.12 in /usr/local/lib/python3.8/dist-packages (from sphinx->stable-baselines3[docs,extra,tests]>=1.7.0->-r requirements.txt (line 2)) (0.16)\n",
            "Collecting livereload\n",
            "  Downloading livereload-2.6.3-py2.py3-none-any.whl (24 kB)\n",
            "Collecting colorama\n",
            "  Downloading colorama-0.4.6-py2.py3-none-any.whl (25 kB)\n",
            "Collecting sphinx\n",
            "  Downloading sphinx-6.1.3-py3-none-any.whl (3.0 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.0/3.0 MB\u001b[0m \u001b[31m80.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading sphinx-6.1.2-py3-none-any.whl (3.0 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.0/3.0 MB\u001b[0m \u001b[31m95.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading sphinx-6.1.1-py3-none-any.whl (3.0 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.0/3.0 MB\u001b[0m \u001b[31m85.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading sphinx-6.1.0-py3-none-any.whl (3.0 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.0/3.0 MB\u001b[0m \u001b[31m91.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading sphinx-6.0.1-py3-none-any.whl (3.0 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.0/3.0 MB\u001b[0m \u001b[31m90.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading sphinx-6.0.0-py3-none-any.whl (3.0 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.0/3.0 MB\u001b[0m \u001b[31m80.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting pygments<3.0.0,>=2.6.0\n",
            "  Downloading Pygments-2.14.0-py3-none-any.whl (1.1 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.1/1.1 MB\u001b[0m \u001b[31m65.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting sphinx\n",
            "  Downloading sphinx-5.3.0-py3-none-any.whl (3.2 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.2/3.2 MB\u001b[0m \u001b[31m65.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting PyEnchant>=3.1.1\n",
            "  Downloading pyenchant-3.2.2-py3-none-any.whl (55 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m55.7/55.7 KB\u001b[0m \u001b[31m7.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting pyperclip>=1.6\n",
            "  Downloading pyperclip-1.8.2.tar.gz (20 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting smmap<6,>=3.0.1\n",
            "  Downloading smmap-5.0.0-py3-none-any.whl (24 kB)\n",
            "Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.8/dist-packages (from google-auth<3,>=1.6.3->tensorboard>=2.9.1->stable-baselines3[docs,extra,tests]>=1.7.0->-r requirements.txt (line 2)) (4.9)\n",
            "Requirement already satisfied: cachetools<6.0,>=2.0.0 in /usr/local/lib/python3.8/dist-packages (from google-auth<3,>=1.6.3->tensorboard>=2.9.1->stable-baselines3[docs,extra,tests]>=1.7.0->-r requirements.txt (line 2)) (5.2.0)\n",
            "Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.8/dist-packages (from google-auth<3,>=1.6.3->tensorboard>=2.9.1->stable-baselines3[docs,extra,tests]>=1.7.0->-r requirements.txt (line 2)) (0.2.8)\n",
            "Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.8/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard>=2.9.1->stable-baselines3[docs,extra,tests]>=1.7.0->-r requirements.txt (line 2)) (1.3.1)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2>=3.1.2->pytype->stable-baselines3[docs,extra,tests]>=1.7.0->-r requirements.txt (line 2)) (2.0.1)\n",
            "Collecting typing-inspect>=0.4.0\n",
            "  Downloading typing_inspect-0.8.0-py3-none-any.whl (8.7 kB)\n",
            "Requirement already satisfied: patsy>=0.5 in /usr/local/lib/python3.8/dist-packages (from statsmodels>=0.11->arch==5.3.0->rliable>=1.0.5->-r requirements.txt (line 14)) (0.5.3)\n",
            "Collecting pbr!=2.1.0,>=2.0.0\n",
            "  Downloading pbr-5.11.1-py2.py3-none-any.whl (112 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m112.7/112.7 KB\u001b[0m \u001b[31m16.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting libtorrent\n",
            "  Using cached libtorrent-2.0.7-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (8.6 MB)\n",
            "Requirement already satisfied: tornado in /usr/local/lib/python3.8/dist-packages (from livereload->sphinx-autobuild->stable-baselines3[docs,extra,tests]>=1.7.0->-r requirements.txt (line 2)) (6.0.4)\n",
            "Collecting pytest\n",
            "  Downloading pytest-4.3.1-py2.py3-none-any.whl (219 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m219.4/219.4 KB\u001b[0m \u001b[31m26.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.8/dist-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard>=2.9.1->stable-baselines3[docs,extra,tests]>=1.7.0->-r requirements.txt (line 2)) (0.4.8)\n",
            "Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard>=2.9.1->stable-baselines3[docs,extra,tests]>=1.7.0->-r requirements.txt (line 2)) (3.2.2)\n",
            "Building wheels for collected packages: gym, box2d-py, pathtools, pytest-env, AutoROM.accept-rom-license, pyperclip\n",
            "  Building wheel for gym (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for gym: filename=gym-0.21.0-py3-none-any.whl size=1616825 sha256=af67f48688590d1f1257d13529be92bdeed6411d3dda9dcc1242914bcd1d3183\n",
            "  Stored in directory: /root/.cache/pip/wheels/27/6d/b3/a3a6e10704795c9b9000f1ab2dc480dfe7bed42f5972806e73\n",
            "  Building wheel for box2d-py (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for box2d-py: filename=box2d_py-2.3.8-cp38-cp38-linux_x86_64.whl size=2046033 sha256=b26a71a45b32ed285acc7fc00abdc5c5470e6ce5a6b7a08d86940bd1d05fdb2a\n",
            "  Stored in directory: /root/.cache/pip/wheels/cc/4f/d6/44eb0a9e6fea384e58f19cb0c4125e46a23af2b33fe3a7e81c\n",
            "  Building wheel for pathtools (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for pathtools: filename=pathtools-0.1.2-py3-none-any.whl size=8806 sha256=7c6096534dacd4be716a77dc5bd7c0510ef2c8f345dbd7701d8e5a54766b987c\n",
            "  Stored in directory: /root/.cache/pip/wheels/4c/8e/7e/72fbc243e1aeecae64a96875432e70d4e92f3d2d18123be004\n",
            "  Building wheel for pytest-env (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for pytest-env: filename=pytest_env-0.6.2-py3-none-any.whl size=2372 sha256=83f0a4407638241a6baf65cf06ea59c4ba83f4bf34de76edfc4e6562262f8442\n",
            "  Stored in directory: /root/.cache/pip/wheels/37/01/73/bbe6b92b251c90dbe27c15b8e5b57085b9b53dfe018e9b6cea\n",
            "  \u001b[1;31merror\u001b[0m: \u001b[1msubprocess-exited-with-error\u001b[0m\n",
            "  \n",
            "  \u001b[31m×\u001b[0m \u001b[32mBuilding wheel for AutoROM.accept-rom-license \u001b[0m\u001b[1;32m(\u001b[0m\u001b[32mpyproject.toml\u001b[0m\u001b[1;32m)\u001b[0m did not run successfully.\n",
            "  \u001b[31m│\u001b[0m exit code: \u001b[1;36m1\u001b[0m\n",
            "  \u001b[31m╰─>\u001b[0m See above for output.\n",
            "  \n",
            "  \u001b[1;35mnote\u001b[0m: This error originates from a subprocess, and is likely not a problem with pip.\n",
            "  Building wheel for AutoROM.accept-rom-license (pyproject.toml) ... \u001b[?25l\u001b[?25herror\n",
            "\u001b[31m  ERROR: Failed building wheel for AutoROM.accept-rom-license\u001b[0m\u001b[31m\n",
            "\u001b[0m  Building wheel for pyperclip (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for pyperclip: filename=pyperclip-1.8.2-py3-none-any.whl size=11136 sha256=2f20579330a8e270db2df527c8cd67fa4de5a7143d2f0f7795aa507ffb41cf97\n",
            "  Stored in directory: /root/.cache/pip/wheels/7f/1a/65/84ff8c386bec21fca6d220ea1f5498a0367883a78dd5ba6122\n",
            "Successfully built gym box2d-py pathtools pytest-env pyperclip\n",
            "Failed to build AutoROM.accept-rom-license\n",
            "\u001b[31mERROR: Could not build wheels for AutoROM.accept-rom-license, which is required to install pyproject.toml-based projects\u001b[0m\u001b[31m\n",
            "\u001b[0m"
          ]
        }
      ],
      "source": [
        "!pip install -r requirements.txt"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5iPgzluo9z-u"
      },
      "source": [
        "## Train our Deep Q-Learning Agent to Play Space Invaders 👾\n",
        "\n",
        "To train an agent with RL-Baselines3-Zoo, we just need to do two things:\n",
        "1. We define the hyperparameters in `rl-baselines3-zoo/hyperparams/dqn.yml`\n",
        "\n",
        "<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/unit3/hyperparameters.png\" alt=\"DQN Hyperparameters\">\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_VjblFSVDQOj"
      },
      "source": [
        "Here we see that:\n",
        "- We use the `Atari Wrapper` that preprocess the input (Frame reduction ,grayscale, stack 4 frames)\n",
        "- We use `CnnPolicy`, since we use Convolutional layers to process the frames\n",
        "- We train it for 10 million `n_timesteps` \n",
        "- Memory (Experience Replay) size is 100000, aka the amount of experience steps you saved to train again your agent with.\n",
        "\n",
        "💡 My advice is to **reduce the training timesteps to 1M,** which will take about 90 minutes on a P100. `!nvidia-smi` will tell you what GPU you're using. At 10 million steps, this will take about 9 hours, which could likely result in Colab timing out. I recommend running this on your local computer (or somewhere else). Just click on: `File>Download`. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5qTkbWrkECOJ"
      },
      "source": [
        "In terms of hyperparameters optimization, my advice is to focus on these 3 hyperparameters:\n",
        "- `learning_rate`\n",
        "- `buffer_size (Experience Memory size)`\n",
        "- `batch_size`\n",
        "\n",
        "As a good practice, you need to **check the documentation to understand what each hyperparameters does**: https://stable-baselines3.readthedocs.io/en/master/modules/dqn.html#parameters\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Hn8bRTHvERRL"
      },
      "source": [
        "2. We run `train.py` and save the models on `logs` folder 📁"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install optuna"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pvehzbDjvRA-",
        "outputId": "1292b239-77b7-4104-dd1a-bbcf1a40c967"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting optuna\n",
            "  Using cached optuna-3.0.5-py3-none-any.whl (348 kB)\n",
            "Collecting importlib-metadata<5.0.0\n",
            "  Using cached importlib_metadata-4.13.0-py3-none-any.whl (23 kB)\n",
            "Requirement already satisfied: sqlalchemy>=1.3.0 in /usr/local/lib/python3.8/dist-packages (from optuna) (1.4.46)\n",
            "Requirement already satisfied: PyYAML in /usr/local/lib/python3.8/dist-packages (from optuna) (6.0)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.8/dist-packages (from optuna) (4.64.1)\n",
            "Collecting alembic>=1.5.0\n",
            "  Using cached alembic-1.9.1-py3-none-any.whl (210 kB)\n",
            "Requirement already satisfied: scipy<1.9.0,>=1.7.0 in /usr/local/lib/python3.8/dist-packages (from optuna) (1.7.3)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.8/dist-packages (from optuna) (21.3)\n",
            "Collecting cmaes>=0.8.2\n",
            "  Using cached cmaes-0.9.1-py3-none-any.whl (21 kB)\n",
            "Collecting colorlog\n",
            "  Using cached colorlog-6.7.0-py2.py3-none-any.whl (11 kB)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.8/dist-packages (from optuna) (1.21.6)\n",
            "Collecting cliff\n",
            "  Using cached cliff-4.1.0-py3-none-any.whl (81 kB)\n",
            "Requirement already satisfied: importlib-resources in /usr/local/lib/python3.8/dist-packages (from alembic>=1.5.0->optuna) (5.10.2)\n",
            "Collecting Mako\n",
            "  Using cached Mako-1.2.4-py3-none-any.whl (78 kB)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.8/dist-packages (from importlib-metadata<5.0.0->optuna) (3.11.0)\n",
            "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging>=20.0->optuna) (3.0.9)\n",
            "Requirement already satisfied: greenlet!=0.4.17 in /usr/local/lib/python3.8/dist-packages (from sqlalchemy>=1.3.0->optuna) (2.0.1)\n",
            "Collecting cmd2>=1.0.0\n",
            "  Using cached cmd2-2.4.2-py3-none-any.whl (147 kB)\n",
            "Collecting autopage>=0.4.0\n",
            "  Using cached autopage-0.5.1-py3-none-any.whl (29 kB)\n",
            "Requirement already satisfied: PrettyTable>=0.7.2 in /usr/local/lib/python3.8/dist-packages (from cliff->optuna) (3.5.0)\n",
            "Collecting stevedore>=2.0.1\n",
            "  Using cached stevedore-4.1.1-py3-none-any.whl (50 kB)\n",
            "Collecting pyperclip>=1.6\n",
            "  Using cached pyperclip-1.8.2-py3-none-any.whl\n",
            "Requirement already satisfied: wcwidth>=0.1.7 in /usr/local/lib/python3.8/dist-packages (from cmd2>=1.0.0->cliff->optuna) (0.2.5)\n",
            "Requirement already satisfied: attrs>=16.3.0 in /usr/local/lib/python3.8/dist-packages (from cmd2>=1.0.0->cliff->optuna) (22.2.0)\n",
            "Collecting pbr!=2.1.0,>=2.0.0\n",
            "  Using cached pbr-5.11.1-py2.py3-none-any.whl (112 kB)\n",
            "Requirement already satisfied: MarkupSafe>=0.9.2 in /usr/local/lib/python3.8/dist-packages (from Mako->alembic>=1.5.0->optuna) (2.0.1)\n",
            "Installing collected packages: pyperclip, pbr, Mako, importlib-metadata, colorlog, cmd2, cmaes, autopage, stevedore, alembic, cliff, optuna\n",
            "  Attempting uninstall: importlib-metadata\n",
            "    Found existing installation: importlib-metadata 6.0.0\n",
            "    Uninstalling importlib-metadata-6.0.0:\n",
            "      Successfully uninstalled importlib-metadata-6.0.0\n",
            "Successfully installed Mako-1.2.4 alembic-1.9.1 autopage-0.5.1 cliff-4.1.0 cmaes-0.9.1 cmd2-2.4.2 colorlog-6.7.0 importlib-metadata-4.13.0 optuna-3.0.5 pbr-5.11.1 pyperclip-1.8.2 stevedore-4.1.1\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import optuna\n",
        "from optuna.pruners import MedianPruner\n",
        "from optuna.samplers import TPESampler\n",
        "from optuna.visualization import plot_optimization_history, plot_param_importances"
      ],
      "metadata": {
        "id": "4PzI75mtvIQO"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install stable-baselines3"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dRfy4AV7ztiE",
        "outputId": "cc52a5ad-aefc-4d97-8cb4-b72b40c41372"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting stable-baselines3\n",
            "  Using cached stable_baselines3-1.7.0-py3-none-any.whl (171 kB)\n",
            "Requirement already satisfied: importlib-metadata~=4.13 in /usr/local/lib/python3.8/dist-packages (from stable-baselines3) (4.13.0)\n",
            "Requirement already satisfied: torch>=1.11 in /usr/local/lib/python3.8/dist-packages (from stable-baselines3) (1.13.0+cu116)\n",
            "Collecting gym==0.21\n",
            "  Using cached gym-0.21.0-py3-none-any.whl\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.8/dist-packages (from stable-baselines3) (1.3.5)\n",
            "Requirement already satisfied: matplotlib in /usr/local/lib/python3.8/dist-packages (from stable-baselines3) (3.2.2)\n",
            "Requirement already satisfied: cloudpickle in /usr/local/lib/python3.8/dist-packages (from stable-baselines3) (2.2.0)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.8/dist-packages (from stable-baselines3) (1.21.6)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.8/dist-packages (from importlib-metadata~=4.13->stable-baselines3) (3.11.0)\n",
            "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.8/dist-packages (from torch>=1.11->stable-baselines3) (4.4.0)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.8/dist-packages (from matplotlib->stable-baselines3) (0.11.0)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.8/dist-packages (from matplotlib->stable-baselines3) (1.4.4)\n",
            "Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.8/dist-packages (from matplotlib->stable-baselines3) (2.8.2)\n",
            "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.8/dist-packages (from matplotlib->stable-baselines3) (3.0.9)\n",
            "Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas->stable-baselines3) (2022.7)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.8/dist-packages (from python-dateutil>=2.1->matplotlib->stable-baselines3) (1.15.0)\n",
            "Installing collected packages: gym, stable-baselines3\n",
            "  Attempting uninstall: gym\n",
            "    Found existing installation: gym 0.25.2\n",
            "    Uninstalling gym-0.25.2:\n",
            "      Successfully uninstalled gym-0.25.2\n",
            "Successfully installed gym-0.21.0 stable-baselines3-1.7.0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install sb3-contrib"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "n4WOpnl-vYNc",
        "outputId": "eeb30156-d053-43ec-88fb-01de24972c80"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting sb3-contrib\n",
            "  Using cached sb3_contrib-1.7.0-py3-none-any.whl (79 kB)\n",
            "Requirement already satisfied: stable-baselines3>=1.7.0 in /usr/local/lib/python3.8/dist-packages (from sb3-contrib) (1.7.0)\n",
            "Requirement already satisfied: matplotlib in /usr/local/lib/python3.8/dist-packages (from stable-baselines3>=1.7.0->sb3-contrib) (3.2.2)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.8/dist-packages (from stable-baselines3>=1.7.0->sb3-contrib) (1.3.5)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.8/dist-packages (from stable-baselines3>=1.7.0->sb3-contrib) (1.21.6)\n",
            "Requirement already satisfied: torch>=1.11 in /usr/local/lib/python3.8/dist-packages (from stable-baselines3>=1.7.0->sb3-contrib) (1.13.0+cu116)\n",
            "Requirement already satisfied: gym==0.21 in /usr/local/lib/python3.8/dist-packages (from stable-baselines3>=1.7.0->sb3-contrib) (0.21.0)\n",
            "Requirement already satisfied: importlib-metadata~=4.13 in /usr/local/lib/python3.8/dist-packages (from stable-baselines3>=1.7.0->sb3-contrib) (4.13.0)\n",
            "Requirement already satisfied: cloudpickle in /usr/local/lib/python3.8/dist-packages (from stable-baselines3>=1.7.0->sb3-contrib) (2.2.0)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.8/dist-packages (from importlib-metadata~=4.13->stable-baselines3>=1.7.0->sb3-contrib) (3.11.0)\n",
            "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.8/dist-packages (from torch>=1.11->stable-baselines3>=1.7.0->sb3-contrib) (4.4.0)\n",
            "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.8/dist-packages (from matplotlib->stable-baselines3>=1.7.0->sb3-contrib) (3.0.9)\n",
            "Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.8/dist-packages (from matplotlib->stable-baselines3>=1.7.0->sb3-contrib) (2.8.2)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.8/dist-packages (from matplotlib->stable-baselines3>=1.7.0->sb3-contrib) (0.11.0)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.8/dist-packages (from matplotlib->stable-baselines3>=1.7.0->sb3-contrib) (1.4.4)\n",
            "Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas->stable-baselines3>=1.7.0->sb3-contrib) (2022.7)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.8/dist-packages (from python-dateutil>=2.1->matplotlib->stable-baselines3>=1.7.0->sb3-contrib) (1.15.0)\n",
            "Installing collected packages: sb3-contrib\n",
            "Successfully installed sb3-contrib-1.7.0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install huggingface_hub"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jQNpdTBH0TsU",
        "outputId": "ed42f286-fea4-4449-87d8-5742a5a7e3c6"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting huggingface_hub\n",
            "  Using cached huggingface_hub-0.11.1-py3-none-any.whl (182 kB)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.8/dist-packages (from huggingface_hub) (6.0)\n",
            "Requirement already satisfied: packaging>=20.9 in /usr/local/lib/python3.8/dist-packages (from huggingface_hub) (21.3)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.8/dist-packages (from huggingface_hub) (2.25.1)\n",
            "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.8/dist-packages (from huggingface_hub) (4.4.0)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.8/dist-packages (from huggingface_hub) (3.9.0)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.8/dist-packages (from huggingface_hub) (4.64.1)\n",
            "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging>=20.9->huggingface_hub) (3.0.9)\n",
            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.8/dist-packages (from requests->huggingface_hub) (1.24.3)\n",
            "Requirement already satisfied: chardet<5,>=3.0.2 in /usr/local/lib/python3.8/dist-packages (from requests->huggingface_hub) (4.0.0)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.8/dist-packages (from requests->huggingface_hub) (2.10)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.8/dist-packages (from requests->huggingface_hub) (2022.12.7)\n",
            "Installing collected packages: huggingface_hub\n",
            "Successfully installed huggingface_hub-0.11.1\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install huggingface_sb3"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GylU7LPV0cnm",
        "outputId": "0b2547b5-7b1d-4c64-82f2-0ffb5c2b1301"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting huggingface_sb3\n",
            "  Using cached huggingface_sb3-2.2.4-py3-none-any.whl (9.4 kB)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.8/dist-packages (from huggingface_sb3) (1.21.6)\n",
            "Requirement already satisfied: wasabi in /usr/local/lib/python3.8/dist-packages (from huggingface_sb3) (0.10.1)\n",
            "Requirement already satisfied: pyyaml~=6.0 in /usr/local/lib/python3.8/dist-packages (from huggingface_sb3) (6.0)\n",
            "Requirement already satisfied: cloudpickle>=1.6 in /usr/local/lib/python3.8/dist-packages (from huggingface_sb3) (2.2.0)\n",
            "Requirement already satisfied: huggingface-hub~=0.8 in /usr/local/lib/python3.8/dist-packages (from huggingface_sb3) (0.11.1)\n",
            "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.8/dist-packages (from huggingface-hub~=0.8->huggingface_sb3) (4.4.0)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.8/dist-packages (from huggingface-hub~=0.8->huggingface_sb3) (4.64.1)\n",
            "Requirement already satisfied: packaging>=20.9 in /usr/local/lib/python3.8/dist-packages (from huggingface-hub~=0.8->huggingface_sb3) (21.3)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.8/dist-packages (from huggingface-hub~=0.8->huggingface_sb3) (2.25.1)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.8/dist-packages (from huggingface-hub~=0.8->huggingface_sb3) (3.9.0)\n",
            "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging>=20.9->huggingface-hub~=0.8->huggingface_sb3) (3.0.9)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.8/dist-packages (from requests->huggingface-hub~=0.8->huggingface_sb3) (2.10)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.8/dist-packages (from requests->huggingface-hub~=0.8->huggingface_sb3) (2022.12.7)\n",
            "Requirement already satisfied: chardet<5,>=3.0.2 in /usr/local/lib/python3.8/dist-packages (from requests->huggingface-hub~=0.8->huggingface_sb3) (4.0.0)\n",
            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.8/dist-packages (from requests->huggingface-hub~=0.8->huggingface_sb3) (1.24.3)\n",
            "Installing collected packages: huggingface_sb3\n",
            "Successfully installed huggingface_sb3-2.2.4\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from huggingface_hub import HfApi, snapshot_download\n",
        "from huggingface_hub.repocard import metadata_eval_result, metadata_save\n",
        "\n",
        "from pathlib import Path\n",
        "import datetime\n",
        "import json"
      ],
      "metadata": {
        "id": "n3u_yTIT0Ry_"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!python train.py --algo ppo --env MountainCar-v0 -n 50000 -optimize --n-trials 1000 --n-jobs 2 \\\n",
        "  --sampler tpe --pruner median"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4uguOXWSuxKF",
        "outputId": "42136edf-8b60-4b20-dc8f-ef0fa79999bd"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "========== MountainCar-v0 ==========\n",
            "Seed: 1768576774\n",
            "Loading hyperparameters from: /content/rl-baselines3-zoo/hyperparams/ppo.yml\n",
            "Default hyperparameters for environment (ones being tuned will be overridden):\n",
            "OrderedDict([('ent_coef', 0.0),\n",
            "             ('gae_lambda', 0.98),\n",
            "             ('gamma', 0.99),\n",
            "             ('n_envs', 16),\n",
            "             ('n_epochs', 4),\n",
            "             ('n_steps', 16),\n",
            "             ('n_timesteps', 1000000.0),\n",
            "             ('normalize', True),\n",
            "             ('policy', 'MlpPolicy')])\n",
            "Using 16 environments\n",
            "Overwriting n_timesteps with n=50000\n",
            "Doing 1 intermediate evaluations for pruning based on the number of timesteps. (1 evaluation every 100k timesteps)\n",
            "Normalization activated: {'gamma': 0.99}\n",
            "Optimizing hyperparameters\n",
            "/usr/local/lib/python3.8/dist-packages/optuna/samplers/_tpe/sampler.py:281: ExperimentalWarning: ``multivariate`` option is an experimental feature. The interface can change in the future.\n",
            "  warnings.warn(\n",
            "Sampler: tpe - Pruner: median\n",
            "\u001b[32m[I 2023-01-14 15:07:17,552]\u001b[0m A new study created in memory with name: no-name-61aae8f5-b630-4838-afb0-c9ca9d230c47\u001b[0m\n",
            "/content/rl-baselines3-zoo/rl_zoo3/hyperparams_opt.py:21: FutureWarning: suggest_loguniform has been deprecated in v3.0.0. This feature will be removed in v6.0.0. See https://github.com/optuna/optuna/releases/tag/v3.0.0. Use :func:`~optuna.trial.Trial.suggest_float` instead.\n",
            "  learning_rate = trial.suggest_loguniform(\"learning_rate\", 1e-5, 1)\n",
            "/content/rl-baselines3-zoo/rl_zoo3/hyperparams_opt.py:30: FutureWarning: suggest_uniform has been deprecated in v3.0.0. This feature will be removed in v6.0.0. See https://github.com/optuna/optuna/releases/tag/v3.0.0. Use :func:`~optuna.trial.Trial.suggest_float` instead.\n",
            "  vf_coef = trial.suggest_uniform(\"vf_coef\", 0, 1)\n",
            "/content/rl-baselines3-zoo/rl_zoo3/hyperparams_opt.py:25: FutureWarning: suggest_loguniform has been deprecated in v3.0.0. This feature will be removed in v6.0.0. See https://github.com/optuna/optuna/releases/tag/v3.0.0. Use :func:`~optuna.trial.Trial.suggest_float` instead.\n",
            "  ent_coef = trial.suggest_loguniform(\"ent_coef\", 0.00000001, 0.1)\n",
            "Normalization activated: {'gamma': 0.99}\n",
            "Normalization activated: {'gamma': 0.99}\n",
            "Normalization activated: {'gamma': 0.99, 'norm_reward': False, 'training': False}\n",
            "Normalization activated: {'gamma': 0.99, 'norm_reward': False, 'training': False}\n",
            "\u001b[32m[I 2023-01-14 15:07:55,594]\u001b[0m Trial 0 finished with value: -200.0 and parameters: {'batch_size': 512, 'n_steps': 1024, 'gamma': 0.9999, 'learning_rate': 0.01618908421102656, 'ent_coef': 2.8845476667052317e-08, 'clip_range': 0.4, 'n_epochs': 5, 'gae_lambda': 0.98, 'max_grad_norm': 0.8, 'vf_coef': 0.777717585845103, 'net_arch': 'small', 'activation_fn': 'relu'}. Best is trial 0 with value: -200.0.\u001b[0m\n",
            "Normalization activated: {'gamma': 0.99}\n",
            "Normalization activated: {'gamma': 0.99, 'norm_reward': False, 'training': False}\n",
            "Number of finished trials:  3\n",
            "Best trial:\n",
            "Value:  -200.0\n",
            "Params: \n",
            "    batch_size: 512\n",
            "    n_steps: 1024\n",
            "    gamma: 0.9999\n",
            "    learning_rate: 0.01618908421102656\n",
            "    ent_coef: 2.8845476667052317e-08\n",
            "    clip_range: 0.4\n",
            "    n_epochs: 5\n",
            "    gae_lambda: 0.98\n",
            "    max_grad_norm: 0.8\n",
            "    vf_coef: 0.777717585845103\n",
            "    net_arch: small\n",
            "    activation_fn: relu\n",
            "Writing report to logs/ppo/report_MountainCar-v0_1000-trials-50000-tpe-median_1673708909\n",
            "^C\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Xr1TVW4xfbz3",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "8af96e66-23d6-4382-e613-4956cc465eb6"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Traceback (most recent call last):\n",
            "  File \"train.py\", line 4, in <module>\n",
            "    train()\n",
            "  File \"/content/rl-baselines3-zoo/rl_zoo3/train.py\", line 180, in train\n",
            "    raise ValueError(f\"{env_id} not found in gym registry, you maybe meant {closest_match}?\")\n",
            "ValueError: SpaceInvadersNoFrameskip-v4 not found in gym registry, you maybe meant 'no close match found...'?\n"
          ]
        }
      ],
      "source": [
        "!python train.py --algo dqn --env SpaceInvadersNoFrameskip-v4 -n 50000 -optimize --n-trials 1000 --n-jobs 2 -f logs/"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SeChoX-3SZfP"
      },
      "source": [
        "#### Solution"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "PuocgdokSab9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "20bb3673-0d89-4905-b9ce-d36d2b0c9ef0"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[1;30;43mStreaming output truncated to the last 5000 lines.\u001b[0m\n",
            "|    ep_rew_mean      | 385      |\n",
            "|    exploration_rate | 0.429    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2736     |\n",
            "|    fps              | 298      |\n",
            "|    time_elapsed     | 1932     |\n",
            "|    total_timesteps  | 577217   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0165   |\n",
            "|    n_updates        | 119304   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.63e+03 |\n",
            "|    ep_rew_mean      | 384      |\n",
            "|    exploration_rate | 0.428    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2740     |\n",
            "|    fps              | 298      |\n",
            "|    time_elapsed     | 1936     |\n",
            "|    total_timesteps  | 578261   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0204   |\n",
            "|    n_updates        | 119565   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.6e+03  |\n",
            "|    ep_rew_mean      | 380      |\n",
            "|    exploration_rate | 0.426    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2744     |\n",
            "|    fps              | 298      |\n",
            "|    time_elapsed     | 1940     |\n",
            "|    total_timesteps  | 579359   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0708   |\n",
            "|    n_updates        | 119839   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.6e+03  |\n",
            "|    ep_rew_mean      | 375      |\n",
            "|    exploration_rate | 0.426    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2748     |\n",
            "|    fps              | 298      |\n",
            "|    time_elapsed     | 1943     |\n",
            "|    total_timesteps  | 580247   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0174   |\n",
            "|    n_updates        | 120061   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.61e+03 |\n",
            "|    ep_rew_mean      | 379      |\n",
            "|    exploration_rate | 0.424    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2752     |\n",
            "|    fps              | 298      |\n",
            "|    time_elapsed     | 1947     |\n",
            "|    total_timesteps  | 581335   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0332   |\n",
            "|    n_updates        | 120333   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.6e+03  |\n",
            "|    ep_rew_mean      | 378      |\n",
            "|    exploration_rate | 0.423    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2756     |\n",
            "|    fps              | 298      |\n",
            "|    time_elapsed     | 1952     |\n",
            "|    total_timesteps  | 582783   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0169   |\n",
            "|    n_updates        | 120695   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.63e+03 |\n",
            "|    ep_rew_mean      | 383      |\n",
            "|    exploration_rate | 0.421    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2760     |\n",
            "|    fps              | 298      |\n",
            "|    time_elapsed     | 1959     |\n",
            "|    total_timesteps  | 584676   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0298   |\n",
            "|    n_updates        | 121168   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.62e+03 |\n",
            "|    ep_rew_mean      | 382      |\n",
            "|    exploration_rate | 0.42     |\n",
            "| time/               |          |\n",
            "|    episodes         | 2764     |\n",
            "|    fps              | 298      |\n",
            "|    time_elapsed     | 1966     |\n",
            "|    total_timesteps  | 586083   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0312   |\n",
            "|    n_updates        | 121520   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.59e+03 |\n",
            "|    ep_rew_mean      | 381      |\n",
            "|    exploration_rate | 0.419    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2768     |\n",
            "|    fps              | 297      |\n",
            "|    time_elapsed     | 1970     |\n",
            "|    total_timesteps  | 587051   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0631   |\n",
            "|    n_updates        | 121762   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.59e+03 |\n",
            "|    ep_rew_mean      | 383      |\n",
            "|    exploration_rate | 0.417    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2772     |\n",
            "|    fps              | 297      |\n",
            "|    time_elapsed     | 1978     |\n",
            "|    total_timesteps  | 589213   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0176   |\n",
            "|    n_updates        | 122303   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.61e+03 |\n",
            "|    ep_rew_mean      | 386      |\n",
            "|    exploration_rate | 0.415    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2776     |\n",
            "|    fps              | 297      |\n",
            "|    time_elapsed     | 1983     |\n",
            "|    total_timesteps  | 590495   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0311   |\n",
            "|    n_updates        | 122623   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.61e+03 |\n",
            "|    ep_rew_mean      | 390      |\n",
            "|    exploration_rate | 0.414    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2780     |\n",
            "|    fps              | 297      |\n",
            "|    time_elapsed     | 1989     |\n",
            "|    total_timesteps  | 592091   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0358   |\n",
            "|    n_updates        | 123022   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.64e+03 |\n",
            "|    ep_rew_mean      | 391      |\n",
            "|    exploration_rate | 0.412    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2784     |\n",
            "|    fps              | 297      |\n",
            "|    time_elapsed     | 1995     |\n",
            "|    total_timesteps  | 593890   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0119   |\n",
            "|    n_updates        | 123472   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.61e+03 |\n",
            "|    ep_rew_mean      | 386      |\n",
            "|    exploration_rate | 0.411    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2788     |\n",
            "|    fps              | 297      |\n",
            "|    time_elapsed     | 2000     |\n",
            "|    total_timesteps  | 595284   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0386   |\n",
            "|    n_updates        | 123820   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.62e+03 |\n",
            "|    ep_rew_mean      | 389      |\n",
            "|    exploration_rate | 0.41     |\n",
            "| time/               |          |\n",
            "|    episodes         | 2792     |\n",
            "|    fps              | 297      |\n",
            "|    time_elapsed     | 2004     |\n",
            "|    total_timesteps  | 596413   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0443   |\n",
            "|    n_updates        | 124103   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.63e+03 |\n",
            "|    ep_rew_mean      | 392      |\n",
            "|    exploration_rate | 0.408    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2796     |\n",
            "|    fps              | 297      |\n",
            "|    time_elapsed     | 2009     |\n",
            "|    total_timesteps  | 597617   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0256   |\n",
            "|    n_updates        | 124404   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.62e+03 |\n",
            "|    ep_rew_mean      | 393      |\n",
            "|    exploration_rate | 0.407    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2800     |\n",
            "|    fps              | 297      |\n",
            "|    time_elapsed     | 2015     |\n",
            "|    total_timesteps  | 599426   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.026    |\n",
            "|    n_updates        | 124856   |\n",
            "----------------------------------\n",
            "Eval num_timesteps=600000, episode_reward=423.00 +/- 161.26\n",
            "Episode length: 3457.80 +/- 769.09\n",
            "----------------------------------\n",
            "| eval/               |          |\n",
            "|    mean_ep_length   | 3.46e+03 |\n",
            "|    mean_reward      | 423      |\n",
            "| rollout/            |          |\n",
            "|    exploration_rate | 0.406    |\n",
            "| time/               |          |\n",
            "|    total_timesteps  | 600000   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0403   |\n",
            "|    n_updates        | 124999   |\n",
            "----------------------------------\n",
            "New best mean reward!\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.64e+03 |\n",
            "|    ep_rew_mean      | 396      |\n",
            "|    exploration_rate | 0.406    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2804     |\n",
            "|    fps              | 296      |\n",
            "|    time_elapsed     | 2026     |\n",
            "|    total_timesteps  | 600263   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.00925  |\n",
            "|    n_updates        | 125065   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.61e+03 |\n",
            "|    ep_rew_mean      | 394      |\n",
            "|    exploration_rate | 0.405    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2808     |\n",
            "|    fps              | 296      |\n",
            "|    time_elapsed     | 2030     |\n",
            "|    total_timesteps  | 601273   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0865   |\n",
            "|    n_updates        | 125318   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.6e+03  |\n",
            "|    ep_rew_mean      | 391      |\n",
            "|    exploration_rate | 0.404    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2812     |\n",
            "|    fps              | 296      |\n",
            "|    time_elapsed     | 2033     |\n",
            "|    total_timesteps  | 602061   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.02     |\n",
            "|    n_updates        | 125515   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.58e+03 |\n",
            "|    ep_rew_mean      | 391      |\n",
            "|    exploration_rate | 0.403    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2816     |\n",
            "|    fps              | 296      |\n",
            "|    time_elapsed     | 2036     |\n",
            "|    total_timesteps  | 602911   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.015    |\n",
            "|    n_updates        | 125727   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.54e+03 |\n",
            "|    ep_rew_mean      | 390      |\n",
            "|    exploration_rate | 0.402    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2820     |\n",
            "|    fps              | 296      |\n",
            "|    time_elapsed     | 2039     |\n",
            "|    total_timesteps  | 603625   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0182   |\n",
            "|    n_updates        | 125906   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.55e+03 |\n",
            "|    ep_rew_mean      | 391      |\n",
            "|    exploration_rate | 0.402    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2824     |\n",
            "|    fps              | 295      |\n",
            "|    time_elapsed     | 2042     |\n",
            "|    total_timesteps  | 604495   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0117   |\n",
            "|    n_updates        | 126123   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.53e+03 |\n",
            "|    ep_rew_mean      | 389      |\n",
            "|    exploration_rate | 0.401    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2828     |\n",
            "|    fps              | 295      |\n",
            "|    time_elapsed     | 2045     |\n",
            "|    total_timesteps  | 605239   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0292   |\n",
            "|    n_updates        | 126309   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.5e+03  |\n",
            "|    ep_rew_mean      | 386      |\n",
            "|    exploration_rate | 0.4      |\n",
            "| time/               |          |\n",
            "|    episodes         | 2832     |\n",
            "|    fps              | 295      |\n",
            "|    time_elapsed     | 2048     |\n",
            "|    total_timesteps  | 606294   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.00908  |\n",
            "|    n_updates        | 126573   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.49e+03 |\n",
            "|    ep_rew_mean      | 384      |\n",
            "|    exploration_rate | 0.399    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2836     |\n",
            "|    fps              | 295      |\n",
            "|    time_elapsed     | 2051     |\n",
            "|    total_timesteps  | 607031   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0161   |\n",
            "|    n_updates        | 126757   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.49e+03 |\n",
            "|    ep_rew_mean      | 386      |\n",
            "|    exploration_rate | 0.398    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2840     |\n",
            "|    fps              | 295      |\n",
            "|    time_elapsed     | 2054     |\n",
            "|    total_timesteps  | 607940   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0147   |\n",
            "|    n_updates        | 126984   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.48e+03 |\n",
            "|    ep_rew_mean      | 383      |\n",
            "|    exploration_rate | 0.397    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2844     |\n",
            "|    fps              | 295      |\n",
            "|    time_elapsed     | 2057     |\n",
            "|    total_timesteps  | 608716   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0283   |\n",
            "|    n_updates        | 127178   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.47e+03 |\n",
            "|    ep_rew_mean      | 382      |\n",
            "|    exploration_rate | 0.397    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2848     |\n",
            "|    fps              | 295      |\n",
            "|    time_elapsed     | 2060     |\n",
            "|    total_timesteps  | 609414   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0248   |\n",
            "|    n_updates        | 127353   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.43e+03 |\n",
            "|    ep_rew_mean      | 374      |\n",
            "|    exploration_rate | 0.395    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2852     |\n",
            "|    fps              | 295      |\n",
            "|    time_elapsed     | 2064     |\n",
            "|    total_timesteps  | 610623   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.00741  |\n",
            "|    n_updates        | 127655   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.44e+03 |\n",
            "|    ep_rew_mean      | 378      |\n",
            "|    exploration_rate | 0.394    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2856     |\n",
            "|    fps              | 295      |\n",
            "|    time_elapsed     | 2068     |\n",
            "|    total_timesteps  | 611650   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0156   |\n",
            "|    n_updates        | 127912   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.43e+03 |\n",
            "|    ep_rew_mean      | 377      |\n",
            "|    exploration_rate | 0.394    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2860     |\n",
            "|    fps              | 295      |\n",
            "|    time_elapsed     | 2072     |\n",
            "|    total_timesteps  | 612619   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.016    |\n",
            "|    n_updates        | 128154   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.41e+03 |\n",
            "|    ep_rew_mean      | 374      |\n",
            "|    exploration_rate | 0.393    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2864     |\n",
            "|    fps              | 295      |\n",
            "|    time_elapsed     | 2074     |\n",
            "|    total_timesteps  | 613044   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.024    |\n",
            "|    n_updates        | 128260   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.39e+03 |\n",
            "|    ep_rew_mean      | 366      |\n",
            "|    exploration_rate | 0.392    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2868     |\n",
            "|    fps              | 295      |\n",
            "|    time_elapsed     | 2078     |\n",
            "|    total_timesteps  | 614252   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.00566  |\n",
            "|    n_updates        | 128562   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.39e+03 |\n",
            "|    ep_rew_mean      | 367      |\n",
            "|    exploration_rate | 0.391    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2872     |\n",
            "|    fps              | 295      |\n",
            "|    time_elapsed     | 2082     |\n",
            "|    total_timesteps  | 615254   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0145   |\n",
            "|    n_updates        | 128813   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.4e+03  |\n",
            "|    ep_rew_mean      | 368      |\n",
            "|    exploration_rate | 0.39     |\n",
            "| time/               |          |\n",
            "|    episodes         | 2876     |\n",
            "|    fps              | 295      |\n",
            "|    time_elapsed     | 2085     |\n",
            "|    total_timesteps  | 616221   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0368   |\n",
            "|    n_updates        | 129055   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.37e+03 |\n",
            "|    ep_rew_mean      | 366      |\n",
            "|    exploration_rate | 0.389    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2880     |\n",
            "|    fps              | 295      |\n",
            "|    time_elapsed     | 2089     |\n",
            "|    total_timesteps  | 617338   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0427   |\n",
            "|    n_updates        | 129334   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.34e+03 |\n",
            "|    ep_rew_mean      | 365      |\n",
            "|    exploration_rate | 0.388    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2884     |\n",
            "|    fps              | 295      |\n",
            "|    time_elapsed     | 2092     |\n",
            "|    total_timesteps  | 618155   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0408   |\n",
            "|    n_updates        | 129538   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.33e+03 |\n",
            "|    ep_rew_mean      | 362      |\n",
            "|    exploration_rate | 0.387    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2888     |\n",
            "|    fps              | 295      |\n",
            "|    time_elapsed     | 2096     |\n",
            "|    total_timesteps  | 619184   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.00915  |\n",
            "|    n_updates        | 129795   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.29e+03 |\n",
            "|    ep_rew_mean      | 354      |\n",
            "|    exploration_rate | 0.386    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2892     |\n",
            "|    fps              | 295      |\n",
            "|    time_elapsed     | 2099     |\n",
            "|    total_timesteps  | 620069   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0134   |\n",
            "|    n_updates        | 130017   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.28e+03 |\n",
            "|    ep_rew_mean      | 353      |\n",
            "|    exploration_rate | 0.385    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2896     |\n",
            "|    fps              | 295      |\n",
            "|    time_elapsed     | 2104     |\n",
            "|    total_timesteps  | 621178   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0144   |\n",
            "|    n_updates        | 130294   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.27e+03 |\n",
            "|    ep_rew_mean      | 352      |\n",
            "|    exploration_rate | 0.384    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2900     |\n",
            "|    fps              | 295      |\n",
            "|    time_elapsed     | 2107     |\n",
            "|    total_timesteps  | 622017   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0131   |\n",
            "|    n_updates        | 130504   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.26e+03 |\n",
            "|    ep_rew_mean      | 351      |\n",
            "|    exploration_rate | 0.383    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2904     |\n",
            "|    fps              | 295      |\n",
            "|    time_elapsed     | 2110     |\n",
            "|    total_timesteps  | 622992   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0177   |\n",
            "|    n_updates        | 130747   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.23e+03 |\n",
            "|    ep_rew_mean      | 344      |\n",
            "|    exploration_rate | 0.383    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2908     |\n",
            "|    fps              | 295      |\n",
            "|    time_elapsed     | 2112     |\n",
            "|    total_timesteps  | 623547   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0134   |\n",
            "|    n_updates        | 130886   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.22e+03 |\n",
            "|    ep_rew_mean      | 340      |\n",
            "|    exploration_rate | 0.382    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2912     |\n",
            "|    fps              | 295      |\n",
            "|    time_elapsed     | 2116     |\n",
            "|    total_timesteps  | 624573   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0328   |\n",
            "|    n_updates        | 131143   |\n",
            "----------------------------------\n",
            "Eval num_timesteps=625000, episode_reward=572.00 +/- 250.99\n",
            "Episode length: 3967.20 +/- 1507.64\n",
            "----------------------------------\n",
            "| eval/               |          |\n",
            "|    mean_ep_length   | 3.97e+03 |\n",
            "|    mean_reward      | 572      |\n",
            "| rollout/            |          |\n",
            "|    exploration_rate | 0.381    |\n",
            "| time/               |          |\n",
            "|    total_timesteps  | 625000   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0198   |\n",
            "|    n_updates        | 131249   |\n",
            "----------------------------------\n",
            "New best mean reward!\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.21e+03 |\n",
            "|    ep_rew_mean      | 340      |\n",
            "|    exploration_rate | 0.381    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2916     |\n",
            "|    fps              | 293      |\n",
            "|    time_elapsed     | 2129     |\n",
            "|    total_timesteps  | 625355   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0578   |\n",
            "|    n_updates        | 131338   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.2e+03  |\n",
            "|    ep_rew_mean      | 339      |\n",
            "|    exploration_rate | 0.379    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2920     |\n",
            "|    fps              | 293      |\n",
            "|    time_elapsed     | 2137     |\n",
            "|    total_timesteps  | 626979   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0127   |\n",
            "|    n_updates        | 131744   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.18e+03 |\n",
            "|    ep_rew_mean      | 337      |\n",
            "|    exploration_rate | 0.378    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2924     |\n",
            "|    fps              | 293      |\n",
            "|    time_elapsed     | 2140     |\n",
            "|    total_timesteps  | 627835   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0134   |\n",
            "|    n_updates        | 131958   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.18e+03 |\n",
            "|    ep_rew_mean      | 337      |\n",
            "|    exploration_rate | 0.377    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2928     |\n",
            "|    fps              | 293      |\n",
            "|    time_elapsed     | 2144     |\n",
            "|    total_timesteps  | 628967   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0048   |\n",
            "|    n_updates        | 132241   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.19e+03 |\n",
            "|    ep_rew_mean      | 337      |\n",
            "|    exploration_rate | 0.376    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2932     |\n",
            "|    fps              | 293      |\n",
            "|    time_elapsed     | 2149     |\n",
            "|    total_timesteps  | 630121   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0312   |\n",
            "|    n_updates        | 132530   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.19e+03 |\n",
            "|    ep_rew_mean      | 334      |\n",
            "|    exploration_rate | 0.375    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2936     |\n",
            "|    fps              | 293      |\n",
            "|    time_elapsed     | 2153     |\n",
            "|    total_timesteps  | 631396   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.025    |\n",
            "|    n_updates        | 132848   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.2e+03  |\n",
            "|    ep_rew_mean      | 336      |\n",
            "|    exploration_rate | 0.373    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2940     |\n",
            "|    fps              | 293      |\n",
            "|    time_elapsed     | 2159     |\n",
            "|    total_timesteps  | 632992   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0225   |\n",
            "|    n_updates        | 133247   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.21e+03 |\n",
            "|    ep_rew_mean      | 338      |\n",
            "|    exploration_rate | 0.372    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2944     |\n",
            "|    fps              | 293      |\n",
            "|    time_elapsed     | 2165     |\n",
            "|    total_timesteps  | 634434   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0262   |\n",
            "|    n_updates        | 133608   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.2e+03  |\n",
            "|    ep_rew_mean      | 337      |\n",
            "|    exploration_rate | 0.371    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2948     |\n",
            "|    fps              | 292      |\n",
            "|    time_elapsed     | 2170     |\n",
            "|    total_timesteps  | 635810   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0197   |\n",
            "|    n_updates        | 133952   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.23e+03 |\n",
            "|    ep_rew_mean      | 339      |\n",
            "|    exploration_rate | 0.369    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2952     |\n",
            "|    fps              | 292      |\n",
            "|    time_elapsed     | 2175     |\n",
            "|    total_timesteps  | 637292   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0328   |\n",
            "|    n_updates        | 134322   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.2e+03  |\n",
            "|    ep_rew_mean      | 334      |\n",
            "|    exploration_rate | 0.368    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2956     |\n",
            "|    fps              | 292      |\n",
            "|    time_elapsed     | 2181     |\n",
            "|    total_timesteps  | 638784   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.00944  |\n",
            "|    n_updates        | 134695   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.23e+03 |\n",
            "|    ep_rew_mean      | 337      |\n",
            "|    exploration_rate | 0.366    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2960     |\n",
            "|    fps              | 292      |\n",
            "|    time_elapsed     | 2186     |\n",
            "|    total_timesteps  | 640252   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0512   |\n",
            "|    n_updates        | 135062   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.24e+03 |\n",
            "|    ep_rew_mean      | 338      |\n",
            "|    exploration_rate | 0.365    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2964     |\n",
            "|    fps              | 292      |\n",
            "|    time_elapsed     | 2191     |\n",
            "|    total_timesteps  | 641544   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0353   |\n",
            "|    n_updates        | 135385   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.24e+03 |\n",
            "|    ep_rew_mean      | 341      |\n",
            "|    exploration_rate | 0.364    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2968     |\n",
            "|    fps              | 292      |\n",
            "|    time_elapsed     | 2195     |\n",
            "|    total_timesteps  | 642606   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0423   |\n",
            "|    n_updates        | 135651   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.24e+03 |\n",
            "|    ep_rew_mean      | 342      |\n",
            "|    exploration_rate | 0.363    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2972     |\n",
            "|    fps              | 292      |\n",
            "|    time_elapsed     | 2199     |\n",
            "|    total_timesteps  | 643822   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0426   |\n",
            "|    n_updates        | 135955   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.27e+03 |\n",
            "|    ep_rew_mean      | 346      |\n",
            "|    exploration_rate | 0.362    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2976     |\n",
            "|    fps              | 292      |\n",
            "|    time_elapsed     | 2203     |\n",
            "|    total_timesteps  | 644880   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0434   |\n",
            "|    n_updates        | 136219   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.27e+03 |\n",
            "|    ep_rew_mean      | 344      |\n",
            "|    exploration_rate | 0.361    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2980     |\n",
            "|    fps              | 292      |\n",
            "|    time_elapsed     | 2206     |\n",
            "|    total_timesteps  | 645670   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0193   |\n",
            "|    n_updates        | 136417   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.28e+03 |\n",
            "|    ep_rew_mean      | 348      |\n",
            "|    exploration_rate | 0.359    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2984     |\n",
            "|    fps              | 292      |\n",
            "|    time_elapsed     | 2213     |\n",
            "|    total_timesteps  | 647401   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0137   |\n",
            "|    n_updates        | 136850   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.3e+03  |\n",
            "|    ep_rew_mean      | 351      |\n",
            "|    exploration_rate | 0.358    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2988     |\n",
            "|    fps              | 292      |\n",
            "|    time_elapsed     | 2217     |\n",
            "|    total_timesteps  | 648536   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0656   |\n",
            "|    n_updates        | 137133   |\n",
            "----------------------------------\n",
            "Eval num_timesteps=650000, episode_reward=355.00 +/- 159.75\n",
            "Episode length: 3345.20 +/- 660.23\n",
            "----------------------------------\n",
            "| eval/               |          |\n",
            "|    mean_ep_length   | 3.35e+03 |\n",
            "|    mean_reward      | 355      |\n",
            "| rollout/            |          |\n",
            "|    exploration_rate | 0.357    |\n",
            "| time/               |          |\n",
            "|    total_timesteps  | 650000   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0199   |\n",
            "|    n_updates        | 137499   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.3e+03  |\n",
            "|    ep_rew_mean      | 351      |\n",
            "|    exploration_rate | 0.356    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2992     |\n",
            "|    fps              | 291      |\n",
            "|    time_elapsed     | 2232     |\n",
            "|    total_timesteps  | 650302   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0638   |\n",
            "|    n_updates        | 137575   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.29e+03 |\n",
            "|    ep_rew_mean      | 353      |\n",
            "|    exploration_rate | 0.355    |\n",
            "| time/               |          |\n",
            "|    episodes         | 2996     |\n",
            "|    fps              | 291      |\n",
            "|    time_elapsed     | 2237     |\n",
            "|    total_timesteps  | 651685   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0221   |\n",
            "|    n_updates        | 137921   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.32e+03 |\n",
            "|    ep_rew_mean      | 354      |\n",
            "|    exploration_rate | 0.353    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3000     |\n",
            "|    fps              | 291      |\n",
            "|    time_elapsed     | 2245     |\n",
            "|    total_timesteps  | 653945   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0731   |\n",
            "|    n_updates        | 138486   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.35e+03 |\n",
            "|    ep_rew_mean      | 360      |\n",
            "|    exploration_rate | 0.351    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3004     |\n",
            "|    fps              | 291      |\n",
            "|    time_elapsed     | 2250     |\n",
            "|    total_timesteps  | 655364   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0176   |\n",
            "|    n_updates        | 138840   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.35e+03 |\n",
            "|    ep_rew_mean      | 359      |\n",
            "|    exploration_rate | 0.349    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3008     |\n",
            "|    fps              | 291      |\n",
            "|    time_elapsed     | 2258     |\n",
            "|    total_timesteps  | 657363   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.01     |\n",
            "|    n_updates        | 139340   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.39e+03 |\n",
            "|    ep_rew_mean      | 364      |\n",
            "|    exploration_rate | 0.348    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3012     |\n",
            "|    fps              | 291      |\n",
            "|    time_elapsed     | 2262     |\n",
            "|    total_timesteps  | 658691   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0161   |\n",
            "|    n_updates        | 139672   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.39e+03 |\n",
            "|    ep_rew_mean      | 363      |\n",
            "|    exploration_rate | 0.347    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3016     |\n",
            "|    fps              | 291      |\n",
            "|    time_elapsed     | 2267     |\n",
            "|    total_timesteps  | 659915   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0353   |\n",
            "|    n_updates        | 139978   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.43e+03 |\n",
            "|    ep_rew_mean      | 366      |\n",
            "|    exploration_rate | 0.345    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3020     |\n",
            "|    fps              | 290      |\n",
            "|    time_elapsed     | 2275     |\n",
            "|    total_timesteps  | 661998   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0339   |\n",
            "|    n_updates        | 140499   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.44e+03 |\n",
            "|    ep_rew_mean      | 367      |\n",
            "|    exploration_rate | 0.344    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3024     |\n",
            "|    fps              | 290      |\n",
            "|    time_elapsed     | 2279     |\n",
            "|    total_timesteps  | 663085   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0199   |\n",
            "|    n_updates        | 140771   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.46e+03 |\n",
            "|    ep_rew_mean      | 367      |\n",
            "|    exploration_rate | 0.342    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3028     |\n",
            "|    fps              | 290      |\n",
            "|    time_elapsed     | 2284     |\n",
            "|    total_timesteps  | 664386   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0136   |\n",
            "|    n_updates        | 141096   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.47e+03 |\n",
            "|    ep_rew_mean      | 371      |\n",
            "|    exploration_rate | 0.341    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3032     |\n",
            "|    fps              | 290      |\n",
            "|    time_elapsed     | 2289     |\n",
            "|    total_timesteps  | 665825   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0314   |\n",
            "|    n_updates        | 141456   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.48e+03 |\n",
            "|    ep_rew_mean      | 374      |\n",
            "|    exploration_rate | 0.339    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3036     |\n",
            "|    fps              | 290      |\n",
            "|    time_elapsed     | 2294     |\n",
            "|    total_timesteps  | 667228   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0423   |\n",
            "|    n_updates        | 141806   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.46e+03 |\n",
            "|    ep_rew_mean      | 376      |\n",
            "|    exploration_rate | 0.338    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3040     |\n",
            "|    fps              | 290      |\n",
            "|    time_elapsed     | 2303     |\n",
            "|    total_timesteps  | 669017   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0161   |\n",
            "|    n_updates        | 142254   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.47e+03 |\n",
            "|    ep_rew_mean      | 374      |\n",
            "|    exploration_rate | 0.336    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3044     |\n",
            "|    fps              | 290      |\n",
            "|    time_elapsed     | 2309     |\n",
            "|    total_timesteps  | 670565   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0438   |\n",
            "|    n_updates        | 142641   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.47e+03 |\n",
            "|    ep_rew_mean      | 380      |\n",
            "|    exploration_rate | 0.335    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3048     |\n",
            "|    fps              | 290      |\n",
            "|    time_elapsed     | 2315     |\n",
            "|    total_timesteps  | 672206   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0241   |\n",
            "|    n_updates        | 143051   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.45e+03 |\n",
            "|    ep_rew_mean      | 377      |\n",
            "|    exploration_rate | 0.333    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3052     |\n",
            "|    fps              | 290      |\n",
            "|    time_elapsed     | 2320     |\n",
            "|    total_timesteps  | 673441   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.00878  |\n",
            "|    n_updates        | 143360   |\n",
            "----------------------------------\n",
            "Eval num_timesteps=675000, episode_reward=449.00 +/- 194.59\n",
            "Episode length: 4670.20 +/- 163.21\n",
            "----------------------------------\n",
            "| eval/               |          |\n",
            "|    mean_ep_length   | 4.67e+03 |\n",
            "|    mean_reward      | 449      |\n",
            "| rollout/            |          |\n",
            "|    exploration_rate | 0.332    |\n",
            "| time/               |          |\n",
            "|    total_timesteps  | 675000   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.051    |\n",
            "|    n_updates        | 143749   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.45e+03 |\n",
            "|    ep_rew_mean      | 377      |\n",
            "|    exploration_rate | 0.331    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3056     |\n",
            "|    fps              | 288      |\n",
            "|    time_elapsed     | 2339     |\n",
            "|    total_timesteps  | 675488   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.013    |\n",
            "|    n_updates        | 143871   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.44e+03 |\n",
            "|    ep_rew_mean      | 378      |\n",
            "|    exploration_rate | 0.33     |\n",
            "| time/               |          |\n",
            "|    episodes         | 3060     |\n",
            "|    fps              | 288      |\n",
            "|    time_elapsed     | 2343     |\n",
            "|    total_timesteps  | 676640   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.115    |\n",
            "|    n_updates        | 144159   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.51e+03 |\n",
            "|    ep_rew_mean      | 383      |\n",
            "|    exploration_rate | 0.328    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3064     |\n",
            "|    fps              | 288      |\n",
            "|    time_elapsed     | 2351     |\n",
            "|    total_timesteps  | 678752   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0812   |\n",
            "|    n_updates        | 144687   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.49e+03 |\n",
            "|    ep_rew_mean      | 383      |\n",
            "|    exploration_rate | 0.326    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3068     |\n",
            "|    fps              | 288      |\n",
            "|    time_elapsed     | 2359     |\n",
            "|    total_timesteps  | 681092   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0157   |\n",
            "|    n_updates        | 145272   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.51e+03 |\n",
            "|    ep_rew_mean      | 387      |\n",
            "|    exploration_rate | 0.324    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3072     |\n",
            "|    fps              | 288      |\n",
            "|    time_elapsed     | 2364     |\n",
            "|    total_timesteps  | 682454   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0163   |\n",
            "|    n_updates        | 145613   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.57e+03 |\n",
            "|    ep_rew_mean      | 397      |\n",
            "|    exploration_rate | 0.322    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3076     |\n",
            "|    fps              | 288      |\n",
            "|    time_elapsed     | 2373     |\n",
            "|    total_timesteps  | 684800   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0203   |\n",
            "|    n_updates        | 146199   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.58e+03 |\n",
            "|    ep_rew_mean      | 400      |\n",
            "|    exploration_rate | 0.32     |\n",
            "| time/               |          |\n",
            "|    episodes         | 3080     |\n",
            "|    fps              | 288      |\n",
            "|    time_elapsed     | 2380     |\n",
            "|    total_timesteps  | 686674   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0186   |\n",
            "|    n_updates        | 146668   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.6e+03  |\n",
            "|    ep_rew_mean      | 398      |\n",
            "|    exploration_rate | 0.319    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3084     |\n",
            "|    fps              | 288      |\n",
            "|    time_elapsed     | 2385     |\n",
            "|    total_timesteps  | 688001   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0653   |\n",
            "|    n_updates        | 147000   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.59e+03 |\n",
            "|    ep_rew_mean      | 394      |\n",
            "|    exploration_rate | 0.318    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3088     |\n",
            "|    fps              | 288      |\n",
            "|    time_elapsed     | 2389     |\n",
            "|    total_timesteps  | 689125   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0621   |\n",
            "|    n_updates        | 147281   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.6e+03  |\n",
            "|    ep_rew_mean      | 398      |\n",
            "|    exploration_rate | 0.316    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3092     |\n",
            "|    fps              | 288      |\n",
            "|    time_elapsed     | 2396     |\n",
            "|    total_timesteps  | 690810   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0174   |\n",
            "|    n_updates        | 147702   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.61e+03 |\n",
            "|    ep_rew_mean      | 398      |\n",
            "|    exploration_rate | 0.314    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3096     |\n",
            "|    fps              | 288      |\n",
            "|    time_elapsed     | 2402     |\n",
            "|    total_timesteps  | 692464   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0301   |\n",
            "|    n_updates        | 148115   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.65e+03 |\n",
            "|    ep_rew_mean      | 402      |\n",
            "|    exploration_rate | 0.313    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3100     |\n",
            "|    fps              | 288      |\n",
            "|    time_elapsed     | 2408     |\n",
            "|    total_timesteps  | 694107   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0159   |\n",
            "|    n_updates        | 148526   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.67e+03 |\n",
            "|    ep_rew_mean      | 405      |\n",
            "|    exploration_rate | 0.311    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3104     |\n",
            "|    fps              | 288      |\n",
            "|    time_elapsed     | 2413     |\n",
            "|    total_timesteps  | 695469   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0261   |\n",
            "|    n_updates        | 148867   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.68e+03 |\n",
            "|    ep_rew_mean      | 407      |\n",
            "|    exploration_rate | 0.31     |\n",
            "| time/               |          |\n",
            "|    episodes         | 3108     |\n",
            "|    fps              | 288      |\n",
            "|    time_elapsed     | 2418     |\n",
            "|    total_timesteps  | 696746   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.00873  |\n",
            "|    n_updates        | 149186   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.71e+03 |\n",
            "|    ep_rew_mean      | 412      |\n",
            "|    exploration_rate | 0.308    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3112     |\n",
            "|    fps              | 288      |\n",
            "|    time_elapsed     | 2425     |\n",
            "|    total_timesteps  | 698524   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0136   |\n",
            "|    n_updates        | 149630   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.72e+03 |\n",
            "|    ep_rew_mean      | 412      |\n",
            "|    exploration_rate | 0.307    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3116     |\n",
            "|    fps              | 287      |\n",
            "|    time_elapsed     | 2429     |\n",
            "|    total_timesteps  | 699662   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0547   |\n",
            "|    n_updates        | 149915   |\n",
            "----------------------------------\n",
            "Eval num_timesteps=700000, episode_reward=450.00 +/- 171.70\n",
            "Episode length: 3552.80 +/- 384.62\n",
            "----------------------------------\n",
            "| eval/               |          |\n",
            "|    mean_ep_length   | 3.55e+03 |\n",
            "|    mean_reward      | 450      |\n",
            "| rollout/            |          |\n",
            "|    exploration_rate | 0.307    |\n",
            "| time/               |          |\n",
            "|    total_timesteps  | 700000   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0149   |\n",
            "|    n_updates        | 149999   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.73e+03 |\n",
            "|    ep_rew_mean      | 415      |\n",
            "|    exploration_rate | 0.305    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3120     |\n",
            "|    fps              | 286      |\n",
            "|    time_elapsed     | 2445     |\n",
            "|    total_timesteps  | 701519   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0275   |\n",
            "|    n_updates        | 150379   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.76e+03 |\n",
            "|    ep_rew_mean      | 420      |\n",
            "|    exploration_rate | 0.304    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3124     |\n",
            "|    fps              | 286      |\n",
            "|    time_elapsed     | 2450     |\n",
            "|    total_timesteps  | 702902   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0258   |\n",
            "|    n_updates        | 150725   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.77e+03 |\n",
            "|    ep_rew_mean      | 420      |\n",
            "|    exploration_rate | 0.303    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3128     |\n",
            "|    fps              | 286      |\n",
            "|    time_elapsed     | 2455     |\n",
            "|    total_timesteps  | 704274   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0314   |\n",
            "|    n_updates        | 151068   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.79e+03 |\n",
            "|    ep_rew_mean      | 424      |\n",
            "|    exploration_rate | 0.301    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3132     |\n",
            "|    fps              | 286      |\n",
            "|    time_elapsed     | 2462     |\n",
            "|    total_timesteps  | 706033   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0165   |\n",
            "|    n_updates        | 151508   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.82e+03 |\n",
            "|    ep_rew_mean      | 431      |\n",
            "|    exploration_rate | 0.299    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3136     |\n",
            "|    fps              | 286      |\n",
            "|    time_elapsed     | 2469     |\n",
            "|    total_timesteps  | 707731   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.04     |\n",
            "|    n_updates        | 151932   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.85e+03 |\n",
            "|    ep_rew_mean      | 436      |\n",
            "|    exploration_rate | 0.298    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3140     |\n",
            "|    fps              | 286      |\n",
            "|    time_elapsed     | 2477     |\n",
            "|    total_timesteps  | 709503   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0211   |\n",
            "|    n_updates        | 152375   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.86e+03 |\n",
            "|    ep_rew_mean      | 437      |\n",
            "|    exploration_rate | 0.296    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3144     |\n",
            "|    fps              | 286      |\n",
            "|    time_elapsed     | 2482     |\n",
            "|    total_timesteps  | 710715   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0208   |\n",
            "|    n_updates        | 152678   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.85e+03 |\n",
            "|    ep_rew_mean      | 436      |\n",
            "|    exploration_rate | 0.295    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3148     |\n",
            "|    fps              | 286      |\n",
            "|    time_elapsed     | 2485     |\n",
            "|    total_timesteps  | 711690   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0316   |\n",
            "|    n_updates        | 152922   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.85e+03 |\n",
            "|    ep_rew_mean      | 435      |\n",
            "|    exploration_rate | 0.294    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3152     |\n",
            "|    fps              | 286      |\n",
            "|    time_elapsed     | 2489     |\n",
            "|    total_timesteps  | 712742   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0515   |\n",
            "|    n_updates        | 153185   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.85e+03 |\n",
            "|    ep_rew_mean      | 434      |\n",
            "|    exploration_rate | 0.293    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3156     |\n",
            "|    fps              | 286      |\n",
            "|    time_elapsed     | 2494     |\n",
            "|    total_timesteps  | 713799   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0118   |\n",
            "|    n_updates        | 153449   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.87e+03 |\n",
            "|    ep_rew_mean      | 436      |\n",
            "|    exploration_rate | 0.292    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3160     |\n",
            "|    fps              | 286      |\n",
            "|    time_elapsed     | 2498     |\n",
            "|    total_timesteps  | 715074   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0142   |\n",
            "|    n_updates        | 153768   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.88e+03 |\n",
            "|    ep_rew_mean      | 439      |\n",
            "|    exploration_rate | 0.291    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3164     |\n",
            "|    fps              | 286      |\n",
            "|    time_elapsed     | 2504     |\n",
            "|    total_timesteps  | 716568   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0143   |\n",
            "|    n_updates        | 154141   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.91e+03 |\n",
            "|    ep_rew_mean      | 443      |\n",
            "|    exploration_rate | 0.289    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3168     |\n",
            "|    fps              | 286      |\n",
            "|    time_elapsed     | 2512     |\n",
            "|    total_timesteps  | 718542   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0523   |\n",
            "|    n_updates        | 154635   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.92e+03 |\n",
            "|    ep_rew_mean      | 446      |\n",
            "|    exploration_rate | 0.288    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3172     |\n",
            "|    fps              | 286      |\n",
            "|    time_elapsed     | 2514     |\n",
            "|    total_timesteps  | 719310   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0363   |\n",
            "|    n_updates        | 154827   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.94e+03 |\n",
            "|    ep_rew_mean      | 451      |\n",
            "|    exploration_rate | 0.286    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3176     |\n",
            "|    fps              | 285      |\n",
            "|    time_elapsed     | 2521     |\n",
            "|    total_timesteps  | 720979   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0263   |\n",
            "|    n_updates        | 155244   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.95e+03 |\n",
            "|    ep_rew_mean      | 453      |\n",
            "|    exploration_rate | 0.284    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3180     |\n",
            "|    fps              | 285      |\n",
            "|    time_elapsed     | 2529     |\n",
            "|    total_timesteps  | 723160   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0608   |\n",
            "|    n_updates        | 155789   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.96e+03 |\n",
            "|    ep_rew_mean      | 454      |\n",
            "|    exploration_rate | 0.283    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3184     |\n",
            "|    fps              | 285      |\n",
            "|    time_elapsed     | 2533     |\n",
            "|    total_timesteps  | 724300   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0126   |\n",
            "|    n_updates        | 156074   |\n",
            "----------------------------------\n",
            "Eval num_timesteps=725000, episode_reward=518.00 +/- 168.12\n",
            "Episode length: 3632.40 +/- 604.64\n",
            "----------------------------------\n",
            "| eval/               |          |\n",
            "|    mean_ep_length   | 3.63e+03 |\n",
            "|    mean_reward      | 518      |\n",
            "| rollout/            |          |\n",
            "|    exploration_rate | 0.282    |\n",
            "| time/               |          |\n",
            "|    total_timesteps  | 725000   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0138   |\n",
            "|    n_updates        | 156249   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.96e+03 |\n",
            "|    ep_rew_mean      | 456      |\n",
            "|    exploration_rate | 0.282    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3188     |\n",
            "|    fps              | 284      |\n",
            "|    time_elapsed     | 2547     |\n",
            "|    total_timesteps  | 725492   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0426   |\n",
            "|    n_updates        | 156372   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.99e+03 |\n",
            "|    ep_rew_mean      | 460      |\n",
            "|    exploration_rate | 0.28     |\n",
            "| time/               |          |\n",
            "|    episodes         | 3192     |\n",
            "|    fps              | 284      |\n",
            "|    time_elapsed     | 2555     |\n",
            "|    total_timesteps  | 727657   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0526   |\n",
            "|    n_updates        | 156914   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.98e+03 |\n",
            "|    ep_rew_mean      | 460      |\n",
            "|    exploration_rate | 0.278    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3196     |\n",
            "|    fps              | 284      |\n",
            "|    time_elapsed     | 2560     |\n",
            "|    total_timesteps  | 728948   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.052    |\n",
            "|    n_updates        | 157236   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.99e+03 |\n",
            "|    ep_rew_mean      | 465      |\n",
            "|    exploration_rate | 0.277    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3200     |\n",
            "|    fps              | 284      |\n",
            "|    time_elapsed     | 2566     |\n",
            "|    total_timesteps  | 730528   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.00933  |\n",
            "|    n_updates        | 157631   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.96e+03 |\n",
            "|    ep_rew_mean      | 464      |\n",
            "|    exploration_rate | 0.275    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3204     |\n",
            "|    fps              | 284      |\n",
            "|    time_elapsed     | 2572     |\n",
            "|    total_timesteps  | 732059   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0252   |\n",
            "|    n_updates        | 158014   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.99e+03 |\n",
            "|    ep_rew_mean      | 464      |\n",
            "|    exploration_rate | 0.274    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3208     |\n",
            "|    fps              | 284      |\n",
            "|    time_elapsed     | 2575     |\n",
            "|    total_timesteps  | 733050   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0124   |\n",
            "|    n_updates        | 158262   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.99e+03 |\n",
            "|    ep_rew_mean      | 467      |\n",
            "|    exploration_rate | 0.272    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3212     |\n",
            "|    fps              | 284      |\n",
            "|    time_elapsed     | 2583     |\n",
            "|    total_timesteps  | 734945   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0202   |\n",
            "|    n_updates        | 158736   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 4e+03    |\n",
            "|    ep_rew_mean      | 471      |\n",
            "|    exploration_rate | 0.271    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3216     |\n",
            "|    fps              | 284      |\n",
            "|    time_elapsed     | 2590     |\n",
            "|    total_timesteps  | 736822   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0302   |\n",
            "|    n_updates        | 159205   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.97e+03 |\n",
            "|    ep_rew_mean      | 468      |\n",
            "|    exploration_rate | 0.27     |\n",
            "| time/               |          |\n",
            "|    episodes         | 3220     |\n",
            "|    fps              | 284      |\n",
            "|    time_elapsed     | 2593     |\n",
            "|    total_timesteps  | 737718   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0131   |\n",
            "|    n_updates        | 159429   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.95e+03 |\n",
            "|    ep_rew_mean      | 469      |\n",
            "|    exploration_rate | 0.269    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3224     |\n",
            "|    fps              | 284      |\n",
            "|    time_elapsed     | 2597     |\n",
            "|    total_timesteps  | 738733   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.016    |\n",
            "|    n_updates        | 159683   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.97e+03 |\n",
            "|    ep_rew_mean      | 472      |\n",
            "|    exploration_rate | 0.266    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3228     |\n",
            "|    fps              | 284      |\n",
            "|    time_elapsed     | 2606     |\n",
            "|    total_timesteps  | 741032   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0126   |\n",
            "|    n_updates        | 160257   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.98e+03 |\n",
            "|    ep_rew_mean      | 477      |\n",
            "|    exploration_rate | 0.265    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3232     |\n",
            "|    fps              | 284      |\n",
            "|    time_elapsed     | 2611     |\n",
            "|    total_timesteps  | 742533   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0222   |\n",
            "|    n_updates        | 160633   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.99e+03 |\n",
            "|    ep_rew_mean      | 479      |\n",
            "|    exploration_rate | 0.264    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3236     |\n",
            "|    fps              | 284      |\n",
            "|    time_elapsed     | 2616     |\n",
            "|    total_timesteps  | 743873   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0135   |\n",
            "|    n_updates        | 160968   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 4e+03    |\n",
            "|    ep_rew_mean      | 482      |\n",
            "|    exploration_rate | 0.262    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3240     |\n",
            "|    fps              | 284      |\n",
            "|    time_elapsed     | 2624     |\n",
            "|    total_timesteps  | 745958   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0418   |\n",
            "|    n_updates        | 161489   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 4e+03    |\n",
            "|    ep_rew_mean      | 487      |\n",
            "|    exploration_rate | 0.26     |\n",
            "| time/               |          |\n",
            "|    episodes         | 3244     |\n",
            "|    fps              | 284      |\n",
            "|    time_elapsed     | 2632     |\n",
            "|    total_timesteps  | 747957   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0145   |\n",
            "|    n_updates        | 161989   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 4e+03    |\n",
            "|    ep_rew_mean      | 492      |\n",
            "|    exploration_rate | 0.258    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3248     |\n",
            "|    fps              | 283      |\n",
            "|    time_elapsed     | 2639     |\n",
            "|    total_timesteps  | 749532   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0309   |\n",
            "|    n_updates        | 162382   |\n",
            "----------------------------------\n",
            "Eval num_timesteps=750000, episode_reward=292.00 +/- 117.03\n",
            "Episode length: 3133.60 +/- 394.12\n",
            "----------------------------------\n",
            "| eval/               |          |\n",
            "|    mean_ep_length   | 3.13e+03 |\n",
            "|    mean_reward      | 292      |\n",
            "| rollout/            |          |\n",
            "|    exploration_rate | 0.258    |\n",
            "| time/               |          |\n",
            "|    total_timesteps  | 750000   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.032    |\n",
            "|    n_updates        | 162499   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.99e+03 |\n",
            "|    ep_rew_mean      | 489      |\n",
            "|    exploration_rate | 0.256    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3252     |\n",
            "|    fps              | 282      |\n",
            "|    time_elapsed     | 2654     |\n",
            "|    total_timesteps  | 751094   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0212   |\n",
            "|    n_updates        | 162773   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.98e+03 |\n",
            "|    ep_rew_mean      | 491      |\n",
            "|    exploration_rate | 0.255    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3256     |\n",
            "|    fps              | 282      |\n",
            "|    time_elapsed     | 2659     |\n",
            "|    total_timesteps  | 752469   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0145   |\n",
            "|    n_updates        | 163117   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.99e+03 |\n",
            "|    ep_rew_mean      | 493      |\n",
            "|    exploration_rate | 0.253    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3260     |\n",
            "|    fps              | 282      |\n",
            "|    time_elapsed     | 2666     |\n",
            "|    total_timesteps  | 754264   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0374   |\n",
            "|    n_updates        | 163565   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.95e+03 |\n",
            "|    ep_rew_mean      | 490      |\n",
            "|    exploration_rate | 0.252    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3264     |\n",
            "|    fps              | 282      |\n",
            "|    time_elapsed     | 2672     |\n",
            "|    total_timesteps  | 755826   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0257   |\n",
            "|    n_updates        | 163956   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.95e+03 |\n",
            "|    ep_rew_mean      | 490      |\n",
            "|    exploration_rate | 0.251    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3268     |\n",
            "|    fps              | 282      |\n",
            "|    time_elapsed     | 2676     |\n",
            "|    total_timesteps  | 756912   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0188   |\n",
            "|    n_updates        | 164227   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.96e+03 |\n",
            "|    ep_rew_mean      | 490      |\n",
            "|    exploration_rate | 0.25     |\n",
            "| time/               |          |\n",
            "|    episodes         | 3272     |\n",
            "|    fps              | 282      |\n",
            "|    time_elapsed     | 2679     |\n",
            "|    total_timesteps  | 757755   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0166   |\n",
            "|    n_updates        | 164438   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.92e+03 |\n",
            "|    ep_rew_mean      | 491      |\n",
            "|    exploration_rate | 0.248    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3276     |\n",
            "|    fps              | 282      |\n",
            "|    time_elapsed     | 2686     |\n",
            "|    total_timesteps  | 759515   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0481   |\n",
            "|    n_updates        | 164878   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.93e+03 |\n",
            "|    ep_rew_mean      | 496      |\n",
            "|    exploration_rate | 0.246    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3280     |\n",
            "|    fps              | 282      |\n",
            "|    time_elapsed     | 2692     |\n",
            "|    total_timesteps  | 761129   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0158   |\n",
            "|    n_updates        | 165282   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.92e+03 |\n",
            "|    ep_rew_mean      | 496      |\n",
            "|    exploration_rate | 0.245    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3284     |\n",
            "|    fps              | 282      |\n",
            "|    time_elapsed     | 2698     |\n",
            "|    total_timesteps  | 762728   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0246   |\n",
            "|    n_updates        | 165681   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.91e+03 |\n",
            "|    ep_rew_mean      | 491      |\n",
            "|    exploration_rate | 0.244    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3288     |\n",
            "|    fps              | 282      |\n",
            "|    time_elapsed     | 2703     |\n",
            "|    total_timesteps  | 763931   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0491   |\n",
            "|    n_updates        | 165982   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.93e+03 |\n",
            "|    ep_rew_mean      | 495      |\n",
            "|    exploration_rate | 0.242    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3292     |\n",
            "|    fps              | 282      |\n",
            "|    time_elapsed     | 2710     |\n",
            "|    total_timesteps  | 765786   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0342   |\n",
            "|    n_updates        | 166446   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.91e+03 |\n",
            "|    ep_rew_mean      | 493      |\n",
            "|    exploration_rate | 0.24     |\n",
            "| time/               |          |\n",
            "|    episodes         | 3296     |\n",
            "|    fps              | 282      |\n",
            "|    time_elapsed     | 2716     |\n",
            "|    total_timesteps  | 767285   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0563   |\n",
            "|    n_updates        | 166821   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.91e+03 |\n",
            "|    ep_rew_mean      | 492      |\n",
            "|    exploration_rate | 0.238    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3300     |\n",
            "|    fps              | 282      |\n",
            "|    time_elapsed     | 2724     |\n",
            "|    total_timesteps  | 769423   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0226   |\n",
            "|    n_updates        | 167355   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.91e+03 |\n",
            "|    ep_rew_mean      | 493      |\n",
            "|    exploration_rate | 0.237    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3304     |\n",
            "|    fps              | 282      |\n",
            "|    time_elapsed     | 2730     |\n",
            "|    total_timesteps  | 770882   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0218   |\n",
            "|    n_updates        | 167720   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.89e+03 |\n",
            "|    ep_rew_mean      | 489      |\n",
            "|    exploration_rate | 0.236    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3308     |\n",
            "|    fps              | 282      |\n",
            "|    time_elapsed     | 2733     |\n",
            "|    total_timesteps  | 771852   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0366   |\n",
            "|    n_updates        | 167962   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.87e+03 |\n",
            "|    ep_rew_mean      | 487      |\n",
            "|    exploration_rate | 0.234    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3312     |\n",
            "|    fps              | 282      |\n",
            "|    time_elapsed     | 2739     |\n",
            "|    total_timesteps  | 773371   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0254   |\n",
            "|    n_updates        | 168342   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.82e+03 |\n",
            "|    ep_rew_mean      | 484      |\n",
            "|    exploration_rate | 0.233    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3316     |\n",
            "|    fps              | 282      |\n",
            "|    time_elapsed     | 2744     |\n",
            "|    total_timesteps  | 774645   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0166   |\n",
            "|    n_updates        | 168661   |\n",
            "----------------------------------\n",
            "Eval num_timesteps=775000, episode_reward=534.00 +/- 98.46\n",
            "Episode length: 3693.00 +/- 736.93\n",
            "----------------------------------\n",
            "| eval/               |          |\n",
            "|    mean_ep_length   | 3.69e+03 |\n",
            "|    mean_reward      | 534      |\n",
            "| rollout/            |          |\n",
            "|    exploration_rate | 0.233    |\n",
            "| time/               |          |\n",
            "|    total_timesteps  | 775000   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0347   |\n",
            "|    n_updates        | 168749   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.8e+03  |\n",
            "|    ep_rew_mean      | 482      |\n",
            "|    exploration_rate | 0.231    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3320     |\n",
            "|    fps              | 281      |\n",
            "|    time_elapsed     | 2762     |\n",
            "|    total_timesteps  | 777051   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0241   |\n",
            "|    n_updates        | 169262   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.76e+03 |\n",
            "|    ep_rew_mean      | 474      |\n",
            "|    exploration_rate | 0.229    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3324     |\n",
            "|    fps              | 281      |\n",
            "|    time_elapsed     | 2767     |\n",
            "|    total_timesteps  | 778289   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0247   |\n",
            "|    n_updates        | 169572   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.72e+03 |\n",
            "|    ep_rew_mean      | 470      |\n",
            "|    exploration_rate | 0.228    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3328     |\n",
            "|    fps              | 281      |\n",
            "|    time_elapsed     | 2772     |\n",
            "|    total_timesteps  | 779578   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0191   |\n",
            "|    n_updates        | 169894   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.73e+03 |\n",
            "|    ep_rew_mean      | 471      |\n",
            "|    exploration_rate | 0.227    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3332     |\n",
            "|    fps              | 281      |\n",
            "|    time_elapsed     | 2778     |\n",
            "|    total_timesteps  | 781095   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0102   |\n",
            "|    n_updates        | 170273   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.72e+03 |\n",
            "|    ep_rew_mean      | 469      |\n",
            "|    exploration_rate | 0.225    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3336     |\n",
            "|    fps              | 281      |\n",
            "|    time_elapsed     | 2784     |\n",
            "|    total_timesteps  | 782818   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0234   |\n",
            "|    n_updates        | 170704   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.73e+03 |\n",
            "|    ep_rew_mean      | 469      |\n",
            "|    exploration_rate | 0.224    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3340     |\n",
            "|    fps              | 281      |\n",
            "|    time_elapsed     | 2789     |\n",
            "|    total_timesteps  | 784017   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0164   |\n",
            "|    n_updates        | 171004   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.73e+03 |\n",
            "|    ep_rew_mean      | 470      |\n",
            "|    exploration_rate | 0.222    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3344     |\n",
            "|    fps              | 280      |\n",
            "|    time_elapsed     | 2796     |\n",
            "|    total_timesteps  | 785786   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0169   |\n",
            "|    n_updates        | 171446   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.72e+03 |\n",
            "|    ep_rew_mean      | 470      |\n",
            "|    exploration_rate | 0.221    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3348     |\n",
            "|    fps              | 280      |\n",
            "|    time_elapsed     | 2800     |\n",
            "|    total_timesteps  | 786699   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0295   |\n",
            "|    n_updates        | 171674   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.75e+03 |\n",
            "|    ep_rew_mean      | 472      |\n",
            "|    exploration_rate | 0.22     |\n",
            "| time/               |          |\n",
            "|    episodes         | 3352     |\n",
            "|    fps              | 280      |\n",
            "|    time_elapsed     | 2807     |\n",
            "|    total_timesteps  | 788290   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0171   |\n",
            "|    n_updates        | 172072   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.74e+03 |\n",
            "|    ep_rew_mean      | 472      |\n",
            "|    exploration_rate | 0.218    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3356     |\n",
            "|    fps              | 280      |\n",
            "|    time_elapsed     | 2813     |\n",
            "|    total_timesteps  | 789658   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.029    |\n",
            "|    n_updates        | 172414   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.73e+03 |\n",
            "|    ep_rew_mean      | 469      |\n",
            "|    exploration_rate | 0.217    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3360     |\n",
            "|    fps              | 280      |\n",
            "|    time_elapsed     | 2818     |\n",
            "|    total_timesteps  | 790849   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0112   |\n",
            "|    n_updates        | 172712   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.73e+03 |\n",
            "|    ep_rew_mean      | 470      |\n",
            "|    exploration_rate | 0.216    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3364     |\n",
            "|    fps              | 280      |\n",
            "|    time_elapsed     | 2824     |\n",
            "|    total_timesteps  | 792285   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.00929  |\n",
            "|    n_updates        | 173071   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.74e+03 |\n",
            "|    ep_rew_mean      | 472      |\n",
            "|    exploration_rate | 0.214    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3368     |\n",
            "|    fps              | 280      |\n",
            "|    time_elapsed     | 2831     |\n",
            "|    total_timesteps  | 794080   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0255   |\n",
            "|    n_updates        | 173519   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.73e+03 |\n",
            "|    ep_rew_mean      | 470      |\n",
            "|    exploration_rate | 0.212    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3372     |\n",
            "|    fps              | 280      |\n",
            "|    time_elapsed     | 2837     |\n",
            "|    total_timesteps  | 795805   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0581   |\n",
            "|    n_updates        | 173951   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.71e+03 |\n",
            "|    ep_rew_mean      | 466      |\n",
            "|    exploration_rate | 0.21     |\n",
            "| time/               |          |\n",
            "|    episodes         | 3376     |\n",
            "|    fps              | 280      |\n",
            "|    time_elapsed     | 2845     |\n",
            "|    total_timesteps  | 797754   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0823   |\n",
            "|    n_updates        | 174438   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.71e+03 |\n",
            "|    ep_rew_mean      | 462      |\n",
            "|    exploration_rate | 0.208    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3380     |\n",
            "|    fps              | 280      |\n",
            "|    time_elapsed     | 2852     |\n",
            "|    total_timesteps  | 799575   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0278   |\n",
            "|    n_updates        | 174893   |\n",
            "----------------------------------\n",
            "Eval num_timesteps=800000, episode_reward=543.00 +/- 149.42\n",
            "Episode length: 3430.60 +/- 413.27\n",
            "----------------------------------\n",
            "| eval/               |          |\n",
            "|    mean_ep_length   | 3.43e+03 |\n",
            "|    mean_reward      | 543      |\n",
            "| rollout/            |          |\n",
            "|    exploration_rate | 0.208    |\n",
            "| time/               |          |\n",
            "|    total_timesteps  | 800000   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0416   |\n",
            "|    n_updates        | 174999   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.69e+03 |\n",
            "|    ep_rew_mean      | 467      |\n",
            "|    exploration_rate | 0.207    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3384     |\n",
            "|    fps              | 279      |\n",
            "|    time_elapsed     | 2868     |\n",
            "|    total_timesteps  | 801474   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0451   |\n",
            "|    n_updates        | 175368   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.69e+03 |\n",
            "|    ep_rew_mean      | 468      |\n",
            "|    exploration_rate | 0.205    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3388     |\n",
            "|    fps              | 279      |\n",
            "|    time_elapsed     | 2873     |\n",
            "|    total_timesteps  | 802727   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0414   |\n",
            "|    n_updates        | 175681   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.68e+03 |\n",
            "|    ep_rew_mean      | 467      |\n",
            "|    exploration_rate | 0.204    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3392     |\n",
            "|    fps              | 279      |\n",
            "|    time_elapsed     | 2877     |\n",
            "|    total_timesteps  | 803705   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0129   |\n",
            "|    n_updates        | 175926   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.7e+03  |\n",
            "|    ep_rew_mean      | 469      |\n",
            "|    exploration_rate | 0.203    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3396     |\n",
            "|    fps              | 279      |\n",
            "|    time_elapsed     | 2881     |\n",
            "|    total_timesteps  | 804874   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0266   |\n",
            "|    n_updates        | 176218   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.7e+03  |\n",
            "|    ep_rew_mean      | 470      |\n",
            "|    exploration_rate | 0.202    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3400     |\n",
            "|    fps              | 279      |\n",
            "|    time_elapsed     | 2886     |\n",
            "|    total_timesteps  | 806236   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0119   |\n",
            "|    n_updates        | 176558   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.68e+03 |\n",
            "|    ep_rew_mean      | 468      |\n",
            "|    exploration_rate | 0.2      |\n",
            "| time/               |          |\n",
            "|    episodes         | 3404     |\n",
            "|    fps              | 279      |\n",
            "|    time_elapsed     | 2893     |\n",
            "|    total_timesteps  | 807836   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0206   |\n",
            "|    n_updates        | 176958   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.64e+03 |\n",
            "|    ep_rew_mean      | 467      |\n",
            "|    exploration_rate | 0.199    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3408     |\n",
            "|    fps              | 279      |\n",
            "|    time_elapsed     | 2898     |\n",
            "|    total_timesteps  | 809264   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.031    |\n",
            "|    n_updates        | 177315   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.65e+03 |\n",
            "|    ep_rew_mean      | 470      |\n",
            "|    exploration_rate | 0.198    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3412     |\n",
            "|    fps              | 279      |\n",
            "|    time_elapsed     | 2903     |\n",
            "|    total_timesteps  | 810413   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0272   |\n",
            "|    n_updates        | 177603   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.64e+03 |\n",
            "|    ep_rew_mean      | 470      |\n",
            "|    exploration_rate | 0.196    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3416     |\n",
            "|    fps              | 279      |\n",
            "|    time_elapsed     | 2911     |\n",
            "|    total_timesteps  | 812547   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0224   |\n",
            "|    n_updates        | 178136   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.66e+03 |\n",
            "|    ep_rew_mean      | 471      |\n",
            "|    exploration_rate | 0.194    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3420     |\n",
            "|    fps              | 279      |\n",
            "|    time_elapsed     | 2917     |\n",
            "|    total_timesteps  | 814147   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0203   |\n",
            "|    n_updates        | 178536   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.66e+03 |\n",
            "|    ep_rew_mean      | 471      |\n",
            "|    exploration_rate | 0.193    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3424     |\n",
            "|    fps              | 279      |\n",
            "|    time_elapsed     | 2922     |\n",
            "|    total_timesteps  | 815358   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0323   |\n",
            "|    n_updates        | 178839   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.63e+03 |\n",
            "|    ep_rew_mean      | 463      |\n",
            "|    exploration_rate | 0.192    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3428     |\n",
            "|    fps              | 278      |\n",
            "|    time_elapsed     | 2925     |\n",
            "|    total_timesteps  | 816184   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0322   |\n",
            "|    n_updates        | 179045   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.64e+03 |\n",
            "|    ep_rew_mean      | 467      |\n",
            "|    exploration_rate | 0.189    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3432     |\n",
            "|    fps              | 278      |\n",
            "|    time_elapsed     | 2936     |\n",
            "|    total_timesteps  | 818959   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0138   |\n",
            "|    n_updates        | 179739   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.63e+03 |\n",
            "|    ep_rew_mean      | 463      |\n",
            "|    exploration_rate | 0.188    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3436     |\n",
            "|    fps              | 278      |\n",
            "|    time_elapsed     | 2939     |\n",
            "|    total_timesteps  | 819883   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0615   |\n",
            "|    n_updates        | 179970   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.63e+03 |\n",
            "|    ep_rew_mean      | 460      |\n",
            "|    exploration_rate | 0.187    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3440     |\n",
            "|    fps              | 278      |\n",
            "|    time_elapsed     | 2945     |\n",
            "|    total_timesteps  | 821247   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0281   |\n",
            "|    n_updates        | 180311   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.66e+03 |\n",
            "|    ep_rew_mean      | 470      |\n",
            "|    exploration_rate | 0.185    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3444     |\n",
            "|    fps              | 278      |\n",
            "|    time_elapsed     | 2953     |\n",
            "|    total_timesteps  | 823509   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0677   |\n",
            "|    n_updates        | 180877   |\n",
            "----------------------------------\n",
            "Eval num_timesteps=825000, episode_reward=518.00 +/- 82.26\n",
            "Episode length: 3883.40 +/- 843.19\n",
            "----------------------------------\n",
            "| eval/               |          |\n",
            "|    mean_ep_length   | 3.88e+03 |\n",
            "|    mean_reward      | 518      |\n",
            "| rollout/            |          |\n",
            "|    exploration_rate | 0.183    |\n",
            "| time/               |          |\n",
            "|    total_timesteps  | 825000   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.042    |\n",
            "|    n_updates        | 181249   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.67e+03 |\n",
            "|    ep_rew_mean      | 476      |\n",
            "|    exploration_rate | 0.182    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3448     |\n",
            "|    fps              | 277      |\n",
            "|    time_elapsed     | 2975     |\n",
            "|    total_timesteps  | 826244   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0529   |\n",
            "|    n_updates        | 181560   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.64e+03 |\n",
            "|    ep_rew_mean      | 471      |\n",
            "|    exploration_rate | 0.18     |\n",
            "| time/               |          |\n",
            "|    episodes         | 3452     |\n",
            "|    fps              | 277      |\n",
            "|    time_elapsed     | 2982     |\n",
            "|    total_timesteps  | 827798   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0364   |\n",
            "|    n_updates        | 181949   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.65e+03 |\n",
            "|    ep_rew_mean      | 473      |\n",
            "|    exploration_rate | 0.179    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3456     |\n",
            "|    fps              | 277      |\n",
            "|    time_elapsed     | 2987     |\n",
            "|    total_timesteps  | 828922   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0131   |\n",
            "|    n_updates        | 182230   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.65e+03 |\n",
            "|    ep_rew_mean      | 473      |\n",
            "|    exploration_rate | 0.178    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3460     |\n",
            "|    fps              | 277      |\n",
            "|    time_elapsed     | 2993     |\n",
            "|    total_timesteps  | 830441   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0371   |\n",
            "|    n_updates        | 182610   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.66e+03 |\n",
            "|    ep_rew_mean      | 472      |\n",
            "|    exploration_rate | 0.176    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3464     |\n",
            "|    fps              | 277      |\n",
            "|    time_elapsed     | 3000     |\n",
            "|    total_timesteps  | 832409   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0174   |\n",
            "|    n_updates        | 183102   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.67e+03 |\n",
            "|    ep_rew_mean      | 471      |\n",
            "|    exploration_rate | 0.175    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3468     |\n",
            "|    fps              | 277      |\n",
            "|    time_elapsed     | 3005     |\n",
            "|    total_timesteps  | 833530   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0461   |\n",
            "|    n_updates        | 183382   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.66e+03 |\n",
            "|    ep_rew_mean      | 468      |\n",
            "|    exploration_rate | 0.173    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3472     |\n",
            "|    fps              | 277      |\n",
            "|    time_elapsed     | 3012     |\n",
            "|    total_timesteps  | 835348   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0537   |\n",
            "|    n_updates        | 183836   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.68e+03 |\n",
            "|    ep_rew_mean      | 466      |\n",
            "|    exploration_rate | 0.171    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3476     |\n",
            "|    fps              | 277      |\n",
            "|    time_elapsed     | 3019     |\n",
            "|    total_timesteps  | 837109   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0212   |\n",
            "|    n_updates        | 184277   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.68e+03 |\n",
            "|    ep_rew_mean      | 468      |\n",
            "|    exploration_rate | 0.17     |\n",
            "| time/               |          |\n",
            "|    episodes         | 3480     |\n",
            "|    fps              | 277      |\n",
            "|    time_elapsed     | 3023     |\n",
            "|    total_timesteps  | 838323   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0202   |\n",
            "|    n_updates        | 184580   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.67e+03 |\n",
            "|    ep_rew_mean      | 463      |\n",
            "|    exploration_rate | 0.168    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3484     |\n",
            "|    fps              | 277      |\n",
            "|    time_elapsed     | 3031     |\n",
            "|    total_timesteps  | 840220   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0153   |\n",
            "|    n_updates        | 185054   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.68e+03 |\n",
            "|    ep_rew_mean      | 464      |\n",
            "|    exploration_rate | 0.166    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3488     |\n",
            "|    fps              | 277      |\n",
            "|    time_elapsed     | 3039     |\n",
            "|    total_timesteps  | 842295   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0429   |\n",
            "|    n_updates        | 185573   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.67e+03 |\n",
            "|    ep_rew_mean      | 463      |\n",
            "|    exploration_rate | 0.164    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3492     |\n",
            "|    fps              | 277      |\n",
            "|    time_elapsed     | 3047     |\n",
            "|    total_timesteps  | 844355   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0255   |\n",
            "|    n_updates        | 186088   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.73e+03 |\n",
            "|    ep_rew_mean      | 471      |\n",
            "|    exploration_rate | 0.16     |\n",
            "| time/               |          |\n",
            "|    episodes         | 3496     |\n",
            "|    fps              | 276      |\n",
            "|    time_elapsed     | 3062     |\n",
            "|    total_timesteps  | 848024   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0696   |\n",
            "|    n_updates        | 187005   |\n",
            "----------------------------------\n",
            "Eval num_timesteps=850000, episode_reward=479.00 +/- 110.25\n",
            "Episode length: 3716.40 +/- 510.85\n",
            "----------------------------------\n",
            "| eval/               |          |\n",
            "|    mean_ep_length   | 3.72e+03 |\n",
            "|    mean_reward      | 479      |\n",
            "| rollout/            |          |\n",
            "|    exploration_rate | 0.159    |\n",
            "| time/               |          |\n",
            "|    total_timesteps  | 850000   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0148   |\n",
            "|    n_updates        | 187499   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.78e+03 |\n",
            "|    ep_rew_mean      | 485      |\n",
            "|    exploration_rate | 0.158    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3500     |\n",
            "|    fps              | 276      |\n",
            "|    time_elapsed     | 3080     |\n",
            "|    total_timesteps  | 850244   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0407   |\n",
            "|    n_updates        | 187560   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.8e+03  |\n",
            "|    ep_rew_mean      | 486      |\n",
            "|    exploration_rate | 0.157    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3504     |\n",
            "|    fps              | 275      |\n",
            "|    time_elapsed     | 3086     |\n",
            "|    total_timesteps  | 851839   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0345   |\n",
            "|    n_updates        | 187959   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.82e+03 |\n",
            "|    ep_rew_mean      | 488      |\n",
            "|    exploration_rate | 0.155    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3508     |\n",
            "|    fps              | 275      |\n",
            "|    time_elapsed     | 3093     |\n",
            "|    total_timesteps  | 853485   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.027    |\n",
            "|    n_updates        | 188371   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.82e+03 |\n",
            "|    ep_rew_mean      | 484      |\n",
            "|    exploration_rate | 0.153    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3512     |\n",
            "|    fps              | 275      |\n",
            "|    time_elapsed     | 3101     |\n",
            "|    total_timesteps  | 855670   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0216   |\n",
            "|    n_updates        | 188917   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.82e+03 |\n",
            "|    ep_rew_mean      | 482      |\n",
            "|    exploration_rate | 0.152    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3516     |\n",
            "|    fps              | 275      |\n",
            "|    time_elapsed     | 3106     |\n",
            "|    total_timesteps  | 856996   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0706   |\n",
            "|    n_updates        | 189248   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.82e+03 |\n",
            "|    ep_rew_mean      | 490      |\n",
            "|    exploration_rate | 0.15     |\n",
            "| time/               |          |\n",
            "|    episodes         | 3520     |\n",
            "|    fps              | 275      |\n",
            "|    time_elapsed     | 3114     |\n",
            "|    total_timesteps  | 859048   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0156   |\n",
            "|    n_updates        | 189761   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.81e+03 |\n",
            "|    ep_rew_mean      | 489      |\n",
            "|    exploration_rate | 0.147    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3524     |\n",
            "|    fps              | 275      |\n",
            "|    time_elapsed     | 3123     |\n",
            "|    total_timesteps  | 861154   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0391   |\n",
            "|    n_updates        | 190288   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.81e+03 |\n",
            "|    ep_rew_mean      | 493      |\n",
            "|    exploration_rate | 0.146    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3528     |\n",
            "|    fps              | 275      |\n",
            "|    time_elapsed     | 3130     |\n",
            "|    total_timesteps  | 862951   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0191   |\n",
            "|    n_updates        | 190737   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.81e+03 |\n",
            "|    ep_rew_mean      | 490      |\n",
            "|    exploration_rate | 0.145    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3532     |\n",
            "|    fps              | 275      |\n",
            "|    time_elapsed     | 3134     |\n",
            "|    total_timesteps  | 864055   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0421   |\n",
            "|    n_updates        | 191013   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.8e+03  |\n",
            "|    ep_rew_mean      | 487      |\n",
            "|    exploration_rate | 0.143    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3536     |\n",
            "|    fps              | 275      |\n",
            "|    time_elapsed     | 3139     |\n",
            "|    total_timesteps  | 865154   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0581   |\n",
            "|    n_updates        | 191288   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.81e+03 |\n",
            "|    ep_rew_mean      | 497      |\n",
            "|    exploration_rate | 0.141    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3540     |\n",
            "|    fps              | 275      |\n",
            "|    time_elapsed     | 3149     |\n",
            "|    total_timesteps  | 867182   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0579   |\n",
            "|    n_updates        | 191795   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.82e+03 |\n",
            "|    ep_rew_mean      | 498      |\n",
            "|    exploration_rate | 0.139    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3544     |\n",
            "|    fps              | 275      |\n",
            "|    time_elapsed     | 3158     |\n",
            "|    total_timesteps  | 869556   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0434   |\n",
            "|    n_updates        | 192388   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.83e+03 |\n",
            "|    ep_rew_mean      | 499      |\n",
            "|    exploration_rate | 0.138    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3548     |\n",
            "|    fps              | 275      |\n",
            "|    time_elapsed     | 3162     |\n",
            "|    total_timesteps  | 870538   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0205   |\n",
            "|    n_updates        | 192634   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.85e+03 |\n",
            "|    ep_rew_mean      | 504      |\n",
            "|    exploration_rate | 0.137    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3552     |\n",
            "|    fps              | 275      |\n",
            "|    time_elapsed     | 3167     |\n",
            "|    total_timesteps  | 871973   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.00871  |\n",
            "|    n_updates        | 192993   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.85e+03 |\n",
            "|    ep_rew_mean      | 510      |\n",
            "|    exploration_rate | 0.134    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3556     |\n",
            "|    fps              | 275      |\n",
            "|    time_elapsed     | 3178     |\n",
            "|    total_timesteps  | 874663   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0294   |\n",
            "|    n_updates        | 193665   |\n",
            "----------------------------------\n",
            "Eval num_timesteps=875000, episode_reward=604.00 +/- 190.75\n",
            "Episode length: 4555.20 +/- 1486.00\n",
            "----------------------------------\n",
            "| eval/               |          |\n",
            "|    mean_ep_length   | 4.56e+03 |\n",
            "|    mean_reward      | 604      |\n",
            "| rollout/            |          |\n",
            "|    exploration_rate | 0.134    |\n",
            "| time/               |          |\n",
            "|    total_timesteps  | 875000   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0389   |\n",
            "|    n_updates        | 193749   |\n",
            "----------------------------------\n",
            "New best mean reward!\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.86e+03 |\n",
            "|    ep_rew_mean      | 512      |\n",
            "|    exploration_rate | 0.132    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3560     |\n",
            "|    fps              | 274      |\n",
            "|    time_elapsed     | 3200     |\n",
            "|    total_timesteps  | 877222   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0193   |\n",
            "|    n_updates        | 194305   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.89e+03 |\n",
            "|    ep_rew_mean      | 516      |\n",
            "|    exploration_rate | 0.131    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3564     |\n",
            "|    fps              | 274      |\n",
            "|    time_elapsed     | 3202     |\n",
            "|    total_timesteps  | 877908   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0164   |\n",
            "|    n_updates        | 194476   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.87e+03 |\n",
            "|    ep_rew_mean      | 515      |\n",
            "|    exploration_rate | 0.13     |\n",
            "| time/               |          |\n",
            "|    episodes         | 3568     |\n",
            "|    fps              | 274      |\n",
            "|    time_elapsed     | 3206     |\n",
            "|    total_timesteps  | 878873   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0154   |\n",
            "|    n_updates        | 194718   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.87e+03 |\n",
            "|    ep_rew_mean      | 518      |\n",
            "|    exploration_rate | 0.128    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3572     |\n",
            "|    fps              | 274      |\n",
            "|    time_elapsed     | 3212     |\n",
            "|    total_timesteps  | 880383   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0178   |\n",
            "|    n_updates        | 195095   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.87e+03 |\n",
            "|    ep_rew_mean      | 522      |\n",
            "|    exploration_rate | 0.127    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3576     |\n",
            "|    fps              | 273      |\n",
            "|    time_elapsed     | 3218     |\n",
            "|    total_timesteps  | 881931   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0609   |\n",
            "|    n_updates        | 195482   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.84e+03 |\n",
            "|    ep_rew_mean      | 520      |\n",
            "|    exploration_rate | 0.126    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3580     |\n",
            "|    fps              | 273      |\n",
            "|    time_elapsed     | 3223     |\n",
            "|    total_timesteps  | 883080   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.013    |\n",
            "|    n_updates        | 195769   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.83e+03 |\n",
            "|    ep_rew_mean      | 520      |\n",
            "|    exploration_rate | 0.124    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3584     |\n",
            "|    fps              | 273      |\n",
            "|    time_elapsed     | 3229     |\n",
            "|    total_timesteps  | 884638   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0207   |\n",
            "|    n_updates        | 196159   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.83e+03 |\n",
            "|    ep_rew_mean      | 523      |\n",
            "|    exploration_rate | 0.123    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3588     |\n",
            "|    fps              | 273      |\n",
            "|    time_elapsed     | 3235     |\n",
            "|    total_timesteps  | 886220   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0233   |\n",
            "|    n_updates        | 196554   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.84e+03 |\n",
            "|    ep_rew_mean      | 523      |\n",
            "|    exploration_rate | 0.122    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3592     |\n",
            "|    fps              | 273      |\n",
            "|    time_elapsed     | 3240     |\n",
            "|    total_timesteps  | 887349   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0633   |\n",
            "|    n_updates        | 196837   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.89e+03 |\n",
            "|    ep_rew_mean      | 531      |\n",
            "|    exploration_rate | 0.119    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3596     |\n",
            "|    fps              | 273      |\n",
            "|    time_elapsed     | 3251     |\n",
            "|    total_timesteps  | 890152   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0198   |\n",
            "|    n_updates        | 197537   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.89e+03 |\n",
            "|    ep_rew_mean      | 529      |\n",
            "|    exploration_rate | 0.117    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3600     |\n",
            "|    fps              | 273      |\n",
            "|    time_elapsed     | 3258     |\n",
            "|    total_timesteps  | 891937   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0366   |\n",
            "|    n_updates        | 197984   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.93e+03 |\n",
            "|    ep_rew_mean      | 542      |\n",
            "|    exploration_rate | 0.114    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3604     |\n",
            "|    fps              | 273      |\n",
            "|    time_elapsed     | 3271     |\n",
            "|    total_timesteps  | 895341   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0872   |\n",
            "|    n_updates        | 198835   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 4.02e+03 |\n",
            "|    ep_rew_mean      | 554      |\n",
            "|    exploration_rate | 0.111    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3608     |\n",
            "|    fps              | 273      |\n",
            "|    time_elapsed     | 3282     |\n",
            "|    total_timesteps  | 897947   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0149   |\n",
            "|    n_updates        | 199486   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 4.01e+03 |\n",
            "|    ep_rew_mean      | 561      |\n",
            "|    exploration_rate | 0.11     |\n",
            "| time/               |          |\n",
            "|    episodes         | 3612     |\n",
            "|    fps              | 273      |\n",
            "|    time_elapsed     | 3287     |\n",
            "|    total_timesteps  | 899287   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0181   |\n",
            "|    n_updates        | 199821   |\n",
            "----------------------------------\n",
            "Eval num_timesteps=900000, episode_reward=739.00 +/- 45.54\n",
            "Episode length: 4337.60 +/- 475.36\n",
            "----------------------------------\n",
            "| eval/               |          |\n",
            "|    mean_ep_length   | 4.34e+03 |\n",
            "|    mean_reward      | 739      |\n",
            "| rollout/            |          |\n",
            "|    exploration_rate | 0.109    |\n",
            "| time/               |          |\n",
            "|    total_timesteps  | 900000   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0204   |\n",
            "|    n_updates        | 199999   |\n",
            "----------------------------------\n",
            "New best mean reward!\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 4.03e+03 |\n",
            "|    ep_rew_mean      | 561      |\n",
            "|    exploration_rate | 0.107    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3616     |\n",
            "|    fps              | 272      |\n",
            "|    time_elapsed     | 3312     |\n",
            "|    total_timesteps  | 902447   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0183   |\n",
            "|    n_updates        | 200611   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 4.04e+03 |\n",
            "|    ep_rew_mean      | 559      |\n",
            "|    exploration_rate | 0.105    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3620     |\n",
            "|    fps              | 272      |\n",
            "|    time_elapsed     | 3318     |\n",
            "|    total_timesteps  | 903683   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0318   |\n",
            "|    n_updates        | 200920   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 4.06e+03 |\n",
            "|    ep_rew_mean      | 566      |\n",
            "|    exploration_rate | 0.103    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3624     |\n",
            "|    fps              | 272      |\n",
            "|    time_elapsed     | 3327     |\n",
            "|    total_timesteps  | 905823   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0144   |\n",
            "|    n_updates        | 201455   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 4.05e+03 |\n",
            "|    ep_rew_mean      | 567      |\n",
            "|    exploration_rate | 0.102    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3628     |\n",
            "|    fps              | 272      |\n",
            "|    time_elapsed     | 3330     |\n",
            "|    total_timesteps  | 906787   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0451   |\n",
            "|    n_updates        | 201696   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 4.08e+03 |\n",
            "|    ep_rew_mean      | 571      |\n",
            "|    exploration_rate | 0.101    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3632     |\n",
            "|    fps              | 272      |\n",
            "|    time_elapsed     | 3335     |\n",
            "|    total_timesteps  | 908027   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0151   |\n",
            "|    n_updates        | 202006   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 4.09e+03 |\n",
            "|    ep_rew_mean      | 573      |\n",
            "|    exploration_rate | 0.0986   |\n",
            "| time/               |          |\n",
            "|    episodes         | 3636     |\n",
            "|    fps              | 272      |\n",
            "|    time_elapsed     | 3345     |\n",
            "|    total_timesteps  | 910462   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0104   |\n",
            "|    n_updates        | 202615   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 4.11e+03 |\n",
            "|    ep_rew_mean      | 573      |\n",
            "|    exploration_rate | 0.0972   |\n",
            "| time/               |          |\n",
            "|    episodes         | 3640     |\n",
            "|    fps              | 272      |\n",
            "|    time_elapsed     | 3351     |\n",
            "|    total_timesteps  | 911940   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0157   |\n",
            "|    n_updates        | 202984   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 4.1e+03  |\n",
            "|    ep_rew_mean      | 572      |\n",
            "|    exploration_rate | 0.0944   |\n",
            "| time/               |          |\n",
            "|    episodes         | 3644     |\n",
            "|    fps              | 272      |\n",
            "|    time_elapsed     | 3362     |\n",
            "|    total_timesteps  | 914712   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0125   |\n",
            "|    n_updates        | 203677   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 4.11e+03 |\n",
            "|    ep_rew_mean      | 578      |\n",
            "|    exploration_rate | 0.0929   |\n",
            "| time/               |          |\n",
            "|    episodes         | 3648     |\n",
            "|    fps              | 271      |\n",
            "|    time_elapsed     | 3368     |\n",
            "|    total_timesteps  | 916293   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0326   |\n",
            "|    n_updates        | 204073   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 4.12e+03 |\n",
            "|    ep_rew_mean      | 580      |\n",
            "|    exploration_rate | 0.0917   |\n",
            "| time/               |          |\n",
            "|    episodes         | 3652     |\n",
            "|    fps              | 271      |\n",
            "|    time_elapsed     | 3373     |\n",
            "|    total_timesteps  | 917475   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.031    |\n",
            "|    n_updates        | 204368   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 4.12e+03 |\n",
            "|    ep_rew_mean      | 582      |\n",
            "|    exploration_rate | 0.0898   |\n",
            "| time/               |          |\n",
            "|    episodes         | 3656     |\n",
            "|    fps              | 271      |\n",
            "|    time_elapsed     | 3381     |\n",
            "|    total_timesteps  | 919421   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0179   |\n",
            "|    n_updates        | 204855   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 4.13e+03 |\n",
            "|    ep_rew_mean      | 581      |\n",
            "|    exploration_rate | 0.0885   |\n",
            "| time/               |          |\n",
            "|    episodes         | 3660     |\n",
            "|    fps              | 271      |\n",
            "|    time_elapsed     | 3386     |\n",
            "|    total_timesteps  | 920703   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0322   |\n",
            "|    n_updates        | 205175   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 4.11e+03 |\n",
            "|    ep_rew_mean      | 579      |\n",
            "|    exploration_rate | 0.0857   |\n",
            "| time/               |          |\n",
            "|    episodes         | 3664     |\n",
            "|    fps              | 271      |\n",
            "|    time_elapsed     | 3397     |\n",
            "|    total_timesteps  | 923515   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0201   |\n",
            "|    n_updates        | 205878   |\n",
            "----------------------------------\n",
            "Eval num_timesteps=925000, episode_reward=526.00 +/- 72.28\n",
            "Episode length: 3557.00 +/- 787.44\n",
            "----------------------------------\n",
            "| eval/               |          |\n",
            "|    mean_ep_length   | 3.56e+03 |\n",
            "|    mean_reward      | 526      |\n",
            "| rollout/            |          |\n",
            "|    exploration_rate | 0.0843   |\n",
            "| time/               |          |\n",
            "|    total_timesteps  | 925000   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0289   |\n",
            "|    n_updates        | 206249   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 4.06e+03 |\n",
            "|    ep_rew_mean      | 572      |\n",
            "|    exploration_rate | 0.0834   |\n",
            "| time/               |          |\n",
            "|    episodes         | 3668     |\n",
            "|    fps              | 271      |\n",
            "|    time_elapsed     | 3416     |\n",
            "|    total_timesteps  | 925864   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.058    |\n",
            "|    n_updates        | 206465   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 4.06e+03 |\n",
            "|    ep_rew_mean      | 572      |\n",
            "|    exploration_rate | 0.0818   |\n",
            "| time/               |          |\n",
            "|    episodes         | 3672     |\n",
            "|    fps              | 270      |\n",
            "|    time_elapsed     | 3422     |\n",
            "|    total_timesteps  | 927437   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0158   |\n",
            "|    n_updates        | 206859   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 4.06e+03 |\n",
            "|    ep_rew_mean      | 572      |\n",
            "|    exploration_rate | 0.081    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3676     |\n",
            "|    fps              | 270      |\n",
            "|    time_elapsed     | 3426     |\n",
            "|    total_timesteps  | 928316   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0255   |\n",
            "|    n_updates        | 207078   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 4.05e+03 |\n",
            "|    ep_rew_mean      | 572      |\n",
            "|    exploration_rate | 0.0791   |\n",
            "| time/               |          |\n",
            "|    episodes         | 3680     |\n",
            "|    fps              | 270      |\n",
            "|    time_elapsed     | 3433     |\n",
            "|    total_timesteps  | 930234   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0184   |\n",
            "|    n_updates        | 207558   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 4.08e+03 |\n",
            "|    ep_rew_mean      | 578      |\n",
            "|    exploration_rate | 0.0773   |\n",
            "| time/               |          |\n",
            "|    episodes         | 3684     |\n",
            "|    fps              | 270      |\n",
            "|    time_elapsed     | 3440     |\n",
            "|    total_timesteps  | 932016   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0234   |\n",
            "|    n_updates        | 208003   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 4.03e+03 |\n",
            "|    ep_rew_mean      | 572      |\n",
            "|    exploration_rate | 0.0761   |\n",
            "| time/               |          |\n",
            "|    episodes         | 3688     |\n",
            "|    fps              | 270      |\n",
            "|    time_elapsed     | 3445     |\n",
            "|    total_timesteps  | 933190   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0188   |\n",
            "|    n_updates        | 208297   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 4.01e+03 |\n",
            "|    ep_rew_mean      | 576      |\n",
            "|    exploration_rate | 0.0739   |\n",
            "| time/               |          |\n",
            "|    episodes         | 3692     |\n",
            "|    fps              | 270      |\n",
            "|    time_elapsed     | 3454     |\n",
            "|    total_timesteps  | 935462   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0251   |\n",
            "|    n_updates        | 208865   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 4.02e+03 |\n",
            "|    ep_rew_mean      | 579      |\n",
            "|    exploration_rate | 0.0713   |\n",
            "| time/               |          |\n",
            "|    episodes         | 3696     |\n",
            "|    fps              | 270      |\n",
            "|    time_elapsed     | 3465     |\n",
            "|    total_timesteps  | 938054   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.126    |\n",
            "|    n_updates        | 209513   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 4.03e+03 |\n",
            "|    ep_rew_mean      | 584      |\n",
            "|    exploration_rate | 0.0693   |\n",
            "| time/               |          |\n",
            "|    episodes         | 3700     |\n",
            "|    fps              | 270      |\n",
            "|    time_elapsed     | 3473     |\n",
            "|    total_timesteps  | 940137   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0952   |\n",
            "|    n_updates        | 210034   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 4.05e+03 |\n",
            "|    ep_rew_mean      | 587      |\n",
            "|    exploration_rate | 0.0681   |\n",
            "| time/               |          |\n",
            "|    episodes         | 3704     |\n",
            "|    fps              | 270      |\n",
            "|    time_elapsed     | 3478     |\n",
            "|    total_timesteps  | 941263   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0128   |\n",
            "|    n_updates        | 210315   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 4.07e+03 |\n",
            "|    ep_rew_mean      | 592      |\n",
            "|    exploration_rate | 0.0667   |\n",
            "| time/               |          |\n",
            "|    episodes         | 3708     |\n",
            "|    fps              | 270      |\n",
            "|    time_elapsed     | 3485     |\n",
            "|    total_timesteps  | 942713   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0123   |\n",
            "|    n_updates        | 210678   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 4.07e+03 |\n",
            "|    ep_rew_mean      | 592      |\n",
            "|    exploration_rate | 0.0645   |\n",
            "| time/               |          |\n",
            "|    episodes         | 3712     |\n",
            "|    fps              | 270      |\n",
            "|    time_elapsed     | 3495     |\n",
            "|    total_timesteps  | 944961   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0161   |\n",
            "|    n_updates        | 211240   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.96e+03 |\n",
            "|    ep_rew_mean      | 580      |\n",
            "|    exploration_rate | 0.0622   |\n",
            "| time/               |          |\n",
            "|    episodes         | 3716     |\n",
            "|    fps              | 270      |\n",
            "|    time_elapsed     | 3503     |\n",
            "|    total_timesteps  | 947223   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0142   |\n",
            "|    n_updates        | 211805   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.97e+03 |\n",
            "|    ep_rew_mean      | 579      |\n",
            "|    exploration_rate | 0.0603   |\n",
            "| time/               |          |\n",
            "|    episodes         | 3720     |\n",
            "|    fps              | 270      |\n",
            "|    time_elapsed     | 3511     |\n",
            "|    total_timesteps  | 949195   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0206   |\n",
            "|    n_updates        | 212298   |\n",
            "----------------------------------\n",
            "Eval num_timesteps=950000, episode_reward=462.00 +/- 182.06\n",
            "Episode length: 3844.80 +/- 1030.57\n",
            "----------------------------------\n",
            "| eval/               |          |\n",
            "|    mean_ep_length   | 3.84e+03 |\n",
            "|    mean_reward      | 462      |\n",
            "| rollout/            |          |\n",
            "|    exploration_rate | 0.0595   |\n",
            "| time/               |          |\n",
            "|    total_timesteps  | 950000   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0379   |\n",
            "|    n_updates        | 212499   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.97e+03 |\n",
            "|    ep_rew_mean      | 583      |\n",
            "|    exploration_rate | 0.0594   |\n",
            "| time/               |          |\n",
            "|    episodes         | 3724     |\n",
            "|    fps              | 269      |\n",
            "|    time_elapsed     | 3525     |\n",
            "|    total_timesteps  | 950092   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.00906  |\n",
            "|    n_updates        | 212522   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.94e+03 |\n",
            "|    ep_rew_mean      | 580      |\n",
            "|    exploration_rate | 0.0587   |\n",
            "| time/               |          |\n",
            "|    episodes         | 3728     |\n",
            "|    fps              | 269      |\n",
            "|    time_elapsed     | 3528     |\n",
            "|    total_timesteps  | 950778   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0168   |\n",
            "|    n_updates        | 212694   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.92e+03 |\n",
            "|    ep_rew_mean      | 577      |\n",
            "|    exploration_rate | 0.0574   |\n",
            "| time/               |          |\n",
            "|    episodes         | 3732     |\n",
            "|    fps              | 269      |\n",
            "|    time_elapsed     | 3534     |\n",
            "|    total_timesteps  | 952147   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0319   |\n",
            "|    n_updates        | 213036   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.94e+03 |\n",
            "|    ep_rew_mean      | 583      |\n",
            "|    exploration_rate | 0.0557   |\n",
            "| time/               |          |\n",
            "|    episodes         | 3736     |\n",
            "|    fps              | 269      |\n",
            "|    time_elapsed     | 3540     |\n",
            "|    total_timesteps  | 953827   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0243   |\n",
            "|    n_updates        | 213456   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.92e+03 |\n",
            "|    ep_rew_mean      | 577      |\n",
            "|    exploration_rate | 0.0539   |\n",
            "| time/               |          |\n",
            "|    episodes         | 3740     |\n",
            "|    fps              | 269      |\n",
            "|    time_elapsed     | 3548     |\n",
            "|    total_timesteps  | 955685   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0106   |\n",
            "|    n_updates        | 213921   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.93e+03 |\n",
            "|    ep_rew_mean      | 574      |\n",
            "|    exploration_rate | 0.052    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3744     |\n",
            "|    fps              | 269      |\n",
            "|    time_elapsed     | 3555     |\n",
            "|    total_timesteps  | 957548   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0278   |\n",
            "|    n_updates        | 214386   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.94e+03 |\n",
            "|    ep_rew_mean      | 575      |\n",
            "|    exploration_rate | 0.0502   |\n",
            "| time/               |          |\n",
            "|    episodes         | 3748     |\n",
            "|    fps              | 269      |\n",
            "|    time_elapsed     | 3563     |\n",
            "|    total_timesteps  | 959364   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.047    |\n",
            "|    n_updates        | 214840   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.93e+03 |\n",
            "|    ep_rew_mean      | 574      |\n",
            "|    exploration_rate | 0.0486   |\n",
            "| time/               |          |\n",
            "|    episodes         | 3752     |\n",
            "|    fps              | 269      |\n",
            "|    time_elapsed     | 3569     |\n",
            "|    total_timesteps  | 960987   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0158   |\n",
            "|    n_updates        | 215246   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.93e+03 |\n",
            "|    ep_rew_mean      | 578      |\n",
            "|    exploration_rate | 0.0465   |\n",
            "| time/               |          |\n",
            "|    episodes         | 3756     |\n",
            "|    fps              | 269      |\n",
            "|    time_elapsed     | 3578     |\n",
            "|    total_timesteps  | 963085   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0113   |\n",
            "|    n_updates        | 215771   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.91e+03 |\n",
            "|    ep_rew_mean      | 568      |\n",
            "|    exploration_rate | 0.0456   |\n",
            "| time/               |          |\n",
            "|    episodes         | 3760     |\n",
            "|    fps              | 269      |\n",
            "|    time_elapsed     | 3582     |\n",
            "|    total_timesteps  | 964077   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.016    |\n",
            "|    n_updates        | 216019   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.89e+03 |\n",
            "|    ep_rew_mean      | 566      |\n",
            "|    exploration_rate | 0.0437   |\n",
            "| time/               |          |\n",
            "|    episodes         | 3764     |\n",
            "|    fps              | 269      |\n",
            "|    time_elapsed     | 3589     |\n",
            "|    total_timesteps  | 965989   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0278   |\n",
            "|    n_updates        | 216497   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.9e+03  |\n",
            "|    ep_rew_mean      | 573      |\n",
            "|    exploration_rate | 0.041    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3768     |\n",
            "|    fps              | 269      |\n",
            "|    time_elapsed     | 3601     |\n",
            "|    total_timesteps  | 968723   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0114   |\n",
            "|    n_updates        | 217180   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.9e+03  |\n",
            "|    ep_rew_mean      | 571      |\n",
            "|    exploration_rate | 0.0391   |\n",
            "| time/               |          |\n",
            "|    episodes         | 3772     |\n",
            "|    fps              | 268      |\n",
            "|    time_elapsed     | 3608     |\n",
            "|    total_timesteps  | 970580   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0152   |\n",
            "|    n_updates        | 217644   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.91e+03 |\n",
            "|    ep_rew_mean      | 572      |\n",
            "|    exploration_rate | 0.0373   |\n",
            "| time/               |          |\n",
            "|    episodes         | 3776     |\n",
            "|    fps              | 268      |\n",
            "|    time_elapsed     | 3615     |\n",
            "|    total_timesteps  | 972382   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0175   |\n",
            "|    n_updates        | 218095   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.86e+03 |\n",
            "|    ep_rew_mean      | 564      |\n",
            "|    exploration_rate | 0.0365   |\n",
            "| time/               |          |\n",
            "|    episodes         | 3780     |\n",
            "|    fps              | 268      |\n",
            "|    time_elapsed     | 3619     |\n",
            "|    total_timesteps  | 973210   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0148   |\n",
            "|    n_updates        | 218302   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.85e+03 |\n",
            "|    ep_rew_mean      | 566      |\n",
            "|    exploration_rate | 0.0349   |\n",
            "| time/               |          |\n",
            "|    episodes         | 3784     |\n",
            "|    fps              | 268      |\n",
            "|    time_elapsed     | 3625     |\n",
            "|    total_timesteps  | 974864   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.00954  |\n",
            "|    n_updates        | 218715   |\n",
            "----------------------------------\n",
            "Eval num_timesteps=975000, episode_reward=604.00 +/- 111.91\n",
            "Episode length: 4096.40 +/- 533.09\n",
            "----------------------------------\n",
            "| eval/               |          |\n",
            "|    mean_ep_length   | 4.1e+03  |\n",
            "|    mean_reward      | 604      |\n",
            "| rollout/            |          |\n",
            "|    exploration_rate | 0.0348   |\n",
            "| time/               |          |\n",
            "|    total_timesteps  | 975000   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0183   |\n",
            "|    n_updates        | 218749   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.88e+03 |\n",
            "|    ep_rew_mean      | 568      |\n",
            "|    exploration_rate | 0.034    |\n",
            "| time/               |          |\n",
            "|    episodes         | 3788     |\n",
            "|    fps              | 268      |\n",
            "|    time_elapsed     | 3640     |\n",
            "|    total_timesteps  | 975799   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0269   |\n",
            "|    n_updates        | 218949   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.9e+03  |\n",
            "|    ep_rew_mean      | 569      |\n",
            "|    exploration_rate | 0.0315   |\n",
            "| time/               |          |\n",
            "|    episodes         | 3792     |\n",
            "|    fps              | 267      |\n",
            "|    time_elapsed     | 3651     |\n",
            "|    total_timesteps  | 978274   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0145   |\n",
            "|    n_updates        | 219568   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.92e+03 |\n",
            "|    ep_rew_mean      | 573      |\n",
            "|    exploration_rate | 0.0297   |\n",
            "| time/               |          |\n",
            "|    episodes         | 3796     |\n",
            "|    fps              | 267      |\n",
            "|    time_elapsed     | 3660     |\n",
            "|    total_timesteps  | 980142   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0286   |\n",
            "|    n_updates        | 220035   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.93e+03 |\n",
            "|    ep_rew_mean      | 579      |\n",
            "|    exploration_rate | 0.0276   |\n",
            "| time/               |          |\n",
            "|    episodes         | 3800     |\n",
            "|    fps              | 267      |\n",
            "|    time_elapsed     | 3668     |\n",
            "|    total_timesteps  | 982197   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0316   |\n",
            "|    n_updates        | 220549   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.96e+03 |\n",
            "|    ep_rew_mean      | 583      |\n",
            "|    exploration_rate | 0.0267   |\n",
            "| time/               |          |\n",
            "|    episodes         | 3804     |\n",
            "|    fps              | 267      |\n",
            "|    time_elapsed     | 3672     |\n",
            "|    total_timesteps  | 983146   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0163   |\n",
            "|    n_updates        | 220786   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.95e+03 |\n",
            "|    ep_rew_mean      | 585      |\n",
            "|    exploration_rate | 0.0256   |\n",
            "| time/               |          |\n",
            "|    episodes         | 3808     |\n",
            "|    fps              | 267      |\n",
            "|    time_elapsed     | 3677     |\n",
            "|    total_timesteps  | 984255   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0292   |\n",
            "|    n_updates        | 221063   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.96e+03 |\n",
            "|    ep_rew_mean      | 587      |\n",
            "|    exploration_rate | 0.0242   |\n",
            "| time/               |          |\n",
            "|    episodes         | 3812     |\n",
            "|    fps              | 267      |\n",
            "|    time_elapsed     | 3682     |\n",
            "|    total_timesteps  | 985672   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0114   |\n",
            "|    n_updates        | 221417   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.95e+03 |\n",
            "|    ep_rew_mean      | 586      |\n",
            "|    exploration_rate | 0.0215   |\n",
            "| time/               |          |\n",
            "|    episodes         | 3816     |\n",
            "|    fps              | 267      |\n",
            "|    time_elapsed     | 3693     |\n",
            "|    total_timesteps  | 988335   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0206   |\n",
            "|    n_updates        | 222083   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.94e+03 |\n",
            "|    ep_rew_mean      | 581      |\n",
            "|    exploration_rate | 0.0187   |\n",
            "| time/               |          |\n",
            "|    episodes         | 3820     |\n",
            "|    fps              | 267      |\n",
            "|    time_elapsed     | 3705     |\n",
            "|    total_timesteps  | 991204   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.014    |\n",
            "|    n_updates        | 222800   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.93e+03 |\n",
            "|    ep_rew_mean      | 582      |\n",
            "|    exploration_rate | 0.0165   |\n",
            "| time/               |          |\n",
            "|    episodes         | 3824     |\n",
            "|    fps              | 267      |\n",
            "|    time_elapsed     | 3714     |\n",
            "|    total_timesteps  | 993398   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0197   |\n",
            "|    n_updates        | 223349   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.91e+03 |\n",
            "|    ep_rew_mean      | 577      |\n",
            "|    exploration_rate | 0.0148   |\n",
            "| time/               |          |\n",
            "|    episodes         | 3828     |\n",
            "|    fps              | 267      |\n",
            "|    time_elapsed     | 3721     |\n",
            "|    total_timesteps  | 995198   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.012    |\n",
            "|    n_updates        | 223799   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.91e+03 |\n",
            "|    ep_rew_mean      | 574      |\n",
            "|    exploration_rate | 0.0125   |\n",
            "| time/               |          |\n",
            "|    episodes         | 3832     |\n",
            "|    fps              | 267      |\n",
            "|    time_elapsed     | 3730     |\n",
            "|    total_timesteps  | 997458   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0432   |\n",
            "|    n_updates        | 224364   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.97e+03 |\n",
            "|    ep_rew_mean      | 580      |\n",
            "|    exploration_rate | 0.0105   |\n",
            "| time/               |          |\n",
            "|    episodes         | 3836     |\n",
            "|    fps              | 267      |\n",
            "|    time_elapsed     | 3739     |\n",
            "|    total_timesteps  | 999523   |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0201   |\n",
            "|    n_updates        | 224880   |\n",
            "----------------------------------\n",
            "Eval num_timesteps=1000000, episode_reward=538.00 +/- 179.01\n",
            "Episode length: 4062.00 +/- 623.02\n",
            "----------------------------------\n",
            "| eval/               |          |\n",
            "|    mean_ep_length   | 4.06e+03 |\n",
            "|    mean_reward      | 538      |\n",
            "| rollout/            |          |\n",
            "|    exploration_rate | 0.01     |\n",
            "| time/               |          |\n",
            "|    total_timesteps  | 1000000  |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0203   |\n",
            "|    n_updates        | 224999   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.97e+03 |\n",
            "|    ep_rew_mean      | 580      |\n",
            "|    exploration_rate | 0.01     |\n",
            "| time/               |          |\n",
            "|    episodes         | 3840     |\n",
            "|    fps              | 266      |\n",
            "|    time_elapsed     | 3755     |\n",
            "|    total_timesteps  | 1000881  |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0115   |\n",
            "|    n_updates        | 225220   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.98e+03 |\n",
            "|    ep_rew_mean      | 586      |\n",
            "|    exploration_rate | 0.01     |\n",
            "| time/               |          |\n",
            "|    episodes         | 3844     |\n",
            "|    fps              | 266      |\n",
            "|    time_elapsed     | 3761     |\n",
            "|    total_timesteps  | 1002362  |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0491   |\n",
            "|    n_updates        | 225590   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.96e+03 |\n",
            "|    ep_rew_mean      | 579      |\n",
            "|    exploration_rate | 0.01     |\n",
            "| time/               |          |\n",
            "|    episodes         | 3848     |\n",
            "|    fps              | 266      |\n",
            "|    time_elapsed     | 3770     |\n",
            "|    total_timesteps  | 1004407  |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0106   |\n",
            "|    n_updates        | 226101   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.97e+03 |\n",
            "|    ep_rew_mean      | 582      |\n",
            "|    exploration_rate | 0.01     |\n",
            "| time/               |          |\n",
            "|    episodes         | 3852     |\n",
            "|    fps              | 266      |\n",
            "|    time_elapsed     | 3777     |\n",
            "|    total_timesteps  | 1006094  |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0197   |\n",
            "|    n_updates        | 226523   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.98e+03 |\n",
            "|    ep_rew_mean      | 586      |\n",
            "|    exploration_rate | 0.01     |\n",
            "| time/               |          |\n",
            "|    episodes         | 3856     |\n",
            "|    fps              | 266      |\n",
            "|    time_elapsed     | 3783     |\n",
            "|    total_timesteps  | 1007599  |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0268   |\n",
            "|    n_updates        | 226899   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.98e+03 |\n",
            "|    ep_rew_mean      | 586      |\n",
            "|    exploration_rate | 0.01     |\n",
            "| time/               |          |\n",
            "|    episodes         | 3860     |\n",
            "|    fps              | 266      |\n",
            "|    time_elapsed     | 3789     |\n",
            "|    total_timesteps  | 1009241  |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0271   |\n",
            "|    n_updates        | 227310   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.99e+03 |\n",
            "|    ep_rew_mean      | 585      |\n",
            "|    exploration_rate | 0.01     |\n",
            "| time/               |          |\n",
            "|    episodes         | 3864     |\n",
            "|    fps              | 266      |\n",
            "|    time_elapsed     | 3792     |\n",
            "|    total_timesteps  | 1009861  |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0213   |\n",
            "|    n_updates        | 227465   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.96e+03 |\n",
            "|    ep_rew_mean      | 580      |\n",
            "|    exploration_rate | 0.01     |\n",
            "| time/               |          |\n",
            "|    episodes         | 3868     |\n",
            "|    fps              | 266      |\n",
            "|    time_elapsed     | 3796     |\n",
            "|    total_timesteps  | 1010870  |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0194   |\n",
            "|    n_updates        | 227717   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.95e+03 |\n",
            "|    ep_rew_mean      | 579      |\n",
            "|    exploration_rate | 0.01     |\n",
            "| time/               |          |\n",
            "|    episodes         | 3872     |\n",
            "|    fps              | 266      |\n",
            "|    time_elapsed     | 3800     |\n",
            "|    total_timesteps  | 1011800  |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0242   |\n",
            "|    n_updates        | 227949   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.95e+03 |\n",
            "|    ep_rew_mean      | 579      |\n",
            "|    exploration_rate | 0.01     |\n",
            "| time/               |          |\n",
            "|    episodes         | 3876     |\n",
            "|    fps              | 266      |\n",
            "|    time_elapsed     | 3805     |\n",
            "|    total_timesteps  | 1012982  |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0372   |\n",
            "|    n_updates        | 228245   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.95e+03 |\n",
            "|    ep_rew_mean      | 577      |\n",
            "|    exploration_rate | 0.01     |\n",
            "| time/               |          |\n",
            "|    episodes         | 3880     |\n",
            "|    fps              | 266      |\n",
            "|    time_elapsed     | 3811     |\n",
            "|    total_timesteps  | 1014542  |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0177   |\n",
            "|    n_updates        | 228635   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 3.98e+03 |\n",
            "|    ep_rew_mean      | 587      |\n",
            "|    exploration_rate | 0.01     |\n",
            "| time/               |          |\n",
            "|    episodes         | 3884     |\n",
            "|    fps              | 266      |\n",
            "|    time_elapsed     | 3822     |\n",
            "|    total_timesteps  | 1016899  |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0136   |\n",
            "|    n_updates        | 229224   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 4e+03    |\n",
            "|    ep_rew_mean      | 590      |\n",
            "|    exploration_rate | 0.01     |\n",
            "| time/               |          |\n",
            "|    episodes         | 3888     |\n",
            "|    fps              | 265      |\n",
            "|    time_elapsed     | 3830     |\n",
            "|    total_timesteps  | 1018530  |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0165   |\n",
            "|    n_updates        | 229632   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 4e+03    |\n",
            "|    ep_rew_mean      | 587      |\n",
            "|    exploration_rate | 0.01     |\n",
            "| time/               |          |\n",
            "|    episodes         | 3892     |\n",
            "|    fps              | 265      |\n",
            "|    time_elapsed     | 3835     |\n",
            "|    total_timesteps  | 1019909  |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.043    |\n",
            "|    n_updates        | 229977   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 4e+03    |\n",
            "|    ep_rew_mean      | 590      |\n",
            "|    exploration_rate | 0.01     |\n",
            "| time/               |          |\n",
            "|    episodes         | 3896     |\n",
            "|    fps              | 265      |\n",
            "|    time_elapsed     | 3841     |\n",
            "|    total_timesteps  | 1021380  |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.00827  |\n",
            "|    n_updates        | 230344   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 4.02e+03 |\n",
            "|    ep_rew_mean      | 591      |\n",
            "|    exploration_rate | 0.01     |\n",
            "| time/               |          |\n",
            "|    episodes         | 3900     |\n",
            "|    fps              | 265      |\n",
            "|    time_elapsed     | 3849     |\n",
            "|    total_timesteps  | 1023151  |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0363   |\n",
            "|    n_updates        | 230787   |\n",
            "----------------------------------\n",
            "Eval num_timesteps=1025000, episode_reward=683.00 +/- 144.38\n",
            "Episode length: 4486.60 +/- 1291.44\n",
            "----------------------------------\n",
            "| eval/               |          |\n",
            "|    mean_ep_length   | 4.49e+03 |\n",
            "|    mean_reward      | 683      |\n",
            "| rollout/            |          |\n",
            "|    exploration_rate | 0.01     |\n",
            "| time/               |          |\n",
            "|    total_timesteps  | 1025000  |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0413   |\n",
            "|    n_updates        | 231249   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 4.1e+03  |\n",
            "|    ep_rew_mean      | 603      |\n",
            "|    exploration_rate | 0.01     |\n",
            "| time/               |          |\n",
            "|    episodes         | 3904     |\n",
            "|    fps              | 264      |\n",
            "|    time_elapsed     | 3873     |\n",
            "|    total_timesteps  | 1026129  |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0143   |\n",
            "|    n_updates        | 231532   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 4.15e+03 |\n",
            "|    ep_rew_mean      | 605      |\n",
            "|    exploration_rate | 0.01     |\n",
            "| time/               |          |\n",
            "|    episodes         | 3908     |\n",
            "|    fps              | 264      |\n",
            "|    time_elapsed     | 3879     |\n",
            "|    total_timesteps  | 1027631  |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0183   |\n",
            "|    n_updates        | 231907   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 4.13e+03 |\n",
            "|    ep_rew_mean      | 605      |\n",
            "|    exploration_rate | 0.01     |\n",
            "| time/               |          |\n",
            "|    episodes         | 3912     |\n",
            "|    fps              | 264      |\n",
            "|    time_elapsed     | 3888     |\n",
            "|    total_timesteps  | 1029742  |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0268   |\n",
            "|    n_updates        | 232435   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 4.13e+03 |\n",
            "|    ep_rew_mean      | 607      |\n",
            "|    exploration_rate | 0.01     |\n",
            "| time/               |          |\n",
            "|    episodes         | 3916     |\n",
            "|    fps              | 264      |\n",
            "|    time_elapsed     | 3890     |\n",
            "|    total_timesteps  | 1030440  |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0299   |\n",
            "|    n_updates        | 232609   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 4.14e+03 |\n",
            "|    ep_rew_mean      | 607      |\n",
            "|    exploration_rate | 0.01     |\n",
            "| time/               |          |\n",
            "|    episodes         | 3920     |\n",
            "|    fps              | 264      |\n",
            "|    time_elapsed     | 3897     |\n",
            "|    total_timesteps  | 1031979  |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0146   |\n",
            "|    n_updates        | 232994   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 4.14e+03 |\n",
            "|    ep_rew_mean      | 608      |\n",
            "|    exploration_rate | 0.01     |\n",
            "| time/               |          |\n",
            "|    episodes         | 3924     |\n",
            "|    fps              | 264      |\n",
            "|    time_elapsed     | 3905     |\n",
            "|    total_timesteps  | 1034029  |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0442   |\n",
            "|    n_updates        | 233507   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 4.17e+03 |\n",
            "|    ep_rew_mean      | 616      |\n",
            "|    exploration_rate | 0.01     |\n",
            "| time/               |          |\n",
            "|    episodes         | 3928     |\n",
            "|    fps              | 264      |\n",
            "|    time_elapsed     | 3912     |\n",
            "|    total_timesteps  | 1035688  |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0158   |\n",
            "|    n_updates        | 233921   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 4.18e+03 |\n",
            "|    ep_rew_mean      | 616      |\n",
            "|    exploration_rate | 0.01     |\n",
            "| time/               |          |\n",
            "|    episodes         | 3932     |\n",
            "|    fps              | 264      |\n",
            "|    time_elapsed     | 3919     |\n",
            "|    total_timesteps  | 1037393  |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0468   |\n",
            "|    n_updates        | 234348   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 4.2e+03  |\n",
            "|    ep_rew_mean      | 618      |\n",
            "|    exploration_rate | 0.01     |\n",
            "| time/               |          |\n",
            "|    episodes         | 3936     |\n",
            "|    fps              | 264      |\n",
            "|    time_elapsed     | 3924     |\n",
            "|    total_timesteps  | 1038768  |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0217   |\n",
            "|    n_updates        | 234691   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 4.19e+03 |\n",
            "|    ep_rew_mean      | 615      |\n",
            "|    exploration_rate | 0.01     |\n",
            "| time/               |          |\n",
            "|    episodes         | 3940     |\n",
            "|    fps              | 264      |\n",
            "|    time_elapsed     | 3933     |\n",
            "|    total_timesteps  | 1040907  |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.146    |\n",
            "|    n_updates        | 235226   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 4.18e+03 |\n",
            "|    ep_rew_mean      | 611      |\n",
            "|    exploration_rate | 0.01     |\n",
            "| time/               |          |\n",
            "|    episodes         | 3944     |\n",
            "|    fps              | 264      |\n",
            "|    time_elapsed     | 3939     |\n",
            "|    total_timesteps  | 1042252  |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.03     |\n",
            "|    n_updates        | 235562   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 4.18e+03 |\n",
            "|    ep_rew_mean      | 607      |\n",
            "|    exploration_rate | 0.01     |\n",
            "| time/               |          |\n",
            "|    episodes         | 3948     |\n",
            "|    fps              | 264      |\n",
            "|    time_elapsed     | 3945     |\n",
            "|    total_timesteps  | 1043738  |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0242   |\n",
            "|    n_updates        | 235934   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 4.17e+03 |\n",
            "|    ep_rew_mean      | 607      |\n",
            "|    exploration_rate | 0.01     |\n",
            "| time/               |          |\n",
            "|    episodes         | 3952     |\n",
            "|    fps              | 264      |\n",
            "|    time_elapsed     | 3950     |\n",
            "|    total_timesteps  | 1044994  |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0536   |\n",
            "|    n_updates        | 236248   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 4.17e+03 |\n",
            "|    ep_rew_mean      | 602      |\n",
            "|    exploration_rate | 0.01     |\n",
            "| time/               |          |\n",
            "|    episodes         | 3956     |\n",
            "|    fps              | 264      |\n",
            "|    time_elapsed     | 3957     |\n",
            "|    total_timesteps  | 1046871  |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0156   |\n",
            "|    n_updates        | 236717   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 4.16e+03 |\n",
            "|    ep_rew_mean      | 601      |\n",
            "|    exploration_rate | 0.01     |\n",
            "| time/               |          |\n",
            "|    episodes         | 3960     |\n",
            "|    fps              | 264      |\n",
            "|    time_elapsed     | 3961     |\n",
            "|    total_timesteps  | 1047836  |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0461   |\n",
            "|    n_updates        | 236958   |\n",
            "----------------------------------\n",
            "Eval num_timesteps=1050000, episode_reward=657.00 +/- 91.96\n",
            "Episode length: 3429.00 +/- 763.97\n",
            "----------------------------------\n",
            "| eval/               |          |\n",
            "|    mean_ep_length   | 3.43e+03 |\n",
            "|    mean_reward      | 657      |\n",
            "| rollout/            |          |\n",
            "|    exploration_rate | 0.01     |\n",
            "| time/               |          |\n",
            "|    total_timesteps  | 1050000  |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0333   |\n",
            "|    n_updates        | 237499   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 4.15e+03 |\n",
            "|    ep_rew_mean      | 596      |\n",
            "|    exploration_rate | 0.01     |\n",
            "| time/               |          |\n",
            "|    episodes         | 3964     |\n",
            "|    fps              | 263      |\n",
            "|    time_elapsed     | 3983     |\n",
            "|    total_timesteps  | 1050823  |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0373   |\n",
            "|    n_updates        | 237705   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 4.15e+03 |\n",
            "|    ep_rew_mean      | 597      |\n",
            "|    exploration_rate | 0.01     |\n",
            "| time/               |          |\n",
            "|    episodes         | 3968     |\n",
            "|    fps              | 263      |\n",
            "|    time_elapsed     | 3996     |\n",
            "|    total_timesteps  | 1053770  |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0732   |\n",
            "|    n_updates        | 238442   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 4.19e+03 |\n",
            "|    ep_rew_mean      | 606      |\n",
            "|    exploration_rate | 0.01     |\n",
            "| time/               |          |\n",
            "|    episodes         | 3972     |\n",
            "|    fps              | 263      |\n",
            "|    time_elapsed     | 4006     |\n",
            "|    total_timesteps  | 1055876  |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0608   |\n",
            "|    n_updates        | 238968   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 4.18e+03 |\n",
            "|    ep_rew_mean      | 606      |\n",
            "|    exploration_rate | 0.01     |\n",
            "| time/               |          |\n",
            "|    episodes         | 3976     |\n",
            "|    fps              | 263      |\n",
            "|    time_elapsed     | 4016     |\n",
            "|    total_timesteps  | 1058407  |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0217   |\n",
            "|    n_updates        | 239601   |\n",
            "----------------------------------\n",
            "----------------------------------\n",
            "| rollout/            |          |\n",
            "|    ep_len_mean      | 4.28e+03 |\n",
            "|    ep_rew_mean      | 620      |\n",
            "|    exploration_rate | 0.01     |\n",
            "| time/               |          |\n",
            "|    episodes         | 3980     |\n",
            "|    fps              | 263      |\n",
            "|    time_elapsed     | 4028     |\n",
            "|    total_timesteps  | 1061351  |\n",
            "| train/              |          |\n",
            "|    learning_rate    | 0.0001   |\n",
            "|    loss             | 0.0172   |\n",
            "|    n_updates        | 240337   |\n",
            "----------------------------------\n",
            "Saving to logs//dqn/SpaceInvadersNoFrameskip-v4_3\n",
            "Traceback (most recent call last):\n",
            "  File \"train.py\", line 4, in <module>\n",
            "    train()\n",
            "  File \"/content/rl-baselines3-zoo/rl_zoo3/train.py\", line 277, in train\n",
            "    exp_manager.save_trained_model(model)\n",
            "  File \"/content/rl-baselines3-zoo/rl_zoo3/exp_manager.py\", line 257, in save_trained_model\n",
            "    model.save(f\"{self.save_path}/{self.env_name}\")\n",
            "  File \"/usr/local/lib/python3.8/dist-packages/stable_baselines3/common/base_class.py\", line 813, in save\n",
            "    save_to_zip_file(path, data=data, params=params_to_save, pytorch_variables=pytorch_variables)\n",
            "  File \"/usr/local/lib/python3.8/dist-packages/stable_baselines3/common/save_util.py\", line 311, in save_to_zip_file\n",
            "    serialized_data = data_to_json(data)\n",
            "  File \"/usr/local/lib/python3.8/dist-packages/stable_baselines3/common/save_util.py\", line 123, in data_to_json\n",
            "    cloudpickle_serialization[variable_name] = str(variable_item)\n",
            "  File \"/usr/local/lib/python3.8/dist-packages/numpy/core/arrayprint.py\", line 1551, in _array_str_implementation\n",
            "    return array2string(a, max_line_width, precision, suppress_small, ' ', \"\")\n",
            "  File \"/usr/local/lib/python3.8/dist-packages/numpy/core/arrayprint.py\", line 698, in array2string\n",
            "    return _array2string(a, options, separator, prefix)\n",
            "  File \"/usr/local/lib/python3.8/dist-packages/numpy/core/arrayprint.py\", line 475, in wrapper\n",
            "    return f(self, *args, **kwargs)\n",
            "  File \"/usr/local/lib/python3.8/dist-packages/numpy/core/arrayprint.py\", line 508, in _array2string\n",
            "    lst = _formatArray(a, format_function, options['linewidth'],\n",
            "  File \"/usr/local/lib/python3.8/dist-packages/numpy/core/arrayprint.py\", line 851, in _formatArray\n",
            "    return recurser(index=(),\n",
            "  File \"/usr/local/lib/python3.8/dist-packages/numpy/core/arrayprint.py\", line 838, in recurser\n",
            "    nested = recurser(index + (-i,), next_hanging_indent,\n",
            "  File \"/usr/local/lib/python3.8/dist-packages/numpy/core/arrayprint.py\", line 827, in recurser\n",
            "    nested = recurser(index + (i,), next_hanging_indent, next_width)\n",
            "  File \"/usr/local/lib/python3.8/dist-packages/numpy/core/arrayprint.py\", line 794, in recurser\n",
            "    s, line = _extendLine_pretty(\n",
            "  File \"/usr/local/lib/python3.8/dist-packages/numpy/core/arrayprint.py\", line 715, in _extendLine_pretty\n",
            "    def _extendLine_pretty(s, line, word, line_width, next_line_prefix, legacy):\n",
            "KeyboardInterrupt\n",
            "^C\n"
          ]
        }
      ],
      "source": [
        "!python train.py --algo dqn  --env SpaceInvadersNoFrameskip-v4 -f logs/"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_dLomIiMKQaf"
      },
      "source": [
        "## Let's evaluate our agent 👀\n",
        "- RL-Baselines3-Zoo provides `enjoy.py`, a python script to evaluate our agent. In most RL libraries, we call the evaluation script `enjoy.py`.\n",
        "- Let's evaluate it for 5000 timesteps 🔥"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "co5um_KeKbBJ"
      },
      "outputs": [],
      "source": [
        "!python enjoy.py  --algo dqn  --env SpaceInvadersNoFrameskip-v4  --no-render  --n-timesteps _________  --folder logs/"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q24K1tyWSj7t"
      },
      "source": [
        "#### Solution"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "P_uSmwGRSk0z",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7545e94b-bfc7-46da-9942-2bff00571af6"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Loading latest experiment, id=1\n",
            "Loading logs/dqn/SpaceInvadersNoFrameskip-v4_1/SpaceInvadersNoFrameskip-v4.zip\n",
            "A.L.E: Arcade Learning Environment (version 0.7.4+069f8bd)\n",
            "[Powered by Stella]\n",
            "Stacking 4 frames\n",
            "Wrapping the env in a VecTransposeImage.\n",
            "Atari Episode Score: 585.00\n",
            "Atari Episode Length 4016\n",
            "Atari Episode Score: 570.00\n",
            "Atari Episode Length 3117\n",
            "Atari Episode Score: 555.00\n",
            "Atari Episode Length 3608\n",
            "Atari Episode Score: 575.00\n",
            "Atari Episode Length 3586\n",
            "Atari Episode Score: 545.00\n",
            "Atari Episode Length 4270\n"
          ]
        }
      ],
      "source": [
        "!python enjoy.py  --algo dqn  --env SpaceInvadersNoFrameskip-v4  --no-render  --n-timesteps 5000  --folder logs/"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "liBeTltiHJtr"
      },
      "source": [
        "## Publish our trained model on the Hub 🚀\n",
        "Now that we saw we got good results after the training, we can publish our trained model on the hub 🤗 with one line of code.\n",
        "\n",
        "<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/unit3/space-invaders-model.gif\" alt=\"Space Invaders model\">"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ezbHS1q3HYVV"
      },
      "source": [
        "By using `rl_zoo3.push_to_hub.py` **you evaluate, record a replay, generate a model card of your agent and push it to the hub**.\n",
        "\n",
        "This way:\n",
        "- You can **showcase our work** 🔥\n",
        "- You can **visualize your agent playing** 👀\n",
        "- You can **share with the community an agent that others can use** 💾\n",
        "- You can **access a leaderboard 🏆 to see how well your agent is performing compared to your classmates** 👉  https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XMSeZRBiHk6X"
      },
      "source": [
        "To be able to share your model with the community there are three more steps to follow:\n",
        "\n",
        "1️⃣ (If it's not already done) create an account to HF ➡ https://huggingface.co/join\n",
        "\n",
        "2️⃣ Sign in and then, you need to store your authentication token from the Hugging Face website.\n",
        "- Create a new token (https://huggingface.co/settings/tokens) **with write role**\n",
        "\n",
        "<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/create-token.jpg\" alt=\"Create HF Token\">"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9O6FI0F8HnzE"
      },
      "source": [
        "- Copy the token \n",
        "- Run the cell below and past the token"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Ppu9yePwHrZX",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 331,
          "referenced_widgets": [
            "0046d9625e0d4c95acf01a02cda7f4e3",
            "f5426df55bb84a93a5324f879f7622ee",
            "42b1b606dd3e4f73b4d9e060ce6a26ff",
            "ffa2b0353f5b438199722e163e594669",
            "3e78aeccf0ef4bb9b8a78cf83e422c68",
            "a663c17108394a5ca036fd26364d35ce",
            "bc68b383d3f94e6d973030124e33c35c",
            "8bc89127573946ed9ffbf2cad0cb108d",
            "8c09e2aeaeff44b5a2ce7ce61e8b70c0",
            "ff70d62b759b4738b3d125ed6eaa4c2f",
            "e2af816ebd154af69218cb10c464eab3",
            "cb990bb2d70f48498958c7e3fee8df5b",
            "bf65637de0844f77bb341f63ae3b82fa",
            "46fc7890d9ce4617a9c456ad6d12cf56",
            "74f8ed71af5e4e23bbfa057970626408",
            "228019b8acb04dc4bdfeb35d8569cbef",
            "7678b80ae0574c4a8c5892db6d4d941c"
          ]
        },
        "outputId": "22c5eb33-2260-483a-9ba2-cbbf28737884"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Token is valid.\n",
            "Your token has been saved in your configured git credential helpers (store).\n",
            "Your token has been saved to /root/.huggingface/token\n",
            "Login successful\n"
          ]
        }
      ],
      "source": [
        "from huggingface_hub import notebook_login # To log to our Hugging Face account to be able to upload models to the Hub.\n",
        "notebook_login()\n",
        "!git config --global credential.helper store"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2RVEdunPHs8B"
      },
      "source": [
        "If you don't want to use a Google Colab or a Jupyter Notebook, you need to use this command instead: `huggingface-cli login`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dSLwdmvhHvjw"
      },
      "source": [
        "3️⃣ We're now ready to push our trained agent to the 🤗 Hub 🔥"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PW436XnhHw1H"
      },
      "source": [
        "Let's run push_to_hub.py file to upload our trained agent to the Hub.\n",
        "\n",
        "`--repo-name `: The name of the repo\n",
        "\n",
        "`-orga`: Your Hugging Face username\n",
        "\n",
        "<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/unit3/select-id.png\" alt=\"Select Id\">"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Ygk2sEktTDEw",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "734a7366-c8d4-46ca-a5de-94351515eb2e"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Loading latest experiment, id=1\n",
            "Loading logs/dqn/SpaceInvadersNoFrameskip-v4_1/SpaceInvadersNoFrameskip-v4.zip\n",
            "A.L.E: Arcade Learning Environment (version 0.7.4+069f8bd)\n",
            "[Powered by Stella]\n",
            "Stacking 4 frames\n",
            "Wrapping the env in a VecTransposeImage.\n",
            "Uploading to Ryukijano/dqn_SINoFrameSkip-v4, make sure to have the rights\n",
            "\u001b[38;5;4mℹ This function will save, evaluate, generate a video of your agent,\n",
            "create a model card and push everything to the hub. It might take up to some\n",
            "minutes if video generation is activated. This is a work in progress: if you\n",
            "encounter a bug, please open an issue.\u001b[0m\n",
            "Cloning https://huggingface.co/Ryukijano/dqn_SINoFrameSkip-v4 into local empty directory.\n",
            "Saving model to: hub/dqn_SINoFrameSkip-v4/dqn-SpaceInvadersNoFrameskip-v4\n",
            "Saving video to /tmp/tmpoddr64wg/-step-0-to-step-1000.mp4\n",
            "ffmpeg version 3.4.11-0ubuntu0.1 Copyright (c) 2000-2022 the FFmpeg developers\n",
            "  built with gcc 7 (Ubuntu 7.5.0-3ubuntu1~18.04)\n",
            "  configuration: --prefix=/usr --extra-version=0ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --enable-gpl --disable-stripping --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-librsvg --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared\n",
            "  libavutil      55. 78.100 / 55. 78.100\n",
            "  libavcodec     57.107.100 / 57.107.100\n",
            "  libavformat    57. 83.100 / 57. 83.100\n",
            "  libavdevice    57. 10.100 / 57. 10.100\n",
            "  libavfilter     6.107.100 /  6.107.100\n",
            "  libavresample   3.  7.  0 /  3.  7.  0\n",
            "  libswscale      4.  8.100 /  4.  8.100\n",
            "  libswresample   2.  9.100 /  2.  9.100\n",
            "  libpostproc    54.  7.100 / 54.  7.100\n",
            "Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/tmp/tmpoddr64wg/-step-0-to-step-1000.mp4':\n",
            "  Metadata:\n",
            "    major_brand     : isom\n",
            "    minor_version   : 512\n",
            "    compatible_brands: isomiso2avc1mp41\n",
            "    encoder         : Lavf57.83.100\n",
            "  Duration: 00:00:33.37, start: 0.000000, bitrate: 49 kb/s\n",
            "    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 160x210, 46 kb/s, 30 fps, 30 tbr, 15360 tbn, 60 tbc (default)\n",
            "    Metadata:\n",
            "      handler_name    : VideoHandler\n",
            "Stream mapping:\n",
            "  Stream #0:0 -> #0:0 (h264 (native) -> h264 (libx264))\n",
            "Press [q] to stop, [?] for help\n",
            "\u001b[1;36m[libx264 @ 0x55ed78eeb900] \u001b[0musing cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2\n",
            "\u001b[1;36m[libx264 @ 0x55ed78eeb900] \u001b[0mprofile High, level 1.2\n",
            "\u001b[1;36m[libx264 @ 0x55ed78eeb900] \u001b[0m264 - core 152 r2854 e9a5903 - H.264/MPEG-4 AVC codec - Copyleft 2003-2017 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=3 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00\n",
            "Output #0, mp4, to 'hub/dqn_SINoFrameSkip-v4/replay.mp4':\n",
            "  Metadata:\n",
            "    major_brand     : isom\n",
            "    minor_version   : 512\n",
            "    compatible_brands: isomiso2avc1mp41\n",
            "    encoder         : Lavf57.83.100\n",
            "    Stream #0:0(und): Video: h264 (libx264) (avc1 / 0x31637661), yuv420p, 160x210, q=-1--1, 30 fps, 15360 tbn, 30 tbc (default)\n",
            "    Metadata:\n",
            "      handler_name    : VideoHandler\n",
            "      encoder         : Lavc57.107.100 libx264\n",
            "    Side data:\n",
            "      cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: -1\n",
            "frame= 1001 fps=0.0 q=-1.0 Lsize=     199kB time=00:00:33.26 bitrate=  49.1kbits/s speed=  42x    \n",
            "video:187kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 6.453969%\n",
            "\u001b[1;36m[libx264 @ 0x55ed78eeb900] \u001b[0mframe I:5     Avg QP:16.07  size:  3445\n",
            "\u001b[1;36m[libx264 @ 0x55ed78eeb900] \u001b[0mframe P:328   Avg QP:23.09  size:   420\n",
            "\u001b[1;36m[libx264 @ 0x55ed78eeb900] \u001b[0mframe B:668   Avg QP:29.49  size:    54\n",
            "\u001b[1;36m[libx264 @ 0x55ed78eeb900] \u001b[0mconsecutive B-frames:  6.0% 11.4% 11.1% 71.5%\n",
            "\u001b[1;36m[libx264 @ 0x55ed78eeb900] \u001b[0mmb I  I16..4: 27.9% 28.7% 43.4%\n",
            "\u001b[1;36m[libx264 @ 0x55ed78eeb900] \u001b[0mmb P  I16..4:  0.6%  1.3%  1.3%  P16..4:  8.1%  3.0%  2.0%  0.0%  0.0%    skip:83.7%\n",
            "\u001b[1;36m[libx264 @ 0x55ed78eeb900] \u001b[0mmb B  I16..4:  0.2%  0.1%  0.1%  B16..8:  9.6%  0.6%  0.0%  direct: 0.1%  skip:89.2%  L0:48.3% L1:51.6% BI: 0.1%\n",
            "\u001b[1;36m[libx264 @ 0x55ed78eeb900] \u001b[0m8x8 transform intra:34.3% inter:5.0%\n",
            "\u001b[1;36m[libx264 @ 0x55ed78eeb900] \u001b[0mcoded y,uvDC,uvAC intra: 24.5% 37.9% 35.7% inter: 1.2% 1.6% 1.3%\n",
            "\u001b[1;36m[libx264 @ 0x55ed78eeb900] \u001b[0mi16 v,h,dc,p: 43% 52%  5%  0%\n",
            "\u001b[1;36m[libx264 @ 0x55ed78eeb900] \u001b[0mi8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 24%  8% 67%  1%  0%  0%  0%  0%  0%\n",
            "\u001b[1;36m[libx264 @ 0x55ed78eeb900] \u001b[0mi4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 38% 12% 38%  2%  2%  3%  2%  2%  2%\n",
            "\u001b[1;36m[libx264 @ 0x55ed78eeb900] \u001b[0mi8c dc,h,v,p: 56% 30% 13%  2%\n",
            "\u001b[1;36m[libx264 @ 0x55ed78eeb900] \u001b[0mWeighted P-Frames: Y:0.0% UV:0.0%\n",
            "\u001b[1;36m[libx264 @ 0x55ed78eeb900] \u001b[0mref P L0: 69.5%  5.9% 11.9% 12.8%\n",
            "\u001b[1;36m[libx264 @ 0x55ed78eeb900] \u001b[0mref B L0: 79.6% 17.8%  2.6%\n",
            "\u001b[1;36m[libx264 @ 0x55ed78eeb900] \u001b[0mref B L1: 96.2%  3.8%\n",
            "\u001b[1;36m[libx264 @ 0x55ed78eeb900] \u001b[0mkb/s:45.78\n",
            "\u001b[38;5;4mℹ Pushing repo dqn_SINoFrameSkip-v4 to the Hugging Face Hub\u001b[0m\n",
            "Upload file dqn-SpaceInvadersNoFrameskip-v4.zip:   0% 3.30k/26.0M [00:00<?, ?B/s]\n",
            "Upload file dqn-SpaceInvadersNoFrameskip-v4/pytorch_variables.pth: 100% 431/431 [00:00<?, ?B/s]\u001b[A\n",
            "\n",
            "Upload file dqn-SpaceInvadersNoFrameskip-v4/policy.pth:   0% 3.30k/12.9M [00:00<?, ?B/s]\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "Upload file dqn-SpaceInvadersNoFrameskip-v4/policy.optimizer.pth:   0% 3.30k/12.9M [00:00<?, ?B/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "\n",
            "Upload file train_eval_metrics.zip:   9% 3.30k/35.6k [00:00<?, ?B/s]\u001b[A\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "Upload file replay.mp4:   2% 3.30k/199k [00:00<?, ?B/s]\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "Upload file dqn-SpaceInvadersNoFrameskip-v4.zip:   2% 619k/26.0M [00:01<00:42, 629kB/s]\n",
            "\n",
            "Upload file dqn-SpaceInvadersNoFrameskip-v4/policy.pth:   1% 147k/12.9M [00:01<01:30, 147kB/s]\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "Upload file dqn-SpaceInvadersNoFrameskip-v4/policy.optimizer.pth:   9% 1.11M/12.9M [00:02<00:20, 600kB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "Upload file dqn-SpaceInvadersNoFrameskip-v4/policy.pth:   6% 735k/12.9M [00:02<00:30, 414kB/s]\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "Upload file dqn-SpaceInvadersNoFrameskip-v4/policy.optimizer.pth:  14% 1.77M/12.9M [00:03<00:18, 640kB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "Upload file dqn-SpaceInvadersNoFrameskip-v4.zip:   3% 707k/26.0M [00:03<02:14, 196kB/s]\n",
            "\n",
            "Upload file dqn-SpaceInvadersNoFrameskip-v4/policy.pth:  16% 2.01M/12.9M [00:04<00:19, 591kB/s]\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "Upload file dqn-SpaceInvadersNoFrameskip-v4.zip:   5% 1.20M/26.0M [00:04<01:22, 314kB/s]\n",
            "\n",
            "\n",
            "Upload file dqn-SpaceInvadersNoFrameskip-v4.zip:   7% 1.84M/26.0M [00:05<00:58, 430kB/s]\n",
            "\n",
            "\n",
            "Upload file dqn-SpaceInvadersNoFrameskip-v4.zip:  10% 2.49M/26.0M [00:06<00:48, 511kB/s]\n",
            "\n",
            "Upload file dqn-SpaceInvadersNoFrameskip-v4/policy.pth:  17% 2.23M/12.9M [00:06<00:33, 338kB/s]\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "Upload file dqn-SpaceInvadersNoFrameskip-v4/policy.optimizer.pth:  32% 4.09M/12.9M [00:07<00:14, 643kB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "Upload file dqn-SpaceInvadersNoFrameskip-v4/policy.pth:  19% 2.40M/12.9M [00:07<00:37, 294kB/s]\u001b[A\u001b[A\n",
            "\n",
            "Upload file dqn-SpaceInvadersNoFrameskip-v4/policy.pth:  24% 3.04M/12.9M [00:08<00:25, 399kB/s]\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "Upload file dqn-SpaceInvadersNoFrameskip-v4/policy.optimizer.pth:  37% 4.73M/12.9M [00:08<00:13, 651kB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "Upload file dqn-SpaceInvadersNoFrameskip-v4/policy.optimizer.pth:  42% 5.38M/12.9M [00:09<00:11, 661kB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "Upload file dqn-SpaceInvadersNoFrameskip-v4.zip:  11% 2.98M/26.0M [00:09<01:16, 315kB/s]\n",
            "\n",
            "\n",
            "Upload file dqn-SpaceInvadersNoFrameskip-v4.zip:  13% 3.50M/26.0M [00:10<01:05, 362kB/s]\n",
            "\n",
            "Upload file dqn-SpaceInvadersNoFrameskip-v4/policy.pth:  30% 3.93M/12.9M [00:10<00:22, 411kB/s]\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "Upload file dqn-SpaceInvadersNoFrameskip-v4/policy.optimizer.pth:  53% 6.76M/12.9M [00:11<00:09, 688kB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "Upload file dqn-SpaceInvadersNoFrameskip-v4/policy.pth:  35% 4.55M/12.9M [00:11<00:18, 484kB/s]\u001b[A\u001b[A\n",
            "\n",
            "Upload file dqn-SpaceInvadersNoFrameskip-v4/policy.pth:  40% 5.20M/12.9M [00:12<00:14, 541kB/s]\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "Upload file dqn-SpaceInvadersNoFrameskip-v4/policy.optimizer.pth:  58% 7.42M/12.9M [00:12<00:08, 687kB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "Upload file dqn-SpaceInvadersNoFrameskip-v4/policy.pth:  46% 5.88M/12.9M [00:13<00:12, 592kB/s]\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "Upload file dqn-SpaceInvadersNoFrameskip-v4.zip:  15% 3.79M/26.0M [00:13<01:37, 238kB/s]\n",
            "\n",
            "Upload file dqn-SpaceInvadersNoFrameskip-v4.zip:  17% 4.41M/26.0M [00:14<01:12, 313kB/s]\n",
            "\n",
            "Upload file dqn-SpaceInvadersNoFrameskip-v4.zip:  19% 5.06M/26.0M [00:15<00:56, 391kB/s]\n",
            "\n",
            "\n",
            "Upload file dqn-SpaceInvadersNoFrameskip-v4.zip:  22% 5.73M/26.0M [00:16<00:45, 462kB/s]\n",
            "\n",
            "\n",
            "Upload file dqn-SpaceInvadersNoFrameskip-v4/policy.optimizer.pth:  64% 8.21M/12.9M [00:16<00:16, 305kB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "Upload file dqn-SpaceInvadersNoFrameskip-v4/policy.pth:  61% 7.85M/12.9M [00:16<00:08, 652kB/s]\u001b[A\u001b[A\n",
            "\n",
            "Upload file dqn-SpaceInvadersNoFrameskip-v4/policy.pth:  64% 8.20M/12.9M [00:17<00:08, 567kB/s]\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "Upload file dqn-SpaceInvadersNoFrameskip-v4/policy.optimizer.pth:  69% 8.83M/12.9M [00:17<00:10, 393kB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "Upload file dqn-SpaceInvadersNoFrameskip-v4/policy.optimizer.pth:  74% 9.53M/12.9M [00:18<00:07, 488kB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "Upload file dqn-SpaceInvadersNoFrameskip-v4.zip:  23% 6.09M/26.0M [00:18<00:58, 355kB/s]\n",
            "\n",
            "Upload file dqn-SpaceInvadersNoFrameskip-v4.zip:  24% 6.26M/26.0M [00:19<01:05, 314kB/s]\n",
            "\n",
            "\n",
            "Upload file dqn-SpaceInvadersNoFrameskip-v4.zip:  27% 6.93M/26.0M [00:20<00:49, 407kB/s]\n",
            "\n",
            "\n",
            "Upload file dqn-SpaceInvadersNoFrameskip-v4.zip:  29% 7.56M/26.0M [00:21<00:40, 473kB/s]\n",
            "\n",
            "\n",
            "Upload file dqn-SpaceInvadersNoFrameskip-v4/policy.optimizer.pth:  86% 11.1M/12.9M [00:21<00:03, 534kB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "Upload file dqn-SpaceInvadersNoFrameskip-v4.zip:  34% 8.89M/26.0M [00:23<00:30, 578kB/s]\n",
            "\n",
            "\n",
            "Upload file dqn-SpaceInvadersNoFrameskip-v4/policy.optimizer.pth:  96% 12.4M/12.9M [00:23<00:00, 616kB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "Upload file dqn-SpaceInvadersNoFrameskip-v4/policy.pth:  76% 9.84M/12.9M [00:23<00:11, 281kB/s]\u001b[A\u001b[A\n",
            "\n",
            "Upload file dqn-SpaceInvadersNoFrameskip-v4.zip:  37% 9.63M/26.0M [00:24<00:27, 634kB/s]\n",
            "\n",
            "Upload file dqn-SpaceInvadersNoFrameskip-v4.zip:  42% 11.0M/26.0M [00:26<00:23, 669kB/s]\n",
            "\n",
            "Upload file dqn-SpaceInvadersNoFrameskip-v4/policy.pth:  89% 11.5M/12.9M [00:26<00:03, 447kB/s]\u001b[A\u001b[A\n",
            "\n",
            "Upload file dqn-SpaceInvadersNoFrameskip-v4.zip:  45% 11.6M/26.0M [00:27<00:22, 670kB/s]\n",
            "\n",
            "Upload file dqn-SpaceInvadersNoFrameskip-v4.zip:  97% 25.2M/26.0M [00:38<00:00, 1.34MB/s]remote: Scanning LFS files for validity, may be slow...        \n",
            "remote: LFS file scan complete.        \n",
            "To https://huggingface.co/Ryukijano/dqn_SINoFrameSkip-v4\n",
            "   65e175f..84dcc7a  main -> main\n",
            "\n",
            "Upload file dqn-SpaceInvadersNoFrameskip-v4.zip: 100% 26.0M/26.0M [00:39<00:00, 697kB/s] \n",
            "\n",
            "Upload file dqn-SpaceInvadersNoFrameskip-v4/pytorch_variables.pth: 100% 431/431 [00:39<?, ?B/s]\u001b[A\n",
            "Upload file dqn-SpaceInvadersNoFrameskip-v4/pytorch_variables.pth: 100% 431/431 [00:39<?, ?B/s]\n",
            "\n",
            "\n",
            "Upload file dqn-SpaceInvadersNoFrameskip-v4/policy.pth: 100% 12.9M/12.9M [00:39<00:00, 118kB/s]\u001b[A\u001b[A\n",
            "\n",
            "Upload file dqn-SpaceInvadersNoFrameskip-v4/policy.pth: 100% 12.9M/12.9M [00:39<00:00, 345kB/s]\n",
            "\n",
            "\n",
            "\n",
            "Upload file dqn-SpaceInvadersNoFrameskip-v4/policy.optimizer.pth: 100% 12.9M/12.9M [00:39<00:00, 106kB/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "Upload file dqn-SpaceInvadersNoFrameskip-v4/policy.optimizer.pth: 100% 12.9M/12.9M [00:39<00:00, 346kB/s]\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "Upload file train_eval_metrics.zip: 100% 35.6k/35.6k [00:39<00:00, 847B/s]\u001b[A\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "\n",
            "Upload file train_eval_metrics.zip: 100% 35.6k/35.6k [00:39<00:00, 847B/s]\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "Upload file replay.mp4: 100% 199k/199k [00:39<00:00, 5.13kB/s]\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "Upload file replay.mp4: 100% 199k/199k [00:39<00:00, 5.13kB/s]\n",
            "\u001b[38;5;4mℹ Your model is pushed to the hub. You can view your model here:\n",
            "https://huggingface.co/Ryukijano/dqn_SINoFrameSkip-v4\u001b[0m\n"
          ]
        }
      ],
      "source": [
        "!python -m rl_zoo3.push_to_hub  --algo dqn  --env SpaceInvadersNoFrameskip-v4  --repo-name dqn_SINoFrameSkip-v4 -orga Ryukijano -f logs/"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "otgpa0rhS9wR"
      },
      "source": [
        "#### Solution"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_HQNlAXuEhci"
      },
      "outputs": [],
      "source": [
        "!python -m rl_zoo3.push_to_hub  --algo dqn  --env SpaceInvadersNoFrameskip-v4  --repo-name dqn-SpaceInvadersNoFrameskip-v4  -orga ThomasSimonini  -f logs/"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0D4F5zsTTJ-L"
      },
      "source": [
        "###."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ff89kd2HL1_s"
      },
      "source": [
        "Congrats 🥳 you've just trained and uploaded your first Deep Q-Learning agent using RL-Baselines-3 Zoo. The script above should have displayed a link to a model repository such as https://huggingface.co/ThomasSimonini/dqn-SpaceInvadersNoFrameskip-v4. When you go to this link, you can:\n",
        "\n",
        "- See a **video preview of your agent** at the right. \n",
        "- Click \"Files and versions\" to see all the files in the repository.\n",
        "- Click \"Use in stable-baselines3\" to get a code snippet that shows how to load the model.\n",
        "- A model card (`README.md` file) which gives a description of the model and the hyperparameters you used.\n",
        "\n",
        "Under the hood, the Hub uses git-based repositories (don't worry if you don't know what git is), which means you can update the model with new versions as you experiment and improve your agent.\n",
        "\n",
        "**Compare the results of your agents with your classmates** using the [leaderboard](https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard) 🏆"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fyRKcCYY-dIo"
      },
      "source": [
        "## Load a powerful trained model 🔥\n",
        "- The Stable-Baselines3 team uploaded **more than 150 trained Deep Reinforcement Learning agents on the Hub**.\n",
        "\n",
        "You can find them here: 👉 https://huggingface.co/sb3\n",
        "\n",
        "Some examples:\n",
        "- Asteroids: https://huggingface.co/sb3/dqn-AsteroidsNoFrameskip-v4\n",
        "- Beam Rider: https://huggingface.co/sb3/dqn-BeamRiderNoFrameskip-v4\n",
        "- Breakout: https://huggingface.co/sb3/dqn-BreakoutNoFrameskip-v4\n",
        "- Road Runner: https://huggingface.co/sb3/dqn-RoadRunnerNoFrameskip-v4\n",
        "\n",
        "Let's load an agent playing Beam Rider: https://huggingface.co/sb3/dqn-BeamRiderNoFrameskip-v4"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "B-9QVFIROI5Y",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 231
        },
        "outputId": "e5fdc5fd-6c65-476d-b2d5-2495a7994081"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "<video controls autoplay><source src=\"https://huggingface.co/sb3/dqn-BeamRiderNoFrameskip-v4/resolve/main/replay.mp4\" type=\"video/mp4\"></video>\n"
            ]
          },
          "metadata": {}
        }
      ],
      "source": [
        "%%html\n",
        "<video controls autoplay><source src=\"https://huggingface.co/sb3/dqn-BeamRiderNoFrameskip-v4/resolve/main/replay.mp4\" type=\"video/mp4\"></video>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7ZQNY_r6NJtC"
      },
      "source": [
        "1. We download the model using `rl_zoo3.load_from_hub`, and place it in a new folder that we can call `rl_trained`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OdBNZHy0NGTR",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ab5611de-3227-45ac-9f1b-32a615378e7b"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Downloading from https://huggingface.co/sb3/dqn-BeamRiderNoFrameskip-v4\n",
            "Downloading: 100% 27.2M/27.2M [00:00<00:00, 82.7MB/s]\n",
            "Downloading: 100% 548/548 [00:00<00:00, 776kB/s]\n",
            "No normalization file\n",
            "Downloading: 100% 887/887 [00:00<00:00, 1.31MB/s]\n",
            "Downloading: 100% 3.00/3.00 [00:00<00:00, 3.32kB/s]\n",
            "Downloading: 100% 244k/244k [00:00<00:00, 5.53MB/s]\n",
            "Saving to rl_trained/dqn/BeamRiderNoFrameskip-v4_1\n"
          ]
        }
      ],
      "source": [
        "# Download model and save it into the logs/ folder\n",
        "!python -m rl_zoo3.load_from_hub --algo dqn --env BeamRiderNoFrameskip-v4 -orga sb3 -f rl_trained/"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LFt6hmWsNdBo"
      },
      "source": [
        "2. Let's evaluate if for 5000 timesteps"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "aOxs0rNuN0uS",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "90464378-4a14-4cda-d43f-eb7a46649b6f"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Loading latest experiment, id=1\n",
            "Loading rl_trained/dqn/BeamRiderNoFrameskip-v4_1/BeamRiderNoFrameskip-v4.zip\n",
            "A.L.E: Arcade Learning Environment (version 0.7.4+069f8bd)\n",
            "[Powered by Stella]\n",
            "Stacking 4 frames\n",
            "/usr/local/lib/python3.8/dist-packages/stable_baselines3/common/save_util.py:166: UserWarning: Could not deserialize object exploration_schedule. Consider using `custom_objects` argument to replace this object.\n",
            "Exception: an integer is required (got type bytes)\n",
            "  warnings.warn(\n",
            "Wrapping the env in a VecTransposeImage.\n",
            "/usr/local/lib/python3.8/dist-packages/gym/envs/atari/environment.py:267: UserWarning: \u001b[33mWARN: We strongly suggest supplying `render_mode` when constructing your environment, e.g., gym.make(ID, render_mode='human'). Using `render_mode` provides access to proper scaling, audio support, and proper framerates.\u001b[0m\n",
            "  logger.warn(\n",
            "Atari Episode Score: 1432.00\n",
            "Atari Episode Length 7401\n"
          ]
        }
      ],
      "source": [
        "!python enjoy.py --algo dqn --env BeamRiderNoFrameskip-v4 -n 5000  -f rl_trained/"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kxMDuDfPON57"
      },
      "source": [
        "Why not trying to train your own **Deep Q-Learning Agent playing BeamRiderNoFrameskip-v4? 🏆.**\n",
        "\n",
        "If you want to try, check https://huggingface.co/sb3/dqn-BeamRiderNoFrameskip-v4#hyperparameters **in the model card, you have the hyperparameters of the trained agent.**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xL_ZtUgpOuY6"
      },
      "source": [
        "But finding hyperparameters can be a daunting task. Fortunately, we'll see in the next Unit, how we can **use Optuna for optimizing the Hyperparameters 🔥.**\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-pqaco8W-huW"
      },
      "source": [
        "## Some additional challenges 🏆\n",
        "The best way to learn **is to try things by your own**!\n",
        "\n",
        "In the [Leaderboard](https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard) you will find your agents. Can you get to the top?\n",
        "\n",
        "Here's a list of environments you can try to train your agent with:\n",
        "- BeamRiderNoFrameskip-v4\n",
        "- BreakoutNoFrameskip-v4 \n",
        "- EnduroNoFrameskip-v4\n",
        "- PongNoFrameskip-v4\n",
        "\n",
        "Also, **if you want to learn to implement Deep Q-Learning by yourself**, you definitely should look at CleanRL implementation: https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/dqn_atari.py\n",
        "\n",
        "<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/unit4/atari-envs.gif\" alt=\"Environments\"/>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "paS-XKo4-kmu"
      },
      "source": [
        "________________________________________________________________________\n",
        "Congrats on finishing this chapter!\n",
        "\n",
        "If you’re still feel confused with all these elements...it's totally normal! **This was the same for me and for all people who studied RL.**\n",
        "\n",
        "Take time to really **grasp the material before continuing and try the additional challenges**. It’s important to master these elements and having a solid foundations.\n",
        "\n",
        "In the next unit, **we’re going to learn about [Optuna](https://optuna.org/)**. One of the most critical task in Deep Reinforcement Learning is to find a good set of training hyperparameters. And Optuna is a library that helps you to automate the search.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5WRx7tO7-mvC"
      },
      "source": [
        "\n",
        "\n",
        "### This is a course built with you 👷🏿‍♀️\n",
        "\n",
        "Finally, we want to improve and update the course iteratively with your feedback. If you have some, please fill this form 👉 https://forms.gle/3HgA7bEHwAmmLfwh9\n",
        "\n",
        "We're constantly trying to improve our tutorials, so **if you find some issues in this notebook**, please [open an issue on the Github Repo](https://github.com/huggingface/deep-rl-class/issues)."
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "See you on Bonus unit 2! 🔥 "
      ],
      "metadata": {
        "id": "Kc3udPT-RcXc"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fS3Xerx0fIMV"
      },
      "source": [
        "### Keep Learning, Stay Awesome 🤗"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.6"
    },
    "varInspector": {
      "cols": {
        "lenName": 16,
        "lenType": 16,
        "lenVar": 40
      },
      "kernels_config": {
        "python": {
          "delete_cmd_postfix": "",
          "delete_cmd_prefix": "del ",
          "library": "var_list.py",
          "varRefreshCmd": "print(var_dic_list())"
        },
        "r": {
          "delete_cmd_postfix": ") ",
          "delete_cmd_prefix": "rm(",
          "library": "var_list.r",
          "varRefreshCmd": "cat(var_dic_list()) "
        }
      },
      "types_to_exclude": [
        "module",
        "function",
        "builtin_function_or_method",
        "instance",
        "_Feature"
      ],
      "window_display": false
    },
    "accelerator": "GPU",
    "gpuClass": "standard",
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "0046d9625e0d4c95acf01a02cda7f4e3": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "VBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "VBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "VBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_f5426df55bb84a93a5324f879f7622ee",
              "IPY_MODEL_42b1b606dd3e4f73b4d9e060ce6a26ff",
              "IPY_MODEL_ffa2b0353f5b438199722e163e594669",
              "IPY_MODEL_3e78aeccf0ef4bb9b8a78cf83e422c68",
              "IPY_MODEL_a663c17108394a5ca036fd26364d35ce"
            ],
            "layout": "IPY_MODEL_bc68b383d3f94e6d973030124e33c35c"
          }
        },
        "f5426df55bb84a93a5324f879f7622ee": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_8bc89127573946ed9ffbf2cad0cb108d",
            "placeholder": "​",
            "style": "IPY_MODEL_8c09e2aeaeff44b5a2ce7ce61e8b70c0",
            "value": "<center> <img\nsrc=https://huggingface.co/front/assets/huggingface_logo-noborder.svg\nalt='Hugging Face'> <br> Copy a token from <a\nhref=\"https://huggingface.co/settings/tokens\" target=\"_blank\">your Hugging Face\ntokens page</a> and paste it below. <br> Immediately click login after copying\nyour token or it might be stored in plain text in this notebook file. </center>"
          }
        },
        "42b1b606dd3e4f73b4d9e060ce6a26ff": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "PasswordModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "PasswordModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "PasswordView",
            "continuous_update": true,
            "description": "Token:",
            "description_tooltip": null,
            "disabled": false,
            "layout": "IPY_MODEL_ff70d62b759b4738b3d125ed6eaa4c2f",
            "placeholder": "​",
            "style": "IPY_MODEL_e2af816ebd154af69218cb10c464eab3",
            "value": ""
          }
        },
        "ffa2b0353f5b438199722e163e594669": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "CheckboxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "CheckboxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "CheckboxView",
            "description": "Add token as git credential?",
            "description_tooltip": null,
            "disabled": false,
            "indent": true,
            "layout": "IPY_MODEL_cb990bb2d70f48498958c7e3fee8df5b",
            "style": "IPY_MODEL_bf65637de0844f77bb341f63ae3b82fa",
            "value": true
          }
        },
        "3e78aeccf0ef4bb9b8a78cf83e422c68": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ButtonModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ButtonModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ButtonView",
            "button_style": "",
            "description": "Login",
            "disabled": false,
            "icon": "",
            "layout": "IPY_MODEL_46fc7890d9ce4617a9c456ad6d12cf56",
            "style": "IPY_MODEL_74f8ed71af5e4e23bbfa057970626408",
            "tooltip": ""
          }
        },
        "a663c17108394a5ca036fd26364d35ce": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_228019b8acb04dc4bdfeb35d8569cbef",
            "placeholder": "​",
            "style": "IPY_MODEL_7678b80ae0574c4a8c5892db6d4d941c",
            "value": "\n<b>Pro Tip:</b> If you don't already have one, you can create a dedicated\n'notebooks' token with 'write' access, that you can then easily reuse for all\nnotebooks. </center>"
          }
        },
        "bc68b383d3f94e6d973030124e33c35c": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": "center",
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": "flex",
            "flex": null,
            "flex_flow": "column",
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": "50%"
          }
        },
        "8bc89127573946ed9ffbf2cad0cb108d": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "8c09e2aeaeff44b5a2ce7ce61e8b70c0": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "ff70d62b759b4738b3d125ed6eaa4c2f": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e2af816ebd154af69218cb10c464eab3": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "cb990bb2d70f48498958c7e3fee8df5b": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "bf65637de0844f77bb341f63ae3b82fa": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "46fc7890d9ce4617a9c456ad6d12cf56": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "74f8ed71af5e4e23bbfa057970626408": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ButtonStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ButtonStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "button_color": null,
            "font_weight": ""
          }
        },
        "228019b8acb04dc4bdfeb35d8569cbef": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "7678b80ae0574c4a8c5892db6d4d941c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}