notebooks/Running-Simulations.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Running Simulations\n",
    "\n",
    "All simulators in POMDPs.jl should be consistent with the [Simulation Standard](https://juliapomdp.github.io/POMDPs.jl/latest/simulation), and the Simulation Standard document should be consulted to answer questions about how a simulation is defined.\n",
    "\n",
    "There are several ways to perform simulations in the POMDPs.jl ecosystem. The [POMDPSimulators package](https://github.com/JuliaPOMDP/POMDPSimulators.jl) provides implementations of several simulators and [a page to help decide which to use](https://juliapomdp.github.io/POMDPSimulators.jl/latest/which). This tutorial will show a few examples demonstrating how to use the simulators, but the POMDPSimulators documentation serves as a more in-depth reference.\n",
    "\n",
    "## Ingredients\n",
    "\n",
    "The [Simulation Standard documentation](https://juliapomdp.github.io/POMDPs.jl/latest/simulation) gives information about ingredients to a simulation. The minimum requirement is for a `POMDP` or `MDP` model and a `Policy`.\n",
    "\n",
    "Models from the [POMDPModels package](https://github.com/JuliaPOMDP/POMDPModels.jl) and policies from the [POMDPPolicies package](https://github.com/JuliaPOMDP/POMDPPolicies.jl) will be used in these examples. For more information on defining models, see the tutorials about defining POMDPs and MDPs; for information on hand-specifying policies see the Heuristic Policies tutorial [TODO: add link]; and for examples of solving for policies, see the tutorials about solving POMDPs online and offline [TODO: add link]. Also consult the docstrings for the various functions used here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "using POMDPs\n",
    "using POMDPSimulators\n",
    "using POMDPModels\n",
    "using POMDPPolicies\n",
    "using POMDPModelTools\n",
    "using BeliefUpdaters\n",
    "using Printf\n",
    "using Random"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below we define some problems, policies, and belief updaters that will be used in the simulations. Use [Julia's help system (type `?`)](https://docs.julialang.org/en/v1/stdlib/REPL/#Help-mode-1) to find out more by reading the docstrings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "tiger = TigerPOMDP()\n",
    "grid = SimpleGridWorld();\n",
    "listen = FunctionPolicy(b->TIGER_LISTEN)\n",
    "tiger_filter = DiscreteUpdater(tiger);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stepthrough\n",
    "\n",
    "The recommended first way to try out POMDPs.jl simulations is with the [`stepthrough` function](https://juliapomdp.github.io/POMDPSimulators.jl/latest/stepthrough/#POMDPSimulators.stepthrough). This function provides a window into the simulation with a for-loop syntax.\n",
    "\n",
    "Within the body of the for loop, we have access to the belief, `b`, the action, `a`, the observation, `o`, and the reward, `r`, in each step. As more information is gathered through listening, confidence about whether the tiger is in the left or right door increases dramatically. The sum of the rewards is also calculated here, but note that this is *not the discounted reward*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "belief that tiger is behind left door: 50.0%\n",
      "action: 0, observation: true\n",
      "belief that tiger is behind left door: 85.0%\n",
      "action: 0, observation: true\n",
      "belief that tiger is behind left door: 97.0%\n",
      "action: 0, observation: false\n",
      "belief that tiger is behind left door: 85.0%\n",
      "action: 0, observation: true\n",
      "belief that tiger is behind left door: 97.0%\n",
      "action: 0, observation: true\n",
      "rsum = -5.0\n"
     ]
    }
   ],
   "source": [
    "rsum = 0.0\n",
    "for (b,a,o,r) in stepthrough(tiger, listen, tiger_filter, \"b,a,o,r\", max_steps=5)\n",
    "    @printf(\"belief that tiger is behind left door: %4.1f%%\\n\",\n",
    "            100*pdf(b, TIGER_LEFT))\n",
    "    @printf(\"action: %s, observation: %s\\n\", a, o)\n",
    "    global rsum += r\n",
    "end\n",
    "@show rsum;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Rollout Simulations\n",
    "\n",
    "While `stepthrough` is a flexible and convenient tool for many user-facing demonstration, it is often less error-prone to use the standard [`simulate`](https://juliapomdp.github.io/POMDPs.jl/latest/api#POMDPs.simulate) function with a [`Simulator`](https://juliapomdp.github.io/POMDPs.jl/latest/concepts#Simulators-1) object. The simplest `Simulator` is the [`RolloutSimulator`](https://juliapomdp.github.io/POMDPSimulators.jl/latest/rollout#POMDPSimulators.RolloutSimulator). It simply runs a simulation as fast as possible and returns the discounted reward. Note that discounted reward for the simulation with the larger number of steps approaches the theoretical reward for a policy of always listening, $\\Sigma_{t=0}^\\infty -5 \\gamma^t$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "short_dr = -4.524381249999999\n",
      "long_dr = -19.999999999999975\n"
     ]
    }
   ],
   "source": [
    "short = RolloutSimulator(max_steps=5)\n",
    "short_dr = simulate(short, tiger, listen)\n",
    "long = RolloutSimulator(max_steps=1_000_000)\n",
    "long_dr = simulate(long, tiger, listen)\n",
    "@show short_dr\n",
    "@show long_dr;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Recording Histories\n",
    "\n",
    "Sometimes it is important to record the entire history of a simulation for further examination. This can be accomplished with a [`HistoryRecorder`](https://juliapomdp.github.io/POMDPSimulators.jl/latest/history_recorder#POMDPSimulators.HistoryRecorder). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "hr = HistoryRecorder(max_steps=5)\n",
    "history = simulate(hr, tiger, listen, tiger_filter);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Histories\n",
    "\n",
    "The history object produced by a `HistoryRecorder` is a [`SimHistory`, documented in the POMDPSimulators package](https://juliapomdp.github.io/POMDPSimulators.jl/latest/histories#Histories-1). The information in this object can be accessed in several ways. For example, there is a function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-4.524381249999999"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "discounted_reward(history)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Accessor functions like `state_hist` and `action_hist` can be used to access parts of the history:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "state_hist(history) = Bool[0, 0, 0, 0, 0, 0]\n",
      "collect(observation_hist(history)) = Bool[0, 0, 0, 0, 0]\n"
     ]
    }
   ],
   "source": [
    "@show state_hist(history)\n",
    "@show collect(observation_hist(history));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, keeping track of which states, actions, and observations belong together can be tricky (for example, since there is a starting state, and ending state, but no action is taken from the ending state, the list of actions has a different length than the list of states; Note the difference in the length of the state and observation histories above). For this reason, it is often better to think of histories in terms of *steps* that include both starting and ending states, i.e. $(s, a, r, s')$/`(s, a, r, sp)` tuples for MDPs.\n",
    "\n",
    "### `eachstep`\n",
    "\n",
    "The most powerful function for accessing the information in a `SimHistory` is the [`eachstep` function](https://juliapomdp.github.io/POMDPSimulators.jl/latest/histories#POMDPSimulators.eachstep) which returns an iterator through [`NamedTuple`](https://docs.julialang.org/en/v1/manual/types/index#Named-Tuple-Types-1)s representing each step in the history. The `eachstep` function is similar to the `stepthrough` function above except that it iterates through the immutable steps of a previously simulated history instead of conducting the simulation as the for loop is being carried out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "belief that tiger is behind left door: 50.0%\n",
      "action: 0, observation: false\n",
      "belief that tiger is behind left door: 15.0%\n",
      "action: 0, observation: false\n",
      "belief that tiger is behind left door:  3.0%\n",
      "action: 0, observation: false\n",
      "belief that tiger is behind left door:  0.5%\n",
      "action: 0, observation: false\n",
      "belief that tiger is behind left door:  0.1%\n",
      "action: 0, observation: false\n",
      "rsum = -5.0\n"
     ]
    }
   ],
   "source": [
    "rsum = 0.0\n",
    "for (b,a,o,r) in eachstep(history, \"b,a,o,r\")\n",
    "    @printf(\"belief that tiger is behind left door: %4.1f%%\\n\",\n",
    "            100*pdf(b, TIGER_LEFT))\n",
    "    @printf(\"action: %s, observation: %s\\n\", a, o)\n",
    "    global rsum += r\n",
    "end\n",
    "@show rsum;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since each step returned by `eachstep` is a [`NamedTuple`](https://docs.julialang.org/en/v1/manual/types/index.html#Named-Tuple-Types-1), the elements in it can also be accessed as fields:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "belief that tiger is behind left door: 50.0%\n",
      "action: 0, observation: false\n",
      "belief that tiger is behind left door: 15.0%\n",
      "action: 0, observation: false\n",
      "belief that tiger is behind left door:  3.0%\n",
      "action: 0, observation: false\n",
      "belief that tiger is behind left door:  0.5%\n",
      "action: 0, observation: false\n",
      "belief that tiger is behind left door:  0.1%\n",
      "action: 0, observation: false\n",
      "rsum = -5.0\n"
     ]
    }
   ],
   "source": [
    "rsum = 0.0\n",
    "for step in eachstep(history)\n",
    "    @printf(\"belief that tiger is behind left door: %4.1f%%\\n\",\n",
    "            100*pdf(step.b, TIGER_LEFT))\n",
    "    @printf(\"action: %s, observation: %s\\n\", step.a, step.o)\n",
    "    global rsum += step.r\n",
    "end\n",
    "@show rsum;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parallel Simulations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is often useful to evaluate a policy by running many simulations.\n",
    "The parallel simulator is the most effective tool for this.\n",
    "To use the parallel simulator, one should first create a list of `Sim` objects, each of which contains all of the information needed to run a simulation, and then run the simulations using `run_parallel`, which will return a `DataFrame` with the results.\n",
    "\n",
    "Details can be found in the [POMDPSimulators package documentation](https://juliapomdp.github.io/POMDPSimulators.jl/stable/parallel/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "┌ Warning: run_parallel(...) was started with only 1 worker in the pool, so simulations will be run in serial.\n",
      "│ \n",
      "│ To supress this warning, use run_parallel(..., proc_warn=false).\n",
      "│ \n",
      "│ To use multiple processes, use addprocs() or the -p option (e.g. julia -p 4) and make sure the correct worker pool is assigned to argument `pool` in the call to run_parallel.\n",
      "└ @ POMDPSimulators /home/zach/.julia/packages/POMDPSimulators/Ipuzk/src/parallel.jl:123\n",
      "\u001b[32mSimulating...100%|██████████████████████████████████████| Time: 0:00:01\u001b[39m\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>policy</th><th>reward</th></tr><tr><th></th><th>String⍰</th><th>Float64⍰</th></tr></thead><tbody><p>2 rows × 2 columns</p><tr><th>1</th><td>feed when crying</td><td>-4.5874</td></tr><tr><th>2</th><td>random</td><td>-27.4139</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cc}\n",
       "\t& policy & reward\\\\\n",
       "\t\\hline\n",
       "\t& String⍰ & Float64⍰\\\\\n",
       "\t\\hline\n",
       "\t1 & feed when crying & -4.5874 \\\\\n",
       "\t2 & random & -27.4139 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "2×2 DataFrames.DataFrame\n",
       "│ Row │ policy           │ reward   │\n",
       "│     │ \u001b[90mString⍰\u001b[39m          │ \u001b[90mFloat64⍰\u001b[39m │\n",
       "├─────┼──────────────────┼──────────┤\n",
       "│ 1   │ feed when crying │ -4.5874  │\n",
       "│ 2   │ random           │ -27.4139 │"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pomdp = BabyPOMDP()\n",
    "fwc = FeedWhenCrying()\n",
    "rnd = solve(RandomSolver(MersenneTwister(7)), pomdp)\n",
    "\n",
    "q = [] # vector of the simulations to be run\n",
    "push!(q, Sim(pomdp, fwc, max_steps=32, rng=MersenneTwister(4), metadata=Dict(:policy=>\"feed when crying\")))\n",
    "push!(q, Sim(pomdp, rnd, max_steps=32, rng=MersenneTwister(4), metadata=Dict(:policy=>\"random\")))\n",
    "\n",
    "# this creates two simulations, one with the feed-when-crying policy and one with a random policy\n",
    "\n",
    "data = run_parallel(q)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default, the parallel simulator only returns the reward from each simulation, but more information can be gathered by specifying a function to analyze the `Sim`-history pair and record additional statistics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "finished a simulation - final state was false\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "┌ Warning: run_parallel(...) was started with only 1 worker in the pool, so simulations will be run in serial.\n",
      "│ \n",
      "│ To supress this warning, use run_parallel(..., proc_warn=false).\n",
      "│ \n",
      "│ To use multiple processes, use addprocs() or the -p option (e.g. julia -p 4) and make sure the correct worker pool is assigned to argument `pool` in the call to run_parallel.\n",
      "└ @ POMDPSimulators /home/zach/.julia/packages/POMDPSimulators/Ipuzk/src/parallel.jl:123\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "finished a simulation - final state was false\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>policy</th><th>reward</th><th>steps</th></tr><tr><th></th><th>String⍰</th><th>Float64⍰</th><th>Float64⍰</th></tr></thead><tbody><p>2 rows × 3 columns</p><tr><th>1</th><td>feed when crying</td><td>-18.2874</td><td>32.0</td></tr><tr><th>2</th><td>random</td><td>-17.7054</td><td>32.0</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ccc}\n",
       "\t& policy & reward & steps\\\\\n",
       "\t\\hline\n",
       "\t& String⍰ & Float64⍰ & Float64⍰\\\\\n",
       "\t\\hline\n",
       "\t1 & feed when crying & -18.2874 & 32.0 \\\\\n",
       "\t2 & random & -17.7054 & 32.0 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "2×3 DataFrames.DataFrame\n",
       "│ Row │ policy           │ reward   │ steps    │\n",
       "│     │ \u001b[90mString⍰\u001b[39m          │ \u001b[90mFloat64⍰\u001b[39m │ \u001b[90mFloat64⍰\u001b[39m │\n",
       "├─────┼──────────────────┼──────────┼──────────┤\n",
       "│ 1   │ feed when crying │ -18.2874 │ 32.0     │\n",
       "│ 2   │ random           │ -17.7054 │ 32.0     │"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# to perform additional analysis on each of the simulations one can define a processing function with the `do` syntax:\n",
    "data2 = run_parallel(q, show_progress=false) do sim, hist\n",
    "    println(\"finished a simulation - final state was $(last(state_hist(hist)))\")\n",
    "    return [:steps=>n_steps(hist), :reward=>discounted_reward(hist)]\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `sim()` [This tutorial is a work in progress]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The [`sim` function](https://juliapomdp.github.io/POMDPSimulators.jl/latest/sim) provides a convenient way to interact with a `POMDP` or `MDP` environment from the perspective of an agent acting in that environment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TODO: Add Example"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Julia 1.2.0",
   "language": "julia",
   "name": "julia-1.2"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "1.2.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}