examples/setting_up_environment_full.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  Setting the environment: full power.\n",
    "_Or: making gym environment happy  with your very own backtrader engine._\n",
    "****"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **This example assumes close familarity with  Backtrader conceptions and operation worflow.**\n",
    "> \n",
    ">  **One should at least run through Quickstart tutorial:  https://www.backtrader.com/docu/quickstart/quickstart.html**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Typical workfolw for traditional Backtrader backtesting procedure (recap):\n",
    "- Define backtrader core engine:\n",
    "  \n",
    "``` python\n",
    "    import backtrader as bt\n",
    "    import backtrader.feeds as btfeeds\n",
    "    engine = bt.Cerebro()\n",
    "```\n",
    "\n",
    "- Add some starategy class, wich has been prepared in advance as backtrader base Strategy() subclass and should define  decision-making logic:\n",
    " \n",
    "``` python\n",
    "    engine.addstrategy(MyStrategy)\n",
    "```\n",
    "     \n",
    "- Set broker options, such as: cash, commission, slippage, etc.:\n",
    " \n",
    "``` python\n",
    "    engine.setcash(100000)\n",
    "    engine.setcommission(0.001)\n",
    "```\n",
    "    \n",
    "- Add analyzers, observers, sizers, writers to own needs:\n",
    " \n",
    "``` python\n",
    "    engine.addobserver(bt.observers.Trades)\n",
    "    engine.addobserver(bt.observers.BuySell)\n",
    "    engine.addanalyzer(bt.analyzers.DrawDown, _name='drawdown')\n",
    "    engine.addsizer(bt.sizers.SizerFix, stake=1000)\n",
    "```\n",
    "     \n",
    "- Define and add data feed from one or another source (live feed is possible):\n",
    " \n",
    "``` python\n",
    "    MyData = btfeeds.GenericCSVData(dataname=CSVfilename.csv)\n",
    "    engine.addata(MyData)\n",
    "```\n",
    "     \n",
    "- Now backtrader enigine is ready to run backtesting:\n",
    " \n",
    "``` python\n",
    "   results = engine.run()\n",
    "```\n",
    " \n",
    "- After that you can print, analyze and think on results:\n",
    " \n",
    "``` python\n",
    "    engine.plot()\n",
    "    my_disaster_drowdown = results[0].analyzers.drawdown.get_analysis()\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "### For BTgym,  same principles apply with several differences:\n",
    "\n",
    " - strategy you prepare will be subclass of base BTgymStrategy,\n",
    "   wich contains specific to RL setup methods and parameters;\n",
    "   \n",
    "    \n",
    " - this startegy will not contain buy/sell decision-making logic - this part will go to RL agent;\n",
    "   \n",
    " \n",
    " - you define you dataset by creating BTgymDataset class instance;\n",
    "   \n",
    " \n",
    " - you don't add data to your bt.cerebro(). Just pass dataset to environment, BTgym server will do the rest.\n",
    "   \n",
    " \n",
    " - you dont run backtrader engine manually via run() method, server will do.\n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "****\n",
    "### There are three levels to  BTgym configuration:\n",
    "#### [Light:](https://github.com/Kismuz/btgym/blob/master/examples/setting_up_environment_basic.ipynb) \n",
    " - use kwargs when making envronment. See 'basic' example for details;\n",
    "  \n",
    "   \n",
    "#### [3/4:](#3/4) \n",
    "- subclass BTgymStrategy: override **`get_state(), get_done(), get_reward(), get_info()`**\n",
    "  and [maybe] **`next()`** methods to get own state, reward definition, order execution logic, actions etc;\n",
    "- pass this strategy to environment via **`strategy`** kwarg along with other parameters;\n",
    "- [optionally] make instance of **`BTgymDataset`** class as your custom dataset an pass it via **`dataset`** kwarg.\n",
    " \n",
    "  \n",
    "#### [Full throttle:](#full)\n",
    " - subclass strategy as in '3/4';\n",
    " - define bt.Cerebro(): set broker parameters, add all required observers,\n",
    "   analysers, stakes and other bells and whistles;\n",
    " - attach your '3/4'-strategy;\n",
    " - pass that snowball to environment via **`engine`** kwarg;\n",
    " - [opt.] make and pass dataset as in '3/4'.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "### Environment kwargs reference:\n",
    "**as for v0.0.4**\n",
    "```python\n",
    "\n",
    "# Dataset parameters:\n",
    "filename=None,  # Source CSV data file;\n",
    "    # Episode data params:\n",
    "start_weekdays=[0, 1, 2, ],  # Only weekdays from the list will be used for episode start.\n",
    "start_00=True,  # Episode start time will be set to first record of the day (usually 00:00).\n",
    "episode_duration={'days': 1, 'hours': 23, 'minutes': 55},   # Maximum episode time duration in d:h:m:\n",
    "time_gap={'hours': 5},  # Maximum data time gap allowed within sample in d:h.\n",
    "                        # If set < 1 day, samples containing weekends and holidays gaps will be rejected.\n",
    "\n",
    "\n",
    "# Backtrader engine parameters:\n",
    "start_cash=10.0,  # initial trading capital.\n",
    "broker_commission=0.001,  # trade execution commission, default is 0.1% of operation value.\n",
    "fixed_stake=10,  # single trade stake is fixed type by def.\n",
    "\n",
    "\n",
    "# Strategy related parameters:\n",
    "# Observation state shape is dictionary of Gym spaces,\n",
    "# at least should contain `raw_state` field.\n",
    "# By convention first dimension of every Gym Box space is time embedding one;\n",
    "# one can define any shape; should match env.observation_space.shape.\n",
    "# observation space state min/max values,\n",
    "# For `raw_state' - absolute min/max values from BTgymDataset will be used.\n",
    "state_shape=dict(\n",
    "    raw_state=spaces.Box(\n",
    "        shape=(10, 4),\n",
    "        low=-100,\n",
    "        high=100,\n",
    "    )\n",
    "),\n",
    "drawdown_call=90,  # episode maximum drawdown threshold, default is 90% of initial value.\n",
    "portfolio_actions=('hold', 'buy', 'sell', 'close'),\n",
    "    # agent actions,\n",
    "    # should consist with BTgymStrategy order execution logic;\n",
    "    # defaults are (env.side): 0 - 'do nothing', 1 - 'buy', 2 - 'sell', 3 - 'close position'.\n",
    "skip_frame=1,\n",
    "    # Number of environment steps to skip before returning next response,\n",
    "    # e.g. if set to 10 -- agent will interact with environment every 10th episode step;\n",
    "    # Every other step agent's action is assumed to be 'hold'.\n",
    "    # Note: INFO part of environment response is a list of all skipped frame's info's,\n",
    "    #       i.e. [info[-9], info[-8], ..., info[0].\n",
    "\n",
    "# Rendering controls:\n",
    "render_state_as_image = True\n",
    "render_state_channel=0\n",
    "render_size_human = (6, 3.5)\n",
    "render_size_statet = (7, 3.5)\n",
    "render_size_episode = (12,8)\n",
    "render_dpi=75\n",
    "render_plotstyle = 'seaborn'\n",
    "render_cmap = 'PRGn'\n",
    "render_xlabel = 'Relative timesteps'\n",
    "render_ylabel = 'Value'\n",
    "render_title = 'step: {}, state observation min: {:.4f}, max: {:.4f}'\n",
    "render_boxtext = dict(fontsize=12,\n",
    "                      fontweight='bold',\n",
    "                      color='w',\n",
    "                      bbox={'facecolor': 'k', 'alpha': 0.3, 'pad': 3},\n",
    "                      )\n",
    "\n",
    "# Other:\n",
    "port=5500,  # network port to use.\n",
    "network_address='tcp://127.0.0.1:',  # using localhost.\n",
    "verbose=0,  # verbosity mode: 0 - silent, 1 - info level, 2 - debugging level (lot of traffic!).\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "### Kwargs applying logic:\n",
    "    \n",
    "        if <engine> kwarg is given:\n",
    "            do not use default engine and strategy parameters;\n",
    "            ignore <startegy> kwarg and all startegy and engine-related kwargs;\n",
    "        \n",
    "        else (no <engine>):\n",
    "            use default engine parameters;\n",
    "            if any engine-related kwarg is given:\n",
    "                override corresponding default parameter;\n",
    "            \n",
    "            if <strategy> is given:\n",
    "                do not use default strategy parameters;\n",
    "                if any strategy related kwarg is given:\n",
    "                    override corresponding strategy parameter;\n",
    "                \n",
    "            else (no <strategy>):\n",
    "                use default strategy parameters;\n",
    "                if any strategy related kwarg is given:\n",
    "                    override corresponding strategy parameter;\n",
    "        \n",
    "       if <dataset> kwarg is given:\n",
    "            do not use default dataset parameters;\n",
    "            ignore dataset related kwargs;\n",
    "                    \n",
    "        else (no <dataset>):\n",
    "            use default dataset parameters;\n",
    "                if  any dataset related kwarg is given:\n",
    "                    override corresponding dataset parameter;\n",
    "        \n",
    "        If any <other> kwarg is given:\n",
    "            override corr. default parameter."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "### <a name=\"3/4\"></a> 3/4. 'State and Reward' with BTgymStrategy.\n",
    "- There are parameters BTgymStrategy class holds.\n",
    "- Point it out: it's strategy parameters, not environment ones (though names are the same as above)!\n",
    "```python\n",
    "# NEW at v0.6: Note that btgym uses new OPenAI Gym space defined in: `gym.spaces.Dict` which is in fact\n",
    "# [possibly nested] dictionary of base Gym spaces. You can use `gym.spaces.Dict` if you have \n",
    "# latest Gym version from repo or use equivalent `btgym.spaces.DictSpace` wrapper instead.\n",
    "# Thus, `space_shape` param directly translites into Dict space.\n",
    "#\n",
    "# Observation state shape is dictionary of Gym spaces,\n",
    "# at least should contain `raw_state` field.\n",
    "# By convention first dimension of every Gym Box space is time embedding one;\n",
    "# one can define any shape; should match env.observation_space.shape.\n",
    "# observation space state min/max values,\n",
    "# For `raw_state' - absolute min/max values from BTgymDataset will be used.\n",
    "state_shape=dict(\n",
    "    raw_state=spaces.Box(\n",
    "        shape=(10, 4),\n",
    "        low=-100,\n",
    "        high=100,\n",
    "    )\n",
    "),\n",
    "drawdown_call=90,  # episode maximum drawdown threshold, default is 90% of initial value.\n",
    "portfolio_actions=('hold', 'buy', 'sell', 'close'),\n",
    "    # agent actions,\n",
    "    # should consist with BTgymStrategy order execution logic;\n",
    "    # defaults are (env.side): 0 - 'do nothing', 1 - 'buy', 2 - 'sell', 3 - 'close position'.\n",
    "skip_frame=1,\n",
    "    # Number of environment steps to skip before returning next response,\n",
    "    # e.g. if set to 10 -- agent will interact with environment every 10th episode step;\n",
    "    # Every other step agent's action is assumed to be 'hold'.\n",
    "    # Note: INFO part of environment response is a list of all skipped frame's info's,\n",
    "    #       i.e. [info[-9], info[-8], ..., info[0].\n",
    "```\n",
    "When maiking own subclass, it's one's responsibility to set those in consistency.\n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.insert(0,'..')\n",
    "\n",
    "import IPython.display as Display\n",
    "import PIL.Image as Image\n",
    "import numpy as np\n",
    "import random\n",
    "\n",
    "from gym import spaces\n",
    "\n",
    "from btgym import BTgymEnv, BTgymBaseStrategy, BTgymDataset\n",
    "\n",
    "# Handy functions:\n",
    "\n",
    "def show_rendered_image(rgb_array):\n",
    "    \"\"\"\n",
    "    Convert numpy array to RGB image using PILLOW and\n",
    "    show it inline using IPykernel.\n",
    "    \"\"\"\n",
    "    Display.display(Image.fromarray(rgb_array))\n",
    "\n",
    "def render_all_modes(env):\n",
    "    \"\"\"\n",
    "    Retrieve and show environment renderings\n",
    "    for all supported modes.\n",
    "    \"\"\"\n",
    "    for mode in env.metadata['render.modes']:\n",
    "        print('[{}] mode:'.format(mode))\n",
    "        show_rendered_image(env.render(mode))\n",
    "\n",
    "def take_some_steps(env, some_steps):\n",
    "    \"\"\"Just does it. Acting randomly.\"\"\"\n",
    "    for step in range(some_steps):\n",
    "        rnd_action = env.action_space.sample()\n",
    "        o, r, d, i = env.step(rnd_action)\n",
    "        if d:\n",
    "            print('Episode finished,')\n",
    "            break\n",
    "    print(step+1, 'actions made.\\n')\n",
    "    \n",
    "def under_the_hood(env):\n",
    "    \"\"\"Shows environment internals.\"\"\"\n",
    "    for attr in ['dataset','strategy','engine','renderer','network_address']:\n",
    "        print ('\\nEnv.{}: {}'.format(attr, getattr(env, attr)))\n",
    "\n",
    "    for params_name, params_dict in env.params.items():\n",
    "        print('\\nParameters [{}]: '.format(params_name))\n",
    "        for key, value in params_dict.items():\n",
    "            print('{} : {}'.format(key,value))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define simple custom strategy:\n",
    "Note using of inner startegy variable **`raw_state`**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class MyStrategy(BTgymBaseStrategy):\n",
    "    \"\"\"\n",
    "    Example subclass of BTgym inner computation startegy,\n",
    "    overrides default get_state() and get_reward() methods.\n",
    "    \"\"\"\n",
    "    \n",
    "    def get_price_gradients_state(self):\n",
    "        \"\"\"\n",
    "        This method follows naming cinvention: get_[state_modality_name]_state\n",
    "        Returns normalized environment observation state\n",
    "        by computing time-embedded vector\n",
    "        of price gradients.\n",
    "        \"\"\"\n",
    "        # Prepare:\n",
    "        sigmoid = lambda x: 1/(1 + np.exp(-x))\n",
    "        \n",
    "        # T is 'gamma-like' signal hyperparameter\n",
    "        # for our signal to be in about [-5,+5] range before passing it to sigmoid;\n",
    "        # tweak it by hand to add/remove \"peaks supressing\":\n",
    "        T = 1.2e+4\n",
    "        \n",
    "        # Use default strategy observation variable to get\n",
    "        # time-embedded state observation as [m,4] numpy matrix, where\n",
    "        # 4 - number of signal features  == state_shape[-1],\n",
    "        # m - time-embedding length  == state_shape[0] == <set by user>.\n",
    "        X = self.raw_state\n",
    "        \n",
    "        # ...while iterating, inner _get_raw_state() method is called just before this one,\n",
    "        # so variable `self.raw_state` is fresh and ready to use.\n",
    "\n",
    "        # Compute gradients with respect to time-embedding (last) dimension:\n",
    "        dX = np.gradient(X)[0]\n",
    "        \n",
    "        # Squash values in [0,1]:\n",
    "        return sigmoid(dX * T)\n",
    "    \n",
    "    def get_reward(self):\n",
    "        \"\"\"\n",
    "        Computes reward as log utility of current to initial portfolio value ratio.\n",
    "        \"\"\"\n",
    "        return float(np.log(self.stats.broker.value[0] / self.env.broker.startingcash))\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure environment:\n",
    "- All strategy parameters shown above that are not meant to be left defaults should be passed to environmnet as kwargs.\n",
    "- when `verbose=1`, pay attention to log output what classes been used (base or custom)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define dataset:\n",
    "MyDataset = BTgymDataset(\n",
    "    filename='../examples/data/DAT_ASCII_EURUSD_M1_2016.csv',\n",
    "     start_weekdays=[0, 1,],\n",
    "     # leave all other to defaults,\n",
    ") \n",
    "\n",
    "env = BTgymEnv(\n",
    "    dataset=MyDataset,\n",
    "    strategy=MyStrategy,\n",
    "    state_shape={\n",
    "       'raw': spaces.Box(low=-10, high=10, shape=(4,4)),  # renered under 'human' name\n",
    "       'price_gradients': spaces.Box(low=0, high=1, shape=(4,4))\n",
    "    },\n",
    "    drawdown_call=30,\n",
    "    skip_frame=5,\n",
    "    # use default agent actions,\n",
    "    # use default engine,\n",
    "    start_cash=100.0,\n",
    "    # use default commission,\n",
    "    # use default stake,\n",
    "    # use default network port,\n",
    "    render_modes=['episode', 'human', 'price_gradients'],\n",
    "    render_state_as_image = False,\n",
    "    render_ylabel = 'Price Gradient',\n",
    "    # leave other rendering p. to dedaults,\n",
    "    verbose=1,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Take a look..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "under_the_hood(env)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "### Time to run:\n",
    "- Play with number of steps. Comment out **`env.reset()`** not to restart episode every time you run th cell.\n",
    "- Refer to 'rendering howto' to get sense of how renerings are updated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.reset()\n",
    "take_some_steps(env, 100)\n",
    "render_all_modes(env)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "****\n",
    "### <a name=\"full\"></a>Full Throttle setup:\n",
    "- Summon Backtrader power;\n",
    "- Wich-is-what: pay attention to arguments being used or ignored."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Clean up:\n",
    "env.close()\n",
    "\n",
    "# Now we need it:\n",
    "import backtrader as bt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define dataset:\n",
    "MyDataset = BTgymDataset(\n",
    "    filename='../examples/data/DAT_ASCII_EURUSD_M1_2016.csv',\n",
    "    start_weekdays=[0, 1,],\n",
    "    episode_duration={'days': 2, 'hours': 23, 'minutes': 55},  # episode duration set to about 3 days (2:23:55),\n",
    "    # leave all other to defaults,\n",
    ") \n",
    "\n",
    "\n",
    "# Configure backtesting engine:\n",
    "MyCerebro = bt.Cerebro()\n",
    "\n",
    "# Note (again): all kwargs here will go stright to strategy parameters dict,\n",
    "# that is our responsibility to consisit observation shape / bounds with what our get_state() computes.\n",
    "MyCerebro.addstrategy(\n",
    "    MyStrategy,\n",
    "    state_shape={\n",
    "        'raw': spaces.Box(low=-10, high=10, shape=(4,4)),\n",
    "        'price_gradients': spaces.Box(low=0, high=1, shape=(4,4))\n",
    "    },\n",
    "    drawdown_call=99,\n",
    "    skip_frame=5,\n",
    ")\n",
    "\n",
    "# Than everything is very backtrader'esque:\n",
    "MyCerebro.broker.setcash(100.0)\n",
    "MyCerebro.broker.setcommission(commission=0.002)\n",
    "MyCerebro.addsizer(bt.sizers.SizerFix, stake=20)\n",
    "MyCerebro.addanalyzer(bt.analyzers.DrawDown)\n",
    "\n",
    "# Finally:\n",
    "env = BTgymEnv(\n",
    "    dataset=MyDataset,\n",
    "    episode_duration={'days': 0, 'hours': 5, 'minutes': 55}, # ignored!\n",
    "    engine=MyCerebro,\n",
    "    strategy='NotUsed',  # ignored!\n",
    "    state_shape=(9, 99), # ignored!\n",
    "    start_cash=1.0,  # ignored!\n",
    "    render_modes=['episode', 'human', 'price_gradients'],\n",
    "    render_state_as_image=True,\n",
    "    render_ylabel='Price Gradient',   \n",
    "    render_size_human=(10,4),\n",
    "    render_size_state=(10,4),\n",
    "    render_plotstyle='ggplot',\n",
    "    verbose=0,\n",
    ")\n",
    "\n",
    "# Look again...\n",
    "under_the_hood(env)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "env.reset()\n",
    "take_some_steps(env, 100)\n",
    "render_all_modes(env)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Clean up:\n",
    "env.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}