diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index f7263974b..49d0ead04 100755 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -70,6 +70,7 @@ nav: - user_guide/evaluate/create_counterfactual.md - user_guide/evaluate/feature_importance_permutation.md - user_guide/evaluate/ftest.md + - user_guide/evaluate/GroupTimeSeriesSplit.md - user_guide/evaluate/lift_score.md - user_guide/evaluate/mcnemar_table.md - user_guide/evaluate/mcnemar_tables.md diff --git a/docs/sources/CHANGELOG.md b/docs/sources/CHANGELOG.md index 11790f8c1..0d620df85 100755 --- a/docs/sources/CHANGELOG.md +++ b/docs/sources/CHANGELOG.md @@ -24,6 +24,7 @@ The CHANGELOG for the current development version is available at - The `mlxtend.evaluate.bootstrap_point632_score` now supports `fit_params`. ([#861](https://github.com/rasbt/mlxtend/pull/861)) - The `mlxtend/plotting/decision_regions.py` function now has a `contourf_kwargs` for matplotlib to change the look of the decision boundaries if desired. ([#881](https://github.com/rasbt/mlxtend/pull/881) via [[pbloem](https://github.com/pbloem)]) - Add a `norm_colormap` parameter to `mlxtend.plotting.plot_confusion_matrix`, to allow normalizing the colormap, e.g., using `matplotlib.colors.LogNorm()` ([#895](https://github.com/rasbt/mlxtend/pull/895)) +- Add new `GroupTimeSeriesSplit` class for evaluation in time series tasks with support of custom groups and additional parameters in comparison with scikit-learn's `TimeSeriesSplit`. ([#915](https://github.com/rasbt/mlxtend/pull/915) via [Dmitry Labazkin](https://github.com/labdmitriy)) ##### Changes diff --git a/docs/sources/USER_GUIDE_INDEX.md b/docs/sources/USER_GUIDE_INDEX.md index 28b17cca6..ef264ffa9 100755 --- a/docs/sources/USER_GUIDE_INDEX.md +++ b/docs/sources/USER_GUIDE_INDEX.md @@ -36,6 +36,7 @@ - [create_counterfactual](user_guide/evaluate/create_counterfactual.md) - [feature_importance_permutation](user_guide/evaluate/feature_importance_permutation.md) - [ftest](user_guide/evaluate/ftest.md) +- [GroupTimeSeriesSplit](user_guide/evaluate/GroupTimeSeriesSplit.md) - [lift_score](user_guide/evaluate/lift_score.md) - [mcnemar_table](user_guide/evaluate/mcnemar_table.md) - [mcnemar_tables](user_guide/evaluate/mcnemar_tables.md) diff --git a/docs/sources/user_guide/evaluate/GroupTimeSeriesSplit.ipynb b/docs/sources/user_guide/evaluate/GroupTimeSeriesSplit.ipynb new file mode 100644 index 000000000..291466add --- /dev/null +++ b/docs/sources/user_guide/evaluate/GroupTimeSeriesSplit.ipynb @@ -0,0 +1,809 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# GroupTimeSeriesSplit: A scikit-learn compatible version of the time series validation with groups" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "A scikit-learn-compatible time series cross-validator that supports non-overlapping groups." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "> `from mlxtend.evaluate import GroupTimeSeriesSplit` " + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Overview" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Time series tasks in machine learning require special type of validation, because the time order of the objects is important for a fairer evaluation of an ML model’s quality. \n", + "Also there can be different time units for splitting the data for different tasks - hours, days, months etc. \n", + "\n", + "Here, we use time series validation with support of the groups which can be flexibly configured along with other parameters:\n", + "\n", + "- Test size\n", + "- Train size\n", + "- Number of splits\n", + "- Gap size\n", + "- Shift size \n", + "- Window type \n", + "\n", + "This `GroupTimeSeriesSplit` implementation is inspired by scikit-learn's [TimeSeriesSplit](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.TimeSeriesSplit.html) but it has several advantages:\n", + "\n", + "- `GroupTimeSeriesSplit` lets you group data as you want before splitting, while `TimeSeriesSplit` only considers the record level.\n", + "- It can be used for both holdout validation (n_splits=1) and cross-validation (n_splits>=2), whereas `TimeSeriesSplit` can be used only for the latter case.\n", + "- `TimeSeriesSplit` uses only an expanding window, while for this implementation you can choose between both rolling and expanding window types.\n", + "- `GroupTimeSeriesSplit` offers additional control for splitting using an additional `shift size` parameter.\n", + "\n", + "**There are several features that need to be taken into account:**\n", + "\n", + "- `GroupTimeSeriesSplit` is compatible with sklearn-learn API.\n", + "- Numbers or custom non-numeric values can be used as groups\n", + "- However, groups should be consecutive\n", + "- Specifying the test size with either a) the train size or b) the number of splits is required parameters for splitting\n", + "- If full data can’t be used with specific parameters, the most recent data is considered for splitting\n", + "- If splitting is impossible (e.g., there is not enough data to split) using specified parameters, an exception will be raised " + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Before we illustrate the usage of `GroupTimeSeriesSplit` in the following examples below, let's set up a `DummyClassifier` that we will reuse in the following sections. Also, let's import the libraries we will be using in the following examples:" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [], + "source": [ + "import numpy as np\n", + "import pandas as pd\n", + "from sklearn.dummy import DummyClassifier\n", + "from sklearn.model_selection import cross_val_score\n", + "\n", + "from mlxtend.evaluate.time_series import (\n", + " GroupTimeSeriesSplit,\n", + " plot_splits,\n", + " print_cv_info,\n", + " print_split_info,\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "tags": [] + }, + "source": [ + "## Prepare sample data" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "For the following examples, we are creating sample dataset consisting of 16 training data points with corresponding targets." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Features and targets" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Let's assume that we have one numeric feature and target for the binary classification task." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [], + "source": [ + "data = [[0], [7], [6], [4], [4], [8], [0], [6], [2], [0], [5], [9], [7], [7], [7], [7]]\n", + "target = [1, 0, 1, 0, 1, 0, 0, 1, 1, 1, 0, 1, 1, 0, 0, 0]\n", + "\n", + "X = pd.DataFrame(data, columns=[\"num_feature\"])\n", + "y = pd.Series(target, name=\"target\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Group numbers" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We create 6 different groups so that the first training example belongs to group 0, the next 4 to group 1, and so forth. \n", + "These groups do not have to be in ascending order (as in this dataset), **but they must be consecutive.**" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "array([0, 1, 1, 1, 1, 2, 2, 2, 3, 3, 4, 4, 5, 5, 5, 5])" + ] + }, + "execution_count": 3, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "groups = np.array([0, 1, 1, 1, 1, 2, 2, 2, 3, 3, 4, 4, 5, 5, 5, 5])\n", + "groups" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Note that the following i another example of a **correct** group ordering (not sorted but consecutive):\n", + "```python\n", + "np.array([5, 5, 5, 5, 1, 1, 1, 1, 3, 3, 2, 2, 2, 4, 4, 0])\n", + "```\n", + "\n", + "However, the example below shows an **incorrect** group ordering (not consecutive), which is not compatible with `GroupTimeSeriesSplit`:\n", + "```python\n", + "np.array([0, 1, 1, 1, 1, 2, 2, 2, 3, 3, 4, 4, 2, 2, 2, 2])\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Group names (months)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We will add months as the index according to the specified groups for a more illustrative example." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "array(['2021-01', '2021-02', '2021-02', '2021-02', '2021-02', '2021-03',\n", + " '2021-03', '2021-03', '2021-04', '2021-04', '2021-05', '2021-05',\n", + " '2021-06', '2021-06', '2021-06', '2021-06'], dtype='" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Train indices: [0 1 2 3 4 5 6 7]\n", + "Test indices: [8 9]\n", + "Train length: 8\n", + "Test length: 2\n", + "Train groups: [0 1 1 1 1 2 2 2]\n", + "Test groups: [3 3]\n", + "Train group size: 3\n", + "Test group size: 1\n", + "Train group months: ['2021-01' '2021-02' '2021-02' '2021-02' '2021-02' '2021-03' '2021-03'\n", + " '2021-03']\n", + "Test group months: ['2021-04' '2021-04']\n", + "\n", + "Train indices: [1 2 3 4 5 6 7 8 9]\n", + "Test indices: [10 11]\n", + "Train length: 9\n", + "Test length: 2\n", + "Train groups: [1 1 1 1 2 2 2 3 3]\n", + "Test groups: [4 4]\n", + "Train group size: 3\n", + "Test group size: 1\n", + "Train group months: ['2021-02' '2021-02' '2021-02' '2021-02' '2021-03' '2021-03' '2021-03'\n", + " '2021-04' '2021-04']\n", + "Test group months: ['2021-05' '2021-05']\n", + "\n", + "Train indices: [ 5 6 7 8 9 10 11]\n", + "Test indices: [12 13 14 15]\n", + "Train length: 7\n", + "Test length: 4\n", + "Train groups: [2 2 2 3 3 4 4]\n", + "Test groups: [5 5 5 5]\n", + "Train group size: 3\n", + "Test group size: 1\n", + "Train group months: ['2021-03' '2021-03' '2021-03' '2021-04' '2021-04' '2021-05' '2021-05']\n", + "Test group months: ['2021-06' '2021-06' '2021-06' '2021-06']\n", + "\n" + ] + } + ], + "source": [ + "cv_args = {\"test_size\": 1, \"train_size\": 3}\n", + "\n", + "plot_splits(X, y, groups, **cv_args)\n", + "print_split_info(X, y, groups, **cv_args)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Please note that if we specify the number of groups for both the training and the test set, the split size is determined automatically, and the number of splits naturally changes with the groups sizes. For example, increasing the number of training groups will naturally result in a lower number of splits as shown below." + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1gAAAEYCAYAAABBWFftAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAA6FUlEQVR4nO3deZyVdd3/8fcbBhCYAdk3WUzRUhIU9M7SX97qXWkumbc3WVmoaS6puZCm3aWmpWnm7ZaJ5ZZ2m6hpau5hmuEdiCCSKMiisonsDALDfH5/XNfRw3H2OWfOmeH1fDzOY8651vc5F5y5PvP9Xt/LESEAAAAAQPO1K3YAAAAAAGgrKLAAAAAAIE8osAAAAAAgTyiwAAAAACBPKLAAAAAAIE8osAAAAAAgTyiwALR5tr9i+0nb79veZPtd2/9r+3PFzpZhe5jtaMAjs9z3WjBbV9s/tT3b9gbbS20/Z/vEPO5jvu2r87W9rO0ebvvvtlfZXmP7Nds32y5v5HbGpZ97efo6cxwOy1rmB7YPyO87AAC0NmXFDgAAhWT7V5LOlHSnpF9Lel/SUElfk/SC7Z0jYm4RI2YslrRv1utPSLpb0umSXq5huXktF033S9pT0mWSZkrqK+n/STpU0m/ztI+jlBybvLF9rKR7JP1GSfaQtIekb0vaXtK6Zmw+cxxez5r2A0k3SJrUjO0CAFo5c6NhAG2V7SMl/UnS8RFxew3zD5c0NSIW1TCvc0RsKHjIWtgeIelVSf8eEZOKmGO4pDck/VdE3Jczz9HMXyKF/Jxt/13Sqoj4cg3zGpXd9jhJt0mqiIgaCzPbyyXdEBEXNy0xAKAtoIsggLbs+5L+WVNxJUkR8edMcZV29zrH9rW231NS3Mh2b9t3pN0LK21Psj0mezs1ddmzfXF6wp15nelitrft59Oudm/YPqqxbyp3f2mmibaPtz3P9jrbd9nuZHsf2/+XTptke0jOtraz/Qvbb9veaHu67UOzFtk+/bmkhs9vqwLF9pC06+WK9LN6wvauWfMz3eq+YftO26sk/Tmd97Eugrb3S7siVqaf/wTbFVnzt7d9q+1Ftj+wvdD2hJzsH8udmz3r2P9Pmn2V7ettd6xp3Zz3clgmv6Rekn6S1Z3zgNrWBwC0XRRYANok22VKunA92YjVxksaIOk4Jd0KpaQF7IuSzpM0Vsn35l9t79zEaPdKekjSV5UUcffZHtnEbWX7jJKub2co6ar2X5KulzRB0v9I+qaSboe35Kw3UdI4ST+TdLikf0p62PaodP5sSeslXWv7C7a3q2nntntKekHSrpJOSfffVdLTtjvnLH61pLWSjkn3W9P2PifpGSUF0n8qKZYPVdKKlHGNpP0kna3kGF2opBtgxsuSjrX9PdsDa9pPlnMl7SDpG0q6E54s6fJ61sl2lKTVSrpM7ps+Xq5zDQBAm8Q1WADaql6SOkl6O3uibUtqnzVpS1ZrxpKIGJu17JckfU7SARHxXDrtWUnzlRRj321Crlsj4up0W09ImiXph0quCWuOcklHRsTqdNsHSDpJ0ucj4m/ptIGSbrTdJSIqbR8k6cvKen+SnrS9i6SLJB0TEWtsn6SkUHtC0mbbkyXdlb6XzGd3tpKCalRErEj393cln9UJkm7Myjo5Ik6v5/1cIenFnOPxrqRnbI+IiJmS9pF0Y0Tcm7Xe77OeXyjp00oKzettz1NSMP8iInJbttam77da0l9sd5J0ke2fZ95PXSJimu0qSe9ExOT6lgcAtF20YAFoq5z+zL3O5lxJm7Me2Sf6j+Ysu4+k97KKD0XEekmPKGk5aYoHs7ZVraQ1a58mbivblExxlZojaZOSVqXsaZKUac05WEkL0d9tl2UeSlqOPuwGGRF/UDIwyAmS/lfSLkpawu7J2vbBkp6StCZrO2slTc3eVir3c96K7S5KWoD+mJPrBSXHbHS66CuSxts+LS0KtxIRb6fLHizpl5JWKCkEZ9jeIWfxh9LjkfGApM6SRtSVFQCAXBRYANqq5ZI2Kun2le0uSXunj1xLc14PqGFaZrmeTcy1rIbXA5q4rWyrcl5vkrQ2p2jYlP7MdPPrLam/ti44N0u6WNLg7I1FxPsRcVtEfCudd5ukr2V1b+ytpAtl7rb+PXdbqvkzzdZDSSvjTTnb2iipQ9b2vqekRerHkmbbftP2Vi2BEbElIp6JiPMiYoySroQ9lRTa2Wo6LlJ+jg0AYBtCF0EAbVJEVNn+h6QvKDkBz0xfqvQEP+ktuPVqOa8XKxmSPFc/Ja0hGRsl5Q6IUFsB1ldbD0feN91PMayQ9K6krzRmpYjY7GT4++MlfVLS9HRbD0v6aQ2rrM3dRD27WJUuc7Gkx2qYvyjNsUrJtXJn2t5DybVnd9ueERGzasn+pO3pae5succ587pYxwYA0ErRggWgLbtW0r/ZPq6J678kqa/t/5eZkHZf+7K27nr3jqRPZS3TTtKBtWzzqJzljpT0f03M11zPKGnBWhcRU3IfacaKGgapkKTh6c+lWdvaXdJrNWxrdmNCpd0wJ0vataZcNQ2rHxEzlFwX105p8WT7Y8VxOkjHDvp4K9qR6fHI+KqkDUru+9VQm/RR6yAAYBtFCxaANisiHrJ9raTbbf+7kiHBlysZAOM/0sVqvdlsRDyRDtRwr+0LlLQ8nafk2pyrshZ9UNLptqdJekvSdyR1q2Wz37G9ScmJ+0mSdpZ0bNPeYbM9pWTgiqdsXynpNSW5R0naLiJ+qGRUwIdt/07Si5Iq0/kXKbkGKlNoXqNkpMJnbV+vpGWsn6TPS3ohvY6rMX6gZECLaiUjHa6VNERJcXtRRLxh+wUln/1MJS1eJykZ8TBTsD5h+3Ulx/1tJcXk95R0QfxNzv4qlIzoOEFJofhjJfe0qneAiyyvS/qy7ceV/LuaHRG5rXcAgDaOAgtAmxYRZ9v+m6TTlAyhXSHpPUn/kHRoRPylnk0cpWSAhGuVtE78n6QDI2JO1jKXKOlSdpmSVowblJz0b3VvrNTXJP0qXfYdSWMjYlqT3lwzRUTY/qqS0fa+r6SAWaGkcLo+XWyupFuVXLt0qpLicqGk30m6MiKq0m0tt/0ZJUOb/0rJPagWKynAZjQh2wtpy+ElSq6bay9pgaTH9VHr0z+UDDE/TNIWSdMkHRIR76Tzf6Hk875SyfF5T8nQ6ftFRG6r4S+VDGP/ByWtYLemn0tjjFcyWuKjkroouf5sUiO3AQBo5dyIG9kDAJrI9jglA0NUREStrWZoebZD0hkRcUOxswAAWj+uwQIAAACAPKHAAgAAAIA8oYsgAAAAAOQJLVgAAAAAkCcUWAAAAACQJxRYAFqU7VG2J9teb/u2nHlh+4D0eUfbF9seVYAMBdm27Um2J+Z5mweko9w1dr0+tq+z/X+2N9me34wM47LXt72P7Yubur069hO2axravmBK6ZjlbKOr7XfSz2REzrxf2d5ge4btQ5uXFgCQbxRYAFraZZIGSfpG+rw2HSX9RMlNbfOtUNs+TdIP87zNphokaaykJUrua5VP+yj5/PJtX0n3FWC7dSmlY5btItV+r8prldyfbbOSe64BAEoINxoG0NJ2kfRIRPyp2EHyLSJmFTtDlhkR0U+SbF8t6T9bOoBtS+oUER80ZPmImFzgSDXts5SOmSTJ9s6SzpR0nqRf586PiAWSFqTLXW+7Y0RsauGYAIBa0IIFoKV1lLS+AcutTX/elnaTCtvDJMn2drZ/Yftt2xttT8/tKmX7CNtT066IK22/ZPvz9W27LrZ3t/247RXpdv9l+/Ss+Vt1N8vadu5jXNYy+9l+znal7fdtT7Bd0YDPp04RUd3cbdQkzX59+jzzfialry+2vTx9T/+U9IGkY9LubjfYnp2+z3m2b7TdLWfbW3URzHyetr9ue47tNbb/YnuHRuRtNccsy7WSbpX0ej3LZW5Y3TGP+wYANBMtWABKRkQ46+WBkp5V0o3w0XTa4vTnRH3UTW2upP+S9LDtMRHxiu2d0mX+R9J4SdtJGi2pZwO2XZeHlZz0flPSRkm7SupWx/L75rz+uqTvSZojSbY/J+kZSX9S0sLUS9IVknqkrxURkyRlfy5Kr4eaFBHjGpC52SLidkm3py8flfRLSefqo/e3JmvxLpLukPQLSW9IWpROa6+k29t7kganz++T9MV6dv9vkgam++us5JjeIqmh1x61qmOW/qHgM5KOkzSyzncGAChJFFgAWtr2algL1j/Tn3Ozu47ZPkjSlyUdEBHPpZOftL2LkpP2YyTtKWltRIzP2t5j9W27LrZ7S/qEpK9ExKvp5GfqWicn916STpJ0SUS8kE6+QtKLETE2a7l3JT1je0REzKxl01WStjQkd75FxHuZAS9q+ew6SzonIh7KmX5q5ontMknzJL1ge0hELKxjl90kfTkiVqbr9pf0K9udI2JDXVlb2zGz3VFJAfnjiFiZ9LCsU2X6s7s+as0CABQZXQQBtIi0m9gJSk4GpzZjUwcrGbjh77bLMg8lJ85j0mVeldTd9h22v2C7a7PCJ1ZIelvSzbbH2u7b0BVt95H0oKSnJV2aTuuipLXkjznv4wUlgxeMrm17EbFzRJzY9LdSUCHpL7kTbR9ne5rtdUreX6Zg2aWe7f0zU1ylMtdMDWpAltZ2zM5R0q3yNw2MOU1StaTTbXdzAyoyAEDhUWABKLj0+qZ1kn4r6fqIeLgZm+stqb+SE9rsx8VKup4pImZLOlJJ68Vjkpbbvic9aW6S9JqmLygp7n4naYnt523vWdd66Qn4HyVtkvTNiMgM391DSbe5m3Lex0ZJHTLvpRVamTvggu2jJN0p6R9KWhg/o2QUPCnpvlmXVTmvM9uub71WdczSf5sXKfl3XGF7e0nl6eyKmv5IEBFvSjpd0g8krZb0+dxlAAAtjy6CAFrCIkn7K7lW6jTbd0fES03c1gpJ70r6Sl0LRcSjkh613V1Jl8JrlQzO8LUm7lcR8bqko213UPJ+rkz3sUMdg0pcLWlvSZ+JiNVZ01cpae25WFt3X8xY1NScRVbT/Z+OkfRSRJyWmeCPBhwpbJjWc8wGKSmoaron14tKWmgPzp6YtshdKekhSddJerkZ+wcA5AkFFoCCS1s0XrA9WdKJSloV6iuwamupeEbJgAfr0pPn+va9WtI96Ql9ZgCDBreC1LLNzZKetX2NpHuUXFe2Inc528dJOkvS13KvzYmI9ennsWtEXNqUHEW0SUpGc2zgEOydlbTyZPtG3lPVoRUcszmS/j1n2ihJv5J0gmounj6n5Bq1H0TE3DznAQA0EQUWgBYTEVW2l6vuUdwyy26yPU/Sf9meqeTalBmSnpL0hKSnbF8p6bV0e6MkbRcRP7T9XSXF1ONKWhWGK2lFubOubdd1LyHbeyhp2bhX0ltKuoudL2l6RNR0or6TktHu/qLknkWfyZo9NyLeU9K16xnb1UpaLtZKGqKkxe2iiHijlixzJD1X3zU9tjP3vtpFUpes18+l+5ft25UMGDKsrm3lyBS2Z9l+VtKatFtmbZ6SdKPti5QU1odKOqgR+2uS1nTMImKdpEk562Se/rOWwTMy/4+W1bRNAEBxUGABaGmhnCGs63CKkhPkpyV1krRjRMy3/VVJF0r6vpKT2xWSXlF6fyYlhdgRkq5RMjT7YkkTJP24rm1Lml9HliWSliq5Tmagku5if1Vywl6TwUpayA5JH9mOl3R7RLxg+/9JukTSXUqu71mgpDBcWkeWsnTZ+txXy+t/10cn813U+BP05yVdpaSl5+eS/ibpgDqW/42S6+HOUvKZPKVk+PNC31i4NR6zxsj8P6qpWyYAoEj80bW7AFB4tudKeiwizih2Fki2FygZFvyOYmdB46QttTdL6tzArpoAgBZACxaAlvaOpP1sD5S0pI6BBlBg6THoIOkPxc6ChrPdTkl3xy9JWk5xBZSOqVOndmzXrt2p7du3Pz4iuqvhPTbQOoTt1Vu2bLmturr616NHj67x0gIKLAAt7RpJdysZCfAOSeOKmiaVnrTWeuuKiKhqwTgtIiIWKek61ypti8csdY2S7pabVXt3RwBFUFZWNqFbt26fGzhw4PqOHTu+z+3p2paI0KZNmzosWrTojDVr1uwl6ds1Lcd9sAC0qIh4SMm9rEYoGe66VPxOH7+31oeP9F5eKC3b6jH7paQ9JPWMiF8VOwyArew3dOjQ1Z06ddpMcdX22FanTp02Dx06dLWk/WpbjhYsAC0u7dL0WrFz5LhY0g11zG+t96Vqyy7WNnjMIuJtSW8XOweAGrVv164dAxy0cekxrnXgIgqsGvTu3TuGDRtW7BgAWtDo0aPrXWbMmDEtkAQNxTEDUGhTp05dHhF9ip0DrQsFVg2GDRumKVOmFDsGAAAAiigdaRVoFAosAAAAoIB+9odNIys3Fv68u0snVV14bMfphd5Prl//+tc9r7322v6zZ8+eVYjlWxsGuQAAAAAKqCWKq8bsp0uXLntmHmVlZXuVlZXtlT2tsfs99dRTVzSmWGrs8i1p9uzZHW2Pnjt3boemboMWLAAAAGAbUllZOS3zfOzYsUOrqqp8//33z69p2Y0bN7pTp04M3NEItGABAAAAkCTts88+u55wwgmDDz744J3Ky8v3vOSSS/rNnTu3w/777z+8R48eIysqKkaNHj161+eff75LZp3rrruu15AhQ0Zkb+Okk07a4Ytf/OJOXbt23XPw4MEjfv/732/f1OWrq6t1wQUX9O/Xr98e3bt3H3XiiScO3nfffXc555xzaryX4wcffOBjjz12aM+ePUeWl5fvOWzYsBG33XZbj8z8xx9/vHz06NG7du/efdTgwYNH/OQnP+lXXV0tSRozZsxukjRixIgRXbp02XP8+PEDGvsZUmABAAAA+NAf//jH3meeeeayNWvWTPvhD3+4rLq62qeccsqyhQsXvrp48eLpe+yxR+XYsWN32rhxY603+5o4cWKvc889d8maNWumfec731l26qmnDlu7dm2ttUddy9900029JkyY0O+BBx54c9myZdMHDBiw6Z///Gd5bdu64YYber3yyitdZ82aNXPdunXTnn766dl77LHHBkmaMmXKdkcfffTws88+e+n777//ysMPPzzn1ltv7XvTTTf1SufPkqSZM2fOrKysnHbVVVctbuznR4EFAAAA4EOHHHLIyiOOOGJtu3btVFFRUT18+PBN3/jGN1ZXVFRUl5eXxzXXXPPu4sWLO86cObNTbds47LDDVn7hC19Y3759e5199tnL161b176py99zzz29jjvuuPc+97nPbejUqVNceumlS/v27bu5tm117NgxKisr273yyiudN2/erJ133nnz6NGjP5Ck6667ru+hhx668pvf/OaqsrIy7bnnnh985zvfWXbPPff0as5nlo1rsAAAAAB8aOjQoRuzXy9evLjstNNO22Hy5MkVa9euLbMdkrRkyZJaa4kBAwZ8WAB169atWpLWrFlT681561p+yZIlHYYOHbopM79du3YaMGDApo9vJXHqqae+v3Tp0g7nnXfe4AULFnTad999115zzTXvjBgxYuPChQs7Tp48uVtFRcX2meUjwv379691e41FCxYAAACAD7Vrt3WJcPbZZw9aunRph8mTJ7++bt26aQsXLpwhJYVJS+Tp37//5gULFnTMvK6urtbixYs71rZ8hw4ddPnlly+ZOXPmv+bNm/dq586dq8eNGzdMknbYYYdNxxxzzPK1a9e+knmsW7du2pw5c16TPv7em4ICCwAAAECt1q5d275z587Vffr0qVq9enW7M844Y4eW3P+xxx77/u9///s+//jHPzpv3LjRl1xySb9ly5bVOoz6ww8/XPH888932bhxo7t27VrdtWvX6rKyspCks846a9mf//znnvfcc0/3jRs3evPmzZo6dep2jz76aLmUtKS1a9dOs2bN2q6peSmwAAAAgALq0klVrXk/l1122aL333+/Q69evUbtvvvuu3/2s59d1759rb398u70009///jjj1925JFHDu/bt+/Id955p+PIkSPXd+rUqbqm5RcvXtxh3LhxO/bo0WNU//79R7799tsdJ0yYsECS9t577w8mTpz45vXXX9+vf//+e/Tu3XvUuHHjdswUbOXl5TF+/Ph3jz/++E9UVFSMOv/88/s3Nq8jGNY+15gxY2LKlCnFjgEAAIAisj01IsY0dPnp06fPHzly5PJCZoK0ZcsWDRw4cI9LLrnknVNOOWVFMTJMnz6998iRI4fVNI8WLAAAAAAlbcKECT0qKyu9du3aduecc87ADRs2tDv66KNXFztXTSiwAAAAAJS0m2++uW+/fv1GDhw4cI/nn3++2wMPPPBmnz59thQ7V00Yph0AAABASZs6dersYmdoKFqwAAAAACBPKLAAAAAAIE8osAAAAAAgTyiwAAAAACBPKLAAAAAAIE8YRRAAAAAooFXXnDcyNqwv+Hm3O3et2v6cq6cXej+NsfPOO+9+/vnnLzrppJNWFjtLS6HAAgAAAAqoJYqrxuynS5cue2aeb9q0yZLUsWPHyEyrrKyc1th9z549u+MnP/nJT8+ZM2fGTjvttDkzfc6cOa81dlst5eijjx5WVlYW995774J8bpcCCwAAANiGZBdQY8eOHVpVVeX7779/fhEjtSlcgwUAAABAkvTmm292/NKXvvSJPn367NGnT589jj322KErV65sJ0nV1dU644wzBvXt23ePrl277jlo0KBPX3755X0lacyYMbtJ0ogRI0Z06dJlz/Hjxw+QpEGDBn36pptu6ilJjzzySEVZWdnoCRMm9Bg8ePCIioqKUYceeugnMtuXpBkzZnTae++9dy0vL99z11133e2nP/1pX9uja8v7pz/9qeJTn/rUbuXl5Xv26NFj5Gc/+9ldMvPWrl3b7uSTT95h0KBBn+7evfuo/ffff/jMmTM7SdKPfvSjfg899FDP+++/v1eXLl327NKly55VVVV5+QwpsAAAAACosrLSBx100C6f/OQnP5g3b96rs2bNem3RokUdTj755CGS9Kc//anbH//4x14vvvjiv9avXz9t8uTJ//r85z+/TpKmTJkyS5Jmzpw5s7KyctpVV121uKZ9bNmyRU899VS3mTNnzpo1a9bM1157rcsVV1zRV5I2b96sI444Yvjuu+9euWTJklcefPDBOXfeeWefujKffPLJO55yyilL16xZM23RokUzfvjDH364369//etD33zzze0mT578r6VLl04fM2bM+sMPP3znjRs3+rLLLlt65JFHrjj66KPfr6ysnFZZWTmtrCw/nfsosAAAAADo3nvv3T4idO211y4qLy+PPn36bLnssssWPfTQQz2rqqrUsWPH2LRpk1955ZXOlZWVHjx4cNV+++1X2dj9XHvtte927969evDgwVVf+tKXVr388stdJenZZ58tX7RoUccbbrjhnfLy8thtt902nXbaaUvr2laHDh1i7ty5nd55552yzp07x+GHH75WkhYvXlz2yCOP9LzlllsWDh48uGq77baLq6++etHy5cs7TJo0qWvTPqGGocACAAAAoLfeeqvj4sWLO1ZUVIzKPA499NBdbOvtt9/ucNhhh6296KKL3v35z38+oG/fvqP222+/4X/729+6NGYf7du318CBAz/si9e1a9fq9evXt5ekhQsXdujZs2dVeXn5hwNu7Ljjjhvr2t7EiRPnzJ07d7uRI0fuvtNOO+1+6aWX9pWkN954o6Mk7bXXXrtl3sv2228/qqqqyvPnz+/YmMyNxSAXAAAAADR06NBNw4YN21jXyH/nnXfe8vPOO2/52rVr240fP37g2LFjd1q8ePGr7do1v91myJAhm1euXFm2bt06Z4qs+oqhfffdd8Ojjz76VnV1tZ588snyr3zlK7uMGjVqw+jRozdI0uzZs2dmF3TZ8pG5xu0WZKsAAAAAWpWxY8eurqqq8gUXXNB/5cqV7aqrqzVv3rwOd9555/aS9Nxzz3V54oknyjds2ODOnTtXV1RUbGnfvr0kacCAAZvbtWunWbNmbdfU/R944IHrBgwYsOnMM8/cobKy0q+//nrHm266qV9ty3/wwQe+/vrrey1evLisXbt26tWrV5Xt6NChQwwaNKjq8MMPX3HiiScOmTdvXgdJWr58efs777xz+9WrV7eTpH79+m1esGBBpy1btjQ1co0osAAAAIACcueu+RmersD7qaioqH766adn/+tf/+q86667jujWrdueBx544C7Tpk3rIkmrV69uf/bZZw/u3bv3qB49eox69tlnu911111zJam8vDzGjx//7vHHH/+JioqKUeeff37/xu6/Q4cOevDBB+fMmDGjS58+fUYdeeSRO48dO/b9Dh06RG3rTJw4scduu+22e5cuXfY86qijho8fP37RIYccsk6S7r777gXDhw//4IADDti1a9eue44YMWL3++67r4dtSdJpp532XmVlZbsePXqMqqioGJWvUQQdUWvebdaYMWNiypQpxY4BAACAIrI9NSLGNHT56dOnzx85cuTyQmba1lx11VW9b7zxxv7z58+fWews2aZPn9575MiRw2qaRwsWAAAAgJLw5JNPdn3ttdc6VVdX66WXXup83XXX9T/qqKNWFDtXYzDIBQAAAICSMG/evE7f/va3P7Fq1aqyHj16VB122GErL7/88hrvqVWqKLAAAAAAlITvfve7K7773e+2qharXBRYefLTPzTtorj/PrZlDkFT80mln5F8iVLPJ5V+RvI1D98zzbfmmu83ab1u51yb1xy1KfV8AFAKuAYLAAAAyI/q6upqFzsECis9xtW1zW/zBZbt9ravsv2e7bW277fdu9i5AAAA0LbYXrJhw4Ym3wcKrcOGDRu2s72ktvltvsCSdIGkIyX9m6Qd0ml3FS8OAAAA2qKqqqpL5s+f33H9+vWdaclqe6qrq71+/frO8+fP71hVVXVJbcttC9dgnSzp0oh4S5Js/0DSHNvDImJ+UZMBAACgzdhrr72eePnll783d+7cn0REf20bjRnbkmrbS6qqqi7Za6+9nqhtoTZdYNnuLmmIpKmZaREx1/YaSXtImp+17MlKijENGTKkZYMCAACgTUhPvGs9+Ubb19ar6m7pz9U501dlzZMkRcQtETEmIsb06dOnJbIBAAAAaGPaeoG1Nv3ZPWf69pLWtGwUAAAAAG1dmy6wImKVpIWS9spMs/0JJa1XM4oUCwAAAEAb1aYLrNQtks63vaPtbpKulPQEA1wAAAAAyDdHRLEzFJTt9kqKqnGSOkl6StLJEbG8tnXGjBkTU6ZMaZmAAAAAKEm2p0bEmGLnQOvSpkcRlKSI2CLpvPQBAAAAAAWzLXQRBAAAAIAWQYEFAAAAAHlCgQUAAAAAeUKBBQAAAAB5QoEFAAAAAHlCgQUAAAAAeUKBBQAAAAB5QoEFAAAAAHlCgQUAAAAAeUKBBQAAAAB5QoEFAAAAAHlCgQUAAAAAeUKBBQAAAAB5UtaQhWxb0tckjZFUkT0vIk4uQC4AAAAAaHUa2oL1a0k3SBosqUPOAwAAAACgBrZgSTpG0j4RMbeQYQAAAACgNWtoC1alpIWFDAIAAAAArV1DC6xfSPpxei0WAAAAAKAGDe0ieKakoZLOsL0se0ZE7JL3VAAAAADQCjW0wLqsoCkAAAAAoA1oUIEVEXcUOggAAAAAtHYNbcGS7b0lnaBkqPa3Jf0uIv5ZqGAAAAAA0No0aJAL21+R9DdJ3SVNk9RN0nO2jypcNAAAAABoXRragvUTSUdHxGOZCbYPkXSFpAcLEQwAAAAAWpuGDtM+TNLjOdOeUDKyIAAAAABADS+wFkg6OGfaQeLmwwAAAADwoYZ2EfyppIdsT5T0lqQdJR0t6duFCgYAAAAArU2DWrAi4n5JB0qqlLS3pA2SDo6IiQXMBgAAAACtSoOHaY+If0j6RwGzAAAAAECrVmuBZfvfIuKl9Plna1suIl4sRDAAAAAAaG3qasF6WlJF+vyFWpYJSe3zmggAAAAAWqlaC6yIqMh63tDRBgEAAABgm9Wgwsn2cbVM/0Z+4wAAAABA69XQlqkba5l+fb6CAAAAAEBr19ACyx+bYA+TVJXXNAAAAADQitU5TLvtzUoHsrC9KWd2e0k3FSoYAAAAALQ29d0H62AlrVePSToka3q1pCUR8WahggEAAABAa1NngRURz0mS7Z0iYnHLRAIAAACA1qm+FixJUkQstt1L0t6S+ijrmqyIuLNA2QAAAACgVWlQgWX7YEn3S9okaXtJq9Kf8yRRYAEAAACAGj6K4BWSLo2IPpLWpT9/KunmgiUDAAAAgFamoQXWcEnXps8z3QOvlPT9POcBAAAAgFaroQVWpaRO6fP3bQ+R1FFSj4KkAgAAAIBWqKEF1ouSvpI+f0zSw5KelvSPAmQCAAAAgFapQYNcSPqmPirGxks6V1KFpGsKEQoAAAAAWqN6CyzbZZKuVlJUKSI+kHR5gXMhz376h6omr/vfxza0Dm+epmYkX6LU80mln5F8zcP3TPORr3nWXPP9Jq/b7Zxr85ajLk3NSL5EqecDpAZ0EYyIKklfk7Sx8HEAAAAAoPVq6DVYD0k6upBBCsH212w/b3uN7ab/aRUAAAAAGqChbfYdJf3e9imS5kuqzsyIiJMLkCtfVkq6SVJnSbcUOQsAAACANq6hBdZmSX9In7dPHyUvIp6QJNsHFDcJAAAAgG1BgwqsiDi+0EGKzfbJkk6WpCFDhhQ5DQAAAIDWqKHXYMl2d9tft/2D9HV/2wMLF61lRcQtETEmIsb06dOn2HEAAAAAtEINKrBs7yVpjqQLJP13OnkPSdcXKBcAAAAAtDoNbcH6H0k/iIg9JGVG43tR0mcKkgoAAAAAWqGGDnKxu6Tb0+chSRGxznbXQoTKF9vtJXVQMgqibG+XztoYEVG0YAAAAADaJDekzrA9W9IXImKB7RUR0dP2zpIeiojdC56yiWyPk3RbDbN2jIj5ta03ZsyYmDJlSqFiAQAAoBWwPTUixhQ7B1qXhnYRvEPS/9reT5Jtj5Z0q6QJBUuWBxFxe0S4hsf8YmcDAAAA0PY0tIvglZLKJT2W/vyrkuuyGOQCAAAAAFINvQ/WFkkXSrrQdu+IWF7YWAAAAADQ+jR0mPbHM8+ziyvbjxYiFAAAAAC0Rg29BuuztUxnmHYAAAAASNXZRdD21zPL2T5WkrNmD5e0slDBAAAAAKC1qe8arMvTn50k/SxrerWkJZLOKEQoAAAAAGiN6iywImJHSbL9cEQc0TKRAAAAAKB1atA1WBRXAAAAAFC/WluwbF8XEWemz2+pbbmIOLkQwQAAAACgtamri2CHWp4DAAAAAGpQa4EVEadmPT++ZeIAAAAAQOvV0PtgAQAAAADqQYEFAAAAAHlCgQUAAAAAeUKBBQAAAAB5UmeBZfth24fadksFAgAAAIDWqr4WrPcl/VHSfNv/bXtQC2QCAAAAgFapzgIrHZ59oKRfSPqqpHm2/2T7EFq1AAAAAGBr9V6DFRFrIuLGiNhT0v76qFVrnu0fFTogAAAAALQWjRrkIiJeiogTlRRaWyRdUpBUAAAAANAKNbjAsl1m+xjbT0maIukdSccVLBkAAAAAtDJl9S1gexdJJ0n6lqT2ku6SdEZEvF7gbAAAAADQqtRZYNl+TtJ+kl6UdK6k+yJiY0sEAwAAAIDWpr4WrGmSTomIf7VEGAAAAABozeq7BusKSZ+uaUZ6PVbf/EcCAAAAgNapvgLrB5KG1zJvJ0nn5zcOAAAAALRe9RVYh0q6tZZ5v5N0WH7jAAAAAEDrVV+B1T8iltY0IyKWSeqf/0gAAAAA0DrVV2Btsj2gphnp9M35jwQAAAAArVN9BdbfJZ1Ry7zTJT2f3zgAAAAA0HrVN0z75ZKet91H0h8kvStpkKRjJX1DyT2yAAAAAACqp8CKiCm2j5B0o6QTJYUkS5oj6YiIeLnwEQEAAACgdaivBUsR8ZSkXWwPl9RH0nsR8WbBkwEAAABAK1NvgZWRFlUUVgAAAABQi/oGuQAAAAAANBAFFgAAAADkCQUWAAAAAOQJBRYAAAAA5AkFFgAAAADkCQUWAAAAAOQJBRYAAAAA5AkFFgAAAADkCQUWAAAAAOQJBRYAAAAA5AkFFgAAAADkSVmxA7QVwy54tEnrzb/iy3lOUrNP3/HpJq/76rdfzWOS2jU1I/kSpZ5Pkm485dkmrXf6zQfmOUnNyNc871zwfJPX3eGK/fOYpHZNzdhS+Z55dqcmrXfQgXPznKRmpZ7vtjcXNXnd44cPzGOS2jU1I/kSpZ4PkGjBAgAAAIC8ocACAAAAgDwpyQLLdodiZwAAAACAxmqxAst2f9t/tr3a9hu2T7QdtofZvt323bZvs71C0nXpOqfanp2uM9n2/lnbu9j20zn7mGT7R+nzA2xX2f627QW2V6T7KW+p9wwAAABg29KSLVh3S9okabCk/SQdlzP/GEmPS+oj6Vzbx0r6qaRvSeolaYKkx20PbcQ+20s6XNIekj4laRdJv6xpQdsn255ie8p7773XiF0AAAAAQKJFCizbO0g6UNL4iFgTEcuUFE/ZXoiIeyNiS0RUSjpe0m8i4qWIqIqI30qaIenrjdz9+RGxOiKWSvqxpG/b/tj7johbImJMRIzp06dPo98jAAAAALRUC9ag9OfCrGkLcpaZn/N6sKS3cqbNTac3RvZ+5kvqJKl3I7cBAAAAAPVqqQLr3fTnkKxpQ3KWqc55/bakHXOmfSKdLknrJHXNmV/TTQ6yuxQOk7RR0vI6sgIAAABAk7RIgRUR70iaJOkK2xW2+0r6UT2r3S7pu7b3sV1me5ykUZL+kM6fImkv26PT+d/TxwsySfq57W7pPi+WdFdE5BZzAAAAANBsjoiW2ZE9QMlAFftLWirpV5JuUtLq9HNJVRHxnZx1vifpTEn9JM2W9IOImJQ1/2eSTkpf3pxu++mIuMz2AZKelnSipEsldZP0Z0mnR8TaurKOGTMmpkyZ0ox3CwAAgNbO9tSIGFPsHGhdWqzA+tiO7S9KekhS5yhAiEyBFRFljV2XAgsAAAAUWGiKRhcfTWV7pKSQ9KqSrnyXSbq3EMUVAAAAABRDS94Hq6ekB5QMTvGCkiHXz2rB/QMAAABAQbVYC1ZE/FXSzi24v0lqwfcHAAAAAC3ZggUAAAAAbRoFFgAAAADkCQUWAAAAAOQJBRYAAAAA5AkFFgAAAADkCQUWAAAAAOQJBRYAAAAA5AkFFgAAAADkCQUWAAAAAOQJBRYAAAAA5AkFFgAAAADkCQUWAAAAAOQJBRYAAAAA5AkFFgAAAADkCQUWAAAAAOQJBRYAAAAA5AkFFgAAAADkCQUWAAAAAOQJBRYAAAAA5AkFFgAAAADkCQUWAAAAAOQJBRYAAAAA5AkFFgAAAADkCQUWAAAAAOQJBRYAAAAA5AkFFgAAAADkCQUWAAAAAOQJBRYAAAAA5AkFFgAAAADkCQUWAAAAAOSJI6LYGUqO7fckLcjjJntLWp7H7eVbqeeTSj8j+Zqv1DOSr3lKPZ9U+hnJ13ylnpF8zVOIfEMjok+et4k2jgKrBdieEhFjip2jNqWeTyr9jORrvlLPSL7mKfV8UulnJF/zlXpG8jVPqefDtoMuggAAAACQJxRYAAAAAJAnFFgt45ZiB6hHqeeTSj8j+Zqv1DOSr3lKPZ9U+hnJ13ylnpF8zVPq+bCN4BosAAAAAMgTWrAAAAAAIE8osAAAAAAgTyiwAAAAACBPKLAKyHZ721fZfs/2Wtv32+5d7FwZtr9m+3nba2xXFTtPLttX2n4tzbfI9gTbPYudK5vty23PSzMusz3R9pBi58plu53tF22H7R2KnUeSbN9ue7PtdVmP04qdqya2D7Y9Oc243PZNxc4kSen/j+zPb0N6jPcqdrYM2/1t35t+D660/aztkcXOlWG7l+07bC+xvdr2PbZ7FDFPnd/Ltr9le67tStsv2R5dKvlsj7T9F9uL03+H+7VktgZm/Fb6Xbgy/b/8F9ufLqF8x9iemeZbafsF258vlXw5y12ZHudvtmS+dN91fYbjbFfnfDf+oaUzYttGgVVYF0g6UtK/Scqc1N5VvDgfs1LSTZK+X+Qctdki6ZuSekkaqeQzvK2oiT7uLkmjIqKbpGGSFkr636ImqtnZkiqLHaIGd0REedajJAqXbLYPkDRR0tVK/i3uIOnWIkb6UETsnv35SbpG0qyIeLnY2bLcJKmnpF0l9ZM0RdIjtl3UVB+5U1K5pOGSdlRyjIv5PV3r93JasPxa0qmSeki6X9JjtruVQj5JmyQ9IOmIFsxTk7oyVkj6iZL/x4MkvSzpSdudWyxd3fkmS/qPiOih5N/idUqO8fYtlq4B5wa295F0iKTFLZQpV30Z38r53XJsy0UDpLJiB2jjTpZ0aUS8JUm2fyBpju1hETG/qMkkRcQT0ocnkCUnIi7Mevme7Rsk3VOsPDWJiNezXlpStZITyZJhexdJp0k6WtK0IsdpjX4u6eaImJg1rZQKGEmS7TJJJyjJW0p2lnRDRKyQJNu/lTReycnj8mIGs91VyUninhGxNp32M0mTbA+NiAUtname7+WTJD0QEU+my1wl6XuSjpJ0R7HzRcS/JP0rnd8ScWpUT8Ybs1+nx/tCSZ9UC30/1pPv7ex4Sv7Q2EXSYEmrCp+u/nMD250k/VbJOU5RWoZK/fwFoAWrQGx3lzRE0tTMtIiYK2mNpD2KlauVO0jSjGKHyGX767ZXS1on6SxJFxc30Udst5P0OyUntKuKm6ZGR9teYfsNJ91py4sdKFt6Ar6PpA9sv5x2KZpke0yxs9XgK5K6K2mRKSVXKTnOvW1vp+Sk7IWIKGpxlXLWIyPze7FkujFmGamtf6eEkqKgFLO2Fgcpad2fU+wgGbaH2F6lpEVwoqR7I+LV4qbaysWSno2IfxQ7SB0Gp91+37b9v7Z3LHYgbFsosAon02Vjdc70VVnz0EC2j1by19uzip0lV0TcExHdJQ1Q8ounlH4RniVpSUQ8UOwgNbheyV+Neyv5C/znJU0oaqKP66Hke/IkSeMkDZT0pFq+y05DfFfJidiqYgfJ8XdJ7SW9p+SPEF9V8nkWXUSskzRJ0sW2t7fdR0lrhlSa39MV4ndK3qSt+7dKOjfTglkKImJhRGyv5LgeL+mvxU30kfSPS8dIuqjYWerwN0mfVvJ9vbekDyQ9lf7BDGgRFFiFk/my7p4zfXslrVhoINvHKDnxPqLEri3ZSkQsUZLzEZfAYBy2d5Z0rpIuRCUnIqZGxNKIqI6I15RcJ/afafeTUpH5f3xbRMyIiE1KuuB1kPTZ4sXamu2dlPwl/uZiZ8mWtqA+LekNJd+FXSRdLul52/2KmS3LNyVtVNK17f8kPZROL4UWtlxrxe+UvLC9m5LC5eqIKKn/NxkRsT4ibpd0lu0vFjuP7Y5KroM+Pf3jREmKiLci4o30d8sSJX/QGSjpM0WOhm0IBVaBpH9FXijpw9G8bH9CyV+kSq6bW6myfbyk30g6PCJK5q94dSiT1FXJl3mx7Sepj6SZtpfro+uGZrg0R+urTn+WyuAHiojVkuZLippmt2yaOn1X0vSIeKnYQXL0VDJwxPURsSYiNkXErUp+95TEyU5EvBsRYyNiQETsKGmekr94Ty5ytJpM19a/UyxpVDodDeRklM1Jkq6IiF8UOU5DlCkZhKXYBkraXdLdaXfp5UquDfu17buLG61OkT5K5ncL2j4KrMK6RdL5tndMR3m6UtITpTDAhfThMPLbSeqYvt4ufZTEl5DtM5WM3PbFiPh7sfPkcjL0+fds901f7yDpRiUn5K/XtW4L+aOknZScgI2SdGg6/Qsqget00mF2t0+fD5f0S0kPR8QHRQ32cTdJOt72bulAEuOVnIC/WNxYifSvyuNUYq1XkpReZ/WGpNNsd7VdZvsEJV3dSqIrre1dbfdM/z/vLelaJSfeq4qUp67v5QmSvmr7oPS4nytpO0kPlkK+1HbpfEnqmL5u31L5GpDxc5KekfSjiLi+JXM1MN+3bO+c/nussP1jJddzP1vsfJLeTrOMynosUtKt9syWyldXxvQz/LLtHdLnPZX8Xl6u0vyjCdqqiOBRoIeS6w6uVvIfe62S4Wt7FztXVr5x+ugvO9mPYcXOluYLSZuVXLfx4aPYubLytZP0mKRlktZLelfS3ZJ2Kna2WvIOSz/THYqdJc0zSdKK9LObp2SI8W7FzlVDTku6VNISJde7/FXJ0PxFz5bm+1r6/VJe7Cy15PuUpEfS78HVSgZpOLLYubLynaRkqOlKSW9KOqvIeer8Xpb0LUlvSdqgpEvj6FLJl/Udk/sYV0IZ/6qktXxdzmP/Esl3maQF6ffie0qKwYNK5fOrYdn5kr7Zkvka8BlepaTwW5/+354oaZeWzshj2344opR6uQAAAABA60UXQQAAAADIEwosAAAAAMgTCiwAAAAAyBMKLAAAAADIEwosAAAAAMgTCiwAAAAAyBMKLADYhtm+2PbTzVj/Qtt/bmaGObbHNWcbAACUCgosAGghtj9h+z7bS2yvs/227Qdtdyx2tqaKiJ9FxOHFzgEAQKmgwAKAlvOYpMWSdpVUIWlfSU9IcjFDAQCA/KHAAoAWYLuXksLq5ohYHYl3IuLmiNiYLjPS9nO2l9teafsvtnfK2sbttu+y/Tvbq2y/a/tY26Ns/9P2Wtt/tT0wa535tn9s+4W01WyK7b3ryNnF9tW259leYftx2zvXsfxWXQzT/V1o+5l0fzNtfzZrfgfb19helrbknV/DNvdP866wPdf2ubadzrvL9pO226WvD7C9xvaIhh4LAAAKiQILAFpARLwv6TVJt9r+lu3dMkVD9mKSLpY0SNIwSesk/T5nmf+UdL+knpJ+KmmCpEslHSWpX9Y2sp0i6ax0nYmSHrPdrZaot0r6pKTPSOov6SVJj9ju0PB3qxMknSmpu6SnJN2RNe8CSYdJ+qykHdP3OTQz0/buSlr6rpLUR9KXJX1P0nFZ72UHST+y3U/SPZLOioiZjcgHAEDBUGABQMs5QNIkSd+X9Iqkpbb/O1NoRcSMiPhrRGyMiNWSLpH0Gdtds7bxbEQ8GhHVku6U1FXSXWlrWKWSAiq3heq3ETE1IjZJulLSBiVFzlZs95Z0rKTTImJpuvwlkgZI+rdGvM/fRMRrEbFFScG2s+3u6bxvSboyIuZExAZJ5ykpCjNOlXRfRDwUEVsi4nVJN6TrKSLWKykyz1NSvD0REbc1IhsAAAVVVuwAALCtiIjlki6UdKHtLpL+S0kL1LuSfpd2B7xKSTFToY8Kj96S1qfPF2dtrzKtzT6cJqkyXTfb/Kx1wvZCJa1AuXZMf87IaVzrIGlwg95kTsas3BWSVqf7zc6z3vaynAwH2v5q1rR2kt7OWmeW7b9KOlzSVxqRCwCAgqMFCwCKICIqI+J2STMkjUon3yxpraQ9IqKbpM+l05s7CMawzJO0tWyIpHdqWG5B+nN4RGyf9egSEX9oZoaMd3PydJXUNyfD73L23y0ids9a5xtKujBOlPRb2+3zlA0AgGajwAKAFmC7h+2f2x6RDvRQZvtoSSMkPZ8u1k1Ji8+qtLvepXna/Qm290qvoxovqYukR3MXiohlSq5pusn2oDT39raPsl2epyx3SRpveyfbnSX9QlsXkDdJ+prtw7M+p91sfz7N86l0mW9I+raS1r2f5CkbAADNRoEFAC1jk5KWmgckrZD0nqQfSTojIu5Llzlb0v6S1igpuh7J075vkXSdpJWSxkr6cnqNV01OkjRb0iTbayW9KukYbX2dVHP8XMnQ9JMlzZO0UB+1nCkdrOIwJdepLZa0TNLtkvqk3Srvk/SriHg6vYbrGEln2f6PPOUDAKBZHJGv35kAgFJje76kH0VE7miEAACgAGjBAgAAAIA8ocACAAAAgDyhiyAAAAAA5AktWAAAAACQJxRYAAAAAJAnFFgAAAAAkCcUWAAAAACQJxRYAAAAAJAn/x/F6l1J7P6RnQAAAABJRU5ErkJggg==\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], + "source": [ + "cv_args = {\"test_size\": 1, \"train_size\": 4}\n", + "\n", + "plot_splits(X, y, groups, **cv_args)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Usage in CV" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The example below illustrates how we can use the time series splitter with scikit-learn, i.e., using `cross_val_score`:" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Split number: 1\n", + "Train true target: [1 0 1 0 1 0 0 1]\n", + "Train predicted target: [0 0 0 0 0 0 0 0]\n", + "Test true target: [1 1]\n", + "Test predicted target: [0 0]\n", + "Accuracy: 0.0\n", + "\n", + "Split number: 2\n", + "Train true target: [0 1 0 1 0 0 1 1 1]\n", + "Train predicted target: [1 1 1 1 1 1 1 1 1]\n", + "Test true target: [0 1]\n", + "Test predicted target: [1 1]\n", + "Accuracy: 0.5\n", + "\n", + "Split number: 3\n", + "Train true target: [0 0 1 1 1 0 1]\n", + "Train predicted target: [1 1 1 1 1 1 1]\n", + "Test true target: [1 0 0 0]\n", + "Test predicted target: [1 1 1 1]\n", + "Accuracy: 0.25\n", + "\n" + ] + } + ], + "source": [ + "cv = GroupTimeSeriesSplit(**cv_args)\n", + "clf = DummyClassifier(strategy=\"most_frequent\")\n", + "\n", + "scores = cross_val_score(clf, X, y, groups=groups, scoring=\"accuracy\", cv=cv)\n", + "print_cv_info(cv, X, y, groups, clf, scores)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Example 2 -- Multiple training groups (with number of splits specified)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now let's take a look at an example where we don't specify the number of training groups. Here, we will split the dataset with test size (2 groups) and a specified number of splits (3 groups), which is sufficient for calculating the training size automatically." + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1gAAAEYCAYAAABBWFftAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAA8d0lEQVR4nO3deXycZb3+8c+VpnvSfYcuLAWFQlsaULafHOCoIIvI4VQUtYjsmyyVRVRWBeFUDiAiRUAQECwgCMomFEEsx5ZCKYVCS1uWbpTuTWmb5vv743kCw5A9k8wkvd6v17wy8yz3c81MO5lv7vu5H0UEZmZmZmZm1nRF+Q5gZmZmZmbWVrjAMjMzMzMzyxEXWGZmZmZmZjniAsvMzMzMzCxHXGCZmZmZmZnliAssMzMzMzOzHHGBZWZtnqSvS3pC0oeSNkp6X9IfJe2d72xVJA2TFPW4VW13Wgtm6yrpMkmzJa2XtETSs5KOy+Ex5ku6JlftZbR7qKR/SlopabWk1yTdJKmkge2MS1/3kvRx1ftwSMY2P5K0X26fgZmZtTbF+Q5gZtacJP0KOAO4A/gN8CEwFPgm8Lyk7SNibh4jVlkE7JnxeFvgLuBU4KVqtpvXctG4HxgNXA7MBPoB/w84GPhdjo5xBMl7kzOSjgbuBn5Lkj2AXYHvAT2AtU1ovup9eCNj2Y+AG4DJTWjXzMxaOflCw2bWVkk6HPgzcGxE3F7N+kOBaRGxsJp1nSNifbOHrIGkEcCrwH9ExOQ85hgOvAn8d0T8KWudoom/RJrzdZb0T2BlRHytmnUNyi5pHHAbUBoR1RZmkpYBN0TExY1LbGZmbYGHCJpZW/ZD4N/VFVcAEfGXquIqHe51tqRrJX1AUtwgqY+k36fDC8slTZZUltlOdUP2JF2cfuGuelw1xGx3Sc+lQ+3elHREQ59U9vHSTJMkHStpnqS1ku6U1FHSHpL+L102WdKQrLY6SfqlpHclbZD0iqSDMzbpkf5cXM3r96kCRdKQdOjl8vS1elzSjhnrq4bVfVvSHZJWAn9J131miKCkfdKhiOXp6z9RUmnG+h6SbpG0UNJHkt6RNDEr+2dyZ2fPeO//N82+UtL1kjpUt2/WczmkKj/QG/hZxnDO/Wra38zM2i4XWGbWJkkqJhnC9UQDdhsPDAS+QzKsEJIesK8A5wJjST43n5G0fSOj3Qs8BHyDpIj7k6SRjWwr0xdJhr6dTjJU7b+B64GJwP8Cx5AMO7w5a79JwDjg58ChwL+BhyWNStfPBtYB10r6sqRO1R1cUi/geWBH4KT0+F2BpyR1ztr8GmANcFR63Ora2xv4O0mB9F8kxfLBJL1IVSYA+wBnkbxHF5IMA6zyEnC0pNMkDaruOBnOAbYGvk0ynPAE4Io69sl0BLCKZMjknuntpVr3MDOzNsnnYJlZW9Ub6Ai8m7lQkoB2GYs2Z/RmLI6IsRnbfhXYG9gvIp5Nlz0NzCcpxk5sRK5bIuKatK3HgVnABSTnhDVFCXB4RKxK294POB74UkT8I102CPi1pC4RUS7pAOBrZDw/4AlJOwA/Bo6KiNWSjicp1B4HNkmaAtyZPpeq1+4skoJqVEQsT4/3T5LX6vvArzOyTomIU+t4PlcCL2S9H+8Df5c0IiJmAnsAv46IezP2+0PG/QuBXUgKzeslzSMpmH8ZEdk9W2vS51sJ/E1SR+DHkn5R9XxqExHTJVUA70XElLq2NzOztss9WGbWVin9mX2ezTnApoxb5hf9R7O23QP4IKP4ICLWAY+Q9Jw0xoMZbVWS9Gbt0ci2Mk2tKq5Sc4CNJL1KmcsAqnpzDiTpIfqnpOKqG0nP0cfDICPiHpKJQb4P/BHYgaQn7O6Mtg8EngRWZ7SzBpiW2VYq+3X+FEldSHqA7svK9TzJezYm3fRlYLykU9Ki8FMi4t102wOB/wGWkxSCMyRtnbX5Q+n7UeUBoDMworasZmZm2VxgmVlbtQzYQDLsK9OdwO7pLduSrMcDq1lWtV2vRuZaWs3jgY1sK9PKrMcbgTVZRcPG9GfVML8+wAA+XXBuAi4GBmc2FhEfRsRtEfHddN1twDczhjf2IRlCmd3Wf2S3RfWvaaaeJL2MN2a1tQFon9HeaSQ9Uj8FZkt6S9KnegIjYnNE/D0izo2IMpKhhL1ICu1M1b0vkJv3xszMtiAeImhmbVJEVEj6F/Blki/gVcuXkH7BT0YLfnq3rMeLSKYkz9afpDekygYge0KEmgqwfnx6OvJ+6XHyYTnwPvD1huwUEZuUTH9/LPA54JW0rYeBy6rZZU12E3UcYmW6zcXAX6tZvzDNsZLkXLkzJO1Kcu7ZXZJmRMSsGrI/IemVNHem7Pe56nG+3hszM2ul3INlZm3ZtcAXJH2nkfu/CPST9P+qFqTD177Gp4fevQd8PmObImD/Gto8Imu7w4H/a2S+pvo7SQ/W2oiYmn1LM5ZWM0kFwPD055KMtnYGXqumrdkNCZUOw5wC7Fhdruqm1Y+IGSTnxRWRFk+SPlMcp5N0bM1ne9EOT9+PKt8A1pNc96u+NvJJ76CZmW2h3INlZm1WRDwk6Vrgdkn/QTIl+DKSCTD+M92sxovNRsTj6UQN90o6n6Tn6VySc3Ouztj0QeBUSdOBt4EfAN1qaPYHkjaSfHE/HtgeOLpxz7DJniSZuOJJSVcBr5HkHgV0iogLSGYFfFjSrcALQHm6/sck50BVFZoTSGYqfFrS9SQ9Y/2BLwHPp+dxNcSPSCa0qCSZ6XANMISkuP1xRLwp6XmS134mSY/X8SQzHlYVrI9LeoPkfX+XpJg8jWQI4m+zjldKMqPjRJJC8ack17Sqc4KLDG8AX5P0GMm/q9kRkd17Z2ZmbZwLLDNr0yLiLEn/AE4hmUK7FPgA+BdwcET8rY4mjiCZIOFakt6J/wP2j4g5GdtcQjKk7HKSXowbSL70f+raWKlvAr9Kt30PGBsR0xv15JooIkLSN0hm2/shSQGznKRwuj7dbC5wC8m5SyeTFJfvALcCV0VERdrWMklfJJna/Fck16BaRFKAzWhEtufTnsNLSM6bawcsAB7jk96nf5FMMT8M2AxMBw6KiPfS9b8keb2vInl/PiCZOn2fiMjuNfwfkmns7yHpBbslfV0aYjzJbImPAl1Izj+b3MA2zMyslVMDLmRvZmaNJGkcycQQpRFRY6+ZtTxJAZweETfkO4uZmbV+PgfLzMzMzMwsR1xgmZmZmZmZ5YiHCJqZmZmZmeWIe7DMzMzMzMxyxAWWmZmZmZlZjrjAMrOCIWmUpCmS1km6LWtdSNovvd9B0sWSRjVDhmZpW9JkSZNy3OZ+6Qx4Dd3vQEn3SlogqVzSTEmnSWrXiLbGSZrf0P1aQvZrnr6vyzIe75Au69HAdhv1nCX9l6QXJH0o6SNJsyVdJKlD1nYPpv8HXpS0e0OPY2Zm+eUCy8wKyeXAVsC30/s16QD8jOSCt7nWXG2fAlyQ4zYb6wSgK3ARcDDwR5LrQP0yn6FaQNX1vKrsQPJe92ih4/cGniG5EPVBJNcS+zHJRZoznQd8i+TaXVe1UDYzM8sRX2jYzArJDsAjEfHnfAfJtYiYle8MGU6JiGUZjydL6gKcJenCiNiQr2DNKb0A8Xt1bth8x/9t1qJnJHUDTpV0eqSzTkXEm8CbkvYlKbTMzKwVcQ+WmRWSDsC6emy3Jv15Wzp0MCQNA5DUSdIvJb0raYOkVyQdnLmzpMMkTUuHYa1Ih2J9qa62ayNpZ0mPSVqetvu6pFMz1mcPV4sabuMyttlH0rPpML4PJU2UVFqP16dWWcVVlelAJ6BbU9pOh8+FpF0kPZm+Fm9I+kYD2mgv6RpJ76Tv4cJ02FyHrGPsLuk5SeslvSnpiDra/XiIYDrc9C/pqnlpe/PTdT0k3ZIe96M0x8RGvBz18SHJv/vqrK1lnZmZFSgXWGbWKkSEImJy+nD/9OflwJ7pbVG6bBIwDvg5cCjwb+DhqnOqJG2XbvN0uv7bwCNAr3q0XZuHgc3AMcBhwPVAbcXQnlm364EA5qQ59wb+DiwG/gv4Iclwvo/PTYuIyRGhzEYlzZd0ez3yZtsLWBYRHzRkp4i4PSKGVbPqbpLX5AjgLeCPkrauZ7MXkLwvPwH+k+S5rwKyzxG7F3gI+AbwKvAnSSPreYyXgHPT+98geQ+qCrQJwD7AWSRDCi8keW+A6p9zWqBdXJ8DS2onqYukfYAzgN+Er5liZtZmeIigmRWSHtSvB+vf6c+5ETGlaqGkA4CvAftFxLPp4ick7UByrstRwGhgTUSMz2jvr3W1XRtJfYBtga9HxKvp4r/Xtk9W7t2A44FLIuL5dPGVwAsRMTZju/eBv0saEREza2i6gqTQqzdJOwEnkdvzfX4VEbem7U8DlgCHADfVY989gLsj4vcZy+6rZrtbIuKa9BiPA7NIirNv1nWAiFgtaXb6cHpEzM86/q8j4t6MZX+oo8nNQGVdx02tAzqm9+8AxtewXTlQKkkuwMzMWg/3YJlZ3knqKun7QHdgWhOaOpCkx+efkoqrbiTFTlm6zatAd0m/l/RlSV2bFD6xHHgXuEnSWEn96rujpL7Ag8BTwKXpsi4kPSr3ZT2P54FNwJia2ouI7SPiuAYcvydwPzCDpNcvV57IyPQhsBSobw/Wy8A4ST+StKsk1bDdgxnHqCTpzdqjcXE/c/zxkk5Ji/M6RURxRFxaz/b3AvYFzgEOB26oYbupJEMET87Rv1MzM2sBLrDMLK/S85vWAr8Dro+Ih5vQXB9gAEkRknm7GBgMEBGzSb7UbkvSc7VM0t1podMo6Zf7L5MUd7cCi9Nzg0bXtl9aNN0HbASOyeil6EkyHO7GrOexAWhf9VyaSlInkqKkI3BYRGzMRbuplVmPN5Kc41UflwO/Jpl58RXgXUlnVrPd0moeD2xAxpqcBvwZ+CkwW9JbkursFauviHgpIp6PiAkkQwRPToeuZm/3NEnR+2tgbX3OBTQzs/xzgWVm+baQ5K/51wOnSPpCE9paDrwP7F7N7YtVG0XEoxGxL8m02ceR9Hxd34TjEhFvRMSRJMMcDyQpJh6VVNvn7DVptiMiYlXG8pUk5/z8rIbncmtTskJyHhDJeVI7AwdFxJKmtpkrEfFRRPw0Pc9pB5Jzra6V9NWsTbN7CvtRv/Pl6jr+yog4IyIGACOBF4G70qGUufZS+nOb7BWSdiEZPngTyf+Rhc1wfDMzyzEXWGaWVxGxMT3v6GySHpov12O3qp6W7B6Rv5P0YK2NiKnZt2qOvSoi7iYZalb15bmmtuslIjalPQ8TSHpTelS3naTvAGcCx2WfTxUR64ApwI7VPY+IyMUX7RuBrwKHpr16BSki3iKZjGIDn7xHVT6eNTAtZA8H/q8Bzdf5XkfEDJIipwj4XAParq+905/zqll3YHrcH6Y9XrnsYTQzs2biSS7MrCBEREU6hXad04RHxEZJ84D/ljQT+IjkHKIngceBJyVdBbyWtjcK6BQRF0g6keT8psdIegSGk0x+cUdtbdf25VbSriS9UfcCb5MM8TsPeCUillez/XbAzcDfgAWSvpixem46k9+PSCa0qCSZ9XANMIRkEo8fp9dKqi7LHODZ2s7DknQhycWGfwFUZh1/VkSsTre7nWTCkGE1tdUcJD1Ici7edGA9ySyKxcA/sjb9gaSNwEySSUK2B45uwKGqCssTJf0RKI+IVyU9T1J0zyTpSTyeZGKKGos3SRXApbWdhyXpMZJz7V4jmRRjb5LzsO6NiLnV7NINWNVWr0tmZtZWucAys0ISQE0TGmQ7iaSoeYrkHKJtImJ+er2lC0mm9h5CMmzwZT4ZAjiDZBr1CSRTsy8CJpKcb1Nj28D8WrIsJpkl78fAIJIhfs+QFFnVGUzSa3JQest0LHB7RDwv6f8BlwB3kpyTtYCkMKxtOF8xn53OPFtVL+EF6S3TfwCT0/td+Ox5Ti3hBWAsn/QczQKOrKYX8pvAr0jO2XoPGBsR0+t7kIhYIOlckvOgTk/bGAb8i2Sq/2EkhdB0kmGUtV2kuB11jwr5d0a7FSTF+AXUPLOiyJge3szMWgd55lczKxSS5gJ/jYjT853FQNIC4KdZ06XnnZKLMd8GlEbE2jzHaTaSfgEcm54LZmZmrYR7sMyskLwH7CNpELA4nZ3P8iB9D9oD9+Q7y5YmnYCkP0lvYm29ZmZWYKZNm9ahqKjo5Hbt2h0bEd2p/6gMax1C0qrNmzffVllZ+ZsxY8ZUe/qACywzKyQTgLtIZgL8PclwqrxLJ1CocfhXRFS0YJwWkU6kMSjX7abXtKptCGOlC2vuJ5mwo5xkyKiZtRLFxcUTu3XrtvegQYPWdejQ4cOaL+NnrVFEsHHjxvYLFy48ffXq1bsB36tuO88iaGYFIyIeIrmW1QiSa1cVilv57LW1Pr75+kQN8iVqeS359Llw1YqI2yNCbXh44Dkk0+f3joj78h3GzBpkn6FDh67q2LHjJhdXbY8kOnbsuGno0KGrgH1q2s49WGZWUCLiI5JZ1grJxcANtaz39YnqbxrJtbxqssW/ljXMKGhmrUO7oqIiT3DQxqXvcY2jMVxgVaNPnz4xbNiwfMcwswIxZsyYOrcpKytrgSStn19LM2tNpk2btiwi+uY7h7UuLrCqMWzYMKZO/cw1Sc3MzMxsC5LOpmrWIC6wzMzMzMya0c/v2TiyfEPzf+/u0pGKC4/u8EpzHyfbb37zm17XXnvtgNmzZ89qju1bG09yYWZmZmbWjFqiuGrIcbp06TK66lZcXLxbcXHxbpnLGnrck08+eXlDiqWGbt+SZs+e3UHSmLlz57ZvbBvuwTIzMzMz24KUl5dPr7o/duzYoRUVFbr//vvnV7fthg0b1LFjR0/c0QDuwTIzMzMzMwD22GOPHb///e8PPvDAA7crKSkZfckll/SfO3du+3333Xd4z549R5aWlo4aM2bMjs8991yXqn2uu+663kOGDBmR2cbxxx+/9Ve+8pXtunbtOnrw4MEj/vCHP/Ro7PaVlZWcf/75A/r3779r9+7dRx133HGD99xzzx3OPvvsaq/X+NFHH+noo48e2qtXr5ElJSWjhw0bNuK2227rWbX+scceKxkzZsyO3bt3HzV48OARP/vZz/pXViaXYCwrK9sJYMSIESO6dOkyevz48QMb+hq6wDIzMzMzs4/dd999fc4444ylq1evnn7BBRcsrays1EknnbT0nXfeeXXRokWv7LrrruVjx47dbsOGDTVe7GvSpEm9zznnnMWrV6+e/oMf/GDpySefPGzNmjU11h61bX/jjTf2njhxYv8HHnjgraVLl74ycODAjf/+979Lamrrhhtu6P3yyy93nTVr1sy1a9dOf+qpp2bvuuuu6wGmTp3a6cgjjxx+1llnLfnwww9ffvjhh+fccsst/W688cbe6fpZADNnzpxZXl4+/eqrr17U0NfPBZaZmZmZmX3soIMOWnHYYYetKSoqorS0tHL48OEbv/3tb68qLS2tLCkpiQkTJry/aNGiDjNnzuxYUxuHHHLIii9/+cvr2rVrx1lnnbVs7dq17Rq7/d133937O9/5zgd77733+o4dO8all166pF+/fptqaqtDhw5RXl5e9PLLL3fetGkT22+//aYxY8Z8BHDdddf1O/jgg1ccc8wxK4uLixk9evRHP/jBD5befffdvZvymmXyOVhmZmZmZvaxoUOHbsh8vGjRouJTTjll6ylTppSuWbOmWFIALF68uMZaYuDAgR8XQN26dasEWL16dY0X561t+8WLF7cfOnToxqr1RUVFDBw4cONnW0mcfPLJHy5ZsqT9ueeeO3jBggUd99xzzzUTJkx4b8SIERveeeedDlOmTOlWWlrao2r7iNCAAQNqbK+h3INlZmZmZmYfKyr6dIlw1llnbbVkyZL2U6ZMeWPt2rXT33nnnRmQFCYtkWfAgAGbFixY0KHqcWVlJYsWLepQ0/bt27fniiuuWDxz5szX582b92rnzp0rx40bNwxg66233njUUUctW7NmzctVt7Vr106fM2fOa/DZ594YLrDMzMzMzKxGa9asade5c+fKvn37Vqxataro9NNP37olj3/00Ud/+Ic//KHvv/71r84bNmzQJZdc0n/p0qU1TqP+8MMPlz733HNdNmzYoK5du1Z27dq1sri4OADOPPPMpX/5y1963X333d03bNigTZs2MW3atE6PPvpoCSQ9aUVFRcyaNatTY/O6wDIzMzMza0ZdOlLRmo9z+eWXL/zwww/b9+7de9TOO++881577bW2XbsaR/vl3Kmnnvrhscceu/Twww8f3q9fv5Hvvfdeh5EjR67r2LFjZXXbL1q0qP24ceO26dmz56gBAwaMfPfddztMnDhxAcDuu+/+0aRJk966/vrr+w8YMGDXPn36jBo3btw2VQVbSUlJjB8//v1jjz1229LS0lHnnXfegIbmVYSntc9WVlYWU6dOzXcMMzMzM8sjSdMioqy+27/yyivzR44cuaw5Mxls3ryZQYMG7XrJJZe8d9JJJy3PR4ZXXnmlz8iRI4dVt849WGZmZmZmVtAmTpzYs7y8XGvWrCk6++yzB61fv77oyCOPXJXvXNVxgWVmZmZmZgXtpptu6te/f/+RgwYN2vW5557r9sADD7zVt2/fzfnOVR1P025mZmZmZgVt2rRps/Odob7cg2VmZmZmZpYjLrDMzMzMzMxyxAWWmZmZmZlZjrjAMjMzMzMzyxEXWGZmZmZmZjniWQTNzMzMzJrRygnnjoz165r9e7c6d63ocfY1rzT3cRpi++233/m8885bePzxx6/Id5aW4gLLzMzMzKwZtURx1ZDjdOnSZXTV/Y0bNwqgQ4cOUbWsvLx8ekOPPXv27A6f+9zndpkzZ86M7bbbblPV8jlz5rzW0LZaypFHHjmsuLg47r333gW5bNcFlpmZmZnZFiSzgBo7duzQiooK3X///fPzGKlN8TlYZmZmZmYGwFtvvdXhq1/96rZ9+/bdtW/fvrseffTRQ1esWFEEUFlZyemnn75Vv379du3atevorbbaapcrrriiH0BZWdlOACNGjBjRpUuX0ePHjx8IsNVWW+1y44039gJ45JFHSouLi8dMnDix5+DBg0eUlpaOOvjgg7etah9gxowZHXffffcdS0pKRu+44447XXbZZf0kjakp75///OfSz3/+8zuVlJSM7tmz58i99tprh6p1a9asKTrhhBO23mqrrXbp3r37qH333Xf4zJkzOwJcdNFF/R966KFe999/f+8uXbqM7tKly+iKioqcvIYusMzMzMzMjPLych1wwAE7fO5zn/to3rx5r86aNeu1hQsXtj/hhBOGAPz5z3/udt999/V+4YUXXl+3bt30KVOmvP6lL31pLcDUqVNnAcycOXNmeXn59KuvvnpRdcfYvHkzTz75ZLeZM2fOmjVr1szXXnuty5VXXtkPYNOmTRx22GHDd9555/LFixe//OCDD8654447+taW+YQTTtjmpJNOWrJ69erpCxcunHHBBRd8fNxvfetbQ996661OU6ZMeX3JkiWvlJWVrTv00EO337Bhgy6//PIlhx9++PIjjzzyw/Ly8unl5eXTi4tzM7jPQwRz5LJ7Glfx/uTolnkLGpsPCj+j8yUKPR/A6gk/bNR+3c6+Nqc5auJ8TdPYfFD4GVsqn5lZPt177709IoJrr712IUBJScnmyy+/fOEBBxzwuYqKivkdOnSIjRs36uWXX+48aNCgisGDB1cMHjy4wV9Arr322ve7d+9e2b1798qvfvWrK1966aWuAE8//XTJwoULO9xwww3vlZSUxE477bTxlFNOWXL22WcPramt9u3bx9y5czu+9957xUOGDKk49NBD1wAsWrSo+JFHHun15ptvvlqV8Zprrll4yy239Js8eXLXr3zlK2sb9yrVrc33YElqJ+lqSR9IWiPpfkl98p3LzMzMzKyQvP322x0WLVrUobS0dFTV7eCDD95BEu+++277Qw45ZM2Pf/zj93/xi18M7Nev36h99tln+D/+8Y8uDTlGu3btGDRo0MdFWdeuXSvXrVvXDuCdd95p36tXr4qSkpKPJ9zYZpttNtTW3qRJk+bMnTu308iRI3febrvtdr700kv7Abz55psdAHbbbbedqp5Ljx49RlVUVGj+/PkdGpK5obaEHqzzgcOBLwAfArcCdwIH5TOUmZmZmVkhGTp06MZhw4ZtqG3mv3PPPXfZueeeu2zNmjVF48ePHzR27NjtFi1a9GpRUdP7bYYMGbJpxYoVxWvXrlVVkVVXMbTnnnuuf/TRR9+urKzkiSeeKPn617++w6hRo9aPGTNmPcDs2bNnZhZ0mXKRudp2m6XVwnICcFVEvB0Rq4AfAV+VNCy/sczMzMzMCsfYsWNXVVRU6Pzzzx+wYsWKosrKSubNm9f+jjvu6AHw7LPPdnn88cdL1q9fr86dO1eWlpZubteuHQADBw7cVFRUxKxZszo19vj777//2oEDB24844wzti4vL9cbb7zR4cYbb+xf0/YfffSRrr/++t6LFi0qLioqonfv3hWSon379rHVVltVHHroocuPO+64IfPmzWsPsGzZsnZ33HFHj1WrVhUB9O/ff9OCBQs6bt68ubGRq9WmCyxJ3YEhwLSqZRExF1gN7JqvXGZmZma25VDnrrmZnq6Zj1NaWlr51FNPzX799dc777jjjiO6des2ev/9999h+vTpXQBWrVrV7qyzzhrcp0+fUT179hz19NNPd7vzzjvnApSUlMT48ePfP/bYY7ctLS0ddd555w1o6PHbt2/Pgw8+OGfGjBld+vbtO+rwww/ffuzYsR+2b98+atpn0qRJPXfaaaedu3TpMvqII44YPn78+IUHHXTQWoC77rprwfDhwz/ab7/9duzatevoESNG7PynP/2ppyQATjnllA/Ky8uLevbsOaq0tHRUrmYRbOtDBLulP1dlLV+ZsQ4ASSeQ9HYxZMiQZg9mZmZmZluGHmdf80q+M9Qk+yK722+//aaHHnpoXnXbHnbYYWsOO+yw12tq68orr1x85ZVXLs5c9v77779adf+QQw5ZU1FRMS1z/YQJExZmPh49evRHU6dOnV31+Oqrr+4zaNCgjdUdr1OnTvHss8/OqSlPaWlp5XXXXbfwuuuuW1jd+p122mnjjBkz3qhp/8Zq0z1YwJr0Z/es5T1IerE+FhE3R0RZRJT17VvrbJBmZmZmZtYMnnjiia6vvfZax8rKSl588cXO11133YAjjjhieb5zNUSb7sGKiJWS3gF2A14GkLQtSe/VjDxGMzMzMzOzLPPmzev4ve99b9uVK1cW9+zZs+KQQw5ZccUVV1R7Ta1C1aYLrNTNwHmSniGZRfAq4PGImJ/XVGZmZmZm9iknnnji8hNPPLFV9Vhl2xIKrCuBnsC/gY7Ak8AxeU1kZmZmZmZtkiJqnJRji1VWVhZTp07NdwwzMzMzyyNJ0yKirL7bv/LKK2/vsssuK4qKivwFuw2rrKzUq6++2nPkyJHbVre+rU9yYWZmZmbWIiQtXr9+faOvA2Wtw/r16ztJWlzTehdYZmZmZmY5UFFRccn8+fM7rFu3rnNlZaXyncdyq7KyUuvWres8f/78DhUVFZfUtN2WcA6WmZmZmVmz22233R5/6aWXTps7d+7PImIA7sxoayolLa6oqLhkt912e7ymjVxgmZmZmZnlSPrFu8Yv39b2uao2MzMzMzPLERdYZmZmZmZmOeICy8zMzMzMLEdcYJmZmZmZmeWICywzMzMzM7MccYFlZmZmZmaWIy6wzMzMzMzMcsQFlpmZmZmZWY64wDIzMzMzM8sRF1hmZmZmZmY54gLLzMzMzMwsR1xgmZmZmZmZ5YgLLDMzMzMzsxxxgWVmZmZmZpYjLrDMzMzMzMxyxAWWmZmZmZlZjrjAMjMzMzMzyxEXWGZmZmZmZjniAsvMzMzMzCxHiuuzkSQB3wTKgNLMdRFxQjPkMjMzMzMza3Xq24P1G+AGYDDQPutmZmZmZmZm1LMHCzgK2CMi5jZnGDMzMzMzs9asvj1Y5cA7zRnEzMzMzMystatvgfVL4KfpuVhmZmZmZmZWjfoOETwDGAqcLmlp5oqI2CHnqczMzMzMzFqh+hZYlzdrCjMzMzMzszagXgVWRPy+uYOYmZmZmZm1dvXtwULS7sD3SaZqfxe4NSL+3VzBzMzMzMzMWpt6TXIh6evAP4DuwHSgG/CspCOaL5qZmZmZmVnrUt8erJ8BR0bEX6sWSDoIuBJ4sDmCWW5ddk9Fo/f9ydH17uhsksZmdL5EoeeDws/ofE2zesIPG71vt7OvzVmO2jQ2o/MlCj2fmVkhqO807cOAx7KWPU4ys6CZmZmZmZlR/wJrAXBg1rIDKPCLD0v6pqTnJK2W1PguHDMzMzMzs3qo77iRy4CHJE0C3ga2AY4EvtdcwXJkBXAj0Bm4Oc9ZzMzMzMysjavvNO33S1oIjAN2J5lF8MCI+FczZmuyiHgcQNJ++U1iZmZmZmZbgnqf+ZwWUwVdUDWFpBOAEwCGDBmS5zRmZmZmZtYa1VhgSfpCRLyY3t+rpu0i4oXmCNbSIuJm0mGEZWVlkec4ZmZmZmbWCtXWg/UUUJref76GbQJol9NEZmZmZmZmrVSNBVZElGbcr+9sg2ZmZmZmZlusehVOkr5Tw/Jv5zZObklqJ6kT0CF93Cm9Kc/RzMzMzMysDVJE3acbSVodEd2qWb48Ino1S7IckDQOuK2aVdtExPya9isrK4upU6c2VywzMzMzawUkTYuIsnznsNalvkP/PtPjI2kYUNAX742I2yNC1dzm5zubmZmZmZm1PbVO0y5pE+lEFpI2Zq1uR3IRXzMzMzMzM6Pu62AdSNJ79VfgoIzllcDiiHiruYKZmZmZmZm1NrUWWBHxLICk7SJiUctEMjMzMzMza53q6sECICIWSeoN7A70JeOcrIi4o5mymZmZmZmZtSr1KrAkHQjcD2wEegAr05/zABdYZmZmZmZm1H8WwSuBSyOiL7A2/XkZcFOzJTMzMzMzM2tl6ltgDQeuTe9XDQ+8CvhhjvOYmZmZmZm1WvUtsMqBjun9DyUNAToAPZsllZmZmZmZWStU3wLrBeDr6f2/Ag8DTwH/aoZMZmZmZmZmrVK9JrkAjuGTYmw8cA5QCkxojlBmZmZmZmatUZ0FlqRi4BqSooqI+Ai4oplzmZmZmZmZtTp1DhGMiArgm8CG5o9jZmZmZmbWetX3HKyHgCObM4iZmZmZmVlrV99zsDoAf5B0EjAfqKxaEREnNEMuMzMzMzOzVqe+BdYm4J70frv0ZmZmZmZmZhnqVWBFxLHNHcTMzMzMzKy1q+85WEjqLulbkn6UPh4gaVDzRTMzMzMzM2td6lVgSdoNmAOcD/wkXbwrcH0z5TIzMzMzM2t16tuD9b/AjyJiV6AiXfYC8MVmSWVmZmZmZtYK1bfA2hm4Pb0fABGxFujaDJnMzMzMzMxapfoWWB8AQzIXSNoeeD/niczMzMzMzFqp+hZYvwf+KGkfQJLGALcAE5stmZmZmZmZWStT3+tgXQWUAH9Nfz5Dcl6WJ7kwMzMzMzNL1fc6WJuBC4ELJfWJiGXNG8vMzMzMzKz1qe807Y9V3c8sriQ92hyhzMzMzMzMWqP6noO1Vw3LPU27mZmZmZlZqtYhgpK+VbWdpKMBZaweDqxormBmZmZmZmatTV3nYF2R/uwI/DxjeSWwGDi9OUKZmZmZmZm1RrUWWBGxDYCkhyPisJaJZGZmZmZm1jrV6xwsF1dmZmZmZmZ1q7EHS9J1EXFGev/mmraLiBOaI5iZmZmZmVlrU9sQwfY13DczMzMzM7Nq1FhgRcTJGfePbZk4ZtaWXXZPRaP2+8nR9bomepM5X9M0Nh8UfsaWyrd6wg8btV+3s6/NaY6atNV8UPgZnS9R6PnMoP7XwTIzMzMzM7M6tOkCS9JVkl6TtFrSQkkTJfXKdy4zMzMzM2ub2nSBBWwGjgF6AyOBrYHb8prIzMzMzMzarJYZVJ4nEXFhxsMPJN0A3J2vPGZmZmZm1rbV2oMl6WFJB0tSSwVqZgcAM6pbIekESVMlTf3ggw9aOJaZmZmZmbUFdQ0R/BC4D5gv6SeStmqBTM1C0pHA8cCZ1a2PiJsjoiwiyvr27duy4czMzMzMrE2otcBKp2cfBPwS+AYwT9KfJR3Umnq1JB0FTAQOi4iX8p3HzMzMzMzapjonuYiI1RHx64gYDezLJ71a8yRd1NwBm0rSscBvgUMj4pl85zEzMzMzs7arQbMIRsSLEXEcSaG1GbikWVLliKQzgGuAr0TEP/Odx8zMzMzM2jZFRP02lIqBI4ATgP8A/gn8NiIKdlY+SQFUABsyl0dESW37lZWVxdSpU5szmpmZmZkVOEnTIqIs3zmsdalzmnZJO5BMDvFdoB1wJ3B6RLzRzNmaLCJazXliZmZmZmbW+tVaYEl6FtgHeAE4B/hTRGyobR8zMzMzM7MtVV09WNOBkyLi9ZYIY2ZmZmZm1prVNcnFlcAu1a2QdJSkfrmPZGZmZmZm1jrVVWD9CBhew7rtgPNyG8fMzMzMzKz1qqvAOhi4pYZ1twKH5DaOmZmZmZlZ61VXgTUgIpZUtyIilgIDch/JzMzMzMysdaqrwNooaWB1K9Llm3IfyczMzMzMrHWqq8D6J3B6DetOBZ7LbRwzMzMzM7PWq65p2q8AnpPUF7gHeB/YCjga+DbJNbLMzMzMzMyMOgqsiJgq6TDg18BxQAAC5gCHRcRLzR/RzMzMzMysdairB4uIeBLYQdJwoC/wQUS81ezJzMzMzMzMWpk6C6wqaVHlwsrMzMzMzKwGdU1yYWZmZmZmZvXkAsvMzMzMzCxHXGCZmZmZmZnliAssMzMzMzOzHHGBZWZmZmZmliMusMzMzMzMzHLEBZaZmZmZmVmOuMAyMzMzMzPLERdYZmZmZmZmOeICy8zMzMzMLEdcYJmZmZmZmeWICywzMzMzM7MccYFlZmZmZmaWIy6wzMzMzMzMcsQFlpmZmZmZWY64wDIzMzMzM8sRF1hmZmZmZmY54gLLzMzMzMwsR1xgmZmZmZmZ5YgLLDMzMzMzsxwpzneAtmLY+Y82ar/5V34tx0mqt8vvd2n0vq9+79UcJqlZYzM6X6LQ8wH8+qSnG7XfqTftn+Mk1XO+pnnv/Ocave/WV+6bwyQ1a2zGlsr396e3a9R+B+w/N8dJqlfo+W57a2Gj9z12+KAcJqlZYzM6X6LQ85mBe7DMzMzMzMxyxgWWmZmZmZlZjhRkgSWpfb4zmJmZmZmZNVSLFViSBkj6i6RVkt6UdJykkDRM0u2S7pJ0m6TlwHXpPidLmp3uM0XSvhntXSzpqaxjTJZ0UXp/P0kVkr4naYGk5elxSlrqOZuZmZmZ2ZalJXuw7gI2AoOBfYDvZK0/CngM6AucI+lo4DLgu0BvYCLwmKShDThmO+BQYFfg88AOwP9Ut6GkEyRNlTT1gw8+aMAhzMzMzMzMEi1SYEnaGtgfGB8RqyNiKUnxlOn5iLg3IjZHRDlwLPDbiHgxIioi4nfADOBbDTz8eRGxKiKWAD8FvifpM887Im6OiLKIKOvbt2+Dn6OZmZmZmVlL9WBtlf58J2PZgqxt5mc9Hgy8nbVsbrq8ITKPMx/oCPRpYBtmZmZmZmZ1aqkC6/3055CMZUOytqnMevwusE3Wsm3T5QBrga5Z66u7yEHmkMJhwAZgWS1ZzczMzMzMGqVFCqyIeA+YDFwpqVRSP+CiOna7HThR0h6SiiWNA0YB96TrpwK7SRqTrj+NzxZkAL+Q1C095sXAnRGRXcyZmZmZmZk1mSKiZQ4kDSSZqGJfYAnwK+BGkl6nXwAVEfGDrH1OA84A+gOzgR9FxOSM9T8Hjk8f3pS2/VREXC5pP+Ap4DjgUqAb8Bfg1IhYU1vWsrKymDp1ahOerZmZmZm1dpKmRURZvnNY69JiBdZnDix9BXgI6BzNEKKqwIqI4obu6wLLzMzMzFxgWWM0uPhoLEkjgQBeJRnKdzlwb3MUV2ZmZmZmZvnQktfB6gU8QDI5xfMkU66f2YLHNzMzMzMza1Yt1oMVEc8A27fg8SbTgs/PzMzMzMysJXuwzMzMzMzM2jQXWGZmZmZmZjniAsvMzMzMzCxHXGCZmZmZmZnliAssMzMzMzOzHHGBZWZmZmZmliMusMzMzMzMzHLEBZaZmZmZmVmOuMAyMzMzMzPLERdYZmZmZmZmOeICy8zMzMzMLEdcYJmZmZmZmeWICywzMzMzM7MccYFlZmZmZmaWIy6wzMzMzMzMcsQFlpmZmZmZWY64wDIzMzMzM8sRF1hmZmZmZmY54gLLzMzMzMwsR1xgmZmZmZmZ5YgLLDMzMzMzsxxRROQ7Q8GR9AGwIIdN9gGW5bC9XCv0fFD4GZ2v6Qo9o/M1TaHng8LP6HxNV+gZna9pmiPf0Ijom+M2rY1zgdUCJE2NiLJ856hJoeeDws/ofE1X6Bmdr2kKPR8Ufkbna7pCz+h8TVPo+WzL4SGCZmZmZmZmOeICy8zMzMzMLEdcYLWMm/MdoA6Fng8KP6PzNV2hZ3S+pin0fFD4GZ2v6Qo9o/M1TaHnsy2Ez8EyMzMzMzPLEfdgmZmZmZmZ5YgLLDMzMzMzsxxxgWVmZmZmZpYjLrCakaR2kq6W9IGkNZLul9Qn37mqSPqmpOckrZZUke882SRdJem1NN9CSRMl9cp3rkySrpA0L824VNIkSUPynSubpCJJL0gKSVvnOw+ApNslbZK0NuN2Sr5zVUfSgZKmpBmXSbox35kA0v8fma/f+vQ93i3f2apIGiDp3vRzcIWkpyWNzHeuKpJ6S/q9pMWSVkm6W1LPPOap9XNZ0nclzZVULulFSWMKJZ+kkZL+JmlR+u9wn5bMVs+M300/C1ek/5f/JmmXAsp3lKSZab4Vkp6X9KVCyZe13VXp+3xMS+ZLj13bazhOUmXWZ+M9LZ3RtmwusJrX+cDhwBeAqi+1d+YvzmesAG4EfpjnHDXZDBwD9AZGkryGt+U10WfdCYyKiG7AMOAd4I95TVS9s4DyfIeoxu8joiTjVhCFSyZJ+wGTgGtI/i1uDdySx0gfi4idM18/YAIwKyJeyne2DDcCvYAdgf7AVOARScprqk/cAZQAw4FtSN7jfH5O1/i5nBYsvwFOBnoC9wN/ldStEPIBG4EHgMNaME91astYCvyM5P/xVsBLwBOSOrdYutrzTQH+MyJ6kvxbvI7kPe7RYunq8d1A0h7AQcCiFsqUra6Mb2f9bjm65aKZQXG+A7RxJwCXRsTbAJJ+BMyRNCwi5uc1GRARj8PHXyALTkRcmPHwA0k3AHfnK091IuKNjIcCKkm+SBYMSTsApwBHAtPzHKc1+gVwU0RMylhWSAUMAJKKge+T5C0k2wM3RMRyAEm/A8aTfHlcls9gkrqSfEkcHRFr0mU/ByZLGhoRC1o6Ux2fy8cDD0TEE+k2VwOnAUcAv893voh4HXg9Xd8ScapVR8ZfZz5O3+8Lgc/RQp+PdeR7NzMeyR8auwCDgZXNn67u7waSOgK/I/mOk5eeoUL//mLmHqxmIqk7MASYVrUsIuYCq4Fd85WrlTsAmJHvENkkfUvSKmAtcCZwcX4TfUJSEXAryRfalflNU60jJS2X9KaS4bQl+Q6UKf0CvgfwkaSX0iFFkyWV5TtbNb4OdCfpkSkkV5O8z30kdSL5UvZ8ROS1uEop41al6vdiwQxjzDCST/9OCZKioBCzthYHkPTuz8l3kCqShkhaSdIjOAm4NyJezW+qT7kYeDoi/pXvILUYnA77fVfSHyVtk+9AtmVxgdV8qoZsrMpavjJjndWTpCNJ/np7Zr6zZIuIuyOiOzCQ5BdPIf0iPBNYHBEP5DtINa4n+atxH5K/wH8JmJjXRJ/Vk+Rz8nhgHDAIeIKWH7JTHyeSfBFbme8gWf4JtAM+IPkjxDdIXs+8i4i1wGTgYkk9JPUl6c2AwvycLsW/U3Im7d2/BTinqgezEETEOxHRg+R9PRZ4Jr+JPpH+ceko4Mf5zlKLfwC7kHxe7w58BDyZ/sHMrEW4wGo+VR/W3bOW9yDpxbJ6knQUyRfvwwrs3JJPiYjFJDkfUQFMxiFpe+AckiFEBScipkXEkoiojIjXSM4T+690+EmhqPp/fFtEzIiIjSRD8NoDe+Uv1qdJ2o7kL/E35TtLprQH9SngTZLPwi7AFcBzkvrnM1uGY4ANJEPb/g94KF1eCD1s2dbg3yk5IWknksLlmogoqP83VSJiXUTcDpwp6Sv5ziOpA8l50Kemf5woSBHxdkS8mf5uWUzyB51BwBfzHM22IC6wmkn6V+R3gI9n85K0LclfpApumFuhknQs8Fvg0IgomL/i1aIY6EryYZ5v+wB9gZmSlvHJeUMzVJiz9VWmPwtl8gMiYhUwH4jqVrdsmlqdCLwSES/mO0iWXiQTR1wfEasjYmNE3ELyu6cgvuxExPsRMTYiBkbENsA8kr94T8lztOq8wqd/pwgYlS63elIyy+Zk4MqI+GWe49RHMckkLPk2CNgZuCsdLr2M5Nyw30i6K7/RahXprWB+t1jb5wKred0MnCdpm3SWp6uAxwthggv4eBr5TkCH9HGn9FYQH0KSziCZue0rEfHPfOfJpmTq89Mk9Usfbw38muQL+Ru17dtC7gO2I/kCNgo4OF3+ZQrgPJ10mt0e6f3hwP8AD0fER3kN9lk3AsdK2imdSGI8yRfwF/IbK5H+VXkcBdZ7BZCeZ/UmcIqkrpKKJX2fZKhbQQyllbSjpF7p/+fdgWtJvnivzFOe2j6XJwLfkHRA+r6fA3QCHiyEfKlO6XqADunjdi2Vrx4Z9wb+DlwUEde3ZK565vuupO3Tf4+lkn5Kcj730/nOB7ybZhmVcVtIMqz2jJbKV1vG9DX8mqSt0/u9SH4vL6Mw/2hibVVE+NZMN5LzDq4h+Y+9hmT62j75zpWRbxyf/GUn8zYs39nSfAFsIjlv4+NbvnNl5CsC/gosBdYB7wN3AdvlO1sNeYelr+nW+c6S5pkMLE9fu3kkU4x3y3euanIKuBRYTHK+yzMkU/PnPVua75vp50tJvrPUkO/zwCPp5+AqkkkaDs93rox8x5NMNV0OvAWcmec8tX4uA98F3gbWkwxpHFMo+TI+Y7Jv4woo4zMkveVrs277Fki+y4EF6efiByTF4AGF8vpVs+184JiWzFeP1/BqksJvXfp/exKwQ0tn9G3LvimikEa5mJmZmZmZtV4eImhmZmZmZpYjLrDMzMzMzMxyxAWWmZmZmZlZjrjAMjMzMzMzyxEXWGZmZmZmZjniAsvMzMzMzCxHXGCZmW3BJF0s6akm7H+hpL80McMcSeOa0oaZmVmhcIFlZtZCJG0r6U+SFktaK+ldSQ9K6pDvbI0VET+PiEPzncPMzKxQuMAyM2s5fwUWATsCpcCewOOA8hnKzMzMcscFlplZC5DUm6SwuikiVkXivYi4KSI2pNuMlPSspGWSVkj6m6TtMtq4XdKdkm6VtFLS+5KOljRK0r8lrZH0jKRBGfvMl/RTSc+nvWZTJe1eS84ukq6RNE/SckmPSdq+lu0/NcQwPd6Fkv6eHm+mpL0y1reXNEHS0rQn77xq2tw3zbtc0lxJ50hSuu5OSU9IKkof7ydptaQR9X0vzMzMmpMLLDOzFhARHwKvAbdI+q6knaqKhszNgIuBrYBhwFrgD1nb/BdwP9ALuAyYCFwKHAH0z2gj00nAmek+k4C/SupWQ9RbgM8BXwQGAC8Cj0hqX/9ny/eBM4DuwJPA7zPWnQ8cAuwFbJM+z6FVKyXtTNLTdzXQF/gacBrwnYznsjVwkaT+wN3AmRExswH5zMzMmo0LLDOzlrMfMBn4IfAysETST6oKrYiYERHPRMSGiFgFXAJ8UVLXjDaejohHI6ISuAPoCtyZ9oaVkxRQ2T1Uv4uIaRGxEbgKWE9S5HyKpD7A0cApEbEk3f4SYCDwhQY8z99GxGsRsZmkYNteUvd03XeBqyJiTkSsB84lKQqrnAz8KSIeiojNEfEGcEO6HxGxjqTIPJekeHs8Im5rQDYzM7NmVZzvAGZmW4qIWAZcCFwoqQvw3yQ9UO8Dt6bDAa8mKWZK+aTw6AOsS+8vymivPK3NPl4GlKf7ZpqfsU9IeoekFyjbNunPGVmda+2BwfV6klkZM3KXAqvS42bmWSdpaVaG/SV9I2NZEfBuxj6zJD0DHAp8vQG5zMzMmp17sMzM8iAiyiPidmAGMCpdfBOwBtg1IroBe6fLmzoJxrCqO2lv2RDgvWq2W5D+HB4RPTJuXSLiniZmqPJ+Vp6uQL+sDLdmHb9bROycsc+3SYYwTgJ+J6ldjrKZmZk1mQssM7MWIKmnpF9IGpFO9FAs6UhgBPBculk3kh6flelwvUtzdPjvS9otPY9qPNAFeDR7o4hYSnJO042Stkpz95B0hKSSHGW5ExgvaTtJnYFf8ukC8kbgm5IOzXiddpL0pTTP59Ntvg18j6R372c5ymZmZtZkLrDMzFrGRpKemgeA5cAHwEXA6RHxp3Sbs4B9gdUkRdcjOTr2zcB1wApgLPC19Byv6hwPzAYmS1oDvAocxafPk2qKX5BMTT8FmAe8wyc9Z6STVRxCcp7aImApcDvQNx1W+SfgVxHxVHoO11HAmZL+M0f5zMzMmkQRufqdaWZmhUbSfOCiiMiejdDMzMyagXuwzMzMzMzMcsQFlpmZmZmZWY54iKCZmZmZmVmOuAfLzMzMzMwsR1xgmZmZmZmZ5YgLLDMzMzMzsxxxgWVmZmZmZpYjLrDMzMzMzMxy5P8D5ELV5xIwI3MAAAAASUVORK5CYII=\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Train indices: [0 1 2 3 4]\n", + "Test indices: [5 6 7 8 9]\n", + "Train length: 5\n", + "Test length: 5\n", + "Train groups: [0 1 1 1 1]\n", + "Test groups: [2 2 2 3 3]\n", + "Train group size: 2\n", + "Test group size: 2\n", + "Train group months: ['2021-01' '2021-02' '2021-02' '2021-02' '2021-02']\n", + "Test group months: ['2021-03' '2021-03' '2021-03' '2021-04' '2021-04']\n", + "\n", + "Train indices: [1 2 3 4 5 6 7]\n", + "Test indices: [ 8 9 10 11]\n", + "Train length: 7\n", + "Test length: 4\n", + "Train groups: [1 1 1 1 2 2 2]\n", + "Test groups: [3 3 4 4]\n", + "Train group size: 2\n", + "Test group size: 2\n", + "Train group months: ['2021-02' '2021-02' '2021-02' '2021-02' '2021-03' '2021-03' '2021-03']\n", + "Test group months: ['2021-04' '2021-04' '2021-05' '2021-05']\n", + "\n", + "Train indices: [5 6 7 8 9]\n", + "Test indices: [10 11 12 13 14 15]\n", + "Train length: 5\n", + "Test length: 6\n", + "Train groups: [2 2 2 3 3]\n", + "Test groups: [4 4 5 5 5 5]\n", + "Train group size: 2\n", + "Test group size: 2\n", + "Train group months: ['2021-03' '2021-03' '2021-03' '2021-04' '2021-04']\n", + "Test group months: ['2021-05' '2021-05' '2021-06' '2021-06' '2021-06' '2021-06']\n", + "\n" + ] + } + ], + "source": [ + "cv_args = {\"test_size\": 2, \"n_splits\": 3}\n", + "\n", + "plot_splits(X, y, groups, **cv_args)\n", + "print_split_info(X, y, groups, **cv_args)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Usage in CV" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Again, let's take a look at how this looks in a scikit-learn context using `cross_val_score`:" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Split number: 1\n", + "Train true target: [1 0 1 0 1]\n", + "Train predicted target: [1 1 1 1 1]\n", + "Test true target: [0 0 1 1 1]\n", + "Test predicted target: [1 1 1 1 1]\n", + "Accuracy: 0.6\n", + "\n", + "Split number: 2\n", + "Train true target: [0 1 0 1 0 0 1]\n", + "Train predicted target: [0 0 0 0 0 0 0]\n", + "Test true target: [1 1 0 1]\n", + "Test predicted target: [0 0 0 0]\n", + "Accuracy: 0.25\n", + "\n", + "Split number: 3\n", + "Train true target: [0 0 1 1 1]\n", + "Train predicted target: [1 1 1 1 1]\n", + "Test true target: [0 1 1 0 0 0]\n", + "Test predicted target: [1 1 1 1 1 1]\n", + "Accuracy: 0.33\n", + "\n" + ] + } + ], + "source": [ + "cv = GroupTimeSeriesSplit(**cv_args)\n", + "clf = DummyClassifier(strategy=\"most_frequent\")\n", + "\n", + "scores = cross_val_score(clf, X, y, groups=groups, scoring=\"accuracy\", cv=cv)\n", + "print_cv_info(cv, X, y, groups, clf, scores)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Example 3 -- Defining the gap size between training and test datasets" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "`GroupTimeSeriesSplit` let's you specify a gap size greater than 1 in order to skip a specified number of groups between training and test folds (the default gap size is 0). In the example below, we use a gap of 1 group to illustrate this." + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1gAAAEYCAYAAABBWFftAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAABAPElEQVR4nO3deZyd893/8dc7qyyTPZGEROwtIQnhLuVXxd2WWqrqTrXaUrW2qCWlullbyq1uVFW0lNLaqpTWvpRqVCISEYJIYskmZJ9IMpnP74/vdTiOWc7MnJkzM3k/H4/zmDnX+j7nzDlzfc73e30vRQRmZmZmZmbWdB3KHcDMzMzMzKy9cIFlZmZmZmZWIi6wzMzMzMzMSsQFlpmZmZmZWYm4wDIzMzMzMysRF1hmZmZmZmYl4gLLzNo9SV+S9KCkdyWtlfS2pD9L+nS5s+VIGiEpirjllvteC2brIel8STMlrZa0UNITko4u4T7mSLq0VNvL2+6Bkv4laamk5ZJelHSNpJ4N3M6R2fPeM7ufex0OyFvmB5L2Ku0jMDOztqZTuQOYmTUnSb8CTgZuBH4DvAtsBnwVeErSVhExq4wRc+YDu+Xd3wK4Gfgu8FwNy81uuWjcCYwBLgCmA4OA/wfsD/yuRPs4hPTalIykw4FbgN+SsgewI/AtoA+wsgmbz70OL+dN+wFwFfB4E7ZrZmZtnHyhYTNrryQdDPwVOCoibqhh/oHA5IiYV8O8bhGxutlD1kLSSOAF4LMR8XgZc2wNvAL8T0TcXjBP0cR/Is35PEv6F7A0Ir5Yw7wGZZd0JHA9UBERNRZmkhYDV0XEOY1LbGZm7YG7CJpZe/Z94NmaiiuAiPhbrrjKunudJulySe+QihskDZD0h6x7YaWkxyWNzd9OTV32JJ2THXDn7ue6mO0i6cmsq90rkg5p6IMq3F+W6Q5JR0maLWmlpJskdZW0q6T/ZNMelzS8YFsbSfqlpDclrZE0VdL+eYv0yX4uqOH5+0iBIml41vXyvey5ekDStnnzc93qvi7pRklLgb9l8z7WRVDSHllXxMrs+Z8gqSJvfh9J10maJ+l9SW9ImlCQ/WO5C7Pnvfb/l2VfKulKSV1qWrfgsRyQyw/0B36W151zr9rWNzOz9ssFlpm1S5I6kbpwPdiA1cYDQ4BvkLoVQmoB+zxwBjCO9Ln5mKStGhntVuBu4MukIu52SaMaua18nyJ1fTuJ1FXtf4ArgQnA/wFHkLodXluw3h3AkcDPgQOBZ4F7JI3O5s8EVgGXS/qcpI1q2rmkfsBTwLbA8dn+ewAPS+pWsPilwArgsGy/NW3v08AjpALpK6RieX9SK1LOZcAewKmk1+hsUjfAnOeAwyV9T9LQmvaT53RgU+DrpO6ExwIX1rNOvkOAZaQuk7tlt+fqXMPMzNoln4NlZu1Vf6Ar8Gb+REkCOuZNWp/XmrEgIsblLfsF4NPAXhHxRDbtUWAOqRg7rhG5rouIS7NtPQDMAH5IOiesKXoCB0fEsmzbewHHAJ+JiH9m04YCv5bUPSIqJe0DfJG8xwc8KGkb4EfAYRGxXNIxpELtAWCdpInATdljyT13p5IKqtER8V62v3+RnqtvA7/OyzoxIr5bz+O5CHi64PV4G3hE0siImA7sCvw6Im7NW++Peb+fDexAKjSvlDSbVDD/MiIKW7ZWZI+3GviHpK7AjyT9Ivd46hIRUyRVAW9FxMT6ljczs/bLLVhm1l4p+1l4ns3pwLq8W/6B/n0Fy+4KvJNXfBARq4B7SS0njXFX3raqSa1ZuzZyW/km5YqrzGvAWlKrUv40gFxrzr6kFqJ/SeqUu5Fajj7oBhkRfyINDPJt4M/ANqSWsFvytr0v8BCwPG87K4DJ+dvKFD7PHyGpO6kF6LaCXE+RXrOds0WfB8ZLOjErCj8iIt7Mlt0X+F/gPVIhOE3SpgWL3529Hjl/AboBI+vKamZmVsgFlpm1V4uBNaRuX/luAnbJboUWFtwfUsO03HL9GplrUQ33hzRyW/mWFtxfC6woKBrWZj9z3fwGAIP5aMG5DjgHGJa/sYh4NyKuj4hvZvOuB76a171xAKkLZeG2Plu4LWp+TvP1JbUyXl2wrTVA57ztfY/UIvVTYKakVyV9pCUwItZHxCMRcUZEjCV1JexHKrTz1fS6QGleGzMz24C4i6CZtUsRUSXp38DnSAfguekLyQ7wU2/Bj65WcH8+aUjyQhuTWkNy1gCFAyLUVoAN4qPDkQ/K9lMO7wFvA19qyEoRsU5p+PujgE8AU7Nt3QOcX8MqKwo3Uc8ulmbLnAP8vYb587IcS0nnyp0saUfSuWc3S5oWETNqyf6gpKlZ7nyFr3PufrleGzMza6PcgmVm7dnlwH9J+kYj138GGCTp/+UmZN3XvshHu969BXwyb5kOwN61bPOQguUOBv7TyHxN9QipBWtlREwqvGUZK2oYpAJg6+znwrxtbQ+8WMO2ZjYkVNYNcyKwbU25ahpWPyKmkc6L60BWPEn6WHGcDdKxKR9vRTs4ez1yvgysJl33q1hr+bB10MzMNlBuwTKzdisi7pZ0OXCDpM+ShgRfTBoA47+zxWq92GxEPJAN1HCrpLNILU9nkM7NuSRv0buA70qaArwOfAfoVctmvyNpLenA/RhgK+Dwxj3CJnuINHDFQ5IuBl4k5R4NbBQRPySNCniPpN8DTwOV2fwfkc6ByhWal5FGKnxU0pWklrGNgc8AT2XncTXED0gDWlSTRjpcAQwnFbc/iohXJD1Feu6nk1q8jiGNeJgrWB+Q9DLpdX+TVEx+j9QF8bcF+6sgjeg4gVQo/pR0Tat6B7jI8zLwRUn3k/6uZkZEYeudmZm1cy6wzKxdi4hTJf0TOJE0hHYF8A7wb2D/iPhHPZs4hDRAwuWk1on/AHtHxGt5y5xL6lJ2AakV4yrSQf9Hro2V+Srwq2zZt4BxETGlUQ+uiSIiJH2ZNNre90kFzHukwunKbLFZwHWkc5dOIBWXbwC/By6OiKpsW4slfYo0tPmvSNegmk8qwKY1IttTWcvhuaTz5joCc4H7+bD16d+kIeZHAOuBKcB+EfFWNv+XpOf7YtLr8w5p6PQ9IqKw1fB/ScPY/4nUCnZd9rw0xHjSaIn3Ad1J55893sBtmJlZG6cGXMjezMwaSdKRpIEhKiKi1lYza3mSAjgpIq4qdxYzM2v7fA6WmZmZmZlZibjAMjMzMzMzKxF3ETQzMzMzMysRt2CZmZmZmZmViAssMzMzMzOzEnGBZdbGSRotaaKkVZKuL5gXkvbKfu8i6RxJo5shQ7NsW9Ljku4o8Tb3ykaNa+h6AyVdIek/ktZKmtOEDEc2Zf3mVPicZ6/r4rz722TT+jRwu416zJK+IulpSe9Kel/STEk/ltSlEds6R9LjDV2vtZB0g6RJJd7miOxzYkQD1+si6RJJT0paXdt7StL3s/fLK9lImmZm7Z4LLLO27wJgE+Dr2e+16QL8jHSR2FJrrm2fCPywxNtsrE2AccAC0nWiNhS5a2DlbEN6rfu00P77A4+RLt68H+n6Wz8iXdh4Q3M+6bpfrUF30mtSSboAdW3+THrdZgNXS9qoBbKZmZWVLzRs1vZtA9wbEX8td5BSi4gZ5c6QZ1pEbAwg6VLgK2XO0yKyi/a+Ve+Czbf/3xZMekxSL+C7kk6KDWikpoiYVe4MORGxVFK/7GLV3wP2rmW5BcACSd2Az5EuZv1KC0Y1M2txbsEya/u6AKuKWG5F9vP6rEvQB92CJG0k6ZeS3pS0RtJUSfvnryzpIEmTs66ISyQ9I+kz9W27LpK2l3S/pPey7b4k6bt58wu7q0UttyPzltlD0hOSKrNuZRMkVRTx/NQpIqqbuo3aZN3nQtIOkh7KnouXJX25AdvoLOlSSW9kr+E8SXflutLl7WOXvG5dr0g6pJ7tftBFMOtu+rds1uxse3OyeX0kXZft9/0sx4RGPB3FeJf0d99kkg6T9Gr2fDwmaUwNf1PflPRU9ne6JFtubMF2bpA0SdKXstfu/Wyd7RqQZVNJt0lalOWZJen8wn3k3Z9Ty/vhnLxlRkq6T9KK7Ha7pMGNfb7yNbC4zV1cuySvm5lZa+YCy6wdiwhFxOPZ3dw3zBcAu2W3+dm0O0hdj34OHAg8C9yj7JwqSVtmyzyazf86cC/Qr4ht1+UeYD1wBHAQcCVQVzG0W8HtSiCA17KcnwYeIXXj+wrwfWB/4INz0yLi8YhQ/kazA9UbishbEhFxQ0SMqGHWLaTn5BDgVeDPkjYtcrM/JL0uPwH+m/TYlwEdC5a7Fbgb+DLwAnC7pFFF7uM54Izs9y+TXoNcgXYZsAdwKqlL4dmk1wao+TEXFgN1kdRRUndJewAnA79paOtVRJwTEXvlbXMsqQvbc9njuIf0/BQaAdwIHAZ8jdSi909JWxQstxnpeTg/W6438ICK7xZ3IzAMOJbUre5CoGsdyx/CR98PZ2fTX8ke31bAv4CNgG+Q3uPbA3+TJICImJN9TszJbTT7YuPxIjObmVkBdxE0a/v6UFwL1rPZz1kRMTE3UdI+wBeBvSLiiWzyg5K2IZ3rchgwBlgREePztvf3+rZdF0kDgC2AL0XEC9nkR+papyD3TsAxwLkR8VQ2+SLg6YgYl7fc28AjkkZGxPRaNl1FKvTK7VcR8XsASZOBhcABwDVFrLsrcEtE/CFv2m01LHddRFya7eMBYAapOPtqfTuIiOWSZmZ3p+QflGf7/3VE5Bcof6xnk+uBYlsGV/FhsXEjML6OZYt1JvAS8NWsWLtfUmfg4vyFIuK83O+SOgAPAbuQvhg4L2/RAcDBEfF0tuxkYBapsCn2NTw8InKthI/XtXBETMnLNRw4DbghIm7JJv+M9GXDfhGxNltuGvAy6YuH+2rZdHO8Fyqzn72bYdtmZq2KW7DM2ihJPSR9m3TAMrkJm9qXdBD2L0mdcjdSsZPrBvUC0FvSHyR9TlKPJoVP3gPeBK6RNE7SoGJXlDQQuAt4mOwAV1J30rf4txU8jqeAdcDOtW0vIraKiKMb/1BK5sHcLxHxLrAIKLYF63ngSEk/kLRjroWiBnfl7aOa1Jq1a+Pifmz/4yWdmBXn9YqITvnFSz12B/YETgcOBq5qVMqP2gX4W0FL2D2FC0n6ZNbdciGp+FgHbEs6/zHfolxxBRARc0nvzWKf3+eBX2TdOYcX+yCUzm+6C3gDOCFv1r7Z9Oq898NsYA4fvrc/JiL2iYh9it1/kV4iFcnHSepfx9+nmVmb5wLLrA1SOr9pJfA74MqI+NhBYQMMAAaTDhrzb+eQuisRETNJB7VbkFquFku6JSt0GiU7uP8cqbj7PelE+Ccljalrvewg8TZgLXBE3sFxX1J3uKsLHscaoHPusbRySwvuryV17yrGBcCvSSMvTgXelHRKDcstquH+kAZkrM33gL8CPwVmZuc11dsqVqyIeC4inoqIy0hdBE/Iuq42xWDgnYJpH7mvdP7eg6S/n9NIRd4upOe48LUpfG5z04p9fscBk4BfAXMlPZ+1MNfnWlL3xC9HxPt50weQWukK39tb0MLvh4hYQeo2+RVgMfCtlty/mVlLchdBs7ZpHulA73+AEyXdHBHPNHJb7wFvA1+qa6GIuA+4T1JvUpfCy0nnQDX6IDoiXgYOzbpl7UnqmnWfpE3rGFTiUtIB7qciYlne9KWkc37O4aPdF3PmNTZnW5AdWP8U+KmkrYHjgcslzYyI+/MWHUQaJCL/fjHny9W3/6WkwudkSTsCPwBuljStGUaDfC77uTmpC15jLQAKvyQovL8bqRXxv7O/VwCy90GhmlphBwEvFhMmIt4mtUJ2ILV6nUM6F3J41qL5MZK+DxwOfCFrMcv3HqkF67oaVl1cw7Rmk73HryIVkOcBU+pew8ys7XILllkbFBFrs/OOTiO10HyuiNXWZj8Lv3V/hPRN/sqImFR4q2Hfy7JzPO4CciOk1bbtokTEuoh4lDRAwBBqucaSpG8ApwBHF55PFRGrgInAtjU9joho1wVWvoh4lTQYxRo+fI1yPhg1MDuQPxj4TwM2X+9rHRHTSOdIdQA+0YBtF+vT2c/ZTdzOs8CBBd3VDipYplv2c01ugqTdSQNfFBqUzcstNxzYiYY9v0REdXa+4bmk601tVtNykj4LXAKcHREP17DII8BIYHIN74c5DclUAiNJrWbnRsTDtRWMZmbtgVuwzNqwiKhSGkK7VxHLrpU0G/gfSdOB94FppBP2HwAeknQx6dv2XqSLBm8UET+UdBzpm/z7SS1BW5MGv7ixrm3nTqyvSdbKcSlp1LbXSV38zgSmRsR7NSy/Jakr1D9I3ac+lTd7VkS8Q2o1eURSNWnUwxWk6+58EfhRRNR4/R1JrwFP1HcelqTcta+2Abrn3X8i2z/ZaIR71TJKYLORdBfpfJ8pwGpSV6xOwD8LFv2OpLXAdNIgIVuRWkCKlRvk4jhJfwYqI+IFSU+Riu7ppJbEY0jn3NRaXEiqAs6r6zwsSfeTzrV7kXT+06dJ52Hdmn9dqNyod/mjBBbhYuAZ0miN1wOfzHLDh4NvTCR1x50g6Zek1qxzSK2+hRYDN0n6Cek1OI/URfCG+oJkLWIPkN5Tr5AG9Did1Mr2Ui3L30Z6vv9Z8H54K9L1y84hPf/3Sfp9lm8T0iiTN+SNMFq47UcgnYtVT+b9gB5kFxjPez88W0NrWu4zqqZulGZm7YoLLLO2L4BiTxg/nlTUPEw6gNs8IuYoXW/pbNLQ3sNJXYueJ3UBhFSIHURqYepH6lI2gdQlrdZtk06mr80C0ih5PwKGkrr4PUYqsmoyjNRqsl92y3cU6YDxKUn/j/TN/02kc7LmkgrDhXVk6cTHhzOvye213P8sH4741p3yHEQ+TTqHJ9dyNAM4tIZWyK+SzvG5gDTc+Lj80ejqExFzJZ1B6g54UraNEcC/SaPljSAVQlNIo9fVdZHijtTfk+LZvO1WkYrxH/LxUfm6U/ff28dExCRJh5MuT3AwqfvaCaQvHZZnyyyUdBjpb/tu0vD5x5OK+UJzs21dRGp1mkQaFfD9GpYt9D5pMJlTSH/rlaTi7nMRsbqG5fuSzrEaQHru850LnBMRr2SF1wWkLye6kQrDR8gubVCLYt4LAL/ho61ruffDUXy8qMx9Rm0wF4Y2sw2XGngZETNrZSTNAv4eESeVO4uBpLnATwuGSy87pQvnXg9URMTKehZvUyR1JRVEn8u71EBjt3UEqTjfIiKK7oKYtVyOjIhaR+fbkEn6POmLjk/mn8tmZtYeuQXLrO17C9hD0lBgQR2DQ1gzy16DzsCfyp1lAzMWmN6Y4krSb0gtVktI50v9GLivIcWV1S47v62C1AJeTTsfbMZs8uTJXTp06HBCx44dj4qI3hTfw8TahpC0bP369ddXV1f/Zuedd67xVAgXWGZt32XAzaSuP38gdacqu2wAhVq7f0VEVQvGaRHZQBpDS73d7CC1rm5b1RtyYR0R/6KO65zVoz9paP/+pNEVb6Xm7n9NUt/7AVgf7bNLySmkLqnVwP9FxPIy5zFrVp06dZrQq1evTw8dOnRVly5d3vUl39qXiGDt2rWd582bd9Ly5ct3opZLTriLoFk7IGkjYEtgVRlGB6tR1mWqrmvdbN5asrZ2kvYinZ9Wm3Mj4pwWCWONIukc4Gd1LPLZ2gadaMuyC4gPBt6MiCXlzmPW3KZOnTprhx12WNqhQwcfYLdj1dXVeuGFF/qMGjWqxusxugXLrB3ITqIv6lo7Legc0nVvauOuQsWbTLr2V238XLZ+1wL31jF/Zh3z2qyIWIRHDrQNS0cXV+1f9hrX2rPELVg1GDBgQIwYMaLcMczMzMysjCZPnrw4IgovQF6rqVOnzhk1alSLXsjbymPq1KkDRo0aNaKmeW7BqsGIESOYNOlj11c1MzMzsw1INjKsWYO4wDIzMzMza0Y//9PaUZVrmv+4u3tXqs4+vMvU5t5Pod/85jf9Lr/88sEzZ86c0RzLtzX1XeDRzMzMzMyaoCWKq4bsp3v37mNyt06dOu3UqVOnnfKnNXS/J5xwwnsNKZYaunxLmjlzZhdJO8+aNatzY7fhFiwzMzMzsw1IZWXllNzv48aN26yqqkp33nnnnJqWXbNmjbp27epBGxrALVhmZmZmZgbArrvuuu23v/3tYfvuu++WPXv2HHPuueduPGvWrM577rnn1n379h1VUVExeuedd972ySef7J5b54orrug/fPjwkfnbOOaYYzb9/Oc/v2WPHj3GDBs2bOQf//jHPo1dvrq6mrPOOmvwxhtvvGPv3r1HH3300cN22223bU477bQarz35/vvv6/DDD9+sX79+o3r27DlmxIgRI6+//vq+ufn3339/z5133nnb3r17jx42bNjIn/3sZxtXV6fLSY4dO3Y7gJEjR47s3r37mPHjxw9p6HPoAsvMzMzMzD5w2223DTj55JMXLV++fMoPf/jDRdXV1Tr++OMXvfHGGy/Mnz9/6o477lg5bty4LdesWVPrlZTvuOOO/qeffvqC5cuXT/nOd76z6IQTThixYsWKWmuPupa/+uqr+0+YMGHjv/zlL68uWrRo6pAhQ9Y+++yzPWvb1lVXXdX/+eef7zFjxozpK1eunPLwww/P3HHHHVcDTJo0aaNDDz1061NPPXXhu++++/w999zz2nXXXTfo6quv7p/NnwEwffr06ZWVlVMuueSS+Q19/lxgmZmZmZnZB/bbb78lBx100IoOHTpQUVFRvfXWW6/9+te/vqyioqK6Z8+ecdlll709f/78LtOnT+9a2zYOOOCAJZ/73OdWdezYkVNPPXXxypUrOzZ2+VtuuaX/N77xjXc+/elPr+7atWucd955CwcNGrSutm116dIlKisrOzz//PPd1q1bx1ZbbbVu5513fh/giiuuGLT//vsvOeKII5Z26tSJMWPGvP+d73xn0S233NK/Kc9ZPp+DZWZmZmZmH9hss83W5N+fP39+pxNPPHHTiRMnVqxYsaKTpABYsGBBrbXEkCFDPiiAevXqVQ2wfPnyWi/OW9fyCxYs6LzZZputzc3v0KEDQ4YMWfvxrSQnnHDCuwsXLux8xhlnDJs7d27X3XbbbcVll1321siRI9e88cYbXSZOnNiroqKiT275iNDgwYNr3V5DuQXLzMzMzMw+0KHDR0uEU089dZOFCxd2njhx4ssrV66c8sYbb0yDVJi0RJ7Bgwevmzt3bpfc/erqaubPn9+ltuU7d+7MhRdeuGD69OkvzZ49+4Vu3bpVH3nkkSMANt1007WHHXbY4hUrVjyfu61cuXLKa6+99iJ8/LE3hgssMzMzMzOr1YoVKzp269ateuDAgVXLli3rcNJJJ23akvs//PDD3/3jH/848N///ne3NWvW6Nxzz9140aJFtQ6jfs8991Q8+eST3desWaMePXpU9+jRo7pTp04BcMoppyz629/+1u+WW27pvWbNGq1bt47JkydvdN999/WE1JLWoUMHZsyYsVFj87rAMjMzMzNrRt27UtWW93PBBRfMe/fddzv3799/9Pbbb7/97rvvvrJjx1p7+5Xcd7/73XePOuqoRQcffPDWgwYNGvXWW291GTVq1KquXbtW17T8/PnzOx955JGb9+3bd/TgwYNHvfnmm10mTJgwF2CXXXZ5/4477nj1yiuv3Hjw4ME7DhgwYPSRRx65ea5g69mzZ4wfP/7to446aouKiorRZ5555uCG5lWEh7UvNHbs2Jg0aVK5Y5iZmZlZGUmaHBFji11+6tSpc0aNGrW4OTMZrF+/nqFDh+547rnnvnX88ce/V44MU6dOHTBq1KgRNc1zC5aZmZmZmbVqEyZM6FtZWakVK1Z0OO2004auXr26w6GHHrqs3Llq4gLLzMzMzMxatWuuuWbQxhtvPGro0KE7Pvnkk73+8pe/vDpw4MD15c5VEw/TbmZmZmZmrdrkyZNnljtDsdyCZWZmZmZmViIusMzMzMzMzErEBZaZmZmZmVmJuMAyMzMzMzMrERdYZmZmZmZmJeJRBM3MzMzMmtHSy84YFatXNftxt7r1qOpz2qVTm3s/DbHVVlttf+aZZ8475phjlpQ7S0txgWVmZmZm1oxaorhqyH66d+8+Jvf72rVrBdClS5fITausrJzS0H3PnDmzyyc+8YkdXnvttWlbbrnlutz011577cWGbqulHHrooSM6deoUt95669xSbtcFlpmZmZnZBiS/gBo3btxmVVVVuvPOO+eUMVK74nOwzMzMzMwMgFdffbXLF77whS0GDhy448CBA3c8/PDDN1uyZEkHgOrqak466aRNBg0atGOPHj3GbLLJJjtceOGFgwDGjh27HcDIkSNHdu/efcz48eOHAGyyySY7XH311f0A7r333opOnTrtPGHChL7Dhg0bWVFRMXr//fffIrd9gGnTpnXdZZddtu3Zs+eYbbfddrvzzz9/kKSda8v717/+teKTn/zkdj179hzTt2/fUbvvvvs2uXkrVqzocOyxx266ySab7NC7d+/Re+6559bTp0/vCvDjH/9447vvvrvfnXfe2b979+5junfvPqaqqqokz6ELLDMzMzMzo7KyUvvss882n/jEJ96fPXv2CzNmzHhx3rx5nY899tjhAH/961973Xbbbf2ffvrpl1atWjVl4sSJL33mM59ZCTBp0qQZANOnT59eWVk55ZJLLplf0z7Wr1/PQw891Gv69OkzZsyYMf3FF1/sftFFFw0CWLduHQcddNDW22+/feWCBQuev+uuu1678cYbB9aV+dhjj938+OOPX7h8+fIp8+bNm/bDH/7wg/1+7Wtf2+zVV1/daOLEiS8tXLhw6tixY1cdeOCBW61Zs0YXXHDBwoMPPvi9Qw899N3KysoplZWVUzp1Kk3nPncRLJHz/9S4ivcnh7fMS9DYfND6Mzpf0trzWfu3/LLvN3rdXqddXrIcdWlsxpbKZ2ZWTrfeemufiODyyy+fB9CzZ8/1F1xwwbx99tnnE1VVVXO6dOkSa9eu1fPPP99t6NChVcOGDasaNmxYgw9ALr/88rd79+5d3bt37+ovfOELS5977rkeAI8++mjPefPmdbnqqqve6tmzZ2y33XZrTzzxxIWnnXbaZrVtq3PnzjFr1qyub731Vqfhw4dXHXjggSsA5s+f3+nee+/t98orr7yQy3jppZfOu+666wY9/vjjPT7/+c+vbNyzVL9234IlqaOkSyS9I2mFpDslDSh3LjMzMzOz1uT111/vMn/+/C4VFRWjc7f9999/G0m8+eabnQ844IAVP/rRj97+xS9+MWTQoEGj99hjj63/+c9/dm/IPjp27MjQoUM/KMp69OhRvWrVqo4Ab7zxRud+/fpV9ezZ84MBNzbffPM1dW3vjjvueG3WrFkbjRo1avstt9xy+/POO28QwCuvvNIFYKeddtou91j69OkzuqqqSnPmzOnSkMwNtSF8dX0WcDDwX8C7wO+Bm4D9yhnKzMzMzKw12WyzzdaOGDFiTV0j/51xxhmLzzjjjMUrVqzoMH78+KHjxo3bcv78+S906ND0dpvhw4evW7JkSaeVK1cqV2TVVwzttttuq++7777Xq6urefDBB3t+6Utf2mb06NGrd95559UAM2fOnJ5f0OUrReYat9ssW21djgUujojXI2IZ8APgC5JGlDeWmZmZmVnrMW7cuGVVVVU666yzBi9ZsqRDdXU1s2fP7nzjjTf2AXjiiSe6P/DAAz1Xr16tbt26VVdUVKzv2LEjAEOGDFnXoUMHZsyYsVFj97/33nuvHDJkyNqTTz5508rKSr388stdrr766o1rW/7999/XlVde2X/+/PmdOnToQP/+/askRefOnWOTTTapOvDAA987+uijh8+ePbszwOLFizveeOONfZYtW9YBYOONN143d+7cruvXr29s5Bq16wJLUm9gODA5Ny0iZgHLgR3LlcvMzMzMNhzq1qM0w9M1834qKiqqH3744ZkvvfRSt2233XZkr169xuy9997bTJkypTvAsmXLOp566qnDBgwYMLpv376jH3300V433XTTLICePXvG+PHj3z7qqKO2qKioGH3mmWcObuj+O3fuzF133fXatGnTug8cOHD0wQcfvNW4cePe7dy5c9S2zh133NF3u+2227579+5jDjnkkK3Hjx8/b7/99lsJcPPNN8/deuut399rr7227dGjx5iRI0duf/vtt/eVBMCJJ574TmVlZYe+ffuOrqioGF2qUQTbexfBXtnPZQXTl+bNA0DSsaTWLoYPH97swczMzMxsw9DntEunljtDbQovsrvVVlutu/vuu2fXtOxBBx204qCDDnqptm1ddNFFCy666KIF+dPefvvtF3K/H3DAASuqqqom58+/7LLL5uXfHzNmzPuTJk2ambt/ySWXDBg6dOjamva30UYbxRNPPPFabXkqKiqqr7jiinlXXHHFvJrmb7fddmunTZv2cm3rN1a7bsECVmQ/exdM70NqxfpARFwbEWMjYuzAgXWOBmlmZmZmZs3gwQcf7PHiiy92ra6u5plnnul2xRVXDD7kkEPeK3euhmjXLVgRsVTSG8BOwPMAkrYgtV5NK2M0MzMzMzMrMHv27K7f+ta3tli6dGmnvn37Vh1wwAFLLrzwwhqvqdVatesCK3MtcKakx0ijCF4MPBARc8qayszMzMzMPuK4445777jjjmtTLVaFNoQC6yKgL/As0BV4CDiirInMzMzMzKxdUkStg3JssMaOHRuTJk0qdwwzMzMzKyNJkyNibLHLT5069fUddthhSYcOHXyA3Y5VV1frhRde6Dtq1Kgtaprf3ge5MDMzMzNrEZIWrF69utHXgbK2YfXq1RtJWlDbfBdYZmZmZmYlUFVVde6cOXO6rFq1qlt1dbXKncdKq7q6WqtWreo2Z86cLlVVVefWttyGcA6WmZmZmVmz22mnnR547rnnvjdr1qyfRcRg3JjR3lRLWlBVVXXuTjvt9EBtC7nAMjMzMzMrkezAu9aDb2v/XFWbmZmZmZmViAssMzMzMzOzEnGBZWZmZmZmViIusMzMzMzMzErEBZaZmZmZmVmJuMAyMzMzMzMrERdYZmZmZmZmJeICy8zMzMzMrERcYJmZmZmZmZWICywzMzMzM7MScYFlZmZmZmZWIi6wzMzMzMzMSsQFlpmZmZmZWYm4wDIzMzMzMysRF1hmZmZmZmYl4gLLzMzMzMysRFxgmZmZmZmZlYgLLDMzMzMzsxJxgWVmZmZmZlYinYpZSJKArwJjgYr8eRFxbDPkMjMzMzMza3OKbcH6DXAVMAzoXHAzMzMzMzMzimzBAg4Ddo2IWc0ZxszMzMzMrC0rtgWrEnijOYOYmZmZmZm1dcUWWL8Efpqdi2VmZmZmZmY1KLaL4MnAZsBJkhblz4iIbUqeyszMzMzMrA0qtsC6oFlTmJmZmZmZtQNFFVgR8YfmDmJmZmZmZtbWFduChaRdgG+Thmp/E/h9RDzbXMHMzMzMzMzamqIGuZD0JeCfQG9gCtALeELSIc0XzczMzMzMrG0ptgXrZ8ChEfH33ARJ+wEXAXc1RzArrfP/VNXodX9yeNENnU3S2IzOl7T2fND6Mzqfldvyy77fqPV6nXZ5SXPUprXnMzNrDYodpn0EcH/BtAdIIwuamZmZmZkZxRdYc4F9C6btQyu/+LCkr0p6UtJySY1vwjEzMzMzMytCsf1GzgfulnQH8DqwOXAo8K3mClYiS4CrgW7AtWXOYmZmZmZm7Vyxw7TfKWkecCSwC2kUwX0j4t/NmK3JIuIBAEl7lTeJmZmZmZltCIo+8zkrplp1QdUUko4FjgUYPnx4mdOYmZmZmVlbVGuBJem/IuKZ7Pfda1suIp5ujmAtLSKuJetGOHbs2ChzHDMzMzMza4PqasF6GKjIfn+qlmUC6FjSRGZmZmZmZm1UrQVWRFTk/V7saINmZmZmZmYbrKIKJ0nfqGX610sbp7QkdZS0EdAlu79RdlOZo5mZmZmZWTukiPpPN5K0PCJ61TD9vYjo1yzJSkDSkcD1NczaPCLm1Lbe2LFjY9KkSc0Vy8zMzMzaAEmTI2JsuXNY21Js17+PtfhIGgG06ov3RsQNEaEabnPKnc3MzMzMzNqfOodpl7SObCALSWsLZnckXcTXzMzMzMzMqP86WPuSWq/+DuyXN70aWBARrzZXMDMzMzMzs7amzgIrIp4AkLRlRMxvmUhmZmZmZmZtU30tWABExHxJ/YFdgIHknZMVETc2UzYzMzMzM7M2pagCS9K+wJ3AWqAPsDT7ORtwgWVmZmZmZkbxowheBJwXEQOBldnP84Frmi2ZmZmZmZlZG1NsgbU1cHn2e6574MXA90ucx8zMzMzMrM0qtsCqBLpmv78raTjQBejbLKnMzMzMzMzaoGILrKeBL2W//x24B3gY+HczZDIzMzMzM2uTihrkAjiCD4ux8cDpQAVwWXOEMjMzMzMza4vqLbAkdQIuJRVVRMT7wIXNnMvMzMzMzKzNqbeLYERUAV8F1jR/HDMzMzMzs7ar2HOw7gYObc4gZmZmZmZmbV2x52B1Af4o6XhgDlCdmxERxzZDLjMzMzMzszan2AJrHfCn7PeO2c3MzMzMzMzyFFVgRcRRzR3EzMzMzMysrSv2HCwk9Zb0NUk/yO4PljS0+aKZmZmZmZm1LUUVWJJ2Al4DzgJ+kk3eEbiymXKZmZmZmZm1OcW2YP0f8IOI2BGoyqY9DXyqWVKZmZmZmZm1QcUWWNsDN2S/B0BErAR6NEMmMzMzMzOzNqnYAusdYHj+BElbAW+XPJGZmZmZmVkbVWyB9Qfgz5L2ACRpZ+A6YEKzJTMzMzMzM2tjir0O1sVAT+Dv2c/HSOdleZALMzMzMzOzTLHXwVoPnA2cLWlARCxu3lhmZmZmZmZtT7HDtN+f+z2/uJJ0X3OEMjMzMzMza4uKPQdr91qme5h2MzMzMzOzTJ1dBCV9LbecpMMB5c3eGljSXMHMzMzMzMzamvrOwbow+9kV+Hne9GpgAXBSc4QyMzMzMzNri+ossCJicwBJ90TEQS0TyczMzMzMrG0q6hwsF1dmZmZmZmb1q7UFS9IVEXFy9vu1tS0XEcc2RzAzMzMzM7O2pq4ugp1r+d3MzMzMzMxqUGuBFREn5P1+VMvEMbP27Pw/VTVqvZ8cXtQ10ZvM+Zqmsfmg9WdsqXzWNMsv+36j1+112uUly1GXxmZ0vqS15zOD4q+DZWZmZmZmZvVo1wWWpIslvShpuaR5kiZI6lfuXGZmZmZm1j616wILWA8cAfQHRgGbAteXNZGZmZmZmbVb7bpTeUScnXf3HUlXAbeUK4+ZmZmZmbVvdbZgSbpH0v6S1FKBmtk+wLSaZkg6VtIkSZPeeeedFo5lZmZmZmbtQX1dBN8FbgPmSPqJpE1aIFOzkHQocAxwSk3zI+LaiBgbEWMHDhzYsuHMzMzMzKxdqLPAyoZnHwr8EvgyMFvSXyXt15ZatSQdBkwADoqI58qdx8zMzMzM2qd6B7mIiOUR8euIGAPsyYetWrMl/bi5AzaVpKOA3wIHRsRj5c5jZmZmZmbtV4NGEYyIZyLiaFKhtR44t1lSlYikk4FLgc9HxL/KncfMzMzMzNo3RURxC0qdgEOAY4HPAv8CfhsRrXZUPkkBVAFr8qdHRM+61hs7dmxMmjSpOaOZmZmZWSsnaXJEjC13Dmtb6h2mXdI2pMEhvgl0BG4CToqIl5s5W5NFRJs5T8zMzMzMzNq+OgssSU8AewBPA6cDt0fEmrrWMTMzMzMz21DV14I1BTg+Il5qiTBmZmZmZmZtWX2DXFwE7FDTDEmHSRpU+khmZmZmZmZtU30F1g+ArWuZtyVwZmnjmJmZmZmZtV31FVj7A9fVMu/3wAGljWNmZmZmZtZ21VdgDY6IhTXNiIhFwODSRzIzMzMzM2ub6iuw1koaUtOMbPq60kcyMzMzMzNrm+orsP4FnFTLvO8CT5Y2jpmZmZmZWdtV3zDtFwJPShoI/Al4G9gEOBz4OukaWWZmZmZmZkY9BVZETJJ0EPBr4GggAAGvAQdFxHPNH9HMzMzMzKxtqK8Fi4h4CNhG0tbAQOCdiHi12ZOZmZmZmZm1MfUWWDlZUeXCyszMzMzMrBb1DXJhZmZmZmZmRXKBZWZmZmZmViIusMzMzMzMzErEBZaZmZmZmVmJuMAyMzMzMzMrERdYZmZmZmZmJeICy8zMzMzMrERcYJmZmZmZmZWICywzMzMzM7MScYFlZmZmZmZWIi6wzMzMzMzMSsQFlpmZmZmZWYm4wDIzMzMzMysRF1hmZmZmZmYl4gLLzMzMzMysRFxgmZmZmZmZlYgLLDMzMzMzsxJxgWVmZmZmZlYiLrDMzMzMzMxKxAWWmZmZmZlZiXQqd4D2YsRZ9zVqvTkXfbHESWq2wx92aPS6L3zrhRImqV1jMzpf0trzAfz6+Ecbtd53r9m7xElq5nxN89ZZTzZ63U0v2rOESWrX2Iwtle+RR7ds1Hr77D2rxElq1trzXf/qvEave9TWQ0uYpHaNzeh8SWvPZwZuwTIzMzMzMysZF1hmZmZmZmYl0ioLLEmdy53BzMzMzMysoVqswJI0WNLfJC2T9IqkoyWFpBGSbpB0s6TrJb0HXJGtc4Kkmdk6EyXtmbe9cyQ9XLCPxyX9OPt9L0lVkr4laa6k97L99Gypx2xmZmZmZhuWlmzBuhlYCwwD9gC+UTD/MOB+YCBwuqTDgfOBbwL9gQnA/ZI2a8A+OwIHAjsCnwS2Af63pgUlHStpkqRJ77zzTgN2YWZmZmZmlrRIgSVpU2BvYHxELI+IRaTiKd9TEXFrRKyPiErgKOC3EfFMRFRFxO+AacDXGrj7MyNiWUQsBH4KfEvSxx53RFwbEWMjYuzAgQMb/BjNzMzMzMxaqgVrk+znG3nT5hYsM6fg/jDg9YJps7LpDZG/nzlAV2BAA7dhZmZmZmZWr5YqsN7Ofg7Pmza8YJnqgvtvApsXTNsimw6wEuhRML+mixzkdykcAawBFteR1czMzMzMrFFapMCKiLeAx4GLJFVIGgT8uJ7VbgCOk7SrpE6SjgRGA3/K5k8CdpK0czb/e3y8IAP4haRe2T7PAW6KiMJizszMzMzMrMkUES2zI2kIaaCKPYGFwK+Aq0mtTr8AqiLiOwXrfA84GdgYmAn8ICIez5v/c+CY7O412bYfjogLJO0FPAwcDZwH9AL+Bnw3IlbUlXXs2LExadKkJjxaMzMzM2vrJE2OiLHlzmFtS4sVWB/bsfR54G6gWzRDiFyBFRGdGrquCywzMzMzc4FljdHg4qOxJI0CAniB1JXvAuDW5iiuzMzMzMzMyqElr4PVD/gLaXCKp0hDrp/Sgvs3MzMzMzNrVi3WghURjwFbteD+HqcFH5+ZmZmZmVlLtmCZmZmZmZm1ay6wzMzMzMzMSsQFlpmZmZmZWYm4wDIzMzMzMysRF1hmZmZmZmYl4gLLzMzMzMysRFxgmZmZmZmZlYgLLDMzMzMzsxJxgWVmZmZmZlYiLrDMzMzMzMxKxAWWmZmZmZlZibjAMjMzMzMzKxEXWGZmZmZmZiXiAsvMzMzMzKxEXGCZmZmZmZmViAssMzMzMzOzEnGBZWZmZmZmViIusMzMzMzMzErEBZaZmZmZmVmJuMAyMzMzMzMrERdYZmZmZmZmJaKIKHeGVkfSO8DcEm5yALC4hNsrtdaeD1p/Rudrutae0fmaprXng9af0fmarrVndL6maY58m0XEwBJv09o5F1gtQNKkiBhb7hy1ae35oPVndL6ma+0Zna9pWns+aP0Zna/pWntG52ua1p7PNhzuImhmZmZmZlYiLrDMzMzMzMxKxAVWy7i23AHq0drzQevP6HxN19ozOl/TtPZ80PozOl/TtfaMztc0rT2fbSB8DpaZmZmZmVmJuAXLzMzMzMysRFxgmZmZmZmZlYgLLDMzMzMzsxJxgdWMJHWUdImkdyStkHSnpAHlzpUj6auSnpS0XFJVufMUknSxpBezfPMkTZDUr9y58km6UNLsLOMiSXdIGl7uXIUkdZD0tKSQtGm58wBIukHSOkkr824nljtXTSTtK2lilnGxpKvLnQkge3/kP3+rs9d4p3Jny5E0WNKt2efgEkmPShpV7lw5kvpL+oOkBZKWSbpFUt8y5qnzc1nSNyXNklQp6RlJO7eWfJJGSfqHpPnZ3+EeLZmtyIzfzD4Ll2Tv5X9I2qEV5TtM0vQs3xJJT0n6TGvJV7DcxdnrfERL5sv2XddzeKSk6oLPxj+1dEbbsLnAal5nAQcD/wXkDmpvKl+cj1kCXA18v8w5arMeOALoD4wiPYfXlzXRx90EjI6IXsAI4A3gz2VNVLNTgcpyh6jBHyKiZ96tVRQu+STtBdwBXEr6W9wUuK6MkT4QEdvnP3/AZcCMiHiu3NnyXA30A7YFNgYmAfdKUllTfehGoCewNbA56TUu5+d0rZ/LWcHyG+AEoC9wJ/B3Sb1aQz5gLfAX4KAWzFOTujJWAD8jvY83AZ4DHpTUrcXS1Z1vIvDfEdGX9Ld4Bek17tNi6Yo4NpC0K7AfML+FMhWqL+PrBf9bDm+5aGbQqdwB2rljgfMi4nUAST8AXpM0IiLmlDUZEBEPwAcHkK1ORJydd/cdSVcBt5QrT00i4uW8uwKqSQeSrYakbYATgUOBKWWO0xb9ArgmIu7Im9aaChgAJHUCvk3K25psBVwVEe8BSPodMJ508Li4nMEk9SAdJI6JiBXZtJ8Dj0vaLCLmtnSmej6XjwH+EhEPZstcAnwPOAT4Q7nzRcRLwEvZ/JaIU6N6Mv46/372ep8NfIIW+nysJ9+b+fFIXzR2B4YBS5s/Xf3HBpK6Ar8jHeOUpWWotR+/mLkFq5lI6g0MBybnpkXELGA5sGO5crVx+wDTyh2ikKSvSVoGrAROAc4pb6IPSeoA/J50QLu0vGlqdKik9yS9otSdtme5A+XLDsB3Bd6X9FzWpehxSWPLna0GXwJ6k1pkWpNLSK/zAEkbkQ7KnoqIshZXGeXdcnL/F1tNN8Y8o/jo/5QgFQWtMWtbsQ+pdf+1cgfJkTRc0lJSi+AdwK0R8UJ5U33EOcCjEfHvcgepw7Cs2++bkv4safNyB7INiwus5pPrsrGsYPrSvHlWJEmHkr69PaXcWQpFxC0R0RsYQvrH05r+EZ4CLIiIv5Q7SA2uJH1rPID0DfxngAllTfRxfUmfk8cARwJDgQdp+S47xTiOdCC2tNxBCvwL6Ai8Q/oS4suk57PsImIl8DhwjqQ+kgaSWjOgdX5OV+D/KSWTte5fB5yea8FsDSLijYjoQ3pdjwIeK2+iD2VfLh0G/KjcWerwT2AH0uf1LsD7wEPZF2ZmLcIFVvPJfVj3Lpjeh9SKZUWSdBjpwPugVnZuyUdExAJSznvVCgbjkLQVcDqpC1GrExGTI2JhRFRHxIuk88S+knU/aS1y7+PrI2JaRKwldcHrDOxevlgfJWlL0jfx15Q7S76sBfVh4BXSZ2F34ELgSUkblzNbniOANaSubf8B7s6mt4YWtkIr8P+UkpC0HalwuTQiWtX7JiciVkXEDcApkj5f7jySupDOg/5u9uVEqxQRr0fEK9n/lgWkL3SGAp8qczTbgLjAaibZt8hvAB+M5iVpC9I3Uq2um1trJeko4LfAgRHRar7Fq0MnoAfpw7zc9gAGAtMlLebD84amqXWO1led/Wwtgx8QEcuAOUDUNLtl09TpOGBqRDxT7iAF+pEGjrgyIpZHxNqIuI70v6dVHOxExNsRMS4ihkTE5sBs0jfeE8scrSZT+ej/FAGjs+lWJKVRNh8HLoqIX5Y5TjE6kQZhKbehwPbAzVl36cWkc8N+I+nm8karU2S3VvO/xdo/F1jN61rgTEmbZ6M8XQw80BoGuIAPhpHfCOiS3d8ou7WKDyFJJ5NGbvt8RPyr3HkKKQ19/j1Jg7L7mwK/Jh2Qv1zXui3kNmBL0gHYaGD/bPrnaAXn6WTD7PbJft8a+F/gnoh4v6zBPu5q4ChJ22UDSYwnHYA/Xd5YSfat8pG0stYrgOw8q1eAEyX1kNRJ0rdJXd1aRVdaSdtK6pe9n3cBLicdeC8tU566PpcnAF+WtE/2up8ObATc1RryZTbK5gN0ye53bKl8RWT8NPAI8OOIuLIlcxWZ75uStsr+Hisk/ZR0Pvej5c4HvJllGZ13m0fqVntyS+WrK2P2HH5R0qbZ7/1I/5cX0zq/NLH2KiJ8a6Yb6byDS0lv7BWk4WsHlDtXXr4j+fCbnfzbiHJny/IFsI503sYHt3LnysvXAfg7sAhYBbwN3AxsWe5steQdkT2nm5Y7S5bnceC97LmbTRpivFe5c9WQU8B5wALS+S6PkYbmL3u2LN9Xs8+XnuXOUku+TwL3Zp+Dy0iDNBxc7lx5+Y4hDTVdCbwKnFLmPHV+LgPfBF4HVpO6NO7cWvLlfcYU3o5sRRkfI7WWryy47dlK8l0AzM0+F98hFYP7tJbnr4Zl5wBHtGS+Ip7DS0iF36rsvX0HsE1LZ/Rtw74pojX1cjEzMzMzM2u73EXQzMzMzMysRFxgmZmZmZmZlYgLLDMzMzMzsxJxgWVmZmZmZlYiLrDMzMzMzMxKxAWWmZmZmZlZibjAMjPbgEk6R9LDTVj/bEl/a2KG1yQd2ZRtmJmZtRYusMzMWoikLSTdLmmBpJWS3pR0l6Qu5c7WWBHx84g4sNw5zMzMWgsXWGZmLefvwHxgW6AC2A14AFA5Q5mZmVnpuMAyM2sBkvqTCqtrImJZJG9FxDURsSZbZpSkJyQtlrRE0j8kbZm3jRsk3STp95KWSnpb0uGSRkt6VtIKSY9JGpq3zhxJP5X0VNZqNknSLnXk7C7pUkmzJb0n6X5JW9Wx/Ee6GGb7O1vSI9n+pkvaPW9+Z0mXSVqUteSdWcM298zyvidplqTTJSmbd5OkByV1yO7vJWm5pJHFvhZmZmbNyQWWmVkLiIh3gReB6yR9U9J2uaIhfzHgHGATYASwEvhjwTJfAe4E+gHnAxOA84BDgI3ztpHveOCUbJ07gL9L6lVL1OuATwCfAgYDzwD3Supc/KPl28DJQG/gIeAPefPOAg4Adgc2zx7nZrmZkrYntfRdAgwEvgh8D/hG3mPZFPixpI2BW4BTImJ6A/KZmZk1GxdYZmYtZy/gceD7wPPAQkk/yRVaETEtIh6LiDURsQw4F/iUpB5523g0Iu6LiGrgRqAHcFPWGlZJKqAKW6h+FxGTI2ItcDGwmlTkfISkAcDhwIkRsTBb/lxgCPBfDXicv42IFyNiPalg20pS72zeN4GLI+K1iFgNnEEqCnNOAG6PiLsjYn1EvAxcla1HRKwiFZlnkIq3ByLi+gZkMzMza1adyh3AzGxDERGLgbOBsyV1B/6H1AL1NvD7rDvgJaRipoIPC48BwKrs9/l526vMarMPpgGV2br55uStE5LeILUCFdo8+zmtoHGtMzCsqAdZkDEvdwWwLNtvfp5VkhYVZNhb0pfzpnUA3sxbZ4akx4ADgS81IJeZmVmzcwuWmVkZRERlRNwATANGZ5OvAVYAO0ZEL+DT2fSmDoIxIvdL1lo2HHirhuXmZj+3jog+ebfuEfGnJmbIebsgTw9gUEGG3xfsv1dEbJ+3ztdJXRjvAH4nqWOJspmZmTWZCywzsxYgqa+kX0gamQ300EnSocBI4MlssV6kFp+lWXe980q0+29L2ik7j2o80B24r3ChiFhEOqfpakmbZLn7SDpEUs8SZbkJGC9pS0ndgF/y0QLyauCrkg7Me562k/SZLM8ns2W+DnyL1Lr3sxJlMzMzazIXWGZmLWMtqaXmL8B7wDvAj4GTIuL2bJlTgT2B5aSi694S7fta4ApgCTAO+GJ2jldNjgFmAo9LWgG8ABzGR8+TaopfkIamnwjMBt7gw5YzssEqDiCdpzYfWATcAAzMulXeDvwqIh7OzuE6DDhF0n+XKJ+ZmVmTKKJU/zPNzKy1kTQH+HFEFI5GaGZmZs3ALVhmZmZmZmYl4gLLzMzMzMysRNxF0MzMzMzMrETcgmVmZmZmZlYiLrDMzMzMzMxKxAWWmZmZmZlZibjAMjMzMzMzKxEXWGZmZmZmZiXy/wFrDCgb4pzVdQAAAABJRU5ErkJggg==\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Train indices: [0 1 2 3 4]\n", + "Test indices: [8 9]\n", + "Train length: 5\n", + "Test length: 2\n", + "Train groups: [0 1 1 1 1]\n", + "Test groups: [3 3]\n", + "Train group size: 2\n", + "Test group size: 1\n", + "Train group months: ['2021-01' '2021-02' '2021-02' '2021-02' '2021-02']\n", + "Test group months: ['2021-04' '2021-04']\n", + "\n", + "Train indices: [1 2 3 4 5 6 7]\n", + "Test indices: [10 11]\n", + "Train length: 7\n", + "Test length: 2\n", + "Train groups: [1 1 1 1 2 2 2]\n", + "Test groups: [4 4]\n", + "Train group size: 2\n", + "Test group size: 1\n", + "Train group months: ['2021-02' '2021-02' '2021-02' '2021-02' '2021-03' '2021-03' '2021-03']\n", + "Test group months: ['2021-05' '2021-05']\n", + "\n", + "Train indices: [5 6 7 8 9]\n", + "Test indices: [12 13 14 15]\n", + "Train length: 5\n", + "Test length: 4\n", + "Train groups: [2 2 2 3 3]\n", + "Test groups: [5 5 5 5]\n", + "Train group size: 2\n", + "Test group size: 1\n", + "Train group months: ['2021-03' '2021-03' '2021-03' '2021-04' '2021-04']\n", + "Test group months: ['2021-06' '2021-06' '2021-06' '2021-06']\n", + "\n" + ] + } + ], + "source": [ + "cv_args = {\"test_size\": 1, \"n_splits\": 3, \"gap_size\": 1}\n", + "\n", + "plot_splits(X, y, groups, **cv_args)\n", + "print_split_info(X, y, groups, **cv_args)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Usage in CV" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The example below shows how this looks like in a scikit-learn context using `cross_val_score`:" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Split number: 1\n", + "Train true target: [1 0 1 0 1]\n", + "Train predicted target: [1 1 1 1 1]\n", + "Test true target: [1 1]\n", + "Test predicted target: [1 1]\n", + "Accuracy: 1.0\n", + "\n", + "Split number: 2\n", + "Train true target: [0 1 0 1 0 0 1]\n", + "Train predicted target: [0 0 0 0 0 0 0]\n", + "Test true target: [0 1]\n", + "Test predicted target: [0 0]\n", + "Accuracy: 0.5\n", + "\n", + "Split number: 3\n", + "Train true target: [0 0 1 1 1]\n", + "Train predicted target: [1 1 1 1 1]\n", + "Test true target: [1 0 0 0]\n", + "Test predicted target: [1 1 1 1]\n", + "Accuracy: 0.25\n", + "\n" + ] + } + ], + "source": [ + "cv = GroupTimeSeriesSplit(**cv_args)\n", + "clf = DummyClassifier(strategy=\"most_frequent\")\n", + "\n", + "scores = cross_val_score(clf, X, y, groups=groups, scoring=\"accuracy\", cv=cv)\n", + "print_cv_info(cv, X, y, groups, clf, scores)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## API" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "## GroupTimeSeriesSplit\n", + "\n", + "*GroupTimeSeriesSplit(test_size, train_size=None, n_splits=None, gap_size=0, shift_size=1, window_type='rolling')*\n", + "\n", + "Group time series cross-validator.\n", + "\n", + "**Parameters**\n", + "\n", + "- `test_size` : int\n", + "\n", + " Size of test dataset.\n", + "\n", + "- `train_size` : int (default=None)\n", + "\n", + " Size of train dataset.\n", + "\n", + "- `n_splits` : int (default=None)\n", + "\n", + " Number of the splits.\n", + "\n", + "- `gap_size` : int (default=0)\n", + "\n", + " Gap size between train and test datasets.\n", + "\n", + "- `shift_size` : int (default=1)\n", + "\n", + " Step to shift for the next fold.\n", + "\n", + "- `window_type` : str (default=\"rolling\")\n", + "\n", + " Type of the window. Possible values: \"rolling\", \"expanding\".\n", + "\n", + "**Examples**\n", + "\n", + "For usage examples, please see\n", + " http://rasbt.github.io/mlxtend/user_guide/evaluate/GroupTimeSeriesSplit/\n", + "\n", + "### Methods\n", + "\n", + "
\n", + "\n", + "*get_n_splits(X=None, y=None, groups=None)*\n", + "\n", + "Returns the number of splitting iterations in the cross-validator.\n", + "\n", + "**Parameters**\n", + "\n", + "- `X` : object\n", + "\n", + " Always ignored, exists for compatibility.\n", + "\n", + "- `y` : object\n", + "\n", + " Always ignored, exists for compatibility.\n", + "\n", + "- `groups` : object\n", + "\n", + " Always ignored, exists for compatibility.\n", + "\n", + "**Returns**\n", + "\n", + "- `n_splits` : int\n", + "\n", + " Returns the number of splitting iterations in the cross-validator.\n", + "\n", + "
\n", + "\n", + "*split(X, y=None, groups=None)*\n", + "\n", + "Generate indices to split data into training and test set.\n", + "\n", + "**Parameters**\n", + "\n", + "- `X` : array-like\n", + "\n", + " Training data.\n", + "\n", + "- `y` : array-like (default=None)\n", + "\n", + " Always ignored, exists for compatibility.\n", + "\n", + "- `groups` : array-like (default=None)\n", + "\n", + " Array with group names or sequence numbers.\n", + "\n", + "**Yields**\n", + "\n", + "- `train` : ndarray\n", + "\n", + " The training set indices for that split.\n", + "\n", + "- `test` : ndarray\n", + "\n", + " The testing set indices for that split.\n", + "\n", + "\n" + ] + } + ], + "source": [ + "with open(\"../../api_modules/mlxtend.evaluate/GroupTimeSeriesSplit.md\", \"r\") as f:\n", + " s = f.read()\n", + "print(s)" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.8.5" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} diff --git a/mlxtend/evaluate/__init__.py b/mlxtend/evaluate/__init__.py index 10d2eface..7c76fec88 100644 --- a/mlxtend/evaluate/__init__.py +++ b/mlxtend/evaluate/__init__.py @@ -21,6 +21,7 @@ from .permutation import permutation_test from .proportion_difference import proportion_difference from .scoring import scoring +from .time_series import GroupTimeSeriesSplit from .ttest import paired_ttest_5x2cv, paired_ttest_kfold_cv, paired_ttest_resampled __all__ = [ @@ -47,4 +48,5 @@ "bias_variance_decomp", "accuracy_score", "create_counterfactual", + "GroupTimeSeriesSplit", ] diff --git a/mlxtend/evaluate/tests/test_time_series.py b/mlxtend/evaluate/tests/test_time_series.py new file mode 100644 index 000000000..8233e0ba8 --- /dev/null +++ b/mlxtend/evaluate/tests/test_time_series.py @@ -0,0 +1,378 @@ +# mlxtend Machine Learning Library Extensions +# +# Time series cross validation with grouping. +# Author: Dmitry Labazkin +# +# License: BSD 3 clause + +import numpy as np +import pytest +from sklearn.dummy import DummyClassifier +from sklearn.model_selection import cross_val_score + +from mlxtend.evaluate import GroupTimeSeriesSplit + + +@pytest.fixture +def X(): + return np.array( + [[0], [7], [6], [4], [4], [8], [0], [6], [2], [0], [5], [9], [7], [7], [7], [7]] + ) + + +@pytest.fixture +def y(): + return np.array([1, 0, 1, 0, 1, 0, 0, 1, 1, 1, 0, 1, 1, 0, 0, 0]) + + +@pytest.fixture +def group_numbers(): + return np.array([0, 1, 1, 1, 1, 2, 2, 2, 3, 3, 4, 4, 5, 5, 5, 5]) + + +@pytest.fixture +def not_sorted_group_numbers(): + return np.array([5, 5, 5, 5, 1, 1, 1, 1, 3, 3, 2, 2, 2, 4, 4, 0]) + + +@pytest.fixture +def not_consecutive_group_numbers(): + return np.array([0, 1, 1, 1, 1, 2, 2, 2, 3, 3, 4, 4, 2, 2, 2, 2]) + + +@pytest.fixture +def group_names(): + return np.array( + [ + "2021-01", + "2021-02", + "2021-02", + "2021-02", + "2021-02", + "2021-03", + "2021-03", + "2021-03", + "2021-04", + "2021-04", + "2021-05", + "2021-05", + "2021-06", + "2021-06", + "2021-06", + "2021-06", + ] + ) + + +@pytest.fixture +def not_sorted_group_names(): + return np.array( + [ + "2021-06", + "2021-06", + "2021-06", + "2021-06", + "2021-02", + "2021-02", + "2021-02", + "2021-02", + "2021-04", + "2021-04", + "2021-03", + "2021-03", + "2021-03", + "2021-05", + "2021-05", + "2021-01", + ] + ) + + +@pytest.fixture +def not_consecutive_group_names(): + return np.array( + [ + "2021-01", + "2021-02", + "2021-02", + "2021-02", + "2021-02", + "2021-03", + "2021-03", + "2021-03", + "2021-04", + "2021-04", + "2021-05", + "2021-05", + "2021-03", + "2021-03", + "2021-03", + "2021-03", + ] + ) + + +def check_splits(X, y, groups, cv_args, expected_results): + cv = GroupTimeSeriesSplit(**cv_args) + results = list(cv.split(X, y, groups)) + + assert len(results) == len(expected_results) + + for split, expected_split in zip(results, expected_results): + assert np.array_equal(split[0], expected_split[0]) + assert np.array_equal(split[1], expected_split[1]) + + return cv + + +def test_get_n_splits(X, y, group_numbers): + cv_args = {"test_size": 1, "train_size": 3} + expected_results = [ + (np.array([0, 1, 2, 3, 4, 5, 6, 7]), np.array([8, 9])), + (np.array([1, 2, 3, 4, 5, 6, 7, 8, 9]), np.array([10, 11])), + (np.array([5, 6, 7, 8, 9, 10, 11]), np.array([12, 13, 14, 15])), + ] + cv = check_splits(X, y, group_numbers, cv_args, expected_results) + + assert cv.get_n_splits() == len(expected_results) + + +def test_train_size(X, y, group_numbers): + cv_args = {"test_size": 1, "train_size": 3} + expected_results = [ + (np.array([0, 1, 2, 3, 4, 5, 6, 7]), np.array([8, 9])), + (np.array([1, 2, 3, 4, 5, 6, 7, 8, 9]), np.array([10, 11])), + (np.array([5, 6, 7, 8, 9, 10, 11]), np.array([12, 13, 14, 15])), + ] + check_splits(X, y, group_numbers, cv_args, expected_results) + + +def test_train_size_group_names(X, y, group_names): + cv_args = {"test_size": 1, "train_size": 3} + expected_results = [ + (np.array([0, 1, 2, 3, 4, 5, 6, 7]), np.array([8, 9])), + (np.array([1, 2, 3, 4, 5, 6, 7, 8, 9]), np.array([10, 11])), + (np.array([5, 6, 7, 8, 9, 10, 11]), np.array([12, 13, 14, 15])), + ] + check_splits(X, y, group_names, cv_args, expected_results) + + +def test_n_splits(X, y, group_numbers): + cv_args = {"test_size": 2, "n_splits": 3} + expected_results = [ + (np.array([0, 1, 2, 3, 4]), np.array([5, 6, 7, 8, 9])), + (np.array([1, 2, 3, 4, 5, 6, 7]), np.array([8, 9, 10, 11])), + (np.array([5, 6, 7, 8, 9]), np.array([10, 11, 12, 13, 14, 15])), + ] + check_splits(X, y, group_numbers, cv_args, expected_results) + + +def test_n_splits_gap_size(X, y, group_numbers): + cv_args = {"test_size": 1, "n_splits": 3, "gap_size": 1} + expected_results = [ + (np.array([0, 1, 2, 3, 4]), np.array([8, 9])), + (np.array([1, 2, 3, 4, 5, 6, 7]), np.array([10, 11])), + (np.array([5, 6, 7, 8, 9]), np.array([12, 13, 14, 15])), + ] + check_splits(X, y, group_numbers, cv_args, expected_results) + + +def test_n_splits_shift_size(X, y, group_numbers): + cv_args = {"test_size": 1, "n_splits": 3, "gap_size": 1} + expected_results = [ + (np.array([0, 1, 2, 3, 4]), np.array([8, 9])), + (np.array([1, 2, 3, 4, 5, 6, 7]), np.array([10, 11])), + (np.array([5, 6, 7, 8, 9]), np.array([12, 13, 14, 15])), + ] + check_splits(X, y, group_numbers, cv_args, expected_results) + + +def test_n_splits_expanding_window(X, y, group_numbers): + cv_args = {"test_size": 3, "n_splits": 3, "window_type": "expanding"} + expected_results = [ + (np.array([0]), np.array([1, 2, 3, 4, 5, 6, 7, 8, 9])), + (np.array([0, 1, 2, 3, 4]), np.array([5, 6, 7, 8, 9, 10, 11])), + ( + np.array([0, 1, 2, 3, 4, 5, 6, 7]), + np.array([8, 9, 10, 11, 12, 13, 14, 15]), + ), + ] + check_splits(X, y, group_numbers, cv_args, expected_results) + + +def test_full_usage_of_data(X, y, group_numbers): + cv_args = {"test_size": 3, "train_size": 2, "n_splits": 2} + expected_results = [ + (np.array([0, 1, 2, 3, 4]), np.array([5, 6, 7, 8, 9, 10, 11])), + ( + np.array([1, 2, 3, 4, 5, 6, 7]), + np.array([8, 9, 10, 11, 12, 13, 14, 15]), + ), + ] + check_splits(X, y, group_numbers, cv_args, expected_results) + + +def test_partial_usage_of_data(X, y, group_numbers): + cv_args = {"test_size": 2, "train_size": 2, "n_splits": 2} + expected_results = [ + (np.array([1, 2, 3, 4, 5, 6, 7]), np.array([8, 9, 10, 11])), + (np.array([5, 6, 7, 8, 9]), np.array([10, 11, 12, 13, 14, 15])), + ] + check_splits(X, y, group_numbers, cv_args, expected_results) + + +def test_not_sorted_group_numbers(X, y, not_sorted_group_numbers): + cv_args = {"test_size": 1, "train_size": 3} + expected_results = [ + (np.array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]), np.array([10, 11, 12])), + (np.array([4, 5, 6, 7, 8, 9, 10, 11, 12]), np.array([13, 14])), + (np.array([8, 9, 10, 11, 12, 13, 14]), np.array([15])), + ] + + check_splits(X, y, not_sorted_group_numbers, cv_args, expected_results) + + +def test_not_sorted_group_names(X, y, not_sorted_group_names): + cv_args = {"test_size": 1, "train_size": 3} + expected_results = [ + (np.array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]), np.array([10, 11, 12])), + (np.array([4, 5, 6, 7, 8, 9, 10, 11, 12]), np.array([13, 14])), + (np.array([8, 9, 10, 11, 12, 13, 14]), np.array([15])), + ] + + check_splits(X, y, not_sorted_group_names, cv_args, expected_results) + + +def test_not_specified_train_size_n_splits(X, y, group_numbers): + cv_args = {"test_size": 1} + expected_results = None + error_message = "Either train_size or n_splits should be defined" + + with pytest.raises(ValueError, match=error_message): + check_splits(X, y, group_numbers, cv_args, expected_results) + + +def test_bad_window_type(X, y, group_numbers): + cv_args = { + "test_size": 1, + "train_size": 3, + "window_type": "incorrect_window_type", + } + expected_results = None + error_message = 'Window type can be either "rolling" or "expanding"' + + with pytest.raises(ValueError, match=error_message): + check_splits(X, y, group_numbers, cv_args, expected_results) + + +def test_train_size_with_expanding_window(X, y, group_numbers): + cv_args = {"test_size": 1, "train_size": 3, "window_type": "expanding"} + expected_results = None + error_message = "Train size can be specified only with rolling window" + + with pytest.raises(ValueError, match=error_message): + check_splits(X, y, group_numbers, cv_args, expected_results) + + +def test_not_specified_groups(X, y): + cv_args = {"test_size": 1, "train_size": 3} + expected_results = None + error_message = "The groups should be specified" + + with pytest.raises(ValueError, match=error_message): + check_splits(X, y, None, cv_args, expected_results) + + +def test_not_consecutive_group_numbers(X, y, not_consecutive_group_numbers): + cv_args = {"test_size": 1, "train_size": 3} + expected_results = None + error_message = "The groups should be consecutive" + + with pytest.raises(ValueError, match=error_message): + check_splits(X, y, not_consecutive_group_numbers, cv_args, expected_results) + + +def test_not_consecutive_group_names(X, y, not_consecutive_group_names): + cv_args = {"test_size": 1, "train_size": 3} + expected_results = None + error_message = "The groups should be consecutive" + + with pytest.raises(ValueError, match=error_message): + check_splits(X, y, not_consecutive_group_names, cv_args, expected_results) + + +def test_too_large_train_size_(X, y, group_numbers): + cv_args = {"test_size": 1, "train_size": 10} + expected_results = None + error_message = ( + r"Not enough data to split number of groups \(6\)" + r" for number splits \(-4\) with train size \(10\)," + r" test size \(1\), gap size \(0\), shift size \(1\)" + ) + + with pytest.raises(ValueError, match=error_message): + check_splits(X, y, group_numbers, cv_args, expected_results) + + +def test_too_large_n_splits(X, y, group_numbers): + cv_args = {"test_size": 1, "n_splits": 10} + expected_results = None + error_message = ( + r"Not enough data to split number of groups \(6\)" + r" for number splits \(10\) with train size \(-4\)," + r" test size \(1\), gap size \(0\), shift size \(1\)" + ) + + with pytest.raises(ValueError, match=error_message): + check_splits(X, y, group_numbers, cv_args, expected_results) + + +def test_too_large_train_size_n_splits(X, y, group_numbers): + cv_args = {"test_size": 1, "train_size": 10, "n_splits": 10} + expected_results = None + error_message = ( + r"Not enough data to split number of groups \(6\)" + r" for number splits \(10\) with train size \(10\)," + r" test size \(1\), gap size \(0\), shift size \(1\)" + ) + + with pytest.raises(ValueError, match=error_message): + check_splits(X, y, group_numbers, cv_args, expected_results) + + +def test_too_large_shift_size(X, y, group_numbers): + cv_args = {"test_size": 1, "n_splits": 3, "shift_size": 10} + expected_results = None + error_message = ( + r"Not enough data to split number of groups \(6\)" + r" for number splits \(3\) with train size \(-15\)," + r" test size \(1\), gap size \(0\), shift size \(10\)" + ) + + with pytest.raises(ValueError, match=error_message): + check_splits(X, y, group_numbers, cv_args, expected_results) + + +def test_too_large_gap_size(X, y, group_numbers): + cv_args = {"test_size": 1, "n_splits": 3, "gap_size": 10} + expected_results = None + error_message = ( + r"Not enough data to split number of groups \(6\)" + r" for number splits \(3\) with train size \(-7\)," + r" test size \(1\), gap size \(10\), shift size \(1\)" + ) + + with pytest.raises(ValueError, match=error_message): + check_splits(X, y, group_numbers, cv_args, expected_results) + + +def test_cross_val_score(X, y, group_numbers): + cv_args = {"test_size": 1, "train_size": 3} + cv = GroupTimeSeriesSplit(**cv_args) + + expected_scores = np.array([0, 0.5, 0.25]) + clf = DummyClassifier(strategy="most_frequent") + scoring = "accuracy" + cv_scores = cross_val_score(clf, X, y, groups=group_numbers, scoring=scoring, cv=cv) + + assert np.array_equal(cv_scores, expected_scores) diff --git a/mlxtend/evaluate/time_series.py b/mlxtend/evaluate/time_series.py new file mode 100644 index 000000000..242c83b3c --- /dev/null +++ b/mlxtend/evaluate/time_series.py @@ -0,0 +1,349 @@ +# mlxtend Machine Learning Library Extensions +# +# Time series cross validation with grouping. +# Author: Dmitry Labazkin +# +# License: BSD 3 clause + +from itertools import accumulate, chain, groupby, islice + +import matplotlib.pyplot as plt +import numpy as np +from matplotlib.patches import Patch +from matplotlib.ticker import MaxNLocator +from sklearn.utils import indexable + + +class GroupTimeSeriesSplit: + """Group time series cross-validator. + + Parameters + ---------- + test_size : int + Size of test dataset. + train_size : int (default=None) + Size of train dataset. + n_splits : int (default=None) + Number of the splits. + gap_size : int (default=0) + Gap size between train and test datasets. + shift_size : int (default=1) + Step to shift for the next fold. + window_type : str (default="rolling") + Type of the window. Possible values: "rolling", "expanding". + + Examples + ----------- + For usage examples, please see + http://rasbt.github.io/mlxtend/user_guide/evaluate/GroupTimeSeriesSplit/ + """ + + def __init__( + self, + test_size, + train_size=None, + n_splits=None, + gap_size=0, + shift_size=1, + window_type="rolling", + ): + + if (train_size is None) and (n_splits is None): + raise ValueError("Either train_size or n_splits should be defined") + + if window_type not in ["rolling", "expanding"]: + raise ValueError('Window type can be either "rolling" or "expanding"') + + if (train_size is not None) and (window_type == "expanding"): + raise ValueError("Train size can be specified only with rolling window") + + self.test_size = test_size + self.train_size = train_size + self.n_splits = n_splits + self.gap_size = gap_size + self.shift_size = shift_size + self.window_type = window_type + + def split(self, X, y=None, groups=None): + """Generate indices to split data into training and test set. + + Parameters + ---------- + X : array-like + Training data. + y : array-like (default=None) + Always ignored, exists for compatibility. + groups : array-like (default=None) + Array with group names or sequence numbers. + + Yields + ------ + train : ndarray + The training set indices for that split. + test : ndarray + The testing set indices for that split. + """ + test_size = self.test_size + gap = self.gap_size + shift_size = self.shift_size + X, y, groups = indexable(X, y, groups) + + if groups is None: + raise ValueError("The groups should be specified") + + group_names, group_lengths = zip( + *[ + (group_name, len(list(group_seq))) + for group_name, group_seq in groupby(groups) + ] + ) + n_groups = len(group_names) + + if n_groups != len(set(group_names)): + raise ValueError("The groups should be consecutive") + + self._n_groups = n_groups + group_starts_idx = chain( + [0], + islice(accumulate(group_lengths), len(group_lengths) - 1), + ) + groups_dict = dict(zip(group_names, group_starts_idx)) + n_samples = len(X) + + self._calculate_split_params() + train_size = self.train_size + n_splits = self.n_splits + train_start_idx = self._train_start_idx + train_end_idx = train_start_idx + train_size + test_start_idx = train_end_idx + gap + test_end_idx = test_start_idx + test_size + + for _ in range(n_splits): + train_idx = np.r_[ + slice( + groups_dict[group_names[train_start_idx]], + groups_dict[group_names[train_end_idx]], + ) + ] + + if test_end_idx < n_groups: + test_idx = np.r_[ + slice( + groups_dict[group_names[test_start_idx]], + groups_dict[group_names[test_end_idx]], + ) + ] + else: + test_idx = np.r_[ + slice(groups_dict[group_names[test_start_idx]], n_samples) + ] + + yield train_idx, test_idx + + if self.window_type == "rolling": + train_start_idx = train_start_idx + shift_size + + train_end_idx = train_end_idx + shift_size + test_start_idx = test_start_idx + shift_size + test_end_idx = test_end_idx + shift_size + + def get_n_splits(self, X=None, y=None, groups=None): + """Returns the number of splitting iterations in the cross-validator. + + Parameters + ---------- + X : object + Always ignored, exists for compatibility. + y : object + Always ignored, exists for compatibility. + groups : object + Always ignored, exists for compatibility. + + Returns + ------- + n_splits : int + Returns the number of splitting iterations in the cross-validator. + """ + return self.n_splits + + def _calculate_split_params(self): + train_size = self.train_size + test_size = self.test_size + n_splits = self.n_splits + gap = self.gap_size + shift_size = self.shift_size + n_groups = self._n_groups + + not_enough_data_error = ( + "Not enough data to split number of groups ({0})" + " for number splits ({1})" + " with train size ({2}), test size ({3})," + " gap size ({4}), shift size ({5})" + ) + + if (train_size is None) and (n_splits is not None): + train_size = n_groups - gap - test_size - (n_splits - 1) * shift_size + self.train_size = train_size + + if train_size <= 0: + raise ValueError( + not_enough_data_error.format( + n_groups, + n_splits, + train_size, + test_size, + gap, + shift_size, + ) + ) + train_start_idx = 0 + elif (n_splits is None) and (train_size is not None): + n_splits = (n_groups - train_size - gap - test_size) // shift_size + 1 + self.n_splits = n_splits + + if self.n_splits <= 0: + raise ValueError( + not_enough_data_error.format( + n_groups, + n_splits, + train_size, + test_size, + gap, + shift_size, + ) + ) + train_start_idx = ( + n_groups - train_size - gap - test_size - (n_splits - 1) * shift_size + ) + else: + train_start_idx = ( + n_groups - train_size - gap - test_size - (n_splits - 1) * shift_size + ) + + if train_start_idx < 0: + raise ValueError( + not_enough_data_error.format( + n_groups, + n_splits, + train_size, + test_size, + gap, + shift_size, + ) + ) + + self._train_start_idx = train_start_idx + + +def print_split_info(X, y, groups, **cv_args): + """Print information details about splits.""" + cv = GroupTimeSeriesSplit(**cv_args) + groups = np.array(groups) + + for train_idx, test_idx in cv.split(X, groups=groups): + print("Train indices:", train_idx) + print("Test indices:", test_idx) + print("Train length:", len(train_idx)) + print("Test length:", len(test_idx)) + print("Train groups:", groups[train_idx]) + print("Test groups:", groups[test_idx]) + print("Train group size:", len(set(groups[train_idx]))) + print("Test group size:", len(set(groups[test_idx]))) + print("Train group months:", X.index[train_idx].values) + print("Test group months:", X.index[test_idx].values) + print() + + +def plot_split_indices(cv, cv_args, X, y, groups, n_splits, image_file_path=None): + """Create a sample plot for indices of a cross-validation object.""" + fig, ax = plt.subplots(figsize=(12, 4)) + cmap_data = plt.cm.tab20 + cmap_cv = plt.cm.coolwarm + lw = 10 + marker_size = 200 + + for split_idx, (train_idx, test_idx) in enumerate( + cv.split(X=X, y=y, groups=groups) + ): + indices = np.array([np.nan] * len(X)) + indices[test_idx] = 1 + indices[train_idx] = 0 + + ax.scatter( + range(len(X)), + [split_idx + 0.5] * len(X), + c=indices, + marker="_", + lw=lw, + cmap=cmap_cv, + vmin=-0.4, + vmax=1.4, + s=marker_size, + ) + + ax.scatter( + range(len(X)), + [split_idx + 1.5] * len(X), + c=groups, + marker="_", + lw=lw, + cmap=cmap_data, + s=marker_size, + ) + + yticklabels = list(range(n_splits)) + ["group"] + ax.set( + yticks=np.arange(n_splits + 1) + 0.5, + yticklabels=yticklabels, + ylabel="CV iteration", + ylim=[n_splits + 1.2, -0.2], + xlim=[-0.5, len(indices) - 0.5], + ) + + ax.legend( + [Patch(color=cmap_cv(0.2)), Patch(color=cmap_cv(0.8))], + ["Training set", "Testing set"], + loc=(1.02, 0.8), + fontsize=13, + ) + + ax.set_title("{}\n{}".format(type(cv).__name__, cv_args), fontsize=15) + ax.xaxis.set_major_locator(MaxNLocator(min_n_ticks=len(X), integer=True)) + ax.set_xlabel(xlabel="Sample index", fontsize=13) + ax.set_ylabel(ylabel="CV iteration", fontsize=13) + ax.tick_params(axis="both", which="major", labelsize=13) + ax.tick_params(axis="both", which="minor", labelsize=13) + + plt.tight_layout() + + if image_file_path: + plt.savefig(image_file_path, bbox_inches="tight") + + plt.show() + + +def plot_splits(X, y, groups, image_file_path=None, **cv_args): + """Visualize splits by group.""" + cv = GroupTimeSeriesSplit(**cv_args) + cv._n_groups = len(np.unique(groups)) + cv._calculate_split_params() + n_splits = cv.n_splits + + plot_split_indices( + cv, cv_args, X, y, groups, n_splits, image_file_path=image_file_path + ) + + +def print_cv_info(cv, X, y, groups, clf, scores): + """Print information details about cross-validation usage with classifier.""" + for split_idx, (train_idx, test_idx) in enumerate(cv.split(X, y, groups)): + clf.fit(X.iloc[train_idx], y.iloc[train_idx]) + y_train_pred = clf.predict(X.iloc[train_idx]) + y_test_pred = clf.predict(X.iloc[test_idx]) + print(f"Split number: {split_idx + 1}") + print(f"Train true target: {y.iloc[train_idx].values}") + print(f"Train predicted target: {y_train_pred}") + print(f"Test true target: {y.iloc[test_idx].values}") + print(f"Test predicted target: {y_test_pred}") + print(f"Accuracy: {scores[split_idx].round(2)}") + print()