Added sp500_daily strategy, cleanups

TODOs_ROADMAP.rst

@@ -45,7 +45,7 @@ their planned release.
 --------------------------
 
 - [ ] ``DataEstimator`` needs refactoring, too long and complex methods. Target 
-  ``1.0.4``. 
+  ``1.1.1``. 
 - ``Estimator`` could define base logic for on-disk caching. By itself it
   wouldn't do anything, actual functionality implemented by forecasters' base
   class.
@@ -71,10 +71,9 @@ Partially public; only ``cvx.Gamma()`` (no arguments) and ``optimize_hyperparame
 (simple usage) are public, all the rest is not.
 
 - [ ] Clean up interface w/ ``MarketSimulator``, right now it calls private 
-  methods, maybe enough to make them public. Target ``1.0.4``.
+  methods, maybe enough to make them public. Target ``1.1.1``.
 - [ ] Add risk/fine default ``GammaTrade``, ``GammaRisk`` (which are
-  ``RangeHyperParameter``) modeled after original examples from paper. 
-  Target ``1.1.0``.
+  ``RangeHyperParameter``) modeled after original examples from paper.
 - [ ] Add ``Constant`` internal object throughout the library, also in ``DataEstimator``
   in the case of scalar; it resolves to ``current_value`` if you pass a hyper-parameter.
 - [ ] Distinguish integer and positive hyper-parameters (also enforced by Constant).
@@ -90,53 +89,63 @@ Partially public; only ``cvx.Gamma()`` (no arguments) and ``optimize_hyperparame
 Optimization policies
 ~~~~~~~~~~~~~~~~~~~~~
 
-- [ ] Improve behavior for infeasibility/unboundedness/solver error. Target 
-  ``1.1.0``.
-- [ ] Improve ``__repr__`` method, now hard to read. Target ``1.0.4``.
+- [ ] Improve behavior for infeasibility/unboundedness/solver error. Idea:
+  optimization policy gets arguments ``infeasible_fallback``, ... which are
+  policies (default to ``cvx.Hold``), problem is that this breaks
+  compatibility, it doesn't if we don't give defaults (so exceptions are raised
+  all the way to the caller), but then it's extra complication (more 
+  arguments). Consider for ``2.0.0``.
+- [ ] Improve ``__repr__`` method, now hard to read. Target ``1.1.1``.
 
 ``cvxportfolio.constraints``
 ----------------------------
 
-- [ ] Add missing constraints from the paper. Target ``1.1.0``.
+- [ ] Add missing constraints from the paper.
 - [ ] Make ``MarketNeutral`` accept arbitrary benchmark (policy object).
 
 ``cvxportfolio.result``
 -----------------------
 
 - [ ] Make ``BackTestResult`` interface methods with ``MarketSimulator`` 
   public. 
-- [ ] Add a ``backruptcy`` property (boolean). Amend ``sharpe_ratio``
+- [ ] Add a ``bankruptcy`` property (boolean). Amend ``sharpe_ratio``
   and other aggregate statistics (as best as possible) to return ``-np.inf``
   if back-test ended in backruptcy. This is needed specifically for
-  hyper-parameter optimization. Target ``1.0.4``.
-- [ ] Capture **logs** from the back-test; add ``logs`` property that returns
-  then as a string (newline separated, like a .log file). Make log level
+  hyper-parameter optimization. Target ``1.1.1``.
+- [X] Capture **logs** from the back-test; add ``logs`` property that returns
+  them as a string (newline separated, like a .log file). Make log level
   changeable by a module constant (like ``cvxportfolio.result.LOG_LEVEL``) set
   to ``INFO`` by default. Then, improve logs throughout (informative, proactive
   on possible issues). Logs formatter should produce source module and
   timestamp.
 
+Other 
+-----
+
+- [ ] Exceptions are not too good, probably ``cvxportfolio.DataError`` should
+  be ``ValueError``, .... Research this, one option is to simply derive from
+  built-ins (``class DataError(ValueError): pass``), .... No compatibility
+  breaks.
 
 Development & testing
 ---------------------
 
 - [ ] Add extra pylint checkers. 
 
-  - [ ] Code complexity. Target ``1.0.4``. 
+  - [ ] Code complexity. Target ``1.1.1``. 
 - [ ] Consider removing downloaded data from ``test_simulator.py``,
   so only ``test_data.py`` requires internet. 
 
 Documentation
 -------------
 
 - [ ] Improve examples section, also how "Hello world" is mentioned in readme.
-  Target ``1.0.4``, PR #118.
 - [ ] Manual.
 - [ ] Quickstart, probably to merge into manual.
 
 Examples
 --------
 
-- [ ] Restore examples from paper. Target ``1.0.4``, PR #118.
-- [ ] Expose more (all?) examples through HTML docs. Target ``1.0.4``, PR #118.
+- [ ] Finish restore examples from paper. Target ``1.1.1``.
+- [ ] Expose more (all?) examples through HTML docs.
 - [ ] Consider making examples a package that can be pip installed.

cvxportfolio/data.py

@@ -803,6 +803,16 @@ def partial_universe_signature(self, partial_universe):
         """
         return None
 
+# compiled based on Interactive Brokers benchmark rates choices
+# (see https://www.ibkrguides.com/kb/article-2949.htm)
+# and their FRED codes
+RATES = {
+    'USDOLLAR': 'DFF', # Federal funds effective rate
+    'EURO': 'ECBESTRVOLWGTTRMDMNRT', # BCE short term rate
+    'GBPOUND': 'IUDSOIA', # SONIA
+    'JPYEN': 'IRSTCB01JPM156N', # updated monthly
+    }
+
 class MarketDataInMemory(MarketData):
     """Market data that is stored in memory when initialized."""
 
@@ -919,9 +929,10 @@ def _add_cash_column(self, cash_key, grace_period):
         objective term.
         """
 
-        if not cash_key == 'USDOLLAR':
+        if not cash_key in RATES:
             raise NotImplementedError(
-                'Currently the only data pipeline built is for USDOLLAR cash')
+                'Currently the only data pipelines built are for cash_key'
+                f' in {list(RATES)}')
 
         if self.returns.index.tz is None:
             raise DataError(
@@ -937,7 +948,9 @@ def _add_cash_column(self, cash_key, grace_period):
                 + " its name.")
 
         data = Fred(
-            'DFF', base_location=self.base_location, grace_period=grace_period)
+            RATES[cash_key], base_location=self.base_location,
+            grace_period=grace_period)
+
         cash_returns_per_period = resample_returns(
             data.data/100, periods=self.periods_per_year)
 

cvxportfolio/result.py

@@ -233,7 +233,7 @@ def _log_final(self, t, t_next, h, extra_simulator_time):
     #
 
     @property
-    def log(self):
+    def logs(self):
         """Logs from the policy, simulator, market data server, ....
 
         :return: Logs produced during the back-test, newline separated.

cvxportfolio/simulator.py

@@ -200,10 +200,13 @@ def simulate(
 
         # translate to weights
         current_portfolio_value = sum(h)
+        logger.info(
+            'Portfolio value at time %s: %s', t, current_portfolio_value)
         current_weights = pd.to_numeric(h / current_portfolio_value)
 
         # evaluate the policy
         s = time.time()
+        logger.info('Evaluating the policy at time %s', t)
         policy_w = policy.values_in_time_recursive(
             t=t, current_weights=current_weights,
             current_portfolio_value=current_portfolio_value,
@@ -482,6 +485,7 @@ def modify_orig_policy(target_policy):
             print('iteration', i)
             # print('Current optimal hyper-parameters:')
             # print(policy)
+            logger.info('Current policy: %s', policy)
             print('Current objective:')
             print(current_objective)
             # print()

docs/result.rst

@@ -88,7 +88,7 @@ Back-test result
 
     .. autoproperty:: simulator_times
 
-    .. autoproperty:: log
+    .. autoproperty:: logs
 
     .. automethod:: plot
 

examples/strategies/dow30_daily.py

@@ -100,7 +100,6 @@ def policy(gamma_risk, gamma_trade):
         research_sim.optimize_hyperparameters(
             research_policy, start_time=HYPERPAR_OPTIMIZE_START,
             objective='sharpe_ratio')
-            #objective='information_ratio')
 
         result_opt = research_sim.backtest(
             research_policy, start_time=HYPERPAR_OPTIMIZE_START)

examples/strategies/sp500_daily.py

@@ -0,0 +1,94 @@
+# Copyright 2023 Enzo Busseti
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""This is a simple example strategy which we run every day.
+
+It is a long-only, unit leverage, allocation on the Standard and Poor's 500
+universe. It's very similar to the two strategies ``dow30_daily`` and
+``ndx100_daily``, but here we also constrain the allocation to be close
+to our chosen benchmark, :class:`cvxportfolio.MarketBenchmark` (allocation
+proportional to last year's total market volumes in dollars).
+
+This strategy also seems to have outperformed our benchmarks and an index ETF.
+We will see how it performs online.
+
+You run it from the root of the repository in the development environment by:
+
+.. code:: bash
+
+    python -m examples.strategies.sp500_daily
+"""
+
+import cvxportfolio as cvx
+
+from ..universes import SP500
+
+HYPERPAR_OPTIMIZE_START = '2023-01-01'
+
+OBJECTIVE = 'sharpe_ratio'
+
+def policy(gamma_risk, gamma_trade):
+    """Create fresh policy object, also return handles to hyper-parameters.
+
+    :param gamma_risk: Risk aversion multiplier.
+    :type gamma_risk: float
+    :param gamma_trade: Transaction cost aversion multiplier.
+    :type gamma_trade: float, optional
+
+    :return: Policy object and dictionary mapping hyper-parameter names (which
+        must match the arguments of this function) to their respective objects.
+    :rtype: tuple
+    """
+    gamma_risk_hp = cvx.Gamma(initial_value=gamma_risk)
+    gamma_trade_hp = cvx.Gamma(initial_value=gamma_trade)
+    return cvx.SinglePeriodOptimization(
+        cvx.ReturnsForecast()
+        - gamma_risk_hp * cvx.FullCovariance()
+        - gamma_trade_hp * cvx.StocksTransactionCost(),
+        [cvx.LongOnly(), cvx.LeverageLimit(1),
+            cvx.MaxBenchmarkDeviation(0.05),
+            cvx.MinBenchmarkDeviation(-0.05)],
+        benchmark=cvx.MarketBenchmark(),
+        ignore_dpp=True,
+    ), {'gamma_risk': gamma_risk_hp, 'gamma_trade': gamma_trade_hp}
+
+
+if __name__ == '__main__':
+
+    RESEARCH = False
+
+    if RESEARCH:
+        INDEX_ETF = 'SPY'
+
+        research_sim = cvx.StockMarketSimulator(SP500)
+
+        result_unif = research_sim.backtest(
+            cvx.Uniform(), start_time=HYPERPAR_OPTIMIZE_START)
+        print('uniform')
+        print(result_unif)
+
+        result_market = research_sim.backtest(
+            cvx.MarketBenchmark(), start_time=HYPERPAR_OPTIMIZE_START)
+        print('market')
+        print(result_market)
+
+        result_etf = cvx.StockMarketSimulator([INDEX_ETF]).backtest(
+            cvx.Uniform(), start_time=HYPERPAR_OPTIMIZE_START)
+        print(INDEX_ETF)
+        print(result_etf)
+
+    from .strategy_executor import main
+    main(policy=policy, hyperparameter_opt_start=HYPERPAR_OPTIMIZE_START,
+        objective=OBJECTIVE, universe=SP500, initial_values={
+            'gamma_risk': 30., 'gamma_trade': 1.
+        })

examples/strategies/sp500_daily_hyper_parameters.json

@@ -0,0 +1,6 @@
+{
+    "2023-12-29 14:30:00+00:00": {
+        "gamma_risk": 58.46151300000004,
+        "gamma_trade": 1.9487171000000012
+    }
+}