Implementation of generic storage equations (#190)

- Add mapping of time and period
- Update data load for storage mappings
- Update data load and parameter/sets def
- Update model core
- Add unit test and explanation
- Add a document for illustration
- Update and clean time mapping
- Update data load
- Update and clean model core
- Clean sets and parameter definition text
- Add to release notes
- Clean up text in GAMS
- Move storage init to core.py for use in all scenarios
- Cleanup of test_storage
- Remove commodity index from storage bounds and losses
- Add relation_storage as an option (storage_equality removed)
- Map storage_balance to map_tec_storage_level
- Add inline documentation to model_core
- Clean up GAMS files before rebasing
- Small modifications after rebasing
- Changes related to initialization of storage
- Clean up notation and complete documentation
- Improve tests to consider the initial value
- Improve mapping of storage level for storage with no loss
- Update asserts in test_reporting explicitly
- Cleanup of documentation and GAMS core
- Update test_legacy_version for changes in GAMS
- Improve text of test and documentation
- Consider lvl_temporal directly in parameter time_seq
- Make the connection between the last and first time steps (storage cycle)
- Remove second 'year' from index of time_period mapping
- Move init_storage to models.MESSAGE.initialize
- Minor code cleanup
- Use ixmp.Model.initialize_items()
- Update hard coded lengths in test_reporting and test_tutorials
- Rename test_storage and move to tests folder
- Accept and check an optional scheme kwarg to message_ix.Scenario
- Cleanup after rebase
  - Code removed by 56f7bd5
  - Code removed by 3bd21e5
- Restore line endings of period_parameter_assignment.gms
- Simplify the mapping of charge-discharge technologies to storage
- Remove map_time_period from include folder to data_load.gms
- Correct the mapping of initial storage content
- Change the sign of relation_storage and some text edits
- Remove new bound definitions
- Consider commodity in equations
- Correct linter errors
- Update period_parameter_assignment.gms
- Rename and cleanup storage equivalence equation
- Update reporting tests for new lengths
- Update Scenario constructor per iiasa/ixmp#315
  - Ensures storage-related parameters are added to any Scenario loaded;
    this allows test_legacy_scenario to pass.
- Rename storage.PNG to storage.png, consistent with usage in docs
- Correct ReST syntax for storage docs in model_core.gms
  - Reference label with '-' instead of spaces.
  - Underlines match heading length.
  - Space between paragraph text and '..'.
  - Correct relative path to figure storage.png.
- Update coding style guide for docs, GAMS code
- Update release notes with migration note; add storage items to docs
- Add autosummary to model docs
- Remove storage relations; to be modeled by existing parameters
- Update reporting length numbers
- References to message_ix in RELEASE_NOTES.rst
- Add documentation of storage params
- Change test assert values to pass CI
- Add documentation of sets for storage
- Edit nomenclature of storage equations
- Correct typo in COMMODITY_BALANCE equation
- Add one missing underline symbol in params section title
- Update test_reporting.py
- Edit text for parameters of storage
- Address Sphinx warnings

Co-authored-by: Matthew Gidden <[email protected]>
Co-authored-by: Paul Natsuo Kishimoto <[email protected]>
Co-authored-by: Francesco Lovat <[email protected]>

CONTRIBUTING.rst

@@ -117,13 +117,22 @@ Other tips
 Code style
 ----------
 
-- Python: follow `PEP 8 <https://www.python.org/dev/peps/pep-0008/>`_.
-- R: follow the style of the existing code base.
-- Jupyter notebooks (``.ipynb``): see below, under `Contributing tutorials`_.
-- Documentation (``.rst``, ``.md``):
+- **Python:**
+
+   - Follow `PEP 8 <https://www.python.org/dev/peps/pep-0008/>`_.
+   - Docstrings are in the `numpydoc format <https://numpydoc.readthedocs.io/en/latest/format.html>`_.
+
+- **R:** follow the style of the existing code base.
+- Jupyter notebooks (:file:`.ipynb`): see below, under `Contributing tutorials`_.
+- **Documentation** for ReStructuredText in :file:`.rst` files, and inline in :file:`.gms` files:
 
   - Do not hard-wrap lines.
   - Start each sentence on a new line.
+  - Ensure Sphinx does not give warnings about ReST syntax for new or modified documentation.
+
+- GAMS:
+
+  - Wrap lines at 121 characters, except for inline documentation (see above).
 
 - Other (file names, CLI, etc.): follow the style of the existing code base.
 

RELEASE_NOTES.rst

@@ -1,31 +1,40 @@
 Next release
 ============
 
+Migration notes
+---------------
+
+The :ref:`generic storage formulation <gams-storage>` introduces **new ixmp items** (sets, parameters, variables, and equations) to the MESSAGE model scheme.
+When loading a Scenario created with a version of `message_ix` older than x.x.x, these items will be initialized (and left empty), using at most one call to :meth:`~message_ix.Scenario.commit`.
+See :meth:`.MESSAGE.initialize`.
+
+
 All changes
 -----------
 
+- :pull:`190`: Add generic mathematical formulation of :ref:`technologies that store commodities <gams-storage>`, such as water and energy.
 - :pull:`343`, :pull:`345`: Accept :class:`.MESSAGE_MACRO` iteration control parameters through :meth:`.solve`; document how to tune these to avoid numerical issues.
-- :pull:`340`: Allow cplex.opt to be used by message_ix from multiple processes.
+- :pull:`340`: Allow cplex.opt to be used by `message_ix` from multiple processes.
 - :pull:`328`: Expand automatic reporting of emissions prices and mapping sets; improve robustness of :meth:`Reporter.convert_pyam`.
-- :pull:`321`: Move :meth:`.Scenario.to_excel`, :meth:`.read_excel` to :class:`ixmp.Scenario`; they continue to work with message_ix.Scenario.
+- :pull:`321`: Move :meth:`.Scenario.to_excel`, :meth:`.read_excel` to :class:`ixmp.Scenario`; they continue to work with :class:`message_ix.Scenario`.
 - :pull:`323`: Add `units`, `replace_vars` arguments to :meth:`.Reporter.convert_pyam`.
 - :pull:`308`: Expand automatic reporting of add-on technologies.
-- :pull:`313`: Include all tests in the message_ix package.
+- :pull:`313`: Include all tests in the `message_ix` package.
 - :pull:`307`: Adjust to deprecations in ixmp 2.0.
 - :pull:`223`: Add methods for parametrization and calibration of MACRO based on an existing MESSAGE Scenario.
 
 
 v2.0.0 (2020-01-14)
 ===================
 
-message_ix v2.0.0 coincides with ixmp v2.0.0.
+`message_ix` v2.0.0 coincides with `ixmp` v2.0.0.
 
 Migration notes
 ---------------
 
 Support for **Python 2.7 is dropped** as it has reached end-of-life, meaning no further releases will be made even to fix bugs.
 See `PEP-0373 <https://www.python.org/dev/peps/pep-0373/>`_ and https://python3statement.org.
-``message_ix`` users must upgrade to Python 3.
+`message_ix` users must upgrade to Python 3.
 
 **Command-line interface (CLI).** Use ``message-ix`` as the program for all command-line operations:
 

doc/source/api.rst

@@ -82,15 +82,21 @@ Model classes
 
 .. currentmodule:: message_ix.models
 
-.. automodule:: message_ix.models
-   :exclude-members: MESSAGE, MESSAGE_MACRO
+.. autosummary::
+
+   MESSAGE
+   MACRO
+   MESSAGE_MACRO
+   GAMSModel
+   DEFAULT_CPLEX_OPTIONS
+   MESSAGE_ITEMS
 
 .. autodata:: DEFAULT_CPLEX_OPTIONS
 
    These configure the GAMS CPLEX solver (or another solver, if selected); see `the solver documentation <https://www.gams.com/latest/docs/S_CPLEX.html>`_ for possible values.
 
 .. autoclass:: MESSAGE
-   :members:
+   :members: initialize
    :exclude-members: defaults
    :show-inheritance:
 
@@ -125,6 +131,8 @@ Model classes
       **var_list**       :obj:`None`
       ================== ===
 
+.. autoclass:: MACRO
+
 .. autoclass:: MESSAGE_MACRO
    :members:
    :show-inheritance:
@@ -149,6 +157,13 @@ Model classes
    :exclude-members: defaults
    :show-inheritance:
 
+.. autodata:: MESSAGE_ITEMS
+
+   Currently, these only include items required for :ref:`storage <gams-storage>`, in order to ensure that all scenarios are compatible with the MESSAGE GAMS implementation.
+
+   .. seealso:: :meth:`.MESSAGE.initialize`
+
+
 
 .. _utils:
 

doc/source/conf.py

@@ -315,7 +315,7 @@ def setup(app):
 # -- Options for sphinx.ext.intersphinx ---------------------------------------
 
 intersphinx_mapping = {
-    'dask': ('http://docs.dask.org/en/stable/', None),
+    'dask': ('https://docs.dask.org/en/stable/', None),
     'ixmp': ('https://message.iiasa.ac.at/projects/ixmp/en/latest/', None),
     'pandas': ('https://pandas.pydata.org/pandas-docs/stable/', None),
     'pint': ('https://pint.readthedocs.io/en/stable/', None),

message_ix/core.py

@@ -20,21 +20,31 @@ class Scenario(ixmp.Scenario):
     """|MESSAGEix| Scenario.
 
     See :class:`ixmp.TimeSeries` for the meaning of arguments `mp`, `model`,
-    `scenario`, `version`, and `annotation`; :class:`ixmp.Scenario` for the
-    meaning of `cache`. The `scheme` of a newly-created Scenario is always
-    'MESSAGE'.
+    `scenario`, `version`, and `annotation`. The `scheme` of a newly-created
+    Scenario is always 'MESSAGE'.
     """
     def __init__(self, mp, model, scenario=None, version=None, annotation=None,
-                 cache=False):
+                 scheme=None, **kwargs):
         # If not a new scenario, use the scheme stored in the Backend
-        scheme = 'MESSAGE' if version == 'new' else None
-
-        # `ixmp.Scenario` verifies that MESSAGE-scheme scenarios are
-        # initialized as `message_ix.Scenario` for correct API
-        self.is_message_scheme = True
+        if version == 'new':
+            scheme = scheme or 'MESSAGE'
+
+        if scheme not in ('MESSAGE', None):
+            msg = f'Instantiate message_ix.Scenario with scheme {scheme}'
+            raise ValueError(msg)
+
+        super().__init__(
+            mp=mp,
+            model=model,
+            scenario=scenario,
+            version=version,
+            annotation=annotation,
+            scheme=scheme,
+            **kwargs
+        )
 
-        super().__init__(mp, model, scenario, version, scheme, annotation,
-                         cache)
+        # Scheme returned by database
+        assert self.scheme == 'MESSAGE', self.scheme
 
     # Utility methods used by .equ(), .par(), .set(), and .var()
 

message_ix/model/MESSAGE/data_load.gms

@@ -18,6 +18,7 @@ $LOAD inv_tec, renewable_tec
 $LOAD balance_equality
 $LOAD shares
 $LOAD addon, type_addon, cat_addon, map_tec_addon
+$LOAD storage_tec, level_storage, map_tec_storage
 $GDXIN
 
 Execute_load '%in%'
@@ -69,6 +70,8 @@ demand_fixed=demand
 * fixing variables to pre-specified values
 is_fixed_extraction, is_fixed_stock, is_fixed_new_capacity, is_fixed_capacity, is_fixed_activity, is_fixed_land
 fixed_extraction, fixed_stock, fixed_new_capacity, fixed_capacity, fixed_activity, fixed_land
+* storage parameters
+storage_initial, storage_self_discharge, time_order
 ;
 
 
@@ -158,6 +161,17 @@ addon_up(node,tec,year_all,mode,time,type_addon)$(
 emission_scaling(type_emission,emission)$( cat_emission(type_emission,emission)
         and not emission_scaling(type_emission,emission) ) = 1 ;
 
+* mapping of storage technologies to their level and commodity (can be different from level and commodity of storage media)
+map_time_commodity_storage(node,tec,level,commodity,mode,year_all,time)$( storage_tec(tec) AND
+    SUM( (node2,year_all2,time_act), input(node2,tec,year_all,year_all2,mode,node,commodity,level,time_act,time) ) ) = yes;
+
+* mapping of sequence of sub-annual timesteps in a period and temporal level
+map_time_period(year_all,lvl_temporal,time,time2)$( time_order(lvl_temporal,time) AND
+     time_order(lvl_temporal,time) + 1 = time_order(lvl_temporal,time2) ) = yes;
+
+* mapping of sequence of the last sub-annual timestep to the first to create a close the order of timesteps
+map_time_period(year_all,lvl_temporal,time,time2)$( time_order(lvl_temporal,time) AND
+     time_order(lvl_temporal,time) = SMAX(time3,time_order(lvl_temporal,time3) ) AND time_order(lvl_temporal,time2) = 1 ) = yes;
 *----------------------------------------------------------------------------------------------------------------------*
 * sanity checks on the data set                                                                                        *
 *----------------------------------------------------------------------------------------------------------------------*