Merge pull request #327 from behnam-zakeri/cleanup-macro

Clean up message_ix.macro

.github/workflows/publish.yaml

@@ -40,15 +40,15 @@ jobs:
         twine check dist/*
 
     - name: Publish to TestPyPI
-      uses: pypa/gh-action-pypi-publish@v1.4.1
+      uses: pypa/gh-action-pypi-publish@v1.5.0
       if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags')
       with:
         user: __token__
         password: ${{ secrets.TESTPYPI_TOKEN }}
         repository_url: https://test.pypi.org/legacy/
 
     - name: Publish to PyPI
-      uses: pypa/gh-action-pypi-publish@v1.4.1
+      uses: pypa/gh-action-pypi-publish@v1.5.0
       if: github.event_name == 'release'
       with:
         user: __token__

RELEASE_NOTES.rst

@@ -1,9 +1,16 @@
 Next release
 ============
 
+Migration notes
+---------------
+
+The format of input data files for MACRO calibration has been changed in :pull:`327`.
+Files compatible with v3.4.0 and earlier will not work with this version and should be updated; see details of the current data file format in the :doc:`documentation <macro>`.
+
 All changes
 -----------
 
+- Improve configurability of :mod:`.macro`; see the :doc:`documentation <macro>` (:pull:`327`).
 - Split :meth:`.Reporter.add_tasks` for use without an underlying :class:.`Scenario` (:pull:`567`).
 - Allow setting the “model_dir” and “solve_options” options for :class:`.GAMSModel` (and subclasses :class:`.MESSAGE`, :class:`.MACRO`, and :class:`.MESSAGE_MACRO`) through the user's ixmp configuration file; expand documentation (:pull:`557`).
 

doc/contributing.rst

@@ -232,6 +232,7 @@ Documentation
   - Check that intersphinx links are correctly resolved, by building the docs and attempting to click new or modified links.
 
 - Write docstrings in the `numpydoc <https://numpydoc.readthedocs.io/en/latest/format.html>`_ style.
+  This implies also `PEP 257 <https://peps.python.org/pep-0257/>`_; see in particular the format for `multi-line docstrings <https://peps.python.org/pep-0257/#multi-line-docstrings>`_.
 
   Use single backticks to refer to function arguments, and asterisks for italics:
 

doc/macro.rst

@@ -1,17 +1,53 @@
-Tune MESSAGE-MACRO
-******************
+Calibrate and tune MESSAGE-MACRO
+********************************
 
 “MESSAGE-MACRO” refers to the combination of MESSAGE and MACRO, run iteratively in a multi-disciplinary optimization algorithm.
 This combination is activated by calling :meth:`.solve` with the argument `model='MESSAGE-MACRO'`, or using the GAMS :file:`MESSAGE-MACRO_run.gms` script directly (see :ref:`running` for details about these two methods).
 
-This page describes how to solve two numerical issues that can occur in large |MESSAGEix| models.
-
 .. contents::
    :local:
 
+.. _macro-input-data:
+
+Input data file
+===============
+
+Using :meth:`.add_macro` requires an input data file in Microsoft Excel format.
+The format of this file is largely the same as for ixmp :ref:`ixmp:excel-data-format`, and includes the following sheets.
+For example input data files, see the files :file:`message_ix/tests/data/*_macro_input.xlsx` included as part of the :mod:`message_ix` test suite; either in your local installation, or `here on GitHub <https://github.com/iiasa/message_ix/tree/main/message_ix/tests/data>`_.
+
+“``config``” sheet
+   This configuration sheet specifies MACRO-related nodes and years, and mapps MACRO sectors to MESSAGE commodities and levels.
+   The ``config`` sheet has five columns, each of which is a list of labels/codes for a corresponding :ref:`ixmp set <ixmp:data-model-data>`:
+
+   - "node", "year": these can each have any length, depending on the number of regions and years to be included in the MACRO calibration process.
+   - "sector", "commodity", "level": these 3 columns must have equal lengths.
+     They describe a one-to-one mapping between MACRO sectors (entries in the "sector" column) and MESSAGE commodities and levels (paired entries in the "commodity" and "level" columns).
+
+MACRO parameter sheets
+   The remaining sheets each contain data for one MACRO parameter:
+
+   - ``price_ref``: prices of MESSAGE commodities in a reference year.
+     This reference year is a year for which commodity prices and energy system cost are known for a given demand of those commodities.
+   - ``cost_ref``: total cost of the energy system in the reference year.
+   - ``demand_ref``: demand for different commodities in the referenbce year.
+   - ``lotol``: tolerance factor for lower bounds on MACRO variabales.
+   - ``esub``: elasticity between capital-labor and energy.
+   - ``drate``: social discount rate.
+   - ``depr``: annual percent depreciation.
+   - ``kpvs``: capital value share parameter.
+   - ``kgdp``: initial capital to GDP ratio in base year.
+   - ``gdp_calibrate``: trajectory of GDP in optimization years calibrated to energy demand to MESSAGE.
+   - ``aeei``: annual potential decrease of energy intensity in sector sector.
+   - ``MERtoPPP``: conversion factor of GDP from market exchange rates to purchasing power parity.
+
+Numerical issues
+================
+
+This section describes how to solve two numerical issues that can occur in large |MESSAGEix| models.
 
 Oscillation detection in the MESSAGE-MACRO algorithm
-====================================================
+----------------------------------------------------
 The documentation for the :class:`.MESSAGE_MACRO` class describes the algorithm and its three parameters:
 
 - `convergence_criterion`,
@@ -43,21 +79,21 @@ This has the effect of reducing the allowable relative change in demands, until
 
 
 Issue 1: Oscillations not detected
-==================================
+----------------------------------
 
 Oscillation detection can fail, especially when the oscillation is very small.
 When this occurs, MESSAGE-MACRO will iterate until `max_iteration` (default ``50``) and then print a message indicating that it has not converged.
 
 For the MESSAGEix-GLOBIOM global model, this issue can be encountered with scenarios which have stringent carbon budgets (e.g. <1000 Gt CO₂ cumulative) and require more aggressive reductions of demands.
 
 Identifying oscillation
------------------------
+~~~~~~~~~~~~~~~~~~~~~~~
 
 In order to find out whether failure to converge is due to undetected oscillation, check the iteration report in :file:`MsgIterationReport_<{model_name}>_<{scenario_name}>.gdx`.
 The initial iterations will show the objective function value either decreasing or increasing (depending on the model), but after a number of iterations, the objective function will flip-flop between two very similar values.
 
 Preventing oscillation
-----------------------
+~~~~~~~~~~~~~~~~~~~~~~
 
 The issue can be resolved by tuning `max_adjustment` and `convergence_criterion` from their respective default values of ``0.2`` (20%) and ``0.01`` (1%).
 The general approach is to **reduce max_adjustment**.
@@ -74,7 +110,7 @@ These changes can be made in two ways:
 
 
 Issue 2: MESSAGE solves optimally with unscaled infeasibilities
-===============================================================
+---------------------------------------------------------------
 
 By default, :mod:`message_ix` is configured so that the CPLEX solver runs using the `lpmethod` option set to ``2``, selecting the dual simplex method.
 Solving models the size of MESSAGEix-GLOBIOM takes very long with the dual simplex method—scenarios with stringent constraints can take >10 hours on common hardware.
@@ -100,3 +136,13 @@ When this is not possible, there are some workarounds:
 3. Start the MESSAGE-MACRO algorithm with `lpmethod` set to ``4``.
    Manually monitor its progress, and after approximately 10 iterations have passed, delete the file :file:`cplex.opt`.
    When CPLEX can not find its option file, it will revert to using a simplex method (and advanced basis) from thereon.
+
+
+:mod:`message_ix.macro` internals
+=================================
+
+.. currentmodule:: message_ix.macro
+
+.. automodule:: message_ix.macro
+   :members:
+   :exclude-members: MACRO_ITEMS

message_ix/core.py

@@ -566,8 +566,25 @@ def solve(self, model="MESSAGE", solve_options={}, **kwargs):
         super().solve(model=model, solve_options=solve_options, **kwargs)
 
     def add_macro(self, data, scenario=None, check_convergence=True, **kwargs):
-        """Add MACRO parametrization to the Scenario and calibrate."""
-        # TODO document
+        """
+        Add MACRO parametrization to the Scenario and calibrate.
+        Notice: existing MACRO calibration data will be overwritten by running this.
+
+        Parameters
+        ----------
+        data : dict (str -> DataFrame or Series)
+            Dictionary of required data for MACRO calibration.
+        scenario : string, optional, default: None.
+            Scenario name for calibrated MESSAGEix scenario.
+        check_convergence : bool, optional, default: True.
+            The calibrated scenario solves in one iteration.
+        kwargs
+            Solve options when solving the calibrated scenario.
+
+        See also
+        --------
+        :ref:`macro-input-data`
+        """
         from .macro import EXPERIMENTAL, add_model_data, calibrate
         from .models import MACRO