Report CO₂ emissions from transportation

.github/workflows/pytest.yaml

@@ -110,6 +110,7 @@ jobs:
     steps:
     - uses: actions/checkout@v3
     - uses: actions/setup-python@v4
+      with: { python-version: "3.11" }
 
     - name: Force recreation of pre-commit virtual environment for mypy
       if: github.event_name == 'schedule'  # Comment this line to run on a PR

.gitignore

@@ -139,6 +139,8 @@ dmypy.json
 # Generated and temporary data files
 debug/
 cache/
+# LibreOffice lock files
+.~lock*#
 
 # VSCode settings
 .vscode

doc/Makefile

@@ -12,9 +12,12 @@ BUILDDIR      = _build
 help:
 	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
 
-.PHONY: help Makefile
+clean-autosummary:
+	find . -name "$(BUILDDIR)" -prune -o -iname _autosummary -print0 | xargs -0 rmdir
+
+.PHONY: help clean-autosummary Makefile
 
 # Catch-all target: route all unknown targets to Sphinx using the new
 # "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
-%: Makefile
+.DEFAULT: Makefile
 	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

doc/api/report/index.rst

@@ -1,39 +1,54 @@
 Reporting (:mod:`~.message_ix_models.report`)
 *********************************************
 
+On this page:
+
 .. contents::
    :local:
 
-See also:
+Elsewhere:
 
 - ``global.yaml``, the :doc:`default-config`.
 - Documentation for :mod:`genno` (:doc:`genno:index`), :mod:`ixmp.reporting`, and :mod:`message_ix.reporting`.
+- Reporting of specific model variants:
+
+  - :mod:`.water.reporting`
+  - (Private) :doc:`Reporting of message_data.model.transport <m-data:reference/model/transport/report>`
 
 .. toctree::
    :hidden:
 
    default-config
 
-Not public:
-
-- `“Reporting” project board <https://github.com/orgs/iiasa/projects/3>`_ on GitHub for the initial implementation of these features.
-- :doc:`m-data:/reference/tools/post_processing`, still in use.
-- Documentation for reporting specific to certain model variants:
-
-  - :doc:`m-data:/reference/model/transport/report`
+.. _report-intro:
 
 Introduction
 ============
 
 See :doc:`the discussion in the MESSAGEix docs <message-ix:reporting>` about the stack.
-In short, :mod:`message_ix` must not contain reporting code that references ``coal_ppl``, because not every model built on the MESSAGE framework will have a technology with this name.
-Any reporting specific to ``coal_ppl`` must be in :mod:`message_ix_models`, since all models in the MESSAGEix-GLOBIOM family will have this technology.
+In short, for instance:
+
+- :mod:`message_ix` **must not** contain reporting code that references :py:`technology="coal_ppl"`, because not every model built on the MESSAGE framework will have a technology with this name.
+- Any model in the MESSAGEix-GLOBIOM family—built with :mod:`message_ix_models` and/or :mod:`message_data`—**should**, with few exceptions, have a :py:`technology="coal_ppl"`, since this appears in the common list of :ref:`technology-yaml`.
+  Reporting specific to this technology ID, *as it is represented* in this model family, should be in :mod:`message_ix_models` or user code.
 
 The basic **design pattern** of :mod:`message_ix_models.report` is:
 
-- A ``global.yaml`` file (i.e. in `YAML <https://en.wikipedia.org/wiki/YAML#Example>`_ format) that contains a *concise* yet *explicit* description of the reporting computations needed for a MESSAGE-GLOBIOM model.
-- :func:`~.report.prepare_reporter` reads the file and a Scenario object, and uses it to populate a new Reporter.
-  This function mostly relies on the :doc:`configuration handlers <genno:config>` built in to Genno to handle the different sections of the file.
+- :func:`~.report.prepare_reporter` populates a new :class:`~.message_ix.Reporter` for a given |Scenario| with many keys to report all quantities of interest in a MESSAGEix-GLOBIOM–family model.
+- This function relies on *callbacks* defined in multiple submodules to add keys and tasks for general or tailored reporting calculations and actions.
+  Additional modules **should** define callback functions and register them with :func:`~report.register` when they are to be used.
+  For example:
+
+  1. The module :mod:`message_ix_models.report.plot` defines :func:`.plot.callback` that adds standard plots to the Reporter.
+  2. The module :mod:`message_data.model.transport.report` defines :func:`~.message_data.model.transport.report.callback` that adds tasks specific to MESSAGEix-Transport.
+  3. The module :mod:`message_data.projects.navigate.report` defines :func:`~.message_data.projects.navigate.report.callback` that add tasks specific to the ‘NAVIGATE’ research project.
+
+  The callback (1) is always registered, because these plots are always applicable and can be expected to function correctly for all models in the family. In contrast, (2) and (3) **should** only be registered and run for the specific model variants for which they are developed/intended.
+
+  Modules with tailored reporting configuration **may** also be indicated on the :ref:`command line <report-cli>` by using the :program:`-m/--modules` option: :program:`mix-models report -m model.transport`.
+
+- A file :file:`global.yaml` file (in `YAML <https://en.wikipedia.org/wiki/YAML#Example>`_ format) contains a description of some of the reporting computations needed for a MESSAGE-GLOBIOM model.
+  :func:`~.report.prepare_reporter` uses the :doc:`configuration handlers <genno:config>` built into :mod:`genno` (and some extensions specific to :mod:`message_ix_models`) to handle the different sections of the file.
 
 Features
 ========
@@ -67,24 +82,6 @@ Units
        base: example_var:a-b-c
        units: kJ
 
-Continuous reporting
-====================
-
-.. note:: This section is no longer current.
-
-The IIASA TeamCity build server is configured to automatically run the full (:file:`global.yaml`) reporting on the following scenarios:
-
-.. literalinclude:: ../../ci/report.yaml
-   :caption: :file:`ci/report.yaml`
-   :language: yaml
-
-This takes place:
-
-- every morning at 07:00 IIASA time, and
-- for every commit on every pull request branch, *if* the branch name includes ``report`` anywhere, e.g. ``feature/improve-reporting``.
-
-The results are output to Excel files that are preserved and made available as 'build artifacts' via the TeamCity web interface.
-
 API reference
 =============
 
@@ -170,6 +167,40 @@ Utilities
       collapse_gwp_info
       copy_ts
 
+.. _report-legacy:
+.. currentmodule:: message_ix_models.report.compat
+
+Compatibility with :mod:`.message_data`
+---------------------------------------
+
+.. automodule:: message_ix_models.report.compat
+   :members:
+
+   :mod:`.message_data` contains :doc:`m-data:reference/tools/post_processing`.
+   This code predates :mod:`genno` and the stack of tools built on it (:ref:`described above <report-intro>`); these were designed to avoid issues with performance and extensibility in the older code. [1]_
+   :mod:`.report.compat` prepares a Reporter to perform the same calculations as :mod:`message_data.tools.post_processing`, except using :mod:`genno`.
+
+   .. warning:: This code is **under development** and **incomplete**.
+      It is not yet a full or exact replacement for the legacy reporting code.
+      Use with caution.
+
+   Main API:
+
+   .. autosummary::
+      TECH_FILTERS
+      callback
+      prepare_techs
+      get_techs
+
+   Utility functions:
+
+   .. autosummary::
+      inp
+      eff
+      emi
+      out
+
+.. _report-cli:
 
 Command-line interface
 ======================
@@ -210,7 +241,20 @@ Command-line interface
 Testing
 =======
 
-
 .. currentmodule:: message_ix_models.report.sim
 .. automodule:: message_ix_models.report.sim
    :members:
+
+Continuous reporting
+--------------------
+
+As part of the :ref:`test-suite`, reporting is run on the same events (pushes and daily schedule) on publicly-available :doc:`model snapshots </api/model-snapshot>`.
+One goal of these tests *inter alia* is to ensure that adjustments and improvements to the reporting code do not disturb manually-verified model outputs.
+
+As part of the (private) :mod:`message_data` test suite, multiple workflows run on regular schedules; some of these include a combination of :mod:`message_ix_models`-based and :ref:`‘legacy’ reporting <report-legacy>`.
+These workflows:
+
+- Operate on specific scenarios within IIASA databases.
+- Create files in CSV, Excel, and/or PDF formats that are that are preserved and made available as 'build artifacts' via the GitHub Actions web interface and API.
+
+.. [1] See a (non-public) `“Reporting” project board <https://github.com/orgs/iiasa/projects/3>`_ on GitHub for details of the initial implementation of these features.

doc/api/testing.rst

@@ -1,8 +1,8 @@
+.. currentmodule:: message_ix_models.testing
+
 Test utilities and fixtures (:mod:`.testing`)
 *********************************************
 
-.. currentmodule:: message_ix_models.testing
-
 :doc:`Fixtures <pytest:explanation/fixtures>`:
 
 .. autosummary::

doc/repro.rst

@@ -20,6 +20,8 @@ This is a Scenario that has the same *structure* (ixmp 'sets') as actual instanc
 Code that operates on the global model can be tested on the bare RES; if it works on that scenario, this is one indication (necessary, but not always sufficient) that it should work on fully-populated scenarios.
 Such tests are faster and lighter than testing on fully-populated scenarios, and make it easier to isolate errors in the code that is being tested.
 
+.. _test-suite:
+
 Test suite (:mod:`message_ix_models.tests`)
 ===========================================
 
@@ -58,13 +60,16 @@ In either case:
 - Running the test suite with ``--local-cache`` causes the local cache to be populated, and this will affect subsequent runs.
 - The continuous integration (below) services don't preserve caches, so code always runs.
 
+.. _ci:
+
 Continuous testing
 ==================
 
-The test suite (:mod:`message_ix_models.tests`) is run using GitHub Actions for new commits on the ``main`` branch, or on any branch associated with a pull request.
-
-Because it is closed-source and requires access to internal IIASA resources, including databases, continuous integration for :mod:`message_data` is handled by an internal server running `TeamCity <https://www.jetbrains.com/teamcity/>`_: https://ene-builds.iiasa.ac.at/project/message (requires authorization)
+The test suite (:mod:`message_ix_models.tests`) is run using GitHub Actions for new commits on the ``main`` branch; new commits on any branch associated with a pull request; and on a daily schedule.
+These ensure that the code is functional and produces expected outputs, even as upstream dependencies evolve.
+Workflow runs and their outputs can be viewed `here <https://github.com/iiasa/message-ix-models/actions/workflows/pytest.yaml>`__.
 
+Because it is closed-source and requires access to internal IIASA resources, including databases, continuous integration for :mod:`.message_data` is handled by GitHub Actions `self-hosted runners <https://docs.github.com/en/actions/hosting-your-own-runners>`__ running on IIASA systems.
 
 .. _export-test-data:
 

doc/whatsnew.rst

@@ -1,8 +1,10 @@
 What's new
 **********
 
-.. Next release
-.. ============
+Next release
+============
+
+- Add :mod:`message_ix_models.report.compat` :ref:`for emulating legacy reporting <report-legacy>` (:pull:`134`).
 
 v2023.10.16
 ===========

message_ix_models/data/technology.yaml

@@ -1224,21 +1224,21 @@ gas_rc:
   name: gas_rc
   description: Gas heating in residential/commercial sector
   type: final
-  sector: gas
+  sector: residential/commercial
   input: [gas, secondary]
   output: [gas, final]
 
-gas_t/d:
-  name: gas_t/d
+gas_t_d:
+  name: gas_t_d
   description: Transmission/Distribution of gas
   type: final
   vintaged: TRUE
   sector: gas
   input: [gas, secondary]
   output: [gas, final]
 
-gas_t/d_ch4:
-  name: gas_t/d_ch4
+gas_t_d_ch4:
+  name: gas_t_d_ch4
   description: Transmission/Distribution of gas with CH4 mitigation
   type: final
   vintaged: TRUE

message_ix_models/model/config.py

@@ -1,4 +1,4 @@
-from dataclasses import dataclass, fields
+from dataclasses import dataclass, field, fields
 
 from message_ix_models.model.structure import codelists
 from message_ix_models.util.context import _ALIAS
@@ -39,6 +39,11 @@ class Config:
     #: :func:`.bare.get_spec`.
     res_with_dummies: bool = False
 
+    #: Default or preferred units for model quantities and reporting.
+    units: dict = field(
+        default_factory=lambda: {"energy": "GWa", "CO2 emissions": "Mt / a"}
+    )
+
     def check(self):
         """Check the validity of :attr:`regions`, :attr:`relations`, :attr:`years`."""
         for attr, name in [

message_ix_models/report/__init__.py

@@ -146,20 +146,24 @@ def cb(rep: Reporter, ctx: Context):
     """
     if isinstance(name_or_callback, str):
         # Resolve a string
-        for name in [
+        candidates = [
             # As a fully-resolved package/module name
             name_or_callback,
             # As a submodule of message_ix_models
             f"message_ix_models.{name_or_callback}.report",
             # As a submodule of message_data
             f"message_data.{name_or_callback}.report",
-        ]:
+        ]
+        mod = None
+        for name in candidates:
             try:
                 mod = import_module(name)
             except ModuleNotFoundError:
                 continue
             else:
                 break
+        if mod is None:
+            raise ModuleNotFoundError(" or ".join(candidates))
         callback = mod.callback
     else:
         callback = name_or_callback
@@ -338,11 +342,13 @@ def prepare_reporter(
         rep = Reporter.from_scenario(scenario)
         has_solution = scenario.has_solution()
 
-    # Append the message_data computations
+    # Append the message_data operators
     rep.require_compat("message_ix_models.report.computations")
     try:
+        # TODO Replace usage of operators from this module in favour of .exo_data; then
+        #      remove this line.
         rep.require_compat("message_data.tools.gdp_pop")
-    except ModuleNotFoundError:
+    except ModuleNotFoundError:  # pragma: no cover
         pass  # Currently in message_data
 
     # Force re-installation of the function iamc() in this file as the handler for
@@ -364,6 +370,7 @@ def prepare_reporter(
         **deepcopy(context.report.genno_config),
         fail="raise" if has_solution else logging.NOTSET,
     )
+    rep.configure(model=deepcopy(context.model))
 
     # Apply callbacks for other modules which define additional reporting computations
     for callback in CALLBACKS: