Manual adjustments for migrate-report

.github/workflows/lint.yaml

@@ -23,6 +23,7 @@ jobs:
       type-hint-packages: >-
         genno
         iam-units
+        "mypy < 1.5"
         "pint < 0.21"
         pytest
         sdmx1

doc/api/report/default-config.rst

@@ -1,5 +1,5 @@
 Default reporting configuration
 *******************************
 
-.. literalinclude:: ../../../data/report/global.yaml
+.. literalinclude:: ../../../message_ix_models/data/report/global.yaml
    :language: yaml

doc/api/report/index.rst

@@ -1,32 +1,35 @@
-Reporting
-*********
+Reporting (:mod:`~.message_ix_models.report`)
+*********************************************
 
 .. contents::
    :local:
 
 See also:
 
-- ``global.yaml``, the :doc:`reporting/default-config`.
+- ``global.yaml``, the :doc:`default-config`.
 - Documentation for :mod:`genno` (:doc:`genno:index`), :mod:`ixmp.reporting`, and :mod:`message_ix.reporting`.
-- :doc:`/reference/tools/post_processing`, still in use.
-- Documentation for reporting specific to certain model variants:
-
-  - :doc:`/reference/model/transport/report`
-- `“Reporting” project board <https://github.com/orgs/iiasa/projects/3>`_ on GitHub for the initial development of these features.
 
 .. toctree::
    :hidden:
 
-   reporting/default-config
+   default-config
+
+Not public:
+
+- `“Reporting” project board <https://github.com/orgs/iiasa/projects/3>`_ on GitHub for the initial implementation of these features.
+- :doc:`m-data:/reference/tools/post_processing`, still in use.
+- Documentation for reporting specific to certain model variants:
+
+  - :doc:`m-data:/reference/model/transport/report`
 
 Introduction
 ============
 
 See :doc:`the discussion in the MESSAGEix docs <message_ix:reporting>` about the stack.
-In short, :mod:`message_ix` cannot contain reporting code that references ``coal_ppl``, because not every model built on the MESSAGE framework will have a technology with this name.
-Any reporting specific to ``coal_ppl`` must be in :mod:`message_data`, since all models in the MESSAGEix-GLOBIOM family will have this technology.
+In short, :mod:`message_ix` must not contain reporting code that references ``coal_ppl``, because not every model built on the MESSAGE framework will have a technology with this name.
+Any reporting specific to ``coal_ppl`` must be in :mod:`message_ix_models`, since all models in the MESSAGEix-GLOBIOM family will have this technology.
 
-The basic **design pattern** of :mod:`message_data.reporting` is:
+The basic **design pattern** of :mod:`message_ix_models.report` is:
 
 - A ``global.yaml`` file (i.e. in `YAML <https://en.wikipedia.org/wiki/YAML#Example>`_ format) that contains a *concise* yet *explicit* description of the reporting computations needed for a MESSAGE-GLOBIOM model.
 - :func:`~.reporting.prepare_reporter` reads the file and a Scenario object, and uses it to populate a new Reporter.
@@ -35,7 +38,7 @@ The basic **design pattern** of :mod:`message_data.reporting` is:
 Features
 ========
 
-By combining these genno, ixmp, message_ix, and message_data features, the following functionality is provided.
+By combining these genno, ixmp, message_ix, and message_ix_models features, the following functionality is provided.
 
 .. note:: If any of this does not appear to work as advertised, file a bug!
 
@@ -64,9 +67,10 @@ Units
        base: example_var:a-b-c
        units: kJ
 
+Continuous reporting
+====================
 
-Continous reporting
-===================
+.. note:: This section is no longer current.
 
 The IIASA TeamCity build server is configured to automatically run the full (:file:`global.yaml`) reporting on the following scenarios:
 
@@ -81,31 +85,40 @@ This takes place:
 
 The results are output to Excel files that are preserved and made available as 'build artifacts' via the TeamCity web interface.
 
-
 API reference
 =============
 
-.. currentmodule:: message_data.reporting
+.. currentmodule:: message_ix_models.report
 
-.. automodule:: message_data.reporting
+.. automodule:: message_ix_models.report
    :members:
 
+   .. autosummary::
+
+      prepare_reporter
+      register
+      report
 
-Computations
-------------
+Operators
+---------
 
-.. currentmodule:: message_data.reporting.computations
-.. automodule:: message_data.reporting.computations
+.. currentmodule:: message_ix_models.report.computations
+.. automodule:: message_ix_models.report.computations
    :members:
 
-   :mod:`message_data` provides the following:
+   :mod:`message_ix_models` provides the following:
 
    .. autosummary::
 
+      from_url
+      get_ts
       gwp_factors
+      make_output_path
+      model_periods
+      remove_ts
       share_curtailment
 
-   Other computations are provided by:
+   Other operators are provided by:
 
    - :mod:`message_ix.reporting.computations`
    - :mod:`ixmp.reporting.computations`
@@ -114,11 +127,51 @@ Computations
 Utilities
 ---------
 
-.. currentmodule:: message_data.reporting.util
-.. automodule:: message_data.reporting.util
+.. currentmodule:: message_ix_models.report.util
+.. automodule:: message_ix_models.report.util
    :members:
 
+   .. autosummary::
+
+      add_replacements
+      as_quantity
+      collapse
+      collapse_gwp_info
+      copy_ts
 
-.. currentmodule:: message_data.reporting.cli
-.. automodule:: message_data.reporting.cli
+
+Command-line interface
+----------------------
+
+.. currentmodule:: message_ix_models.report.cli
+.. automodule:: message_ix_models.report.cli
    :members:
+
+
+.. code-block::
+
+   $ mix-models report --help
+
+   Usage: mix-models report [OPTIONS] [KEY]
+
+     Postprocess results.
+
+     KEY defaults to the comprehensive report 'message::default', but may also be
+     the name of a specific model quantity, e.g. 'output'.
+
+     --config can give either the absolute path to a reporting configuration
+     file, or the stem (i.e. name without .yaml extension) of a file in
+     data/report.
+
+     With --from-file, read multiple Scenario identifiers from FILE, and report
+     each one. In this usage, --output-path may only be a directory.
+
+   Options:
+     --dry-run             Only show what would be done.
+     --config TEXT         Path or stem for reporting config file.  [default:
+                           global]
+     -L, --legacy          Invoke legacy reporting.
+     -m, --module MODULES  Add extra reporting for MODULES.
+     -o, --output PATH     Write output to file instead of console.
+     --from-file FILE      Report multiple Scenarios listed in FILE.
+     --help                Show this message and exit.

doc/index.rst

@@ -34,6 +34,7 @@ Among other tasks, the tools allow modelers to:
    api/model-emissions
    api/model-snapshot
    api/disutility
+   api/report/index
    api/tools
    api/util
    api/testing

doc/whatsnew.rst

@@ -4,6 +4,7 @@ What's new
 Next release
 ============
 
+- New module :mod:`message_ix_models.report` for reporting (:pull:`116`).
 - Add documentation on :ref:`migrate-filter-repo` using :program:`git filter-repo` and helper scripts (:pull:`89`).
 
 v2023.7.26

message_ix_models/cli.py

@@ -114,6 +114,7 @@ def debug(ctx):
     "message_ix_models.model.snapshot",
     "message_ix_models.model.structure",
     "message_ix_models.model.water.cli",
+    "message_ix_models.report.cli",
 ]
 
 try:

message_ix_models/report/__init__.py

@@ -1,7 +1,7 @@
 import logging
-import sys
 from copy import deepcopy
 from functools import partial
+from importlib import import_module
 from operator import itemgetter
 from pathlib import Path
 from typing import Callable, List, Optional, Tuple, Union
@@ -13,9 +13,10 @@
 from genno import Key
 from genno.compat.pyam import iamc as handle_iamc
 from message_ix import Reporter, Scenario
+
 from message_ix_models import Context
 from message_ix_models.model.structure import get_codes
-from message_ix_models.util import local_data_path, private_data_path
+from message_ix_models.util import local_data_path, package_data_path
 from message_ix_models.util._logging import mark_time
 
 from .util import add_replacements
@@ -66,12 +67,12 @@ def iamc(c: Reporter, info):
     """
     # FIXME the upstream key "variable" for the configuration is confusing; choose a
     #       better name
-    from message_data.reporting.util import collapse
+    from message_ix_models.report.util import collapse
 
     # Common
-    base_key = Key.from_str_or_key(info["base"])
+    base_key = Key(info["base"])
 
-    # Use message_data custom collapse() method
+    # Use message_ix_models custom collapse() method
     info.setdefault("collapse", {})
 
     # Add standard renames
@@ -128,7 +129,7 @@ def register(name_or_callback: Union[Callable, str]) -> Optional[str]:
 
         from message_ix.reporting import Reporter
         from message_ix_models import Context
-        from message_data.reporting import register
+        from message_ix_models.report import register
 
         def cb(rep: Reporter, ctx: Context):
             # Modify `rep` by calling its methods ...
@@ -139,22 +140,29 @@ def cb(rep: Reporter, ctx: Context):
     Parameters
     ----------
     name_or_callback
-        If a string, this may be a submodule of :mod:`.message_data`, in which case the
-        function :func:`message_data.{name}.report.callback` is used. Or, it may be a
-        fully-resolved package/module name, in which case :func:`{name}.callback` is
+        If a string, this may be a submodule of :mod:`.message_ix_models`, or
+        :mod:`message_data`, in which case the function
+        ``{message_data,message_ix_models}.{name}.report.callback`` is used. Or, it may
+        be a fully-resolved package/module name, in which case ``{name}.callback`` is
         used. If a callable (function), it is used directly.
     """
     if isinstance(name_or_callback, str):
         # Resolve a string
-        try:
-            # …as a submodule of message_data
-            name = f"message_data.{name_or_callback}.report"
-            __import__(name)
-        except ImportError:
-            # …as a fully-resolved package/module name
-            name = name_or_callback
-            __import__(name)
-        callback = sys.modules[name].callback
+        for name in [
+            # As a submodule of message_ix_models
+            f"message_ix_models.{name_or_callback}.report",
+            # As a submodule of message_data
+            f"message_data.{name_or_callback}.report",
+            # As a fully-resolved package/module name
+            name_or_callback,
+        ]:
+            try:
+                mod = import_module(name)
+            except ModuleNotFoundError:
+                continue
+            else:
+                break
+        callback = mod.callback
     else:
         callback = name_or_callback
         name = callback.__name__
@@ -243,7 +251,7 @@ def report(context: Context, *args, **kwargs):
 
     # Default arguments for genno-based reporting
     context.report.setdefault("key", "default")
-    context.report.setdefault("config", private_data_path("report", "global.yaml"))
+    context.report.setdefault("config", package_data_path("report", "global.yaml"))
 
     rep, key = prepare_reporter(context)
 
@@ -362,8 +370,18 @@ def prepare_reporter(
         has_solution = scenario.has_solution()
 
     # Append the message_data computations
-    rep.require_compat("message_data.reporting.computations")
-    rep.require_compat("message_data.tools.gdp_pop")
+    rep.require_compat("message_ix_models.report.computations")
+    try:
+        rep.require_compat("message_data.tools.gdp_pop")
+    except ModuleNotFoundError:
+        pass  # Currently in message_data
+
+    # Force re-installation of the function iamc() in this file as the handler for
+    # "iamc:" sections in global.yaml. Until message_data.reporting is removed, then
+    # importing it will cause the iamc() function in *that* file to override the one
+    # registered above.
+    # TODO Remove, once message_data.reporting is removed.
+    genno.config.handles("iamc")(iamc)
 
     # Handle `report/config` setting passed from calling code
     context.setdefault("report", dict())
@@ -380,7 +398,7 @@ def prepare_reporter(
     p = config.get("path")
     if p and not p.exists() and not p.is_absolute():
         # Try to resolve relative to the data/ directory
-        p = private_data_path("report", p)
+        p = package_data_path("report", p)
         assert p.exists(), p
         config.update(path=p)
 

message_ix_models/report/cli.py

@@ -4,12 +4,12 @@
 
 import click
 import yaml
+
+from message_ix_models.report import register, report
 from message_ix_models.util import local_data_path, private_data_path
 from message_ix_models.util._logging import mark_time
 from message_ix_models.util.click import common_params
 
-from message_data.reporting import register, report
-
 log = logging.getLogger(__name__)