diff --git a/doc/api.rst b/doc/api.rst index fabd7b6a..e2216653 100644 --- a/doc/api.rst +++ b/doc/api.rst @@ -287,6 +287,7 @@ Top-level classes and functions .. autoclass:: genno.Quantity :members: + :inherited-members: pipe, shape, size The :class:`.Quantity` constructor converts its arguments to an internal, :class:`xarray.DataArray`-like data format: @@ -424,3 +425,6 @@ Utilities for testing .. automodule:: genno.testing :members: :exclude-members: parametrize_quantity_class + +.. automodule:: genno.testing.jupyter + :members: diff --git a/doc/cache.rst b/doc/cache.rst index 8b8bbcdd..4d4d0de8 100644 --- a/doc/cache.rst +++ b/doc/cache.rst @@ -31,7 +31,7 @@ A cache key is computed from: 1. the name of `func`. 2. the arguments to `func`, and -3. the compiled bytecode of `func` (see :func:`hash_code`). +3. the compiled bytecode of `func` (see :func:`.hash_code`). If a file exists in ``cache_path`` with a matching key, it is loaded and returned instead of calling `func`. @@ -61,12 +61,12 @@ Consider a function that loads a very large file, or performs some slow processi # … further processing … return Quantity(result) -We want to cache the result of :func:`slow_data_load`, but have the cache refreshed when the file contents change. +We want to cache the result of :py:`slow_data_load`, but have the cache refreshed when the file contents change. We do this using the `_extra_cache_key` argument to the function. This argument is not used in the function, but *does* affect the value of the cache key. When calling the function, pass some value that indicates whether the contents of `path` have changed. -One possibility is the modification time, via :meth:`.Path.stat`: +One possibility is the modification time, via :meth:`pathlib.Path.stat`: .. code-block:: python @@ -74,7 +74,7 @@ One possibility is the modification time, via :meth:`.Path.stat`: return slow_data_load(path, path.stat().st_mtime) Another possibility is to hash the entire file. -:func:`hash_contents` is provided for this purpose: +:func:`.hash_contents` is provided for this purpose: .. code-block:: python diff --git a/doc/compat-pyam.rst b/doc/compat-pyam.rst index 85805567..7de5b1a5 100644 --- a/doc/compat-pyam.rst +++ b/doc/compat-pyam.rst @@ -45,10 +45,8 @@ Pyam (:mod:`.compat.pyam`) as_pyam add_as_pyam - concat - write_report - .. .. autofunction:: as_pyam + This module also registers implementations of :func:`.concat` and :func:`.write_report` that handle :class:`pyam.IamDataFrame` objects. .. autofunction:: add_as_pyam diff --git a/doc/index.rst b/doc/index.rst index 05f7f6fa..4ae40554 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -4,7 +4,7 @@ **genno** is a Python package for describing and executing complex calculations on labelled, multi-dimensional data. It aims to make these calculations efficient, transparent, and easily validated as part of scientific research. -genno is built on high-quality Python data packages including ``dask``, ``xarray``, and ``pint``; and provides (current or planned) compatibility with packages including ``pandas``, ``matplotlib``, ``plotnine``, ``ixmp``, ``sdmx1``, and ``pyam``. +genno is built on high-quality Python data packages including :py:`dask`, :mod:`xarray`, and ```pint``; and provides (current or planned) compatibility with packages including :mod:`pandas`, :mod:`matplotlib`, :mod:`plotnine <.compat.plotnine>`, :mod:`ixmp`, :mod:`sdmx1 <.compat.sdmx>`, and :mod:`pyam <.compat.pyam>`. .. toctree:: :maxdepth: 2 @@ -22,9 +22,9 @@ Compatibility :mod:`.genno` provides built-in support for interaction with: -- :doc:`Plotnine ` (:mod:`.plotnine`), via :mod:`.compat.plotnine`. -- :doc:`Pyam ` (:mod:`.pyam`), via :mod:`.compat.pyam`. -- :doc:`SDMX ` (:mod:`.sdmx`), via :mod:`.compat.sdmx`. +- :doc:`Plotnine ` (:mod:`plotnine`), via :mod:`.compat.plotnine`. +- :doc:`Pyam ` (:mod:`pyam`), via :mod:`.compat.pyam`. +- :doc:`SDMX ` (:mod:`sdmx`), via :mod:`.compat.sdmx`. .. toctree:: :maxdepth: 1 @@ -37,8 +37,8 @@ Compatibility Packages that extend :mod:`genno` include: -- :mod:`ixmp.reporting` -- :mod:`message_ix.reporting` +- :mod:`ixmp.report` +- :mod:`message_ix.report` .. toctree:: :maxdepth: 2 diff --git a/doc/whatsnew.rst b/doc/whatsnew.rst index 0f109f76..3ebf0048 100644 --- a/doc/whatsnew.rst +++ b/doc/whatsnew.rst @@ -123,7 +123,7 @@ v1.17.0 (2023-05-15) ==================== - Bug fix: genno v1.16.1 (:pull:`85`) introduced :class:`ValueError` for some usages of :func:`.computations.sum <.operator.sum>` (:pull:`88`). -- Provide typed signatures for :meth:`Quantity.bfill`, :meth:`~Quantity.cumprod`, :meth:`~Quantity.ffill` for the benefit of downstream applications (:pull:`88`). +- Provide typed signatures for :meth:`.Quantity.bfill`, :meth:`~.Quantity.cumprod`, :meth:`~.Quantity.ffill` for the benefit of downstream applications (:pull:`88`). - Ensure and test that :attr:`.Quantity.name` and :attr:`~.Quantity.units` pass through all :mod:`~genno.computations `, in particular :func:`~.operator.aggregate`, :func:`~.operator.convert_units`, and :func:`~.operator.sum` (:pull:`88`). - Simplify arithmetic operations (:func:`~.operator.div`, :func:`~.operator.mul`, :func:`~.operator.pow`) so they are agnostic as to the :class:`.Quantity` class in use (:pull:`88`). - Ensure :attr:`.AttrSeries.index` is always :class:`pandas.MultiIndex` (:pull:`88`). @@ -144,13 +144,13 @@ v1.16.0 (2023-04-29) - genno supports and is tested on Python 3.11 (:pull:`83`). - Update dependencies (:pull:`83`): - - General: :mod:`importlib_resources` (the independent backport of :mod:`importlib.resources`) is added for Python 3.9 and earlier. + - General: :py:`importlib_resources` (the independent backport of :mod:`importlib.resources`) is added for Python 3.9 and earlier. - ``genno[sparse]``: new set of optional dependencies, including :mod:`sparse`. Install this set in order to use :class:`.SparseDataArray` for :class:`.Quantity`. - Note that sparse depends on :mod:`numba`, and thus :mod:`llvmlite`, and both of these package can lag new Python versions by several months. + Note that sparse depends on :py:`numba`, and thus :py:`llvmlite`, and both of these package can lag new Python versions by several months. For example, as of this release, they do not yet support Python 3.11, and thus :mod:`sparse` and :class:`.SparseDataArray` can only be used with Python 3.10 and earlier. - - ``genno[tests]``: :mod:`ixmp` is removed; :mod:`jupyter` and :mod:`nbclient` are added. + - ``genno[tests]``: :mod:`ixmp` is removed; :py:`jupyter` and :py:`nbclient` are added. Testing utilities in :mod:`genno.testing.jupyter` are duplicated from :mod:`ixmp.testing.jupyter`. - Adjust :meth:`.AttrSeries.interp` for compatibility with pandas 2.0.0 (released 2023-04-03) (:pull:`81`). @@ -268,7 +268,7 @@ v1.9.1 (2022-01-27) Note that installing ``genno[pyam]`` (including via ``genno[compat]``) currently forces the installation of an old version of :mod:`pint`; version 0.17 or earlier. Users wishing to use :mod:`genno.compat.pyam` should first install ``genno[pyam]``, then ``pip install --upgrade pint`` to restore a recent version of pint (0.18 or newer) that is usable with genno. -- :func:`computations.concat` works with :class:`.AttrSeries` with misaligned dimensions (:pull:`53`). +- :func:`.computations.concat <.operator.concat>` works with :class:`.AttrSeries` with misaligned dimensions (:pull:`53`). - Improve typing of :class:`.Quantity` and :class:`.Computer` to help with using `mypy `_ on code that uses :mod:`genno` (:pull:`53`). v1.9.0 (2021-11-23) @@ -307,7 +307,7 @@ v1.7.0 (2021-07-22) v1.6.0 (2021-07-07) =================== -- Add :meth:`Key.permute_dims` (:pull:`47`). +- Add :py:`Key.permute_dims()` (:pull:`47`). - Improve performance of :meth:`.Computer.check_keys` (:pull:`47`). v1.5.2 (2021-07-06) @@ -323,9 +323,9 @@ v1.5.1 (2021-07-01) v1.5.0 (2021-06-27) =================== -- Adjust :meth:`.test_assign_coords` for xarray 0.18.2 (:pull:`43`). +- Adjust :func:`.test_assign_coords` for xarray 0.18.2 (:pull:`43`). - Make :attr:`.Key.dims` order-insensitive so that ``Key("foo", "ab") == Key("foo", "ba")`` (:pull:`42`); make corresponding changes to :class:`.Computer` (:pull:`44`). -- Fix “:class:`AttributeError`: 'COO' object has no attribute 'item'” on :meth:`SparseDataArray.item` (:pull:`41`). +- Fix “:class:`AttributeError`: 'COO' object has no attribute 'item'” on :meth:`.SparseDataArray.item` (:pull:`41`). v1.4.0 (2021-04-26) =================== @@ -339,18 +339,18 @@ v1.3.0 (2021-03-22) - Bump minimum version of :mod:`sparse` from 0.10 to 0.12 and adjust to changes in this version (:pull:`39`) - - Remove :meth:`.SparseDataArray.equals`, obviated by improvements in :mod:`sparse`. + - Remove :py:`SparseDataArray.equals()`, obviated by improvements in :mod:`sparse`. - Improve :class:`.AttrSeries` (:pull:`39`) - Implement :meth:`~.AttrSeries.drop_vars` and :meth:`~.AttrSeries.expand_dims`. - :meth:`~.AttrSeries.assign_coords` can relabel an entire dimension. - - :meth:`~.AttrSeries.sel` can accept :class:`.DataArray` indexers and rename/combine dimensions. + - :meth:`~.AttrSeries.sel` can accept :class:`~xarray.DataArray` indexers and rename/combine dimensions. v1.2.1 (2021-03-08) =================== -- Bug fix: Provide abstract :class:`.Quantity.to_series` method for type checking in packages that depend on :mod:`genno`. +- Bug fix: Provide abstract :meth:`.Quantity.to_series` method for type checking in packages that depend on :mod:`genno`. v1.2.0 (2021-03-08) =================== @@ -377,15 +377,15 @@ v1.1.0 (2021-02-16) v1.0.0 (2021-02-13) =================== -- Adjust for usage by :mod:`ixmp.reporting` and :mod:`message_ix.reporting` (:pull:`28`): +- Adjust for usage by :mod:`ixmp.reporting ` and :mod:`message_ix.reporting ` (:pull:`28`): - Reduce minimum Python version to 3.6. This is lower than the minimum version for xarray (3.7), but matches ixmp, etc. - - Remove :mod:`compat.ixmp`; this code has been moved to :mod:`ixmp.reporting`, replacing what was there. - Likewise, remove :mod:`compat.message_ix`. + - Remove submodule :py:`compat.ixmp`; this code has been moved to :mod:`ixmp.reporting `, replacing what was there. + Likewise, remove submodule :py:`compat.message_ix`. - Simplify the form & parsing of ``iamc:`` section entries in configuration files: - - Remove unused feature to add :func:`group_sum` to the chain of tasks. + - Remove unused feature to add :py:`group_sum()` to the chain of tasks. - Keys now conform more closely to the arguments of :meth:`.Computer.convert_pyam`. - Move argument-checking from :func:`.as_pyam` to :meth:`.convert_pyam()`. @@ -416,11 +416,11 @@ v0.2.0 (2021-01-18) ------------------- - Increase test coverage to 100% (:pull:`12`). -- Port code from :mod:`message_ix.reporting` (:pull:`11`). +- Port code from :mod:`message_ix.reporting ` (:pull:`11`). - Add :mod:`.compat.pyam`. - Add a `name` parameter to :func:`.load_file`. v0.1.0 (2021-01-10) ------------------- -- Initial code port from :mod:`ixmp.reporting`. +- Initial code port from :mod:`ixmp.reporting `. diff --git a/genno/caching.py b/genno/caching.py index 1fa0a2e4..0481f1f0 100644 --- a/genno/caching.py +++ b/genno/caching.py @@ -53,7 +53,7 @@ def ignore(cls, *types): @classmethod def register(cls, func): - """Register a `func` to serialize a type not handled by :class:`.JSONEncoder`. + """Register `func` to serialize a type not handled by :class:`json.JSONEncoder`. `func` should return a type that *is* handled by JSONEncoder; see the docs. @@ -69,12 +69,12 @@ def register(cls, func): return _encode.register(func) def default(self, o): - """For `o`, return an object serializable by the base :class:`.JSONEncoder`. + """For `o`, return an object serializable by the base :class:`json.JSONEncoder`. - :class:`pathlib.Path`: the string representation of `o`. - - :class:`code` objects (from Python's built-in :mod:`inspect` module), e.g. - a function or lambda: :class:`blake2b` hash of the object's bytecode and its - serialized constants. + - :ref:`python:code-objects` (from Python's built-in :mod:`inspect` module), for + instance a function or lambda: :func:`~hashlib.blake2b` hash of the object's + bytecode and its serialized constants. .. warning:: This is not 100% guaranteed to change if the operation of `o` (or other code called in turn by `o`) changes. If relying on this behaviour, @@ -102,7 +102,7 @@ def default(self, o): def hash_args(*args, **kwargs): - """Return a 20-character :class:`blake2b` hex digest of `args` and `kwargs`. + """Return a 20-character :func:`hashlib.blake2b` hex digest of `args` and `kwargs`. Used by :func:`.decorate`. @@ -121,7 +121,7 @@ def hash_args(*args, **kwargs): def hash_code(func: Callable) -> str: - """Return the :class:`.blake2b` hex digest of the compiled bytecode of `func`. + """Return the :func:`hashlib.blake2b` hex digest of the compiled bytecode of `func`. See also -------- @@ -133,11 +133,11 @@ def hash_code(func: Callable) -> str: def hash_contents(path: Union[Path, str], chunk_size=65536) -> str: - """Return the :class:`.blake2b` hex digest of the contents of the file at `path`. + """Return the :func:`hashlib.blake2b` hex digest the file contents of `path`. Parameters ---------- - chunk_size : int, *optional* + chunk_size : int, optional Read the file in chunks of this size; default 64 kB. """ with Path(path).open("rb") as f: @@ -154,13 +154,13 @@ def decorate( Parameters ---------- - computer : Computer, *optional* + computer : .Computer, optional If supplied, the ``config`` dictionary stored in the Computer is used to look up values for `cache_path` and `cache_skip`, at the moment when `func` is called. - cache_path : os.Pathlike, *optional* + cache_path : os.PathLike, optional Directory in which to store cache files. - cache_skip : bool, *optional* + cache_skip : bool, optional If :obj:`True`, ignore existing cache entries and overwrite them with new values from `func`. diff --git a/genno/compat/graphviz.py b/genno/compat/graphviz.py index 4e161eee..6906eaf0 100644 --- a/genno/compat/graphviz.py +++ b/genno/compat/graphviz.py @@ -51,11 +51,11 @@ def visualize( ---------- dsk : The graph to display. - filename : Path or str, *optional* + filename : Path or str, optional The name of the file to write to disk. If the file name does not have a suffix, ".png" is used by default. If `filename` is :data:`None`, no file is written, and dask communicates with :program:`dot` using only pipes. - format : {'png', 'pdf', 'dot', 'svg', 'jpeg', 'jpg'}, *optional* + format : {'png', 'pdf', 'dot', 'svg', 'jpeg', 'jpg'}, optional Format in which to write output file, if not given by the suffix of `filename`. Default "png". data_attributes : @@ -73,7 +73,7 @@ def visualize( edge_attr : Mapping of (attribute, value) pairs set for all edges. Passed directly to :class:`.graphviz.Digraph`. - collapse_outputs : bool, *optional* + collapse_outputs : bool, optional Omit nodes for keys that are the output of intermediate calculations. kwargs : All other keyword arguments are added to `graph_attr`. diff --git a/genno/compat/plotnine/plot.py b/genno/compat/plotnine/plot.py index dba5792c..b7377197 100644 --- a/genno/compat/plotnine/plot.py +++ b/genno/compat/plotnine/plot.py @@ -1,21 +1,19 @@ import logging from abc import ABC, abstractmethod -from typing import TYPE_CHECKING, Hashable, Sequence +from typing import Hashable, Sequence from warnings import warn import plotnine as p9 +from genno.core.computer import Computer +from genno.core.key import KeyLike from genno.core.quantity import Quantity -if TYPE_CHECKING: - from genno.core.computer import Computer - from genno.core.key import KeyLike - log = logging.getLogger(__name__) class Plot(ABC): - """Class for plotting using :mod:`plotnine`.""" + """Class for plotting using :doc:`plotnine `.""" #: Filename base for saving the plot. basename = "" @@ -84,7 +82,7 @@ def make_task(cls, *inputs): Parameters ---------- - inputs : sequence of :class:`.Key`, :class:`str`, or other hashable, *optional* + *inputs : `.Key` or str or hashable, optional If provided, overrides the :attr:`inputs` property of the class. Returns @@ -106,8 +104,8 @@ def make_task(cls, *inputs): @classmethod def add_tasks( - cls, c: "Computer", key: "KeyLike", *inputs, strict: bool = False - ) -> "KeyLike": + cls, c: Computer, key: KeyLike, *inputs, strict: bool = False + ) -> KeyLike: """Add a task to `c` to generate and save the Plot. Analogous to :meth:`.Operator.add_tasks`. @@ -125,8 +123,8 @@ def generate(self, *args, **kwargs): Parameters ---------- - args : sequence of :class:`pandas.DataFrame` - Because :mod:`plotnine` operates on pandas data structures, :meth:`save` - automatically converts :obj:`Quantity` before being provided to - :meth:`generate`. + args : sequence of pandas.DataFrame + Because :doc:`plotnine ` operates on pandas data structures, + :meth:`save` automatically converts :obj:`.Quantity` before they are passed + to :meth:`generate`. """ diff --git a/genno/compat/pyam/operator.py b/genno/compat/pyam/operator.py index 597b94a1..04ecf93a 100644 --- a/genno/compat/pyam/operator.py +++ b/genno/compat/pyam/operator.py @@ -1,7 +1,15 @@ import logging from functools import partial from pathlib import Path -from typing import TYPE_CHECKING, Callable, Collection, Iterable, Optional, Union +from typing import ( + TYPE_CHECKING, + Callable, + Collection, + Iterable, + Mapping, + Optional, + Union, +) from warnings import warn import pyam @@ -27,7 +35,7 @@ def as_pyam( scenario, quantity: "Quantity", *, - rename=dict(), + rename=Mapping[str, str], collapse: Optional[Callable] = None, replace=dict(), drop: Union[Collection[str], str] = "auto", @@ -54,22 +62,24 @@ def as_pyam( Parameters ---------- scenario : - Any object with :attr:`model` and :attr:`scenario` attributes of type - :class:`str`, for instance an :class:`ixmp.Scenario`. - rename : dict (str -> str), *optional* - Mapping from dimension names in `quantity` to column names; either IAMC - dimension names, or others that are consumed by `collapse`. - collapse : callable, *optional* + Any object with :py:`model` and :py:`scenario` attributes of type :class:`str`, + for instance an :class:`ixmp.Scenario` or + :class:`~message_ix_models.util.scenarioinfo.ScenarioInfo`. + rename : dict, optional + Mapping from dimension names in `quantity` (:class:`str`) to column names + (:class:`str`); either IAMC dimension names, or others that are consumed by + `collapse`. + collapse : callable, optional Function that takes a :class:`pandas.DataFrame` and returns the same type. This function **may** collapse 2 or more dimensions, for example to construct labels for the IAMC ``variable`` dimension, or any other. - replace : *optional* + replace : optional Values to be replaced and their replaced. Passed directly to :meth:`pandas.DataFrame.replace`. - drop : str or collection of str, *optional* + drop : str or collection of str, optional Columns to drop. Passed to :func:`.util.drop`, so if not given, all non-IAMC columns are dropped. - unit : str, *optional* + unit : str, optional Label for the IAMC ``unit`` dimension. Passed to :func:`~.pyam.util.clean_units`. @@ -130,9 +140,9 @@ def add_as_pyam( Parameters ---------- - quantities : str or Key or list of (str, Key) + quantities : str or .Key or list of str or .Key Keys for quantities to transform. - tag : str, *optional* + tag : str, optional Tag to append to new Keys. Other parameters @@ -142,7 +152,7 @@ def add_as_pyam( Returns ------- - list of Key + list of .Key Each task converts a :class:`.Quantity` into a :class:`pyam.IamDataFrame`. """ # Handle single vs. iterable of inputs diff --git a/genno/compat/sdmx.py b/genno/compat/sdmx.py index d904cd59..21f6696d 100644 --- a/genno/compat/sdmx.py +++ b/genno/compat/sdmx.py @@ -1,11 +1,12 @@ from typing import TYPE_CHECKING, Iterable, List, Mapping, Optional, Union if TYPE_CHECKING: - from sdmx.model.common import Code, Codelist + import sdmx.model.common def codelist_to_groups( - codes: Union["Codelist", Iterable["Code"]], dim: Optional[str] = None + codes: Union["sdmx.model.common.Codelist", Iterable["sdmx.model.common.Code"]], + dim: Optional[str] = None, ) -> Mapping[str, Mapping[str, List[str]]]: """Convert `codes` into a mapping from parent items to their children. @@ -16,14 +17,14 @@ def codelist_to_groups( codes Either a :class:`sdmx.Codelist ` object or any iterable of :class:`sdmx.Code `. - dim : str, *optional* + dim : str, optional Dimension to aggregate. If `codes` is a code list and `dim` is not given, the ID of the code list is used; otherwise `dim` must be supplied. """ from sdmx.model.common import Codelist if isinstance(codes, Codelist): - items: Iterable["Code"] = codes.items.values() + items: Iterable["sdmx.model.common.Code"] = codes.items.values() dim = dim or codes.id else: items = codes diff --git a/genno/compat/xarray.py b/genno/compat/xarray.py index 7cf8331e..564c1a52 100644 --- a/genno/compat/xarray.py +++ b/genno/compat/xarray.py @@ -85,6 +85,7 @@ def attrs(self) -> Dict[Any, Any]: @property def data(self) -> Any: + """Like :attr:`xarray.DataArray.data`.""" return NotImplemented @property @@ -99,10 +100,12 @@ def dims(self) -> Tuple[Hashable, ...]: @property def shape(self) -> Tuple[int, ...]: + """Like :attr:`xarray.DataArray.shape`.""" return NotImplemented @property def size(self) -> int: + """Like :attr:`xarray.DataArray.size`.""" return NotImplemented def assign_coords( @@ -122,6 +125,7 @@ def astype( copy=None, keep_attrs=True, ): + """Like :meth:`xarray.DataArray.astype`.""" ... def bfill( @@ -129,6 +133,7 @@ def bfill( dim: Hashable, limit: Optional[int] = None, ): + """Like :meth:`xarray.DataArray.bfill`.""" ... def copy( @@ -146,6 +151,7 @@ def cumprod( keep_attrs: Optional[bool] = None, **kwargs: Any, ): + """Like :meth:`xarray.DataArray.cumprod`.""" ... def drop_vars( @@ -169,6 +175,7 @@ def ffill( dim: Hashable, limit: Optional[int] = None, ): + """Like :meth:`xarray.DataArray.ffill`.""" return NotImplemented def groupby( @@ -198,6 +205,7 @@ def pipe( *args: Any, **kwargs: Any, ) -> T: + """Like :meth:`xarray.DataArray.pipe`.""" return NotImplemented def rename( @@ -226,6 +234,7 @@ def shift( fill_value: Any = None, **shifts_kwargs: int, ): + """Like :attr:`xarray.DataArray.shift`.""" ... def sum( diff --git a/genno/config.py b/genno/config.py index 3e323163..7e158d9c 100644 --- a/genno/config.py +++ b/genno/config.py @@ -43,7 +43,7 @@ def configure(path: Optional[Union[Path, str]] = None, **config): Parameters ---------- - path : pathlib.Path, *optional* + path : pathlib.Path, optional Path to a configuration file in JSON or YAML format. **config : Configuration keys/sections and values. @@ -62,11 +62,11 @@ def handles(section_name: str, iterate: bool = True, discard: bool = True): section_name: str The name of the configuration section to handle. Using a name already present in :data:`HANDLERS` overrides that handler. - iterate : bool, *optional* + iterate : bool, optional If :obj:`True`, the handler is called once for each item (either list item, or (key, value) tuple) in the section. If :obj:`False`, the entire section contents, of whatever type, are passed to tha handler. - discard : bool, *optional* + discard : bool, optional If :obj:`True`, configuration section data is discarded after the handler is called. If :obj:`False`, the data is retained and stored on the Computer. """ @@ -264,6 +264,8 @@ def files(c: Computer, info): def general(c: Computer, info): """Handle one entry from the ``general:`` config section.""" # Inputs + # TODO allow to specify a more narrow key and *not* have infer_keys applied; perhaps + # using "*" inputs = c.infer_keys(info.get("inputs", [])) if info["comp"] in ("mul", "product"): diff --git a/genno/core/attrseries.py b/genno/core/attrseries.py index 29d13705..27c86c5b 100644 --- a/genno/core/attrseries.py +++ b/genno/core/attrseries.py @@ -99,10 +99,10 @@ class AttrSeries(pd.Series, Quantity): Parameters ---------- - units : str or pint.Unit, *optional* + units : str or pint.Unit, optional Set the units attribute. The value is converted to :class:`pint.Unit` and added to `attrs`. - attrs : :class:`~collections.abc.Mapping`, *optional* + attrs : :class:`~collections.abc.Mapping`, optional Set the :attr:`~pandas.Series.attrs` of the AttrSeries. This attribute was added in `pandas 1.0 `_, but is not currently supported by the Series constructor. diff --git a/genno/core/computer.py b/genno/core/computer.py index 608a6beb..dc3b3727 100644 --- a/genno/core/computer.py +++ b/genno/core/computer.py @@ -108,9 +108,9 @@ def configure( Parameters ---------- - path : .Path, *optional* + path : pathlib.Path, optional Path to a configuration file in JSON or YAML format. - fail : "raise" or str or :mod:`logging` level, *optional* + fail : "raise" or str or :mod:`logging` level, optional Passed to :meth:`.add_queue`. If not "raise", then log messages are generated for config handlers that fail. The Computer may be only partially configured. @@ -149,7 +149,7 @@ def get_operator(self, name) -> Optional[Callable]: Returns ------- - .callable + callable None If there is no callable with the given `name` in any of :attr:`modules`. """ @@ -251,7 +251,7 @@ def add(self, data, *args, **kwargs) -> Union[KeyLike, Tuple[KeyLike, ...]]: Returns ------- - |KeyLike| or tuple of |KeyLike| + KeyLike or tuple of KeyLike Some or all of the keys added to the Computer. See also @@ -342,12 +342,13 @@ def add_queue( Parameters ---------- - queue : iterable of 2- or N-:class:`tuple` - The members of each tuple are the arguments (:class:`tuple`) and, - optionally, keyword arguments (e.g :class:`dict`) to :meth:`add`. - max_tries : int, *optional* + queue : iterable of tuple + Each item is either a N-:class:`tuple` of positional arguments to + :meth:`add`, or a 2-:class:`tuple` of (:class:`.tuple` of positional + arguments, :class:`dict` of keyword arguments). + max_tries : int, optional Retry adding elements up to this many times. - fail : "raise" or str or :mod:`logging` level, *optional* + fail : "raise" or str or :mod:`logging` level, optional Action to take when a computation from `queue` cannot be added after `max_tries`: "raise" an exception, or log messages on the indicated level and continue. @@ -434,10 +435,10 @@ def add_single( A string, Key, or other value identifying the output of `computation`. computation : object Any computation. See :attr:`graph`. - strict : bool, *optional* + strict : bool, optional If True, `key` must not already exist in the Computer, and any keys referred to by `computation` must exist. - index : bool, *optional* + index : bool, optional If True, `key` is added to the index as a full-resolution key, so it can be later retrieved with :meth:`full_key`. @@ -503,7 +504,7 @@ def apply( Parameters ---------- - generator : .callable + generator : callable Function to apply to `keys`. This function **may** take a first positional argument annotated with :class:`.Computer` or a subtype; if so, then it is provided with a reference to `self`. @@ -608,7 +609,7 @@ def get(self, key=None): Parameters ---------- - key : str, *optional* + key : str, optional If not provided, :attr:`default_key` is used. Raises @@ -646,7 +647,7 @@ def get(self, key=None): # Convenience methods for the graph and its keys def keys(self): - """Return the keys of :attr:`graph`.""" + """Return the keys of :attr:`~genno.Computer.graph`.""" return self.graph.keys() def full_key(self, name_or_key: KeyLike) -> KeyLike: @@ -676,16 +677,16 @@ def check_keys( Parameters ---------- - keys : |KeyLike| + keys : KeyLike Some :class:`Keys ` or strings. - predicate : callable, *optional* + predicate : callable, optional Function to run on each of `keys`; see below. - action : "raise" or any other value + action : "raise" or str Action to take on missing `keys`. Returns ------- - list of |KeyLike| + list of KeyLike One item for each item ``k`` in `keys`: 1. ``k`` itself, unchanged, if `predicate` is given and ``predicate(k)`` @@ -748,16 +749,16 @@ def infer_keys( Parameters ---------- - key_or_keys : |KeyLike| or list of |KeyLike| - dims : list of str, *optional* + key_or_keys : KeyLike or list of KeyLike + dims : list of str, optional Drop all but these dimensions from the returned key(s). Returns ------- - |KeyLike| - If `key_or_keys` is a single |KeyLike|. - list of |KeyLike| - If `key_or_keys` is an iterable of |KeyLike|. + KeyLike + If `key_or_keys` is a single KeyLike. + list of KeyLike + If `key_or_keys` is an iterable of KeyLike. """ single = isinstance(key_or_keys, (Key, Hashable)) keys = [key_or_keys] if single else tuple(cast(Iterable, key_or_keys)) @@ -852,7 +853,7 @@ def add_product(self, *args, **kwargs): """Deprecated. .. deprecated:: 1.18.0 - Instead use :func:`.add_mul` via: + Instead use :func:`.add_binop` via: .. code-block:: python @@ -902,13 +903,13 @@ def aggregate( quantity. dims_or_groups: str or iterable of str or dict Name(s) of the dimension(s) to sum over, or nested dict. - weights : :class:`xarray.DataArray`, *optional* + weights : :class:`xarray.DataArray`, optional Weights for weighted aggregation. - keep : bool, *optional* + keep : bool, optional Passed to :meth:`operator.aggregate `. - sums : bool, *optional* + sums : bool, optional Passed to :meth:`add`. - fail : str or int, *optional* + fail : str or int, optional Passed to :meth:`add_queue` via :meth:`add`. Returns diff --git a/genno/core/describe.py b/genno/core/describe.py index d1a27182..8b7e234f 100644 --- a/genno/core/describe.py +++ b/genno/core/describe.py @@ -83,7 +83,8 @@ def label(arg, max_length=MAX_ITEM_LENGTH) -> str: The label depends on the type of `arg`: - :class:`.xarray.DataArray`: the first line of the string representation. - - :func:`.partial` object: a less-verbose version that omits None arguments. + - :func:`functools.partial` object: a less-verbose version that omits None + arguments. - Item protected with :func:`.dask.core.quote`: its literal value. - A callable, e.g. a function: its name. - Anything else: its :class:`str` representation. diff --git a/genno/core/graph.py b/genno/core/graph.py index f867b77d..d453e7e8 100644 --- a/genno/core/graph.py +++ b/genno/core/graph.py @@ -80,12 +80,14 @@ def __contains__(self, item) -> bool: return False def pop(self, *args): + """Overload :meth:`dict.pop` to also call :meth:`_deindex`.""" try: return super().pop(*args) finally: self._deindex(args[0]) def update(self, arg=None, **kwargs): + """Overload :meth:`dict.pop` to also call :meth:`_index`.""" if isinstance(arg, (Sequence, Generator)): arg0, arg1 = tee(arg) arg_keys = map(itemgetter(0), arg0) @@ -114,14 +116,14 @@ def infer( Parameters ---------- - dims : list of str, *optional* + dims : list of str, optional Drop all but these dimensions from the returned key(s). Returns ------- str If `key` is not found in the Graph. - Key + .Key `key` with either its full dimensions (cf. :meth:`full_key`) or, if `dims` are given, with only these dims. """ diff --git a/genno/core/key.py b/genno/core/key.py index ff8c1be5..e384cfbb 100644 --- a/genno/core/key.py +++ b/genno/core/key.py @@ -113,22 +113,6 @@ def from_str_or_key( ) -> "Key": """Return a new Key from *value*. - Parameters - ---------- - value : str or Key - Value to use to generate a new Key. - drop : list of str or :obj:`True`, *optional* - Existing dimensions of *value* to drop. See :meth:`drop`. - append : list of str, *optional*. - New dimensions to append to the returned Key. See :meth:`append`. - tag : str, *optional* - Tag for returned Key. If *value* has a tag, the two are joined - using a '+' character. See :meth:`add_tag`. - - Returns - ------- - :class:`Key` - .. versionchanged:: 1.18.0 Calling :meth:`from_str_or_key` with a single argument is no longer @@ -141,6 +125,22 @@ def from_str_or_key( k1 = Key("foo:a-b-c:t1", drop="b", append="d", tag="t2") k2 = Key("foo:a-b-c:t1").drop("b").append("d)" + + Parameters + ---------- + value : str or .Key + Value to use to generate a new Key. + drop : list of str or :obj:`True`, optional + Existing dimensions of *value* to drop. See :meth:`drop`. + append : list of str, optional + New dimensions to append to the returned Key. See :meth:`append`. + tag : str, optional + Tag for returned Key. If *value* has a tag, the two are joined + using a '+' character. See :meth:`add_tag`. + + Returns + ------- + :class:`Key` """ base = cls(value) diff --git a/genno/core/operator.py b/genno/core/operator.py index 63d9f656..f999adf0 100644 --- a/genno/core/operator.py +++ b/genno/core/operator.py @@ -1,11 +1,10 @@ from functools import update_wrapper from inspect import signature -from typing import TYPE_CHECKING, Any, Callable, ClassVar, Dict, Optional, Tuple, Union +from typing import Any, Callable, ClassVar, Dict, Optional, Tuple, Union from warnings import warn -if TYPE_CHECKING: - from .computer import Computer - from .key import KeyLike +from .computer import Computer +from .key import KeyLike class Operator: @@ -70,7 +69,7 @@ def define( Parameters ---------- - helper : Callable, *optional* + helper : Callable, optional Equivalent to calling :meth:`helper` on the Operator instance. """ @@ -111,13 +110,13 @@ def decorator(func: Callable) -> "Operator": return decorator def helper( - self, func: Callable[..., Union["KeyLike", Tuple["KeyLike", ...]]] + self, func: Callable[..., Union[KeyLike, Tuple[KeyLike, ...]]] ) -> Callable: """Register `func` as the convenience method for adding task(s).""" self.__class__._add_tasks = staticmethod(func) return func - def add_tasks(self, c: "Computer", *args, **kwargs) -> Tuple["KeyLike", ...]: + def add_tasks(self, c: "Computer", *args, **kwargs) -> Tuple[KeyLike, ...]: """Invoke :attr:`_add_task` to add tasks to `c`.""" if self._add_tasks is None: raise NotImplementedError diff --git a/genno/core/quantity.py b/genno/core/quantity.py index fba183c6..eab65788 100644 --- a/genno/core/quantity.py +++ b/genno/core/quantity.py @@ -16,7 +16,7 @@ class Quantity(DataArrayLike["Quantity"]): """A sparse data structure that behaves like :class:`xarray.DataArray`. - Depending on the value of :data:`CLASS`, Quantity is either :class:`.AttrSeries` or + Depending on the value of :data:`.CLASS`, Quantity is either :class:`.AttrSeries` or :class:`.SparseDataArray`. """ diff --git a/genno/core/sparsedataarray.py b/genno/core/sparsedataarray.py index 0050dd44..24658dab 100644 --- a/genno/core/sparsedataarray.py +++ b/genno/core/sparsedataarray.py @@ -82,7 +82,7 @@ def COO_data(self): @property def dense(self): - """Return a copy with dense (:class:`.ndarray`) data.""" + """Return a copy with dense (:class:`numpy.ndarray`) data.""" try: # Use existing method xr.Variable._to_dense() return self.da._replace(variable=self.da.variable._to_dense()) @@ -92,7 +92,7 @@ def dense(self): @property def dense_super(self): - """Return a proxy to a :class:`.ndarray`-backed :class:`.DataArray`.""" + """Return a proxy to a :class:`numpy.ndarray`-backed :class:`xarray.DataArray`.""" return super(SparseDataArray, self.dense) @@ -118,10 +118,11 @@ class SparseDataArray(OverrideItem, xr.DataArray, Quantity): """:class:`~xarray.DataArray` with sparse data. SparseDataArray uses :class:`sparse.COO` for storage with :data:`numpy.nan` - as its :attr:`sparse.COO.fill_value`. Some methods of :class:`~xarray.DataArray` - are overridden to ensure data is in sparse, or dense, format as necessary, to - provide expected functionality not currently supported by :mod:`sparse`, and to - avoid exhausting memory for some operations that require dense data. + as its :attr:`sparse.SparseArray.fill_value`. Some methods of + :class:`~xarray.DataArray` are overridden to ensure data is in sparse, or dense, + format as necessary, to provide expected functionality not currently supported by + :mod:`sparse`, and to avoid exhausting memory for some operations that require dense + data. """ __slots__: Tuple[str, ...] = tuple() @@ -245,7 +246,7 @@ def to_dataframe( name: Optional[Hashable] = None, dim_order: Optional[Sequence[Hashable]] = None, ) -> pd.DataFrame: - """Convert this array and its coords into a :class:`~xarray.DataFrame`. + """Convert this array and its coords into a :class:`pandas.DataFrame`. Overrides :meth:`~xarray.DataArray.to_dataframe`. """ diff --git a/genno/operator.py b/genno/operator.py index 2a9ae8f6..5cdb0974 100644 --- a/genno/operator.py +++ b/genno/operator.py @@ -95,15 +95,15 @@ def add_binop(func, c: "Computer", key, *quantities, **kwargs) -> Key: Parameters ---------- - key : str or Key + key : str or .Key Key or name of the new quantity. If a Key, any dimensions are ignored; the dimensions of the result are the union of the dimensions of `quantities`. - sums : bool, *optional* + sums : bool, optional If :obj:`True`, all partial sums of the new quantity are also added. Returns ------- - Key + .Key The full key of the new quantity. Example @@ -146,7 +146,7 @@ def add(*quantities: Quantity, fill_value: float = 0.0) -> Quantity: Returns ------- - Quantity + .Quantity Units are the same as the first of `quantities`. See also @@ -329,8 +329,8 @@ def broadcast_map( """Broadcast `quantity` using a `map`. The `map` must be a 2-dimensional Quantity with dimensions (``d1``, ``d2``), such as - returned by :func:`map_as_qty`. `quantity` must also have a dimension ``d1``. - Typically ``len(d2) > len(d1)``. + returned by :func:`ixmp.report.operator.map_as_qty`. `quantity` must also have a + dimension ``d1``. Typically ``len(d2) > len(d1)``. `quantity` is 'broadcast' by multiplying it with `map`, and then summing on the common dimension ``d1``. The result has the dimensions of `quantity`, but with @@ -338,9 +338,10 @@ def broadcast_map( Parameters ---------- - rename : dict (str -> str), *optional* - Dimensions to rename on the result. - strict : bool, *optional* + rename : dict, optional + Dimensions to rename on the result; mapping from original dimension + (:class:`str`) to target name (:class:`str`). + strict : bool, optional Require that each element of ``d2`` is mapped from exactly 1 element of ``d1``. """ if strict and int(map.sum().item()) != len(map.coords[map.dims[1]]): @@ -354,11 +355,11 @@ def combine( select: Optional[List[Mapping]] = None, weights: Optional[List[float]] = None, ) -> Quantity: # noqa: F811 - """Sum distinct *quantities* by *weights*. + """Sum distinct `quantities` by `weights`. Parameters ---------- - *quantities : Quantity + *quantities : .Quantity The quantities to be added. select : list of dict Elements to be selected from each quantity. Must have the same number of @@ -370,7 +371,7 @@ def combine( Raises ------ ValueError - If the *quantities* have mismatched units. + If the `quantities` have mismatched units. """ # Handle arguments if select is None: @@ -495,8 +496,8 @@ def div(numerator: Union[Quantity, float], denominator: Quantity) -> Quantity: Parameters ---------- - numerator : Quantity - denominator : Quantity + numerator : .Quantity + denominator : .Quantity See also -------- @@ -505,7 +506,7 @@ def div(numerator: Union[Quantity, float], denominator: Quantity) -> Quantity: return numerator / denominator -#: Alias of :func:`div`, for backwards compatibility. +#: Alias of :func:`~genno.operator.div`, for backwards compatibility. #: #: .. note:: This may be deprecated and possibly removed in a future version. ratio = div @@ -605,7 +606,7 @@ def load_file( units: Optional[UnitLike] = None, name: Optional[str] = None, ) -> Any: - """Read the file at `path` and return its contents as a :class:`.Quantity`. + """Read the file at `path` and return its contents as a :class:`~genno.Quantity`. Some file formats are automatically converted into objects for direct use in genno computations: @@ -624,7 +625,7 @@ def load_file( ---------- path : pathlib.Path Path to the file to read. - dims : collections.abc.Collection or collections.abc.Mapping, *optional* + dims : collections.abc.Collection or collections.abc.Mapping, optional If a collection of names, other columns besides these and 'value' are discarded. If a mapping, the keys are the column labels in `path`, and the values are the target dimension names. @@ -660,7 +661,7 @@ def add_load_file(func, c: "Computer", path, key=None, **kwargs): ---------- path : os.PathLike Path to the file, e.g. '/path/to/foo.ext'. - key : str or Key, *optional* + key : str or .Key, optional Key for the quantity read from the file. Other parameters @@ -767,7 +768,7 @@ def mul(*quantities: Quantity) -> Quantity: return reduce(operator.mul, quantities) -#: Alias of :func:`mul`, for backwards compatibility. +#: Alias of :func:`~genno.operator.mul`, for backwards compatibility. #: #: .. note:: This may be deprecated and possibly removed in a future version. product = mul @@ -778,7 +779,7 @@ def pow(a: Quantity, b: Union[Quantity, int]) -> Quantity: Returns ------- - Quantity + .Quantity If `b` is :class:`int` or a Quantity with all :class:`int` values that are equal to one another, then the quantity has the units of `a` raised to this power; for example, "kg²" → "kg⁴" if `b` is 2. In other cases, there are no meaningful @@ -894,13 +895,17 @@ def select( Parameters ---------- - indexers : dict (str -> xarray.DataArray or list of str) - Elements to be selected from `qty`. Mapping from dimension names to coords along - the respective dimension of `qty`, or to xarray-style indexers. Values not - appearing in the dimension coords are silently ignored. - inverse : bool, *optional* + indexers : dict + Elements to be selected from `qty`. Mapping from dimension names (:class:`str`) + to either: + + - :class:`list` of `str`: coords along the respective dimension of `qty`, or + - :class:`xarray.DataArray`: xarray-style indexers. + + Values not appearing in the dimension coords are silently ignored. + inverse : bool, optional If :obj:`True`, *remove* the items in indexers instead of keeping them. - drop : bool, *optional* + drop : bool, optional If :obj:`True`, drop dimensions that are indexed by a scalar value (for instance, :py:`"foo"` or :py:`999`) in `indexers`. Note that dimensions indexed by a length-1 list of labels (for instance :py:`["foo"]`) are not dropped; this @@ -958,10 +963,10 @@ def sum( Parameters ---------- - weights : .Quantity, *optional* + weights : .Quantity, optional If `dimensions` is given, `weights` must have at least these dimensions. Otherwise, any dimensions are valid. - dimensions : list of str, *optional* + dimensions : list of str, optional If not provided, sum over all dimensions. If provided, sum over these dimensions. """ diff --git a/genno/testing/__init__.py b/genno/testing/__init__.py index 553effe1..7e015c5c 100644 --- a/genno/testing/__init__.py +++ b/genno/testing/__init__.py @@ -60,7 +60,7 @@ def add_large_data(c: Computer, num_params, N_dims=6, N_data=0): The result is a matrix wherein the Cartesian product of all the keys is very large— about 2e17 elements for N_dim = 6—but the contents are very sparse. This can be handled by :class:`.SparseDataArray`, but not by :class:`xarray.DataArray` backed - by :class:`np.array`. + by :class:`numpy.ndarray`. """ def _fib(): @@ -234,7 +234,7 @@ def assert_logs(caplog, message_or_messages=None, at_level=None): The pytest caplog fixture. message_or_messages : str or list of str String(s) that must appear in log messages. - at_level : int, *optional* + at_level : int, optional Messages must appear on 'genno' or a sub-logger with at least this level. """ __tracebackhide__ = True @@ -289,12 +289,12 @@ def assert_qty_equal( Parameters ---------- - check_type : bool, *optional* + check_type : bool, optional Assert that `a` and `b` are both :class:`.Quantity` instances. If :obj:`False`, the arguments are converted to Quantity. - check_attrs : bool, *optional* + check_attrs : bool, optional Also assert that check that attributes are identical. - ignore_extra_coords : bool, *optional* + ignore_extra_coords : bool, optional Ignore extra coords that are not dimensions. Only meaningful when Quantity is :class:`.SparseDataArray`. """ @@ -345,12 +345,12 @@ def assert_qty_allclose( Parameters ---------- - check_type : bool, *optional* + check_type : bool, optional Assert that `a` and `b` are both :class:`.Quantity` instances. If :obj:`False`, the arguments are converted to Quantity. - check_attrs : bool, *optional* + check_attrs : bool, optional Also assert that check that attributes are identical. - ignore_extra_coords : bool, *optional* + ignore_extra_coords : bool, optional Ignore extra coords that are not dimensions. Only meaningful when Quantity is :class:`.SparseDataArray`. """ @@ -392,19 +392,20 @@ def assert_units(qty: Quantity, exp: str) -> None: ).dimensionless, f"Units '{qty.units:~}'; expected {repr(exp)}" -def random_qty(shape: Dict[str, int], **kwargs): +def random_qty(shape: Dict[str, int], **kwargs) -> Quantity: """Return a Quantity with `shape` and random contents. Parameters ---------- - shape : dict (str -> int) - Mapping from dimension names to lengths along each dimension. + shape : dict + Mapping from dimension names (:class:`str`) to lengths along each dimension + (:class:`int`). **kwargs - Other keyword arguments to :class:`Quantity`. + Other keyword arguments to :class:`.Quantity`. Returns ------- - Quantity + .Quantity Random data with one dimension for each key in `shape`, and coords along those dimensions like "foo1", "foo2", with total length matching the value from `shape`. If `shape` is empty, a scalar (0-dimensional) Quantity. diff --git a/genno/testing/jupyter.py b/genno/testing/jupyter.py index e2f8bd13..0819065c 100644 --- a/genno/testing/jupyter.py +++ b/genno/testing/jupyter.py @@ -16,11 +16,11 @@ def run_notebook(nb_path, tmp_path, env=None, **kwargs): Parameters ---------- - nb_path : path-like + nb_path : os.PathLike The notebook file to execute. - tmp_path : path-like + tmp_path : os.PathLike A directory in which to create temporary output. - env : dict-like, *optional* + env : mapping, optional Execution environment for :mod:`nbclient`. Default: :obj:`os.environ`. kwargs : Keyword arguments for :class:`nbclient.NotebookClient`. Defaults are set for: @@ -102,7 +102,7 @@ def get_cell_output(nb, name_or_index, kind="data"): Parameters ---------- - kind : str, *optional* + kind : str, optional Kind of cell output to retrieve. For 'data', the data in format 'text/plain' is run through :func:`eval`. To retrieve an exception message, use 'evalue'. """ diff --git a/genno/util.py b/genno/util.py index d918aabb..57e8879c 100644 --- a/genno/util.py +++ b/genno/util.py @@ -23,7 +23,7 @@ #: - The '%' symbol cannot be supported by pint, because it is a Python operator; it is #: replaced with “percent”. #: -#: Additional values can be added with :func:`configure`; see :ref:`config-units`. +#: Additional values can be added with :func:`.configure`; see :ref:`config-units`. REPLACE_UNITS = { "%": "percent", } @@ -176,7 +176,7 @@ def free_parameters(func: Callable) -> Mapping: """Retrieve information on the free parameters of `func`. Identical to :py:`inspect.signature(func).parameters`; that is, to - :attr:`inspect.Signature.parameters`. :func:`free_pars` also: + :attr:`inspect.Signature.parameters`. :py:`free_parameters` also: - Handles functions that have been :func:`functools.partial`'d, returning only the parameters that have *not* already been assigned a value by the