diff --git a/.gitignore b/.gitignore index bd1177490..3a48aad59 100644 --- a/.gitignore +++ b/.gitignore @@ -53,7 +53,7 @@ rixmp/source/.Rhistory # pytest and related *.pid -.coverage +.coverage* .pytest_cache/ coverage.xml htmlcov/ diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md index 785003744..7eaee7422 100644 --- a/RELEASE_NOTES.md +++ b/RELEASE_NOTES.md @@ -1,6 +1,7 @@ # Next Release +- [#182](https://github.com/iiasa/ixmp/pull/182): Add new Backend, Model APIs and JDBCBackend, GAMSModel classes. - [#188](https://github.com/iiasa/ixmp/pull/188): Enhance reporting. - [#177](https://github.com/iiasa/ixmp/pull/177): add ability to pass `gams_args` through `Scenario.solve()` - [#175](https://github.com/iiasa/ixmp/pull/175): Drop support for Python 2. diff --git a/doc/source/api-backend.rst b/doc/source/api-backend.rst new file mode 100644 index 000000000..43840882f --- /dev/null +++ b/doc/source/api-backend.rst @@ -0,0 +1,145 @@ +.. currentmodule:: ixmp.backend + +Storage back ends (:mod:`ixmp.backend`) +======================================= + +By default, the |ixmp| is installed with :class:`ixmp.backend.jdbc.JDBCBackend`, which can store data in many types of relational database management systems (RDBMS) that have Java DataBase Connector (JDBC) interfaces—hence its name. + +However, |ixmp| is extensible to support other methods of storing data: in non-JDBC RDBMS, non-relational databases, local files, memory, or other ways. +Developers wishing to add such capabilities may subclass :class:`ixmp.backend.base.Backend` and implement its methods. + + +Provided backends +----------------- + +.. automodule:: ixmp.backend + :members: BACKENDS + +.. currentmodule:: ixmp.backend.jdbc + +.. autoclass:: ixmp.backend.jdbc.JDBCBackend + :members: read_gdx, write_gdx + + JDBCBackend supports: + + - ``dbtype='HSQLDB'``: HyperSQL databases in local files. + - Remote databases. This is accomplished by creating a :class:`ixmp.Platform` with the ``dbprops`` argument pointing a file that specifies JDBC information. For instance:: + + jdbc.driver = oracle.jdbc.driver.OracleDriver + jdbc.url = jdbc:oracle:thin:@database-server.example.com:1234:SCHEMA + jdbc.user = USER + jdbc.pwd = PASSWORD + + It has the following methods that are not part of the overall :class:`Backend` API: + + .. autosummary:: + :nosignatures: + + read_gdx + write_gdx + +.. automethod:: ixmp.backend.jdbc.start_jvm + +Backend API +----------- + +- :class:`ixmp.Platform` implements a *user-friendly* API for scientific programming. + This means its methods can take many types of arguments, check, and transform them—in a way that provides modeler-users with easy, intuitive workflows. +- In contrast, :class:`Backend` has a *very simple* API that accepts arguments and returns values in basic Python data types and structures. +- As a result: + + - :class:`Platform ` code is not affected by where and how data is stored; it merely handles user arguments and then makes, usually, a single :class:`Backend` call. + - :class:`Backend` code does not need to perform argument checking; merely store and retrieve data reliably. + +.. autodata:: ixmp.backend.FIELDS + +.. currentmodule:: ixmp.backend.base + +.. autoclass:: ixmp.backend.base.Backend + :members: + + In the following, the bold-face words **required**, **optional**, etc. have specific meanings as described in `IETF RFC 2119 `_. + + Backend is an **abstract** class; this means it **must** be subclassed. + Most of its methods are decorated with :meth:`abc.abstractmethod`; this means they are **required** and **must** be overridden by subclasses. + + Others, marked below with "OPTIONAL:", are not so decorated. + For these methods, the behaviour in the base Backend—often, nothing—is an acceptable default behaviour. + Subclasses **may** extend or replace this behaviour as desired, so long as the methods still perform the actions described in the description. + + Backends: + + - **must** only raise standard Python exceptions. + + Methods related to :class:`ixmp.Platform`: + + .. autosummary:: + :nosignatures: + + close_db + get_auth + get_nodes + get_scenarios + get_units + open_db + set_log_level + set_node + set_unit + + Methods related to :class:`ixmp.TimeSeries`: + + - Each method has an argument `ts`, a reference to the TimeSeries object being manipulated. + - ‘Geodata’ is otherwise identical to regular timeseries data, except value are :class:`str` rather than :class:`float`. + + .. autosummary:: + :nosignatures: + + check_out + commit + delete + delete_geo + discard_changes + get + get_data + get_geo + init_ts + is_default + last_update + preload + run_id + set_data + set_as_default + set_geo + + Methods related to :class:`ixmp.Scenario`: + + - Each method has an argument `s`, a reference to the Scenario object being manipulated. + + .. autosummary:: + :nosignatures: + + clone + delete_item + get_meta + has_solution + init_s + init_item + item_delete_elements + item_get_elements + item_set_elements + item_index + list_items + set_meta + + Methods related to :class:`message_ix.Scenario`: + + - Each method has an argument `ms`, a reference to the Scenario object being manipulated. + + .. warning:: These methods may be moved to ixmp in a future release. + + .. autosummary:: + :nosignatures: + + cat_get_elements + cat_list + cat_set_elements diff --git a/doc/source/api-model.rst b/doc/source/api-model.rst new file mode 100644 index 000000000..17cf1cad9 --- /dev/null +++ b/doc/source/api-model.rst @@ -0,0 +1,36 @@ +.. currentmodule:: ixmp.model + +Mathematical models (:mod:`ixmp.model`) +======================================= + +By default, the |ixmp| is installed with :class:`ixmp.model.gams.GAMSModel`, which performs calculations by executing code stored in GAMS files. + +However, |ixmp| is extensible to support other methods of performing calculations or optimization. +Developers wishing to add such capabilities may subclass :class:`ixmp.model.base.Model` and implement its methods. + + +Provided models +--------------- + +.. automodule:: ixmp.model + :members: get_model, MODELS + +.. autoclass:: ixmp.model.gams.GAMSModel + :members: + + +Model API +--------- + +.. autoclass:: ixmp.model.base.Model + :members: name, __init__, run + + In the following, the words REQUIRED, OPTIONAL, etc. have specific meanings as described in `IETF RFC 2119 `_. + + Model is an **abstract** class; this means it MUST be subclassed. + It has two REQURIED methods that MUST be overridden by subclasses: + + .. autosummary:: + name + __init__ + run diff --git a/doc/source/api-python.rst b/doc/source/api-python.rst index 21b96401f..491463bef 100644 --- a/doc/source/api-python.rst +++ b/doc/source/api-python.rst @@ -3,8 +3,188 @@ Python (:mod:`ixmp` package) ============================ -.. automodule:: ixmp - :members: Platform, TimeSeries, Scenario +The |ixmp| application progamming interface (API) is organized around three classes: + +.. autosummary:: + + Platform + TimeSeries + Scenario + +Platform +-------- + +.. autoclass:: Platform + :members: + + Platforms have the following methods: + + .. autosummary:: + add_region + add_region_synonym + add_unit + check_access + regions + scenario_list + set_log_level + + +TimeSeries +---------- + +.. autoclass:: TimeSeries + :members: + + A TimeSeries is uniquely identified on its :class:`Platform` by three + values: + + 1. `model`: the name of a model used to perform calculations between input + and output data. + + - In TimeSeries storing non-model data, arbitrary strings can be used. + - In a :class:`Scenario`, the `model` is a reference to a GAMS program + registered to the :class:`Platform` that can be solved with + :meth:`Scenario.solve`. See :attr:`ixmp.model.MODELS`. + + 2. `scenario`: the name of a specific, coherent description of the real- + world system being modeled. Any `model` may be used to represent mutiple + alternate, or 'counter-factual', `scenarios`. + 3. `version`: an integer identifying a specific iteration of a + (`model`, `scenario`). A new `version` is created by: + + - Instantiating a new TimeSeries with the same `model` and `scenario` as + an existing TimeSeries. + - Calling :meth:`Scenario.clone`. + + Optionally, one `version` may be set as a **default version**. See + :meth:`set_as_default`. + + TimeSeries objects have the following methods: + + .. autosummary:: + add_geodata + add_timeseries + check_out + commit + discard_changes + get_geodata + is_default + last_update + preload_timeseries + remove_geodata + remove_timeseries + run_id + set_as_default + timeseries + + +Scenario +-------- + +.. autoclass:: Scenario + :show-inheritance: + :members: + + A Scenario is a :class:`TimeSeries` associated with a particular model that + can be run on the current :class:`Platform` by calling :meth:`solve`. The + Scenario also stores the output, or 'solution' of a model run; this + includes the 'level' and 'marginal' values of GAMS equations and variables. + + Data in a Scenario are closely related to different types in the GAMS data + model: + + - A **set** is a named collection of labels. See :meth:`init_set`, + :meth:`add_set`, and :meth:`set`. There are two types of sets: + + 1. Sets that are lists of labels. + 2. Sets that are 'indexed' by one or more other set(s). For this type of + set, each member is an ordered tuple of the labels in the index sets. + + - A **scalar** is a named, single, numerical value. See + :meth:`init_scalar`, :meth:`change_scalar`, and :meth:`scalar`. + + - **Parameters**, **variables**, and **equations** are multi-dimensional + arrays of values that are indexed by one or more sets (i.e. with + dimension 1 or greater). The Scenario methods for handling these types + are very similar; they mainly differ in how they are used within GAMS + models registered with ixmp: + + - **Parameters** are generic data that can be defined before a model run. + They may be altered by the model solution. See :meth:`init_par`, + :meth:`remove_par`, :meth:`par_list`, :meth:`add_par`, and :meth:`par`. + - **Variables** are calculated during or after a model run by GAMS code, + so they cannot be modified by a Scenario. See :meth:`init_var`, + :meth:`var_list`, and :meth:`var`. + - **Equations** describe fundamental relationships between other types + (parameters, variables, and scalars) in a model. They are defined in + GAMS code, so cannot be modified by a Scenario. See :meth:`init_equ`, + :meth:`equ_list`, and :meth:`equ`. + + .. autosummary:: + add_par + add_set + change_scalar + clear_cache + clone + equ + equ_list + get_meta + has_equ + has_par + has_set + has_solution + has_var + idx_names + idx_sets + init_equ + init_par + init_scalar + init_set + init_var + load_scenario_data + par + par_list + remove_par + remove_set + remove_solution + scalar + set + set_list + set_meta + solve + var + var_list + +Configuration +------------- + +.. currentmodule:: ixmp.config + +.. autoclass:: Config + + When imported, :mod:`ixmp` reads configuration from the first file named + ``config.json`` found in one of the following directories: + + 1. The directory given by the environment variable ``IXMP_DATA``, if + defined, + 2. ``${XDG_DATA_HOME}/ixmp``, if the environment variable is defined, + 3. ``$HOME/.local/share/ixmp``, or + 4. ``$HOME/.local/ixmp`` (deprecated; retained for compatibility with ixmp + <= 1.1). + + The file may define either or both of the following configuration keys, in + JSON format: + + - `DB_CONFIG_PATH`: location for database properties files. A + :class:`ixmp.Platform` instantiated with a relative path name for the + `dbprops` argument will locate the file first in the current working + directory, then in `DB_CONFIG_PATH`, then in the four directories above. + - `DEFAULT_DBPROPS_FILE`: path to a default database properties file. + A :class:`ixmp.Platform` instantiated with no arguments will use this + file. + - `DEFAULT_LOCAL_DB_PATH`: path to a directory where a local directory + should be created. A :class:`ixmp.Platform` instantiated with + `dbtype='HSQLDB'` will create or reuse a database in this path. Testing utilities diff --git a/doc/source/api.rst b/doc/source/api.rst index 0a6cd3e6a..5208d50b0 100644 --- a/doc/source/api.rst +++ b/doc/source/api.rst @@ -9,6 +9,8 @@ On separate pages: :maxdepth: 2 api-python + api-backend + api-model reporting On this page: @@ -28,6 +30,7 @@ An R interface to the `ixmp` is provided by the ``rixmp`` package. # Load the rixmp package library(rixmp) + ixmp <- import('ixmp') # An 'ixmp' object is added to the global namespace. # It can be used in the same way as the Python ixmp package. @@ -75,46 +78,3 @@ With ``rixmp`` the user can load entire sets of strings or dataframes, which req scen$init_par("a", c("i")) a.df = data.frame( i = i.set, value = c(350 , 600) , unit = 'cases') scen$add_par("a", adapt_to_ret(a.df)) - - -.. _gams-api: - -GAMS ----- - -The *ixmp* :doc:`tutorials ` use a common example problem from -Dantzig :cite:`dantzig-1963`, implemented in GAMS and available -`from the GAMS website `_. -The file ``tutorial/transport/transport_ixmp.gms`` illustrates how such an -existing GAMS model can be simply adapted to work with the |ixmp|. - -The steps are: - -1. The GAMS definitions of sets ``i`` and ``j``, and parameters ``a``, ``b``, - ``d``, and ``f``, are modified to **remove explicit values**. -2. The following lines are added **before** the model definition and solution:: - - $if not set in $setglobal in 'ix_transport_data.gdx' - $if not set out $setglobal out 'ix_transport_results.gdx' - - $gdxin '%in%' - $load i, j, a, b, d, f - $gdxin - -3. The following line is added **after** the model's ``solve ...;`` statement:: - - execute_unload '%out%'; - -*ixmp* uses GAMS command-line options to pass the values of the compile-time -variables ``in`` and ``out``. This causes the model to read its input data from -a GDX-format file created by *ixmp*, and write its output data to a GDX file -specified by *ixmp*. *ixmp* then automatically retrieves the model solution and -other information from the output file, updating the :class:`ixmp.Scenario` and -database. - - -Java ----- - -The `ixmp` is powered by a Java interface to connect a database instance with -the scientific programming interfaces. diff --git a/doc/source/conf.py b/doc/source/conf.py index ff5728af2..6c92cc201 100644 --- a/doc/source/conf.py +++ b/doc/source/conf.py @@ -35,14 +35,14 @@ sys.path.append(os.path.abspath('exts')) extensions = [ 'sphinx.ext.autodoc', + 'sphinx.ext.autosummary', + 'sphinx.ext.coverage', 'sphinx.ext.doctest', 'sphinx.ext.intersphinx', - 'sphinx.ext.todo', - 'sphinx.ext.coverage', 'sphinx.ext.mathjax', + 'sphinx.ext.napoleon', + 'sphinx.ext.todo', 'sphinxcontrib.bibtex', - 'sphinx.ext.autosummary', - 'sphinx.ext.napoleon' ] # Add any paths that contain templates here, relative to this directory. diff --git a/doc/source/tutorials.rst b/doc/source/tutorials.rst index c3db975de..d187fb0c3 100644 --- a/doc/source/tutorials.rst +++ b/doc/source/tutorials.rst @@ -1 +1,36 @@ .. include:: ../../tutorial/README.rst + +Adapting GAMS models for :mod:`ixmp` +------------------------------------ + +The common example optimization from Dantzig :cite:`dantzig-1963`, is available in a GAMS implementation +`from the GAMS website `_. +The file ``tutorial/transport/transport_ixmp.gms`` illustrates how an +existing GAMS model can be adapted to work with the |ixmp|. +The same, simple procedure can be applied to any GAMS code. + +The steps are: + +1. Modify the definitions of GAMS sets (``i`` and ``j``) and parameters (``a``, ``b``, ``d``, and ``f``) to **remove explicit values**. +2. Add lines to **read the model input data passed by ixmp**. + + The following lines are added *before* the code that defines and solves the model:: + + * These two lines let the model code be run outside of ixmp, if needed + $if not set in $setglobal in 'ix_transport_data.gdx' + $if not set out $setglobal out 'ix_transport_results.gdx' + + $gdxin '%in%' + $load i, j, a, b, d, f + $gdxin + + +3. Add a line to **write the model output data**. + + The following line is added *after* the model's ``solve ...;`` statement:: + + execute_unload '%out%'; + +*ixmp*'s :class:`~.GAMSModel` class uses command-line options to pass the values of the variables ``in`` and ``out``. +This causes the model to read its input data from a GDX-format file created by *ixmp*, and write its output data to a GDX file specified by *ixmp*. +*ixmp* then automatically retrieves the model solution and other information from the output file, updating the :class:`~.Scenario` and storage :class:`~.Backend`. diff --git a/ixmp/__init__.py b/ixmp/__init__.py index 1936d1389..1aa7c62fd 100644 --- a/ixmp/__init__.py +++ b/ixmp/__init__.py @@ -1,25 +1,24 @@ -import sys - from ._version import get_versions -__version__ = get_versions()['version'] -del get_versions - -from ixmp.core import * - -from ixmp import ( - model_settings, - utils, +from ixmp.core import ( # noqa: F401 + IAMC_IDX, + Platform, + TimeSeries, + Scenario, ) +from .backend import BACKENDS +from .backend.jdbc import JDBCBackend +from .model import MODELS +from .model.gams import GAMSModel +from ixmp.reporting import Reporter # noqa: F401 +__version__ = get_versions()['version'] +del get_versions -if sys.version_info[0] == 3: - from ixmp.reporting import Reporter # noqa: F401 - +# Register Backends provided by ixmp +BACKENDS['jdbc'] = JDBCBackend -model_settings.register_model( - 'default', - model_settings.ModelConfig(model_file='"{model}.gms"', - inp='{model}_in.gdx', - outp='{model}_out.gdx', - args=['--in="{inp}"', '--out="{outp}"']) -) +# Register Models provided by ixmp +MODELS.update({ + 'default': GAMSModel, + 'gams': GAMSModel, +}) diff --git a/ixmp/_version.py b/ixmp/_version.py index d1d4fbcd3..f5206fab9 100644 --- a/ixmp/_version.py +++ b/ixmp/_version.py @@ -40,7 +40,7 @@ def get_config(): # _version.py cfg = VersioneerConfig() cfg.VCS = "git" - cfg.style = "pep440" + cfg.style = "pep440-pre" cfg.tag_prefix = "v" cfg.parentdir_prefix = "ixmp-" cfg.versionfile_source = "ixmp/_version.py" diff --git a/ixmp/backend/__init__.py b/ixmp/backend/__init__.py new file mode 100644 index 000000000..bf363d02b --- /dev/null +++ b/ixmp/backend/__init__.py @@ -0,0 +1,16 @@ +#: Lists of field names for tuples returned by Backend API methods. +FIELDS = { + 'get_nodes': ('region', 'mapped_to', 'parent', 'hierarchy'), + 'get_scenarios': ('model', 'scenario', 'scheme', 'is_default', + 'is_locked', 'cre_user', 'cre_date', 'upd_user', + 'upd_date', 'lock_user', 'lock_date', 'annotation', + 'version'), + 'ts_get': ('region', 'variable', 'unit', 'year', 'value'), + 'ts_get_geo': ('region', 'variable', 'time', 'year', 'value', 'unit', + 'meta'), +} + + +#: Mapping from names to available backends. To register additional backends, +#: add elements to this variable. +BACKENDS = {} diff --git a/ixmp/backend/base.py b/ixmp/backend/base.py new file mode 100644 index 000000000..a0d8f49fc --- /dev/null +++ b/ixmp/backend/base.py @@ -0,0 +1,761 @@ +from abc import ABC, abstractmethod + +from ixmp.core import TimeSeries, Scenario + + +class Backend(ABC): + """Abstract base class for backends.""" + # NB non-abstract methods like close_db() are marked '# pragma: no cover'. + # In order to cover these with tests, define a MemoryBackend or similar + # that provides implementations of all the abstract methods but does + # NOT override the non-abstract methods; then call those. + + def __init__(self): # pragma: no cover + """OPTIONAL: Initialize the backend.""" + pass + + def __call__(self, obj, method, *args, **kwargs): + """Call the backend method *method* for *obj*. + + The class attribute obj._backend_prefix is used to determine a prefix + for the method name, e.g. 'ts_{method}'. + """ + return getattr(self, method)(obj, *args, **kwargs) + + def close_db(self): # pragma: no cover + """OPTIONAL: Close database connection(s). + + Close any database connection(s), if open. + """ + pass + + def get_auth(self, user, models, kind): # pragma: no cover + """OPTIONAL: Return user authorization for *models*. + + If the Backend implements access control, this method **must** indicate + whether *user* has permission *kind* for each of *models*. + + *kind* **may** be 'read'/'view', 'write'/'modify', or other values; + :meth:`get_auth` **should** raise exceptions on invalid values. + + Parameters + ---------- + user : str + User name or identifier. + models : list of str + Model names. + kind : str + Type of permission being requested + + Returns + ------- + dict + Mapping of *model name (str)* → :class:`bool`; :obj:`True` if the + user is authorized for the model. + """ + return {model: True for model in models} + + @abstractmethod + def get_nodes(self): + """Iterate over all nodes stored on the Platform. + + Yields + ------- + tuple + The members of each tuple are: + + ========= =========== === + ID Type Description + ========= =========== === + region str Node name or synonym for node + mapped_to str or None Node name + parent str Parent node name + hierarchy str Node hierarchy ID + ========= =========== === + + See also + -------- + set_node + """ + + @abstractmethod + def get_scenarios(self, default, model, scenario): + """Iterate over TimeSeries stored on the Platform. + + Scenarios, as subclasses of TimeSeries, are also included. + + Parameters + ---------- + default : bool + :obj:`True` to include only TimeSeries versions marked as default. + model : str or None + Model name to filter results. + scenario : str or None + Scenario name to filter results. + + Yields + ------ + tuple + The members of each tuple are: + + ========== ==== === + ID Type Description + ========== ==== === + model str Model name + scenario str Scenario name + scheme str Scheme name + is_default bool :obj:`True` if `version` is the default + is_locked bool :obj:`True` if read-only + cre_user str Name of user who created the TimeSeries + cre_date str Creation datetime + upd_user str Name of user who last modified the TimeSeries + upd_date str Modification datetime + lock_user str Name of user who locked the TimeSeries + lock_date str Lock datetime + annotation str Description of the TimeSeries + version int Version + ========== ==== === + """ + + @abstractmethod + def get_units(self): + """Return all registered symbols for units of measurement. + + Returns + ------- + list of str + + See also + -------- + set_unit + """ + + def open_db(self): # pragma: no cover + """OPTIONAL: (Re-)open database connection(s). + + A backend **may** connect to a database server. This method opens the + database connection if it is closed. + """ + pass + + def set_log_level(self, level): # pragma: no cover + """OPTIONAL: Set logging level for the backend and other code. + + Parameters + ---------- + level : int or Python logging level + """ + pass + + @abstractmethod + def set_node(self, name, parent=None, hierarchy=None, synonym=None): + """Add a node name to the Platform. + + This method **must** have one of two effects, depending on the + arguments: + + - With `parent` and `hierarchy`: `name` is added as a child of `parent` + in the named `hierarchy`. + - With `synonym`: `synonym` is added as an alias for `name`. + + Parameters + ---------- + name : str + Node name. + parent : str, optional + Parent node name. + hierarchy : str, optional + Node hierarchy ID. + synonym : str, optional + Synonym for node. + + See also + -------- + get_nodes + """ + + @abstractmethod + def set_unit(self, name, comment): + """Add a unit of measurement to the Platform. + + Parameters + ---------- + name : str + Symbol of the unit. + comment : str + Description of the change or of the unit. + + See also + -------- + get_units + """ + + # Methods for ixmp.TimeSeries + + @abstractmethod + def init_ts(self, ts: TimeSeries, annotation=None): + """Initialize the TimeSeries *ts*. + + ts_init **may** modify the :attr:`~TimeSeries.version` attribute of + *ts*. + + Parameters + ---------- + annotation : str + If *ts* is newly-created, the Backend **must** store this + annotation with the TimeSeries. + + Returns + ------- + None + """ + + @abstractmethod + def get(self, ts: TimeSeries, version): + """Retrieve the existing TimeSeries or Scenario *ts*. + + The TimeSeries is identified based on its (:attr:`~.TimeSeries.model`, + :attr:`~.TimeSeries.scenario`) and *version*. + + If *ts* is a Scenario, :meth:`get` **must** set the + :attr:`~.Scenario.scheme` attribute on it. + + Parameters + ---------- + version : str or None + If :obj:`None`, the version marked as the default is returned, and + ts_get **must** set :attr:`.TimeSeries.version` attribute on *ts*. + + Returns + ------- + None + + See also + -------- + ts_set_as_default + """ + + @abstractmethod + def check_out(self, ts: TimeSeries, timeseries_only): + """Check out *ts* for modification. + + Parameters + ---------- + timeseries_only : bool + ??? + + Returns + ------- + None + """ + + @abstractmethod + def commit(self, ts: TimeSeries, comment): + """Commit changes to *ts*. + + ts_init **may** modify the :attr:`~.TimeSeries.version` attribute of + *ts*. + + Parameters + ---------- + comment : str + Description of the changes being committed. + + Returns + ------- + None + """ + + @abstractmethod + def get_data(self, ts: TimeSeries, region, variable, unit, year): + """Retrieve time-series data. + + Parameters + ---------- + region : list of str + Region names to filter results. + variable : list of str + Variable names to filter results. + unit : list of str + Unit symbols to filter results. + year : list of str + Years to filter results. + + Yields + ------ + tuple + The members of each tuple are: + + ======== ===== === + ID Type Description + ======== ===== === + region str Region name + variable str Variable name + unit str Unit symbol + year int Year + value float Data value + ======== ===== === + """ + + @abstractmethod + def get_geo(self, ts: TimeSeries): + """Retrieve time-series 'geodata'. + + Yields + ------ + tuple + The members of each tuple are: + + ======== ==== === + ID Type Description + ======== ==== === + region str Region name + variable str Variable name + time str Time period + year int Year + value str Value + unit str Unit symbol + meta bool :obj:`True` if the data is marked as metadata + ======== ==== === + """ + + @abstractmethod + def set_data(self, ts: TimeSeries, region, variable, data, unit, meta): + """Store *data*. + + Parameters + ---------- + region : str + Region name. + variable : str + Variable name. + time : str + Time period. + unit : str + Unit symbol. + data : dict (int -> float) + Mapping from year to value. + meta : bool + :obj:`True` to mark *data* as metadata. + """ + + @abstractmethod + def set_geo(self, ts: TimeSeries, region, variable, time, year, value, + unit, meta): + """Store time-series 'geodata'. + + Parameters + ---------- + region : str + Region name. + variable : str + Variable name. + time : str + Time period. + year : int + Year. + value : str + Data value. + unit : str + Unit symbol. + meta : bool + :obj:`True` to mark *data* as metadata. + """ + + @abstractmethod + def delete(self, ts: TimeSeries, region, variable, years, unit): + """Remove data values. + + Parameters + ---------- + region : str + Region name. + variable : str + Variable name. + years : Iterable of int + Years. + unit : str + Unit symbol. + + Returns + ------- + None + """ + + @abstractmethod + def delete_geo(self, ts: TimeSeries, region, variable, time, years, unit): + """Remove 'geodata' values. + + Parameters + ---------- + region : str + Region name. + variable : str + Variable name. + time : str + Time period. + years : Iterable of int + Years. + unit : str + Unit symbol. + + Returns + ------- + None + """ + + @abstractmethod + def discard_changes(self, ts: TimeSeries): + """Discard changes to *ts* since the last :meth:`ts_check_out`. + + Returns + ------- + None + """ + + @abstractmethod + def set_as_default(self, ts: TimeSeries): + """Set the current :attr:`.TimeSeries.version` as the default. + + Returns + ------- + None + + See also + -------- + ts_is_default + ts_get + s_get + """ + + @abstractmethod + def is_default(self, ts: TimeSeries): + """Return :obj:`True` if *ts* is the default version. + + Returns + ------- + bool + + See also + -------- + ts_set_as_default + ts_get + s_get + """ + + @abstractmethod + def last_update(self, ts: TimeSeries): + """Return the date of the last modification of the *ts*. + + Returns + ------- + str + """ + + @abstractmethod + def run_id(self, ts: TimeSeries): + """Return the run ID for the *ts*. + + Returns + ------- + int + """ + + def preload(self, ts: TimeSeries): # pragma: no cover + """OPTIONAL: Load *ts* data into memory.""" + pass + + # Methods for ixmp.Scenario + + @abstractmethod + def init_s(self, s: Scenario, scheme, annotation): + """Initialize the Scenario *s*. + + s_init **may** modify the :attr:`~.TimeSeries.version` attribute of + *s*. + + Parameters + ---------- + scheme : str + Description of the scheme of the Scenario. + annotation : str + Description of the Scenario. + + Returns + ------- + None + """ + + @abstractmethod + def clone(self, s: Scenario, platform_dest, model, scenario, annotation, + keep_solution, first_model_year=None): + """Clone *s*. + + Parameters + ---------- + platform_dest : :class:`.Platform` + Target backend. May be the same as *s.platform*. + model : str + New model name. + scenario : str + New scenario name. + annotation : str + Description for the creation of the new scenario. + keep_solution : bool + If :obj:`True`, model solution data is also cloned. If + :obj:`False`, it is discarded. + first_model_year : int or None + If :class:`int`, must be greater than the first model year of *s*. + + Returns + ------- + Same class as *s* + The cloned Scenario. + """ + + @abstractmethod + def has_solution(self, s: Scenario): + """Return `True` if Scenario *s* has been solved. + + If :obj:`True`, model solution data is available from the Backend. + """ + + @abstractmethod + def list_items(self, s: Scenario, type): + """Return a list of items of *type*. + + Parameters + ---------- + type : 'set' or 'par' or 'equ' + + Return + ------ + list of str + """ + + @abstractmethod + def init_item(self, s: Scenario, type, name): + """Initialize an item *name* of *type*. + + Parameters + ---------- + type : 'set' or 'par' or 'equ' + name : str + Name for the new item. + + Return + ------ + None + """ + + @abstractmethod + def delete_item(self, s: Scenario, type, name): + """Remove an item *name* of *type*. + + Parameters + ---------- + type : 'set' or 'par' or 'equ' + name : str + Name of the item to delete. + + Return + ------ + None + """ + + @abstractmethod + def item_index(self, s: Scenario, name, sets_or_names): + """Return the index sets or names of item *name*. + + Parameters + ---------- + sets_or_names : 'sets' or 'names' + + Returns + ------- + list of str + """ + + @abstractmethod + def item_get_elements(self, s: Scenario, type, name, filters=None): + """Return elements of item *name*. + + Parameters + ---------- + type : 'equ' or 'par' or 'set' or 'var' + name : str + Name of the item. + filters : dict (str -> list of str), optional + If provided, a mapping from dimension names to allowed values + along that dimension. + + Returns + ------- + pandas.Series + When *type* is 'set' and *name* an index set (not indexed by other + sets). + dict + When *type* is 'equ', 'par', or 'set' and *name* is scalar (zero- + dimensional). The value has the keys 'value' and 'unit' (for 'par') + or 'lvl' and 'mrg' (for 'equ' or 'var'). + pandas.DataFrame + For mapping sets, or all 1+-dimensional values. The dataframe has + one column per index name with dimension values; plus the columns + 'value' and 'unit' (for 'par') or 'lvl' and 'mrg' (for 'equ' or + 'var'). + """ + + @abstractmethod + def item_set_elements(self, s: Scenario, type, name, elements): + """Add keys or values to item *name*. + + Parameters + ---------- + type : 'par' or 'set' + name : str + Name of the items. + elements : iterable of 4-tuple + The members of each tuple are: + + ======= ========================== === + ID Type Description + ======= ========================== === + key str or list of str or None Set elements or value indices + value float or None Parameter value + unit str or None Unit symbol + comment str or None Description of the change + ======= ========================== === + + If *name* is indexed by other set(s), then the number of elements + of each *key*, and their contents, must match the index set(s). + When *type* is 'set', *value* and *unit* **must** be :obj:`None`. + + Raises + ------ + ValueError + If *elements* contain invalid values, e.g. key values not in the + index set(s). + Exception + If the Backend encounters any error adding the elements. + + See also + -------- + s_init_item + s_item_delete_elements + """ + + @abstractmethod + def item_delete_elements(self, s: Scenario, type, name, keys): + """Remove elements of item *name*. + + Parameters + ---------- + type : 'par' or 'set' + name : str + keys : iterable of iterable of str + If *name* is indexed by other set(s), then the number of elements + of each key in *keys*, and their contents, must match the index + set(s). + If *name* is a basic set, then each key must be a list containing a + single str, which must exist in the set. + + See also + -------- + s_init_item + s_item_set_elements + """ + + @abstractmethod + def get_meta(self, s: Scenario): + """Return all metadata. + + Returns + ------- + dict (str -> any) + Mapping from metadata keys to values. + + See also + -------- + s_get_meta + """ + + @abstractmethod + def set_meta(self, s: Scenario, name, value): + """Set a single metadata key. + + Parameters + ---------- + name : str + Metadata key name. + value : any + Value for *name*. + + Returns + ------- + None + """ + + @abstractmethod + def clear_solution(self, s: Scenario, from_year=None): + """Remove data associated with a model solution. + + .. todo:: Document. + """ + + # Methods for message_ix.Scenario + + @abstractmethod + def cat_list(self, ms: Scenario, name): + """Return list of categories in mapping *name*. + + Parameters + ---------- + name : str + Name of the category mapping set. + + Returns + ------- + list of str + All categories in *name*. + """ + + @abstractmethod + def cat_get_elements(self, ms: Scenario, name, cat): + """Get elements of a category mapping. + + Parameters + ---------- + name : str + Name of the category mapping set. + cat : str + Name of the category within *name*. + + Returns + ------- + list of str + All set elements mapped to *cat* in *name*. + """ + + @abstractmethod + def cat_set_elements(self, ms: Scenario, name, cat, keys, is_unique): + """Add elements to category mapping. + + Parameters + ---------- + name : str + Name of the category mapping set. + cat : str + Name of the category within *name*. + keys : iterable of str or list of str + Keys to add to *cat*. + is_unique : bool + If :obj:`True`: + + - *keys* **must** contain only one key. + - The Backend **must** remove any existing member of *cat*, so that + it has only one element. + + Returns + ------- + None + """ diff --git a/ixmp/backend/jdbc.py b/ixmp/backend/jdbc.py new file mode 100644 index 000000000..1d19b88db --- /dev/null +++ b/ixmp/backend/jdbc.py @@ -0,0 +1,647 @@ +from collections import ChainMap +from collections.abc import Collection, Iterable +from functools import lru_cache +import os +from pathlib import Path +import re +from types import SimpleNamespace + +import jpype +from jpype import JClass +import numpy as np +import pandas as pd + +from ixmp.config import _config +from ixmp.core import Scenario +from ixmp.utils import islistable, logger +from . import FIELDS +from .base import Backend + + +# Map of Python to Java log levels +LOG_LEVELS = { + 'CRITICAL': 'ALL', + 'ERROR': 'ERROR', + 'WARNING': 'WARN', + 'INFO': 'INFO', + 'DEBUG': 'DEBUG', + 'NOTSET': 'OFF', +} + +# Java classes, loaded by start_jvm(). These become available as e.g. +# java.IxException or java.HashMap. +java = SimpleNamespace() + +JAVA_CLASSES = [ + 'at.ac.iiasa.ixmp.exceptions.IxException', + 'at.ac.iiasa.ixmp.objects.Scenario', + 'at.ac.iiasa.ixmp.objects.TimeSeries.TimeSpan', + 'at.ac.iiasa.ixmp.Platform', + 'java.lang.Double', + 'java.lang.Integer', + 'java.math.BigDecimal', + 'java.util.HashMap', + 'java.util.LinkedHashMap', + 'java.util.LinkedList', +] + + +class JDBCBackend(Backend): + """Backend using JPype/JDBC to connect to Oracle and HyperSQLDB instances. + + Parameters + ---------- + dbtype : 'HSQLDB', optional + Database type to use. If :obj:`None`, a remote database is accessed. If + 'HSQLDB', a local database is created and used at the path given by + `dbprops`. + + dbprops : path-like, optional + If `dbtype` is :obj:`None`, the name of a *database properties file* + (default: ``default.properties``) in the properties file directory + (see :class:`.Config`) or a path to a properties file. + + If `dbtype` is 'HSQLDB'`, the path of a local database, + (default: ``$HOME/.local/ixmp/localdb/default``) or name of a + database file in the local database directory (default: + ``$HOME/.local/ixmp/localdb/``). + + jvmargs : str, optional + Java Virtual Machine arguments. See :func:`.start_jvm`. + """ + # NB Much of the code of this backend is in Java, in the iiasa/ixmp_source + # Github repository. + # + # Among other abstractions, this backend: + # + # - Handles any conversion between Java and Python types that is not + # done automatically by JPype. + # - Catches Java exceptions such as ixmp.exceptions.IxException, and + # re-raises them as appropriate Python exceptions. + # + # Limitations: + # + # - s_clone() is only supported when target_backend is JDBCBackend. + + #: Reference to the at.ac.iiasa.ixmp.Platform Java object + jobj = None + + #: Mapping from ixmp.TimeSeries object to the underlying + #: at.ac.iiasa.ixmp.Scenario object (or subclasses of either) + jindex = {} + + def __init__(self, dbprops=None, dbtype=None, jvmargs=None): + start_jvm(jvmargs) + self.dbtype = dbtype + + try: + # if no dbtype is specified, launch Platform with properties file + if dbtype is None: + dbprops = _config.find_dbprops(dbprops) + if dbprops is None: + raise ValueError("Not found database properties file " + "to launch platform") + logger().info("launching ixmp.Platform using config file at " + "'{}'".format(dbprops)) + self.jobj = java.Platform('Python', str(dbprops)) + # if dbtype is specified, launch Platform with local database + elif dbtype == 'HSQLDB': + dbprops = dbprops or _config.get('DEFAULT_LOCAL_DB_PATH') + logger().info("launching ixmp.Platform with local {} database " + "at '{}'".format(dbtype, dbprops)) + self.jobj = java.Platform('Python', str(dbprops), dbtype) + else: + raise ValueError('Unknown dbtype: {}'.format(dbtype)) + except TypeError: + msg = ("Could not launch the JVM for the ixmp.Platform." + "Make sure that all dependencies of ixmp.jar" + "are included in the 'ixmp/lib' folder.") + logger().info(msg) + raise + + # Platform methods + + def set_log_level(self, level): + self.jobj.setLogLevel(LOG_LEVELS[level]) + + def open_db(self): + """(Re-)open the database connection.""" + self.jobj.openDB() + + def close_db(self): + """Close the database connection. + + A HSQL database can only be used by one :class:`Backend` instance at a + time. Any existing connection must be closed before a new one can be + opened. + """ + self.jobj.closeDB() + + def get_auth(self, user, models, kind): + return self.jobj.checkModelAccess(user, kind, to_jlist2(models)) + + def set_node(self, name, parent=None, hierarchy=None, synonym=None): + if parent and hierarchy and not synonym: + self.jobj.addNode(name, parent, hierarchy) + elif synonym and not (parent or hierarchy): + self.jobj.addNodeSynonym(synonym, name) + + def get_nodes(self): + for r in self.jobj.listNodes('%'): + n, p, h = r.getName(), r.getParent(), r.getHierarchy() + yield (n, None, p, h) + yield from [(s, n, p, h) for s in (r.getSynonyms() or [])] + + def get_scenarios(self, default, model, scenario): + # List> + scenarios = self.jobj.getScenarioList(default, model, scenario) + + for s in scenarios: + data = [] + for field in FIELDS['get_scenarios']: + data.append(int(s[field]) if field == 'version' else s[field]) + yield data + + def set_unit(self, name, comment): + self.jobj.addUnitToDB(name, comment) + + def get_units(self): + return to_pylist(self.jobj.getUnitList()) + + # Timeseries methods + + def _common_init(self, ts, klass, *args): + """Common code for ts_init and s_init.""" + method = getattr(self.jobj, 'new' + klass) + # Create a new TimeSeries + jobj = method(ts.model, ts.scenario, *args) + + # Add to index + self.jindex[ts] = jobj + + # Retrieve initial version + ts.version = jobj.getVersion() + + def init_ts(self, ts, annotation=None): + self._common_init(ts, 'TimeSeries', annotation) + + def get(self, ts, version): + args = [ts.model, ts.scenario] + if isinstance(version, int): + # Load a TimeSeries of specific version + args.append(version) + + # either getTimeSeries or getScenario + method = getattr(self.jobj, 'get' + ts.__class__.__name__) + jobj = method(*args) + # Add to index + self.jindex[ts] = jobj + + if version is None: + # Update the version attribute + ts.version = jobj.getVersion() + else: + assert version == jobj.getVersion() + + if isinstance(ts, Scenario): + # Also retrieve the scheme + ts.scheme = jobj.getScheme() + + def check_out(self, ts, timeseries_only): + self.jindex[ts].checkOut(timeseries_only) + + def commit(self, ts, comment): + self.jindex[ts].commit(comment) + if ts.version == 0: + ts.version = self.jindex[ts].getVersion() + + def discard_changes(self, ts): + self.jindex[ts].discardChanges() + + def set_as_default(self, ts): + self.jindex[ts].setAsDefaultVersion() + + def is_default(self, ts): + return bool(self.jindex[ts].isDefault()) + + def last_update(self, ts): + return self.jindex[ts].getLastUpdateTimestamp().toString() + + def run_id(self, ts): + return self.jindex[ts].getRunId() + + def preload(self, ts): + self.jindex[ts].preloadAllTimeseries() + + def get_data(self, ts, region, variable, unit, year): + # Convert the selectors to Java lists + r = to_jlist2(region) + v = to_jlist2(variable) + u = to_jlist2(unit) + y = to_jlist2(year) + + # Field types + ftype = { + 'year': int, + 'value': float, + } + + # Iterate over returned rows + for row in self.jindex[ts].getTimeseries(r, v, u, None, y): + # Get the value of each field and maybe convert its type + yield tuple(ftype.get(f, str)(row.get(f)) + for f in FIELDS['ts_get']) + + def get_geo(self, ts): + # NB the return type of getGeoData() requires more processing than + # getTimeseries. It also accepts no selectors. + + # Field types + ftype = { + 'meta': int, + 'time': lambda ord: timespans()[int(ord)], # Look up the name + 'year': lambda obj: obj, # Pass through; handled later + } + + # Returned names in Java data structure do not match API column names + jname = { + 'meta': 'meta', + 'region': 'nodeName', + 'time': 'time', + 'unit': 'unitName', + 'variable': 'keyString', + 'year': 'yearlyData' + } + + # Iterate over rows from the Java backend + for row in self.jindex[ts].getGeoData(): + data1 = {f: ftype.get(f, str)(row.get(jname.get(f, f))) + for f in FIELDS['ts_get_geo'] if f != 'value'} + + # At this point, the 'year' key is a not a single value, but a + # year -> value mapping with multiple entries + yv_entries = data1.pop('year').entrySet() + + # Construct a chain map: look up in data1, then data2 + data2 = {'year': None, 'value': None} + cm = ChainMap(data1, data2) + + for yv in yv_entries: + # Update data2 + data2['year'] = yv.getKey() + data2['value'] = yv.getValue() + + # Construct a row with a single value + yield tuple(cm[f] for f in FIELDS['ts_get_geo']) + + def set_data(self, ts, region, variable, data, unit, meta): + # Convert *data* to a Java data structure + jdata = java.LinkedHashMap() + for k, v in data.items(): + # Explicit cast is necessary; otherwise java.lang.Long + jdata.put(java.Integer(k), v) + + self.jindex[ts].addTimeseries(region, variable, None, jdata, unit, + meta) + + def set_geo(self, ts, region, variable, time, year, value, unit, meta): + self.jindex[ts].addGeoData(region, variable, time, java.Integer(year), + value, unit, meta) + + def delete(self, ts, region, variable, years, unit): + years = to_jlist2(years, java.Integer) + self.jindex[ts].removeTimeseries(region, variable, None, years, unit) + + def delete_geo(self, ts, region, variable, time, years, unit): + years = to_jlist2(years, java.Integer) + self.jindex[ts].removeGeoData(region, variable, time, years, unit) + + # Scenario methods + + def init_s(self, s, scheme, annotation): + self._common_init(s, 'Scenario', scheme, annotation) + + def clone(self, s, platform_dest, model, scenario, annotation, + keep_solution, first_model_year=None): + # Raise exceptions for limitations of JDBCBackend + if not isinstance(platform_dest._backend, self.__class__): + raise NotImplementedError(f'Clone between {self.__class__} and' + f'{platform_dest._backend.__class__}') + elif platform_dest._backend is not self: + msg = 'Cross-platform clone of {}.Scenario with'.format( + s.__class__.__module__.split('.')[0]) + if keep_solution is False: + raise NotImplementedError(msg + ' `keep_solution=False`') + elif 'message_ix' in msg and first_model_year is not None: + raise NotImplementedError(msg + ' first_model_year != None') + + # Prepare arguments + args = [platform_dest._backend.jobj, model, scenario, annotation, + keep_solution] + if first_model_year: + args.append(first_model_year) + + # Reference to the cloned Java object + jclone = self.jindex[s].clone(*args) + + # Instantiate same class as the original object + return s.__class__(platform_dest, model, scenario, + version=jclone.getVersion(), cache=s._cache) + + def has_solution(self, s): + return self.jindex[s].hasSolution() + + def list_items(self, s, type): + return to_pylist(getattr(self.jindex[s], f'get{type.title()}List')()) + + def init_item(self, s, type, name, idx_sets, idx_names): + # generate index-set and index-name lists + if isinstance(idx_sets, set) or isinstance(idx_names, set): + raise ValueError('index dimension must be string or ordered lists') + idx_sets = to_jlist(idx_sets) + idx_names = to_jlist(idx_names if idx_names is not None else idx_sets) + + # Initialize the Item + func = getattr(self.jindex[s], f'initialize{type.title()}') + + # The constructor returns a reference to the Java Item, but these + # aren't exposed by Backend, so don't return here + func(name, idx_sets, idx_names) + + def delete_item(self, s, type, name): + getattr(self.jindex[s], f'remove{type.title()}')() + + def item_index(self, s, name, sets_or_names): + jitem = self._get_item(s, 'item', name, load=False) + return list(getattr(jitem, f'getIdx{sets_or_names.title()}')()) + + def item_get_elements(self, s, type, name, filters=None): + # Retrieve the item + item = self._get_item(s, type, name, load=True) + + # Get list of elements, using filters if provided + if filters is not None: + jFilter = java.HashMap() + for dim, values in filters.items(): + jFilter.put(dim, to_jlist(values)) + jList = item.getElements(jFilter) + else: + jList = item.getElements() + + if item.getDim() > 0: + # Mapping set or multi-dimensional equation, parameter, or variable + idx_names = list(item.getIdxNames()) + idx_sets = list(item.getIdxSets()) + + data = {} + types = {} + + # Retrieve index columns + for d, (d_name, d_set) in enumerate(zip(idx_names, idx_sets)): + data[d_name] = item.getCol(d, jList) + if d_set == 'year': + # Record column for later type conversion + types[d_name] = int + + # Retrieve value columns + if type == 'par': + data['value'] = item.getValues(jList) + data['unit'] = item.getUnits(jList) + + if type in ('equ', 'var'): + data['lvl'] = item.getLevels(jList) + data['mrg'] = item.getMarginals(jList) + + return pd.DataFrame.from_dict(data, orient='columns') \ + .astype(types) + elif type == 'set': + # Index sets + return pd.Series(item.getCol(0, jList)) + elif type == 'par': + # Scalar parameters + return dict(value=item.getScalarValue().floatValue(), + unit=str(item.getScalarUnit())) + elif type in ('equ', 'var'): + # Scalar equations and variables + return dict(lvl=item.getScalarLevel().floatValue(), + mrg=item.getScalarMarginal().floatValue()) + + def item_set_elements(self, s, type, name, elements): + jobj = self._get_item(s, type, name) + + try: + for key, value, unit, comment in elements: + # Prepare arguments + args = [to_jlist2(key)] if key else [] + if type == 'par': + args.extend([java.Double(value), unit]) + if comment: + args.append(comment) + + # Activates one of 5 signatures for addElement: + # - set: (key) + # - set: (key, comment) + # - par: (key, value, unit, comment) + # - par: (key, value, unit) + # - par: (value, unit, comment) + jobj.addElement(*args) + except java.IxException as e: + msg = e.message() + if 'does not have an element' in msg: + # Re-raise as Python ValueError + raise ValueError(msg) from e + else: + raise RuntimeError('Unhandled Java exception') from e + + def item_delete_elements(self, s, type, name, keys): + jitem = self._get_item(s, type, name, load=False) + for key in keys: + jitem.removeElement(to_jlist2(key)) + + def get_meta(self, s): + def unwrap(v): + """Unwrap metadata numeric value (BigDecimal -> Double)""" + return v.doubleValue() if isinstance(v, java.BigDecimal) else v + + return {entry.getKey(): unwrap(entry.getValue()) + for entry in self.jindex[s].getMeta().entrySet()} + + def set_meta(self, s, name, value): + self.jindex[s].setMeta(name, value) + + def clear_solution(self, s, from_year=None): + from ixmp.core import Scenario + + if from_year: + if type(s) is not Scenario: + raise TypeError('s_clear_solution(from_year=...) only valid ' + 'for ixmp.Scenario; not subclasses') + self.jindex[s].removeSolution(from_year) + else: + self.jindex[s].removeSolution() + + # MsgScenario methods + + def cat_list(self, ms, name): + return to_pylist(self.jindex[ms].getTypeList(name)) + + def cat_get_elements(self, ms, name, cat): + return to_pylist(self.jindex[ms].getCatEle(name, cat)) + + def cat_set_elements(self, ms, name, cat, keys, is_unique): + self.jindex[ms].addCatEle(name, cat, to_jlist2(keys), is_unique) + + # Helpers; not part of the Backend interface + + def write_gdx(self, s, path): + """Write the Scenario to a GDX file at *path*.""" + # include_var_equ=False -> do not include variables/equations in GDX + self.jindex[s].toGDX(str(path.parent), path.name, False) + + def read_gdx(self, s, path, check_solution, comment, equ_list, var_list): + """Read the Scenario from a GDX file at *path*. + + Parameters + ---------- + check_solution : bool + If True, raise an exception if the GAMS solver did not reach + optimality. (Only for MESSAGE-scheme Scenarios.) + comment : str + Comment added to Scenario when importing the solution. + equ_list : list of str + Equations to be imported. + var_list : list of str + Variables to be imported. + """ + self.jindex[s].readSolutionFromGDX( + str(path.parent), path.name, comment, var_list, equ_list, + check_solution) + + def _get_item(self, s, ix_type, name, load=True): + """Return the Java object for item *name* of *ix_type*. + + Parameters + ---------- + load : bool, optional + If *ix_type* is 'par', 'var', or 'equ', the elements of the item + are loaded from the database before :meth:`_item` returns. If + :const:`False`, the elements can be loaded later using + ``item.loadItemElementsfromDB()``. + """ + # getItem is not overloaded to accept a second bool argument + args = [name] + ([load] if ix_type != 'item' else []) + try: + return getattr(self.jindex[s], f'get{ix_type.title()}')(*args) + except java.IxException as e: + if re.match('No item [^ ]* exists in this Scenario', e.args[0]): + # Re-raise as a Python KeyError + raise KeyError(f'No {ix_type.title()} {name!r} exists in this ' + 'Scenario!') from None + else: + raise RuntimeError('Unhandled Java exception') from e + + +def start_jvm(jvmargs=None): + """Start the Java Virtual Machine via :mod:`JPype`. + + Parameters + ---------- + jvmargs : str or list of str, optional + Additional arguments for launching the JVM, passed to + :func:`jpype.startJVM`. + + For instance, to set the maximum heap space to 4 GiB, give + ``jvmargs=['-Xmx4G']``. See the `JVM documentation`_ for a list of + options. + + .. _`JVM documentation`: https://docs.oracle.com/javase/7/docs + /technotes/tools/windows/java.html) + """ + # TODO change the jvmargs default to [] instead of None + if jpype.isJVMStarted(): + return + + jvmargs = jvmargs or [] + + # Arguments + args = [jpype.getDefaultJVMPath()] + + # Add the ixmp root directory, ixmp.jar and bundled .jar and .dll files to + # the classpath + module_root = Path(__file__).parents[1] + jarfile = module_root / 'ixmp.jar' + module_jars = list(module_root.glob('lib/*')) + classpath = map(str, [module_root, jarfile] + list(module_jars)) + + sep = ';' if os.name == 'nt' else ':' + args.append('-Djava.class.path={}'.format(sep.join(classpath))) + + # Add user args + args.extend(jvmargs if isinstance(jvmargs, list) else [jvmargs]) + + # For JPype 0.7 (raises a warning) and 0.8 (default is False). + # 'True' causes Java string objects to be converted automatically to Python + # str(), as expected by ixmp Python code. + kwargs = dict(convertStrings=True) + + jpype.startJVM(*args, **kwargs) + + # define auxiliary references to Java classes + global java + for class_name in JAVA_CLASSES: + setattr(java, class_name.split('.')[-1], JClass(class_name)) + + +# Conversion methods + +def to_pylist(jlist): + """Transforms a Java.Array or Java.List to a :class:`numpy.array`.""" + # handling string array + try: + return np.array(jlist[:]) + # handling Java LinkedLists + except Exception: + return np.array(jlist.toArray()[:]) + + +def to_jlist(pylist, idx_names=None): + """Convert *pylist* to a jLinkedList.""" + if pylist is None: + return None + + jList = java.LinkedList() + if idx_names is None: + if islistable(pylist): + for key in pylist: + jList.add(str(key)) + else: + jList.add(str(pylist)) + else: + # pylist must be a dict + for idx in idx_names: + jList.add(str(pylist[idx])) + return jList + + +def to_jlist2(arg, convert=None): + """Simple conversion of :class:`list` *arg* to java.LinkedList.""" + jlist = java.LinkedList() + + if convert: + arg = map(convert, arg) + + if isinstance(arg, Collection): + # Sized collection can be used directly + jlist.addAll(arg) + elif isinstance(arg, Iterable): + # Transfer items from an iterable, generator, etc. to the LinkedList + [jlist.add(value) for value in arg] + else: + raise ValueError(arg) + return jlist + + +@lru_cache(1) +def timespans(): + # Mapping for the enums of at.ac.iiasa.ixmp.objects.TimeSeries.TimeSpan + return {t.ordinal(): t.name() for t in java.TimeSpan.values()} diff --git a/ixmp/config.py b/ixmp/config.py index 4ff93560d..481c501d2 100644 --- a/ixmp/config.py +++ b/ixmp/config.py @@ -14,32 +14,10 @@ class Config: """Configuration for ixmp. - When imported, :mod:`ixmp` reads a configuration file `config.json` in the - first of the following directories: - - 1. `IXMP_DATA`, if defined. - 2. `${XDG_DATA_HOME}/ixmp`, if defined. - 3. `$HOME/.local/share/ixmp`. - 4. `$HOME/.local/ixmp` (used by ixmp <= 1.1). - - The file may define either or both of the following configuration keys, in - JSON format: - - - `DB_CONFIG_PATH`: location for database properties files. A - :class:`ixmp.Platform` instantiated with a relative path name for the - `dbprops` argument will locate the file first in the current working - directory, then in `DB_CONFIG_PATH`, then in the four directories above. - - `DEFAULT_DBPROPS_FILE`: path to a default database properties file. A - :class:`ixmp.Platform` instantiated with no arguments will use this file. - - `DEFAULT_LOCAL_DB_PATH`: path to a directory where a local directory - should be created. A :class:`ixmp.Platform` instantiated with - `dbtype='HSQLDB'` will create or reuse a database in this path. - Parameters ---------- read : bool Read `config.json` on startup. - """ # User configuration keys _keys = [ diff --git a/ixmp/core.py b/ixmp/core.py index ab86bd4d6..3e324df10 100644 --- a/ixmp/core.py +++ b/ixmp/core.py @@ -1,148 +1,92 @@ -# coding=utf-8 -import os -import sys -from pathlib import Path -from subprocess import check_call -import warnings - -import jpype -from jpype import ( - JClass, - JPackage as java, -) -import numpy as np +from functools import partial +from itertools import repeat, zip_longest +import logging +from warnings import warn + import pandas as pd -import ixmp as ix -from ixmp import model_settings -from ixmp.config import _config -from ixmp.utils import logger, islistable, check_year, harmonize_path +from .backend import BACKENDS, FIELDS +from .model import get_model +from .utils import ( + as_str_list, + check_year, + filtered, + logger, +) # %% default settings for column headers IAMC_IDX = ['model', 'scenario', 'region', 'variable', 'unit'] -def start_jvm(jvmargs=None): - """Start the Java Virtual Machine via JPype. - - Parameters - ---------- - jvmargs : str or list of str, optional - Additional arguments to pass to :meth:`jpype.startJVM`. - """ - # TODO change the jvmargs default to [] instead of None - if jpype.isJVMStarted(): - return - - jvmargs = jvmargs or [] - - # Arguments - args = [jpype.getDefaultJVMPath()] - - # Add the ixmp root directory, ixmp.jar and bundled .jar and .dll files to - # the classpath - module_root = Path(__file__).parent - jarfile = module_root / 'ixmp.jar' - module_jars = list(module_root.glob('lib/*')) - classpath = map(str, [module_root, jarfile] + list(module_jars)) - - sep = ';' if os.name == 'nt' else ':' - args.append('-Djava.class.path={}'.format(sep.join(classpath))) - - # Add user args - args.extend(jvmargs if isinstance(jvmargs, list) else [jvmargs]) - - # For JPype 0.7 (raises a warning) and 0.8 (default is False). - # 'True' causes Java string objects to be converted automatically to Python - # str(), as expected by ixmp Python code. - kwargs = dict(convertStrings=True) - - jpype.startJVM(*args, **kwargs) - - # define auxiliary references to Java classes - java.ixmp = java('at.ac.iiasa.ixmp') - java.Integer = java('java.lang').Integer - java.Double = java('java.lang').Double - java.LinkedList = java('java.util').LinkedList - java.HashMap = java('java.util').HashMap - java.LinkedHashMap = java('java.util').LinkedHashMap - - class Platform: - """Database-backed instance of the ixmp. + """Instance of the modeling platform. - Each Platform connects three components: + A Platform connects two key components: - 1. A **database** for storing model inputs and outputs. This may either be - a local file (``dbtype='HSQLDB'``) or a database server accessed via a - network connection. In the latter case, connection information is read - from a `properties file`. - 2. A Java Virtual Machine (**JVM**) to run core ixmp logic and access the - database. - 3. One or more **model(s)**, implemented in GAMS or another language or - framework. + 1. A **back end** for storing data such as model inputs and outputs. + 2. One or more **model(s)**; codes in Python or other languages or + frameworks that run, via :meth:`Scenario.solve`, on the data stored in + the Platform. - The constructor parameters control these components. :class:`TimeSeries` - and :class:`Scenario` objects are specific to one Platform; to move data - between platforms, see :meth:`Scenario.clone`. + The Platform parameters control these components. :class:`TimeSeries` and + :class:`Scenario` objects tied to a single Platform; to move data between + platforms, see :meth:`Scenario.clone`. Parameters ---------- - dbprops : path-like, optional - If `dbtype` is :obj:`None`, the name of a database properties file - (default: 'default.properties') in the properties file directory - (default: ???) or the path of a properties file. - - If `dbtype == 'HSQLDB'`, the path of a local database, - (default: "$HOME/.local/ixmp/localdb/default") or name of a - database file in the local database directory (default: - "$HOME/.local/ixmp/localdb/"). - + backend : 'jdbc' + Storage backend type. 'jdbc' corresponds to the built-in + :class:`.JDBCBackend`; see :obj:`.BACKENDS`. + backend_args + Keyword arguments to specific to the `backend`. The “Other Parameters” + shown below are specific to :class:`.JDBCBackend`. + + Other parameters + ---------------- dbtype : 'HSQLDB', optional - Database type to use. If `None`, a remote database is accessed. If + Database type to use. If :obj:`None`, a remote database is accessed. If 'HSQLDB', a local database is created and used at the path given by `dbprops`. - jvmargs : str, optional - Options for launching the Java Virtual Machine, e.g., the maximum heap - space: "-Xmx4G". See the `JVM documentation`_ for a list of options. + dbprops : path-like, optional + If `dbtype` is :obj:`None`, the name of a *database properties file* + (default: ``default.properties``) in the properties file directory + (see :class:`.Config`) or a path to a properties file. - .. _`JVM documentation`: https://docs.oracle.com/javase/7/docs - /technotes/tools/windows/java.html) + If `dbtype` is 'HSQLDB'`, the path of a local database, + (default: ``$HOME/.local/ixmp/localdb/default``) or name of a + database file in the local database directory (default: + ``$HOME/.local/ixmp/localdb/``). + + jvmargs : str, optional + Java Virtual Machine arguments. See :func:`.start_jvm`. """ - def __init__(self, dbprops=None, dbtype=None, jvmargs=None): - start_jvm(jvmargs) - self.dbtype = dbtype - - try: - # if no dbtype is specified, launch Platform with properties file - if dbtype is None: - dbprops = _config.find_dbprops(dbprops) - if dbprops is None: - raise ValueError("Not found database properties file " - "to launch platform") - logger().info("launching ixmp.Platform using config file at " - "'{}'".format(dbprops)) - self._jobj = java.ixmp.Platform("Python", str(dbprops)) - # if dbtype is specified, launch Platform with local database - elif dbtype == 'HSQLDB': - dbprops = dbprops or _config.get('DEFAULT_LOCAL_DB_PATH') - logger().info("launching ixmp.Platform with local {} database " - "at '{}'".format(dbtype, dbprops)) - self._jobj = java.ixmp.Platform("Python", str(dbprops), dbtype) - else: - raise ValueError('Unknown dbtype: {}'.format(dbtype)) - except TypeError: - msg = ("Could not launch the JVM for the ixmp.Platform." - "Make sure that all dependencies of ixmp.jar" - "are included in the 'ixmp/lib' folder.") - logger().info(msg) - raise + # List of method names which are handled directly by the backend + _backend_direct = [ + 'open_db', + 'close_db', + ] + + def __init__(self, *args, backend='jdbc', **backend_args): + if backend != 'jdbc': + raise ValueError(f'unknown ixmp backend {backend!r}') + else: + # Copy positional args for the default JDBC backend + for i, arg in enumerate(['dbprops', 'dbtype', 'jvmargs']): + if len(args) > i: + backend_args[arg] = args[i] + + backend_cls = BACKENDS[backend] + self._backend = backend_cls(**backend_args) + + def __getattr__(self, name): + """Convenience for methods of Backend.""" + return getattr(self._backend, name) def set_log_level(self, level): - """Set global logger level (for both Python and Java) + """Set global logger level. Parameters ---------- @@ -150,41 +94,15 @@ def set_log_level(self, level): set the logger level if specified, see https://docs.python.org/3/library/logging.html#logging-levels """ - py_to_java = { - 'CRITICAL': 'ALL', - 'ERROR': 'ERROR', - 'WARNING': 'WARN', - 'INFO': 'INFO', - 'DEBUG': 'DEBUG', - 'NOTSET': 'OFF', - } - if level not in py_to_java.keys(): + if level not in dir(logging): msg = '{} not a valid Python logger level, see ' + \ 'https://docs.python.org/3/library/logging.html#logging-level' raise ValueError(msg.format(level)) logger().setLevel(level) - self._jobj.setLogLevel(py_to_java[level]) - - def open_db(self): - """(Re-)open the database connection. - - The database connection is opened automatically for many operations. - After calling :meth:`close_db`, it must be re-opened. - - """ - self._jobj.openDB() - - def close_db(self): - """Close the database connection. - - A HSQL database can only be used by one :class:`Platform` instance at a - time. Any existing connection must be closed before a new one can be - opened. - """ - self._jobj.closeDB() + self._backend.set_log_level(level) def scenario_list(self, default=True, model=None, scen=None): - """Return information on TimeSeries and Scenarios in the database. + """Return information about TimeSeries and Scenarios on the Platform. Parameters ---------- @@ -217,24 +135,8 @@ def scenario_list(self, default=True, model=None, scen=None): lock time. - ``annotation``: description of the Scenario or changelog. """ - mod_scen_list = self._jobj.getScenarioList(default, model, scen) - - mod_range = range(mod_scen_list.size()) - cols = ['model', 'scenario', 'scheme', 'is_default', 'is_locked', - 'cre_user', 'cre_date', 'upd_user', 'upd_date', - 'lock_user', 'lock_date', 'annotation'] - - data = {} - for i in cols: - data[i] = [str(mod_scen_list.get(j).get(i)) for j in mod_range] - - data['version'] = [int(str(mod_scen_list.get(j).get('version'))) - for j in mod_range] - cols.append("version") - - df = pd.DataFrame.from_dict(data, orient='columns', dtype=None) - df = df[cols] - return df + return pd.DataFrame(self._backend.get_scenarios(default, model, scen), + columns=FIELDS['get_scenarios']) def Scenario(self, model, scen, version=None, scheme=None, annotation=None, cache=False): @@ -248,20 +150,11 @@ def Scenario(self, model, scen, version=None, >>> ixmp.Scenario(mp, …) """ - warnings.warn('The constructor `mp.Scenario()` is deprecated, ' - 'please use `ixmp.Scenario(mp, ...)`') + warn('The constructor `mp.Scenario()` is deprecated, please use ' + '`ixmp.Scenario(mp, ...)`') return Scenario(self, model, scen, version, scheme, annotation, cache) - def units(self): - """Return all units described in the database. - - Returns - ------- - list - """ - return to_pylist(self._jobj.getUnitList()) - def add_unit(self, unit, comment='None'): """Define a unit. @@ -273,11 +166,15 @@ def add_unit(self, unit, comment='None'): Annotation describing the unit or why it was added. The current database user and timestamp are appended automatically. """ - if unit not in self.units(): - self._jobj.addUnitToDB(unit, comment) - else: + if unit in self.units(): msg = 'unit `{}` is already defined in the platform instance' logger().info(msg.format(unit)) + return + + self._backend.set_unit(unit, comment) + + def units(self): + return self._backend.get_units() def regions(self): """Return all regions defined for the IAMC-style timeseries format @@ -287,14 +184,8 @@ def regions(self): ------- :class:`pandas.DataFrame` """ - lst = [] - for r in self._jobj.listNodes('%'): - n, p, h = (r.getName(), r.getParent(), r.getHierarchy()) - lst.extend([(n, None, p, h)]) - lst.extend([(s, n, p, h) for s in (r.getSynonyms() or [])]) - region = pd.DataFrame(lst) - region.columns = ['region', 'mapped_to', 'parent', 'hierarchy'] - return region + return pd.DataFrame(self._backend.get_nodes(), + columns=FIELDS['get_nodes']) def add_region(self, region, hierarchy, parent='World'): """Define a region including a hierarchy level and a 'parent' region. @@ -308,16 +199,17 @@ def add_region(self, region, hierarchy, parent='World'): ---------- region : str Name of the region. - hierarchy : str - Hierarchy level of the region (e.g., country, R11, basin) parent : str, default 'World' Assign a 'parent' region. + hierarchy : str + Hierarchy level of the region (e.g., country, R11, basin) """ - _regions = self.regions() - if region not in list(_regions['region']): - self._jobj.addNode(region, parent, hierarchy) - else: - _logger_region_exists(_regions, region) + for r in self._backend.get_nodes(): + if r[1] == region: + _logger_region_exists(self.regions(), region) + return + + self._backend.set_node(region, parent, hierarchy) def add_region_synonym(self, region, mapped_to): """Define a synonym for a `region`. @@ -332,14 +224,15 @@ def add_region_synonym(self, region, mapped_to): mapped_to : str Name of the region to which the synonym should be mapped. """ - _regions = self.regions() - if region not in list(_regions['region']): - self._jobj.addNodeSynonym(mapped_to, region) - else: - _logger_region_exists(_regions, region) + for r in self._backend.get_nodes(): + if r[1] == region: + _logger_region_exists(self.regions(), region) + return + + self._backend.set_node(region, synonym=mapped_to) def check_access(self, user, models, access='view'): - """Check access to specific model + """Check access to specific models. Parameters ---------- @@ -349,19 +242,17 @@ def check_access(self, user, models, access='view'): Model(s) name access : str, optional Access type - view or edit - """ + Returns + ------- + bool or dict of bool + """ + models_list = as_str_list(models) + result = self._backend.get_auth(user, models_list, access) if isinstance(models, str): - return self._jobj.checkModelAccess(user, access, models) + return result[models] else: - models_list = java.LinkedList() - for model in models: - models_list.add(model) - access_map = self._jobj.checkModelAccess(user, access, models_list) - result = {} - for model in models: - result[model] = access_map.get(model) == 1 - return result + return {model: result.get(model) == 1 for model in models_list} def _logger_region_exists(_regions, r): @@ -377,35 +268,10 @@ def _logger_region_exists(_regions, r): class TimeSeries: - """Generic collection of data in time series format. + """Collection of data in time series format. TimeSeries is the parent/super-class of :class:`Scenario`. - A TimeSeries is uniquely identified on its :class:`Platform` by three - values: - - 1. `model`: the name of a model used to perform calculations between input - and output data. - - - In TimeSeries storing non-model data, arbitrary strings can be used. - - In a :class:`Scenario`, the `model` is a reference to a GAMS program - registered to the :class:`Platform` that can be solved with - :meth:`Scenario.solve`. See - :meth:`ixmp.model_settings.register_model`. - - 2. `scenario`: the name of a specific, coherent description of the real- - world system being modeled. Any `model` may be used to represent mutiple - alternate, or 'counter-factual', `scenarios`. - 3. `version`: an integer identifying a specific iteration of a - (`model`, `scenario`). A new `version` is created by: - - - Instantiating a new TimeSeries with the same `model` and `scenario` as - an existing TimeSeries. - - Calling :meth:`Scenario.clone`. - - Optionally, one `version` may be set as a **default version**. See - :meth:`set_as_default`. - Parameters ---------- mp : :class:`Platform` @@ -422,79 +288,86 @@ class TimeSeries: annotation : str, optional A short annotation/comment used when ``version='new'``. """ + #: Name of the model associated with the TimeSeries + model = None - # Version of the TimeSeries + #: Name of the scenario associated with the TimeSeries + scenario = None + + #: Version of the TimeSeries. Immutable for a specific instance. version = None - _timespans = {} def __init__(self, mp, model, scenario, version=None, annotation=None): if not isinstance(mp, Platform): raise ValueError('mp is not a valid `ixmp.Platform` instance') - if version == 'new': - self._jobj = mp._jobj.newTimeSeries(model, scenario, annotation) - elif isinstance(version, int): - self._jobj = mp._jobj.getTimeSeries(model, scenario, version) - else: - self._jobj = mp._jobj.getTimeSeries(model, scenario) - + # Set attributes self.platform = mp self.model = model self.scenario = scenario - self.version = self._jobj.getVersion() + + if version == 'new': + self._backend('init_ts', annotation) + elif isinstance(version, int) or version is None: + self._backend('get', version) + else: + raise ValueError(f'version={version!r}') + + def _backend(self, method, *args, **kwargs): + """Convenience for calling *method* on the backend.""" + return self.platform._backend(self, method, *args, **kwargs) # functions for platform management def check_out(self, timeseries_only=False): - """check out from the ixmp database instance for making changes""" + """Check out the TimeSeries for modification.""" if not timeseries_only and self.has_solution(): raise ValueError('This Scenario has a solution, ' 'use `Scenario.remove_solution()` or ' '`Scenario.clone(..., keep_solution=False)`' ) - self._jobj.checkOut(timeseries_only) + self._backend('check_out', timeseries_only) def commit(self, comment): """Commit all changed data to the database. - :attr:`version` is not incremented. + If the TimeSeries was newly created (with ``version='new'``), + :attr:`version` is updated with a new version number assigned by the + backend. Otherwise, :meth:`commit` does not change the :attr:`version`. + + Parameters + ---------- + comment : str + Description of the changes being committed. """ - self._jobj.commit(comment) - # if version == 0, this is a new instance - # and a new version number was assigned after the initial commit - if self.version == 0: - self.version = self._jobj.getVersion() + self._backend('commit', comment) def discard_changes(self): """Discard all changes and reload from the database.""" - self._jobj.discardChanges() + self._backend('discard_changes') def set_as_default(self): """Set the current :attr:`version` as the default.""" - self._jobj.setAsDefaultVersion() + self._backend('set_as_default') def is_default(self): """Return :obj:`True` if the :attr:`version` is the default version.""" - return bool(self._jobj.isDefault()) + return self._backend('is_default') def last_update(self): """get the timestamp of the last update/edit of this TimeSeries""" - return self._jobj.getLastUpdateTimestamp().toString() + return self._backend('last_update') def run_id(self): """get the run id of this TimeSeries""" - return self._jobj.getRunId() - - def version(self): - """get the version number of this TimeSeries""" - return self._jobj.getVersion() + return self._backend('run_id') # functions for importing and retrieving timeseries data def preload_timeseries(self): """Preload timeseries data to in-memory cache. Useful for bulk updates. """ - self._jobj.preloadAllTimeseries() + self._backend('preload') def add_timeseries(self, df, meta=False): """Add data to the TimeSeries. @@ -518,54 +391,36 @@ def add_timeseries(self, df, meta=False): specially when :meth:`Scenario.clone` is called for Scenarios created with ``scheme='MESSAGE'``. """ - meta = 1 if meta else 0 + meta = bool(meta) + # Ensure consistent column names df = to_iamc_template(df) - # if in tabular format - if ("value" in df.columns): - df = df.sort_values(by=['region', 'variable', 'unit', 'year'])\ - .reset_index(drop=True) - - region = df.region[0] - variable = df.variable[0] - unit = df.unit[0] - time = None - jData = java.LinkedHashMap() - - for i in df.index: - if not (region == df.region[i] and variable == df.variable[i] - and unit == df.unit[i]): - # if new 'line', pass to Java interface, start a new - # LinkedHashMap - self._jobj.addTimeseries(region, variable, time, jData, - unit, meta) - - region = df.region[i] - variable = df.variable[i] - unit = df.unit[i] - jData = java.LinkedHashMap() - - jData.put(java.Integer(int(df.year[i])), - java.Double(float(df.value[i]))) - # add the final iteration of the loop - self._jobj.addTimeseries(region, variable, time, jData, unit, meta) - - # if in 'IAMC-style' format + if 'value' in df.columns: + # Long format; pivot to wide + df = pd.pivot_table(df, + values='value', + index=['region', 'variable', 'unit'], + columns=['year']) else: - for i in df.index: - jData = java.LinkedHashMap() + # Wide format: set index columns + df.set_index(['region', 'variable', 'unit'], inplace=True) + + # Discard non-numeric columns, e.g. 'model', 'scenario' + num_cols = [pd.api.types.is_numeric_dtype(dt) for dt in df.dtypes] + df = df.iloc[:, num_cols] - for j in ix.utils.numcols(df): - jData.put(java.Integer(int(j)), - java.Double(float(df[j][i]))) + # Columns (year) as integer + df.columns = df.columns.astype(int) - time = None - self._jobj.addTimeseries(df.region[i], df.variable[i], time, - jData, df.unit[i], meta) + # Add one time series per row + for (r, v, u), data in df.iterrows(): + # Values as float; exclude NA + self._backend('set_data', r, v, data.astype(float).dropna(), u, + meta) - def timeseries(self, iamc=False, region=None, variable=None, level=None, - unit=None, year=None, **kwargs): + def timeseries(self, region=None, variable=None, unit=None, year=None, + iamc=False): """Retrieve TimeSeries data. Parameters @@ -587,44 +442,20 @@ def timeseries(self, iamc=False, region=None, variable=None, level=None, :class:`pandas.DataFrame` Specified data. """ - # convert filter lists to Java objects - region = ix.to_jlist(region) - variable = ix.to_jlist(variable) - unit = ix.to_jlist(unit) - year = ix.to_jlist(year) - - # retrieve data, convert to pandas.DataFrame - data = self._jobj.getTimeseries(region, variable, unit, None, year) - dictionary = {} - - # if in tabular format - ts_range = range(data.size()) - - cols = ['region', 'variable', 'unit'] - for i in cols: - dictionary[i] = [str(data.get(j).get(i)) for j in ts_range] - - dictionary['year'] = [data.get(j).get('year').intValue() - for j in ts_range] - cols.append("year") - - dictionary['value'] = [data.get(j).get('value').floatValue() - for j in ts_range] - cols.append("value") - - df = pd.DataFrame - df = df.from_dict(dictionary, orient='columns', dtype=None) - + # Retrieve data, convert to pandas.DataFrame + df = pd.DataFrame(self._backend('get_data', + as_str_list(region) or [], + as_str_list(variable) or [], + as_str_list(unit) or [], + as_str_list(year) or []), + columns=FIELDS['ts_get']) df['model'] = self.model df['scenario'] = self.scenario - df = df[['model', 'scenario'] + cols] - if iamc: - df = df.pivot_table(index=IAMC_IDX, columns='year')['value'] - df.reset_index(inplace=True) - df.columns = [c if isinstance(c, str) else int(c) - for c in df.columns] + # Convert to wide format + df = df.pivot_table(index=IAMC_IDX, columns='year')['value'] \ + .reset_index() df.columns.names = [None] return df @@ -642,15 +473,17 @@ def remove_timeseries(self, df): - `unit` - `year` """ + # Ensure consistent column names df = to_iamc_template(df) + if 'year' not in df.columns: + # Reshape from wide to long format df = pd.melt(df, id_vars=['region', 'variable', 'unit'], var_name='year', value_name='value') - for name, data in df.groupby(['region', 'variable', 'unit']): - years = java.LinkedList() - for y in data['year']: - years.add(java.Integer(y)) - self._jobj.removeTimeseries(name[0], name[1], None, years, name[2]) + + # Remove all years for a given (r, v, u) combination at once + for (r, v, u), data in df.groupby(['region', 'variable', 'unit']): + self._backend('delete', r, v, data['year'].tolist(), u) def add_geodata(self, df): """Add geodata (layers) to the TimeSeries. @@ -668,14 +501,9 @@ def add_geodata(self, df): - `value` - `meta` """ - for i in df.index: - self._jobj.addGeoData(df.region[i], - df.variable[i], - df.time[i], - java.Integer(int(df.year[i])), - df.value[i], - df.unit[i], - int(df.meta[i])) + for _, row in df.astype({'year': int, 'meta': int}).iterrows(): + self._backend('set_geo', row.region, row.variable, row.time, + row.year, row.value, row.unit, row.meta) def remove_geodata(self, df): """Remove geodata from the TimeSeries instance. @@ -691,26 +519,10 @@ def remove_geodata(self, df): - `time` - `year` """ - for values, _df in df.groupby(['region', 'variable', 'time', 'unit']): - years = java.LinkedList() - for year in _df['year']: - years.add(java.Integer(year)) - self._jobj.removeGeoData(values[0], values[1], values[2], years, - values[3]) - - def _get_timespans(self): - self._timespans = {} - timespan_cls = JClass('at.ac.iiasa.ixmp.objects.TimeSeries$TimeSpan') - for timespan in timespan_cls.values(): - self._timespans[timespan.ordinal()] = timespan.name() - - def _get_timespan_name(self, ordinal): - """Access the enums of at.ac.iiasa.ixmp.objects.TimeSeries.TimeSpan""" - if isinstance(ordinal, str): - ordinal = int(ordinal) - if not self._timespans: - self._get_timespans() - return self._timespans[ordinal] + # Remove all years for a given (r, v, t, u) combination at once + for (r, v, t, u), data in df.groupby(['region', 'variable', 'time', + 'unit']): + self._backend('delete_geo', r, v, t, data['year'].tolist(), u) def get_geodata(self): """Fetch geodata and return it as dataframe. @@ -720,159 +532,54 @@ def get_geodata(self): :class:`pandas.DataFrame` Specified data. """ - java_geodata = self._jobj.getGeoData() - geodata_range = range(java_geodata.size()) - # auto_cols can be added without any conversion - auto_cols = {'nodeName': 'region', - 'unitName': 'unit', - 'meta': 'meta', - 'time': 'time', - 'keyString': 'variable'} - # Initialize empty dict to have correct columns also for empty results - cols = list(auto_cols.values()) + ['year', 'value'] - geodata = {col: [] for col in cols} - - for i in geodata_range: - year_values = java_geodata.get(i).get('yearlyData').entrySet() - for year_value in year_values: - geodata['year'].append(year_value.getKey()) - geodata['value'].append(year_value.getValue()) - for (java_key, python_key) in auto_cols.items(): - geodata[python_key].append(str(java_geodata.get(i).get( - java_key))) - - geodata['meta'] = [int(_meta) for _meta in geodata['meta']] - geodata['time'] = [self._get_timespan_name(time) for time in - geodata['time']] - - return pd.DataFrame.from_dict(geodata) + return pd.DataFrame(self._backend('get_geo'), + columns=FIELDS['ts_get_geo']) # %% class Scenario class Scenario(TimeSeries): - """Collection of model-related input and output data. - - A Scenario is a :class:`TimeSeries` associated with a particular model that - can be run on the current :class:`Platform` by calling :meth:`solve`. The - Scenario also stores the output, or 'solution' of a model run; this - includes the 'level' and 'marginal' values of GAMS equations and variables. - - Data in a Scenario are closely related to different types in the GAMS data - model: - - - A **set** is a named collection of labels. See :meth:`init_set`, - :meth:`add_set`, and :meth:`set`. There are two types of sets: - - 1. Sets that are lists of labels. - 2. Sets that are 'indexed' by one or more other set(s). For this type of - set, each member is an ordered tuple of the labels in the index sets. - - - A **scalar** is a named, single, numerical value. See - :meth:`init_scalar`, :meth:`change_scalar`, and :meth:`scalar`. - - - **Parameters**, **variables**, and **equations** are multi-dimensional - arrays of values that are indexed by one or more sets (i.e. with - dimension 1 or greater). The Scenario methods for handling these types - are very similar; they mainly differ in how they are used within GAMS - models registered with ixmp: - - - **Parameters** are generic data that can be defined before a model run. - They may be altered by the model solution. See :meth:`init_par`, - :meth:`remove_par`, :meth:`par_list`, :meth:`add_par`, and :meth:`par`. - - **Variables** are calculated during or after a model run by GAMS code, - so they cannot be modified by a Scenario. See :meth:`init_var`, - :meth:`var_list`, and :meth:`var`. - - **Equations** describe fundamental relationships between other types - (parameters, variables, and scalars) in a model. They are defined in - GAMS code, so cannot be modified by a Scenario. See :meth:`init_equ`, - :meth:`equ_list`, and :meth:`equ`. + """Collection of model-related data. + + See :class:`TimeSeries` for the meaning of parameters `mp`, `model`, + `scenario`, `version`, and `annotation`. Parameters ---------- - mp : :class:`Platform` - ixmp instance in which to store data. - model : str - Model name; must be a registered model. - scenario : str - Scenario name. - version : str or int or at.ac.iiasa.ixmp.objects.Scenario, optional - If omitted, load the default version of the (`model`, `scenario`). - If :class:`int`, load the specified version. - If ``'new'``, initialize a new Scenario. scheme : str, optional Use an explicit scheme for initializing a new scenario. - annotation : str, optional - A short annotation/comment used when ``version='new'``. cache : bool, optional Store data in memory and return cached values instead of repeatedly - querying the database. + querying the backend. """ - # Name of the model associated with the Scenario - model = None - - # Name of the Scenario - scenario = None - - _java_kwargs = { - 'set': {}, - 'par': {'has_value': True}, - 'var': {'has_level': True}, - 'equ': {'has_level': True}, - } + #: Scheme of the Scenario. + scheme = None def __init__(self, mp, model, scenario, version=None, scheme=None, annotation=None, cache=False): if not isinstance(mp, Platform): raise ValueError('mp is not a valid `ixmp.Platform` instance') - if version == 'new': - self._jobj = mp._jobj.newScenario(model, scenario, scheme, - annotation) - elif isinstance(version, int): - self._jobj = mp._jobj.getScenario(model, scenario, version) - # constructor for `message_ix.Scenario.__init__` or `clone()` function - elif isinstance(version, JClass('at.ac.iiasa.ixmp.objects.Scenario')): - self._jobj = version - elif version is None: - self._jobj = mp._jobj.getScenario(model, scenario) - else: - raise ValueError('Invalid `version` arg: `{}`'.format(version)) - + # Set attributes self.platform = mp self.model = model self.scenario = scenario - self.version = self._jobj.getVersion() - self.scheme = scheme or self._jobj.getScheme() + + if version == 'new': + self._backend('init_s', scheme, annotation) + elif isinstance(version, int) or version is None: + self._backend('get', version) + else: + raise ValueError(f'version={version!r}') + if self.scheme == 'MESSAGE' and not hasattr(self, 'is_message_scheme'): - warnings.warn('Using `ixmp.Scenario` for MESSAGE-scheme scenarios ' - 'is deprecated, please use `message_ix.Scenario`') + warn('Using `ixmp.Scenario` for MESSAGE-scheme scenarios is ' + 'deprecated, please use `message_ix.Scenario`') + # Initialize cache self._cache = cache self._pycache = {} - def _item(self, ix_type, name, load=True): - """Return the Java object for item *name* of *ix_type*. - - Parameters - ---------- - load : bool, optional - If *ix_type* is 'par', 'var', or 'equ', the elements of the item - are loaded from the database before :meth:`_item` returns. If - :const:`False`, the elements can be loaded later using - ``item.loadItemElementsfromDB()``. - """ - funcs = { - 'item': self._jobj.getItem, - 'set': self._jobj.getSet, - 'par': self._jobj.getPar, - 'var': self._jobj.getVar, - 'equ': self._jobj.getEqu, - } - # getItem is not overloaded to accept a second bool argument - args = [name] + ([load] if ix_type != 'item' else []) - return funcs[ix_type](*args) - def load_scenario_data(self): """Load all Scenario data into memory. @@ -897,19 +604,18 @@ def load_scenario_data(self): def _element(self, ix_type, name, filters=None, cache=None): """Return a pd.DataFrame of item elements.""" - item = self._item(ix_type, name) cache_key = (ix_type, name) # if dataframe in python cache, retrieve from there if cache_key in self._pycache: return filtered(self._pycache[cache_key], filters) - # if no cache, retrieve from Java with filters + # if no cache, retrieve from Backend with filters if filters is not None and not self._cache: - return _get_ele_list(item, filters, **self._java_kwargs[ix_type]) + return self._backend('item_get_elements', ix_type, name, filters) # otherwise, retrieve from Java and keep in python cache - df = _get_ele_list(item, None, **self._java_kwargs[ix_type]) + df = self._backend('item_get_elements', ix_type, name, None) # save if using memcache if self._cache: @@ -925,7 +631,7 @@ def idx_sets(self, name): name : str name of the item """ - return to_pylist(self._item('item', name).getIdxSets()) + return self._backend('item_index', name, 'sets') def idx_names(self, name): """return the list of index names for an item (set, par, var, equ) @@ -935,7 +641,20 @@ def idx_names(self, name): name : str name of the item """ - return to_pylist(self._item('item', name).getIdxNames()) + return self._backend('item_index', name, 'names') + + def _keys(self, name, key_or_keys): + if isinstance(key_or_keys, (list, pd.Series)): + return as_str_list(key_or_keys) + elif isinstance(key_or_keys, (pd.DataFrame, dict)): + if isinstance(key_or_keys, dict): + key_or_keys = pd.DataFrame.from_dict(key_or_keys, + orient='columns') + idx_names = self.idx_names(name) + return [as_str_list(row, idx_names) + for _, row in key_or_keys.iterrows()] + else: + return [str(key_or_keys)] def cat_list(self, name): raise DeprecationWarning('function was migrated to `message_ix` class') @@ -948,11 +667,11 @@ def cat(self, name, cat): def set_list(self): """List all defined sets.""" - return to_pylist(self._jobj.getSetList()) + return self._backend('list_items', 'set') def has_set(self, name): - """check whether the scenario has a set with that name""" - return self._jobj.hasSet(name) + """Check whether the scenario has a set *name*.""" + return name in self.set_list() def init_set(self, name, idx_sets=None, idx_names=None): """Initialize a new set. @@ -971,7 +690,7 @@ def init_set(self, name, idx_sets=None, idx_names=None): :class:`jpype.JavaException` If the set (or another object with the same *name*) already exists. """ - self._jobj.initializeSet(name, *make_dims(idx_sets, idx_names)) + return self._backend('init_item', 'set', name, idx_sets, idx_names) def set(self, name, filters=None, **kwargs): """Return the (filtered) elements of a set. @@ -1003,54 +722,97 @@ def add_set(self, name, key, comment=None): Element(s) to be added. If *name* exists, the elements are appended to existing elements. comment : str or iterable of str, optional - Comment describing the element(s). Only used if *key* is a string - or list/range. + Comment describing the element(s). If given, there must be the + same number of comments as elements. Raises ------ - :class:`jpype.JavaException` + KeyError If the set *name* does not exist. :meth:`init_set` must be called before :meth:`add_set`. + ValueError + For invalid forms or combinations of *key* and *comment*. """ + # TODO expand docstring (here or in doc/source/api.rst) with examples, + # per test_core.test_add_set. self.clear_cache(name=name, ix_type='set') - jSet = self._item('set', name) + # Get index names for set *name*, may raise KeyError + idx_names = self.idx_names(name) - if sys.version_info[0] > 2 and isinstance(key, range): - key = list(key) + # Check arguments and convert to two lists: keys and comments + if len(idx_names) == 0: + # Basic set. Keys must be strings. + if isinstance(key, (dict, pd.DataFrame)): + raise ValueError('dict, DataFrame keys invalid for ' + f'basic set {name!r}') - if (jSet.getDim() == 0) and isinstance(key, list): - for i in range(len(key)): - if comment and i < len(comment): - jSet.addElement(str(key[i]), str(comment[i])) - else: - jSet.addElement(str(key[i])) - elif isinstance(key, pd.DataFrame) or isinstance(key, dict): - if isinstance(key, dict): - key = pd.DataFrame.from_dict(key, orient='columns', dtype=None) - idx_names = self.idx_names(name) - if "comment" in list(key): - for i in key.index: - jSet.addElement(to_jlist(key.loc[i], idx_names), - str(key['comment'][i])) - else: - for i in key.index: - jSet.addElement(to_jlist(key.loc[i], idx_names)) - elif isinstance(key, list): - if isinstance(key[0], list): - for i in range(len(key)): - if comment and i < len(comment): - jSet.addElement(to_jlist( - key[i]), str(comment[i])) - else: - jSet.addElement(to_jlist(key[i])) - else: - if comment: - jSet.addElement(to_jlist(key), str(comment[i])) - else: - jSet.addElement(to_jlist(key)) + # Ensure keys is a list of str + keys = as_str_list(key) else: - jSet.addElement(str(key), str(comment)) + # Set defined over 1+ other sets + + # Check for ambiguous arguments + if comment and isinstance(key, (dict, pd.DataFrame)) and \ + 'comment' in key: + raise ValueError("ambiguous; both key['comment'] and comment " + "given") + + if isinstance(key, pd.DataFrame): + # DataFrame of key values and perhaps comments + try: + # Pop a 'comment' column off the DataFrame, convert to list + comment = key.pop('comment').to_list() + except KeyError: + pass + + # Convert key to list of list of key values + keys = [] + for row in key.to_dict(orient='records'): + keys.append(as_str_list(row, idx_names=idx_names)) + elif isinstance(key, dict): + # Dict of lists of key values + + # Pop a 'comment' list from the dict + comment = key.pop('comment', None) + + # Convert to list of list of key values + keys = list(map(as_str_list, + zip(*[key[i] for i in idx_names]))) + elif isinstance(key[0], str): + # List of key values; wrap + keys = [as_str_list(key)] + elif isinstance(key[0], list): + # List of lists of key values; convert to list of list of str + keys = list(map(as_str_list, key)) + elif isinstance(key, str) and len(idx_names) == 1: + # Bare key given for a 1D set; wrap for convenience + keys = [[key]] + else: + # Other, invalid value + raise ValueError(key) + + # Process comments to a list of str, or let them all be None + comments = as_str_list(comment) if comment else repeat(None, len(keys)) + + # Combine iterators to tuples. If the lengths are mismatched, the + # sentinel value 'False' is filled in + to_add = list(zip_longest(keys, comments, fillvalue=False)) + + # Check processed arguments + for e, c in to_add: + # Check for sentinel values + if e is False: + raise ValueError(f'Comment {c!r} without matching key') + elif c is False: + raise ValueError(f'Key {e!r} without matching comment') + elif len(idx_names) and len(idx_names) != len(e): + raise ValueError(f'{len(e)}-D key {e!r} invalid for ' + f'{len(idx_names)}-D set {name}{idx_names!r}') + + # Send to backend + elements = ((kc[0], None, None, kc[1]) for kc in to_add) + self._backend('item_set_elements', 'set', name, elements) def remove_set(self, name, key=None): """delete a set from the scenario @@ -1066,17 +828,18 @@ def remove_set(self, name, key=None): self.clear_cache(name=name, ix_type='set') if key is None: - self._jobj.removeSet(name) + self._backend('delete_item', 'set', name) else: - _remove_ele(self._jobj.getSet(name), key) + self._backend('item_delete_elements', 'set', name, + self._keys(name, key)) def par_list(self): """List all defined parameters.""" - return to_pylist(self._jobj.getParList()) + return self._backend('list_items', 'par') def has_par(self, name): """check whether the scenario has a parameter with that name""" - return self._jobj.hasPar(name) + return name in self.par_list() def init_par(self, name, idx_sets, idx_names=None): """Initialize a new parameter. @@ -1090,7 +853,7 @@ def init_par(self, name, idx_sets, idx_names=None): idx_names : list of str, optional Names of the dimensions indexed by `idx_sets`. """ - self._jobj.initializePar(name, *make_dims(idx_sets, idx_names)) + return self._backend('init_item', 'par', name, idx_sets, idx_names) def par(self, name, filters=None, **kwargs): """return a dataframe of (filtered) elements for a specific parameter @@ -1104,80 +867,111 @@ def par(self, name, filters=None, **kwargs): """ return self._element('par', name, filters, **kwargs) - def add_par(self, name, key, val=None, unit=None, comment=None): + def add_par(self, name, key_or_data=None, value=None, unit=None, + comment=None, key=None, val=None): """Set the values of a parameter. Parameters ---------- name : str Name of the parameter. - key : str, list/range of strings/values, dictionary, dataframe - element(s) to be added - val : values, list/range of values, optional - element values (only used if 'key' is a string or list/range) - unit : str, list/range of strings, optional - element units (only used if 'key' is a string or list/range) - comment : str or list/range of strings, optional - comment (optional, only used if 'key' is a string or list/range) + key_or_data : str or iterable of str or range or dict or \ + pandas.DataFrame + Element(s) to be added. + value : numeric or iterable of numeric, optional + Values. + unit : str or iterable of str, optional + Unit symbols. + comment : str or iterable of str, optional + Comment(s) for the added values. """ - self.clear_cache(name=name, ix_type='par') - - jPar = self._item('par', name) - - if sys.version_info[0] > 2 and isinstance(key, range): - key = list(key) + # Number of dimensions in the index of *name* + idx_names = self.idx_names(name) + N_dim = len(idx_names) + + if key: + warn("Scenario.add_par(key=...) deprecated and will be removed in " + "ixmp 2.0; use key_or_data", DeprecationWarning) + key_or_data = key + if val: + warn("Scenario.add_par(val=...) deprecated and will be removed in " + "ixmp 2.0; use value", DeprecationWarning) + value = val + + # Convert valid forms of arguments to pd.DataFrame + if isinstance(key_or_data, dict): + # dict containing data + data = pd.DataFrame.from_dict(key_or_data, orient='columns') + elif isinstance(key_or_data, pd.DataFrame): + data = key_or_data + if 'value' in data.columns and value is not None: + raise ValueError('both key_or_data.value and value supplied') + else: + # One or more keys; convert to a list of strings + if isinstance(key_or_data, range): + key_or_data = list(key_or_data) + keys = self._keys(name, key_or_data) + + # Check the type of value + if isinstance(value, (float, int)): + # Single value + values = [float(value)] + + if N_dim > 1 and len(keys) == N_dim: + # Ambiguous case: ._key() above returns ['dim_0', 'dim_1'], + # when we really want [['dim_0', 'dim_1']] + keys = [keys] + else: + # Multiple values + values = value + + data = pd.DataFrame(zip_longest(keys, values), + columns=['key', 'value']) + if data.isna().any(axis=None): + raise ValueError('Length mismatch between keys and values') + + # Column types + types = { + 'key': str if N_dim == 1 else object, + 'value': float, + 'unit': str, + 'comment': str, + } - if isinstance(key, pd.DataFrame) and "key" in list(key): - if "comment" in list(key): - for i in key.index: - jPar.addElement(str(key['key'][i]), - _jdouble(key['value'][i]), - str(key['unit'][i]), - str(key['comment'][i])) + # Further handle each column + if 'key' not in data.columns: + # Form the 'key' column from other columns + if N_dim > 1: + data['key'] = data.apply(partial(as_str_list, + idx_names=idx_names), + axis=1) else: - for i in key.index: - jPar.addElement(str(key['key'][i]), - _jdouble(key['value'][i]), - str(key['unit'][i])) - - elif isinstance(key, pd.DataFrame) or isinstance(key, dict): - if isinstance(key, dict): - key = pd.DataFrame.from_dict(key, orient='columns', dtype=None) - idx_names = self.idx_names(name) - if "comment" in list(key): - for i in key.index: - jPar.addElement(to_jlist(key.loc[i], idx_names), - _jdouble(key['value'][i]), - str(key['unit'][i]), - str(key['comment'][i])) + data['key'] = data[idx_names[0]] + + if 'unit' not in data.columns: + # Broadcast single unit across all values. pandas raises ValueError + # if *unit* is iterable but the wrong length + data['unit'] = unit or '???' + + if 'comment' not in data.columns: + if comment: + # Broadcast single comment across all values. pandas raises + # ValueError if *comment* is iterable but the wrong length + data['comment'] = comment else: - for i in key.index: - jPar.addElement(to_jlist(key.loc[i], idx_names), - _jdouble(key['value'][i]), - str(key['unit'][i])) - elif isinstance(key, list) and isinstance(key[0], list): - unit = unit or ["???"] * len(key) - for i in range(len(key)): - if comment and i < len(comment): - jPar.addElement(to_jlist(key[i]), _jdouble(val[i]), - str(unit[i]), str(comment[i])) - else: - jPar.addElement(to_jlist(key[i]), _jdouble(val[i]), - str(unit[i])) - elif isinstance(key, list) and isinstance(val, list): - unit = unit or ["???"] * len(key) - for i in range(len(key)): - if comment and i < len(comment): - jPar.addElement(str(key[i]), _jdouble(val[i]), - str(unit[i]), str(comment[i])) - else: - jPar.addElement(str(key[i]), _jdouble(val[i]), - str(unit[i])) - elif isinstance(key, list) and not isinstance(val, list): - jPar.addElement(to_jlist( - key), _jdouble(val), unit, comment) - else: - jPar.addElement(str(key), _jdouble(val), unit, comment) + # Store a 'None' comment + data['comment'] = None + types.pop('comment') + + # Convert types, generate tuples + elements = ((e.key, e.value, e.unit, e.comment) + for e in data.astype(types).itertuples()) + + # Store + self._backend('item_set_elements', 'par', name, elements) + + # Clear cache + self.clear_cache(name=name, ix_type='par') def init_scalar(self, name, val, unit, comment=None): """Initialize a new scalar. @@ -1193,8 +987,8 @@ def init_scalar(self, name, val, unit, comment=None): comment : str, optional Description of the scalar. """ - jPar = self._jobj.initializePar(name, None, None) - jPar.addElement(_jdouble(val), unit, comment) + self.init_par(name, None, None) + self.change_scalar(name, val, unit, comment) def scalar(self, name): """Return the value and unit of a scalar. @@ -1208,7 +1002,7 @@ def scalar(self, name): ------- {'value': value, 'unit': unit} """ - return _get_ele_list(self._jobj.getPar(name), None, has_value=True) + return self._backend('item_get_elements', 'par', name, None) def change_scalar(self, name, val, unit, comment=None): """Set the value and unit of a scalar. @@ -1225,7 +1019,8 @@ def change_scalar(self, name, val, unit, comment=None): Description of the change. """ self.clear_cache(name=name, ix_type='par') - self._item('par', name).addElement(_jdouble(val), unit, comment) + self._backend('item_set_elements', 'par', name, + [(None, float(val), unit, comment)]) def remove_par(self, name, key=None): """Remove parameter values or an entire parameter. @@ -1240,17 +1035,18 @@ def remove_par(self, name, key=None): self.clear_cache(name=name, ix_type='par') if key is None: - self._jobj.removePar(name) + self._backend('delete_item', 'par', name) else: - _remove_ele(self._jobj.getPar(name), key) + self._backend('item_delete_elements', 'par', name, + self._keys(name, key)) def var_list(self): """List all defined variables.""" - return to_pylist(self._jobj.getVarList()) + return self._backend('list_items', 'var') def has_var(self, name): """check whether the scenario has a variable with that name""" - return self._jobj.hasVar(name) + return name in self.var_list() def init_var(self, name, idx_sets=None, idx_names=None): """initialize a new variable in the scenario @@ -1264,7 +1060,7 @@ def init_var(self, name, idx_sets=None, idx_names=None): idx_names : list of str, optional index name list """ - self._jobj.initializeVar(name, *make_dims(idx_sets, idx_names)) + return self._backend('init_item', 'var', name, idx_sets, idx_names) def var(self, name, filters=None, **kwargs): """return a dataframe of (filtered) elements for a specific variable @@ -1280,7 +1076,7 @@ def var(self, name, filters=None, **kwargs): def equ_list(self): """List all defined equations.""" - return to_pylist(self._jobj.getEquList()) + return self._backend('list_items', 'equ') def init_equ(self, name, idx_sets=None, idx_names=None): """Initialize a new equation. @@ -1294,11 +1090,11 @@ def init_equ(self, name, idx_sets=None, idx_names=None): idx_names : list of str, optional index name list """ - self._jobj.initializeEqu(name, *make_dims(idx_sets, idx_names)) + return self._backend('init_item', 'equ', name, idx_sets, idx_names) def has_equ(self, name): """check whether the scenario has an equation with that name""" - return self._jobj.hasEqu(name) + return name in self.equ_list() def equ(self, name, filters=None, **kwargs): """return a dataframe of (filtered) elements for a specific equation @@ -1313,7 +1109,7 @@ def equ(self, name, filters=None, **kwargs): return self._element('equ', name, filters, **kwargs) def clone(self, model=None, scenario=None, annotation=None, - keep_solution=True, first_model_year=None, platform=None, + keep_solution=True, shift_first_model_year=None, platform=None, **kwargs): """Clone the current scenario and return the clone. @@ -1350,81 +1146,45 @@ def clone(self, model=None, scenario=None, annotation=None, Platform to clone to (default: current platform) """ if 'keep_sol' in kwargs: - warnings.warn( - '`keep_sol` is deprecated and will be removed in the next' + - ' release, please use `keep_solution`') + warn('`keep_sol` is deprecated and will be removed in the next' + ' release, please use `keep_solution`') keep_solution = kwargs.pop('keep_sol') if 'scen' in kwargs: - warnings.warn( - '`scen` is deprecated and will be removed in the next' + - ' release, please use `scenario`') + warn('`scen` is deprecated and will be removed in the next' + ' release, please use `scenario`') scenario = kwargs.pop('scen') - if keep_solution and first_model_year is not None: - raise ValueError('Use `keep_solution=False` when cloning with ' - '`first_model_year`!') + if 'first_model_year' in kwargs: + warn('`first_model_year` is deprecated and will be removed in the' + ' next release. Use `shift_first_model_year`.') + shift_first_model_year = kwargs.pop('first_model_year') - if platform is not None and not keep_solution: - raise ValueError('Cloning across platforms is only possible ' - 'with `keep_solution=True`!') + if len(kwargs): + raise ValueError('Invalid arguments {!r}'.format(kwargs)) + + if shift_first_model_year is not None: + if keep_solution: + logger().warn('Overriding keep_solution=True for ' + 'shift_first_model_year') + keep_solution = False platform = platform or self.platform model = model or self.model scenario = scenario or self.scenario - args = [platform._jobj, model, scenario, annotation, keep_solution] - if check_year(first_model_year, 'first_model_year'): - args.append(first_model_year) - - scenario_class = self.__class__ - return scenario_class(platform, model, scenario, cache=self._cache, - version=self._jobj.clone(*args)) - - def to_gdx(self, path, filename, include_var_equ=False): - """export the scenario data to GAMS gdx - - Parameters - ---------- - path : str - path to the folder - filename : str - name of the gdx file - include_var_equ : boolean, optional - indicator whether to include variables/equations in gdx - """ - self._jobj.toGDX(path, filename, include_var_equ) - def read_sol_from_gdx(self, path, filename, comment=None, - var_list=None, equ_list=None, check_solution=True): - """read solution from GAMS gdx and import it to the scenario + args = [platform, model, scenario, annotation, keep_solution] + if check_year(shift_first_model_year, 'first_model_year'): + args.append(shift_first_model_year) - Parameters - ---------- - path : str - path to the folder - filename : str - name of the gdx file - comment : str - comment to be added to the changelog - var_list : list of str - variables (levels and marginals) to be imported from gdx - equ_list : list of str - equations (levels and marginals) to be imported from gdx - check_solution : boolean, optional - raise an error if GAMS did not solve to optimality - (only applicable for a MESSAGE-scheme scenario) - """ - self.clear_cache() # reset Python data cache - self._jobj.readSolutionFromGDX(path, filename, comment, - to_jlist(var_list), to_jlist(equ_list), - check_solution) + return self._backend('clone', *args) def has_solution(self): """Return :obj:`True` if the Scenario has been solved. If ``has_solution() == True``, model solution data exists in the db. """ - return self._jobj.hasSolution() + return self._backend('has_solution') def remove_solution(self, first_model_year=None): """Remove the solution from the scenario @@ -1446,24 +1206,24 @@ def remove_solution(self, first_model_year=None): """ if self.has_solution(): self.clear_cache() # reset Python data cache - if check_year(first_model_year, 'first_model_year'): - self._jobj.removeSolution(first_model_year) - else: - self._jobj.removeSolution() + check_year(first_model_year, 'first_model_year') + self._backend('clear_solution', first_model_year) else: raise ValueError('This Scenario does not have a solution!') - def solve(self, model, case=None, model_file=None, in_file=None, - out_file=None, solve_args=None, comment=None, var_list=None, - equ_list=None, check_solution=True, callback=None, - gams_args=['LogOption=4'], cb_kwargs={}): + def solve(self, model=None, callback=None, cb_kwargs={}, **model_options): """Solve the model and store output. - ixmp 'solves' a model using the following steps: + ixmp 'solves' a model by invoking the run() method of a :class:`.Model` + subclass—for instance, :meth:`.GAMSModel.run`. Depending on the + underlying model code, different steps are taken; see each model class + for details. In general: - 1. Write all Scenario data to a GDX model input file. - 2. Run GAMS for the specified `model` to perform calculations. - 3. Read the model output, or 'solution', into the database. + 1. Data from the Scenario are written to a **model input file**. + 2. Code or an external program is invoked to perform calculations or + optimizations, **solving the model**. + 3. Data representing the model outputs or solution are read from a + **model output file** and stored in the Scenario. If the optional argument `callback` is given, then additional steps are performed: @@ -1471,44 +1231,21 @@ def solve(self, model, case=None, model_file=None, in_file=None, 4. Execute the `callback` with the Scenario as an argument. The Scenario has an `iteration` attribute that stores the number of times the underlying model has been solved (#2). - 5. If the `callback` returns :obj:`False` or similar, go to #1; - otherwise exit. + 5. If the `callback` returns :obj:`False` or similar, iterate by + repeating from step #1. Otherwise, exit. Parameters ---------- model : str model (e.g., MESSAGE) or GAMS file name (excluding '.gms') - case : str - identifier of gdx file names, defaults to 'model_name_scen_name' - model_file : str, optional - path to GAMS file (including '.gms' extension) - in_file : str, optional - path to GAMS gdx input file (including '.gdx' extension) - out_file : str, optional - path to GAMS gdx output file (including '.gdx' extension) - solve_args : str, optional - arguments to be passed to GAMS (input/output file names, etc.) - comment : str, optional - additional comment added to changelog when importing the solution - var_list : list of str, optional - variables to be imported from the solution - equ_list : list of str, optional - equations to be imported from the solution - check_solution : boolean, optional - flag whether a non-optimal solution raises an exception - (only applies to MESSAGE runs) callback : callable, optional Method to execute arbitrary non-model code. Must accept a single - argument, the Scenario. Must return a non-:obj:`False` value to + argument: the Scenario. Must return a non-:obj:`False` value to indicate convergence. - gams_args : list of str, optional - Additional arguments for the CLI call to GAMS. See, e.g., - https://www.gams.com/latest/docs/UG_GamsCall.html#UG_GamsCall_ListOfCommandLineParameters - - - `LogOption=4` prints output to stdout (not console) and the log - file. cb_kwargs : dict, optional Keyword arguments to pass to `callback`. + model_options : + Keyword arguments specific to the `model`. See :class:`.GAMSModel`. Warns ----- @@ -1526,27 +1263,8 @@ def solve(self, model, case=None, model_file=None, in_file=None, raise ValueError('This Scenario has already been solved, ', 'use `remove_solution()` first!') - model = str(harmonize_path(model)) - config = model_settings.model_config(model) \ - if model_settings.model_registered(model) \ - else model_settings.model_config('default') - - # define case name for gdx export/import, replace spaces by '_' - case = case or '{}_{}'.format(self.model, self.scenario) - case = case.replace(" ", "_") - - model_file = model_file or config.model_file.format(model=model) - - # define paths for writing to gdx, running GAMS, and reading a solution - inp = in_file or config.inp.format(model=model, case=case) - outp = out_file or config.outp.format(model=model, case=case) - args = solve_args or [arg.format(model=model, case=case, inp=inp, - outp=outp) for arg in config.args] - - ipth = os.path.dirname(inp) - ingdx = os.path.basename(inp) - opth = os.path.dirname(outp) - outgdx = os.path.basename(outp) + # Instantiate a model + model = get_model(model, **model_options) # Validate *callback* argument if callback is not None and not callable(callback): @@ -1556,19 +1274,12 @@ def solve(self, model, case=None, model_file=None, in_file=None, def callback(scenario, **kwargs): return True + # Flag to warn if the *callback* appears not to return anything warn_none = True # Iterate until convergence while True: - # Write model data to file - self.to_gdx(ipth, ingdx) - - # Invoke GAMS - run_gams(model_file, args, gams_args=gams_args) - - # Read model solution - self.read_sol_from_gdx(opth, outgdx, comment, - var_list, equ_list, check_solution) + model.run(self) # Store an iteration number to help the callback if not hasattr(self, 'iteration'): @@ -1580,9 +1291,8 @@ def callback(scenario, **kwargs): cb_result = callback(self, **cb_kwargs) if cb_result is None and warn_none: - warnings.warn('solve(callback=...) argument returned None;' - ' will loop indefinitely unless True is' - ' returned.') + warn('solve(callback=...) argument returned None; will loop ' + 'indefinitely unless True is returned.') # Don't repeat the warning warn_none = False @@ -1619,21 +1329,6 @@ def clear_cache(self, name=None, ix_type=None): if key is not None: self._pycache.pop(key) - def years_active(self, node, tec, yr_vtg): - """return a list of years in which a technology of certain vintage - at a specific node can be active - - Parameters - ---------- - node : str - node name - tec : str - name of the technology - yr_vtg : str - vintage year - """ - return to_pylist(self._jobj.getTecActYrs(node, tec, str(yr_vtg))) - def get_meta(self, name=None): """get scenario metadata @@ -1642,14 +1337,8 @@ def get_meta(self, name=None): name : str, optional metadata attribute name """ - def unwrap(value): - """Unwrap metadata numeric value (BigDecimal -> Double)""" - if type(value).__name__ == 'java.math.BigDecimal': - return value.doubleValue() - return value - meta = np.array(self._jobj.getMeta().entrySet().toArray()[:]) - meta = {x.getKey(): unwrap(x.getValue()) for x in meta} - return meta if name is None else meta[name] + all_meta = self._backend('get_meta') + return all_meta[name] if name else all_meta def set_meta(self, name, value): """set scenario metadata @@ -1661,62 +1350,36 @@ def set_meta(self, name, value): value : str or number or bool metadata attribute value """ - self._jobj.setMeta(name, value) + self._backend('set_meta', name, value) # %% auxiliary functions for class Scenario - -def filtered(df, filters): - """Returns a filtered dataframe based on a filters dictionary""" - if filters is None: - return df - - mask = pd.Series(True, index=df.index) - for k, v in filters.items(): - isin = df[k].isin(v) - mask = mask & isin - return df[mask] - - -def _jdouble(val): - """Returns a Java.Double""" - return java.Double(float(val)) - - -def to_pylist(jlist): - """Transforms a Java.Array or Java.List to a python list""" - # handling string array - try: - return np.array(jlist[:]) - # handling Java LinkedLists - except Exception: - return np.array(jlist.toArray()[:]) - - -def to_jlist(pylist, idx_names=None): - """Transforms a python list to a Java.LinkedList""" - if pylist is None: - return None - - jList = java.LinkedList() - if idx_names is None: - if islistable(pylist): - for key in pylist: - jList.add(str(key)) - else: - jList.add(str(pylist)) - else: - for idx in idx_names: - jList.add(str(pylist[idx])) - return jList - - def to_iamc_template(df): - """Formats a pd.DataFrame to an IAMC-compatible table""" - if "time" in df.columns: - msg = "sub-annual time slices not supported by the Python interface!" - raise ValueError(msg) + """Format pd.DataFrame *df* in IAMC style. + + Parameters + ---------- + df : pandas.DataFrame + May have a 'node' column, which will be renamed to 'region'. + + Returns + ------- + pandas.DataFrame + The returned object has: + + - Any (Multi)Index levels reset as columns. + - Lower-case column names 'region', 'variable', and 'unit'. + + Raises + ------ + ValueError + If 'time' is among the column names; or 'region', 'variable', or 'unit' + is not. + """ + if 'time' in df.columns: + raise ValueError('sub-annual time slices not supported by ' + 'ixmp.TimeSeries') # reset the index if meaningful entries are included there if not list(df.index.names) == [None]: @@ -1732,109 +1395,3 @@ def to_iamc_template(df): raise ValueError("missing required columns `{}`!".format(missing)) return df - - -def make_dims(sets, names): - """Wrapper of `to_jlist()` to generate an index-name and index-set list""" - if isinstance(sets, set) or isinstance(names, set): - raise ValueError('index dimension must be string or ordered lists!') - return to_jlist(sets), to_jlist(names if names is not None else sets) - - -def _get_ele_list(item, filters=None, has_value=False, has_level=False): - - # get list of elements, with filter HashMap if provided - if filters is not None: - jFilter = java.HashMap() - for idx_name in filters.keys(): - jFilter.put(idx_name, to_jlist(filters[idx_name])) - jList = item.getElements(jFilter) - else: - jList = item.getElements() - - # return a dataframe if this is a mapping or multi-dimensional parameter - dim = item.getDim() - if dim > 0: - idx_names = np.array(item.getIdxNames().toArray()[:]) - idx_sets = np.array(item.getIdxSets().toArray()[:]) - - data = {} - for d in range(dim): - ary = np.array(item.getCol(d, jList)[:]) - if idx_sets[d] == "year": - # numpy tricks to avoid extra copy - # _ary = ary.view('int') - # _ary[:] = ary - ary = ary.astype('int') - data[idx_names[d]] = ary - - if has_value: - data['value'] = np.array(item.getValues(jList)[:]) - data['unit'] = np.array(item.getUnits(jList)[:]) - - if has_level: - data['lvl'] = np.array(item.getLevels(jList)[:]) - data['mrg'] = np.array(item.getMarginals(jList)[:]) - - df = pd.DataFrame.from_dict(data, orient='columns', dtype=None) - return df - - else: - # for index sets - if not (has_value or has_level): - return pd.Series(item.getCol(0, jList)[:]) - - data = {} - - # for parameters as scalars - if has_value: - data['value'] = item.getScalarValue().floatValue() - data['unit'] = str(item.getScalarUnit()) - - # for variables as scalars - elif has_level: - data['lvl'] = item.getScalarLevel().floatValue() - data['mrg'] = item.getScalarMarginal().floatValue() - - return data - - -def _remove_ele(item, key): - """auxiliary """ - if item.getDim() > 0: - if isinstance(key, list) or isinstance(key, pd.Series): - item.removeElement(to_jlist(key)) - elif isinstance(key, pd.DataFrame) or isinstance(key, dict): - if isinstance(key, dict): - key = pd.DataFrame.from_dict(key, orient='columns', dtype=None) - idx_names = to_pylist(item.getIdxNames()) - for i in key.index: - item.removeElement(to_jlist(key.loc[i], idx_names)) - else: - item.removeElement(str(key)) - - else: - if isinstance(key, list) or isinstance(key, pd.Series): - item.removeElement(to_jlist(key)) - else: - item.removeElement(str(key)) - - -def run_gams(model_file, args, gams_args=['LogOption=4']): - """Parameters - ---------- - model : str - the path to the gams file - args : list - arguments related to the GAMS code (input/output gdx paths, etc.) - gams_args : list of str - additional arguments for the CLI call to gams - - `LogOption=4` prints output to stdout (not console) and the log file - """ - cmd = ['gams', model_file] + args + gams_args - cmd = cmd if os.name != 'nt' else ' '.join(cmd) - - file_path = os.path.dirname(model_file).strip('"') - file_path = None if file_path == '' else file_path - - check_call(cmd, shell=os.name == 'nt', cwd=file_path) diff --git a/ixmp/model/__init__.py b/ixmp/model/__init__.py new file mode 100644 index 000000000..b8c4b76bd --- /dev/null +++ b/ixmp/model/__init__.py @@ -0,0 +1,11 @@ +#: Mapping from names to available models. To register additional models, +#: add elements to this variable. +MODELS = {} + + +def get_model(name, **model_options): + """Return a model for *name* (or the default) with *model_options*.""" + try: + return MODELS[name](**model_options) + except KeyError: + return MODELS['default'](name=name, **model_options) diff --git a/ixmp/model/base.py b/ixmp/model/base.py new file mode 100644 index 000000000..0ce98c817 --- /dev/null +++ b/ixmp/model/base.py @@ -0,0 +1,30 @@ +from abc import ABC, abstractmethod + + +class Model(ABC): # pragma: no cover + #: Name of the model. + name = 'base' + + @abstractmethod + def __init__(self, name, **kwargs): + """Constructor. + + Parameters + ---------- + kwargs : + Model options, passed directly from :meth:`.Scenario.solve`. + + Model subclasses MUST document acceptable option values. + """ + pass + + @abstractmethod + def run(self, scenario): + """Execute the model. + + Parameters + ---------- + scenario : .Scenario + Scenario object to solve by running the Model. + """ + pass diff --git a/ixmp/model/gams.py b/ixmp/model/gams.py new file mode 100644 index 000000000..8ade3abc5 --- /dev/null +++ b/ixmp/model/gams.py @@ -0,0 +1,144 @@ +import os +from pathlib import Path +from subprocess import check_call + + +from ixmp.backend.jdbc import JDBCBackend +from ixmp.model.base import Model +from ixmp.utils import as_str_list + + +class GAMSModel(Model): + """General class for ixmp models using `GAMS `_. + + GAMSModel solves a Scenario using the following steps: + + 1. All Scenario data is written to a model input file in GDX format. + 2. A GAMS program is run to perform calculations, producing output in a + GDX file. + 3. Output, or solution, data is read from the GDX file and stored in the + Scenario. + + When created and :meth:`run`, GAMSModel constructs file paths and other + necessary values using format strings. The :attr:`defaults` may be + overridden by the keyword arguments to the constructor: + + Other parameters + ---------------- + name : str, optional + Override the :attr:`name` attribute to provide the `model_name` for + format strings. + model_file : str, optional + Path to GAMS file, including '.gms' extension. + Default: ``'{model_name}.gms'`` (in the current directory). + case : str, optional + Run or case identifier to use in GDX file names. Default: + ``'{scenario.model}_{scenario.name}'``, where `scenario` is the + :class:`.Scenario` object passed to :meth:`run`. + Formatted using `model_name` and `scenario`. + in_file : str, optional + Path to write GDX input file. Default: ``'{model_name}_in.gdx'``. + Formatted using `model_name`, `scenario`, and `case`. + out_file : str, optional + Path to read GDX output file. Default: ``'{model_name}_out.gdx'``. + Formatted using `model_name`, `scenario`, and `case`. + solve_args : list of str, optional + Arguments to be passed to GAMS, e.g. to identify the model input and + output files. Each formatted using `model_file`, `scenario`, `case`, + `in_file`, and `out_file`. Default: + + - ``'--in="{in_file}"'`` + - ``'--out="{out_file}"'`` + gams_args : list of str, optional + Additional arguments passed directly to GAMS without formatting, e.g. + to control solver options or behaviour. See the `GAMS + Documentation `_. + For example: + + - ``'LogOption=4'`` prints output to stdout (not console) and the log + file. + check_solution : bool, optional + If :obj:`True`, raise an exception if the GAMS solver did not reach + optimality. (Only for MESSAGE-scheme Scenarios.) + comment : str, optional + Comment added to Scenario when importing the solution. If omitted, no + comment is added. + equ_list : list of str, optional + Equations to be imported from the `out_file`. Default: all. + var_list : list of str, optional + Variables to be imported from the `out_file`. Default: all. + """ # noqa: E501 + + #: Model name. + name = 'default' + + #: Default model options. + defaults = { + 'model_file': '{model_name}.gms', + 'case': "{scenario.model}_{scenario.scenario}", + 'in_file': '{model_name}_in.gdx', + 'out_file': '{model_name}_out.gdx', + 'solve_args': ['--in="{in_file}"', '--out="{out_file}"'], + # Not formatted + 'gams_args': ['LogOption=4'], + 'check_solution': True, + 'comment': None, + 'equ_list': None, + 'var_list': None, + } + + def __init__(self, name=None, **model_options): + self.model_name = name or self.name + for arg_name, default in self.defaults.items(): + setattr(self, arg_name, model_options.get(arg_name, default)) + + def run(self, scenario): + """Execute the model.""" + if not isinstance(scenario.platform._backend, JDBCBackend): + raise ValueError('GAMSModel can only solve Scenarios with ' + 'JDBCBackend') + + self.scenario = scenario + + def format(key): + value = getattr(self, key) + try: + return value.format(**self.__dict__) + except AttributeError: + # Something like a Path; don't format it + return value + + # Process args in order + command = ['gams'] + + model_file = Path(format('model_file')).resolve() + command.append('"{}"'.format(model_file)) + + self.case = format('case').replace(' ', '_') + self.in_file = Path(format('in_file')).resolve() + self.out_file = Path(format('out_file')).resolve() + + for arg in self.solve_args: + command.append(arg.format(**self.__dict__)) + + command.extend(self.gams_args) + + if os.name == 'nt': + command = ' '.join(command) + + # Write model data to file + scenario._backend('write_gdx', self.in_file) + + # Invoke GAMS + check_call(command, shell=os.name == 'nt', cwd=model_file.parent) + + # Reset Python data cache + scenario.clear_cache() + + # Read model solution + scenario._backend('read_gdx', self.out_file, + self.check_solution, + self.comment, + as_str_list(self.equ_list), + as_str_list(self.var_list), + ) diff --git a/ixmp/model_settings.py b/ixmp/model_settings.py deleted file mode 100644 index 8ad7dd1b3..000000000 --- a/ixmp/model_settings.py +++ /dev/null @@ -1,48 +0,0 @@ -# -*- coding: utf-8 -*- - -from collections import namedtuple - -ModelConfig = namedtuple('ModelConfig', ['model_file', 'inp', 'outp', 'args']) - -_MODEL_REGISTRY = {} - - -def register_model(name, config): - """Register a new model with ixmp. - - Parameters - ---------- - name : str - Model name. - config : dict - Configuration settings for the model, with the following keys. Each key - is a format string that may use certain named values: - - - `model_file` (:class:`str`): GAMS source file (``.gms``) containing - the model. E.g. "{model}_run.gms". Available values: model, case. - - `inp` (:class:`str`): input path; location where the model expects a - GDX file containing input data. E.g. "{model}_{case}_in.gdx". - Available values: model, case. - - `outp` (:class:`str`): output path; location where the model will - create a GDX file containing output data. E.g. - "{model}_{case}_out.gdx". Available values: model, case. - - `args` (:class:`list` of :class:`str`): additional GAMS command-line - args to be passed when invoking the model. Available values: model, - case, inp, outp. - - The `model` and `case` formatting values are generated from - :attr:`ixmp.Scenario.model` and :attr:`ixmp.Scenario.scenario`, - respectively, with spaces (“ ”) converted to underscores ("_"). - """ - global _MODEL_REGISTRY - _MODEL_REGISTRY[name] = config - - -def model_registered(name): - global _MODEL_REGISTRY - return name in _MODEL_REGISTRY - - -def model_config(name): - global _MODEL_REGISTRY - return _MODEL_REGISTRY[name] diff --git a/ixmp/reporting/__init__.py b/ixmp/reporting/__init__.py index 2d4ed5552..ecf189740 100644 --- a/ixmp/reporting/__init__.py +++ b/ixmp/reporting/__init__.py @@ -140,7 +140,7 @@ def from_scenario(cls, scenario, **kwargs): except AttributeError: # pd.DataFrame for a multidimensional set; store as-is pass - rep.add(name, elements) + rep.add(RENAME_DIMS.get(name, name), elements) # Add the scenario itself rep.add('scenario', scenario) @@ -611,7 +611,7 @@ def _config_args(path, keys, sections={}): extra_sections = set(result.keys()) - sections - {'config_dir'} if len(extra_sections): warn(('Unrecognized sections {!r} in reporting configuration will ' - 'have no effect').format(extra_sections)) + 'have no effect').format(sorted(extra_sections))) return result diff --git a/ixmp/reporting/utils.py b/ixmp/reporting/utils.py index d60f95abe..a973a9bee 100644 --- a/ixmp/reporting/utils.py +++ b/ixmp/reporting/utils.py @@ -65,8 +65,14 @@ def collect_units(*args): return [arg.attrs['_unit'] for arg in args] -def _find_dims(data): - """Return the list of dimensions for *data*.""" +def dims_for_qty(data): + """Return the list of dimensions for *data*. + + If *data* is a :class:`pandas.DataFrame`, its columns are processed; + otherwise it must be a list. + + ixmp.reporting.RENAME_DIMS is used to rename dimensions. + """ if isinstance(data, pd.DataFrame): # List of the dimensions dims = data.columns.tolist() @@ -86,12 +92,8 @@ def _find_dims(data): def keys_for_quantity(ix_type, name, scenario): """Iterate over keys for *name* in *scenario*.""" - # Retrieve names of the indices of the low-level/Java object, *without* - # loading the associated data - # NB this is used instead of .getIdxSets, since the same set may index more - # than one dimension of the same variable. - dims = _find_dims(scenario._item(ix_type, name, load=False) - .getIdxNames().toArray()) + # Retrieve names of the indices of the ixmp item, without loading the data + dims = dims_for_qty(scenario.idx_names(name)) # Column for retrieving data column = 'value' if ix_type == 'par' else 'lvl' @@ -195,7 +197,7 @@ def data_for_quantity(ix_type, name, column, scenario, filters=None): data = pd.DataFrame.from_records([data]) # List of the dimensions - dims = _find_dims(data) + dims = dims_for_qty(data) # Remove the unit from the DataFrame try: diff --git a/ixmp/utils.py b/ixmp/utils.py index d5a14f348..dc2c2c32b 100644 --- a/ixmp/utils.py +++ b/ixmp/utils.py @@ -1,10 +1,5 @@ import collections import logging -import os -try: - from pathlib import Path -except ImportError: - from pathlib2 import Path import pandas as pd import six @@ -26,6 +21,27 @@ def logger(): return _LOGGER +def as_str_list(arg, idx_names=None): + """Convert various *arg* to list of str. + + Several types of arguments are handled: + - None: returned as None. + - str: returned as a length-1 list of str. + - list of values: returned as a list with each value converted to str + - dict, with list of idx_names: the idx_names are used to look up values + in the dict, the resulting list has the corresponding values in the same + order. + + """ + if arg is None: + return None + elif idx_names is None: + # arg must be iterable + return list(map(str, arg)) if islistable(arg) else [str(arg)] + else: + return [str(arg[idx]) for idx in idx_names] + + def isstr(x): """Returns True if x is a string""" return isinstance(x, six.string_types) @@ -75,11 +91,24 @@ def pd_write(df, f, *args, **kwargs): def numcols(df): + """Return the indices of the numeric columns of *df*.""" dtypes = df.dtypes return [i for i in dtypes.index if dtypes.loc[i].name.startswith(('float', 'int'))] +def filtered(df, filters): + """Returns a filtered dataframe based on a filters dictionary""" + if filters is None: + return df + + mask = pd.Series(True, index=df.index) + for k, v in filters.items(): + isin = df[k].isin(v) + mask = mask & isin + return df[mask] + + def import_timeseries(mp, data, model, scenario, version=None, firstyear=None, lastyear=None): if not isinstance(mp, ixmp.Platform): @@ -110,15 +139,3 @@ def import_timeseries(mp, data, model, scenario, version=None, if lastyear is not None: annot = '{} until {}'.format(annot, lastyear) scen.commit(annot) - - -def harmonize_path(path_or_str): - """Harmonize mixed '\' and '/' separators in pathlib.Path or str. - - On Windows, R's file.path(...) uses '/', not '\', as a path separator. - Python's str(WindowsPath(...)) uses '\'. Mixing outputs from the two - functions (e.g. through rixmp) produces path strings with both kinds of - separators. - """ - args = ('/', '\\') if os.name == 'nt' else ('\\', '/') - return Path(str(path_or_str).replace(*args)) diff --git a/setup.py b/setup.py index 5941eda91..6f7fdfe22 100644 --- a/setup.py +++ b/setup.py @@ -18,7 +18,7 @@ 'pandas', 'pint', 'PyYAML', - 'xarray<0.12', + 'xarray', 'xlsxwriter', 'xlrd', ] diff --git a/tests/test_access.py b/tests/test_access.py index 54f666f44..f4dd0767b 100644 --- a/tests/test_access.py +++ b/tests/test_access.py @@ -1,5 +1,5 @@ -import sys from subprocess import Popen +import sys from time import sleep from pretenders.client.http import HTTPMock diff --git a/tests/test_cli.py b/tests/test_cli.py index 8f2a21db1..9689364f9 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -22,7 +22,8 @@ def test_jvm_warn(recwarn): """ # Start the JVM for the first time in the test session - ix.start_jvm() + from ixmp.backend.jdbc import start_jvm + start_jvm() if jpype.__version__ > '0.7': # Zero warnings were recorded diff --git a/tests/test_core.py b/tests/test_core.py index 466259295..f8f2d7ccb 100644 --- a/tests/test_core.py +++ b/tests/test_core.py @@ -76,11 +76,20 @@ def test_has_set(test_mp): assert not scen.has_set('k') -def test_init_par_35(test_mp): +def test_range(test_mp): scen = ixmp.Scenario(test_mp, *can_args, version='new') + scen.init_set('ii') + ii = range(1, 20, 2) + + # range instance is automatically converted to list of str in add_set + scen.add_set('ii', ii) + scen.init_par('new_par', idx_sets='ii') + # range instance is a valid key argument to add_par + scen.add_par('new_par', ii, [1.2] * len(ii)) + def test_get_scalar(test_mp): scen = ixmp.Scenario(test_mp, *can_args) @@ -123,15 +132,92 @@ def test_init_set(test_mp): scen.init_set('foo') -def test_add_set(test_mp): - """Test ixmp.Scenario.add_set().""" +def test_set(test_mp): + """Test ixmp.Scenario.add_set(), .set(), and .remove_set().""" scen = ixmp.Scenario(test_mp, *can_args) # Add element to a non-existent set - with pytest.raises(jpype.JException, - match="No Set 'foo' exists in this Scenario!"): + with pytest.raises(KeyError, + match="No Item 'foo' exists in this Scenario!"): scen.add_set('foo', 'bar') + scen.remove_solution() + scen.check_out() + + # Add elements to a 0-D set + scen.add_set('i', 'i1') # Name only + scen.add_set('i', 'i2', 'i2 comment') # Name and comment + scen.add_set('i', ['i3']) # List of names, length 1 + scen.add_set('i', ['i4', 'i5']) # List of names, length >1 + scen.add_set('i', range(0, 3)) # Generator (range object) + # Lists of names and comments, length 1 + scen.add_set('i', ['i6'], ['i6 comment']) + # Lists of names and comments, length >1 + scen.add_set('i', ['i7', 'i8'], ['i7 comment', 'i8 comment']) + + # Incorrect usage + + # Lists of different length + with pytest.raises(ValueError, + match="Comment 'extra' without matching key"): + scen.add_set('i', ['i9'], ['i9 comment', 'extra']) + with pytest.raises(ValueError, + match="Key 'extra' without matching comment"): + scen.add_set('i', ['i9', 'extra'], ['i9 comment']) + + # Add elements to a 1D set + scen.init_set('foo', 'i', 'dim_i') + scen.add_set('foo', ['i1']) # Single key + scen.add_set('foo', ['i2'], 'i2 in foo') # Single key and comment + scen.add_set('foo', 'i3') # Bare name automatically wrapped + # Lists of names and comments, length 1 + scen.add_set('foo', ['i6'], ['i6 comment']) + # Lists of names and comments, length >1 + scen.add_set('foo', [['i7'], ['i8']], ['i7 comment', 'i8 comment']) + # Dict + scen.add_set('foo', dict(dim_i=['i7', 'i8'])) + + # Incorrect usage + # Improperly wrapped keys + with pytest.raises(ValueError, match=r"2-D key \['i4', 'i5'\] invalid for " + r"1-D set foo\['dim_i'\]"): + scen.add_set('foo', ['i4', 'i5']) + with pytest.raises(ValueError): + scen.add_set('foo', range(0, 3)) + # Lists of different length + with pytest.raises(ValueError, + match="Comment 'extra' without matching key"): + scen.add_set('i', ['i9'], ['i9 comment', 'extra']) + with pytest.raises(ValueError, + match="Key 'extra' without matching comment"): + scen.add_set('i', ['i9', 'extra'], ['i9 comment']) + # Missing element in the index set + with pytest.raises(ValueError, match="The index set 'i' does not have an " + "element 'bar'!"): + scen.add_set('foo', 'bar') + + # Retrieve set elements + i = {'seattle', 'san-diego', 'i1', 'i2', 'i3', 'i4', 'i5', '0', '1', '2', + 'i6', 'i7', 'i8'} + assert i == set(scen.set('i')) + + # Remove elements from an 0D set: bare name + scen.remove_set('i', 'i2') + i -= {'i2'} + assert i == set(scen.set('i')) + + # Remove elements from an 0D set: Iterable of names, length >1 + scen.remove_set('i', ['i4', 'i5']) + i -= {'i4', 'i5'} + assert i == set(scen.set('i')) + + # Remove elements from a 1D set: Dict + scen.remove_set('foo', dict(dim_i=['i7', 'i8'])) + # Added elements from above; minus directly removed elements; minus i2 + # because it was removed from the set i that indexes dim_i of foo + foo = {'i1', 'i2', 'i3', 'i6', 'i7', 'i8'} - {'i2'} - {'i7', 'i8'} + assert foo == set(scen.set('foo')['dim_i']) + # make sure that changes to a scenario are copied over during clone def test_add_clone(test_mp): diff --git a/tests/test_feature_timeseries.py b/tests/test_feature_timeseries.py index 64dfdbffd..a2d07aea6 100644 --- a/tests/test_feature_timeseries.py +++ b/tests/test_feature_timeseries.py @@ -24,7 +24,7 @@ def test_get_timeseries(test_mp): def test_get_timeseries_iamc(test_mp): scen = ixmp.TimeSeries(test_mp, *test_args) - obs = scen.timeseries(iamc=True, regions='World', variables='Testing') + obs = scen.timeseries(region='World', variable='Testing', iamc=True) exp = TS_DF.pivot_table(index=['region', 'variable', 'unit'], columns='year')['value'].reset_index() diff --git a/tests/test_integration.py b/tests/test_integration.py index a1788f04f..9dc8cb08c 100644 --- a/tests/test_integration.py +++ b/tests/test_integration.py @@ -10,7 +10,7 @@ TS_DF_CLEARED.loc[0, 2005] = np.nan -def test_run_clone(tmpdir, test_data_path): +def test_run_clone(tmpdir, test_data_path, caplog): # this test is designed to cover the full functionality of the GAMS API # - initialize a new platform instance # - creates a new scenario and exports a gdx file @@ -27,8 +27,10 @@ def test_run_clone(tmpdir, test_data_path): assert np.isclose(scen2.var('z')['lvl'], 153.675) pdt.assert_frame_equal(scen2.timeseries(iamc=True), TS_DF) - # cloning with `keep_solution=True` and `first_model_year` raises an error - pytest.raises(ValueError, scen.clone, first_model_year=2005) + # cloning with `keep_solution=True` and `first_model_year` raises a warning + scen.clone(keep_solution=True, shift_first_model_year=2005) + assert ('Overriding keep_solution=True for shift_first_model_year' + == caplog.records[-1].message) # cloning with `keep_solution=False` drops the solution and only keeps # timeseries set as `meta=True` @@ -100,7 +102,8 @@ def test_multi_db_run(tmpdir, test_data_path): mp2.add_region('wrong_region', 'country') # check that cloning across platforms must copy the full solution - pytest.raises(ValueError, scen1.clone, platform=mp2, keep_solution=False) + pytest.raises(NotImplementedError, scen1.clone, platform=mp2, + keep_solution=False) # clone solved model across platforms (with default settings) scen1.clone(platform=mp2, keep_solution=True) diff --git a/tests/test_reporting.py b/tests/test_reporting.py index 019bd12b5..6d3f0a6d3 100644 --- a/tests/test_reporting.py +++ b/tests/test_reporting.py @@ -11,14 +11,16 @@ import ixmp.reporting from ixmp.reporting import ( + UNITS, + RENAME_DIMS, ComputationError, KeyExistsError, MissingKeyError, Key, Reporter, + configure, computations, ) -from ixmp.reporting import UNITS from ixmp.reporting.utils import Quantity from ixmp.testing import make_dantzig, assert_qty_allclose, assert_qty_equal @@ -41,10 +43,24 @@ def scenario(test_mp): return scen -def test_reporting_configure(): - # TODO test: All supported configuration keys can be handled - # TODO test: Unsupported keys raise warnings or errors - pass +def test_reporting_configure(test_mp, test_data_path): + # TODO test: configuration keys 'units', 'replace_units' + + # Configure globally; reads 'rename_dims' section + configure(rename_dims={'i': 'i_renamed'}) + + # Reporting uses the RENAME_DIMS mapping of 'i' to 'i_renamed' + scen = make_dantzig(test_mp) + rep = Reporter.from_scenario(scen) + assert 'd:i_renamed-j' in rep, rep.graph.keys() + assert ['seattle', 'san-diego'] == rep.get('i_renamed') + + # Original name 'i' are not found in the reporter + assert 'd:i-j' not in rep, rep.graph.keys() + pytest.raises(KeyError, rep.get, 'i') + + # Remove the configuration for renaming 'i', so that other tests work + RENAME_DIMS.pop('i') def test_reporter_add(): @@ -168,8 +184,10 @@ def test_reporter_read_config(test_mp, test_data_path): scen = make_dantzig(test_mp) rep = Reporter.from_scenario(scen) + + # Warning is raised when reading configuration with unrecognized section(s) with pytest.warns(UserWarning, - match=r"Unrecognized sections {'notarealsection'}"): + match=r"Unrecognized sections \['notarealsection'\]"): rep.read_config(test_data_path / 'report-config-0.yaml') # Data from configured file is available diff --git a/tutorial/README.rst b/tutorial/README.rst index dc48e1bf9..31e417757 100644 --- a/tutorial/README.rst +++ b/tutorial/README.rst @@ -28,7 +28,4 @@ The tutorials are provided as Jupyter notebooks for both Python and R, and are i in `Python `__, or in `R `__. - This tutorial creates an alternate or ‘counterfactual’ scenario of the transport problem; solve it; and compare the results to the original or reference scenario. - - -See :ref:`the GAMS API description ` for an explanation of how the standard GAMS transport problem code is easily adapted for use with *ixmp*. + This tutorial creates an alternate or ‘counterfactual’ scenario of the transport problem; solves it; and compares the results to the original or reference scenario.