Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Backend, JDBCBackend, Model, GAMSModel classes #182

Merged
merged 73 commits into from
Oct 31, 2019
Merged
Show file tree
Hide file tree
Changes from 68 commits
Commits
Show all changes
73 commits
Select commit Hold shift + click to select a range
0cf9ca5
Initial Backend file/class tree
khaeru Aug 13, 2019
b3694b5
Move __init__ code to JDBCBackend, pass tests
khaeru Aug 13, 2019
900a9cb
Move open_db, close_db, units to Backend
khaeru Aug 13, 2019
09c8edf
Move set_log_level to Backend
khaeru Aug 13, 2019
29ee636
Add TimeSeries, Scenario methods to Backend; warnings
khaeru Aug 13, 2019
8843ef6
Move Java/Python conversions to backend.jdbc
khaeru Aug 13, 2019
514065c
Add 4 methods to Backend
khaeru Aug 13, 2019
f5217fd
Add s_init_item to Backend
khaeru Aug 13, 2019
09a4d2e
Remove xarray version spec of <0.12
khaeru Aug 15, 2019
4e197f8
Add s_add_set_elements to Backend, rewrite add_set, expand tests
khaeru Aug 15, 2019
c43acf0
Add ts_check_out, ts_commit to Backend; clean loading of JClass
khaeru Aug 15, 2019
db2e476
Add s_add_par_elements to Backend, JDBCBackend
khaeru Aug 15, 2019
986bec6
Add Backend documentation
khaeru Aug 15, 2019
25fe9ba
Add get_units to Backend
khaeru Aug 15, 2019
789fd85
Add set_unit to Backend
khaeru Aug 15, 2019
5923d1a
Add set_node to Backend
khaeru Aug 15, 2019
e3635d5
Avoid loading item to get indices in JDBCBackend
khaeru Aug 15, 2019
aeb694f
Use Scenario.idx_names in reporting.utils.keys_for_quantity
khaeru Aug 15, 2019
5edf5b3
Add get_auth to Backend
khaeru Aug 15, 2019
554f764
Add clone, get_scenarios to Backend
khaeru Aug 15, 2019
35b6f21
Implement ts_set / simplify add_timeseries
khaeru Aug 16, 2019
6769989
Implement JDBCBackend.ts_delete
khaeru Aug 16, 2019
767af5e
Fix test_get_timeseries_iamc
khaeru Aug 16, 2019
f0cda2a
Implement JDBCBackend.ts_get
khaeru Aug 16, 2019
26d0c97
Remove direct imports from ixmp.backend.jdbc in ixmp.core
khaeru Aug 16, 2019
9a5869e
Tidy warn() import in ixmp.core
khaeru Aug 16, 2019
c7f63e7
Move ._jobj() shim from TimeSeries to Scenario
khaeru Aug 16, 2019
a73a96d
Appease stickler
khaeru Aug 21, 2019
7d98df3
Appease stickler 2
khaeru Aug 21, 2019
47eda71
Merge branch 'master' into feature/backend
khaeru Aug 21, 2019
55fb0c8
Merge branch 'master' into feature/backend
khaeru Oct 24, 2019
37ab47a
Add ixmp.model.gams.GAMSModel
khaeru Oct 24, 2019
ddc8b21
.gitignore more coverage-related files
khaeru Oct 24, 2019
4e07a0e
Add ts_get_geo, ts_set_geo, ts_delete_geo to Backend
khaeru Oct 24, 2019
e94fa0d
Check backend type in GAMSModel.run()
khaeru Oct 24, 2019
113cf33
Move GDX-related calls to JDBCBackend
khaeru Oct 24, 2019
52c1937
Add s_set_meta to Backend
khaeru Oct 24, 2019
82758ae
Add s_clear_solution to Backend
khaeru Oct 24, 2019
53279ec
Add s_get_meta to Backend
khaeru Oct 24, 2019
61b9776
Add s_delete_item to Backend
khaeru Oct 24, 2019
2d7839d
Add s_item_delete_elements to Backend
khaeru Oct 24, 2019
f83e0f1
Remove shims Scenario._jobj and ._item
khaeru Oct 24, 2019
896b3b8
Lint core.py
khaeru Oct 24, 2019
eadde3d
Remove Scenario.years_active (should be in message_ix)
khaeru Oct 24, 2019
6f08736
Changes caught by message_ix, yet not ixmp, tests 1
khaeru Oct 24, 2019
1230170
Add ms_cat_set_elements to Backend for message_ix
khaeru Oct 24, 2019
df146b2
Add ms_cat_list, _cat_get_elements, _year_first_model, _years_active
khaeru Oct 24, 2019
d03f2a5
Expand documentation and docstrings
khaeru Oct 25, 2019
1ff416b
Document base.Backend methods for Platform
khaeru Oct 25, 2019
2f1c58e
Revert accidental overwrite of model.base
khaeru Oct 25, 2019
d415df4
Handle clone limitations in JDBCBackend
khaeru Oct 25, 2019
3eecdfb
Test reporting.RENAME_DIMS
khaeru Oct 26, 2019
88c614d
Also use RENAME_DIMS in Reporter.from_scenario()
khaeru Oct 26, 2019
b4a7402
Remove pass in base.Backend abstract methods
khaeru Oct 26, 2019
b2fea05
Update ixmp._version.get_config to match setup.cfg
khaeru Oct 26, 2019
99cf7f4
Remove unused utils.harmonize_path
khaeru Oct 26, 2019
962f739
Expand test_set() to cover remove_set(), Scenario._keys()
khaeru Oct 26, 2019
bf1d433
Document more Backend methods
khaeru Oct 26, 2019
6bcaccf
Update RELEASE_NOTES
khaeru Oct 26, 2019
84158cc
Remove _backend method for TimeSeries subclasses
khaeru Oct 27, 2019
d2a987b
Move call conveniences into base.Backend
khaeru Oct 27, 2019
3c37020
Remove Scenario._java_kwargs
khaeru Oct 27, 2019
e7703a1
Streamline Scenario.add_par to use only a single backend call
khaeru Oct 27, 2019
6de5ab9
Re-add range as valid key type for Scenario.add_par
khaeru Oct 27, 2019
65468cd
Document more Backend methods
khaeru Oct 27, 2019
d542adb
Remove ms_years_active from Backend
khaeru Oct 27, 2019
4978a37
Merge Backend.s_add_set_elements and .add_par_values
khaeru Oct 27, 2019
797f3c1
Streamline and document Backend.s_item_get_elements
khaeru Oct 27, 2019
05629ca
Remove 'Java' section in API docs
khaeru Oct 31, 2019
d73f8b1
Correct R description in API docs
khaeru Oct 31, 2019
8225302
Move GAMS adapter docs
khaeru Oct 31, 2019
2421788
Remove prefixes on backend names; streamline import hierarchy
khaeru Oct 31, 2019
c6f60b8
Combine Backend.get_ts and .get_s
khaeru Oct 31, 2019
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ rixmp/source/.Rhistory

# pytest and related
*.pid
.coverage
.coverage*
.pytest_cache/
coverage.xml
htmlcov/
Expand Down
1 change: 1 addition & 0 deletions RELEASE_NOTES.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@

# Next Release

- [#182](https://github.com/iiasa/ixmp/pull/182): Add new Backend, Model APIs and JDBCBackend, GAMSModel classes.
- [#188](https://github.com/iiasa/ixmp/pull/188): Enhance reporting.
- [#177](https://github.com/iiasa/ixmp/pull/177): add ability to pass `gams_args` through `Scenario.solve()`
- [#175](https://github.com/iiasa/ixmp/pull/175): Drop support for Python 2.
Expand Down
144 changes: 144 additions & 0 deletions doc/source/api-backend.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,144 @@
.. currentmodule:: ixmp.backend

Storage back ends (:mod:`ixmp.backend`)
=======================================

By default, the |ixmp| is installed with :class:`ixmp.backend.jdbc.JDBCBackend`, which can store data in many types of relational database management systems (RDBMS) that have Java DataBase Connector (JDBC) interfaces—hence its name.

However, |ixmp| is extensible to support other methods of storing data: in non-JDBC RDBMS, non-relational databases, local files, memory, or other ways.
Developers wishing to add such capabilities may subclass :class:`ixmp.backend.base.Backend` and implement its methods.


Provided backends
-----------------

.. automodule:: ixmp.backend
:members: BACKENDS

.. currentmodule:: ixmp.backend.jdbc

.. autoclass:: ixmp.backend.jdbc.JDBCBackend
:members: s_write_gdx, s_read_gdx

JDBCBackend supports:

- ``dbtype='HSQLDB'``: HyperSQL databases in local files.
- Remote databases. This is accomplished by creating a :class:`ixmp.Platform` with the ``dbprops`` argument pointing a file that specifies JDBC information. For instance::

jdbc.driver = oracle.jdbc.driver.OracleDriver
jdbc.url = jdbc:oracle:thin:@database-server.example.com:1234:SCHEMA
jdbc.user = USER
jdbc.pwd = PASSWORD

It has the following methods that are not part of the overall :class:`Backend` API:

.. autosummary::
:nosignatures:

s_write_gdx
s_read_gdx

.. automethod:: ixmp.backend.jdbc.start_jvm

Backend API
-----------

- :class:`ixmp.Platform` implements a *user-friendly* API for scientific programming.
This means its methods can take many types of arguments, check, and transform them—in a way that provides modeler-users with easy, intuitive workflows.
- In contrast, :class:`Backend` has a *very simple* API that accepts arguments and returns values in basic Python data types and structures.
- As a result:

- :class:`Platform <ixmp.Platform>` code is not affected by where and how data is stored; it merely handles user arguments and then makes, usually, a single :class:`Backend` call.
- :class:`Backend` code does not need to perform argument checking; merely store and retrieve data reliably.

.. currentmodule:: ixmp.backend.base

.. autodata:: ixmp.backend.base.FIELDS

.. autoclass:: ixmp.backend.base.Backend
:members:

In the following, the bold-face words **required**, **optional**, etc. have specific meanings as described in `IETF RFC 2119 <https://tools.ietf.org/html/rfc2119>`_.

Backend is an **abstract** class; this means it **must** be subclassed.
Most of its methods are decorated with :meth:`abc.abstractmethod`; this means they are **required** and **must** be overridden by subclasses.

Others, marked below with "OPTIONAL:", are not so decorated.
For these methods, the behaviour in the base Backend—often, nothing—is an acceptable default behaviour.
Subclasses **may** extend or replace this behaviour as desired, so long as the methods still perform the actions described in the description.

Backends:

- **must** only raise standard Python exceptions.

Methods related to :class:`ixmp.Platform`:

.. autosummary::
:nosignatures:

close_db
get_auth
get_nodes
get_scenarios
get_units
open_db
set_log_level
set_node
set_unit

Methods related to :class:`ixmp.TimeSeries`:

- Each method has an argument `ts`, a reference to the TimeSeries object being manipulated.
- ‘Geodata’ is otherwise identical to regular timeseries data, except value are :class:`str` rather than :class:`float`.

.. autosummary::
:nosignatures:

ts_check_out
ts_commit
ts_delete
ts_delete_geo
ts_discard_changes
ts_get
ts_get_data
ts_get_geo
ts_init
ts_is_default
ts_last_update
ts_preload
ts_run_id
ts_set_data
ts_set_as_default
ts_set_geo

Methods related to :class:`ixmp.Scenario`:

- Each method has an argument `s`, a reference to the Scenario object being manipulated.

.. autosummary::
:nosignatures:

s_clone
s_delete_item
s_get
s_get_meta
s_has_solution
s_init
s_init_item
s_item_delete_elements
s_item_get_elements
s_item_set_elements
s_item_index
s_list_items
s_set_meta

Methods related to :class:`message_ix.Scenario`:

- Each method has an argument `ms`, a reference to the Scenario object being manipulated.

.. autosummary::
:nosignatures:

ms_cat_get_elements
ms_cat_list
ms_cat_set_elements
36 changes: 36 additions & 0 deletions doc/source/api-model.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
.. currentmodule:: ixmp.model

Mathematical models (:mod:`ixmp.model`)
=======================================

By default, the |ixmp| is installed with :class:`ixmp.model.gams.GAMSModel`, which performs calculations by executing code stored in GAMS files.

However, |ixmp| is extensible to support other methods of performing calculations or optimization.
Developers wishing to add such capabilities may subclass :class:`ixmp.model.base.Model` and implement its methods.


Provided models
---------------

.. automodule:: ixmp.model
:members: get_model, MODELS

.. autoclass:: ixmp.model.gams.GAMSModel
:members:


Model API
---------

.. autoclass:: ixmp.model.base.Model
:members: name, __init__, run

In the following, the words REQUIRED, OPTIONAL, etc. have specific meanings as described in `IETF RFC 2119 <https://tools.ietf.org/html/rfc2119>`_.

Model is an **abstract** class; this means it MUST be subclassed.
It has two REQURIED methods that MUST be overridden by subclasses:

.. autosummary::
name
__init__
run
184 changes: 182 additions & 2 deletions doc/source/api-python.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,188 @@
Python (:mod:`ixmp` package)
============================

.. automodule:: ixmp
:members: Platform, TimeSeries, Scenario
The |ixmp| application progamming interface (API) is organized around three classes:

.. autosummary::

Platform
TimeSeries
Scenario

Platform
--------

.. autoclass:: Platform
:members:

Platforms have the following methods:

.. autosummary::
add_region
add_region_synonym
add_unit
check_access
regions
scenario_list
set_log_level


TimeSeries
----------

.. autoclass:: TimeSeries
:members:

A TimeSeries is uniquely identified on its :class:`Platform` by three
values:

1. `model`: the name of a model used to perform calculations between input
and output data.

- In TimeSeries storing non-model data, arbitrary strings can be used.
- In a :class:`Scenario`, the `model` is a reference to a GAMS program
registered to the :class:`Platform` that can be solved with
:meth:`Scenario.solve`. See :attr:`ixmp.model.MODELS`.

2. `scenario`: the name of a specific, coherent description of the real-
world system being modeled. Any `model` may be used to represent mutiple
alternate, or 'counter-factual', `scenarios`.
3. `version`: an integer identifying a specific iteration of a
(`model`, `scenario`). A new `version` is created by:

- Instantiating a new TimeSeries with the same `model` and `scenario` as
an existing TimeSeries.
- Calling :meth:`Scenario.clone`.

Optionally, one `version` may be set as a **default version**. See
:meth:`set_as_default`.

TimeSeries objects have the following methods:

.. autosummary::
add_geodata
add_timeseries
check_out
commit
discard_changes
get_geodata
is_default
last_update
preload_timeseries
remove_geodata
remove_timeseries
run_id
set_as_default
timeseries


Scenario
--------

.. autoclass:: Scenario
:show-inheritance:
:members:

A Scenario is a :class:`TimeSeries` associated with a particular model that
can be run on the current :class:`Platform` by calling :meth:`solve`. The
Scenario also stores the output, or 'solution' of a model run; this
includes the 'level' and 'marginal' values of GAMS equations and variables.

Data in a Scenario are closely related to different types in the GAMS data
model:

- A **set** is a named collection of labels. See :meth:`init_set`,
:meth:`add_set`, and :meth:`set`. There are two types of sets:

1. Sets that are lists of labels.
2. Sets that are 'indexed' by one or more other set(s). For this type of
set, each member is an ordered tuple of the labels in the index sets.

- A **scalar** is a named, single, numerical value. See
:meth:`init_scalar`, :meth:`change_scalar`, and :meth:`scalar`.

- **Parameters**, **variables**, and **equations** are multi-dimensional
arrays of values that are indexed by one or more sets (i.e. with
dimension 1 or greater). The Scenario methods for handling these types
are very similar; they mainly differ in how they are used within GAMS
models registered with ixmp:

- **Parameters** are generic data that can be defined before a model run.
They may be altered by the model solution. See :meth:`init_par`,
:meth:`remove_par`, :meth:`par_list`, :meth:`add_par`, and :meth:`par`.
- **Variables** are calculated during or after a model run by GAMS code,
so they cannot be modified by a Scenario. See :meth:`init_var`,
:meth:`var_list`, and :meth:`var`.
- **Equations** describe fundamental relationships between other types
(parameters, variables, and scalars) in a model. They are defined in
GAMS code, so cannot be modified by a Scenario. See :meth:`init_equ`,
:meth:`equ_list`, and :meth:`equ`.

.. autosummary::
add_par
add_set
change_scalar
clear_cache
clone
equ
equ_list
get_meta
has_equ
has_par
has_set
has_solution
has_var
idx_names
idx_sets
init_equ
init_par
init_scalar
init_set
init_var
load_scenario_data
par
par_list
remove_par
remove_set
remove_solution
scalar
set
set_list
set_meta
solve
var
var_list

Configuration
-------------

.. currentmodule:: ixmp.config

.. autoclass:: Config

When imported, :mod:`ixmp` reads configuration from the first file named
``config.json`` found in one of the following directories:

1. The directory given by the environment variable ``IXMP_DATA``, if
defined,
2. ``${XDG_DATA_HOME}/ixmp``, if the environment variable is defined,
3. ``$HOME/.local/share/ixmp``, or
4. ``$HOME/.local/ixmp`` (deprecated; retained for compatibility with ixmp
<= 1.1).

The file may define either or both of the following configuration keys, in
JSON format:

- `DB_CONFIG_PATH`: location for database properties files. A
:class:`ixmp.Platform` instantiated with a relative path name for the
`dbprops` argument will locate the file first in the current working
directory, then in `DB_CONFIG_PATH`, then in the four directories above.
- `DEFAULT_DBPROPS_FILE`: path to a default database properties file.
A :class:`ixmp.Platform` instantiated with no arguments will use this
file.
- `DEFAULT_LOCAL_DB_PATH`: path to a directory where a local directory
should be created. A :class:`ixmp.Platform` instantiated with
`dbtype='HSQLDB'` will create or reuse a database in this path.


Testing utilities
Expand Down
2 changes: 2 additions & 0 deletions doc/source/api.rst
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@ On separate pages:
:maxdepth: 2

api-python
api-backend
api-model
reporting

On this page:
Expand Down
Loading