Disentangle message_ix and ixmp_source

[khaeru]

One goal of iiasa/ixmp#182 / #249 was to clarify what core functionality of the message_ix modeling framework currently resides in the Java code of ixmp_source (i.e. behind ixmp's JDBCBackend).

As the name suggests, it is not the task of a storage backend to provide core model/framework logic; this logic should instead live in this repo. (In the current situation, new, non-Java backends would each be forced to re-implement that logic, which would be duplicative and increase points of failure.)

This issue is for:

• @danielhuppmann @gidden @zikolach myself and possibly others to identify these pieces, and
• a broader set of people to divide and coordinate (in further issues/PRs) the work of porting them from ixmp_source to message_ix.

Pieces

(@khaeru to flesh these out/keep them updated based on discussion below; others feel free to edit)

1. Scenario initialization. message_ix.Scenario(..., version='new') returns a Scenario pre-populated with certain sets and parameters.
   • Connect MACRO explicitly to MESSAGEix Scenarios #223 sets an example here: the MACRO sets and parameters are visible in the Python code. As a further iteration @gidden @khaeru feel they could be moved to a YAML file.
   • Add Model.initialize ixmp#212 adds this functionality to ixmp.
2. Mapping sets. message_ix.Scenario.cat_list(...), .cat(...) and .add_cat(...) operate by creating and manipulating related sets named type_* and map_*.
   • More intuitive treatment of type_* and cat_* mappings #21 could be addressed at the same time.
3. GDX output. The ixmp_source method Scenario.toGDX() writes Scenario data to a GDX file.
   • Depends on item 2.
4. GDX input. The ixmp_source method Scenario.readSolutionFromGDX() reads the solution (levels of specific variables and marginals of specific equations) from a GDX file. It does not read any structure (i.e., sets) or input data (i.e., parameters).

Notes

• @zikolach pointed out that some of the Java code is used by other builds from ixmp_source, e.g. behind the Scenario Explorer. So, re-implementing the functionality in Python does not always mean it can be removed from ixmp_source. On a case-by-case basis, we will need to decide this; and if the code is in both places, then there must be clear warnings to future developers that the behaviour must be kept in sync, perhaps by marking one or the other as authoritative.
• Items 3 and 4 are needed to make ixmp's GAMSModel non-backend-specific. Currently it only works with JDBCBackend.
• See also Update MESSAGEix scheme defaults in the ixmp core ixmp#70.

[danielhuppmann]

I'm not sure why you think that item 3 depends on item 2?

[khaeru]

I see the relationships as follows:

• In order to write to GDX (item 3), message_ix (as opposed to ixmp_source) needs a full list of which sets and parameters to write (item 1), plus or including any ‘hidden’, ‘implicit’, or ‘auto-generated’ sets and parameters (item 2).
• In order to read from GDX (item 4), message_ix would either:
  • use the same full list (item 1 + item 2) plus another list of solution-related items (equ, var, and possibly set/par), or
  • just the list of solution-related items, or
  • implicitly determine that any items found in the GDX file which are not in the (item 1 + item 2) list are solution-related.

As the latter point shows, this is somewhat implementation-dependent.

[danielhuppmann]

I would not describe type_* and map_* as ‘hidden’, ‘implicit’, or ‘auto-generated’. These are two perfectly normal sets to define the mathematical structure of the equations. The cat_* feature described in term 2 just makes it slightly more user-friendly to work with them.

There are a number of additional mapping sets used as flags for GAMS, generally named is_*. These are only required in the workflow "write to gdx" and are completely unrelated to item 2.

[khaeru]

The cat_* feature described in term 2 just makes it slightly more user-friendly to work with them.

This is good news, then—if the logic is simple and there's no special treatment of any of the categorizations, then it's easy to port.

There are a number of additional mapping sets used as flags for GAMS, generally named is_*.

Okay, thanks for pointing these out. Searching led me to MsgScenario.writeParametersToGDX() where these are defined; this should be the reference for whoever tackles these tasks.

[khaeru]

@francescolovat's question on #275 revealed some additional issues to address/code to be consulted.