Rework math handling

CHANGELOG.md

@@ -4,6 +4,9 @@
 
 |new| Math has been removed from `model.math`, and can now be accessed via `model.math.data`.
 
+|new| Piecewise constraints added to the YAML math with its own unique syntax (#107).
+These constraints will be added to the optimisation problem using Special Ordered Sets of Type 2 (SOS2) variables.
+
 |new| Direct interface to the Gurobi Python API using `!#yaml config.build.backend: gurobi` or `!#python model.build(backend="gurobi")`.
 Tests show that using the gurobi solver via the Python API reduces peak memory consumption and runtime by at least 30% for the combined model build and solve steps.
 This requires the `gurobipy` package which can be installed with `mamba`: `mamba install gurobi::gurobi`.
@@ -36,7 +39,9 @@ Parameter titles from the model definition schema will also propagate to the mod
 
 ### Internal changes
 
-|new| `ModelMath` is a new helper class to handle math additions, including separate methods for pre-defined math, user-defined math and validation checks.
+|changed| `model._model_def_dict` has been removed.
+
+|new| `CalliopeMath` is a new helper class to handle math additions, including separate methods for pre-defined math, user-defined math and validation checks.
 
 |changed| `MathDocumentation` has been extracted from `Model`/`LatexBackend`, and now is a postprocessing module which can take models as input.
 

docs/examples/calliope_model_object.py

@@ -36,33 +36,10 @@
 # Get information on the model
 print(m.info())
 
-# %% [markdown]
-# ## Model definition dictionary
-#
-# `m._model_def_dict` is a python dictionary that holds all the data from the model definition YAML files, restructured into one dictionary.
-#
-# The underscore before the method indicates that it defaults to being hidden (i.e. you wouldn't see it by trying a tab auto-complete and it isn't documented)
-
-# %%
-m._model_def_dict.keys()
-
-# %% [markdown]
-# `techs` hold only the information about a technology that is specific to that node
-
-# %%
-m._model_def_dict["techs"]["pv"]
-
-# %% [markdown]
-# `nodes` hold only the information about a technology that is specific to that node
-
-# %%
-m._model_def_dict["nodes"]["X2"]["techs"]["pv"]
-
 # %% [markdown]
 # ## Model data
 #
-# `m._model_data` is an xarray Dataset.
-# Like `_model_def_dict` it is a hidden prperty of the Model as you are expected to access the data via the public property `inputs`
+# `m._model_data` is an xarray Dataset, a hidden property of the Model as you are expected to access the data via the public property `inputs`
 
 # %%
 m.inputs

docs/examples/piecewise_constraints.py

@@ -67,22 +67,20 @@
 #
 
 # %%
-new_params = {
-    "parameters": {
-        "capacity_steps": {
-            "data": capacity_steps,
-            "index": [0, 1, 2, 3, 4],
-            "dims": "breakpoints",
-        },
-        "cost_steps": {
-            "data": cost_steps,
-            "index": [0, 1, 2, 3, 4],
-            "dims": "breakpoints",
-        },
-    }
-}
+new_params = f"""
+  parameters:
+    capacity_steps:
+      data: {capacity_steps}
+      index: [0, 1, 2, 3, 4]
+      dims: "breakpoints"
+    cost_steps:
+      data: {cost_steps}
+      index: [0, 1, 2, 3, 4]
+      dims: "breakpoints"
+"""
 print(new_params)
-m = calliope.examples.national_scale(override_dict=new_params)
+new_params_as_dict = calliope.AttrDict.from_yaml_string(new_params)
+m = calliope.examples.national_scale(override_dict=new_params_as_dict)
 
 # %%
 m.inputs.capacity_steps
@@ -94,55 +92,48 @@
 # ## Creating our piecewise constraint
 #
 # We create the piecewise constraint by linking decision variables to the piecewise curve we have created.
-# In this example, we require a new decision variable for investment costs that can take on the value defined by the curve at a given value of `flow_cap`.
+# In this example, we need:
+# 1. a new decision variable for investment costs that can take on the value defined by the curve at a given value of `flow_cap`;
+# 1. to link that decision variable to our total cost calculation; and
+# 1. to define the piecewise constraint.
 
 # %%
-m.math["variables"]["piecewise_cost_investment"] = {
-    "description": "Investment cost that increases monotonically",
-    "foreach": ["nodes", "techs", "carriers", "costs"],
-    "where": "[csp] in techs",
-    "bounds": {"min": 0, "max": np.inf},
-    "default": 0,
-}
-
-# %% [markdown]
-# We also need to link that decision variable to our total cost calculation.
-
-# %%
-# Before
-m.math["global_expressions"]["cost_investment_flow_cap"]["equations"]
-
-# %%
-# Updated - we split the equation into two expressions.
-m.math["global_expressions"]["cost_investment_flow_cap"]["equations"] = [
-    {"expression": "$cost_sum * flow_cap", "where": "NOT [csp] in techs"},
-    {"expression": "piecewise_cost_investment", "where": "[csp] in techs"},
-]
-
-# %% [markdown]
-# We then need to define the piecewise constraint:
-
-# %%
-m.math["piecewise_constraints"]["csp_piecewise_costs"] = {
-    "description": "Set investment costs values along a piecewise curve using special ordered sets of type 2 (SOS2).",
-    "foreach": ["nodes", "techs", "carriers", "costs"],
-    "where": "piecewise_cost_investment",
-    "x_expression": "flow_cap",
-    "x_values": "capacity_steps",
-    "y_expression": "piecewise_cost_investment",
-    "y_values": "cost_steps",
-}
-
-# %% [markdown]
-# Then we can build our optimisation problem:
+new_math = """
+  variables:
+    piecewise_cost_investment:
+      description: "Investment cost that increases monotonically"
+      foreach: ["nodes", "techs", "carriers", "costs"]
+      where: "[csp] in techs"
+      bounds:
+        min: 0
+        max: .inf
+      default: 0
+  global_expressions:
+    cost_investment_flow_cap:
+      equations:
+        - expression: "$cost_sum * flow_cap"
+          where: "NOT [csp] in techs"
+        - expression: "piecewise_cost_investment"
+          where: "[csp] in techs"
+  piecewise_constraints:
+    csp_piecewise_costs:
+      description: "Set investment costs values along a piecewise curve using special ordered sets of type 2 (SOS2)."
+      foreach: ["nodes", "techs", "carriers", "costs"]
+      where: "piecewise_cost_investment"
+      x_expression: "flow_cap"
+      x_values: "capacity_steps"
+      y_expression: "piecewise_cost_investment"
+      y_values: "cost_steps"
+"""
 
 # %% [markdown]
 # # Building and checking the optimisation problem
 #
-# With our piecewise constraint defined, we can build our optimisation problem
+# With our piecewise constraint defined, we can build our optimisation problem and inject this new math.
 
 # %%
-m.build()
+new_math_as_dict = calliope.AttrDict.from_yaml_string(new_math)
+m.build(add_math_dict=new_math_as_dict)
 
 # %% [markdown]
 # And we can see that our piecewise constraint exists in the built optimisation problem "backend"
@@ -189,65 +180,6 @@
 )
 fig.show()
 
-# %% [markdown]
-# ## YAML model definition
-# We have updated the model parameters and math interactively in Python in this tutorial, the definition in YAML would look like:
-
-# %% [markdown]
-# ### Math
-#
-# Saved as e.g., `csp_piecewise_math.yaml`.
-#
-# ```yaml
-# variables:
-#   piecewise_cost_investment:
-#     description: Investment cost that increases monotonically
-#     foreach: [nodes, techs, carriers, costs]
-#     where: "[csp] in techs"
-#     bounds:
-#       min: 0
-#       max: .inf
-#     default: 0
-#
-# piecewise_constraints:
-#   csp_piecewise_costs:
-#     description: >
-#       Set investment costs values along a piecewise curve using special ordered sets of type 2 (SOS2).
-#     foreach: [nodes, techs, carriers, costs]
-#     where: "[csp] in techs"
-#     x_expression: flow_cap
-#     x_values: capacity_steps
-#     y_expression: piecewise_cost_investment
-#     y_values: cost_steps
-#
-# global_expressions:
-#   cost_investment_flow_cap.equations:
-#     - expression: "$cost_sum * flow_cap"
-#       where: "NOT [csp] in techs"
-#     - expression: "piecewise_cost_investment"
-#       where: "[csp] in techs"
-# ```
-
-# %% [markdown]
-# ### Scenario definition
-#
-# Loaded into the national-scale example model with: `calliope.examples.national_scale(scenario="piecewise_csp_cost")`
-#
-# ```yaml
-# overrides:
-#   piecewise_csp_cost:
-#     config.init.add_math: [csp_piecewise_math.yaml]
-#     parameters:
-#       capacity_steps:
-#         data: [0, 2500, 5000, 7500, 10000]
-#         index: [0, 1, 2, 3, 4]
-#         dims: "breakpoints"
-#       cost_steps:
-#         data: [0, 3.75e6, 6e6, 7.5e6, 8e6]
-#         index: [0, 1, 2, 3, 4]
-#         dims: "breakpoints"
-# ```
-
 # %% [markdown]
 # ## Troubleshooting
 #

docs/hooks/dummy_model/model.yaml

@@ -2,8 +2,9 @@ overrides:
   storage_inter_cluster:
     config.init:
       name: inter-cluster storage
-      add_math: ["storage_inter_cluster"]
       time_cluster: cluster_days.csv
+    config.build:
+      add_math: ["storage_inter_cluster"]
 
 config.init.name: base
 

docs/hooks/generate_math_docs.py

@@ -145,6 +145,7 @@ def generate_base_math_documentation() -> MathDocumentation:
         MathDocumentation: model math documentation with latex backend.
     """
     model = calliope.Model(model_definition=MODEL_PATH)
+    model.build()
     return MathDocumentation(model)
 
 
@@ -163,10 +164,11 @@ def generate_custom_math_documentation(
         MathDocumentation: model math documentation with latex backend.
     """
     model = calliope.Model(model_definition=MODEL_PATH, scenario=override)
+    model.build()
 
     full_del = []
     expr_del = []
-    for component_group, component_group_dict in model.math.data.items():
+    for component_group, component_group_dict in model.applied_math.data.items():
         for name, component_dict in component_group_dict.items():
             if name in base_documentation.math.data[component_group]:
                 if not component_dict.get("active", True):

docs/pre_defined_math/index.md

@@ -3,25 +3,20 @@
 As of Calliope version 0.7, the math used to build optimisation problems is stored in YAML files.
 The pre-defined math is a re-implementation of the formerly hardcoded math formulation in this YAML format.
 
-The base math is _always_ applied to your model when you `build` the optimisation problem.
+The pre-defined math for your chosen run [mode](../creating/config.md#configbuildmode) is _always_ applied to your model when you `build` the optimisation problem.
 We have also pre-defined some additional math, which you can _optionally_ load into your model.
 For instance, the [inter-cluster storage][inter-cluster-storage-math] math allows you to track storage levels in technologies more accurately when you are using timeseries clustering in your model.
 
 To load optional, pre-defined math on top of the base math, you can reference it by name (_without_ the file extension) in your model configuration:
 
 ```yaml
 config:
-  init:
+  build:
     add_math: [storage_inter_cluster]
 ```
 
-When solving the model in a run mode other than `plan`, some pre-defined additional math will be applied automatically from a file of the same name (e.g., `spores` mode math is stored in [math/spores.yaml](https://github.com/calliope-project/calliope/blob/main/src/calliope/math/spores.yaml)).
-
-!!! note
-
-    Additional math is applied in the order it appears in the `#!yaml config.init.add_math` list.
-    By default, any run mode math will be applied as the final step.
-    If you want to apply your own math *after* the run mode math, you should add the name of the run mode explicitly to the `#!yaml config.init.add_math` list, e.g., `#!yaml config.init.add_math: [operate, user_defined_math.yaml]`.
+If you are running in the `plan` run mode, this will first apply all the [`plan`][base-math] pre-defined math, then the [`storage_inter_cluster`][inter-cluster-storage-math] pre-defined math.
+All pre-defined math YAML files can be found in [`math` directory of the Calliope source code](https://github.com/calliope-project/calliope/blob/main/src/calliope/math/storage_inter_cluster.yaml).
 
 If you want to introduce new constraints, decision variables, or objectives, you can do so as part of the collection of YAML files describing your model.
 See the [user-defined math](../user_defined_math/index.md) section for an in-depth guide to applying your own math.

docs/user_defined_math/customise.md

@@ -4,8 +4,8 @@ Once you understand the [math components](components.md) and the [formulation sy
 
 You can find examples of additional math that we have put together in our [math example gallery](examples/index.md).
 
-Whenever you introduce your own math, it will be applied on top of the [base math][base-math].
-Therefore, you can include base math overrides as well as add new math.
+Whenever you introduce your own math, it will be applied on top of the pre-defined math for your chosen run [mode](../creating/config.md#configbuildmode).
+Therefore, you can override the pre-defined math as well as add new math.
 For example, you may want to introduce a timeseries parameter to the pre-defined `storage_max` constraint to limit maximum storage capacity on a per-timestep basis:
 
 ```yaml
@@ -16,22 +16,39 @@ storage_max:
 
 The other elements of the `storage_max` constraints have not changed (`foreach`, `where`, ...), so we do not need to define them again when adding our own twist on the pre-defined math.
 
-When defining your model, you can reference any number of YAML files containing the math you want to add in `config.init`. The paths are relative to your main model configuration file:
+!!! note
+
+    If you prefer to start from scratch with your math, you can ask Calliope to _not_ load the pre-defined math for your chosen run mode by setting `#!yaml config.build.ignore_mode_math: true`.
+
+When defining your model, you can reference any number of YAML files containing the math you want to add in `config.build`.
+The paths are relative to your main model configuration file:
 
 ```yaml
 config:
-  init:
+  build:
     add_math: [my_new_math_1.yaml, my_new_math_2.yaml]
 ```
 
 You can also define a mixture of your own math and the [pre-defined math](../pre_defined_math/index.md):
 
 ```yaml
 config:
-  init:
+  build:
     add_math: [my_new_math_1.yaml, storage_inter_cluster, my_new_math_2.md]
 ```
 
+Finally, when working in an interactive Python session, you can add math as a dictionary at build time:
+
+```python
+model.build(add_math_dict={...})
+```
+
+This will be applied after the pre-defined mode math and any math from file listed in `config.build.add_math`.
+
+!!! note
+
+    When working in an interactive Python session, you can view the final math dictionary that has been applied to build the optimisation problem by inspecting `model.applied_math` after a successful call to `model.build()`.
+
 ## Adding your parameters to the YAML schema
 
 Our YAML schemas are used to validate user inputs.
@@ -90,15 +107,19 @@ You can write your model's mathematical formulation to view it in a rich-text fo
 To write a LaTeX, reStructuredText, or Markdown file that includes only the math valid for your model:
 
 ```python
+from calliope.postprocess.math_documentation import MathDocumentation
+
 model = calliope.Model("path/to/model.yaml")
-model.math_documentation.build(include="valid")
-model.math_documentation.write(filename="path/to/output/file.[tex|rst|md]")
+model.build()
+
+math_documentation = MathDocumentation(model, include="valid")
+math_documentation.write(filename="path/to/output/file.[tex|rst|md]")
 ```
 
 You can then convert this to a PDF or HTML page using your renderer of choice.
 We recommend you only use HTML as the equations can become too long for a PDF page.
 
 !!! note
 
-    You can add the tabs to flip between rich-text math and the input YAML snippet in your math documentation by using the `mkdocs_tabbed` argument in `model.math_documentation.write`.
+    You can add the tabs to flip between rich-text math and the input YAML snippet in your math documentation by using the `mkdocs_tabbed` argument in `math_documentation.write`.
     We use this functionality in our [pre-defined math](../pre_defined_math/index.md).

docs/user_defined_math/syntax.md

@@ -107,7 +107,7 @@ If you are defining a `constraint`, then you also need to define a comparison op
 You do not need to define the sets of math components in expressions, unless you are actively "slicing" them.
 Behind the scenes, we will make sure that every relevant element of the defined `foreach` sets are matched together when applying the expression (we [merge the underlying xarray DataArrays](https://docs.xarray.dev/en/stable/user-guide/combining.html)).
 Slicing math components involves appending the component with square brackets that contain the slices, e.g. `flow_out[carriers=electricity, nodes=[A, B]]` will slice the `flow_out` decision variable to focus on `electricity` in its `carriers` dimension and only has two nodes (`A` and `B`) on its `nodes` dimension.
-To find out what dimensions you can slice a component on, see your input data (`model.inputs`) for parameters and the definition for decision variables in your loaded math dictionary (`model.math.variables`).
+To find out what dimensions you can slice a component on, see your input data (`model.inputs`) for parameters and the definition for decision variables in your math dictionary.
 
 ## Helper functions