Feature: piecewise constraints #569

brynpickering · 2024-02-14T17:51:53Z

Fixes #107

Summary of changes in this pull request:

Add piecewise_constraints component to YAML math.
Add simple example of its use in the user-defined math gallery.

TODO

Handle piecewise constraints when using verbose YAML - I suspect it won't be possible to incorporate it properly.
Handle piecewise constraints in math docs
Capture piecewise validation errors and pipe them through our own exceptions classes.
~~- [ ] Test values as lists and not as mutable parameters.~~ EDIT: not adding
Add documentation.

Reviewer checklist:

Test(s) added to cover contribution
Documentation updated
Changelog updated
Coverage maintained or improved

codecov · 2024-02-16T18:35:03Z

Codecov Report

Attention: Patch coverage is 98.85057% with 2 lines in your changes missing coverage. Please review.

Project coverage is 95.97%. Comparing base (61e1afa) to head (ff81669).
Report is 25 commits behind head on main.

Files with missing lines	Patch %	Lines
src/calliope/backend/expression_parser.py	50.00%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #569      +/-   ##
==========================================
+ Coverage   95.92%   95.97%   +0.05%     
==========================================
  Files          26       26              
  Lines        3899     3980      +81     
  Branches      838      767      -71     
==========================================
+ Hits         3740     3820      +80     
- Misses         69       70       +1     
  Partials       90       90

Files with missing lines	Coverage Δ
src/calliope/backend/backend_model.py	`97.95% <100.00%> (+0.29%)`	⬆️
src/calliope/backend/gurobi_backend_model.py	`95.66% <100.00%> (+0.06%)`	⬆️
src/calliope/backend/latex_backend_model.py	`96.83% <100.00%> (+0.56%)`	⬆️
src/calliope/backend/parsing.py	`96.99% <100.00%> (+0.10%)`	⬆️
src/calliope/backend/pyomo_backend_model.py	`98.11% <100.00%> (+0.03%)`	⬆️
src/calliope/backend/where_parser.py	`98.18% <100.00%> (-0.01%)`	⬇️
src/calliope/backend/expression_parser.py	`93.73% <50.00%> (-0.27%)`	⬇️

docs/user_defined_math/components.md

src/calliope/backend/backend_model.py

docs/user_defined_math/examples/sos2_piecewise_linear_costs.yaml

docs/user_defined_math/components.md

pyproject.toml

Co-authored-by: Stefan Pfenninger <[email protected]>

docs/user_defined_math/components.md

sjpfenninger

At a high level it looks good to me. If @irm-codebase has capacity for a more detailed technical review that would be great.

irm-codebase

@brynpickering review done!

Overall, it looks good. But I do have some questions regarding new additions to the backend, and potentially removing some lines that do not seem to do anything.
As always, nice-to-haves are just that. Feel free to ignore them.

irm-codebase · 2024-07-11T13:12:12Z

docs/examples/piecewise_constraints.py

Nice-to-have: at the moment, it's a tad hard to find this if you do not know about this functionality.
This relates to a separate issue (#601), but we should consider improving this in the future, maybe by listing these features in the Home directory.

Yeah, as mentioned in another comment, having a key features page is high up on our to-do list. @sjpfenninger is on the case!

irm-codebase · 2024-07-11T13:22:22Z

docs/examples/piecewise_constraints.py

+new_params = {
+    "parameters": {
+        "capacity_steps": {
+            "data": capacity_steps,


Nice-to-have:
For ease of reading, I would either:

re-define capacity_steps and cost_steps above new_params, so users see them when reading this bit of the example.

just pasting the array (e.g., [0, 2500, 5000, 7500, 10000]) at data.

They are defined in the previous Python cell (with only one markdown cell in between). I'd say that's close enough to be visible. Still, I'll print out new_params after defining it.

irm-codebase · 2024-07-11T13:26:48Z

docs/examples/piecewise_constraints.py

+# In this example, we require a new decision variable for investment costs that can take on the value defined by the curve at a given value of `flow_cap`.
+
+# %%
+m.math["variables"]["piecewise_cost_investment"] = {


Nice-to-have:
Consider showing this added math in YAML format. Particularly because the example in Math components is different, and we may not want to incentivise users messing with model.math directly.

hmm, this perhaps exposes a problem with adding math snippets on-the-fly. I'd need to show it in YAML and then also store that in a file AND reference that in config.init.add_math. Would your math fixes make that easier?

irm-codebase · 2024-07-11T13:58:30Z

docs/migrating.md

@@ -977,5 +977,7 @@ Now, all components of our internal math are defined in a readable YAML syntax t

 You can add your own math to update the pre-defined math and to represent the physical system in ways we do not cover in our base math, or to apply new modelling methods and problem types (e.g., pathway or stochastic optimisation)!

+When adding your own math, you can add [piecewise linear constraints](user_defined_math/components.md#piecewise-constraints), which is a new type of constraint compared to what could be defined in v0.6.


Nice-to-have: not sure if this belongs here...

Yeah, we need a quick intro to everything you can do in Calliope v0.7, which we don't have at the moment. Hence why some new functionality is described in migrating.md, as it's where existing users are most likely to go first. I'll leave it in until we have this intro page that @sjpfenninger might be putting together.

irm-codebase · 2024-07-11T15:58:33Z

src/calliope/backend/backend_model.py

-            component_dict = self.inputs.math[component_type][name]
-        if name not in self.inputs.math[component_type]:
-            self.inputs.math[component_type][name] = component_dict
+        if name not in self.inputs.math.get(component_type, {}):


It might be possible to remove this check...

Looking at the order of calls, it seems like this case can never be hit.
name appears to always come from self.inputs.math[components].items(), in self.add_all_math.

Since this is an internal function, removal might be ok. Let me know what you think.

no, you can simply do model.backend.add_constraint(...) and it will add a new constraint to the backend after you've added everything from the math dict. This then appends it to the math dict.

Hmmm... this suggests that we are not arranging logic in the best way. If that condition can only be reached when calling add_x, then this should be handled at that level.

Maybe setting a new math key should happen in add_x (keeping the override off, to catch duplicates), and this function should just add the component to the backend?

Except then you have to handle in every add_x method separately, including across backends (something this add_component helper func was made to deal with). It could be added as a wrapper on all add_x funcs, I guess?

Hmmm... I do not think that is the case, because component_type and component_dict are passed by the caller, but repeating it everywhere seems off, I agree. He probably have an antipattern.

Let's keep it for now, as this is another issue.

irm-codebase · 2024-07-11T16:07:47Z

src/calliope/backend/backend_model.py

+            else:
+                yaml_snippet_attrs[attr] = val
+
+        if yaml_snippet_attrs:


Is there a particular reason why we check/have this in the Model generator and not in the Latex backend?
Seems like this is only ever used on that one.

Might be a case of a subclass polluting parent functionality?

irm-codebase · 2024-07-11T16:42:07Z

src/calliope/backend/expression_parser.py

-        if evaluated.shape:
-            dims = rf"_\text{{{','.join(str(i).removesuffix('s') for i in evaluated.dims)}}}"
+        if evaluated.attrs["obj_type"] != "string":
+            data_var_string = evaluated.attrs["math_repr"]


Similar to the yaml attribute above: this seems like it's only used by the Latex backend, but it's present in the regular backend. Not necessarily something to fix now, but it hints that child classes may be polluting the parent.

Not something to fix in this PR, but is this desired?

the math representation? I guess it is only of use to the LaTeX backend. Definitely something to clean up at some point

Correct!
I do not think it's super bad, these attributes do not use much memory. But it suggests that our logic is getting murky.

src/calliope/backend/parsing.py

src/calliope/backend/pyomo_backend_model.py

tests/test_backend_general.py

brynpickering added 9 commits February 14, 2024 17:48

First pass on introducing piecewise constraints

e052989

Merge branch 'main' into feature-piecewise-constraints

f8c0870

Force to sos2; add verbose strings; start adding latex math

ab3f55b

Add LaTex math for piecewise constraints

4e5e644

Only allow piecewise values to be parameter references

d7e83ac

Update piecewise schema; update latex math; add tests

d90da69

Allow x and y to be expressions

d176dfe

Add docs

fd091e4

Add checks for breakpoints dimension

ba70e5b

brynpickering marked this pull request as ready for review February 16, 2024 18:18

brynpickering requested a review from sjpfenninger February 16, 2024 18:18

Update SOS2 example

2199240

brynpickering added 6 commits February 19, 2024 09:42

Increase test coverage

ec05159

Update docs and changelog

b912279

Merge branch 'main' into feature-piecewise-constraints

d8946ac

Add piecewise constraint tutorial

881cb89

Merge branch 'main' into feature-piecewise-constraints

22559b0

Update pre-commit and ruff config

b2ba3a6

sjpfenninger requested changes Mar 1, 2024

View reviewed changes

pyproject.toml Outdated Show resolved Hide resolved

sjpfenninger reviewed Mar 1, 2024

View reviewed changes

pyproject.toml Outdated Show resolved Hide resolved

brynpickering and others added 2 commits March 1, 2024 14:34

Apply suggestions from code review

1f9c8b1

Co-authored-by: Stefan Pfenninger <[email protected]>

Update config; fix long lines

53e7d44

sjpfenninger reviewed Mar 1, 2024

View reviewed changes

docs/user_defined_math/components.md Outdated Show resolved Hide resolved

sjpfenninger reviewed Mar 1, 2024

View reviewed changes

docs/user_defined_math/components.md Outdated Show resolved Hide resolved

sjpfenninger added 2 commits March 1, 2024 17:52

Update docs/user_defined_math/components.md

e1b17b6

Update docs/user_defined_math/components.md

52043a9

brynpickering requested a review from sjpfenninger June 11, 2024 07:17

Merge branch 'main' into feature-piecewise-constraints

1c596e1

Merge branch 'main' into feature-piecewise-constraints

5b6202e

brynpickering requested a review from irm-codebase July 1, 2024 09:28

brynpickering added 7 commits July 2, 2024 09:36

Merge branch 'main' into feature-piecewise-constraints

3bb0535

Merge branch 'main' into feature-piecewise-constraints

9405c35

Post-merge fixes

c0f8993

Clean up piecewise constraint function

4188d06

Clean up timeseries dtype setting

4c7e198

Fix test

d276c63

Bring coverage back up

33f1f2d

sjpfenninger reviewed Jul 11, 2024

View reviewed changes

docs/user_defined_math/components.md Show resolved Hide resolved

sjpfenninger previously approved these changes Jul 11, 2024

View reviewed changes

irm-codebase reviewed Jul 11, 2024

View reviewed changes

Fixes following review

ff81669

brynpickering dismissed sjpfenninger’s stale review via ff81669 July 19, 2024 08:01

brynpickering merged commit 4fc6b84 into main Jul 19, 2024
13 checks passed

brynpickering deleted the feature-piecewise-constraints branch July 19, 2024 16:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature: piecewise constraints #569

Feature: piecewise constraints #569

brynpickering commented Feb 14, 2024 •

edited by sjpfenninger

Loading

codecov bot commented Feb 16, 2024 •

edited

Loading

sjpfenninger left a comment

irm-codebase left a comment

irm-codebase Jul 11, 2024

brynpickering Jul 19, 2024

irm-codebase Jul 11, 2024

brynpickering Jul 19, 2024

irm-codebase Jul 11, 2024

brynpickering Jul 19, 2024

irm-codebase Jul 11, 2024

brynpickering Jul 19, 2024

irm-codebase Jul 11, 2024

brynpickering Jul 19, 2024

irm-codebase Jul 19, 2024

brynpickering Jul 19, 2024

irm-codebase Jul 19, 2024

irm-codebase Jul 11, 2024

irm-codebase Jul 11, 2024

brynpickering Jul 19, 2024

irm-codebase Jul 19, 2024

		@@ -977,5 +977,7 @@ Now, all components of our internal math are defined in a readable YAML syntax t

		You can add your own math to update the pre-defined math and to represent the physical system in ways we do not cover in our base math, or to apply new modelling methods and problem types (e.g., pathway or stochastic optimisation)!

		When adding your own math, you can add [piecewise linear constraints](user_defined_math/components.md#piecewise-constraints), which is a new type of constraint compared to what could be defined in v0.6.

Feature: piecewise constraints #569

Feature: piecewise constraints #569

Conversation

brynpickering commented Feb 14, 2024 • edited by sjpfenninger Loading

TODO

codecov bot commented Feb 16, 2024 • edited Loading

Codecov Report

sjpfenninger left a comment

Choose a reason for hiding this comment

irm-codebase left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brynpickering commented Feb 14, 2024 •

edited by sjpfenninger

Loading

codecov bot commented Feb 16, 2024 •

edited

Loading