Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(scheduling): Price devices distinctively in scheduling #654

Draft
wants to merge 23 commits into
base: main
Choose a base branch
from
Draft

feat(scheduling): Price devices distinctively in scheduling #654

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
ab72c4f
Adapt FlexContextSchema for pricing devices distinctively
anirudh-ramesh Apr 26, 2023
a9550f0
Add price sensors for individual devices to the CLI
anirudh-ramesh Apr 26, 2023
bea996e
Merge branch 'main' into feature/scheduling
anirudh-ramesh Apr 29, 2023
a80a475
Merge branch 'main' into feature/scheduling
anirudh-ramesh May 9, 2023
014edfa
Add help text for CLI option corresponding to price sensors per device
anirudh-ramesh May 9, 2023
515c31b
Pass prices from the database to the scheduler
anirudh-ramesh May 9, 2023
c34952d
Edit model in order to ingest multiple price sensors
anirudh-ramesh May 9, 2023
be79cf7
Add new index dimensions to linear_optimization.py
anirudh-ramesh May 12, 2023
901b7f3
Update linear_optimization.py
rajath-09 May 12, 2023
fc12afb
Update linear_optimization.py
rajath-09 May 19, 2023
346030c
Update linear_optimization.py
rajath-09 May 19, 2023
be387b5
Update data_add.py
rajath-09 Jun 2, 2023
f830cfc
Update data_add.py
rajath-09 Jun 2, 2023
e2fcb5a
Update data_add.py
rajath-09 Jun 2, 2023
7307256
Merge branch 'main' of https://github.com/irasus-technologies/flexmea…
Jun 2, 2023
0a9c6f0
"Adding tests for dictionary price sensor parameters"
Jun 5, 2023
9fe1831
"Correction in Schedule End date"
Jun 6, 2023
a931838
"Changes Suggested in TestFunction"
Jun 7, 2023
f0919ae
"Flake8 and Black changes"
Jun 8, 2023
41d3a2f
"Flake changes in Storage.py and linear_optimization.py"
Jun 8, 2023
11d2b03
Merge branch 'FlexMeasures:main' into feature/scheduling
rajath-09 Jun 9, 2023
95da3f4
"Changes made according to PR reviews"
Jun 15, 2023
ea323eb
"Revert back to 'd' index dimension"
Jun 26, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
91 changes: 83 additions & 8 deletions flexmeasures/cli/data_add.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -947,6 +947,20 @@ def create_schedule(ctx):
required=False,
help="Optimize production against this sensor. Defaults to the consumption price sensor. The sensor typically records an electricity price (e.g. in EUR/kWh), but this field can also be used to optimize against some emission intensity factor (e.g. in kg CO₂ eq./kWh). Follow up with the sensor's ID.",
)
@click.option(
"--consumption-price-sensor-per-device",
"consumption_price_sensor_per_device",
type=str,
required=False,
help="Optimize consumption against this dictionary of sensors. The sensors typically record electricity prices (e.g. in EUR/kWh), but this field can also be used to optimize against some emission intensity factors (e.g. in kg CO₂ eq./kWh).",
)
@click.option(
"--production-price-sensor-per-device",
"production_price_sensor_per_device",
type=str,
required=False,
help="Optimize production against this dictionary of sensors. Defaults to the consumption price sensor. The sensors typically record electricity prices (e.g. in EUR/kWh), but this field can also be used to optimize against some emission intensity factors (e.g. in kg CO₂ eq./kWh).",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here I would show an example of how to pass a dictionary in the command line.

Moreover, I'm not sure this type (dict) would work straightaway, given that:

  • The dict type is not listed in the click parameters.
  • I created a simple command to test it and I couldn't get to accept any of the formats {"a" : 2}, a=1, "a"=1, a:1 or '{"a":2}'.

Test command:

@fm_add_data.command("test-dict")
@click.option("--dictionary", type=dict)
def test_dict_type(dictionary : dict):
    click.secho(dictionary)

Copy link

@rajath-09 rajath-09 Jun 13, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correctly said.I propose we do something like this?

@click.option("--dictionary", type=str)
def test_dict_type(dictionary: str):
    try:
        dictionary = json.loads(dictionary)
        click.secho(str(dictionary))
    except json.JSONDecodeError:
        click.secho("Invalid dictionary format.", fg='red')

Using this now I can use the format { "19" : 21}
Let me know if that works?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @victorgarcia98
I worked on this now it supports for the following type of input :
flexmeasures add schedule for-storage --sensor-id 1 --consumption-price-sensor-per-device '{"1": 2}' --start ${TOMORROW}T07:00+01:00 --duration PT12H --soc-at-start 50% --roundtrip-efficiency 90% --as-job
Let me know if that's fine?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also given the introduction to the new price sensor parameters, I believe the consumption_price_sensor should no longer be a necessary argument in the CLI and rather an option.It worked for me by omitting the function call to replace_deprecated_argument.Is that how it is supposed to be done?

)
@click.option(
"--optimization-context-id",
"optimization_context_sensor",
Expand Down Expand Up @@ -1029,6 +1043,8 @@ def add_schedule_for_storage(
production_price_sensor: Sensor,
optimization_context_sensor: Sensor,
inflexible_device_sensors: list[Sensor],
consumption_price_sensor_per_device: str,
production_price_sensor_per_device: str,
start: datetime,
duration: timedelta,
soc_at_start: ur.Quantity,
Expand All @@ -1045,13 +1061,54 @@ def add_schedule_for_storage(
- Limited to power sensors (probably possible to generalize to non-electric assets)
- Only supports datetimes on the hour or a multiple of the sensor resolution thereafter
"""
# Convert the 'consumption_price_sensor_per_device' & 'production_price_sensor_per_device' into dictionary
try:
if consumption_price_sensor_per_device is not None:
consumption_price_sensor_per_device = json.loads(
consumption_price_sensor_per_device
)
converted_dict = {}
for sensor_id, value in consumption_price_sensor_per_device.items():
try:
sensor_id = int(sensor_id)
value = int(value)
sensor_obj = SensorIdField()
value = sensor_obj._deserialize(value=value, attr=None, obj=None)
sensor_id = sensor_obj._deserialize(
value=sensor_id, attr=None, obj=None
)
converted_dict[sensor_id] = value
except ValueError:
click.secho(f"Invalid sensor ID: {sensor_id}", fg="red")
consumption_price_sensor_per_device = converted_dict
if production_price_sensor_per_device is not None:
production_price_sensor_per_device = json.loads(
production_price_sensor_per_device
)
converted_dict = {}
for sensor_id, value in production_price_sensor_per_device.items():
try:
sensor_id = int(sensor_id)
value = int(value)
sensor_obj = SensorIdField()
value = sensor_obj._deserialize(value=value, attr=None, obj=None)
sensor_id = sensor_obj._deserialize(
value=sensor_id, attr=None, obj=None
)
converted_dict[sensor_id] = value
except ValueError:
click.secho(f"Invalid sensor ID: {sensor_id}", fg="red")
production_price_sensor_per_device = converted_dict
except json.JSONDecodeError:
click.secho("Invalid dictionary format.", fg="red")

# todo: deprecate the 'optimization-context-id' argument in favor of 'consumption-price-sensor' (announced v0.11.0)
tb_utils.replace_deprecated_argument(
"optimization-context-id",
optimization_context_sensor,
"consumption-price-sensor",
consumption_price_sensor,
)
# tb_utils.replace_deprecated_argument(
# "optimization-context-id",
# optimization_context_sensor,
# # "consumption-price-sensor",
# # consumption_price_sensor,
# )

# Parse input and required sensor attributes
if not power_sensor.measures_power:
Expand Down Expand Up @@ -1106,11 +1163,29 @@ def add_schedule_for_storage(
"roundtrip-efficiency": roundtrip_efficiency,
},
flex_context={
"consumption-price-sensor": consumption_price_sensor.id,
"production-price-sensor": production_price_sensor.id,
"inflexible-device-sensors": [s.id for s in inflexible_device_sensors],
},
)

if consumption_price_sensor is not None:
scheduling_kwargs["flex_context"][
"consumption-price-sensor"
] = consumption_price_sensor.id
if production_price_sensor is not None:
scheduling_kwargs["flex_context"][
"production-price-sensor"
] = production_price_sensor.id
if production_price_sensor_per_device is not None:
scheduling_kwargs["flex_context"]["production-price-sensor-per-device"] = {
power.id: price.id
for power, price in production_price_sensor_per_device.items()
}
if consumption_price_sensor_per_device is not None:
scheduling_kwargs["flex_context"]["consumption-price-sensor-per-device"] = {
power.id: price.id
for power, price in consumption_price_sensor_per_device.items()
}

if as_job:
job = create_scheduling_job(sensor=power_sensor, **scheduling_kwargs)
if job:
Expand Down
83 changes: 50 additions & 33 deletions flexmeasures/data/models/planning/linear_optimization.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
from flask import current_app
import pandas as pd
import numpy as np
from typing import Dict
from pandas.tseries.frequencies import to_offset
from pyomo.core import (
ConcreteModel,
Expand All @@ -29,8 +30,12 @@ def device_scheduler( # noqa C901
device_constraints: List[pd.DataFrame],
ems_constraints: pd.DataFrame,
commitment_quantities: List[pd.Series],
commitment_downwards_deviation_price: Union[List[pd.Series], List[float]],
commitment_upwards_deviation_price: Union[List[pd.Series], List[float]],
consumption_price_sensor_per_device: Dict[int, int],
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if device_scheduler function is used somewhere else (e.g a plugin). In any case, I think we should still support the previous way (i.e commitment_downwards_deviation_price, commitment_upwards_deviation_price). What do you think @Flix6x?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed we should make this backwards compatible instead of introducing a change that potentially breaks things.

At the moment, I think this constitutes a breaking change not only in terms of changing the function signature, but also in terms of breaking a use case. Namely, applying prices to the whole system rather than to each device individually.

production_price_sensor_per_device: Dict[int, int],
commitment_downwards_deviation_price_array: List[
Union[List[pd.Series], List[float]]
],
commitment_upwards_deviation_price_array: List[Union[List[pd.Series], List[float]]],
) -> Tuple[List[pd.Series], float, SolverResults]:
"""This generic device scheduler is able to handle an EMS with multiple devices,
with various types of constraints on the EMS level and on the device level,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The docstring is not updated with the new parameters.

Expand All @@ -54,10 +59,10 @@ def device_scheduler( # noqa C901
Commitments are on an EMS level. Parameter explanations:
commitment_quantities: amounts of flow specified in commitments (both previously ordered and newly requested)
- e.g. in MW or boxes/h
commitment_downwards_deviation_price: penalty for downwards deviations of the flow
commitment_downwards_deviation_price_array: penalty for downwards deviations of the flows
- e.g. in EUR/MW or EUR/(boxes/h)
- either a single value (same value for each flow value) or a Series (different value for each flow value)
commitment_upwards_deviation_price: penalty for upwards deviations of the flow
commitment_upwards_deviation_price_array: penalty for upwards deviations of the flows

All Series and DataFrames should have the same resolution.

Expand Down Expand Up @@ -89,22 +94,26 @@ def device_scheduler( # noqa C901
)

# Turn prices per commitment into prices per commitment flow
if len(commitment_downwards_deviation_price) != 0:
if all(
isinstance(price, float) for price in commitment_downwards_deviation_price
):
commitment_downwards_deviation_price = [
initialize_series(price, start, end, resolution)
for price in commitment_downwards_deviation_price
]
if len(commitment_upwards_deviation_price) != 0:
if all(
isinstance(price, float) for price in commitment_upwards_deviation_price
):
commitment_upwards_deviation_price = [
initialize_series(price, start, end, resolution)
for price in commitment_upwards_deviation_price
]
for i in range(0, len(commitment_downwards_deviation_price_array)):
if len(commitment_downwards_deviation_price_array[i]) != 0:
if all(
isinstance(price, float)
for price in commitment_downwards_deviation_price_array[i]
):
commitment_downwards_deviation_price_array[i] = [
initialize_series(price, start, end, resolution)
for price in commitment_downwards_deviation_price_array[i]
]
for i in range(0, len(commitment_upwards_deviation_price_array)):
if len(commitment_upwards_deviation_price_array[i]) != 0:
if all(
isinstance(price, float)
for price in commitment_upwards_deviation_price_array[i]
):
commitment_upwards_deviation_price_array[i] = [
initialize_series(price, start, end, resolution)
for price in commitment_upwards_deviation_price_array[i]
]

model = ConcreteModel()

Expand All @@ -114,13 +123,14 @@ def device_scheduler( # noqa C901
0, len(device_constraints[0].index.to_pydatetime()) - 1, doc="Set of datetimes"
)
model.c = RangeSet(0, len(commitment_quantities) - 1, doc="Set of commitments")

# Add parameters
def price_down_select(m, c, j):
return commitment_downwards_deviation_price[c].iloc[j]

def price_up_select(m, c, j):
return commitment_upwards_deviation_price[c].iloc[j]
def price_down_select(m, d, c, j):
return commitment_downwards_deviation_price_array[d][c].iloc[j]

def price_up_select(m, d, c, j):
return commitment_upwards_deviation_price_array[d][c].iloc[j]

def commitment_quantity_select(m, c, j):
return commitment_quantities[c].iloc[j]
Expand Down Expand Up @@ -191,8 +201,8 @@ def device_derivative_up_efficiency(m, d, j):
return 1
return eff

model.up_price = Param(model.c, model.j, initialize=price_up_select)
model.down_price = Param(model.c, model.j, initialize=price_down_select)
model.up_price = Param(model.d, model.c, model.j, initialize=price_up_select)
model.down_price = Param(model.d, model.c, model.j, initialize=price_down_select)
model.commitment_quantity = Param(
model.c, model.j, initialize=commitment_quantity_select
)
Expand Down Expand Up @@ -220,10 +230,10 @@ def device_derivative_up_efficiency(m, d, j):
)
model.device_power_up = Var(model.d, model.j, domain=NonNegativeReals, initialize=0)
model.commitment_downwards_deviation = Var(
model.c, model.j, domain=NonPositiveReals, initialize=0
model.d, model.c, model.j, domain=NonPositiveReals, initialize=0
)
model.commitment_upwards_deviation = Var(
model.c, model.j, domain=NonNegativeReals, initialize=0
model.d, model.c, model.j, domain=NonNegativeReals, initialize=0
)

# Add constraints as a tuple of (lower bound, value, upper bound)
Expand Down Expand Up @@ -270,8 +280,8 @@ def ems_flow_commitment_equalities(m, j):
return (
0,
sum(m.commitment_quantity[:, j])
+ sum(m.commitment_downwards_deviation[:, j])
+ sum(m.commitment_upwards_deviation[:, j])
+ sum(m.commitment_downwards_deviation[:, :, j])
+ sum(m.commitment_upwards_deviation[:, :, j])
- sum(m.ems_power[:, j]),
0,
)
Expand Down Expand Up @@ -307,8 +317,15 @@ def cost_function(m):
costs = 0
for c in m.c:
for j in m.j:
costs += m.commitment_downwards_deviation[c, j] * m.down_price[c, j]
costs += m.commitment_upwards_deviation[c, j] * m.up_price[c, j]
for d in m.d:
costs += (
m.commitment_downwards_deviation[d, c, j]
* m.down_price[d, c, j]
)
costs += (
m.commitment_upwards_deviation[d, c, j]
* m.up_price[d, c, j]
)
return costs

model.costs = Objective(rule=cost_function, sense=minimize)
Expand Down
Loading