Min max scaling for observation space #508

kim-mskw · 2024-12-03T15:34:39Z

Pull Request

Description

To Include more robust observation space scaling, a min-max scaling is proposed instead of the formerly introduced max scaling.

Changes Proposed

min/max scaling in learning strategies and advanced learning strategies
not in tutorials, as we want to stick to an easy scaling introduction and not over-engineer it there
changes of actual scaling values are minor

Testing

With example 02a tiny

Checklist

Please check all applicable items:

…o Min-Max-Scaling

Note: Demand sclaing param unequal action sclaing param

- renamed max_bid_price to max_market_price

kim-mskw · 2024-12-03T15:48:40Z

@AndreasEppler I made a couple of quick fixes and pushed them. The scaling of the action space (from the hyperparameters) and the observation space were somewhat mixed up. Could you take care of the remaining points? Specifically, testing with an example that uses both normal and advanced orders.

kim-mskw · 2024-12-03T15:49:48Z

assume/strategies/learning_strategies.py

+            scaled_res_load_forecast = min_max_scale(
+                unit.forecaster[f"residual_load_{market_id}"].loc[start:],
+                lower_scaling_factor_res_load,
+                upper_scaling_factor_res_load,
            )
            scaled_res_load_forecast = np.concatenate(


Why no scaling in this part?

I confusingly marked these lines, but I meant the ones below

oh, this is a mistake, this should be fixed

codecov · 2024-12-03T16:05:42Z

Codecov Report

Attention: Patch coverage is 93.10345% with 4 lines in your changes missing coverage. Please review.

Project coverage is 76.66%. Comparing base (7457a21) to head (e63cbbe).

Files with missing lines	Patch %	Lines
assume/strategies/learning_advanced_orders.py	90.47%	2 Missing ⚠️
assume/strategies/learning_strategies.py	94.28%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #508      +/-   ##
==========================================
+ Coverage   76.55%   76.66%   +0.11%     
==========================================
  Files          51       51              
  Lines        6871     6896      +25     
==========================================
+ Hits         5260     5287      +27     
+ Misses       1611     1609       -2

Flag	Coverage Δ
pytest	`76.66% <93.10%> (+0.11%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

nick-harder

good suggestion and a nice catch with the mistake not scaling some values. I have left some comments to improve the performance and how things are handled

nick-harder · 2024-12-04T08:14:51Z

assume/strategies/learning_advanced_orders.py

+        # stays here as it is unit specific, and different forecasts might apply for different units
+        # different handling would require an extra unit loop at learning role intiliazation and unit specific max/min values
+        # further forecasts might change during the simulation if advanced forecasting is used
+        self.max_market_price = max(unit.forecaster[f"price_{market_id}"])


this can be done in init so it is done only once and not during every call of the function

I disagree for the reasons stated in the lengthly comment

@kim-mskw what lengthy comment? Don't see it

I argue that the market price max might be different for different units in the future, hence it should be unit specific as well

stays here as it is unit specific, and different forecasts might apply for different units

different handling would require an extra unit loop at learning role intiliazation and unit specific max/min values

further forecasts might change during the simulation if advanced forecasting is used

nick-harder · 2024-12-04T08:15:44Z

assume/strategies/learning_advanced_orders.py

        # price forecast
-        scaling_factor_price = self.max_bid_price
+        upper_scaling_factor_price = self.max_market_price


why assign new variable here if we can use self.values directly?

We did the same beforehand to make this crucial part of the learning easily changeable for new users; compare the RL tutorial. We could change it if you think it would be easier to understand.

nick-harder · 2024-12-04T08:16:35Z

assume/strategies/learning_advanced_orders.py

@@ -320,18 +342,30 @@ def create_observation(
        current_costs = unit.calculate_marginal_cost(start, current_volume)

        # scale unit outputs
-        scaled_max_power = current_volume / scaling_factor_total_capacity
-        scaled_marginal_cost = current_costs / scaling_factor_marginal_cost
+        scaled_max_power = min_max_scale(


same here, should be calculated only once in init

I disagree for the reasons stated in the lengthly comment

nick-harder · 2024-12-04T08:20:27Z

docs/source/release_notes.rst

@@ -36,6 +36,8 @@ Upcoming Release
 - **Outputs Role Performance Optimization:** Output role handles dict data directly and only converts to DataFrame on Database write.
 - **Overall Performance Optimization:** The overall performance of the framework has been improved by a factor of 5x to 12x
  depending on the size of the simulation (number of units, markets, and time steps).
+- **Learning Opservation Space Scaling:** Instead of the formerly used max sclaing of the observation space, we added a min-max scaling to the observation space.


Suggested change

- **Learning Opservation Space Scaling:** Instead of the formerly used max sclaing of the observation space, we added a min-max scaling to the observation space.

- **Learning Observation Space Scaling:** Instead of the formerly used max scaling of the observation space, we added a min-max scaling to the observation space.

I thought the new precommit would fix something like this?

nick-harder · 2024-12-04T08:34:05Z

assume/strategies/learning_strategies.py

+            scaled_res_load_forecast = min_max_scale(
+                unit.forecaster[f"residual_load_{market_id}"].loc[start:],
+                lower_scaling_factor_res_load,
+                upper_scaling_factor_res_load,
            )
            scaled_res_load_forecast = np.concatenate(


oh, this is a mistake, this should be fixed

nick-harder · 2024-12-04T08:35:07Z

assume/strategies/learning_strategies.py

+        # stays here as it is unit specific, and different forecasts might apply for different units
+        # different handling would require an extra unit loop at learning role intiliazation and unit specific max/min values
+        # further forecasts might change during the simulation if advanced forecasting is used
+        self.max_market_price = max(unit.forecaster[f"price_{market_id}"])


we should also here compute these things in the init directly to save time. Also no need to assign values from self like upper_scaling_factor_res_load = self.max_residual since we can use directly the self. values

I disagree for the reasons stated in the lengthly comment

nick-harder · 2024-12-04T08:44:41Z

assume/strategies/learning_advanced_orders.py

@@ -10,6 +10,7 @@
 from assume.common.base import SupportsMinMax
 from assume.common.market_objects import MarketConfig, Orderbook, Product
 from assume.common.utils import get_products_index
+from assume.reinforcement_learning.learning_utils import min_max_scale


I don't see this as really a learning util, but just a simple util

AndreasEppler and others added 6 commits December 2, 2024 17:36

Implemented min-max-scaling instead of simple max_scaling.

7db09e9

Merge branch 'main' into Min-Max-Scaling

9ab9e9e

Merge branch 'main' of https://github.com/assume-framework/assume int…

df7f44c

…o Min-Max-Scaling

- added comment for clarification

691ee6b

- readded max_bid_price to be read from config as hyperparameter

1059ba6

Note: Demand sclaing param unequal action sclaing param

- add min max sclaing to learning advanced orders

86e0cd2

- renamed max_bid_price to max_market_price

kim-mskw assigned kim-mskw and AndreasEppler Dec 3, 2024

kim-mskw added 3 commits December 3, 2024 16:38

add release note

88da585

run pre commit

b4ad505

- fixed unused variable

4f0ad60

kim-mskw commented Dec 3, 2024

View reviewed changes

- adjusted sclaing for storages

730e366

nick-harder requested changes Dec 4, 2024

View reviewed changes

Merge branch 'main' into Min-Max-Scaling

e63cbbe

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Min max scaling for observation space #508

Min max scaling for observation space #508

kim-mskw commented Dec 3, 2024 •

edited

Loading

kim-mskw commented Dec 3, 2024

kim-mskw Dec 3, 2024

kim-mskw Dec 3, 2024

nick-harder Dec 4, 2024

codecov bot commented Dec 3, 2024 •

edited

Loading

nick-harder left a comment

nick-harder Dec 4, 2024

kim-mskw Dec 4, 2024

nick-harder Dec 5, 2024

kim-mskw Dec 5, 2024

nick-harder Dec 4, 2024

kim-mskw Dec 4, 2024

nick-harder Dec 4, 2024

kim-mskw Dec 4, 2024

nick-harder Dec 4, 2024

kim-mskw Dec 4, 2024

nick-harder Dec 4, 2024

nick-harder Dec 4, 2024

kim-mskw Dec 4, 2024

nick-harder Dec 4, 2024

	- Learning Opservation Space Scaling: Instead of the formerly used max sclaing of the observation space, we added a min-max scaling to the observation space.
	- Learning Observation Space Scaling: Instead of the formerly used max scaling of the observation space, we added a min-max scaling to the observation space.

Min max scaling for observation space #508

Are you sure you want to change the base?

Min max scaling for observation space #508

Conversation

kim-mskw commented Dec 3, 2024 • edited Loading

Pull Request

Description

Changes Proposed

Testing

Checklist

kim-mskw commented Dec 3, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Dec 3, 2024 • edited Loading

Codecov Report

nick-harder left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stays here as it is unit specific, and different forecasts might apply for different units

different handling would require an extra unit loop at learning role intiliazation and unit specific max/min values

further forecasts might change during the simulation if advanced forecasting is used

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kim-mskw commented Dec 3, 2024 •

edited

Loading

codecov bot commented Dec 3, 2024 •

edited

Loading