-
Notifications
You must be signed in to change notification settings - Fork 10
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Min max scaling for observation space #508
base: main
Are you sure you want to change the base?
Conversation
…o Min-Max-Scaling
Note: Demand sclaing param unequal action sclaing param
- renamed max_bid_price to max_market_price
@AndreasEppler I made a couple of quick fixes and pushed them. The scaling of the action space (from the hyperparameters) and the observation space were somewhat mixed up. Could you take care of the remaining points? Specifically, testing with an example that uses both normal and advanced orders. |
scaled_res_load_forecast = min_max_scale( | ||
unit.forecaster[f"residual_load_{market_id}"].loc[start:], | ||
lower_scaling_factor_res_load, | ||
upper_scaling_factor_res_load, | ||
) | ||
scaled_res_load_forecast = np.concatenate( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why no scaling in this part?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I confusingly marked these lines, but I meant the ones below
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
oh, this is a mistake, this should be fixed
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #508 +/- ##
==========================================
+ Coverage 76.55% 76.66% +0.11%
==========================================
Files 51 51
Lines 6871 6896 +25
==========================================
+ Hits 5260 5287 +27
+ Misses 1611 1609 -2
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
good suggestion and a nice catch with the mistake not scaling some values. I have left some comments to improve the performance and how things are handled
# stays here as it is unit specific, and different forecasts might apply for different units | ||
# different handling would require an extra unit loop at learning role intiliazation and unit specific max/min values | ||
# further forecasts might change during the simulation if advanced forecasting is used | ||
self.max_market_price = max(unit.forecaster[f"price_{market_id}"]) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this can be done in init so it is done only once and not during every call of the function
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I disagree for the reasons stated in the lengthly comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@kim-mskw what lengthy comment? Don't see it
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I argue that the market price max might be different for different units in the future, hence it should be unit specific as well
stays here as it is unit specific, and different forecasts might apply for different units
different handling would require an extra unit loop at learning role intiliazation and unit specific max/min values
further forecasts might change during the simulation if advanced forecasting is used
# price forecast | ||
scaling_factor_price = self.max_bid_price | ||
upper_scaling_factor_price = self.max_market_price |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why assign new variable here if we can use self.values directly?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We did the same beforehand to make this crucial part of the learning easily changeable for new users; compare the RL tutorial. We could change it if you think it would be easier to understand.
@@ -320,18 +342,30 @@ def create_observation( | |||
current_costs = unit.calculate_marginal_cost(start, current_volume) | |||
|
|||
# scale unit outputs | |||
scaled_max_power = current_volume / scaling_factor_total_capacity | |||
scaled_marginal_cost = current_costs / scaling_factor_marginal_cost | |||
scaled_max_power = min_max_scale( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
same here, should be calculated only once in init
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I disagree for the reasons stated in the lengthly comment
@@ -36,6 +36,8 @@ Upcoming Release | |||
- **Outputs Role Performance Optimization:** Output role handles dict data directly and only converts to DataFrame on Database write. | |||
- **Overall Performance Optimization:** The overall performance of the framework has been improved by a factor of 5x to 12x | |||
depending on the size of the simulation (number of units, markets, and time steps). | |||
- **Learning Opservation Space Scaling:** Instead of the formerly used max sclaing of the observation space, we added a min-max scaling to the observation space. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- **Learning Opservation Space Scaling:** Instead of the formerly used max sclaing of the observation space, we added a min-max scaling to the observation space. | |
- **Learning Observation Space Scaling:** Instead of the formerly used max scaling of the observation space, we added a min-max scaling to the observation space. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I thought the new precommit would fix something like this?
scaled_res_load_forecast = min_max_scale( | ||
unit.forecaster[f"residual_load_{market_id}"].loc[start:], | ||
lower_scaling_factor_res_load, | ||
upper_scaling_factor_res_load, | ||
) | ||
scaled_res_load_forecast = np.concatenate( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
oh, this is a mistake, this should be fixed
# stays here as it is unit specific, and different forecasts might apply for different units | ||
# different handling would require an extra unit loop at learning role intiliazation and unit specific max/min values | ||
# further forecasts might change during the simulation if advanced forecasting is used | ||
self.max_market_price = max(unit.forecaster[f"price_{market_id}"]) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we should also here compute these things in the init directly to save time. Also no need to assign values from self like upper_scaling_factor_res_load = self.max_residual since we can use directly the self. values
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I disagree for the reasons stated in the lengthly comment
@@ -10,6 +10,7 @@ | |||
from assume.common.base import SupportsMinMax | |||
from assume.common.market_objects import MarketConfig, Orderbook, Product | |||
from assume.common.utils import get_products_index | |||
from assume.reinforcement_learning.learning_utils import min_max_scale |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't see this as really a learning util, but just a simple util
Pull Request
Description
To Include more robust observation space scaling, a min-max scaling is proposed instead of the formerly introduced max scaling.
Changes Proposed
Testing
With example 02a tiny
Checklist
Please check all applicable items:
doc
folder updates)pyproject.toml
doc/release_notes.rst
of the upcoming release is included