Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove technology_type column from tools.costs #269

Merged
merged 4 commits into from
Dec 16, 2024
Merged

Remove technology_type column from tools.costs #269

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
cf509f6
Remove `technology_type` column from input CSV files
measrainsey Dec 12, 2024
d61e2ec
Remove instances in code that used `technology_type` column
measrainsey Dec 12, 2024
1b61d33
Add #269 to doc/whatsnew
measrainsey Dec 12, 2024
59c4ace
Replace `._append()` with `pd.concat()`
measrainsey Dec 12, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/whatsnew.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,7 @@ Investment and fixed costs
- Add `cooling` technology variant/module (:pull:`222`).
- Add functionality to specify cost reduction values and cost reduction scenarios in a module (:issue:`251`, :pull:`255`).
- Fix technology mapping of ammonia CCS technologies in materials module (:pull:`258`).
- Remove unused `technology_type` column from tool (:pull:`269`).

Others
------
Expand Down
192 changes: 96 additions & 96 deletions message_ix_models/data/costs/energy/cost_reduction.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,96 +1,96 @@
# Cost reduction in 2100,,,,,,
# ,,,,,,
# Units: % ,,,,,,
#,,,,,,
# Data is copied from Sheet1 in https://github.com/iiasa/message_data/blob/dev/data/model/investment_cost/SSP_technology_learning.xlsx,,,,,,
# There are some manually changed assumptions to the original GEA data in the spreadsheet (can be seen in the spreadsheet as marked in yellow),,,,,,
# The initial copied data can be found in gea_reduction_rates.csv,,,,,,
# This file renames the learning rates from GEAL to low and GEAM to medium and GEAH to high,,,,,,
message_technology,technology_type,very_low,low,medium,high,very_high
bio_istig,Biomass,0,0.1,0.3,0.4,0.5
bio_istig_ccs,CCS,0,0.1,0.3,0.4,0.6
bio_ppl,Biomass,0,0.1,0.2,0.3,0.4
bio_ppl_co2scr,CCS,0,0,0,0.3,0.4
biomass_i,Biomass,0,0,0,0,0
c_ppl_co2scr,CCS,0,0,0,0.3,0.4
coal_adv,Coal,0,0.1,0.3,0.5,0.7
coal_adv_ccs,CCS,0,0.1,0.3,0.5,0.7
coal_i,Coal,0,0,0,0,0
coal_ppl,Coal,0,0,0.2,0.5,0.7
coal_ppl_u,Coal,0,0,0,0,0
csp_sm1_ppl,Renewable,0.15,0.3,0.3,0.5,0.7
csp_sm3_ppl,Renewable,0.15,0.3,0.3,0.5,0.7
elec_i,NA,0,0,0,0,0
eth_bio,Biomass,0,0.27,0.27,0.4,0.55
eth_bio_ccs,CCS,0,0.27,0.27,0.4,0.55
eth_i,Biomass,0,0,0,0,0
foil_i,Gas/Oil,0,0,0,0,0
g_ppl_co2scr,CCS,0,0,0,0.3,0.4
gas_cc,Gas/Oil,0.1,0.2,0.29,0.38,0.5
gas_cc_ccs,CCS,0.1,0.2,0.29,0.5,0.7
gas_ct,Gas/Oil,0.1,0.2,0.29,0.38,0.5
gas_i,Gas/Oil,0,0,0,0,0
gas_ppl,Gas/Oil,0.1,0.2,0.29,0.38,0.5
geo_hpl,Renewable,0.1,0.15,0.18,0.25,0.35
geo_ppl,Renewable,0,0.1,0.18,0.25,0.35
h2_bio,Biomass,0,0.25,0.4,0.5,0.7
h2_bio_ccs,CCS,0,0.25,0.4,0.5,0.7
h2_coal,Coal,0,0.25,0.4,0.4,0.5
h2_coal_ccs,CCS,0,0.25,0.4,0.4,0.5
h2_elec,Renewable,0,0,0.1,0.2,0.3
h2_i,NA,0,0,0,0,0
h2_smr,Gas/Oil,0,0.25,0.4,0.5,0.7
h2_smr_ccs,CCS,0,0.25,0.4,0.5,0.7
heat_i,NA,0,0,0,0,0
hp_el_i,Renewable,0.1,0.2,0.5,0.5,0.6
hp_gas_i,Gas/Oil,0.1,0.2,0.4,0.4,0.5
hydro_hc,Renewable,0,0,0,0,0
hydro_lc,Renewable,0,0,0,0,0
igcc,Coal,0,0.1,0.3,0.5,0.7
igcc_ccs,CCS,0,0.1,0.3,0.5,0.7
liq_bio,Biomass,0.15,0.27,0.27,0.4,0.55
liq_bio_ccs,CCS,0.15,0.27,0.27,0.4,0.55
loil_i,Gas/Oil,0,0,0,0,0
meth_coal,Coal,0,0.05,0.1,0.15,0.2
meth_coal_ccs,CCS,0,0.05,0.1,0.15,0.2
meth_i,Coal,0,0,0,0,0
meth_ng,Gas/Oil,0,0.05,0.1,0.15,0.2
meth_ng_ccs,CCS,0,0.05,0.1,0.15,0.2
nuc_hc,Nuclear,0,0,0.15,0.3,0.45
nuc_lc,Nuclear,0,0,0,0,0
solar_i,Renewable,0.1,0.2,0.6,0.9,0.95
solar_pv_I,Renewable,0.15,0.3,0.7,0.9,0.95
solar_pv_RC,Renewable,0.15,0.3,0.7,0.9,0.95
solar_res_hist_2005,Renewable,0.15,0.3,0.7,0.9,0.95
solar_res_hist_2010,Renewable,0.15,0.3,0.7,0.9,0.95
solar_res_hist_2015,Renewable,0.15,0.3,0.7,0.9,0.95
solar_res_hist_2020,Renewable,0.15,0.3,0.7,0.9,0.95
solar_res1,Renewable,0.15,0.3,0.7,0.9,0.95
solar_res2,Renewable,0.15,0.3,0.7,0.9,0.95
solar_res3,Renewable,0.15,0.3,0.7,0.9,0.95
solar_res4,Renewable,0.15,0.3,0.7,0.9,0.95
solar_res5,Renewable,0.15,0.3,0.7,0.9,0.95
solar_res6,Renewable,0.15,0.3,0.7,0.9,0.95
solar_res7,Renewable,0.15,0.3,0.7,0.9,0.95
solar_res8,Renewable,0.15,0.3,0.7,0.9,0.95
solar_th_ppl,Renewable,0.15,0.3,0.3,0.5,0.7
stor_ppl,Renewable,0.15,0.3,0.7,0.9,0.95
syn_liq,Coal,0,0.05,0.1,0.15,0.2
syn_liq_ccs,CCS,0,0.05,0.1,0.15,0.2
wind_ref_hist_2005,Renewable,0.15,0.3,0.53,0.65,0.75
wind_ref_hist_2010,Renewable,0.15,0.3,0.53,0.65,0.75
wind_ref_hist_2015,Renewable,0.15,0.3,0.53,0.65,0.75
wind_ref_hist_2020,Renewable,0.15,0.3,0.53,0.65,0.75
wind_ref1,Renewable,0.15,0.3,0.53,0.65,0.75
wind_ref2,Renewable,0.15,0.3,0.53,0.65,0.75
wind_ref3,Renewable,0.15,0.3,0.53,0.65,0.75
wind_ref4,Renewable,0.15,0.3,0.53,0.65,0.75
wind_ref5,Renewable,0.15,0.3,0.53,0.65,0.75
wind_res_hist_2005,Renewable,0.15,0.3,0.53,0.65,0.75
wind_res_hist_2010,Renewable,0.15,0.3,0.53,0.65,0.75
wind_res_hist_2015,Renewable,0.15,0.3,0.53,0.65,0.75
wind_res_hist_2020,Renewable,0.15,0.3,0.53,0.65,0.75
wind_res1,Renewable,0.15,0.3,0.53,0.65,0.75
wind_res2,Renewable,0.15,0.3,0.53,0.65,0.75
wind_res3,Renewable,0.15,0.3,0.53,0.65,0.75
wind_res4,Renewable,0.15,0.3,0.53,0.65,0.75
# Cost reduction in 2100,,,,,
# ,,,,,
# Units: % ,,,,,
#,,,,,
# Data is copied from Sheet1 in https://github.com/iiasa/message_data/blob/dev/data/model/investment_cost/SSP_technology_learning.xlsx,,,,,
# There are some manually changed assumptions to the original GEA data in the spreadsheet (can be seen in the spreadsheet as marked in yellow),,,,,
# The initial copied data can be found in gea_reduction_rates.csv,,,,,
# This file renames the learning rates from GEAL to low and GEAM to medium and GEAH to high,,,,,
message_technology,very_low,low,medium,high,very_high
bio_istig,0,0.1,0.3,0.4,0.5
bio_istig_ccs,0,0.1,0.3,0.4,0.6
bio_ppl,0,0.1,0.2,0.3,0.4
bio_ppl_co2scr,0,0,0,0.3,0.4
biomass_i,0,0,0,0,0
c_ppl_co2scr,0,0,0,0.3,0.4
coal_adv,0,0.1,0.3,0.5,0.7
coal_adv_ccs,0,0.1,0.3,0.5,0.7
coal_i,0,0,0,0,0
coal_ppl,0,0,0.2,0.5,0.7
coal_ppl_u,0,0,0,0,0
csp_sm1_ppl,0.15,0.3,0.3,0.5,0.7
csp_sm3_ppl,0.15,0.3,0.3,0.5,0.7
elec_i,0,0,0,0,0
eth_bio,0,0.27,0.27,0.4,0.55
eth_bio_ccs,0,0.27,0.27,0.4,0.55
eth_i,0,0,0,0,0
foil_i,0,0,0,0,0
g_ppl_co2scr,0,0,0,0.3,0.4
gas_cc,0.1,0.2,0.29,0.38,0.5
gas_cc_ccs,0.1,0.2,0.29,0.5,0.7
gas_ct,0.1,0.2,0.29,0.38,0.5
gas_i,0,0,0,0,0
gas_ppl,0.1,0.2,0.29,0.38,0.5
geo_hpl,0.1,0.15,0.18,0.25,0.35
geo_ppl,0,0.1,0.18,0.25,0.35
h2_bio,0,0.25,0.4,0.5,0.7
h2_bio_ccs,0,0.25,0.4,0.5,0.7
h2_coal,0,0.25,0.4,0.4,0.5
h2_coal_ccs,0,0.25,0.4,0.4,0.5
h2_elec,0,0,0.1,0.2,0.3
h2_i,0,0,0,0,0
h2_smr,0,0.25,0.4,0.5,0.7
h2_smr_ccs,0,0.25,0.4,0.5,0.7
heat_i,0,0,0,0,0
hp_el_i,0.1,0.2,0.5,0.5,0.6
hp_gas_i,0.1,0.2,0.4,0.4,0.5
hydro_hc,0,0,0,0,0
hydro_lc,0,0,0,0,0
igcc,0,0.1,0.3,0.5,0.7
igcc_ccs,0,0.1,0.3,0.5,0.7
liq_bio,0.15,0.27,0.27,0.4,0.55
liq_bio_ccs,0.15,0.27,0.27,0.4,0.55
loil_i,0,0,0,0,0
meth_coal,0,0.05,0.1,0.15,0.2
meth_coal_ccs,0,0.05,0.1,0.15,0.2
meth_i,0,0,0,0,0
meth_ng,0,0.05,0.1,0.15,0.2
meth_ng_ccs,0,0.05,0.1,0.15,0.2
nuc_hc,0,0,0.15,0.3,0.45
nuc_lc,0,0,0,0,0
solar_i,0.1,0.2,0.6,0.9,0.95
solar_pv_I,0.15,0.3,0.7,0.9,0.95
solar_pv_RC,0.15,0.3,0.7,0.9,0.95
solar_res_hist_2005,0.15,0.3,0.7,0.9,0.95
solar_res_hist_2010,0.15,0.3,0.7,0.9,0.95
solar_res_hist_2015,0.15,0.3,0.7,0.9,0.95
solar_res_hist_2020,0.15,0.3,0.7,0.9,0.95
solar_res1,0.15,0.3,0.7,0.9,0.95
solar_res2,0.15,0.3,0.7,0.9,0.95
solar_res3,0.15,0.3,0.7,0.9,0.95
solar_res4,0.15,0.3,0.7,0.9,0.95
solar_res5,0.15,0.3,0.7,0.9,0.95
solar_res6,0.15,0.3,0.7,0.9,0.95
solar_res7,0.15,0.3,0.7,0.9,0.95
solar_res8,0.15,0.3,0.7,0.9,0.95
solar_th_ppl,0.15,0.3,0.3,0.5,0.7
stor_ppl,0.15,0.3,0.7,0.9,0.95
syn_liq,0,0.05,0.1,0.15,0.2
syn_liq_ccs,0,0.05,0.1,0.15,0.2
wind_ref_hist_2005,0.15,0.3,0.53,0.65,0.75
wind_ref_hist_2010,0.15,0.3,0.53,0.65,0.75
wind_ref_hist_2015,0.15,0.3,0.53,0.65,0.75
wind_ref_hist_2020,0.15,0.3,0.53,0.65,0.75
wind_ref1,0.15,0.3,0.53,0.65,0.75
wind_ref2,0.15,0.3,0.53,0.65,0.75
wind_ref3,0.15,0.3,0.53,0.65,0.75
wind_ref4,0.15,0.3,0.53,0.65,0.75
wind_ref5,0.15,0.3,0.53,0.65,0.75
wind_res_hist_2005,0.15,0.3,0.53,0.65,0.75
wind_res_hist_2010,0.15,0.3,0.53,0.65,0.75
wind_res_hist_2015,0.15,0.3,0.53,0.65,0.75
wind_res_hist_2020,0.15,0.3,0.53,0.65,0.75
wind_res1,0.15,0.3,0.53,0.65,0.75
wind_res2,0.15,0.3,0.53,0.65,0.75
wind_res3,0.15,0.3,0.53,0.65,0.75
wind_res4,0.15,0.3,0.53,0.65,0.75
8 changes: 4 additions & 4 deletions message_ix_models/data/costs/materials/cost_reduction.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Cost reduction in 2100,,,,,,
# ,,,,,,
# Units: % ,,,,,,
message_technology,technology_type,very_low,low,medium,high,very_high
# Cost reduction in 2100,,,,,
# ,,,,,
# Units: % ,,,,,
message_technology,very_low,low,medium,high,very_high
15 changes: 6 additions & 9 deletions message_ix_models/tools/costs/decay.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,9 +81,9 @@ def _get_module_scenarios_reduction(
)
]

# append scen_red_module to scen_red_energy
scenarios_joined = scenarios_energy._append(scenarios_module).reset_index(
drop=True
# concat scenarios_energy and scenarios_module
scenarios_joined = pd.concat(
[scenarios_energy, scenarios_module], ignore_index=True
)
else:
scenarios_joined = scenarios_energy.copy()
Expand Down Expand Up @@ -207,7 +207,6 @@ def _get_module_cost_reduction(
pd.concat([reduction_energy, reduction_energy_no_reduction], ignore_index=True)
.sort_values("message_technology")
.reset_index(drop=True)
.drop(columns=["technology_type"])
)

if module != "energy":
Expand All @@ -224,11 +223,9 @@ def _get_module_cost_reduction(
)
]

# append scen_red_module to scen_red_energy
reduction_joined = (
reduction_energy._append(reduction_module)
.reset_index(drop=True)
.drop(columns=["technology_type"])
# concat reduction_energy and reduction_module
reduction_joined = pd.concat(
[reduction_energy, reduction_module], ignore_index=True
)
else:
reduction_joined = reduction_energy.copy()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In such cases, I prefer this kind of syntax:

reduction_joined = reduction_energy.copy() if module == "energy" else reduction_energy._append(reduction_module).reset_index(drop=True)

I think this is an improvement as it reduces the number of lines requiring tests and more clearly states that all you do is define reduction_joined, just slightly differently.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, but what is ._append() here? I thought pandas deprecated and removed methods like DataFrame.append() in v2.0 (2023), and I don't see this one in the documentation.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah interesting - it seems that function may be a "private" function that isn't supported by pandas 🙈 i'll push a commit to switch to using concat or something else like that!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks @glatterf42 for the suggestion!

the if...else statement is actually not on module == "energy" but on if os.path.exists(ffile). if the file exists, then the code reads in the file as a dataframe, deletes some rows in another dataframe, and then concats the two dataframes together as the output. if the file doesn't exist, then the output is just the second dataframe copied over. i thought about condensing the code like you suggested but seems like the if part requires a few more steps? 😅

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, sorry, I misunderstood what was going on. I guess its technically possible to still make this one line, but I don't know if it will be easier to read. Up to you if you want to try this :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if it's alright, i would prefer to skip editing this part for this PR and move forward with approving/merging -- thank you! :)

Expand Down
Loading