New input files format - split of Generators_data.csv #612

lbonaldo · 2024-01-13T02:14:47Z

Description

This PR splits Generators_data.csv into a few different files, one for each type of generator. This allows each file to contain only the attributes related to the specific type of generator for which it is meant.
A large number of files had to be changed, because all the examples and tests needed to be modified (and so did the associated csv files).

The implementation follows an initial restructuring proposed by @cfe316. The main formulation is defined in the resources.jl file and can be summarized as follows:

A new Resources folder located inside the case folder contains the new input files, one for each type of resource (e.g., vre.csv, storage.csv, hydro.csv, etc.), along with the files that contain the columns related to policies (e.g., Resource_capacity_reserve_margin.csv, Resource_energy_share_requirement.csv etc.)
load_resources_data.jl substitutes load_generators_data.jl and contains the functions that read each input file and create the new data structures
each resource is an instance of a Julia type (e.g., GenX.Vre, GenX.Hydro, GenX.Storage, etc). The types are defined in the resources.jl file along with a set of function interfaces to interact with the resources. In this way, attributes of each resource can be looked up indirectly through the interface or directly with the usual dot notation.
The interface also contains a set of utility functions that can be used to wrap operations that are used several times in the code. For example, the function resources_in_zone_by_rid(gen, zone) proposed by @cfe316 can be called instead of the way it has been done till now, i.e. using dfGen[dfGen.Zone .== zone, :R_ID] to get the ids of the resources in a certain zone. (Julia macros can be used to facilitate the creation of these function interfaces).

Since this PR is a significant change to the code, I'd appreciate any comments, thoughts, or suggestions on how to modify or improve it.

Todo:

Test RealSystemExample
Final cleanup
Update docs

What type of PR is this? (check all applicable)

Code Refactor

Checklist

Code changes are sufficiently documented; i.e. new functions contain docstrings and .md files under /docs/src have been updated if necessary.
The latest changes on the target branch have been incorporated, so that any conflicts are taken care of before merging. This can be accomplished either by merging in the target branch (e.g. 'git merge develop') or by rebasing on top of the target branch (e.g. 'git rebase develop'). Please do not hesitate to reach out to the GenX development team if you need help with this.
Code has been tested to ensure all functionality works as intended.
CHANGELOG.md has been updated (if this is a 'notable' change).
I consent to the release of this PR's code under the GNU General Public license.

How this can be tested

With the tests and the examples provided in the GenX repository.

Post-approval checklist for GenX core developers

After the PR is approved

Check that the latest changes on the target branch are incorporated, either via merge or rebase
Remember to squash and merge if incorporating into develop

lbonaldo · 2024-01-24T00:59:33Z

This PR is now ready for review.

sambuddhac

Here are a few of my clarifying questions, @lbonaldo

The exporting of AbstractResource in line 27 in GenX.jl has been added to facilitate the use of the newly defined abstract type at splitting the Generators_data.csv according to resources?
I need some help understanding the modifications to piecewise fuel file, MGA, Method of Morris, and Multistage loading of inputs. I checked the other files following the guidelines from you and rest everything seems okay by me.
I also tested this branch of the fork manually with a few example cases (real system three zone, real system one zone, simple system one zone, simple system three zone, and multistage real three zone and it works good for all of these except for the multistage model (Error message attached at the end of this message; I couldn't dig for debugging yet though).
Can you please clarify once again the advantage of using the function calls instead of the dataframe referencing?

Error message from multistage real ISONE trizone:


***********
Iteration Number: 1
Upper Bound = Inf
Lower Bound = 64958.37577086839
***********
***********
Forward Pass t = 2
***********
Running HiGHS 1.6.0: Copyright (c) 2023 HiGHS under MIT licence terms
Presolving model
Problem status detected on presolve: Infeasible
Model   status      : Infeasible
Objective value     :  0.0000000000e+00
HiGHS run time      :          0.53
ERROR:   No LP invertible representation for getDualRay
MILP solved for primal
ERROR: LoadError: Result index of attribute MathOptInterface.VariablePrimal(1) out of bounds. There are currently 0 solution(s) in the model.
Stacktrace:
  [1] check_result_index_bounds
    @ ~/.julia/packages/MathOptInterface/DDWnF/src/attributes.jl:207 [inlined]
  [2] get
    @ ~/.julia/packages/HiGHS/k1fEq/src/MOI_wrapper.jl:2131 [inlined]
  [3] get(b::MathOptInterface.Bridges.LazyBridgeOptimizer{HiGHS.Optimizer}, attr::MathOptInterface.VariablePrimal, index::MathOptInterface.VariableIndex)
    @ MathOptInterface.Bridges ~/.julia/packages/MathOptInterface/DDWnF/src/Bridges/bridge_optimizer.jl:1265
  [4] get(model::MathOptInterface.Utilities.CachingOptimizer{MathOptInterface.Bridges.LazyBridgeOptimizer{HiGHS.Optimizer}, MathOptInterface.Utilities.UniversalFallback{MathOptInterface.Utilities.Model{Float64}}}, attr::MathOptInterface.VariablePrimal, index::MathOptInterface.VariableIndex)
    @ MathOptInterface.Utilities ~/.julia/packages/MathOptInterface/DDWnF/src/Utilities/cachingoptimizer.jl:932
  [5] _moi_get_result(::MathOptInterface.Utilities.CachingOptimizer{MathOptInterface.Bridges.LazyBridgeOptimizer{HiGHS.Optimizer}, MathOptInterface.Utilities.UniversalFallback{MathOptInterface.Utilities.Model{Float64}}}, ::MathOptInterface.VariablePrimal, ::Vararg{Any})
    @ JuMP ~/.julia/packages/JuMP/HjlGr/src/optimizer_interface.jl:663
  [6] get(model::JuMP.Model, attr::MathOptInterface.VariablePrimal, v::JuMP.VariableRef)
    @ JuMP ~/.julia/packages/JuMP/HjlGr/src/optimizer_interface.jl:703
  [7] value(v::JuMP.VariableRef; result::Int64)
    @ JuMP ~/.julia/packages/JuMP/HjlGr/src/variables.jl:1703
  [8] value
    @ ~/.julia/packages/JuMP/HjlGr/src/variables.jl:1702 [inlined]
  [9] #55
    @ ./none:0 [inlined]
 [10] iterate
    @ ./generator.jl:47 [inlined]
 [11] _all(f::Base.var"#343#345", itr::Base.Generator{Vector{JuMP.VariableRef}, GenX.var"#55#56"}, #unused#::Colon)
    @ Base ./reduce.jl:1242
 [12] all
    @ ./reduce.jl:1238 [inlined]
 [13] Dict(kv::Base.Generator{Vector{JuMP.VariableRef}, GenX.var"#55#56"})
    @ Base ./dict.jl:131
 [14] fix_integers(jump_model::JuMP.Model)
    @ GenX ~/code/GenXPRTest/GenX/src/model/solve_model.jl:24
 [15] solve_model(EP::JuMP.Model, setup::Dict{Any, Any})
    @ GenX ~/code/GenXPRTest/GenX/src/model/solve_model.jl:73
 [16] run_ddp(models_d::Dict{Any, Any}, setup::Dict{Any, Any}, inputs_d::Dict{Any, Any})
    @ GenX ~/code/GenXPRTest/GenX/src/multi_stage/dual_dynamic_programming.jl:222
 [17] run_genx_case_multistage!(case::String, mysetup::Dict{Any, Any}, optimizer::Type)
    @ GenX ~/code/GenXPRTest/GenX/src/case_runners/case_runner.jl:145
 [18] run_genx_case!(case::String, optimizer::Type)
    @ GenX ~/code/GenXPRTest/GenX/src/case_runners/case_runner.jl:23
 [19] run_genx_case!(case::String)
    @ GenX ~/code/GenXPRTest/GenX/src/case_runners/case_runner.jl:17
 [20] top-level scope
    @ ~/code/GenXPRTest/GenX/Example_Systems/RealSystemExample/ISONE_Trizone_MultiStage/Run.jl:3
 [21] include(fname::String)
    @ Base.MainInclude ./client.jl:476
 [22] top-level scope
    @ REPL[4]:1
in expression starting at /Users/sc87/code/GenXPRTest/GenX/Example_Systems/RealSystemExample/ISONE_Trizone_MultiStage/Run.jl:3

caused by: Result index of attribute MathOptInterface.VariablePrimal(1) out of bounds. There are currently 0 solution(s) in the model.
Stacktrace:
  [1] check_result_index_bounds
    @ ~/.julia/packages/MathOptInterface/DDWnF/src/attributes.jl:207 [inlined]
  [2] get
    @ ~/.julia/packages/HiGHS/k1fEq/src/MOI_wrapper.jl:2131 [inlined]
  [3] get(b::MathOptInterface.Bridges.LazyBridgeOptimizer{HiGHS.Optimizer}, attr::MathOptInterface.VariablePrimal, index::MathOptInterface.VariableIndex)
    @ MathOptInterface.Bridges ~/.julia/packages/MathOptInterface/DDWnF/src/Bridges/bridge_optimizer.jl:1265
  [4] get(model::MathOptInterface.Utilities.CachingOptimizer{MathOptInterface.Bridges.LazyBridgeOptimizer{HiGHS.Optimizer}, MathOptInterface.Utilities.UniversalFallback{MathOptInterface.Utilities.Model{Float64}}}, attr::MathOptInterface.VariablePrimal, index::MathOptInterface.VariableIndex)
    @ MathOptInterface.Utilities ~/.julia/packages/MathOptInterface/DDWnF/src/Utilities/cachingoptimizer.jl:932
  [5] _moi_get_result(::MathOptInterface.Utilities.CachingOptimizer{MathOptInterface.Bridges.LazyBridgeOptimizer{HiGHS.Optimizer}, MathOptInterface.Utilities.UniversalFallback{MathOptInterface.Utilities.Model{Float64}}}, ::MathOptInterface.VariablePrimal, ::Vararg{Any})
    @ JuMP ~/.julia/packages/JuMP/HjlGr/src/optimizer_interface.jl:663
  [6] get(model::JuMP.Model, attr::MathOptInterface.VariablePrimal, v::JuMP.VariableRef)
    @ JuMP ~/.julia/packages/JuMP/HjlGr/src/optimizer_interface.jl:703
  [7] value(v::JuMP.VariableRef; result::Int64)
    @ JuMP ~/.julia/packages/JuMP/HjlGr/src/variables.jl:1703
  [8] value
    @ ~/.julia/packages/JuMP/HjlGr/src/variables.jl:1702 [inlined]
  [9] #55
    @ ./none:0 [inlined]
 [10] iterate
    @ ./generator.jl:47 [inlined]
 [11] Dict{JuMP.VariableRef, Float64}(kv::Base.Generator{Vector{JuMP.VariableRef}, GenX.var"#55#56"})
    @ Base ./dict.jl:103
 [12] dict_with_eltype
    @ ./abstractdict.jl:575 [inlined]
 [13] dict_with_eltype
    @ ./abstractdict.jl:582 [inlined]
 [14] Dict(kv::Base.Generator{Vector{JuMP.VariableRef}, GenX.var"#55#56"})
    @ Base ./dict.jl:129
 [15] fix_integers(jump_model::JuMP.Model)
    @ GenX ~/code/GenXPRTest/GenX/src/model/solve_model.jl:24
 [16] solve_model(EP::JuMP.Model, setup::Dict{Any, Any})
    @ GenX ~/code/GenXPRTest/GenX/src/model/solve_model.jl:73
 [17] run_ddp(models_d::Dict{Any, Any}, setup::Dict{Any, Any}, inputs_d::Dict{Any, Any})
    @ GenX ~/code/GenXPRTest/GenX/src/multi_stage/dual_dynamic_programming.jl:222
 [18] run_genx_case_multistage!(case::String, mysetup::Dict{Any, Any}, optimizer::Type)
    @ GenX ~/code/GenXPRTest/GenX/src/case_runners/case_runner.jl:145
 [19] run_genx_case!(case::String, optimizer::Type)
    @ GenX ~/code/GenXPRTest/GenX/src/case_runners/case_runner.jl:23
 [20] run_genx_case!(case::String)
    @ GenX ~/code/GenXPRTest/GenX/src/case_runners/case_runner.jl:17
 [21] top-level scope
    @ ~/code/GenXPRTest/GenX/Example_Systems/RealSystemExample/ISONE_Trizone_MultiStage/Run.jl:3
 [22] include(fname::String)
    @ Base.MainInclude ./client.jl:476
 [23] top-level scope
    @ REPL[4]:1

sambuddhac

Code reviewed, doubts clarified, successfully tested against both multistage and single stage one zone and three zone cases, and approved !!!

cfe316

Hi Luca, I've made some minor comments - mostly style.

cfe316 · 2024-02-12T20:39:48Z