Re-add links in model definition with v0.7.x format?

[jgu2]

What can be improved?

Description

In model definition of Calliope v0.7.x, the links key got removed and defined as transmission technologies under techs in yaml, the doc - https://calliope.readthedocs.io/en/latest/migrating/#links-transmission-links-defined-in-techs

Would it possible to re-add links but keep current transmission schema definition under techs?

Proposed Change

Add links key and allow users to define transmission technologies, for example,

Current Yaml

techs:
  ccgt:
    name: "Combined cycle gas turbine"
    ...

  csp:
    name: "Concentrating solar power"
    ...

  battery:
    name: "Battery storage"
    ...

  demand_power:
    name: "Power demand"
    ...

  region1_to_region2:
    from: region1
    to: region2
    name: "AC power transmission"
    ...
  region1_to_region1_1:
    from: region1
    to: region1_1
    inherit: free_transmission
  region1_to_region1_2:
    from: region1
    to: region1_2
    inherit: free_transmission
  region1_to_region1_3:
    from: region1
    to: region1_3
    inherit: free_transmission

After the change,

techs:
  ccgt:
    name: "Combined cycle gas turbine"
    ...

  csp:
    name: "Concentrating solar power"
    ...

  battery:
    name: "Battery storage"
    ...

  demand_power:
    name: "Power demand"
    ...

links:
  region1_to_region2:
    from: region1
    to: region2
    name: "AC power transmission"
    ...
  region1_to_region1_1:
    from: region1
    to: region1_1
    inherit: free_transmission
  region1_to_region1_2:
    from: region1
    to: region1_2
    inherit: free_transmission
  region1_to_region1_3:
    from: region1
    to: region1_3
    inherit: free_transmission

This change can let users get links items from model definition easier. And, it looks more natural with links siting along techs.

Minimum Solution

To make this change to be compatible with current schema validation,
In src.calliope.model.py, could add following in to def _init_from_model_def_dict before validate_dict. It would not change any structure of current version after the validation.

if "links" in model_definition:
    model_definition["techs"].update(model_definition.pop("links"))

That is,

    def _init_from_model_def_dict(
        self,
        model_definition: calliope.AttrDict,
        applied_overrides: str,
        scenario: str | None,
        data_source_dfs: dict[str, pd.DataFrame] | None = None,
    ) -> None:
        """Initialise the model using pre-processed YAML files and optional dataframes/dicts.

        Args:
            model_definition (calliope.AttrDict): preprocessed model configuration.
            applied_overrides (str): overrides specified by users
            scenario (str | None): scenario specified by users
            data_source_dfs (dict[str, pd.DataFrame] | None, optional): files with additional model information. Defaults to None.
        """
        
        
        # Pop the `links` and add items into `tech`, making it compatible with current schema.
        if "links" in model_definition:
            model_definition["techs"].update(model_definition.pop("links"))


        # First pass to check top-level keys are all good
        validate_dict(model_definition, CONFIG_SCHEMA, "Model definition")

Version

v0.7.0.dev3

[brynpickering]

The reason for the change was because links were being turned into techs in the pre-processed Calliope model in v0.6 anyway. That is, if you defined a link "ac_transmission" from "A" to "B" there would be an entry "A::ac_transmission:B" in the model data dimension techs.

The change makes it clearer to the user that they are effectively defining technologies when they define transmission links and that they can then expect to find those technologies on the dimension techs in their Calliope model inputs and results.

We tried to work out a way to have them on their own dimension (links), but this causes problems with defining parameters and decision variables that are shared with non-transmission technologies (e.g. flow_out_eff, flow_cap). See this comment for more detail. If you have an idea of how we might achieve that, I'm very open to trying to implement it!

[jgu2]

Thanks for the note.

Yeah, that's why we are trying to make links to be optional, if links appears in yaml, then merge it into techs to align with what current model schema look like. This change is trying to facilitate our current gradient modeling at NREL.

@jmorrisnrel, do you have any input regarding this?

[jmorrisnrel]

Yeah this request would primarily be to help facilitate results processing both for visualizations/analysis and for automating multi-year modeling buildouts. We can easily shift the old behavior of transmission "techs" from techs to tech_groups, but having to place the individual transmission lines in techs means we have to add extra processing to distinguish actual tech definitions in techs from individual transmission lines when processing especially when looking for lines that may have been built in a previous year. Having an optional links tag to specify those in makes the processing cleaner and the YAML a little more readable/familiar to users of v0.6.

The change we implemented doesn't actually impact any of the problem formulation under the hood, instead just looking for links and doing a dictionary update of the model dict after it is loaded to add the content of links to techs. Using it would abstract the YAML definition away from cleanly matching the actual dimensions of the underlying model which may not align with your goals so happy to discuss if there is another alternative.

[brynpickering]

It would be good to know how you're processing results at the moment, to be able to settle on a solution that works best.

There are several ways in python of filtering the data to keep / remove transmission links, the easiest are probably:

in xarray: model.results.flow_out.sel(techs=model.inputs.base_tech == "transmission")
in pandas: df_flow_out.where(df_base_tech == "transmission").dropna()

[jmorrisnrel]

Because we have discrete containerized compute jobs for each solve (with several chained together for a multiyear sequential buildout), we can't easily keep model results in memory between model runs. We are writing all results to disk (currently in the default CSV export format) and create an "output" YAML with all of the input configurations (model, techs, locations, ect.) with inline results sections at the loc_tech/link level in v0.6. This also lets users who aren't as comfortable with python/xarray to review the results in detail in a more human readable format. We could move the transmission techs block to another file and rewrite this code to split the files back out, but having the transmission "links" in a separate section makes this a little more readable for our users.

Our current process for sequential year runs primarily works with this output YAML file as well as the input YAMLs for future years in dictionary form. We load in the results and using those update the future year inputs. Having the full set of inputs right there in the same dict lets us easily copy over any characteristics from the previous run (performance, O&M, ect.) that we need to correctly represent the technology built in that year in the future year run. We'll need to split the outputs YAML into separate files or add some code that either reads one of the input files or hunts through the inheritance in the YAML to determine what techs are transmission so we can determine what was built to copy over to the new year. With a links section we can just loop through that instead which simplifies things. We can certainly adjust our code and likely will do a rewrite of how we process results in the future, but it looked like a relatively small change to add optional support for links back in v0.7 with a quick dict update.

[brynpickering]

I somehow think it would be easier to implement your postprocessing without a links top-level key, since the links look identical to other techs (just with two protected parameters: to and from). You could also save the results as CSV and load them in as data sources (soon to be data tables) so that you remove the YAMLification step entirely.