Data priority not respected by ModelDataFactory

[irm-codebase]

What happened?

ModelDataFactory is affected by the order in which data_tables are defined in the YAML file.
In certain cases, values defined at the techs level will be lost (that is, only stuff at the nodes.techs level will be left).

Below you can find a file that replicates this issue in the pytest tests/test_example_modes.py file.
All it does is 'shuffle' the order of the data_tables section in the nat_model_from_data_tables case to reproduce the issue.

# Model configuration: all settings that affect the built model
config:
  init:
    name: National-scale example model
    # What version of Calliope this model is intended for
    calliope_version: 0.7.0
    time_subset: ["2005-01-01", "2005-01-05"] # Subset of timesteps

  build:
    ensure_feasibility: true # Switches on the "unmet demand" constraint
    mode: plan # Choices: plan, operate

  solve:
    solver: cbc
    zero_threshold: 1e-10 # Any value coming out of the backend that is smaller than this (due to floating point errors, probably) will be set to zero

nodes:
  region1.techs: {demand_power, ccgt}
  region2.techs: {demand_power, battery}
  region1_1.techs: {csp}
  region1_2.techs: {csp}
  region1_3.techs: {csp}

data_tables:
  dimensionless_params:
    data: data_tables/dimensionless_params.csv
    rows: parameters

  techs_costs_monetary:
    data: data_tables/techs_costs_monetary.csv
    rows: techs
    columns: parameters
    add_dims:
      costs: monetary

  costs_params:
    data: data_tables/costs_params.csv
    rows: costs
    columns: parameters

  techs_node_constraints:
    data: data_tables/techs_node_constraints.csv
    rows: [nodes, techs]
    columns: parameters

  techs_constraints:
    data: data_tables/techs_constraints.csv
    rows: techs
    columns: parameters
  # will be loaded from the example model directory in calliope source code.
  time_varying_data_from_df:
    data: time_varying_df
    rows: timesteps
    columns: [comment, nodes, techs, parameters]
    drop: comment

  nodes_base_info:
    data: data_tables/nodes_base_info.csv
    rows: nodes
    columns: parameters

  links:
    data: data_tables/links.csv
    rows: techs
    columns: parameters

  techs_base_info:
    data: data_tables/techs_base_info.csv
    rows: techs
    columns: parameters

  techs_carriers_at_nodes:
    data: data_tables/techs_carriers.csv
    rows: techs
    columns: parameters
    add_dims:
      carriers: power

Which operating systems have you used?

• macOS
• Windows
• Linux

Version

v0.7.0.dev4

Relevant log output

❯ pytest tests/test_example_models.py
======================================================================= test session starts ========================================================================
platform linux -- Python 3.12.7, pytest-8.3.3, pluggy-1.5.0
rootdir: /home/ivanruizmanuel/Documents/git/calliope
configfile: pyproject.toml
plugins: cov-4.1.0, order-1.3.0, xdist-3.6.1
14 workers [39 items]     
sssssssssssssssssss............Fss.F...                                                                                                                      [100%]
============================================================================= FAILURES =============================================================================
___________________________ TestNationalScaleExampleModelSenseChecks.test_nationalscale_example_results_cbc[nat_model_from_data_tables] ____________________________
[gw4] linux -- Python 3.12.7 /home/ivanruizmanuel/miniforge3/envs/calliope/bin/python3.12

self = <tests.test_example_models.TestNationalScaleExampleModelSenseChecks object at 0x7fa7b8fe64b0>
example_tester = <function TestNationalScaleExampleModelSenseChecks.example_tester.<locals>._example_tester at 0x7fa7b883eca0>

    def test_nationalscale_example_results_cbc(self, example_tester):
>       example_tester()

tests/test_example_models.py:114: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

solver = 'cbc', solver_io = None

    def _example_tester(solver="cbc", solver_io=None):
        model = request.getfixturevalue(request.param)
    
        solve_kwargs = {"solver": solver}
        if solver_io:
            solve_kwargs["solver_io"] = solver_io
    
        model.solve(force=True, **solve_kwargs)
    
>       assert model.results.storage_cap.sel(
            nodes="region1_1", techs="csp"
        ) == approx(45129.950)
E       AssertionError: assert <xarray.DataA...s_result:    1 == 45129.95 ± 4.5e-02
E         
E         comparison failed
E         Obtained: <xarray.DataArray 'storage_cap' ()> Size: 8B\narray(0.)\nCoordinates:\n    techs    <U3 12B 'csp'\n    nodes    <U9 36B 'region1_1'\nAttributes:\n    title:        Stored carrier capacity\n    description:  The upper limit on a carrier that can be stored by a techno...\n    default:      0\n    unit:         energy\n    is_result:    1
E         Expected: 45129.95 ± 4.5e-02

tests/test_example_models.py:79: AssertionError
------------------------------------------------------------------------ Captured log call -------------------------------------------------------------------------
WARNING  calliope.postprocess.postprocess:postprocess.py:158 Postprocessing: All values < 1e-10 set to 0 in flow_cap, link_flow_cap, flow_out, flow_in, storage_cap, storage, flow_out_inc_eff, flow_in_inc_eff, cost_operation_variable, cost_investment_storage_cap, cost_investment, cost_investment_annualised, cost
___________________________ TestNationalScaleExampleModelSenseChecks.test_nationalscale_example_results_glpk[nat_model_from_data_tables] ___________________________
[gw4] linux -- Python 3.12.7 /home/ivanruizmanuel/miniforge3/envs/calliope/bin/python3.12

self = <tests.test_example_models.TestNationalScaleExampleModelSenseChecks object at 0x7fa7b8fe6090>
example_tester = <function TestNationalScaleExampleModelSenseChecks.example_tester.<locals>._example_tester at 0x7fa7b7e8ae80>

    def test_nationalscale_example_results_glpk(self, example_tester):
        if shutil.which("glpsol"):
>           example_tester(solver="glpk")

tests/test_example_models.py:129: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

solver = 'glpk', solver_io = None

    def _example_tester(solver="cbc", solver_io=None):
        model = request.getfixturevalue(request.param)
    
        solve_kwargs = {"solver": solver}
        if solver_io:
            solve_kwargs["solver_io"] = solver_io
    
        model.solve(force=True, **solve_kwargs)
    
>       assert model.results.storage_cap.sel(
            nodes="region1_1", techs="csp"
        ) == approx(45129.950)
E       AssertionError: assert <xarray.DataA...s_result:    1 == 45129.95 ± 4.5e-02
E         
E         comparison failed
E         Obtained: <xarray.DataArray 'storage_cap' ()> Size: 8B\narray(0.)\nCoordinates:\n    techs    <U3 12B 'csp'\n    nodes    <U9 36B 'region1_1'\nAttributes:\n    title:        Stored carrier capacity\n    description:  The upper limit on a carrier that can be stored by a techno...\n    default:      0\n    unit:         energy\n    is_result:    1
E         Expected: 45129.95 ± 4.5e-02

tests/test_example_models.py:79: AssertionError
------------------------------------------------------------------------ Captured log call -------------------------------------------------------------------------
WARNING  calliope.postprocess.postprocess:postprocess.py:158 Postprocessing: All values < 1e-10 set to 0 in flow_cap, link_flow_cap, flow_out, flow_in, area_use, source_use, source_cap, storage, flow_out_inc_eff, flow_in_inc_eff, cost_operation_variable, cost_investment_flow_cap, cost_investment_source_cap, cost_investment_area_use, cost_investment_annualised, cost
========================================================================= warnings summary =========================================================================
tests/test_example_models.py::TestNationalScaleExampleModelSenseChecks::test_nationalscale_example_results_cbc[nat_model]
tests/test_example_models.py::TestNationalScaleExampleModelOperate::test_nationalscale_example_results_cbc
tests/test_example_models.py::TestNationalScaleResampledExampleModelSenseChecks::test_nationalscale_resampled_example_results_glpk
tests/test_example_models.py::TestNationalScaleExampleModelSenseChecks::test_nationalscale_example_results_cbc[nat_model_from_data_tables]
tests/test_example_models.py::TestNationalScaleExampleModelSenseChecks::test_nationalscale_example_results_glpk[nat_model]
tests/test_example_models.py::TestNationalScaleExampleModelSenseChecks::test_nationalscale_example_results_glpk[nat_model_from_data_tables]
  /home/ivanruizmanuel/Documents/git/calliope/src/calliope/postprocess/postprocess.py:158: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
    LOGGER.warn(comment)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
===================================================================== short test summary info ======================================================================
SKIPPED [4] tests/test_example_models.py: SPORES mode will fail until the cost max group constraint can be reproduced
SKIPPED [3] tests/test_example_models.py:245: SPORES mode will fail until the cost max group constraint can be reproduced
SKIPPED [12] tests/test_example_models.py:303: SPORES mode will fail until the cost max group constraint can be reproduced
SKIPPED [2] tests/test_example_models.py:125: CPLEX not installed
FAILED tests/test_example_models.py::TestNationalScaleExampleModelSenseChecks::test_nationalscale_example_results_cbc[nat_model_from_data_tables] - AssertionError: assert <xarray.DataA...s_result:    1 == 45129.95 ± 4.5e-02
FAILED tests/test_example_models.py::TestNationalScaleExampleModelSenseChecks::test_nationalscale_example_results_glpk[nat_model_from_data_tables] - AssertionError: assert <xarray.DataA...s_result:    1 == 45129.95 ± 4.5e-02
================================================= 2 failed, 16 passed, 21 skipped, 6 warnings in 101.35s (0:01:41) =================================================

[irm-codebase]

@brynpickering since was discovered in #719, I will attempt to get it fixed there.
I'll also try to include test cases that ensure this does not slip by again.

[brynpickering]

This is something of a feature, not a bug. See point 3 here: https://calliope.readthedocs.io/en/latest/creating/data_tables/#important-considerations. It allows a user to set tech-level parameters and then overlay that with node-level overrides using data tables.

[irm-codebase]

@brynpickering ok... This is still easily missed, and somewhat arbitrary.
I'll change this to a feature request where the 'dimensional order of priority' is clearly defined.
Perhaps as part of the params/dims feature, or a subsequent update.

My worry is that this can easily lead to users accepting results from models with erroneous data because they flipped one data table.