Find out if status2019 and eGon100RE can be created in one pipeline

[ClaraBuettner]

Since the results from osmTGmod are not deterministic, it would help to be able to create both scenarios in one pipeline.

I will test with a Schleswig-Holstein run if that works out.

[ClaraBuettner]

@CarlosEpia Is it fine to use the current dev for this check or should I use another branch?

[ClaraBuettner]

There seems to be only one task that is only in the status quo pipeline:

powerd-data/src/egon/data/airflow/dags/pipeline_status_quo.py

Line 47 in bb84edc

from egon.data.datasets.heat_supply.individual_heating import HeatPumps2019

I guess we can simply add this one.

[ClaraBuettner]

Some tasks run into errors (electricity_demand.temporal.insert-cts-load, industry.temporal.insert-osm-ind-load) because the weather year for status2019 is another one then for eGon100RE.
I think that could be a problem for other tasks as well.
Fixing that would require some large changes.

@CarlosEpia and @ulfmueller: Would you think that it is fine to create a status2019-scenario with the general weather year we have for the other scenarios? At least for the path it would help I guess.

[CarlosEpia]

Some tasks run into errors (electricity_demand.temporal.insert-cts-load, industry.temporal.insert-osm-ind-load) because the weather year for status2019 is another one then for eGon100RE. I think that could be a problem for other tasks as well. Fixing that would require some large changes.

@CarlosEpia and @ulfmueller: Would you think that it is fine to create a status2019-scenario with the general weather year we have for the other scenarios? At least for the path it would help I guess.

I think it is okay to use the same weather data for both scenarios, but I do not know how to justify it, especially because we have weather data for 2019. What do you think if we create the scenarios using 2011 weather data but update it for 2019 once the pipeline is ready?

[ClaraBuettner]

I think it is okay to use the same weather data for both scenarios, but I do not know how to justify it, especially because we have weather data for 2019. What do you think if we create the scenarios using 2011 weather data but update it for 2019 once the pipeline is ready?

I think we can justify it by saying that we need one consistent weather year for the scenario path. That also makes sense for the scaling, demand profiles often depend on the calendar year, so it would be better to not mix up different years.

For the validation, 2019 makes sense, but we would not do that again.

[CarlosEpia]

Sounds good to me.

[ClaraBuettner]

I tried it out but run into problems with demandregio:

  File "/home/clara/powerd-data-36/venv/lib/python3.8/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/home/clara/powerd-data-36/venv/lib/python3.8/site-packages/requests/adapters.py", line 700, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='open***.ffe.de', port=4000): Max retries exceeded with url: /demandregio_spatial?id_spatial=eq.71&&year=eq.2011&&value=gt.0.0 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x78dfdb453850>: Failed to establish a new connection: [Errno 110] Connection timed out'))

The cached data is there. Could it be the case that it was not used when the year changed?

[ClaraBuettner]

The tasks electrical_neighbours.insert-generators-sq, electrical_neighbours.insert-storage-units-sq and electrical_neighbours.insert-loads-sq are not working as expected. The data in the backups includes data for 2019 and the entsoe-API can be used for years after 2015. The year 2011 is not working.

[ClaraBuettner]

I was able to create status2019 and eGon100RE in one pipeline for the test mode.
There is now one consistent weather year (2011) used for all scenarios.
Unfortunately, the foreign demand time series (electrical_neighbours.insert-loads-sq) still uses 2019 and I did not change this since the data on the entsoe-platform is only available for years after 2015.
For the scenario path resp. the intermediate scenarios we could just directly the data from pypsa-eur for foreign loads and not do any scaling. But the status2019 scenario created with this branch is not completely consistent.