Append BNetzA-MaStR renewables to power plants #11

fneum · 2019-06-18T09:20:50Z

I would like to add the wind and solar power plants from the BNetzA-MaStR to the matched power plant dataset (i.e. no matching of solar and wind, just concatenated at the end). I assume I need to create a function in data.py that translates the data to the common format? I read there's already the option to add wind and solar from OPSD. Where can I find this? I could draw some inspiration from that.

The text was updated successfully, but these errors were encountered:

FabianHofmann · 2019-07-09T14:04:46Z

Hey @fneum!

I am sorry for the late reply, yes indeed you can extend your data set by the vres of opsd. If you have some more days of patience, I am working on a new version of ppm which should facilitate the whole usage.

fneum · 2019-07-09T14:20:35Z

Sweet! There is no hurry since the geolocation script for the solar plants in BNetzA-MaStR is still running.

BTW will you consider versioning, pip, maybe conda and so on?

FabianHofmann · 2019-07-09T14:27:18Z

yes, I intended to go for conda, since I wanted to check out how to create a recipe. although pip would be nice as well. perhaps both :)

coroa · 2019-07-10T08:22:33Z

Both!

Recipes based on pip are easiest :) And pip is not hard either.

For conda: https://github.com/conda-forge/pypsa-feedstock/blob/master/recipe/meta.yaml

FabianHofmann · 2019-07-16T11:28:57Z

@fneum (I will just use this thread here) could you test the pip installation of powerplantmatching in a new environment?

pip install powerplantmatching

and if that worked

import powerplantmatching as pm
pm.powerplants()

the last might take a while, it downloads, cleans and merges all the data files. (JRC should be included)

To answer your original question, you can use

import powerplantmatching as pm
pm.powerplants().powerplant.extend_by_VRE(base_year=2016)

which however has a stupid bug atm, which I just encoutered. But more on this later :)

fneum · 2019-07-16T12:52:45Z

Seems to work!

Python 3.7.3 | packaged by conda-forge | (default, Jul  1 2019, 21:52:21) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.6.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import powerplantmatching as pm                                                                                                                                             
WARNING:powerplantmatching.plot:Neither mpl_toolkits.basemap module nor cartopy existent.
INFO:root:Outputs for this configuration will be saved under /home/ws/sp2668/.local/share/powerplantmatching/data/out/default

In [2]: pm.powerplants()                                                                                                                                                            
INFO:powerplantmatching.collection:Collect combined dataset for CARMA, ENTSOE, GEO, GPD, JRC, OPSD
WARNING:powerplantmatching.collection:Forcing update since the cache file is missing
INFO:powerplantmatching:Retrieving data from https://vfs.fias.science/f/e1d9d9d587/?raw=1
INFO:powerplantmatching:Retrieving data from https://data.open-power-system-data.org/conventional_power_plants/2018-12-20/conventional_power_plants_DE.csv
INFO:powerplantmatching:Retrieving data from https://data.open-power-system-data.org/conventional_power_plants/2018-12-20/conventional_power_plants_EU.csv
INFO:powerplantmatching.cleaning:Cleaning plant names in 'CARMA'.
INFO:powerplantmatching.cleaning:Aggregating blocks to entire units in 'CARMA'.
INFO:powerplantmatching.cleaning:Non-existing saved aggregation groups for dataset 'CARMA', continuing by aggregating again
INFO:powerplantmatching:Retrieving data from https://vfs.fias.science/f/ebbdf6ba8c/?raw=1
WARNING:powerplantmatching:Geoparsing not possible as no google api key was found, please add the key to your config.yaml if you want to enable it.
INFO:powerplantmatching.cleaning:Cleaning plant names in 'ENTSOE'.
INFO:powerplantmatching.cleaning:Aggregating blocks to entire units in 'ENTSOE'.
INFO:powerplantmatching.cleaning:Non-existing saved aggregation groups for dataset 'ENTSOE', continuing by aggregating again
INFO:powerplantmatching:Retrieving data from https://vfs.fias.science/f/b4607c76b4/?raw=1
INFO:powerplantmatching.cleaning:Cleaning plant names in 'GEO'.
INFO:powerplantmatching.cleaning:Aggregating blocks to entire units in 'GEO'.
INFO:powerplantmatching.cleaning:Non-existing saved aggregation groups for dataset 'GEO', continuing by aggregating again
INFO:powerplantmatching.cleaning:Cleaning plant names in 'GPD'.
INFO:powerplantmatching.cleaning:Aggregating blocks to entire units in 'GPD'.
INFO:powerplantmatching.cleaning:Non-existing saved aggregation groups for dataset 'GPD', continuing by aggregating again
INFO:powerplantmatching.cleaning:Cleaning plant names in 'JRC'.
INFO:powerplantmatching.cleaning:Aggregating blocks to entire units in 'JRC'.
INFO:powerplantmatching.cleaning:Non-existing saved aggregation groups for dataset 'JRC', continuing by aggregating again
INFO:powerplantmatching.cleaning:Cleaning plant names in 'OPSD'.
INFO:powerplantmatching.cleaning:Aggregating blocks to entire units in 'OPSD'.
INFO:powerplantmatching.cleaning:Non-existing saved aggregation groups for dataset 'OPSD', continuing by aggregating again
INFO:powerplantmatching.matching:Comparing CARMA with ENTSOE
WARNING:powerplantmatching.matching:Non-existing saved matches for dataset 'CARMA', 'ENTSOE' continuing by matching again
INFO:powerplantmatching.matching:Comparing CARMA with GEO
WARNING:powerplantmatching.matching:Non-existing saved matches for dataset 'CARMA', 'GEO' continuing by matching again
INFO:powerplantmatching.matching:Comparing CARMA with GPD
WARNING:powerplantmatching.matching:Non-existing saved matches for dataset 'CARMA', 'GPD' continuing by matching again
INFO:powerplantmatching.matching:Comparing CARMA with JRC
WARNING:powerplantmatching.matching:Non-existing saved matches for dataset 'CARMA', 'JRC' continuing by matching again
INFO:powerplantmatching.matching:Comparing CARMA with OPSD
WARNING:powerplantmatching.matching:Non-existing saved matches for dataset 'CARMA', 'OPSD' continuing by matching again
INFO:powerplantmatching.matching:Comparing ENTSOE with GEO
WARNING:powerplantmatching.matching:Non-existing saved matches for dataset 'ENTSOE', 'GEO' continuing by matching again
INFO:powerplantmatching.matching:Comparing ENTSOE with GPD
WARNING:powerplantmatching.matching:Non-existing saved matches for dataset 'ENTSOE', 'GPD' continuing by matching again
INFO:powerplantmatching.matching:Comparing ENTSOE with JRC
WARNING:powerplantmatching.matching:Non-existing saved matches for dataset 'ENTSOE', 'JRC' continuing by matching again
INFO:powerplantmatching.matching:Comparing ENTSOE with OPSD
WARNING:powerplantmatching.matching:Non-existing saved matches for dataset 'ENTSOE', 'OPSD' continuing by matching again
INFO:powerplantmatching.matching:Comparing GEO with GPD
WARNING:powerplantmatching.matching:Non-existing saved matches for dataset 'GEO', 'GPD' continuing by matching again
INFO:powerplantmatching.matching:Comparing GEO with JRC
WARNING:powerplantmatching.matching:Non-existing saved matches for dataset 'GEO', 'JRC' continuing by matching again
INFO:powerplantmatching.matching:Comparing GEO with OPSD
WARNING:powerplantmatching.matching:Non-existing saved matches for dataset 'GEO', 'OPSD' continuing by matching again
INFO:powerplantmatching.matching:Comparing GPD with JRC
WARNING:powerplantmatching.matching:Non-existing saved matches for dataset 'GPD', 'JRC' continuing by matching again
INFO:powerplantmatching.matching:Comparing GPD with OPSD
WARNING:powerplantmatching.matching:Non-existing saved matches for dataset 'GPD', 'OPSD' continuing by matching again
INFO:powerplantmatching.matching:Comparing JRC with OPSD
WARNING:powerplantmatching.matching:Non-existing saved matches for dataset 'JRC', 'OPSD' continuing by matching again
INFO:powerplantmatching.cleaning:Cleaning plant names in 'OPSD'.
INFO:powerplantmatching.cleaning:Aggregating blocks to entire units in 'OPSD'.
INFO:powerplantmatching.cleaning:Cleaning plant names in 'JRC'.
INFO:powerplantmatching.cleaning:Aggregating blocks to entire units in 'JRC'.
WARNING:powerplantmatching:Geoparsing not possible as no google api key was found, please add the key to your config.yaml if you want to enable it.
INFO:powerplantmatching.cleaning:Cleaning plant names in 'ENTSOE'.
INFO:powerplantmatching.cleaning:Aggregating blocks to entire units in 'ENTSOE'.
INFO:powerplantmatching.cleaning:Cleaning plant names in 'CARMA'.
INFO:powerplantmatching.cleaning:Aggregating blocks to entire units in 'CARMA'.
INFO:powerplantmatching.cleaning:Cleaning plant names in 'GEO'.
INFO:powerplantmatching.cleaning:Aggregating blocks to entire units in 'GEO'.
Out[2]: 
                                                   Name Fueltype  ...                                          projectID                                           eic_code
0                                                  Alta    Hydro  ...  {'CARMA': ['CARMA1122'], 'ENTSOE': ['50WP00000...  {'CARMA': [], 'ENTSOE': ['50WP00000000006N'], ...
1                   Porąbka żar porabka zar porabka zar    Hydro  ...  {'CARMA': ['CARMA35310', 'CARMA35311'], 'ENTSO...  {'CARMA': [], 'ENTSOE': ['19W000000000247X'], ...
2                                Żydowo zydowops zydowo    Hydro  ...  {'CARMA': ['CARMA51126'], 'ENTSOE': ['19W00000...  {'CARMA': [], 'ENTSOE': ['19W0000000002426'], ...
3                         Żarnowiec zarnowiec zarnowiec    Hydro  ...  {'CARMA': ['CARMA50891'], 'ENTSOE': ['19W00000...  {'CARMA': [], 'ENTSOE': ['19W0000000002418'], ...
4                                             Svartisen    Hydro  ...  {'CARMA': ['CARMA43936'], 'ENTSOE': ['50WP0000...  {'CARMA': [], 'ENTSOE': ['50WP00000000695B'], ...
5                                               Skjomen    Hydro  ...  {'CARMA': ['CARMA41924'], 'ENTSOE': ['50WP0000...  {'CARMA': [], 'ENTSOE': ['50WP00000000633X'], ...
6                         Seitevare seitevare seitevare    Hydro  ...  {'CARMA': ['CARMA40432'], 'ENTSOE': ['46WPU000...  {'CARMA': [], 'ENTSOE': ['46WPU0000000033U'], ...
7                                  Solina solina solina    Hydro  ...  {'CARMA': ['CARMA42275'], 'ENTSOE': ['19W00000...  {'CARMA': [], 'ENTSOE': ['19W0000000002515'], ...
8             Monteynard monteynard monteyard avigonnet    Hydro  ...  {'CARMA': ['CARMA29247'], 'ENTSOE': ['17W100P1...  {'CARMA': [], 'ENTSOE': ['17W100P100P02861'], ...
9                    Wien freudenau freudenau freudenau    Hydro  ...  {'CARMA': ['CARMA14110'], 'ENTSOE': ['14W-BFR-...  {'CARMA': [], 'ENTSOE': ['14W-BFR-KW-----A'], ...
10          Foyers falls foyers foye foyers scotland uk    Hydro  ...  {'CARMA': ['CARMA13925', 'CARMA13926'], 'ENTSO...  {'CARMA': [], 'ENTSOE': ['48WSTN00000FOYEQ'], ...
11                                 Dychów dychow dychow    Hydro  ...  {'CARMA': ['CARMA11576'], 'ENTSOE': ['19W00000...  {'CARMA': [], 'ENTSOE': ['19W000000000272Y'], ...
12                            Donzere mondragon bollene    Hydro  ...  {'CARMA': ['CARMA11135'], 'ENTSOE': ['17W100P1...  {'CARMA': [], 'ENTSOE': ['17W100P100P0324J'], ...
13                             Edst dlouhe strane czech    Hydro  ...  {'CARMA': ['CARMA10945', 'CARMA10946'], 'ENTSO...  {'CARMA': [], 'ENTSOE': ['27W-PU-EDST----Z'], ...
14                                  Edal dalesice czech    Hydro  ...  {'CARMA': ['CARMA10090'], 'ENTSOE': ['27W-PU-E...  {'CARMA': [], 'ENTSOE': ['27W-PU-EDAL----O'], ...
15                              Oraison oraison oraison    Hydro  ...  {'CARMA': ['CARMA32690'], 'ENTSOE': ['17W100P1...  {'CARMA': [], 'ENTSOE': ['17W100P100P0288Y'], ...
16                              Pouget pouget pouget le    Hydro  ...  {'CARMA': ['CARMA35556', 'CARMA24759'], 'ENTSO...  {'CARMA': [], 'ENTSOE': ['17W100P100P0290A'], ...
17                                    Porsi porsi porsi    Hydro  ...  {'CARMA': ['CARMA35341'], 'ENTSOE': ['46WPU000...  {'CARMA': [], 'ENTSOE': ['46WPU0000000036O'], ...
18                                 Porjus porjus porjus    Hydro  ...  {'CARMA': ['CARMA35333'], 'ENTSOE': ['46WPU000...  {'CARMA': [], 'ENTSOE': ['46WPU0000000028N'], ...
19                     Pied borne pied borne pied borne    Hydro  ...  {'CARMA': ['CARMA34551'], 'ENTSOE': ['17W100P1...  {'CARMA': [], 'ENTSOE': ['17W100P100P0289W'], ...
20                  Ottmarsheim ottmarsheim ottmarsheim    Hydro  ...  {'CARMA': ['CARMA33091'], 'ENTSOE': ['17W100P1...  {'CARMA': [], 'ENTSOE': ['17W100P100P0306L'], ...
21    Ottensheim wilhering ottensheim wilhering otte...    Hydro  ...  {'CARMA': ['CARMA33078'], 'ENTSOE': ['14W-BOW-...  {'CARMA': [], 'ENTSOE': ['14W-BOW-KW-----9'], ...
22                              Mareges mareges mareges    Hydro  ...  {'CARMA': ['CARMA27194'], 'ENTSOE': ['17W100P1...  {'CARMA': [], 'ENTSOE': ['17W100P100P0328B'], ...
23                      Montezic montezic montezic pump    Hydro  ...  {'CARMA': ['CARMA29248'], 'ENTSOE': ['17W100P1...  {'CARMA': [], 'ENTSOE': ['17W100P100P02772'], ...
24                           Messaure messaure messaure    Hydro  ...  {'CARMA': ['CARMA28173'], 'ENTSOE': ['46WPU000...  {'CARMA': [], 'ENTSOE': ['46WPU0000000032W'], ...
25                                       Melk melk melk    Hydro  ...  {'CARMA': ['CARMA27970'], 'ENTSOE': ['14W-BME-...  {'CARMA': [], 'ENTSOE': ['14W-BME-KW-----Q'], ...
26                        Mayrhofen mayrhofen mayrhofen    Hydro  ...  {'CARMA': ['CARMA27679'], 'ENTSOE': ['14W-TZMA...  {'CARMA': [], 'ENTSOE': ['14W-TZMA-KW----N'], ...
27    Markersbach psw markersbach pss markersbach ge...    Hydro  ...  {'CARMA': ['CARMA27280'], 'ENTSOE': ['11WD8MAR...  {'CARMA': [], 'ENTSOE': ['11WD8MARK1H----7'], ...
28    Presenzano presenzan presenzano domenico cimar...    Hydro  ...  {'CARMA': ['CARMA35733'], 'ENTSOE': ['26WIMPI-...  {'CARMA': [], 'ENTSOE': ['26WIMPI-S15PRSN8'], ...
29                        Pragneres pragneres pragneres    Hydro  ...  {'CARMA': ['CARMA35658'], 'ENTSOE': ['17W100P1...  {'CARMA': [], 'ENTSOE': ['17W100P100P02918'], ...
...                                                 ...      ...  ...                                                ...                                                ...
7128                                         Vals bains    Hydro  ...                          {'CARMA': ['CARMA47689']}                                                 []
7129                                            Vanelle    Hydro  ...                          {'CARMA': ['CARMA23918']}                                                 []
7130                                                Var    Hydro  ...            {'CARMA': ['CARMA24774', 'CARMA24775']}                                                 []
7131                                            Vaufrey    Hydro  ...                          {'CARMA': ['CARMA47840']}                                                 []
7132                                    Vaulnaveys haut    Hydro  ...                          {'CARMA': ['CARMA47843']}                                                 []
7133                                         Vaxoncourt    Hydro  ...                          {'CARMA': ['CARMA47847']}                                                 []
7134                                            Vendets    Hydro  ...                          {'CARMA': ['CARMA47932']}                                                 []
7135                                            Venthon    Hydro  ...                          {'CARMA': ['CARMA47950']}                                                 []
7136                                             Vergne    Hydro  ...                          {'CARMA': ['CARMA24776']}                                                 []
7137                                              Verna    Hydro  ...                          {'CARMA': ['CARMA58915']}                                                 []
7138                                             Vernes    Hydro  ...                          {'CARMA': ['CARMA25032']}                                                 []
7139                                             Verney    Hydro  ...                          {'CARMA': ['CARMA24777']}                                                 []
7140                                          Versilhac    Hydro  ...                          {'CARMA': ['CARMA48021']}                                                 []
7141                                            Vesubie    Hydro  ...                          {'CARMA': ['CARMA38710']}                                                 []
7142                                          Veyrieres    Hydro  ...                          {'CARMA': ['CARMA35203']}                                                 []
7143                                             Veyton    Hydro  ...                          {'CARMA': ['CARMA48061']}                                                 []
7144                                             Vezins    Hydro  ...                          {'CARMA': ['CARMA48062']}                                                 []
7145                                                Via    Hydro  ...                          {'CARMA': ['CARMA57257']}                                                 []
7146                                            Villard    Hydro  ...                          {'CARMA': ['CARMA48276']}                                                 []
7147                              Villarodin mont cenis    Hydro  ...                          {'CARMA': ['CARMA48282']}                                                 []
7148                              Villefranche conflent    Hydro  ...                          {'CARMA': ['CARMA48291']}                                                 []
7149                                       Villejoubert    Hydro  ...                          {'CARMA': ['CARMA48293']}                                                 []
7150                                        Villelongue    Hydro  ...                          {'CARMA': ['CARMA48294']}                                                 []
7151                                         Villey sec    Hydro  ...                          {'CARMA': ['CARMA48301']}                                                 []
7152                                      Villoevocance    Hydro  ...                          {'CARMA': ['CARMA48307']}                                                 []
7153                                         Vinon recy    Hydro  ...                          {'CARMA': ['CARMA48346']}                                                 []
7154                                            Vintrou    Hydro  ...                          {'CARMA': ['CARMA48349']}                                                 []
7155                                             Violle    Hydro  ...                          {'CARMA': ['CARMA23945']}                                                 []
7156                                            Vizille    Hydro  ...            {'CARMA': ['CARMA33965', 'CARMA48438']}                                                 []
7157                                            Hodonin  Lignite  ...                              {'GEO': ['GEO42325']}                                                 []

[7158 rows x 16 columns]

coroa · 2019-07-16T13:27:48Z

Couple of comments:

This conversation should move to a new release PR.
@pandas.api.extensions.register_dataframe_accessor("powerplant") is neat, i didn't know
It would be better to move code from __init__.py into core.py or common.py to avoid juggling with those cyclic dependencies as in f.ex. utils.py:
```
def get_obj_if_Acc(obj):
    from . import PowerPlantAccessor
    [...]
```
Windows doesn't meet the XDG specs. I think it's best to use the env-var APPDATA then.
_data_dir = os.path.join(os.environ.get("XDG_DATA_HOME", os.environ.get("APPDATA", _writable_dir)), 'powerplantmatching')

FabianHofmann · 2019-07-16T15:28:01Z

@fneum thank you for the test!

@coroa thanks for your points. I will add a new PR. And you're right, the cyclic dependencies are annoying and cause high import times ~600ms. I try to set the imports in a more efficient manner.

Ludee · 2019-12-17T13:50:00Z

Hello dear friends,
I just finished a new data download of the MaStR database. The data quality is improving slowly.
Now I am facing some problems in the wind dataset. There are around 10000 double entries I would like to find. Did someone already tried MaStR and PPM?

FabianHofmann · 2019-12-17T14:09:40Z

Hey @Ludee, nice to hear that. I didn't not take any look at it, but as far as I know a colleague of @fneum is working on a similar thing? I think she already formatted the data according to ppm.

FabianHofmann · 2019-12-17T14:13:52Z

I'm closing this in favour of #16 (due to mixed issues here)

fneum changed the title ~~Append BNetzA-MaStR renewables to~~ Append BNetzA-MaStR renewables to power plants Jun 18, 2019

FabianHofmann mentioned this issue Dec 17, 2019

Introduce/Include MaSTR data #16

Closed

FabianHofmann closed this as completed Dec 17, 2019

fneum mentioned this issue Jul 12, 2021

Possible bug: where is Wind in Italy? #47

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Append BNetzA-MaStR renewables to power plants #11

Append BNetzA-MaStR renewables to power plants #11

fneum commented Jun 18, 2019 •

edited

Loading

FabianHofmann commented Jul 9, 2019

fneum commented Jul 9, 2019

FabianHofmann commented Jul 9, 2019

coroa commented Jul 10, 2019

FabianHofmann commented Jul 16, 2019

fneum commented Jul 16, 2019

coroa commented Jul 16, 2019

FabianHofmann commented Jul 16, 2019

Ludee commented Dec 17, 2019

FabianHofmann commented Dec 17, 2019

FabianHofmann commented Dec 17, 2019

Append BNetzA-MaStR renewables to power plants #11

Append BNetzA-MaStR renewables to power plants #11

Comments

fneum commented Jun 18, 2019 • edited Loading

FabianHofmann commented Jul 9, 2019

fneum commented Jul 9, 2019

FabianHofmann commented Jul 9, 2019

coroa commented Jul 10, 2019

FabianHofmann commented Jul 16, 2019

fneum commented Jul 16, 2019

coroa commented Jul 16, 2019

FabianHofmann commented Jul 16, 2019

Ludee commented Dec 17, 2019

FabianHofmann commented Dec 17, 2019

FabianHofmann commented Dec 17, 2019

fneum commented Jun 18, 2019 •

edited

Loading