Skip to content

Export datasets to ETSource

Jump to bottom
kndehaan edited this page Oct 23, 2024 · 13 revisions

To export a dataset from ETLocal to the Energy Transition Model (ETSource):

  1. Make sure your ETLocal .env file contains the correct export root. This either has to be EXPORT_ROOT=localhost:5000 (or different port number depending on your local configuration) if you want to export a dataset from your local database or EXPORT_ROOT=data.energytransitionmodel.com.
  2. If you want to export a dataset from your local database, make sure you start your rails server by entering rails server -p 5000 (or different port number depending on your local configuration) in your terminal (ETLocal folder).
  3. Write down the unique ID of the dataset(s) you're interested in. The unique ID can be found by opening a dataset through the ETLocal interface. See screenshot below. You can open ETLocal on your local machine by typing http://localhost:5000/ in your browser.
  4. In your terminal (ETLocal folder) run rake etsource:export DATASET=<ID1>,<ID2>,<ID3>,....
  5. The dataset(s) will now appear in your local ETSource folder. Test your dataset(s) and make a pull request on ETSource.

Remarks:

  • If your dataset on ETLocal contains empty input fields the exporter may return an error.
  • Each ETLocal region has a parent dataset. This parent dataset is used when exporting an ETLocal dataset to ETSource to scale certain values, add information not in ETLocal (such as curve data, real estate data etc.) and to run certain sparse graph queries. For each ETLocal dataset, the parent dataset is derived from the region's country (specified in the ETLocal database) and the dataset's year. E.g. a Dutch region with a 2017 dataset on ETLocal will have nl2017 as parent dataset. The export will show which parent set it is using. If the parent dataset does not exist on ETSource, the export will default to using the sets country, in this case nl. You can create a missing parent dataset with the mimick_full_dataset rake task on ETSource.
  • If you would like to use custom curves on ETSource (e.g. region specific solar curves), you can simply overwrite the symlinked curves on ETSource with curves of your choice. When exporting the dataset again from ETLocal (e.g. to update some energy data), the custom curves will not be overwritten.

Below the command of step 4 is provided all international regions, provinces and RES regions, and municipalities (as of June 2024). Note to change the port if necessary.

International regions

EXPORT_ROOT=http://localhost:5000/ bundle exec rake etsource:export DATASET=BEGM35013,UKNI01,BEGM13040,SK,AT,BE,BG,CY,CZ,DE,DK,ES,EE,FI,FR,UK,EL,HR,HU,IE,IT,LT,LU,LV,PL,PT,RO,SE,SI,MT,GB,CH,NO,RS,EU27,SG,DKGM751,VUNH5,BEGM11002

Provinces and RES regions

EXPORT_ROOT=http://localhost:5000/ bundle exec rake etsource:export DATASET=PV24,PV27,PV22,PV29,PV20,PV26,PV23,PV21,PV25,PV28,PV30,PV31,RES16,RES17,RES22,RES01,RES02,RES03,RES04,RES05,RES06,RES11,RES12,RES13,RES14,RES18,RES19,RES20,RES21,RES23,RES24,RES25,RES26,RES27,RES28,RES29,RES30,RES31,RES07,RES08,RES09,RES10

Municipalities

EXPORT_ROOT=http://localhost:5000/ bundle exec rake etsource:export DATASET=GM1680,GM0358,GM0197,GM0059,GM0482,GM0613,GM0361,GM0141,GM0034,GM0484,GM1723,GM0060,GM0307,GM0362,GM0363,GM0200,GM0202,GM0744,GM0308,GM0489,GM0203,GM0370,GM1945,GM1724,GM0373,GM0748,GM1859,GM0209,GM0375,GM1728,GM0376,GM0377,GM1901,GM0755,GM1681,GM0147,GM0654,GM0758,GM0501,GM1876,GM0213,GM0312,GM0313,GM0214,GM0502,GM0383,GM0216,GM0148,GM1891,GM0310,GM1940,GM0736,GM1690,GM0503,GM0518,GM0400,GM0762,GM0150,GM0384,GM1774,GM0221,GM0222,GM0766,GM0505,GM0498,GM1719,GM0303,GM0225,GM0226,GM0385,GM0228,GM0317,GM0230,GM0388,GM0153,GM0232,GM0233,GM0777,GM0779,GM0784,GM1924,GM0664,GM0785,GM1942,GM0512,GM0513,GM0014,GM0158,GM0392,GM0394,GM1655,GM0160,GM0243,GM0523,GM0072,GM0244,GM0396,GM0397,GM0246,GM0074,GM0398,GM0399,GM0163,GM0530,GM0531,GM0164,GM0252,GM0797,GM0534,GM0798,GM0402,GM1735,GM1911,GM0405,GM0321,GM0406,GM0677,GM0353,GM1884,GM0166,GM0678,GM0537,GM1598,GM0542,GM1931,GM0415,GM0416,GM1621,GM0417,GM0080,GM0546,GM0547,GM1916,GM0995,GM0327,GM1705,GM0553,GM0262,GM0809,GM0331,GM0168,GM0263,GM0556,GM0420,GM0687,GM1842,GM1731,GM1709,GM1955,GM0335,GM1740,GM0356,GM0569,GM0267,GM0268,GM1930,GM1695,GM1699,GM0171,GM0575,GM0302,GM0579,GM0824,GM1895,GM0269,GM0173,GM1773,GM0175,GM1586,GM0826,GM0085,GM0431,GM0432,GM1509,GM0437,GM0589,GM1734,GM0590,GM0765,GM1926,GM0439,GM0273,GM0177,GM0703,GM0274,GM0339,GM0275,GM0340,GM0597,GM1742,GM0603,GM1674,GM0599,GM0277,GM0840,GM0441,GM0279,GM0606,GM0088,GM1676,GM0796,GM0610,GM1714,GM0090,GM0342,GM0180,GM0532,GM0851,GM1708,GM1904,GM0715,GM0093,GM0448,GM1525,GM0716,GM0281,GM0855,GM0183,GM1700,GM1730,GM0737,GM0450,GM0451,GM0184,GM0344,GM1581,GM0858,GM0345,GM0717,GM0861,GM0453,GM0622,GM0096,GM0718,GM0626,GM0285,GM0867,GM0627,GM0289,GM0629,GM0852,GM0457,GM0668,GM1701,GM0293,GM1783,GM0098,GM0614,GM0189,GM0296,GM1696,GM0352,GM0294,GM0632,GM0880,GM0351,GM0479,GM0297,GM0473,GM0050,GM0355,GM0299,GM0637,GM0638,GM1892,GM0301,GM1896,GM0642,GM0193,GM1948,GM1950,GM1952,GM1961,GM1960,GM1966,GM1959,GM1963,GM1978,GM1970,GM0037,GM0047,GM0086,GM0106,GM0109,GM0114,GM0118,GM0119,GM0743,GM0753,GM0756,GM0757,GM0770,GM0772,GM0786,GM0794,GM0815,GM0820,GM0823,GM0828,GM0845,GM0847,GM0848,GM0856,GM0865,GM0866,GM0873,GM0879,GM0882,GM0888,GM0889,GM0893,GM0899,GM0907,GM0917,GM0928,GM0935,GM0938,GM0944,GM0946,GM0957,GM0965,GM0971,GM0981,GM0983,GM0984,GM0986,GM0988,GM0994,GM1507,GM1640,GM1641,GM1652,GM1658,GM1659,GM1667,GM1669,GM1684,GM1685,GM1702,GM1706,GM1711,GM1721,GM1729,GM1771,GM1883,GM1894,GM1900,GM1903,GM1949,GM1954,GM1969,GM1979,GM0345HA

Clone this wiki locally