This is the source code for the Calculation Engine that is used by the Energy Transition Model and its various interfaces (clients).
It is an online web app that lets you create a future energy scenario for various countries. This software is open source, so you can fork it and alter at your will.
ETEngine does not contain an easy-to-use frontend for creating and editing these energy scenarios; that role is instead fulfilled by separate applications such as ETModel, ETFlex, and the EnergyMixer, which each use ETEngine's REST API for manipulating and calculating scenarios.
The ETE is released under the MIT License.
Installing ETEngine on a local machine can be a bit involved, owing to the number of dependencies. Fortunately, most can be installed automatically using Ruby's "Bundler" library, which is required to use ETEngine.
-
Install the "Graphviz" library; for Mac users with Homebrew, this is as simple as
brew install graphviz. -
Run
bundle installto install the dependencies required by ETEngine. -
Copy "config/config.sample.yml" to "config/config.yml", and "config/database.sample.yml" to "config/database.yml", making any changes -- particularly to the database configuration -- as you see fit.
-
Clone a copy of ETSource –– which contains the data for each region. Edit "config/config.yml" and enter the ETSource directory into the "etsource_export" and "etsource_working_copy" options.
-
Create the database you specified in your "database.yml" file, and run
bundle exec rake db:setup db:seedto create the tables and add an administrator account. -
You're now ready-to-go! Fire up the Rails process with
bundle exec rails sor use Pow.
The ETEngine uses heavily caching of calculated values by using the fetch function that stores and retrieves calculated values. This has some drawbacks, but is necessary to keep performance up.
When the user starts a new scenario, the user has to choose the end_year
and the area for which this scenario applies. This can/should not be
altered later.
The ETEngine uses two graphs that store all the data: one for the present year and one for the future year. In this sense, the ETengine is a 'two state' model: everything is calculated twice: once for the start year, and once for the end year. It is important to note that ETengine therefor does not calculate intermediate years. An exception to this is Merit, a module for ETengine (that can also be used independently which contains time series at a one hour resolution for one year.
A user can alter the start scenario with the use of inputs. Every input has a key and a value can be sent to ETEngine. For example a user can tell ETengine:
number_of_energy_power_nuclear_gen3_uranium_oxide = 2
This means that the user wants to 'set' the number of nuclear power plants to 2
in his/her current scenario.
The current set of inputs can be found on ETSource.
Every times the user requests some output, all the inputs that have been touched by that user for that scenario are applied again. The order in which they are applied can be controlled if necessary.
The priority of every input defaults to 0, and can be set a manual value
(e.g. 100) on inputs which need to be executed first. For example, an input
with priority=100 gets executed before an input with priority=99, etc...
This is someting to keep in mind when designing your input statements.
For example, when you have two inputs:
- input
A: update attributeXto have value1 - input
B: update attributeXto have value2
The outcome of this X will be 1 or 2 depending on the priority of
these inputs (if they both have no priority or the same priority), this will
be randomly determined.
For example, when you have two inputs:
- input
A: update attributeXto increase with1% - input
B: update attributeXto increase with2%
Then the outcome of the X will be 1.01 * 1.02.
The user can request output from his/her scenario with the use of gqueries. A gquery always returns the present and the future output value, although there are exceptions to this.
E.g. when the user sends the dashboard_co2_emissions query to
ETEngine, it will receive the following feedback:
- present: 123
- future: 456
- unit: MJ
A gquery is nothing more then a stored statement. These statements are written in our own language called the Graph Query Language (GQL) and a recent list can be found on ETSource.
Assuming you can run a 'normal' rails application on your local machine, you have to follow these steps to run ET-Engine.
bundle installto install dependencies if you haven't done alreadybundle exec rake db:createto create your databasebundle exec cap staging db2localto fill your database with records from staging server- create a directory to clone etsource into, I advise to use
a dedicated etsource copy in the etengine root folder:
git clone [email protected]:quintel/etsource.git - create an empty directory as a working copy for your etsource directory
mkdir etsource_export cd configcp config.sample.yml config.yml; cp database.sample.yml database.yml- open up these two files in your favorite text-editor and fill in the details of the directories you just created. You can leave the defaults as they are, unless you want something else
- make sure you have memcached running (using
memcached -d) - fire up your local rails server (use
rails son the console or use pow - go to
http://etengine.dev/etsourceor equivalent - (if you run into an dataset error, check out this explanation on CSV files)
- Press 'import' (the latest commit or another one if you like that better)
- you're done!
Sometimes you want to play around or tweak some gqueries. Then, you don't want to create commits every time and import them. Because when you are satisfied, you'll probably have 10 commits, that needs to be cleaned up, squashed.
You can add the option etsource_live_reload: true in your config.yml
file.
Change queries, inputs, datasets, gqueries, inputs or topology directory in your et_source_export folder, and Etengine reloads your changes automatically!
B.t.w. By default your etsource_export directory is not under version control. In order to gain the advantages of Git, just point etsource_export to the etsource directory, either by using a symbolic link or using the same directory in your config.yml file. But be carefull NOT to use the interface's 'import' action on /etsource: that will delete/overwrite your etsource_export directory!
Password for all the screencasts below is quintel.
How to use this documentation.
How to work with different etsource directories, make changes and load them in the gql console.
We build a new etmodel with 3 converters from scratch. This helps you understand how the etsource works.
The result you can find in: etsource/models/sample