diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md index f9ba8cf..784c0ba 100644 --- a/CODE_OF_CONDUCT.md +++ b/CODE_OF_CONDUCT.md @@ -1,9 +1,69 @@ -# Microsoft Open Source Code of Conduct +# Code of Conduct - xl2times -This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). +## Our Pledge -Resources: +In the interest of fostering an open and welcoming environment, we as +contributors and maintainers pledge to make participation in our project and +our community a harassment-free experience for everyone, regardless of age, body +size, disability, ethnicity, sex characteristics, gender identity and expression, +level of experience, education, socio-economic status, nationality, personal +appearance, race, religion, or sexual identity and orientation. -- [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/) -- [Microsoft Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) -- Contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with questions or concerns +## Our Standards + +Examples of behaviour that contributes to a positive environment for our +community include: + +* Demonstrating empathy and kindness toward other people +* Being respectful of differing opinions, viewpoints, and experiences +* Giving and gracefully accepting constructive feedback +* Accepting responsibility and apologising to those affected by our mistakes, + and learning from the experience +* Focusing on what is best not just for us as individuals, but for the + overall community + +Examples of unacceptable behaviour include: + +* The use of sexualised language or imagery, and sexual attention or advances +* Trolling, insulting or derogatory comments, and personal or political attacks +* Public or private harassment +* Publishing others' private information, such as a physical or email + address, without their explicit permission +* Other conduct which could reasonably be considered inappropriate in a + professional setting + +## Our Responsibilities + +Project maintainers are responsible for clarifying and enforcing our standards of +acceptable behaviour and will take appropriate and fair corrective action in +response to any instances of unacceptable behaviour. + +Project maintainers have the right and responsibility to remove, edit, or reject +comments, commits, code, wiki edits, issues, and other contributions that are +not aligned to this Code of Conduct, or to ban +temporarily or permanently any contributor for other behaviours that they deem +inappropriate, threatening, offensive, or harmful. + +## Scope + +This Code of Conduct applies within all community spaces, and also applies when +an individual is officially representing the community in public spaces. +Examples of representing our community include using an official e-mail address, +posting via an official social media account, or acting as an appointed +representative at an online or offline event. + +## Enforcement + +Instances of abusive, harassing, or otherwise unacceptable behaviour may be +reported to the community leaders responsible for enforcement at . +All complaints will be reviewed and investigated promptly and fairly. + +All community leaders are obligated to respect the privacy and security of the +reporter of any incident. + +## Attribution + +This Code of Conduct is adapted from the [Contributor Covenant](https://contributor-covenant.org/), version +[1.4](https://www.contributor-covenant.org/version/1/4/code-of-conduct/code_of_conduct.md) and +[2.0](https://www.contributor-covenant.org/version/2/0/code_of_conduct/code_of_conduct.md), +and was generated by [contributing-gen](https://github.com/bttger/contributing-gen). diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..be520f5 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,159 @@ + +# Contributing to xl2times + +First off, thanks for taking the time to contribute! ❤️ + +All types of contributions are encouraged and valued. See the [Table of Contents](#table-of-contents) for different ways to help and details about how this project handles them. Please make sure to read the relevant section before making your contribution. It will make it a lot easier for us maintainers and smooth out the experience for all involved. The community looks forward to your contributions. 🎉 + +> And if you like the project, but just don't have time to contribute, that's fine. There are other easy ways to support the project and show your appreciation, which we would also be very happy about: +> - Star the project +> - Tweet about it +> - Refer this project in your project's readme +> - Mention the project at local meetups and tell your friends/colleagues + + +## Table of Contents + +- [Code of Conduct](#code-of-conduct) +- [I Have a Question](#i-have-a-question) + - [I Want To Contribute](#i-want-to-contribute) + - [Reporting Bugs](#reporting-bugs) + - [Suggesting Enhancements](#suggesting-enhancements) + - [Your First Code Contribution](#your-first-code-contribution) + - [Improving The Documentation](#improving-the-documentation) +- [Styleguides](#styleguides) + - [Commit Messages](#commit-messages) +- [Join The Project Team](#join-the-project-team) + + +## Code of Conduct + +This project and everyone participating in it is governed by the +[xl2times Code of Conduct](https://github.com/etsap-TIMES/xl2times/blob//CODE_OF_CONDUCT.md). +By participating, you are expected to uphold this code. Please report unacceptable behavior +to . + + +## I Have a Question + +> If you want to ask a question, we assume that you have read the available [Documentation](https://xl2times.readthedocs.io/). + +Before you ask a question, it is best to search for existing [Issues](https://github.com/etsap-TIMES/xl2times/issues) that might help you. In case you have found a suitable issue and still need clarification, you can write your question in this issue. It is also advisable to search the internet for answers first. + +If you then still feel the need to ask a question and need clarification, we recommend the following: + +- Open an [Issue](https://github.com/etsap-TIMES/xl2times/issues/new). +- Provide as much context as you can about what you're running into. +- Provide project and platform versions (at least: OS, python), depending on what seems relevant. + +We will then take care of the issue as soon as possible. + + + +## I Want To Contribute + +> ### Legal Notice +> When contributing to this project, you must agree that you have authored 100% of the content, that you have the necessary rights to the content and that the content you contribute may be provided under the project licence. + +### Reporting Bugs + + +#### Before Submitting a Bug Report + +A good bug report shouldn't leave others needing to chase you up for more information. Therefore, we ask you to investigate carefully, collect information and describe the issue in detail in your report. Please complete the following steps in advance to help us fix any potential bug as fast as possible. + +- Make sure that you are using the latest version. +- Determine if your bug is really a bug and not an error on your side e.g. using incompatible environment components/versions (Make sure that you have read the [documentation](https://xl2times.readthedocs.io/). If you are looking for support, you might want to check [this section](#i-have-a-question)). +- To see if other users have experienced (and potentially already solved) the same issue you are having, check if there is not already a bug report existing for your bug or error in the [bug tracker](https://github.com/etsap-TIMES/xl2times/issues?q=label%3Abug). +- Also make sure to search the internet (including Stack Overflow) to see if users outside of the GitHub community have discussed the issue. +- Collect information about the bug: + - Stack trace (Traceback) + - OS, Platform and Version (Windows, Linux, macOS, x86, ARM) + - Version of the interpreter, compiler, SDK, runtime environment, package manager, depending on what seems relevant. + - Possibly your input and the output + - Can you reliably reproduce the issue? And can you also reproduce it with older versions? + + +#### How Do I Submit a Good Bug Report? + +> You must never report security related issues, vulnerabilities or bugs including sensitive information to the issue tracker, or elsewhere in public. Instead sensitive bugs must be sent by email to . + + +We use GitHub issues to track bugs and errors. If you run into an issue with the project: + +- Open an [Issue](https://github.com/etsap-TIMES/xl2times/issues/new). (Since we can't be sure at this point whether it is a bug or not, we ask you not to talk about a bug yet and not to label the issue.) +- Explain the behavior you would expect and the actual behavior. +- Please provide as much context as possible and describe the *reproduction steps* that someone else can follow to recreate the issue on their own. This usually includes your code. For good bug reports you should isolate the problem and create a reduced test case. +- Provide the information you collected in the previous section. + +Once it's filed: + +- The project team will label the issue accordingly. +- A team member will try to reproduce the issue with your provided steps. If there are no reproduction steps or no obvious way to reproduce the issue, the team will ask you for those steps and mark the issue as `needs-repro`. Bugs with the `needs-repro` tag will not be addressed until they are reproduced. +- If the team is able to reproduce the issue, it will be marked `needs-fix`, as well as possibly other tags (such as `critical`), and the issue will be left to be [implemented by someone](#your-first-code-contribution). + + + + +### Suggesting Enhancements + +This section guides you through submitting an enhancement suggestion for xl2times, **including completely new features and minor improvements to existing functionality**. Following these guidelines will help maintainers and the community to understand your suggestion and find related suggestions. + + +#### Before Submitting an Enhancement + +- Make sure that you are using the latest version. +- Read the [documentation](https://xl2times.readthedocs.io/) carefully and find out if the functionality is already covered, maybe by an individual configuration. +- Perform a [search](https://github.com/etsap-TIMES/xl2times/issues) to see if the enhancement has already been suggested. If it has, add a comment to the existing issue instead of opening a new one. +- Find out whether your idea fits with the scope and aims of the project. It's up to you to make a strong case to convince the project's developers of the merits of this feature. Keep in mind that we want features that will be useful to the majority of our users and not just a small subset. If you're just targeting a minority of users, consider writing an add-on/plugin library. + + +#### How Do I Submit a Good Enhancement Suggestion? + +Enhancement suggestions are tracked as [GitHub issues](https://github.com/etsap-TIMES/xl2times/issues). + +- Use a **clear and descriptive title** for the issue to identify the suggestion. +- Provide a **step-by-step description of the suggested enhancement** in as many details as possible. +- **Describe the current behavior** and **explain which behavior you expected to see instead** and why. At this point you can also tell which alternatives do not work for you. +- You may want to **include screenshots or screen recordings** which help you demonstrate the steps or point out the part which the suggestion is related to. You can use [LICEcap](https://www.cockos.com/licecap/) to record GIFs on macOS and Windows, and the built-in [screen recorder in GNOME](https://help.gnome.org/users/gnome-help/stable/screen-shot-record.html.en) or [SimpleScreenRecorder](https://github.com/MaartenBaert/ssr) on Linux. +- **Explain why this enhancement would be useful** to most xl2times users. You may also want to point out the other projects that solved it better and which could serve as inspiration. + + + +### Your First Code Contribution + + +### Improving The Documentation + + +## Styleguides +### Commit Messages + + +## Join The Project Team + + + +## Attribution +This guide is based on the **contributing-gen**. [Make your own](https://github.com/bttger/contributing-gen)! diff --git a/LICENSE b/LICENSE index 9e841e7..c8b1859 100644 --- a/LICENSE +++ b/LICENSE @@ -1,21 +1,22 @@ - MIT License +MIT License - Copyright (c) Microsoft Corporation. +Copyright (c) 2022-2023 Microsoft Corporation. +Copyright (c) 2023-2024 IEA Energy Technology Systems Analysis Programme. - Permission is hereby granted, free of charge, to any person obtaining a copy - of this software and associated documentation files (the "Software"), to deal - in the Software without restriction, including without limitation the rights - to use, copy, modify, merge, publish, distribute, sublicense, and/or sell - copies of the Software, and to permit persons to whom the Software is - furnished to do so, subject to the following conditions: +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: - The above copyright notice and this permission notice shall be included in all - copies or substantial portions of the Software. +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. - THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR - IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, - FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE - AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER - LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, - OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE - SOFTWARE +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE diff --git a/README.md b/README.md index ec24356..acd5119 100644 --- a/README.md +++ b/README.md @@ -1,13 +1,14 @@ # xl2times -**Note: this tool is a work in progress and not yet in a useful state** +`xl2times` is an open source tool to convert TIMES models specified in Excel to a format ready for processing by [GAMS](https://www.gams.com/). +Development of the tool originally started in a Microsoft [repository](https://github.com/microsoft/times-excel-reader) with an intention to make it easier for anyone to reproduce research results on TIMES models. -`xl2times` is an open source tool to convert TIMES model Excel input files to DD format ready for processing by [GAMS](https://www.gams.com/). The intention is to make it easier for people to reproduce research results on TIMES models. - -[TIMES](https://iea-etsap.org/index.php/etsap-tools/model-generators/times) is an energy systems model generator from the International Energy Agency that is used around the world to inform energy policy. +[TIMES](https://github.com/etsap-TIMES/TIMES_model) is an open source energy systems model generator developed by the [Energy Technology Systems Analysis Program](https://iea-etsap.org/) (ETSAP) of the [International Energy Agency](https://www.iea.org/) (IEA) that is used around the world to inform energy policy. It is fully explained in the [TIMES Model Documentation](https://iea-etsap.org/index.php/documentation). -The Excel input format accepted by this tool is documented in the [TIMES Model Documentation PART IV](https://iea-etsap.org/docs/Documentation_for_the_TIMES_Model-Part-IV.pdf). Additional table types are documented in the [VEDA support forum](https://forum.kanors-emr.org/printthread.php?tid=140). Example inputs are available at https://github.com/kanors-emr/Model_Demo_Adv_Veda +Multiple approaches to using spreadsheets for specifying TIMES models have been developed, e.g. [ANSWER-TIMES](https://iea-etsap.org/index.php/etsap-tools/data-handling-shells/answer) and [VEDA-TIMES](https://iea-etsap.org/index.php/etsap-tools/data-handling-shells/veda). +At present, `xl2times` implements partial support of the Veda approach described in the [TIMES Model Documentation PART IV](https://iea-etsap.org/docs/Documentation_for_the_TIMES_Model-Part-IV.pdf) and [Veda Documentation](https://veda-documentation.readthedocs.io/en/latest/pages/VedaTags.html). +Support of other approaches may be added over time. ## Installation and Basic Usage @@ -78,8 +79,8 @@ you'll have to request access) - if not, comment out the inaccessible benchmarks ```bash mkdir benchmarks -# Get VEDA example models and reference DD files -# XLSX files are in private repo for licensing reasons, please request access or replace with your own licensed VEDA example files. +# Get TIMES DemoS example models and reference DD files +# XLSX files are in private repo for licensing reasons, please request access or replace with your own files distributed with Veda. git clone git@github.com:olejandro/demos-xlsx.git benchmarks/xlsx/ git clone git@github.com:olejandro/demos-dd.git benchmarks/dd/ @@ -137,25 +138,6 @@ python -m build python -m twine upload dist/* ``` - ## Contributing -This project welcomes contributions and suggestions. Most contributions require you to agree to a -Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us -the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com. - -When you submit a pull request, a CLA bot will automatically determine whether you need to provide -a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions -provided by the bot. You will only need to do this once across all repos using our CLA. - -This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). -For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or -contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments. - -## Trademarks - -This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft -trademarks or logos is subject to and must follow -[Microsoft's Trademark & Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general). -Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. -Any use of third-party trademarks or logos are subject to those third-party's policies. +This project welcomes contributions and suggestions. See [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md) and [CONTRIBUTING.md](CONTRIBUTING.md) for more details. diff --git a/SECURITY.md b/SECURITY.md deleted file mode 100644 index a050f36..0000000 --- a/SECURITY.md +++ /dev/null @@ -1,41 +0,0 @@ - - -## Security - -Microsoft takes the security of our software products and services seriously, which includes all source code repositories managed through our GitHub organizations, which include [Microsoft](https://github.com/Microsoft), [Azure](https://github.com/Azure), [DotNet](https://github.com/dotnet), [AspNet](https://github.com/aspnet), [Xamarin](https://github.com/xamarin), and [our GitHub organizations](https://opensource.microsoft.com/). - -If you believe you have found a security vulnerability in any Microsoft-owned repository that meets [Microsoft's definition of a security vulnerability](https://docs.microsoft.com/en-us/previous-versions/tn-archive/cc751383(v=technet.10)), please report it to us as described below. - -## Reporting Security Issues - -**Please do not report security vulnerabilities through public GitHub issues.** - -Instead, please report them to the Microsoft Security Response Center (MSRC) at [https://msrc.microsoft.com/create-report](https://msrc.microsoft.com/create-report). - -If you prefer to submit without logging in, send email to [secure@microsoft.com](mailto:secure@microsoft.com). If possible, encrypt your message with our PGP key; please download it from the [Microsoft Security Response Center PGP Key page](https://www.microsoft.com/en-us/msrc/pgp-key-msrc). - -You should receive a response within 24 hours. If for some reason you do not, please follow up via email to ensure we received your original message. Additional information can be found at [microsoft.com/msrc](https://www.microsoft.com/msrc). - -Please include the requested information listed below (as much as you can provide) to help us better understand the nature and scope of the possible issue: - - * Type of issue (e.g. buffer overflow, SQL injection, cross-site scripting, etc.) - * Full paths of source file(s) related to the manifestation of the issue - * The location of the affected source code (tag/branch/commit or direct URL) - * Any special configuration required to reproduce the issue - * Step-by-step instructions to reproduce the issue - * Proof-of-concept or exploit code (if possible) - * Impact of the issue, including how an attacker might exploit the issue - -This information will help us triage your report more quickly. - -If you are reporting for a bug bounty, more complete reports can contribute to a higher bounty award. Please visit our [Microsoft Bug Bounty Program](https://microsoft.com/msrc/bounty) page for more details about our active programs. - -## Preferred Languages - -We prefer all communications to be in English. - -## Policy - -Microsoft follows the principle of [Coordinated Vulnerability Disclosure](https://www.microsoft.com/en-us/msrc/cvd). - - diff --git a/SUPPORT.md b/SUPPORT.md index 8d7be6a..0935766 100644 --- a/SUPPORT.md +++ b/SUPPORT.md @@ -5,7 +5,3 @@ This project uses GitHub Issues to track bugs and feature requests. Please search the existing issues before filing new issues to avoid duplicates. For new issues, file your bug or feature request as a new Issue. - -## Microsoft Support Policy - -Support for this **PROJECT or PRODUCT** is limited to the resources listed above. diff --git a/TODO.md b/TODO.md deleted file mode 100644 index 2f57f12..0000000 --- a/TODO.md +++ /dev/null @@ -1,248 +0,0 @@ -## High level TODO *(rough priority/dependency order)* - -- Generate all output rows that are in ground truth ~~(~40% complete currently)~~ - - See CI for latest status - - ~~See https://github.com/microsoft/times-excel-reader/wiki/State-of-Progress~~ -- Decide how we should manage work list and assignments - TODO file, Wiki, Issues, Milestones -- Write an overview of how code currently works to help get new people started -- Go over old TODO lists below and work out what is still relevant -- ~~Code formatting~~ -- Integrate type checker into CI (mypy?) -- ~~Decide at what point to reach out to TIMES community and announce project (and ask for help?)~~ -- ~~Figure out how to handle columns that start with * but are not comments (manual isn't clear on this)~~ -- ~~Decide what to do about \~TFM_FILL. These tables look like the Excel files are meant to be updated by the tool~~ -- Create a glossary? -- Remove additional rows in output that are not in ground truth -- ~~Test on other data sets (using CI?)~~ -- Split into scenarios (we are currently just lumping all the data together into one) -- Sort out row ordering (currently ignored) -- Generate (or convert to) DD files -- Measure difference between our generated DD files and ground truth -- Get difference between generated DD files and ground truth to zero -- Code tidying - -## Processing steps -- Read files included in the case / all model files if a subset not specified -- Normalise the tables (e.g. lower/upper case headers, etc.) -- Read settings in syssettings -- Fill in missing values that are independent from info in other files (e.g. process name in FI_T, sets in FI_Process, default year, etc.). -- Process model files in groups: - - All the VT files + BY_TRANS together - - SubRES (incl. trans) one-by-one. Use info on commodities from above. - - Scenario files. Must be processed in the order specified / alphabetic order. Have access to all info specified before them (e.g. upd tables) - - -## OLD Mix of doc and detailed TODO list - -Steps to standardize a commodity table (~FI\_COMM) - -- [x] Extract table by finding tag and blank rows and columns -- [x] Remove comment rows -- [x] Remove table tag row (whilst remembering metadata like its location) -- [x] Convert to data frame -- [x] Remove comment columns and reduce tag\_indent accordingly (see last table in VT\_UK\_IND[IIS]) -- [x] Concatenate all commodity tables by row -- [x] Apply column-specific missing value handling - - [x] Csets: declarations are inherited until the next one is encountered - - [x] Region: By default, it is applied to all regions of the model when not specified - - [x] Get hold of regions from ~BookRegions\_Map, can use comma-separated list here -- [x] LimType: Do nothing, when not specified. TIMES uses LO for all but MAT commodities (Csets column) with a default of FX -- [x] CTSLvl: Do nothing, when not specified. TIMES uses ANNUAL by default. -- [x] PeakTS: Do nothing, when not specified. TIMES uses ANNUAL by default. -- [x] Duplicate rows with comma-separated values (Csets, Region, PeakTS), each row getting only one of the values. -- [ ] After above, check there are no comma-separated values -- [x] Remove rows with disallowed values - - [x] Csets must be one of: NRG (energy), MAT (material), DEM (demand service), ENV (emissions) and FIN (financial) -- [x] Split columns into separate tables, permute/rename/drop columns to match TIMES (using times\_mapping.txt) -- [x] Mark each table as set or parameter (based having a column named "VALUE") -- [ ] Put each table into the correct dd file (syssettings.dd, base.dd, etc). This is based on the name of the Excel file. - -
- -Notes: - -For now, ignore all validity constraints (assume input is valid) - -Try enumerating all possible "missing value handling rules" and mark all columns with one of them - -There are processes in the Excel files that are not mentioned in the dd files - -Row ordering must be preserved by these transformations, but only until the defaults are filled(?) - -
- -Extra steps for process tables: - -- [ ] Remove rows for unused processes i.e., if a process is missing from FT_Process table any data generated for it can be removed. This is has to be checked agains a specific subset of files (e.g. BASE or a SubRES) - -Missing-value handling of process tables (~FI\_PROCESS) - -- [x] Sets: Inherit values from previous row if missing -- [x] Region: all regions if missing -- [ ] Tslvl: default value depends on the value of Sets column -- [x] PrimaryCG: if missing, use Veda rules to determine default - - [x] Can only be done after the technology tables are available - - [ ] ~~Could append the output commodity column first~~ -- [x] Others have constant/fixed default values - -
- -Special handling for flexible import tables (~FI\_T) - -- [x] Create a column named "Attribute" if it does not yet exist - - [x] Same for Year, TimeSlice, LimType, Region, Currency -- [x] If the table tag has a table-level declaration (e.g., ~FI\_T:DEMAND): - - [x] Set every missing value for the Attribute column to the tag value - - [x] Example: ~FI\_T: COM\_PKRSV - - [x] Set every missing value of Attribute to "COM\_PKRSV" -- [x] ~~Add two new data columns~~: - - [x] ~~TOP-IN whose value is "IN"~~ - - [x] ~~TOP-OUT whose value is "OUT"~~ -- [] ~~Use the column position of the table tag to identify the data columns (columns to the right of the tag) for the next step~~ -- [x] For each row, collect all data columns and transpose them to get a single column with multiple rows. The new column is named "VALUE". - - [x] Copy the existing non-data columns across the new rows - - [x] The original column names are now Attribute values. If an Attribute value already exists, append the column name to it, with a tilde in the middle. Example: Attribute was DEMAND, column name was "2010", new Attribute is "DEMAND~2010". -- [x] Concatenate all technology tables by row (they should have the same columns at this point) -- [x] If any Attribute contains tilde (e.g., STOCK~2030): - - [ ] Determine what indices are being given (out of the six above plus Comm-OUT). A number is always Year. A non-number must be matched against the possible values for each index. - - [ ] If the table column for any index does not yet exist, create it - - [x] For each row, move the index string from the Attribute to the index column - - [x] Example: Attribute is STOCK~2030 - - [x] The index is Year - - [x] New Attribute is "STOCK", Year=2030 - - [ ] Example: Attribute is FLOSHAR~UP - - [ ] The index is LimType - - [ ] New Attribute is "FLOSHAR", LimType="UP" - - [ ] Example: Attribute is FLOSHAR~UP~0 - - [ ] The indices are LimType and Year - - [ ] New Attribute is "FLOSHAR", LimType="UP", Year="0" (The data for year “0” are interpolation/extrapolation options) -- [ ] Missing-value handling after the above is done: - - [x] Region: all regions if missing - - [ ] Year: the base year if missing - - [x] TechName: inherited from above - - [ ] LimType: TODO (dd files have limits, called BD, but not sure how this is determined) -- [ ] If Attribute="FLOSHAR": - - [ ] ~~If Other\_Indexes is missing, set it to the value in COM\_GMAP(Comm-IN)~~ - - [ ] If Other\_Indexes is missing, determine default veda pcg based on the process name, commodity type and whether the commodity is an input or output of the process -- [ ] If Attribute="FLO\_EMIS": - - [ ] If Other\_Indexes is missing, set it to "ACT" -- [ ] If Attribute="EFF": - - [ ] ~~Change Comm-IN to "ACT"~~ - - [ ] ~~~Change Attribute to "CEFF"~~ - - [x] Set Other\_Indexes to "ACT" -- [ ] If Attribute="OUTPUT": - - [ ] Change Comm-IN to "Comm-OUT-A" - - [ ] Change Attribute to "CEFF" -- [ ] If Attribute="END": - - [ ] Change Year to the contents of VALUE + 1 - - [ ] Change Other\_Indexes to "EOH" - - [ ] Change Attribute to "PRC\_NOFF" -- [ ] ~~If Attribute="TOP-IN":~~ - - [ ] ~~Change Other\_Indexes to Comm-IN~~ - - [ ] ~~Change Attribute to "IO"~~ -- [ ] ~~If Attribute="TOP-OUT":~~ - - [ ] ~~Change Other\_Indexes to Comm-OUT~~ - - [ ] ~~Change Attribute to "IO"~~ -- [x] Split into separate tables based on Attribute - - [x] In the mapping file, look for ~FI\_T(…,Attribute) or Attribute(…) = ~FI\_T(…). In some cases, Attribute is the literal TIMES variable while in other cases it is an alias. For example, PRC\_RESID sometimes has Attribute="PRC\_RESID" and sometimes it has Attribute="STOCK" (comes with additional assumption and processing). -- [ ] ~~Note: \* can also be used to indicate a wildcard or an operation in some cells~~ - -
- -Special handling for transformation insert tables (~TFM\_INS): - -- Duplicate rows with comma-separated values - probably better not. Comma-separated values should be processed together in order to determine a subset of processes or commodities to which a particular row applies. -- Remove blank rows -- Create a column for every region in ~BookRegions\_Map, if it doesn't already exist -- If the table has an AllRegions column: - - For every region column, fill in missing values by copying from AllRegions -- For each row, collect all region columns and transpose them to get a single column with multiple rows. The new column is named "VALUE". - - [x] Copy all other columns across the new rows. - - [x] The original column names are now Region values for a new Region column. -- [x] Create a TechName column. For each row, replicate it across all TechNames that match the filters specified by the Pset columns. Pset\_PN is a name filter with \* as wildcard. -- [x] Create a Commodity column. For each row, replicate it across all commodities that match the filters specified by the Cset columns. - - [ ] This step can be skipped for attributes that don't involve commodities, but it won't hurt either way -- Create a Year, TimeSlice, etc. columns as for FI_T tables. -- [ ] Now treat it like a ~FI\_T table - -
- -Special handling for transformation update tables (~TFM\_UPD): -- Follow the steps for transformation insert tables, but note that values can be prefixed with an operator -- For each row, find a matching data in some ~FI\_T table or table generated by other TFM tables, and generate an additional table that reflects the update. -- This is easier to do after merging the FI\_T tables, but it does not necesserily apply to all the files (depends on the order in which files are specified). - -
- -Special handling for ~TFM\_COMGRP: - -- Follow the steps for ~TFM\_INS above -- Defines COM\_GMAP in syssettings.dd -- COM\_GMAP(REG,COM\_GRP,COM) = ~TFM\_COMGRP(Region,Name,Commodity) - -
- -Special handling for ~StartYear (in SysSettings.xls): - -- [x] This table always consists of a single cell. It has no column labels. -- [x] Change the column label to VALUE - -
- -Special handling for ~TimePeriods: - -- [x] This table should have a single column. Rename the column to "D". -- [x] Add a column named "B" whose first row is ~StartYear and each subsequent value is the previous value plus the previous D column. -- [x] Add a column named "E" defined as B + D - 1 -- [x] Add a column named "M" defined as round((B+E)/2) = B + round((D-1)/2) -- [x] Add a column named "Year" equal to M - -
- -These tables only involve renaming columns: - -~Currencies - -~ActivePDef - -~BookRegions\_Map - -
- -Special handling for ~TimeSlices (Review the exact implementation): - -- [x] This table is unusual in that it is not really a table, but a set of lists of varying length, corresponding to hierarchy levels -- [x] For each level after the first (Season), add a list that contains all ways of choosing one value from each of the previous levels and concatenating them into a single string. This list should be named ALL\_X, e.g. ALL\_DAYNITE. - - [x] For each such list, make a second list called PARENT\_X that removes the last level from the value -- [x] Add a list named TS\_GROUP that contains "ANNUAL", then the entire first list (Season), then each of the ALL\_X lists - - [x] At the same time, create a list of the same size named TSLVL that contains "ANNUAL", then "SEASON" repeated to the length of the first list, then X repeated to the length of ALL\_X -- [x] Add a list named TS\_MAP that concatenates all of the ALL\_X lists -- [x] Add a list named PARENT that concatenates all of the PARENT\_X lists -- [x] Make a table named TS\_MAP according to times\_mapping.txt -- [x] Make a table named TS\_GROUP according to times\_mapping.txt - -
- -Special handling for ~COMEMI: - -- For each row, collect all columns not in ("Region","Year","CommName") and transpose them to get a single column with multiple rows. The new column is named "EMCB". The old column names become a column named "Other\_Indexes". -- Create columns for Region and Year if they don't exist and apply the usual missing-value handling to them -- Make a table named VDA\_EMCB according to times\_mapping.txt - -
- -TODO: - -Special handling for ~UC\_Sets: - -
- -~DefUnits (seems to be unused) - -~ImpSettings - is not used by Veda2. These settings are in AppData. - -~TFM\_Csets - -~TFM\_Psets - -~COMAGG diff --git a/proposal.md b/proposal.md deleted file mode 100644 index 7da134a..0000000 --- a/proposal.md +++ /dev/null @@ -1,54 +0,0 @@ -# Improving Government/National Energy Modelling - -## Project Description - -Government policy relies on credible economic modelling of the required energy transition. But current tools make it difficult to build large-scale bottom-up economic models. This project will develop open-source software for energy-economy modelling at a much larger scale and realism than previously possible, through better optimization and data management. We will also provide tools to ensure the validity, transparency, and overall trustworthiness of a model. University College London (UCL) energy-economy models are used worldwide and frequently cited in UK government reports (https://www.ucl.ac.uk/drupal/site\_energy-models/policy-impacts-of-our-models, https://www.sciencedirect.com/science/article/pii/S0306261916301672). - -The free and open source energy-economic models we build will result in cheaper, more detailed, and more trustworthy data about the energy economy. This can be used by national governments to formulate more precise and ambitious strategies to decarbonize and achieve a sustainable future. The models we will develop can also be used to evaluate decarbonization progress and our project can increase the accuracy of carbon accounting. - -The project primarily assists decision makers, e.g. governments/legislatures, by providing them with more accurate and trustworthy predictions. We expect this to accelerate net zero commitments from countries. - -Energy-economic models were used to justify and evaluate UKs 2019 amendment to the Climate Change Act that, by committing to net zero, leads to a reduction of 160 MtC02e from the atmosphere in 2050 and beyond. This project will open source such models and enable other countries and corporations to adapt them and accelerate their transitions. - -Modelling allows decision makers (government or corporate) to make better policy decisions, which affect environmental outcomes. The tools and techniques we develop can be transferred to model policy outcomes in other areas such as water. - -## Project objective - -Accelerate net-zero transitions by improving the economic evaluation of interventions in the energy sector (government policies and technology changes). - -## Key results - -1. Improve the transparency and accessibility of UCLs UK TIMES model (https://www.ucl.ac.uk/drupal/site\_energymodels/sites/energy-models/files/uk-times-overview.pdf) by transitioning to open source. In particular the data management, least-cost optimization, and output formatting are currently closed-source. We will design the system with modern software engineering practices. Switching to free and open source software will reduce costs, increase transparency and accelerate modelling efforts through a snowballing effect.

Key Metrics: - - Replication - Are the sample outputs replicated in the FOSS implementation? - - Efficiency - Are the outputs produced in time comparable to the proprietary system? - - Usability - Is the system usable by existing UK TIMES clients? -

- -2. Provide efficiency improvements that enable the model to operate at an hourly timescale (8760 timeslices per year instead of the current 16). These will come from our knowledge of machine learning, compilers, and optimization. It may involve a custom machine learning algorithm and/or custom compiler. - -3. Provide further efficiency improvements that enable the model to operate over multiple countries and regions per country. - -4. Enable propagation of uncertainty through the model, in order to accurately assess risk (of not meeting carbon targets, or not meeting energy demand). This is an opportunity to add significant value to the model. - -5. Provide tools to ensure the trustworthiness of a model (in terms of explainability, uncertainty presentation). - -6. Apply our software stack to other similar energy-economy models (https://www.ucl.ac.uk/energy-models/), to show that it generalizes. - - -## Project milestones/deliverables - -### Q1 - -Set up open source project(s) on GitHub for various components, define interfaces, and build benchmarks. - -### Q2 - -Develop infrastructure to populate the input/output databases and inspect/validate them. - -### Q3 - -Develop open source component that creates model scenarios that are accepted by currently used solvers. - -### Q4 - -Minimum viable product tying in the above, but still using GAMS/CPLEX solvers (stretch goal: use open solver).