-
Notifications
You must be signed in to change notification settings - Fork 7
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add TIMES-NZ to benchmarks and refactor CI (#239)
Move setting up benchmark repos to new script. --------- Co-authored-by: Siddharth Krishna <[email protected]>
- Loading branch information
1 parent
f3ec4e6
commit fc9b478
Showing
5 changed files
with
198 additions
and
69 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -73,38 +73,32 @@ git commit --no-verify | |
|
|
||
| ### Running Benchmarks | ||
|
|
||
| See our GitHub Actions CI `.github/workflows/ci.yml` and the utility script `utils/run_benchmarks.py` to see how to run the tool on the DemoS models. | ||
|
|
||
| In short, use the commands below to clone the benchmarks data into your local `benchmarks` dir. | ||
| Note that this assumes you have access to all these repositories (some are private and | ||
| you'll have to request access) - if not, comment out the inaccessible benchmarks from `benchmakrs.yml` before running. | ||
| We use the TIMES DemoS models and some public TIMES models as benchmarks. | ||
| See our GitHub Actions CI `.github/workflows/ci.yml` and the utility script `utils/run_benchmarks.py` to see how to we benchmark the tool and check PRs automatically for regression. | ||
| If you are a developer, you can use the below instructions to set up and run the benchmarks locally: | ||
|
|
||
| ```bash | ||
| mkdir benchmarks | ||
| # Get TIMES DemoS example models and reference DD files | ||
| # XLSX files are in private repo for licensing reasons, please request access or replace with your own files distributed with Veda. | ||
| git clone [email protected]:olejandro/demos-xlsx.git benchmarks/xlsx/ | ||
| git clone [email protected]:olejandro/demos-dd.git benchmarks/dd/ | ||
|
|
||
| # Get Ireland model and reference DD files | ||
| git clone [email protected]:esma-cgep/tim.git benchmarks/xlsx/Ireland | ||
| git clone [email protected]:esma-cgep/tim-gams.git benchmarks/dd/Ireland | ||
| ./setup-benchmarks.sh | ||
| ``` | ||
| Note that this script assumes you have access to all the relevant repositories (some are private and you'll have to request access) - if not, comment out the inaccessible benchmarks from `benchmarks.yml` before running. | ||
|
|
||
| Then to run the benchmarks: | ||
| ```bash | ||
| # Run a only a single benchmark by name (see benchmarks.yml for name list) | ||
| python utils/run_benchmarks.py benchmarks.yml --verbose --run DemoS_001-all | tee out.txt | ||
| python utils/run_benchmarks.py benchmarks.yml --run DemoS_001-all | ||
|
|
||
| # Run all benchmarks (without GAMS run, just comparing CSV data for regressions) | ||
| # Note: if you have multiple remotes, set etsap-TIMES/xl2times as the first one, as it is used for speed/correctness comparisons. | ||
| python utils/run_benchmarks.py benchmarks.yml --verbose | tee out.txt | ||
| # To see the full output logs, and save it in a file for convenience | ||
| python utils/run_benchmarks.py benchmarks.yml --run DemoS_001-all --verbose | tee out.txt | ||
|
|
||
| # Run all benchmarks (without GAMS run, just comparing CSV data for regressions) | ||
| # Note: if you have multiple remotes, set etsap-TIMES/xl2times as the `origin`, as it is used for speed/correctness comparisons. | ||
| python utils/run_benchmarks.py benchmarks.yml | ||
|
|
||
| # Run benchmarks with regression tests vs main branch | ||
| git branch feature/your_new_changes --checkout | ||
| # ... make your code changes here ... | ||
| git commit -a -m "your commit message" # code must be committed for comparison to `main` branch to run. | ||
| python utils/run_benchmarks.py benchmarks.yml --verbose | tee out.txt | ||
| python utils/run_benchmarks.py benchmarks.yml | ||
| ``` | ||
| At this point, if you haven't broken anything you should see something like: | ||
| ``` | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,78 @@ | ||
| #!/bin/bash | ||
|
|
||
| # A helper script to setup or update the repositories containing benchmark models under `benchmarks/` | ||
|
|
||
| set -eo pipefail | ||
|
|
||
| # Commit SHA for each repository: | ||
| REF_TIMES_model="b488fb07f0899ee8b7e710c230b1a9414fa06f7d" | ||
| REF_demos_xlsx="34a2a5c044cc0bbea1357de50db2f5f02d575181" | ||
| REF_demos_dd="2848a8a8e2fdcf0cdf7f83eefbdd563b0bb74e86" | ||
| REF_tim="e820d8002adc6b1526a3bffcc439219b28d0eed5" | ||
| REF_tim_gams="703f6a4e1d0bedd95c3ebdae534496f3a7e1b7cc" | ||
| REF_TIMES_NZ="c83f2d0e51d692cba27a55032c8f8a2a48e4d425" | ||
|
|
||
| # If no GitHub token is provided, try to clone using SSH | ||
| if [ -z "$GH_PAT_DEMOS_XLSX" ]; then | ||
| echo "Warning: no GitHub token provided, will try to clone private repos using SSH" | ||
| use_SSH=1 | ||
| fi | ||
|
|
||
| # Move to the directory containing this script | ||
| SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd ) | ||
| cd "$SCRIPT_DIR" | ||
|
|
||
| mkdir -p benchmarks | ||
|
|
||
| # Function to check out a repository at a specified commit | ||
| checkout_repo() { | ||
| local repo=$1 | ||
| local dest_dir=$2 | ||
| local commit=$3 | ||
| local private=$4 | ||
|
|
||
| if [ -d "$dest_dir" ]; then | ||
| echo "Directory $dest_dir already exists. Checking if on correct commit." | ||
| pushd "$dest_dir" > /dev/null | ||
| git fetch --depth=1 origin "$commit" | ||
| else | ||
| echo "Directory $dest_dir does not exist. Cloning repository." | ||
| if [ -n "$private" ]; then | ||
| if [ -n "$use_SSH" ]; then | ||
| repo_url="[email protected]:${repo}.git" | ||
| else | ||
| repo_url="https://$GH_PAT_DEMOS_XLSX@github.com/${repo}/" | ||
| fi | ||
| else | ||
| repo_url="https://github.com/${repo}/" | ||
| fi | ||
| git clone --filter=blob:none "$repo_url" "$dest_dir" | ||
| pushd "$dest_dir" > /dev/null | ||
| fi | ||
| git checkout "$commit" || exit 1 | ||
| popd > /dev/null | ||
| echo "$dest_dir: successfully checked out $repo at $commit" | ||
| } | ||
|
|
||
| # Array of repositories to check out, in the form repo|dest_dir|commit|private | ||
| repositories=( | ||
| "etsap-TIMES/TIMES_model|TIMES_model|$REF_TIMES_model" | ||
| "olejandro/demos-dd|benchmarks/dd|$REF_demos_dd" | ||
| "olejandro/demos-xlsx|benchmarks/xlsx|$REF_demos_xlsx|true" | ||
| "esma-cgep/tim|benchmarks/xlsx/Ireland|$REF_tim" | ||
| "esma-cgep/tim-gams|benchmarks/dd/Ireland|$REF_tim_gams" | ||
| "olejandro/TIMES-NZ-Model-Files|benchmarks/TIMES-NZ|$REF_TIMES_NZ" | ||
| ) | ||
|
|
||
| # Setup / update the repositories | ||
| for repo_info in "${repositories[@]}"; do | ||
| IFS='|' read -r repo dest_dir commit private <<< "$repo_info" | ||
| checkout_repo "$repo" "$dest_dir" "$commit" "$private" | ||
| done | ||
|
|
||
| # Create symlinks for TIMES-NZ since xlsx & dd files are in same repo | ||
| ln -s "$SCRIPT_DIR/benchmarks/TIMES-NZ/TIMES-NZ" "$SCRIPT_DIR/benchmarks/xlsx/TIMES-NZ" | ||
| ln -s "$SCRIPT_DIR/benchmarks/TIMES-NZ/TIMES-NZ-GAMS/times_scenarios/kea-v2_1_3" "$SCRIPT_DIR/benchmarks/dd/TIMES-NZ-KEA" | ||
| ln -s "$SCRIPT_DIR/benchmarks/TIMES-NZ/TIMES-NZ-GAMS/times_scenarios/tui-v2_1_3" "$SCRIPT_DIR/benchmarks/dd/TIMES-NZ-TUI" | ||
|
|
||
| echo "All benchmark repositories are set up and up to date :)" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters