Start script 'healthy landscapes paper' and output script for automatic submission of SEALS runs #721
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -21,7 +21,7 @@ Imports: | |
| m4fsdp, | ||
| madrat, | ||
| magclass (>= 6.14.0), | ||
| magpie4 (>= 2.12.0), | ||
| MagpieNCGains, | ||
| magpiesets, | ||
| mip, | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| ;fix_2020;fix_2025 | ||
pvjeetze marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| gms$sm_fix_SSP2;2020;2025 | ||
| gms$sm_fix_cc;2020;2025 | ||
| gms$s15_food_substitution_start;2020;2025 | ||
| gms$s15_exo_foodscen_start;2020;2025 | ||
| gms$s21_trade_tariff_startyear;2020;2025 | ||
| gms$s22_conservation_start;2020;2025 | ||
| gms$s29_snv_scenario_start;2020;2025 | ||
| gms$s29_treecover_scenario_start;2020;2025 | ||
| gms$s29_fallow_scenario_start;2020;2025 | ||
| gms$s30_betr_scenario_start;2020;2025 | ||
| gms$s30_rotation_scenario_start;2020;2025 | ||
| gms$s38_startyear_labor_substitution;2020;2025 | ||
| gms$s42_efp_startyear;2020;2025 | ||
| gms$s44_start_year;2020;2025 | ||
| gms$s70_feed_substitution_start;2020;2025 | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,231 @@ | ||
| # | (C) 2008-2024 Potsdam Institute for Climate Impact Research (PIK) | ||
| # | authors, and contributors see CITATION.cff file. This file is part | ||
| # | of MAgPIE and licensed under AGPL-3.0-or-later. Under Section 7 of | ||
| # | AGPL-3.0, you are granted additional permissions described in the | ||
| # | MAgPIE License Exception, version 1.0 (see LICENSE file). | ||
| # | Contact: [email protected] | ||
|
|
||
| # -------------------------------------------------------------- | ||
| # description: Starts SEALS allocation run based on gridded MAgPIE land cover projections (SEALS) | ||
pvjeetze marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| # comparison script: FALSE | ||
| # --------------------------------------------------------------- | ||
|
|
||
| # Version 1.00 - Patrick v. Jeetze, Pascal Sauer | ||
| # 1.00: first working version | ||
|
|
||
| library(gms) | ||
| library(gdx) | ||
| library(magpie4) | ||
| library(filelock) | ||
|
|
||
| message("Initialising SEALS allocation run") | ||
|
|
||
| ############################# BASIC CONFIGURATION ####################################### | ||
| if (!exists("source_include")) { | ||
| title <- NULL | ||
| outputdir <- NULL | ||
|
|
||
| # Define arguments that can be read from command line | ||
| readArgs("outputdir", "title") | ||
| } | ||
| ######################################################################################### | ||
|
|
||
| # ======================== | ||
| # User settings | ||
| # ======================== | ||
|
|
||
| ### SEALS python environment name | ||
| # see https://justinandrewjohnson.com/earth_economy_devstack/installation.html | ||
| # for instructions on how to set up a python environment for SEALS | ||
| sealsEnv <- "seals_dev" | ||
|
|
||
| ### Path to miniforge installation | ||
| # For instructions on how to install miniforge see https://github.com/conda-forge/miniforge | ||
| miniforgePath <- "/home/vjeetze/miniforge3/bin/activate" | ||
|
There was a problem hiding this comment. Paths into home directories are not ideal, I get permission denied, and even if you give permission this only works as long as your home folder exists. There was a problem hiding this comment. How do you want this script to be used by others, should they install miniforge themselves (into their home folder) and then modify the script here? Or can we setup a shared miniforge path in There was a problem hiding this comment. This goes back to the whole discussion whether miniforge should be available to all (e.g. via the piam module or else). At the moment I could only install it into my home directory. This may be fine for the time being and if any users want to use the script they can install it in their home directory and change the path. I don't expect this script to be used by anyone besides me at the moment. I want this to be part of the release in the current form mainly for reasons of documentation for a paper. However, the script is also fully functional, if the user settings are modified based on user specific installations of miniforge and SEALS. There was a problem hiding this comment. I've adapted the script based on our bilateral discussion. Please have a look whether you are fine with these changes. |
||
|
|
||
| ### Path to SEALS base input file directory | ||
| dirBaseFiles <- "/p/projects/magpie/users/vjeetze/seals/files" | ||
|
|
||
| ### Path to SEALS code base | ||
| dirSEALS <- "/p/projects/magpie/users/vjeetze/seals/files/seals/seals_dev" | ||
pvjeetze marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
|
|
||
| # ======================== | ||
| # Prepare MAgPIE output | ||
| # ======================== | ||
|
|
||
| message("Script started for output directory: ", outputdir) | ||
|
|
||
| cfg <- gms::loadConfig(file.path(outputdir, "config.yml")) | ||
| title <- cfg$title | ||
|
|
||
| if (length(cfg$seals_years) != 0) { | ||
| rep_years <- cfg$seals_years[cfg$seals_years > 2020] | ||
| rep_years <- c(2020, rep_years) | ||
| } else { | ||
| rep_years <- seq(2020, 2050, 5) | ||
| } | ||
|
|
||
| # Restructure data to conform to SEALS | ||
| reportLandUseForSEALS( | ||
| magCellLand = "cell.land_0.5_share.mz", | ||
| outFile = paste0("cell.land_0.5_SEALS_", title, ".nc"), | ||
| dir = outputdir, selectyears = rep_years | ||
| ) | ||
|
|
||
|
|
||
| # ======================== | ||
| # Setup SEALS run | ||
| # ======================== | ||
|
|
||
| # -------------------------------- | ||
| # Prepare SEALS start script | ||
| # -------------------------------- | ||
|
|
||
| .setupSEALSrun <- function(title, dir, dirProject, dirSEALS, dirBaseFiles) { | ||
| if (!dir.exists(file.path(dirProject, "scripts"))) { | ||
| dir.create(file.path(dirProject, "scripts"), recursive = TRUE) | ||
| } | ||
|
|
||
| file.copy( | ||
| from = list.files(file.path(dirSEALS, "seals"), full.names = TRUE), | ||
| to = file.path(dirProject, "scripts"), | ||
| overwrite = TRUE, | ||
| recursive = TRUE | ||
| ) | ||
|
|
||
| if (!dir.exists(file.path(dirProject, "inputs"))) { | ||
| dir.create(file.path(dirProject, "inputs"), recursive = TRUE) | ||
| } | ||
|
|
||
| file.copy( | ||
| from = file.path(dir, paste0("seals_scenario_config_", title, ".csv")), | ||
| to = file.path(dirProject, "inputs", paste0("seals_scenario_config_", title, ".csv")), | ||
| overwrite = TRUE | ||
| ) | ||
|
|
||
| main <- readLines(file.path(dirProject, "scripts/run_test_standard.py")) | ||
pvjeetze marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| main[min(which(grepl(" p.user_dir =", main)))] <- paste0(" p.user_dir = \'", dirBaseFiles, "\'") | ||
| main[min(which(grepl(" p.extra_dirs", main)))] <- paste0(" p.extra_dirs = '.'") | ||
| main[min(which(grepl(" p.project_name =", main)))] <- paste0(" p.project_name = \'", title, "\'") | ||
| main[min(which(grepl(" p.project_dir =", main)))] <- paste0( | ||
| " p.project_dir = \'", normalizePath(dirProject), "\'" | ||
| ) | ||
| main[min(which(grepl("hb.pretty_time()", main)))] <- " " | ||
| main[min(which(grepl(" p.base_data_dir =", main)))] <- paste0( | ||
| " p.base_data_dir = \'", dirBaseFiles, "/base_data\'" | ||
| ) | ||
| main[min(which(grepl(" p.scenario_definitions_filename =", main)))] <- paste0( | ||
| " p.scenario_definitions_filename = \'", paste0("seals_scenario_config_", title, ".csv"), "\'" | ||
| ) | ||
|
|
||
| writeLines(main, file.path(dirProject, "scripts", paste0("run_seals_", title, ".py"))) | ||
|
|
||
|
|
||
| if (!dir.exists(file.path(dirProject, "run_submit"))) { | ||
| dir.create(file.path(dirProject, "run_submit"), recursive = TRUE) | ||
| } | ||
| } | ||
|
|
||
| # -------------------------------- | ||
| # Submit run | ||
| # -------------------------------- | ||
|
|
||
| .submitSEALS <- function(title, dirProject, miniforgePath, sealsEnv, qos = "short", | ||
| slurmID = FALSE, dependsID = NULL) { | ||
| submitFile <- file.path(dirProject, "run_submit", paste0("submit_seals_", title, ".sh")) | ||
|
|
||
| submit <- c( | ||
| "#!/bin/bash", "\n", | ||
| paste0("#SBATCH --qos=", qos), | ||
| ifelse(qos %in% c("priority", "standby"), "#SBATCH --partition=priority", "#SBATCH --partition=standard"), | ||
pvjeetze marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| "#SBATCH --job-name=seals_allocation", | ||
| paste0("#SBATCH --chdir=", normalizePath(file.path(dirProject, "scripts"))), | ||
| "#SBATCH --output=outfile_%j.out", | ||
| "#SBATCH --error=outfile_%j.err", | ||
| "#SBATCH --mail-type=END,FAIL", | ||
| "#SBATCH --time=3:00:00", | ||
| "#SBATCH --nodes=1", | ||
| "#SBATCH --ntasks=1", | ||
| ifelse(!is.null(dependsID), paste0( | ||
| "#SBATCH --dependency=afterok:", dependsID, | ||
| "\n#SBATCH --kill-on-invalid-dep=yes" | ||
| ), | ||
| "#SBATCH --dependency=singleton" | ||
| ), "\n", | ||
| ifelse(qos == "priority", "#SBATCH --cpus-per-task=64", "#SBATCH --cpus-per-task=128"), "\n", | ||
| paste("source", miniforgePath), | ||
| paste("conda activate", sealsEnv), "\n", | ||
| paste("python", paste0("run_seals_", title, ".py")) | ||
| ) | ||
| writeLines(submit, submitFile) | ||
|
|
||
| id <- system(paste("sbatch --parsable", submitFile), intern = TRUE) | ||
| if (slurmID) { | ||
| return(id) | ||
| } | ||
| } | ||
|
|
||
| # create output directory | ||
| dirProject <- "./output/seals" | ||
|
|
||
| if (!dir.exists(file.path(dirProject))) { | ||
| dir.create(file.path(dirProject), recursive = TRUE) | ||
| } | ||
|
|
||
| iniLock <- file.path(dirProject, ".lock") | ||
| lockOn <- filelock::lock(iniLock, exclusive = TRUE, timeout = Inf) | ||
| Sys.chmod(iniLock, mode = "0664") | ||
|
|
||
| # -------------------------------- | ||
| # Run SEALS allocation | ||
| # -------------------------------- | ||
|
|
||
| if (!is.null(lockOn)) { | ||
| sealsLock <- file.path(dirProject, "seals.lock") | ||
|
|
||
| if (!file.exists(sealsLock) || file.size(sealsLock) == 0) { | ||
| message("Starting SEALS allocation with input data creation.") | ||
| .setupSEALSrun( | ||
| title = title, | ||
| dir = outputdir, | ||
| dirProject = dirProject, | ||
| dirSEALS = dirSEALS, | ||
| dirBaseFiles = dirBaseFiles | ||
| ) | ||
|
|
||
| id <- .submitSEALS( | ||
| title = title, | ||
| dirProject = dirProject, | ||
| miniforgePath = miniforgePath, | ||
| sealsEnv = sealsEnv, | ||
| qos = "short", | ||
| slurmID = TRUE | ||
| ) | ||
|
|
||
| writeLines(id, sealsLock) | ||
| } else { | ||
| message("Starting SEALS allocation using existing input data.") | ||
| id <- readLines(sealsLock) | ||
|
|
||
| .setupSEALSrun( | ||
| title = title, | ||
| dir = outputdir, | ||
| dirProject = dirProject, | ||
| dirSEALS = dirSEALS, | ||
| dirBaseFiles = dirBaseFiles | ||
| ) | ||
|
|
||
| .submitSEALS( | ||
| title = title, | ||
| dirProject = dirProject, | ||
| miniforgePath = miniforgePath, | ||
| sealsEnv = sealsEnv, | ||
| qos = "short", | ||
| slurmID = FALSE, | ||
| dependsID = id | ||
| ) | ||
| } | ||
| unlink(iniLock) | ||
| } | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe add to the changelog that the default setting for the SVN policy has changed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I updated the changelog. Thanks for spotting this