Merge pull request remindmodel#1742 from orichters/develop

fail on duplicated column names in scenario config

CHANGELOG.md

@@ -11,35 +11,48 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
     anymore; can be re-activated using `cm_wastelag`
 
 ### added
-- added realizations diffExp2Lin and diffLin2Lin to 45_carbonprice [#1723](https://github.com/remindmodel/remind/pull/1723)
+- **50_damages**, **51_internalizeDamages** add KotzWenz realization based on Kotz & Wenz (2024)
+    [[#1601](https://github.com/remindmodel/remind/pull/1601)]
+- **45_carbonprice** added realizations diffExp2Lin and diffLin2Lin
+    [[#1723](https://github.com/remindmodel/remind/pull/1723)]
+- **scripts** define defaults for script selections in output.R
+    [[#1739](https://github.com/remindmodel/remind/pull/1739)]
+- **scripts** fail transparently on duplicated column names in scenario_config*.csv files
+    [[#1742](https://github.com/remindmodel/remind/pull/1742)]
 
 ### fixed
 - included CCS from plastic waste incineration in CCS mass flows so it is
     subject to injection constraints (but did not add CCS costs, see
     https://github.com/remindmodel/development_issues/issues/274
+- **scripts** fix tax convergence reporting in modelSummary
+    [[#1728](https://github.com/remindmodel/remind/pull/1728)]
+- **scripts** cleanup non-existing realizations from settings_config.csv
+    [[#1718](https://github.com/remindmodel/remind/pull/1718)]
 
 ### removed
 
 ## [3.3.2] - 2024-07-04
 
 ### changed
-- fix output generation [[#1715]] (https://github.com/remindmodel/remind/pull/1715)
+- fix output generation [[#1715](https://github.com/remindmodel/remind/pull/1715)]
 
 ## [3.3.1] - 2024-06-18
 
 ### changed
-- new input data (6.77) including new GDP and population data([#83](https://github.com/pik-piam/mrdrivers/pull/83)) [[#1684]](https://github.com/remindmodel/remind/pull/1684)
+- new input data (6.77) including new GDP and population data
+    [[#83](https://github.com/pik-piam/mrdrivers/pull/83)] [[#1684](https://github.com/remindmodel/remind/pull/1684)]
 - **37_industry** remove subsector-specific shares of SE
-  origins in FE carriers for performance reasons [[#1659]](https://github.com/remindmodel/remind/pull/1659)
-- **37_industry** make process-based steel production model the default over the ces-based model [[#1663]](https://github.com/remindmodel/remind/pull/1663)
+  origins in FE carriers for performance reasons [[#1659](https://github.com/remindmodel/remind/pull/1659)]
+- **37_industry** make process-based steel production model the default over the ces-based model [[#1663](https://github.com/remindmodel/remind/pull/1663)]
 - **37_industry** fixed incineration of plastic and non-plastic waste causing
   non-zero emissions for biomass and synfuels
-  [[#1682]](https://github.com/remindmodel/remind/pull/1682)
-- **core** another change of preference parameters and associated computation of interest rates/mark ups [[#1663]](https://github.com/remindmodel/remind/pull/1663)
+  [[#1682](https://github.com/remindmodel/remind/pull/1682)]
+- **core** another change of preference parameters and associated computation of interest rates/mark ups [[#1663](https://github.com/remindmodel/remind/pull/1663)]
 - **scripts** do not check anymore that MAgPIE uses renv
   [[1646](https://github.com/remindmodel/remind/pull/1646)]
-- **scripts** adjust function calls after moving functionality from `remind2` [[#578]]](https://github.com/pik-piam/remind2/pull/578) to `piamPlotComparison` and `piamutils` [[#1661](https://github.com/remindmodel/remind/pull/1661)
-- **scripts** enhance output script `reportCEScalib` to include additional plot formats [[#1671](https://github.com/remindmodel/remind/pull/1671)
+- **scripts** adjust function calls after moving functionality from `remind2`
+  [[#578](https://github.com/pik-piam/remind2/pull/578)] to `piamPlotComparison` and `piamutils` [[#1661](https://github.com/remindmodel/remind/pull/1661)]
+- **scripts** enhance output script `reportCEScalib` to include additional plot formats [[#1671](https://github.com/remindmodel/remind/pull/1671)]
 
 ### added
 - **24_trade** add optinal trade scenario for EUR hydrogen and e-liquids imports [[#1666](https://github.com/remindmodel/remind/pull/1666)] 
@@ -48,7 +61,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
 ### changed
 - **37_industry** changed industry to have subsector-specific shares of SE
-  origins in FE carriers [[#1620]](https://github.com/remindmodel/remind/pull/1620)
+  origins in FE carriers [[#1620](https://github.com/remindmodel/remind/pull/1620)]
 
 ### added
 - **config** regex tests for many parameters [[#1356](https://github.com/remindmodel/remind/pull/1356)]

scripts/start/readCheckScenarioConfig.R

@@ -26,6 +26,10 @@ readCheckScenarioConfig <- function(filename, remindPath = ".", testmode = FALSE
   scenConf <- scenConf[! is.na(scenConf[1]), ]
   rownames(scenConf) <- scenConf[, 1]
   scenConf[1] <- NULL
+  colduplicates <- grep("\\.[1-9]$", colnames(scenConf), value = TRUE)
+  if (length(colduplicates) > 0) {
+    warning("These colnames are signs of duplicated columns: ", paste(colduplicates, collapse = ", "))
+  }
   toolong <- nchar(rownames(scenConf)) > 75
   if (any(toolong)) {
     warning("These titles are too long: ",
@@ -120,7 +124,7 @@ readCheckScenarioConfig <- function(filename, remindPath = ".", testmode = FALSE
   scenConf[, names(path_gdx_list)[! names(path_gdx_list) %in% names(scenConf)]] <- NA
 
   # collect errors
-  errorsfound <- sum(toolong) + sum(regionname) + sum(nameisNA) + sum(illegalchars) + whitespaceErrors + copyConfigFromErrors + pathgdxerrors + missingRealizations
+  errorsfound <- length(colduplicates) + sum(toolong) + sum(regionname) + sum(nameisNA) + sum(illegalchars) + whitespaceErrors + copyConfigFromErrors + pathgdxerrors + missingRealizations
 
   # check column names
   knownColumnNames <- c(names(cfg$gms), setdiff(names(cfg), "gms"), names(path_gdx_list),

tutorials/10_DebuggingREMIND.md

@@ -124,16 +124,17 @@ An explanation of the modelstat and solvestat numbers can be found in the tables
 |Modelstat = 6 | Intermediate Infeasible|
 |Modelstat = 7 | Intermediate Nonoptimal|
 | | |
-|Solvestat = 1 | Normal Completion|
-|Solvestat = 2 | Iteration Interrupt|
-|Solvestat = 3 | Resource Interrupt|
+|Solvestat = 1 | Normal Completion   |
+|Solvestat = 2 | Iteration Interrupt |
+|Solvestat = 3 | Resource Interrupt  |
 |Solvestat = 4 | Terminated by Solver|
 
 |Desirable Status in REMIND|
 |---|
 |Solve + Model stat = 1 + 2 | solution found|
 |Solve + Model stat = 4 + 7 | feasible but slow convergence|
 
+Further possible error codes can be found in [the GAMS documentation](https://www.gams.com/latest/docs/UG_GAMSOutput.html#UG_GAMSOutput_SolverStatus).
 If infeasibilities show up already in the first iteration, it may be related to a wrong `input.gdx` (specified with `path_gdx` in the `scenario_config_XYZ.csv`) or some general error in the GAMS code. Via the international trade, infeasibilities in one region may propagate to other regions in later iterations, but then it is worth knowing where it started.
 
 There are different types of solver infeasibilities: pre-triangular and optimization infeasibilities. In pre-triangular infeasibilities, GAMS shows you in the solution report the equations that are incompatible with each other. For optimization infeasibilies, the CONOPT solver tries to reduce the infeasibility to the thing that less affects the objective function. It does not show all affected equations, as it is not a simple problem as a non-square system of equations like in the pre-triangular case. You need to check if it is bound-related: the variables bounds, and the equation bounds starting from the infeasibility and going through the variables that have relation with it. You can always force in another run the variable that is infeasible to a feasible value to see what else is affected by it. But this is usually not necessary as just checking the logic behind the equation and the infeasible variable is usually sufficient to find the limitation.