From 6d618297299631b026847fbc916873d99a45f83b Mon Sep 17 00:00:00 2001 From: Courtney Peverley Date: Mon, 9 Dec 2024 09:38:18 -0700 Subject: [PATCH] update regression testing documentation --- .../development_workflow.md | 2 +- docs/conversion/convert-portable-layer.md | 1 + docs/conversion/interstitials.md | 28 +-- docs/development/cam-testing.md | 164 ++++++++++-------- docs/development/git-basics.md | 28 ++- 5 files changed, 126 insertions(+), 97 deletions(-) diff --git a/docs/atmospheric_physics/development_workflow.md b/docs/atmospheric_physics/development_workflow.md index 9866d45..9f2c802 100644 --- a/docs/atmospheric_physics/development_workflow.md +++ b/docs/atmospheric_physics/development_workflow.md @@ -9,7 +9,7 @@ The general workflow for adding a feature, bug-fix, or modification to atmospher 1. [**Open an issue.**](#1-open-an-issue) 2. **[Add your code modifications](#3-update-your-code-with-changes-from-the-official-repo) to a branch on your [fork](#2-create-a-fork-if-you-havent-already).** 3. **[Add/amend unit tests](#5-unit-testing)** - 3. **[Open a PR](#5-creating-a-pull-request-pr) from your branch to the `development` branch.** + 3. **[Open a PR](#6-creating-a-pull-request-pr) from your branch to the `development` branch.** 4. **Respond to any reviewer requests.** 5. **Fix any failing tests.** 6. **[Update `doc/NamesNotInDictionary.txt` file.](#updating-namesnotindictionarytxt-file)** diff --git a/docs/conversion/convert-portable-layer.md b/docs/conversion/convert-portable-layer.md index 56df61c..c293517 100644 --- a/docs/conversion/convert-portable-layer.md +++ b/docs/conversion/convert-portable-layer.md @@ -160,6 +160,7 @@ Do a preliminary look at the variables on the calling lists and make sure that C - If variables are not filled out in the list, highlight the entire line with yellow. - If they don't exist at all in the spreadsheet, enter them into the bottom of the sheet and highlight them with yellow. Note that physconst and diagnostic-only variables *may* not reside in the spreadsheet. - Let Jesse, Courtney or Cheryl know if you've encountered missing CCPP standardnames and/or units and added them to the list, so they can be discussed at an upcoming meeting. +- If you create your own name, add `_TBD` to the end of the name to signify that it has not been discussed yet. !!! note This process can normally take 2-4 weeks, so a preliminary look is advisable. diff --git a/docs/conversion/interstitials.md b/docs/conversion/interstitials.md index 25afa0b..490a158 100644 --- a/docs/conversion/interstitials.md +++ b/docs/conversion/interstitials.md @@ -51,27 +51,13 @@ As mentioned, there are some calculations/conversions/translations that are perf - **physics_tendency_updaters.F90**: apply tendencies output by physics to state variables. You'll need to include a tendency updater in your SDF for any `ptend%X` variables in the CAM-version of your code. -| Scheme name | Description | Inout variable | Input variable | -|:------------|-------------|----------------|----------------| -| apply_tendency_of_
eastward_wind | Apply the eastward wind tendency calculated in the previous physics scheme(s) to the `eastward_wind` state variable | eastward_wind | tendency_of_eastward_wind_
due_to_model_physics
timestep_for_physics | -| apply_tendency_of_
northward_wind| Apply the northward wind tendency calculated in the previous physics scheme(s) to the `northward_wind` state variable | northward_wind | tendency_of_northward_wind_
due_to_model_physics
timestep_for_physics | -| apply_heating_rate | Apply the heating rate (`tendency_of_dry_air_enthalpy_at_constant_pressure`) to the temperature tendency and temperature state variable | air_temperature
tendency_of_air_temperature_
due_to_model_physics | tendency_of_dry_air_enthalpy_
  at_constant_pressure
composition_dependent_specific_heat_
  of_dry_air_at_constant_pressure
timestep_for_physics | -| apply_tendency_of_
air_temperature | Apply the temperature tendency calculated in the previous physics scheme(s) to the `air_temperature` state variable | air_temperature | tendency_of_air_temperature_
due_to_model_physics
timestep_for_physics | - -## Temporary constituent handling -For now, we don't have tendency updaters for the constituents. As a result, if your parameterization includes one or more constituent, you'll have to do a little finagling. - -!!! Note "Problem overview" - CAM outputs a tendency and then passes that tendency to physics_update to be added to state%q, but CAM-SIMA (currently) updates the constituent array directly - -1. Write the scheme to update the constituent data directly (constituent variable passed in with `intent(inout)`) -1. In CAM, pass in a temporary array for the constituent rather than the actual `state%q(:,:,index)` -1. Then, back out the tendency after calling scheme_run and reassign that to `state%q(:,:,index)` - -An example of how this is done can be found ~line 276 in: -``` -$CAM/src/physics/simple/kessler_cam.F90 -``` +| Scheme name | Description | inout variables | onput variables | +|:------------|-------------|-----------------|-----------------| +| apply_tendency_of_
eastward_wind | Apply the eastward wind tendency calculated in the previous physics scheme(s) to the `eastward_wind` state variable | eastward_wind
tendency_of_eastward_wind | tendency_of_eastward_wind_
  due_to_model_physics
timestep_for_physics | +| apply_tendency_of_
northward_wind| Apply the northward wind tendency calculated in the previous physics scheme(s) to the `northward_wind` state variable | northward_wind
tendency_of_northward_wind | tendency_of_northward_wind_
  due_to_model_physics
timestep_for_physics | +| apply_heating_rate | Apply the heating rate (`tendency_of_dry_air_enthalpy_at_constant_pressure`) to the temperature tendency and temperature state variable | air_temperature
tendency_of_air_temperature_
  due_to_model_physics
heating_rate | tendency_of_dry_air_enthalpy_
  at_constant_pressure
composition_dependent_specific_heat_
  of_dry_air_at_constant_pressure
timestep_for_physics | +| apply_tendency_of_
air_temperature | Apply the temperature tendency calculated in the previous physics scheme(s) to the `air_temperature` state variable | air_temperature
tendency_of_air_temperature | tendency_of_air_temperature_
  due_to_model_physics
timestep_for_physics | +| apply_constituent_
tendencies | Apply the constituent tendencies calculated in the previous physics scheme(s) to the constituent state array | ccpp_constituents
ccpp_constituent_tendencies | timestep_for_physics | Proceed to [5 - Create an SDF](create-sdf.md). diff --git a/docs/development/cam-testing.md b/docs/development/cam-testing.md index c77ba56..1d20b58 100644 --- a/docs/development/cam-testing.md +++ b/docs/development/cam-testing.md @@ -2,6 +2,96 @@ This page describes the various automated and manual tests that are run for CAM-SIMA whenever the code is modified, as well as instructions for how to add new tests. +## Regression Testing +### Running the regression tests (manual) + +**NOTE: Regression testing on derecho should be done for every PR before merging!** + +Users can manually run regression tests on Derecho to ensure that the model builds correctly in various configurations. The tests can be run with a local copy of CAM-SIMA by using the `test_driver.sh` script under `$CAM-SIMA/test/system`. To run the tests associated with a particular compiler option one can do the following commands: + +For running GNU tests*: +``` +env CAM_FC=gnu ./test_driver.sh -f +``` + +For running Intel tests*: +``` +env CAM_FC=intel ./test_driver.sh -f +``` + +!!! Note "test_driver.sh default" + By default, `test_driver.sh` will compare against the baselines in `/glade/campaign/cesm/community/amwg/sima_baselines/latest_`. If you wish to compare against different baselines, specify the path to those baselines with the environment variable `BL_TESTDIR` + +!!! Note "Running test_driver.sh with no baseline comparison" + If you do not wish to compare to baselines, don't use the `BL_TESTDIR` environment variable and use the flag `--no-bl-compare` + +Running the script will produce a directory in your scratch space labeled `aux_sima__`, where `` is the compiler you chose, and `` is the timestamp (starting with the date) of when the tests were started, along with a job submitted to the local batch system. + +Inside the directory you should see an executable labeled `cs.status.*`. Running that command after the submitted job has finished will display the test results. **Everything should be labeled `PASS`. Any other label indicates that a test may have failed, and should be investigated.** Expected failures can be found in the `$CAM-SIMA/test/existing-test-failures.txt` file. + +Finally, the tests themselves are listed in `/cime_config/testdefs/testlist_cam.xml`. Any files that need to be included in order for the tests to run properly are located in ` +env CESM_TESTDIR=/glade/derecho/scratch/YourName/aux_cam_yyyymmddsssss CAM_FC=NVHPC ./archive_baseline.sh +``` + +!!! Note "Baseline 'latest' symlink" + By default, `archive_baseline.sh` will create a symlink to the relevant `latest_` directory. If you are archiving old baselines and do not want to create the symlink, use the `--no-symlink` flag. + +### Adding a new regression test + +The test list can be found here: `$CAM-SIMA/cime_config/testdefs/testlist_cam.xml` + +- If you are adding a new machine, compiler or category for an existing test, add a new `` XML entry to that `` entry +- If you are adding a fully new test, add a *new* `` XML entry with the following structure: +``` + + + + + + + + + +``` + +- ``: component set alias (or long name) - you can see more about compsets [here](../usage/creating-a-case.md) +- ``: model grid/resolution you'd like to run on - you can see more about grids [here](../usage/creating-a-case.md) + - **Try to use the lowest/coarsest resolution that will still accomplish your testing goals** +- ``: type of test to be run. You can find the testing types [here](https://esmci.github.io/cime/versions/master/html/users_guide/testing.html#testtype). +- ``: test modifier that changes the default behavior of the test type. More [here](https://esmci.github.io/cime/versions/master/html/users_guide/testing.html#modifiers) + - **Unless a longer run is necessary to exercise the code you are testing, run for 9 timesteps (_Ln9)** +- ``: relative path to the testmods directory for this run; usually looks something like `"cam/some_directory_name/"` + - The testmods directory will contain any namelist mods and XML configuration variable changes for the test (`user_nl_cam` and/or `shell_commands`) + - testmods directories can be found in `$CAM-SIMA/cime_config/testdefs/testmods_dirs/cam/` +- ``: machine name (options: `derecho`, `izumi`, `casper`) +- ``: compiler to be used (options: `gnu`, `nag`, `intel`, `nvhpc`) +- ``: group of tests that this test belongs to - the default run by `test_driver.sh` is `aux_sima` (which is run for each PR to CAM-SIMA) +- `WALLCLOCK_TIME`: maximum amount of time that the job will be allowed to run + +Here is an example test entry for a 2-timestep smoke test of kessler physics on a coarse MPAS grid, run with both intel and gnu +``` + + + + + + + + + + +``` + ## Github continuous integration testing The following tests/linters are run automatically on pull requests to `develop` in github. You can see previous runs of the CI tests [here](https://github.com/ESCOMP/CAM-SIMA/actions) @@ -81,76 +171,4 @@ hemco hemco-cesm2_0_hemco_3_9_0 7bd8358 is out of sync with .gitmodules hemco-ce Sparse checkout file missing (check the path): ``` mpas sparse checkout file .mpas_sparse_checkout not found -``` - -## Regression Testing -### Running the regression tests (manual) - -**NOTE: Regression testing on Derecho should be done for every PR before merging!** - -Users can manually run regression tests on Derecho to ensure that the model builds correctly in various configurations. The tests can be run with a local copy of CAM-SIMA by using the `test_driver.sh` script under `$CAM-SIMA/test/system`. To run the tests associated with a particular compiler option one can do the following commands: - -For running GNU tests*: -``` -env CAM_FC=gnu ./test_driver.sh -f -``` - -For running Intel tests*: -``` -env CAM_FC=intel ./test_driver.sh -f -``` - -Running the script will produce a directory in your scratch space labeled `aux_sima__`, where `` is the compiler you chose, and `` is the timestamp (starting with the date) of when the tests were started, along with a job submitted to the local batch system. - -Inside the directory you should see an executable labeled `cs.status.*`. Running that command after the submitted job has finished will display the test results. Currently for all tests everything should be labeled `PASS` except the `SUBMIT` step, which should be labeled `PEND`. Any other label indicates that a test may have failed, and should be investigated. - -Finally, the tests themselves are listed in `/cime_config/testdefs/testlist_cam.xml`. Any files that need to be included in order for the tests to run properly are located in `` XML entry to that `` entry -- If you are adding a fully new test, add a *new* `` XML entry with the following structure: -``` - - - - - - - - - -``` - -- ``: component set alias (or long name) - you can see more about compsets [here](../usage/creating-a-case.md) -- ``: model grid/resolution you'd like to run on - you can see more about grids [here](../usage/creating-a-case.md) - - **Try to use the lowest/coarsest resolution that will still accomplish your testing goals** -- ``: type of test to be run. You can find the testing types [here](https://esmci.github.io/cime/versions/master/html/users_guide/testing.html#testtype). -- ``: test modifier that changes the default behavior of the test type. More [here](https://esmci.github.io/cime/versions/master/html/users_guide/testing.html#modifiers) - - **Unless a longer run is necessary to exercise the code you are testing, run for 9 timesteps (_Ln9)** -- ``: relative path to the testmods directory for this run; usually looks something like `"cam/some_directory_name/"` - - The testmods directory will contain any namelist mods and XML configuration variable changes for the test (`user_nl_cam` and/or `shell_commands`) - - testmods directories can be found in `$CAM-SIMA/cime_config/testdefs/testmods_dirs/cam/` -- ``: machine name (options: `derecho`, `izumi`, `casper`) -- ``: compiler to be used (options: `gnu`, `nag`, `intel`, `nvhpc`) -- ``: group of tests that this test belongs to - the default run by `test_driver.sh` is `aux_sima` (which is run for each PR to CAM-SIMA) -- `WALLCLOCK_TIME`: maximum amount of time that the job will be allowed to run - -Here is an example test entry for a 2-timestep smoke test of kessler physics on a coarse MPAS grid, run with both intel and gnu -``` - - - - - - - - - - -``` - +``` \ No newline at end of file diff --git a/docs/development/git-basics.md b/docs/development/git-basics.md index f4e478c..bf47b64 100644 --- a/docs/development/git-basics.md +++ b/docs/development/git-basics.md @@ -174,7 +174,7 @@ cd git checkout ``` -note that `` can also be the name of a branch or a commit hash. If you specify the name of a branch, you will check out the head of the branch. If you name a remote branch (e.g., `origin/branch_name`), you will create a detached HEAD but you can still use the code. Please note that if you plan on changing the code, first create a branch (see [Working with branches](#working-with-branches) +note that `` can also be the name of a branch or a commit hash. If you specify the name of a branch, you will check out the head of the branch. If you name a remote branch (e.g., `origin/branch_name`), you will create a detached HEAD but you can still use the code. Please note that if you plan on changing the code, first create a branch (see [Working with branches](#working-with-branches)) ### Working with branches When you create a clone, your clone will contain pointers all the branches that existed at the clone's origin (e.g., the repository at GitHub). While you can check out these branches, however, before attempting to make any changes, you should first create a local version branch (so git can keep track of the local commits). @@ -241,7 +241,7 @@ Note that while this section explains how to update your local branch to the `ES Before starting, you should have either: -- A fresh clone of your fork with the branch you wish to update checked out (see [Create a new clone](#create-a-new-clone) and [Working with branches](#working-with-branches). +- A fresh clone of your fork with the branch you wish to update checked out (see [Create a new clone](#create-a-new-clone) and [Working with branches](#working-with-branches)). - An existing clone with the branch you wish to update checked out and in a clean state (i.e., make sure you do a `git commit` and that `git status` shows no modified files). Add the upstream remote, if you have not already done so (see [Adding remotes](#adding-remote-new-upstream-repository-locations)). @@ -252,6 +252,30 @@ git fetch ESCOMP git merge ESCOMP/development ``` +## Tagging a commit +When a PR is ready to be merged into CAM-SIMA's code base, you will need to consider whether or not a new tag is appropriate. In CAM-SIMA, the standard is whether or not the baselines have changed since the last tag. After you have run the tests (per the [regression testing workflow](cam-testing.md/#running-the-regression-tests-manual)): + +1. If there are no baseline differences (no tests "FAIL" or yield a "DIFF"), no tag is necessary and you can proceed to merge your PR (you will also not archive the baselines) +1. If there are baseline differences (at least one test results in a "DIFF"), you will need to follow the procedure below to create a tag **after** you merge your PR + +### Tag syntax + +We use the following syntax for CAM-SIMA tags: **simaX_YY_ZZZ**, where: + +- X: major version; incremented only during major releases +- YY: minor version; incremented any time the answers for the CAM7 physics suite change (zero-padded) +- ZZZ: patch version; incremented when answers or fieldlists change for non-CAM7 configurations (zero-padded) + +To see the latest tag: [https://github.com/ESCOMP/CAM-SIMA/tags](https://github.com/ESCOMP/CAM-SIMA/tags). + +### Creating a new tag + +1. Clone [https://github.com/ESSCOMP/CAM](https://github.com/ESCOMP/CAM) or update an existing clone with `git fetch development` +1. Find the PR merge commit (``) for the tag + - In your up-to-date CAM clone, find the merge commit with `git log --oneline -1 /development` where the merge commit is the first field in the output. +1. Tag the `` with `git tag -a -m '` where `` is the same as the message for the `` +1. Push the tag to the proper remote with `git push ` + ## Comparing differences using git diff If you have a git clone, you can view differences between commits or tags. As far as `git diff` is concerned, a commit hash is the same as a tag so in the examples below will use ``.