Skip to content

Commit

Permalink
update regression testing documentation
Browse files Browse the repository at this point in the history
  • Loading branch information
@peverwhee
peverwhee committed Dec 9, 2024
1 parent d5c1fbd commit 6d61829
Show file tree
Hide file tree
Showing 5 changed files with 126 additions and 97 deletions.
2 changes: 1 addition & 1 deletion docs/atmospheric_physics/development_workflow.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ The general workflow for adding a feature, bug-fix, or modification to atmospher
1. [**Open an issue.**](#1-open-an-issue)
2. **[Add your code modifications](#3-update-your-code-with-changes-from-the-official-repo) to a branch on your [fork](#2-create-a-fork-if-you-havent-already).**
3. **[Add/amend unit tests](#5-unit-testing)**
3. **[Open a PR](#5-creating-a-pull-request-pr) from your branch to the `development` branch.**
3. **[Open a PR](#6-creating-a-pull-request-pr) from your branch to the `development` branch.**
4. **Respond to any reviewer requests.**
5. **Fix any failing tests.**
6. **[Update `doc/NamesNotInDictionary.txt` file.](#updating-namesnotindictionarytxt-file)**
Expand Down
1 change: 1 addition & 0 deletions docs/conversion/convert-portable-layer.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -160,6 +160,7 @@ Do a preliminary look at the variables on the calling lists and make sure that C
- If variables are not filled out in the list, highlight the entire line with yellow.
- If they don't exist at all in the spreadsheet, enter them into the bottom of the sheet and highlight them with yellow. Note that physconst and diagnostic-only variables *may* not reside in the spreadsheet.
- Let Jesse, Courtney or Cheryl know if you've encountered missing CCPP standardnames and/or units and added them to the list, so they can be discussed at an upcoming meeting.
- If you create your own name, add `_TBD` to the end of the name to signify that it has not been discussed yet.

!!! note
This process can normally take 2-4 weeks, so a preliminary look is advisable.
Expand Down
28 changes: 7 additions & 21 deletions docs/conversion/interstitials.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -51,27 +51,13 @@ As mentioned, there are some calculations/conversions/translations that are perf

- **physics_tendency_updaters.F90**: apply tendencies output by physics to state variables. You'll need to include a tendency updater in your SDF for any `ptend%X` variables in the CAM-version of your code.

| Scheme name | Description | Inout variable | Input variable |
|:------------|-------------|----------------|----------------|
| apply_tendency_of_<br/>eastward_wind | Apply the eastward wind tendency calculated in the previous physics scheme(s) to the `eastward_wind` state variable | eastward_wind | tendency_of_eastward_wind_<br/>due_to_model_physics<br/>timestep_for_physics |
| apply_tendency_of_<br/>northward_wind| Apply the northward wind tendency calculated in the previous physics scheme(s) to the `northward_wind` state variable | northward_wind | tendency_of_northward_wind_<br/>due_to_model_physics<br/>timestep_for_physics |
| apply_heating_rate | Apply the heating rate (`tendency_of_dry_air_enthalpy_at_constant_pressure`) to the temperature tendency and temperature state variable | air_temperature<br/>tendency_of_air_temperature_<br/>due_to_model_physics | tendency_of_dry_air_enthalpy_<br/>&ensp;&ensp;at_constant_pressure<br/>composition_dependent_specific_heat_<br/>&ensp;&ensp;of_dry_air_at_constant_pressure<br/>timestep_for_physics |
| apply_tendency_of_<br/>air_temperature | Apply the temperature tendency calculated in the previous physics scheme(s) to the `air_temperature` state variable | air_temperature | tendency_of_air_temperature_<br/>due_to_model_physics<br/>timestep_for_physics |

## Temporary constituent handling
For now, we don't have tendency updaters for the constituents. As a result, if your parameterization includes one or more constituent, you'll have to do a little finagling.

!!! Note "Problem overview"
CAM outputs a tendency and then passes that tendency to physics_update to be added to state%q, but CAM-SIMA (currently) updates the constituent array directly

1. Write the scheme to update the constituent data directly (constituent variable passed in with `intent(inout)`)
1. In CAM, pass in a temporary array for the constituent rather than the actual `state%q(:,:,index)`
1. Then, back out the tendency after calling scheme_run and reassign that to `state%q(:,:,index)`

An example of how this is done can be found ~line 276 in:
```
$CAM/src/physics/simple/kessler_cam.F90
```
| Scheme name | Description | inout variables | onput variables |
|:------------|-------------|-----------------|-----------------|
| apply_tendency_of_<br/>eastward_wind | Apply the eastward wind tendency calculated in the previous physics scheme(s) to the `eastward_wind` state variable | eastward_wind<br/>tendency_of_eastward_wind | tendency_of_eastward_wind_<br/>&ensp;&ensp;due_to_model_physics<br/>timestep_for_physics |
| apply_tendency_of_<br/>northward_wind| Apply the northward wind tendency calculated in the previous physics scheme(s) to the `northward_wind` state variable | northward_wind<br/>tendency_of_northward_wind | tendency_of_northward_wind_<br/>&ensp;&ensp;due_to_model_physics<br/>timestep_for_physics |
| apply_heating_rate | Apply the heating rate (`tendency_of_dry_air_enthalpy_at_constant_pressure`) to the temperature tendency and temperature state variable | air_temperature<br/>tendency_of_air_temperature_<br/>&ensp;&ensp;due_to_model_physics<br/>heating_rate | tendency_of_dry_air_enthalpy_<br/>&ensp;&ensp;at_constant_pressure<br/>composition_dependent_specific_heat_<br/>&ensp;&ensp;of_dry_air_at_constant_pressure<br/>timestep_for_physics |
| apply_tendency_of_<br/>air_temperature | Apply the temperature tendency calculated in the previous physics scheme(s) to the `air_temperature` state variable | air_temperature<br/>tendency_of_air_temperature | tendency_of_air_temperature_<br/>&ensp;&ensp;due_to_model_physics<br/>timestep_for_physics |
| apply_constituent_<br/>tendencies | Apply the constituent tendencies calculated in the previous physics scheme(s) to the constituent state array | ccpp_constituents<br/>ccpp_constituent_tendencies | timestep_for_physics |

Proceed to [5 - Create an SDF](create-sdf.md).

Expand Down
164 changes: 91 additions & 73 deletions docs/development/cam-testing.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,96 @@

This page describes the various automated and manual tests that are run for CAM-SIMA whenever the code is modified, as well as instructions for how to add new tests.

## Regression Testing
### Running the regression tests (manual)

**NOTE: Regression testing on derecho should be done for every PR before merging!**

Users can manually run regression tests on Derecho to ensure that the model builds correctly in various configurations. The tests can be run with a local copy of CAM-SIMA by using the `test_driver.sh` script under `$CAM-SIMA/test/system`. To run the tests associated with a particular compiler option one can do the following commands:

For running GNU tests*:
```
env CAM_FC=gnu ./test_driver.sh -f
```

For running Intel tests*:
```
env CAM_FC=intel ./test_driver.sh -f
```

!!! Note "test_driver.sh default"
By default, `test_driver.sh` will compare against the baselines in `/glade/campaign/cesm/community/amwg/sima_baselines/latest_<CAM_FC>`. If you wish to compare against different baselines, specify the path to those baselines with the environment variable `BL_TESTDIR`

!!! Note "Running test_driver.sh with no baseline comparison"
If you do not wish to compare to baselines, don't use the `BL_TESTDIR` environment variable and use the flag `--no-bl-compare`

Running the script will produce a directory in your scratch space labeled `aux_sima_<CAM_FC>_<timestamp>`, where `<CAM_FC>` is the compiler you chose, and `<timestamp>` is the timestamp (starting with the date) of when the tests were started, along with a job submitted to the local batch system.

Inside the directory you should see an executable labeled `cs.status.*`. Running that command after the submitted job has finished will display the test results. **Everything should be labeled `PASS`. Any other label indicates that a test may have failed, and should be investigated.** Expected failures can be found in the `$CAM-SIMA/test/existing-test-failures.txt` file.

Finally, the tests themselves are listed in `<CAM-SIMA>/cime_config/testdefs/testlist_cam.xml`. Any files that need to be included in order for the tests to run properly are located in `<CAM-SIMA/cime_config/testdefs/testmods_dirs/cam/outfrq_XXX`, where `XXX` is the name of the test. Additional information on the CIME testing system, which is what this testing infrastructure is built on, can be found [online here](https://esmci.github.io/cime/versions/master/html/users_guide/testing.html).

*Note: you may also have to include the environment variable `CAM_ACCOUNT` on derecho, which points to your account key

### Archiving baselines
After you have run the tests, merged your PR, and created a tag (see [tag workflow](git-basics.md#tagging-a-commit)), you will need to archive your baselines for the next person.

To do this, navigate to `$CAM-SIMA/test/system` on derecho and run:

```
env CESM_TESTDIR=/glade/derecho/scratch/YourName/aux_cam_yyyymmddsssss CAM_FC=INTEL ./archive_baseline.sh <sima tag>
env CESM_TESTDIR=/glade/derecho/scratch/YourName/aux_cam_yyyymmddsssss CAM_FC=NVHPC ./archive_baseline.sh <sima tag>
```

!!! Note "Baseline 'latest' symlink"
By default, `archive_baseline.sh` will create a symlink to the relevant `latest_<CAM_FC>` directory. If you are archiving old baselines and do not want to create the symlink, use the `--no-symlink` flag.

### Adding a new regression test

The test list can be found here: `$CAM-SIMA/cime_config/testdefs/testlist_cam.xml`

- If you are adding a new machine, compiler or category for an existing test, add a new `<machine>` XML entry to that `<test>` entry
- If you are adding a fully new test, add a *new* `<test>` XML entry with the following structure:
```
<test compset="<COMPSET_NAME>" grid="<GRID_ALIAS>" name="<TEST_TYPE>_<TEST_MOD>" testmods="<RELPATH_TO_TESTMODS_DIR>">
<machines>
<machine name="<MACH_NAME>" compiler="<COMPILER>" category="<TEST_CATEGORY>"/>
</machines>
<options>
<option name="comment">COMMENT HERE</option>
<option name="wallclock">WALLCLOCK_TIME</option>
</options>
</test>
```

- `<COMPSET_NAME>`: component set alias (or long name) - you can see more about compsets [here](../usage/creating-a-case.md)
- `<GRID_ALIAS>`: model grid/resolution you'd like to run on - you can see more about grids [here](../usage/creating-a-case.md)
- **Try to use the lowest/coarsest resolution that will still accomplish your testing goals**
- `<TEST_TYPE>`: type of test to be run. You can find the testing types [here](https://esmci.github.io/cime/versions/master/html/users_guide/testing.html#testtype).
- `<TEST_MOD>`: test modifier that changes the default behavior of the test type. More [here](https://esmci.github.io/cime/versions/master/html/users_guide/testing.html#modifiers)
- **Unless a longer run is necessary to exercise the code you are testing, run for 9 timesteps (_Ln9)**
- `<RELPATH_TO_TESTMODS_DIR>`: relative path to the testmods directory for this run; usually looks something like `"cam/some_directory_name/"`
- The testmods directory will contain any namelist mods and XML configuration variable changes for the test (`user_nl_cam` and/or `shell_commands`)
- testmods directories can be found in `$CAM-SIMA/cime_config/testdefs/testmods_dirs/cam/`
- `<MACH_NAME>`: machine name (options: `derecho`, `izumi`, `casper`)
- `<COMPILER>`: compiler to be used (options: `gnu`, `nag`, `intel`, `nvhpc`)
- `<TEST_CATEGORY>`: group of tests that this test belongs to - the default run by `test_driver.sh` is `aux_sima` (which is run for each PR to CAM-SIMA)
- `WALLCLOCK_TIME`: maximum amount of time that the job will be allowed to run

Here is an example test entry for a 2-timestep smoke test of kessler physics on a coarse MPAS grid, run with both intel and gnu
```
<test compset="FKESSLER" grid="mpasa480_mpasa480" name="SMS_Ln2" testmods="cam/outfrq_kessler_mpas_derecho_nooutput/">
<machines>
<machine name="derecho" compiler="intel" category="aux_sima"/>
<machine name="derecho" compiler="gnu" category="aux_sima"/>
</machines>
<options>
<option name="wallclock">00:10:00</option>
<option name="comment">GNU build test for MPAS dycore (with Kessler physics)</option>
</options>
</test>
```

## Github continuous integration testing
The following tests/linters are run automatically on pull requests to `develop` in github. You can see previous runs of the CI tests [here](https://github.com/ESCOMP/CAM-SIMA/actions)

Expand Down Expand Up @@ -81,76 +171,4 @@ hemco hemco-cesm2_0_hemco_3_9_0 7bd8358 is out of sync with .gitmodules hemco-ce
Sparse checkout file missing (check the path):
```
mpas sparse checkout file .mpas_sparse_checkout not found
```

## Regression Testing
### Running the regression tests (manual)

**NOTE: Regression testing on Derecho should be done for every PR before merging!**

Users can manually run regression tests on Derecho to ensure that the model builds correctly in various configurations. The tests can be run with a local copy of CAM-SIMA by using the `test_driver.sh` script under `$CAM-SIMA/test/system`. To run the tests associated with a particular compiler option one can do the following commands:

For running GNU tests*:
```
env CAM_FC=gnu ./test_driver.sh -f
```

For running Intel tests*:
```
env CAM_FC=intel ./test_driver.sh -f
```

Running the script will produce a directory in your scratch space labeled `aux_sima_<CAM_FC>_<timestamp>`, where `<CAM_FC>` is the compiler you chose, and `<timestamp>` is the timestamp (starting with the date) of when the tests were started, along with a job submitted to the local batch system.

Inside the directory you should see an executable labeled `cs.status.*`. Running that command after the submitted job has finished will display the test results. Currently for all tests everything should be labeled `PASS` except the `SUBMIT` step, which should be labeled `PEND`. Any other label indicates that a test may have failed, and should be investigated.

Finally, the tests themselves are listed in `<CAM-SIMA>/cime_config/testdefs/testlist_cam.xml`. Any files that need to be included in order for the tests to run properly are located in `<CAM-SIMA/cime_config/testdefs/testmods_dirs/cam/outfrq_XXX`, where `XXX` is the name of the test. Additional information on the CIME testing system, which is what this testing infrastructure is built on, can be found [online here](https://esmci.github.io/cime/versions/master/html/users_guide/testing.html).

*Note: you may also have to include the environment variable `CAM_ACCOUNT` on derecho, which points to your account key

### Adding a new regression test

The test list can be found here: `$CAM-SIMA/cime_config/testdefs/testlist_cam.xml`

- If you are adding a new machine, compiler or category for an existing test, add a new `<machine>` XML entry to that `<test>` entry
- If you are adding a fully new test, add a *new* `<test>` XML entry with the following structure:
```
<test compset="<COMPSET_NAME>" grid="<GRID_ALIAS>" name="<TEST_TYPE>_<TEST_MOD>" testmods="<RELPATH_TO_TESTMODS_DIR>">
<machines>
<machine name="<MACH_NAME>" compiler="<COMPILER>" category="<TEST_CATEGORY>"/>
</machines>
<options>
<option name="comment">COMMENT HERE</option>
<option name="wallclock">WALLCLOCK_TIME</option>
</options>
</test>
```

- `<COMPSET_NAME>`: component set alias (or long name) - you can see more about compsets [here](../usage/creating-a-case.md)
- `<GRID_ALIAS>`: model grid/resolution you'd like to run on - you can see more about grids [here](../usage/creating-a-case.md)
- **Try to use the lowest/coarsest resolution that will still accomplish your testing goals**
- `<TEST_TYPE>`: type of test to be run. You can find the testing types [here](https://esmci.github.io/cime/versions/master/html/users_guide/testing.html#testtype).
- `<TEST_MOD>`: test modifier that changes the default behavior of the test type. More [here](https://esmci.github.io/cime/versions/master/html/users_guide/testing.html#modifiers)
- **Unless a longer run is necessary to exercise the code you are testing, run for 9 timesteps (_Ln9)**
- `<RELPATH_TO_TESTMODS_DIR>`: relative path to the testmods directory for this run; usually looks something like `"cam/some_directory_name/"`
- The testmods directory will contain any namelist mods and XML configuration variable changes for the test (`user_nl_cam` and/or `shell_commands`)
- testmods directories can be found in `$CAM-SIMA/cime_config/testdefs/testmods_dirs/cam/`
- `<MACH_NAME>`: machine name (options: `derecho`, `izumi`, `casper`)
- `<COMPILER>`: compiler to be used (options: `gnu`, `nag`, `intel`, `nvhpc`)
- `<TEST_CATEGORY>`: group of tests that this test belongs to - the default run by `test_driver.sh` is `aux_sima` (which is run for each PR to CAM-SIMA)
- `WALLCLOCK_TIME`: maximum amount of time that the job will be allowed to run

Here is an example test entry for a 2-timestep smoke test of kessler physics on a coarse MPAS grid, run with both intel and gnu
```
<test compset="FKESSLER" grid="mpasa480_mpasa480" name="SMS_Ln2" testmods="cam/outfrq_kessler_mpas_derecho_nooutput/">
<machines>
<machine name="derecho" compiler="intel" category="aux_sima"/>
<machine name="derecho" compiler="gnu" category="aux_sima"/>
</machines>
<options>
<option name="wallclock">00:10:00</option>
<option name="comment">GNU build test for MPAS dycore (with Kessler physics)</option>
</options>
</test>
```

```
Loading

0 comments on commit 6d61829

Please sign in to comment.