Skip to content

Commit

Permalink
Merge branch 'jgfouca/scream_downstream_2024_11_07' into next (PR #6739)
Browse files Browse the repository at this point in the history
Merge 2 for this PR.

Fixes some PAM stuff.

[BFB]

* jgfouca/scream_downstream_2024_11_07:
  fix formatting of changed md files
  update pam to match new p3 signature
  • Loading branch information
jgfouca committed Nov 11, 2024
2 parents 86d0dc4 + 4cf27bc commit 2c89145
Show file tree
Hide file tree
Showing 13 changed files with 346 additions and 225 deletions.
6 changes: 3 additions & 3 deletions components/eamxx/docs/developer/ci_nightly.md
Original file line number Diff line number Diff line change
@@ -1,17 +1,17 @@
# Continuous Integration and Nightly Testing

## Autotester ##
## Autotester

EAMxx using github actions and a Sandia product called Autotester 2
to run CI testing on a CPU and GPU machine for every github pull
request. By default, we run the e3sm_scream_v1_at suite and the
standalone eamxx tests (test-all-scream).

## Nightly overview, CDash ##
## Nightly overview, CDash

Our nightly testing is much more extensive than the CI testing. You
can see our dashboard here under the section "E3SM_SCREAM":
https://my.cdash.org/index.php?project=E3SM
<https://my.cdash.org/index.php?project=E3SM>

We run a variety of CIME test suites and standalone testing on a number
of platforms. We even do some performance testing on frontier.
33 changes: 21 additions & 12 deletions components/eamxx/docs/developer/cime_testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,43 +4,48 @@ Full model system testing of eamxx is done through CIME test cases
(much like the rest of E3SM).

We offer a number of test suites, including:

* e3sm_scream_v0: Test the full set of V0 (pre-C++) tests
* e3sm_scream_v1: Test the full set of V1 (C++) tests
* e3sm_scream_v1_at: A smaller and quicker set of tests for autotesting
* e3sm_scream_hires: A small number of bigger, longer-running tests to measure performance

Example for running a suite:
```
% cd $repo/cime/scripts
% ./create_test e3sm_scream_v1_at --wait

```shell
cd $repo/cime/scripts
./create_test e3sm_scream_v1_at --wait
```

Example for running a single test case:
```
% cd $repo/cime/scripts
% ./create_test SMS.ne4_ne4.F2010-SCREAMv1 --wait

```shell
cd $repo/cime/scripts
./create_test SMS.ne4_ne4.F2010-SCREAMv1 --wait
```

There are many behavioral tweaks you can make to a test case, like
changing the run length, test type, etc. Most of this is not specific
to eamxx and works for any CIME case. This generic stuff
is well-documentated here:
http://esmci.github.io/cime/versions/master/html/users_guide/testing.html
<http://esmci.github.io/cime/versions/master/html/users_guide/testing.html>

When it comes to things specific to eamxx, you have grids, compsets, and
testmods.

Common EAMxx grids are:

* ne4_ne4 (low resolution)
* ne4pg2_ne4pg2 (low resolution with phys grid)
* ne30_ne30 (med resolution)
* ne30pg2_ne30pg2 (med resolution with phys grid)
* ne1024pg2_ne1024pg2 (ultra high with phys grid)

More grid info can be found here:
https://acme-climate.atlassian.net/wiki/spaces/DOC/pages/933986549/ATM+Grid+Resolution+Summary
<https://acme-climate.atlassian.net/wiki/spaces/DOC/pages/933986549/ATM+Grid+Resolution+Summary>

Common EAMxx compsets are:

* F2010-SCREAM-LR: V0 low res compset with eamxx V0 atmosphere
* F2010-SCREAMv1: V1 standard compset with eamxx V1 atmosphere
* FIOP-SCREAMv1-DP: V1 with dpxx (doubly-periodic lateral boundary condition in C++)
Expand All @@ -50,10 +55,14 @@ Full info on supported compsets can be found by looking at this file:
`$scream_repo/components/eamxx/cime_config/config_compsets.xml`

Common EAMxx testmods are:
* small_kernels: Enable smaller-granularity kernels, can improve performance on some systems
* scream-output-preset-[1-6]: Our 6 output presets. These turn some combination of our three output streams (phys_dyn, phys, and diags),

* small_kernels: Enable smaller-granularity kernels,
can improve performance on some systems
* scream-output-preset-[1-6]: Our 6 output presets.
These turn some combination of our three output streams
(phys_dyn, phys, and diags),
various remaps, etc.
* bfbhash: Turns on bit-for-bit hash output: https://acme-climate.atlassian.net/wiki/spaces/NGDNA/pages/3831923056/EAMxx+BFB+hashing
* bfbhash: Turns on bit-for-bit hash output: <https://acme-climate.atlassian.net/wiki/spaces/NGDNA/pages/3831923056/EAMxx+BFB+hashing>

More info on running EAMxx can be found here:
https://acme-climate.atlassian.net/wiki/spaces/DOC/pages/3386015745/How+To+Run+EAMxx+SCREAMv1
<https://acme-climate.atlassian.net/wiki/spaces/DOC/pages/3386015745/How+To+Run+EAMxx+SCREAMv1>
95 changes: 54 additions & 41 deletions components/eamxx/docs/developer/field.md
Original file line number Diff line number Diff line change
@@ -1,45 +1,58 @@
## Field
# Field

In EAMxx, a `Field` is a data structure holding two things: pointers to the data and pointers to metadata.
Both the data and metadata are stored in `std::shared_ptr` instances, to ensure consistency across all copies
of the field. This allows for fast shallow copy semantic for this class.
In EAMxx, a `Field` is a data structure holding two things: pointers to the
data and pointers to metadata. Both the data and metadata are stored in
`std::shared_ptr` instances, to ensure consistency across all copies of
the field. This allows for fast shallow copy semantic for this class.

The data is stored on both CPU and device memory (these may be the same, depending on the Kokkos
backend). In EAMxx, we always assume and guarantee that the device data is up to date. That implies that the data
must be explicitly synced to host before using it on host, and explicitly synced to device after host manipulation,
in order to ensure correctness. In order to access the data, users must use the `get_view`/'get_strided_view' methods,
which takes two template arguments: the data type, and an enum specifying whether CPU or device data is needed.
The data type is used to reinterpret the generic pointer stored inside to a view of the correct scalar type and layout.
It is a possibly const-qualified type, and if the field was marked as "read-only", the method ensures that the
provided data type is const. A read-only field can be created via the `getConst` method, which returns a shallow
copy of the field, but marked as read-only. The enum specifying host or device data is optional, with device being the default.
The data is stored on both CPU and device memory (these may be the same,
depending on the Kokkos backend). In EAMxx, we always assume and guarantee
that the device data is up to date. That implies that the data must be
explicitly synced to host before using it on host, and explicitly synced
to device after host manipulation, in order to ensure correctness.
In order to access the data, users must use the `get_view`/
`get_strided_view` methods, which takes two template arguments:
the data type, and an enum specifying whether CPU or device data is needed.
The data type is used to reinterpret the generic pointer stored inside
to a view of the correct scalar type and layout. It is a possibly
const-qualified type, and if the field was marked as "read-only",
the method ensures that the provided data type is const. A read-only field
can be created via the `getConst` method, which returns a shallow copy of
the field, but marked as read-only. The enum specifying host or device data
is optional, with device being the default.

The metadata is a collection of information on the field, such as name, layout, units, allocation size, and more.
Part of the metadata is immutable after creation (e.g., name, units, or layout), while some metadata can be
partially or completely modified. The metadata is contained in the `FieldHeader` data structure, which contains
four parts:
The metadata is a collection of information on the field, such as name, layout, units,
allocation size, and more. Part of the metadata is immutable after creation (e.g.,
name, units, or layout), while some metadata can be partially or completely modified.
The metadata is contained in the `FieldHeader` data structure, which contains four
parts:

* `FieldIdentifier`: stores the field's name, layout, units, data type, and name of the grid where it's defined.
These information are condensed in a single string, that can be used to uniquely identify a field,
allowing to distinguish between different version of the same field. The layout is stored in the `FieldLayout`
data structure, which includes:
* the field tags: stored as a `std::vector<FieldTag>`, they give context to the field's extents.
* the field dims: stored both as a `std::vector<int>`, as well as a 1d `Kokkos::View`.
* `FieldTracking`: stores information on the usage of the field, as well as its possible connections to other
fields. In particular, the tracked items are:
* the field time stamp: the time stamp when the field was last updated.
* the field accumulation start time: used for fields that are accumulated over several time steps
(or time step subcycles). For instance, it allows to reconstruct fluxes from raw accumulations.
* the providers/customers: lists of atmosphere processes (see below) that respectively require/compute
the field in their calculations.
* the field groups: a list of field groups that this field belongs too. Field groups are used to access
a group of fields without explicit prior knowledge about the number and/or names of the fields.
* `FieldAllocProp`: stores information about the allocation. While the field is not yet allocated, users can
request special allocations for the field, for instance to accommodate packing (for SIMD), which may
require padding. Upon allocation, this information is then used by the Field structure to extract the
actual data, wrapped in a properly shaped `Kokkos::View`. The alloc props are also responsible of tracking
additional information in case the field is a "slice" of a higher-dimensional one, a fact that can affect
how the data is accessed.
* Extra data: stored as a `std::map<std::string,ekat::any>`, allows to catch any metadata that does not fit
in the above structures. This is a last resort structure, intended to accommodate the most peculiar
corner cases, and should be used sparingly.
* `FieldIdentifier`: stores the field's name, layout, units, data type,
and name of the grid where it's defined. These information are condensed
in a single string, that can be used to uniquely identify a field, allowing
to distinguish between different version of the same field.
The layout is stored in the `FieldLayout` data structure, which includes:
* the field tags: stored as a `std::vector<FieldTag>`, they give context to the
field's extents.
* the field dims: stored both as a `std::vector<int>`, as well as a 1d `Kokkos::View`.
* `FieldTracking`: stores information on the usage of the field, as well as its
possible connections to other fields. In particular, the tracked items are:
* the field time stamp: the time stamp when the field was last updated.
* the field accumulation start time: used for fields that are accumulated over
several time steps (or time step subcycles). For instance, it allows to
reconstruct fluxes from raw accumulations.
* the providers/customers: lists of atmosphere processes (see below) that
respectively require/compute the field in their calculations.
* the field groups: a list of field groups that this field belongs too. Field groups
are used to access a group of fields without explicit prior knowledge about the
number and/or names of the fields.
* `FieldAllocProp`: stores information about the allocation. While the field is not
yet allocated, users can request special allocations for the field, for instance
to accommodate packing (for SIMD), which may require padding. Upon allocation,
this information is then used by the Field structure to extract the actual data,
wrapped in a properly shaped `Kokkos::View`. The alloc props are also
responsible of tracking additional information in case the field is a "slice" of
a higher-dimensional one, a fact that can affect how the data is accessed.
* Extra data: stored as a `std::map<std::string,ekat::any>`, allows to catch any
metadata that does not fit in the above structures. This is a last resort structure,
intended to accommodate the most peculiar corner cases, and should be used sparingly.
43 changes: 25 additions & 18 deletions components/eamxx/docs/developer/grid.md
Original file line number Diff line number Diff line change
@@ -1,22 +1,29 @@
## Grids and Remappers
# Grids and Remappers

In EAMxx, the `AbstractGrid` is an interface used to access information regarding the horizontal and vertical
discretization. The most important information that the grid stores is:
In EAMxx, the `AbstractGrid` is an interface used to access information regarding
the horizontal and vertical discretization. The most important information that
the grid stores is:

* the number of local/global DOFs: these are the degrees of freedom of the horizontal grid only. Here,
local/global refers to the MPI partitioning.
* the DOFs global IDs (GIDs): a list of GIDs of the DOFs on the current MPI rank, stored as a Field
* the local IDs (LIDs) to index list: this list maps the LID of a DOF (that is, the position of the DOF
in the GID list) to a "native" indexing system for that DOF. For instance, a `PointGrid` (a class derived from
`AbstractGrid`) is a simple collection of points, so the "native" indexing system coincides with the LIDs.
However, for a `SEGrid` (a derived class, for spectral element grids), the "native" indexing is a triplet
`(ielem,igp,jgp)`, specifying the element index, and the two indices of the Gauss point within the element.
* geometry data: stored as a `std::map<std::string,Field>`, this represent any data that is intrinsically
linked to the grid (either along the horizontal or vertical direction), such as lat/lon coordinates,
vertical coordinates, area associated with the DOF.
* the number of local/global DOFs: these are the degrees of freedom of the
horizontal grid only. Here, local/global refers to the MPI partitioning.
* the DOFs global IDs (GIDs): a list of GIDs of the DOFs on the current MPI rank,
stored as a Field
* the local IDs (LIDs) to index list: this list maps the LID of a DOF (that is,
the position of the DOF in the GID list) to a "native" indexing system for that
DOF. For instance, a `PointGrid` (a class derived from `AbstractGrid`) is a
simple collection of points, so the "native" indexing system coincides with the
LIDs. However, for a `SEGrid` (a derived class, for spectral element grids),
the "native" indexing is a triplet `(ielem,igp,jgp)`, specifying the element
index, and the two indices of the Gauss point within the element.
* geometry data: stored as a `std::map<std::string,Field>`, this represent any
data that is intrinsically linked to the grid (either along the horizontal or
vertical direction), such as lat/lon coordinates, vertical coordinates, area
associated with the DOF.

Grids can also be used to retrieve the layout of a 2d/3d scalar/vector field, which allows certain downstream
classes to perform certain operations without assuming anything on the horizontal grid.
Grids can also be used to retrieve the layout of a 2d/3d scalar/vector field,
which allows certain downstream classes to perform certain operations without
assuming anything on the horizontal grid.

In general, grid objects are passed around the different parts of EAMxx as const objects (read-only).
The internal data can only be modified during construction, which usually is handled by a `GridsManager` object.
In general, grid objects are passed around the different parts of EAMxx as const
objects (read-only). The internal data can only be modified during construction,
which usually is handled by a `GridsManager` object.
2 changes: 0 additions & 2 deletions components/eamxx/docs/developer/index.md
Original file line number Diff line number Diff line change
@@ -1,3 +1 @@
# SCREAM Developer Guide


6 changes: 3 additions & 3 deletions components/eamxx/docs/developer/io.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# Input-Output

In EAMxx, I/O is handled through the SCORPIO library, currently a submodule of E3SM.
The `scream_io` library within eamxx allows to interface the EAMxx infrastructure classes
with the SCORPIO library.
In EAMxx, I/O is handled through the SCORPIO library, currently a submodule of
E3SM. The `scream_io` library within eamxx allows to interface the EAMxx
infrastructure classes with the SCORPIO library.
Loading

0 comments on commit 2c89145

Please sign in to comment.