Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Tracers infrastructure #133

Merged
merged 5 commits into from
Oct 4, 2024
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions components/omega/configs/Default.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,3 +25,6 @@ Omega:
ViscDel2: 1.0e3
VelHyperDiffTendencyEnable: true
ViscDel4: 1.2e11
Tracers:
Base: [Temp, Salt]
Debug: [Debug1, Debug2, Debug3]
593 changes: 304 additions & 289 deletions components/omega/doc/design/Tracers.md

Large diffs are not rendered by default.

225 changes: 225 additions & 0 deletions components/omega/doc/devGuide/Tracers.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,225 @@
(omega-dev-tracers)=

# Tracers

Tracers refer to either heat (temperature) or material carried by a fluid
parcel (e.g., salt, chemical, or biological constituents).

To manage tracer definitions, users will update the `TracerDefs.inc` file
located in the `omega/src/ocn` directory of E3SM. The file contains
Tracer index variables and `defineAllTracers` C++ function defintion that
contains the calls to `define` function per each tracer as shown below:

```c++
inline static I4 IndxTemp = Tracers::IndxInvalid;
inline static I4 IndxSalt = Tracers::IndxInvalid;
inline static I4 IndxMyBGCTracer = Tracers::IndxInvalid;

// Tracer definitions packaged in a defineAllTracers function
static void defineAllTracers() {

define("Temp", ///< [in] Name of tracer
"Potential Temperature", ///< [in] Long name or description
"degree_C", ///< [in] Units
"sea_water_potential_temperature", ///< [in] CF standard Name
-273.15, ///< [in] min valid field value
100.0, ///< [in] max valid field value
1.e33, ///< [in] value for undef entries
IndxTemp); ///< [out] (optional) static index

define("Salt", "Salinity", "psu", "sea_water_salinity", 0.0, 50.0, 1.e33,
IndxSalt);
define("Debug1", "Debug Tracer 1", "none", "none", 0.0, 100.0, 1.e33);
define("Debug2", "Debug Tracer 2", "none", "none", 0.0, 100.0, 1.e33);
define("Debug3", "Debug Tracer 3", "none", "none", 0.0, 100.0, 1.e33);
}
```

To add a new tracer, simply call the `define` function with the appropriate
arguments. Index argument is optional one that allows to access the tracer
data using the given tracer index variable.

The following sections explain the concept and implementation of OMEGA tracers.

## Concepts

### Arrays in Host and Device Memory

All tracers are stored in a vector of 3-dimensional device and host arrays.

```c++
std::vector<Array3DReal> TracerArrays; // Device: multiple time levels
std::vector<HostArray3DReal> TracerArraysH; // Host: multiple time levels
```

Each device and host array in the vector has dimensions corresponding to the
number of tracers, the number of cells in the rank, and the vertical levels.

The host tracer array (`TracerArraysH`) is internally managed, so users
should not directly modify it in most cases.

### Tracer Groups

Each tracer is assigned to a tracer group defined in the OMEGA YAML
configuration file, such as `Default.yml`. A tracer should be assigned to only one group.

To access the member tracers of a group in the code, users can use the
`getGroupRange` function as shown below:

```c++
// Retrieves a pair of (group start index, group length)
static I4
getGroupRange(std::pair<I4, I4> &GroupRange, ///< [out] Group range
const std::string &GroupName ///< [in] Group name
);
```

A typical use of this function might look like:

```c++
// Get group range
std::pair<I4, I4> GroupRange;
I4 Err = Tracers::getGroupRange(GroupRange, GroupName);

// Unpack group range
auto [StartIndex, GroupLength] = GroupRange;

// Get all tracers at the current time level (0)
Array3DReal TracerArray;
Err = OMEGA::Tracers::getAll(TracerArray, 0);
if (Err != 0)
LOG_ERROR("getAll returns an error code: {}.", Err);


OMEGA::parallelFor(
"ComputeGroupTendency",
{GroupLength, NCells, NVertLayers},
KOKKOS_LAMBDA(Int iIndex, Int iCell, Int iVert) {
int iTracer = TracerArray[iIndex + StartIndex];
// Perform operations on TracerArray(iTracer, iCell, iVert);
});
```

### Time Levels

During the initialization of Tracers, the number of time levels is determined,
and the length of the tracer vectors (`TracerArrays` and `TracerArraysH`)
is set accordingly. The Tracers class internally manages the index of the
current time level in the arrays. To access a specific time level in the
arrays, users will use the integer "0" or negative integers. For example,
the following code retrieves the device tracer arrays for the current and
previous time levels:

```c++
// Get all tracers at the current time level (0)
Array3DReal CurrentTracerArray;
I4 Err1 = OMEGA::Tracers::getAll(CurrentTracerArray, 0);

// Get all tracers at the previous time level (-1)
Array3DReal PreviousTracerArray;
I4 Err2 = OMEGA::Tracers::getAll(PreviousTracerArray, -1);
```

## Initialization and Finalization

To use Tracers, the class should first be initialized by calling the
`init()` function and finalized by calling `finalize()`. These function
calls are typically handled within the initialization and finalization of
OMEGA itself, so users may not need to call them separately.

## Key APIs

### `getAll` and `getAllHost`

These functions return all device and host tracer arrays, respectively. If
the specified `TimeLevel` does not exist, they return a negative integer.

```c++
static HostArray3DReal getAllHost(
HostArray3DReal &TracerArrayH, ///< [out] tracer host array
const I4 TimeLevel ///< [in] Time level index
);

static Array3DReal getAll(
Array3DReal &TracerArray, ///< [out] tracer device array
const I4 TimeLevel ///< [in] Time level index
);
```

### `getGroupRange`

`getGroupRange` returns a pair of two `I4` values representing the group
start index and the group length. If the group is not found, it returns
negative integers.

```c++
static I4 getGroupRange(
std::pair<I4, I4> &GroupRange, ///< [out] Group range
const std::string &GroupName ///< [in] Group name
);
```

### `getByIndex` and `getHostByIndex`

These functions return device and host tracer arrays, respectively, based
on the `TracerIndex`. If the specified `TimeLevel` and/or `TracerIndex`
does not exist, they return a negative integer.

```c++
static Array2DReal getByIndex(
Array2DReal &TracerArray, ///< [out] tracer device array
const I4 TimeLevel, ///< [in] Time level index
const I4 TracerIndex ///< [in] Global tracer index
);

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update docs for the changes in retrieval functions

Copy link
Author

@grnydawn grnydawn Sep 28, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated.

static HostArray2DReal getHostByIndex(
HostArray2DReal &TracerArrayH, ///< [out] tracer host array
const I4 TimeLevel, ///< [in] Time level index
const I4 TracerIndex ///< [in] Global tracer index
);
```

### `getIndex`

`getIndex` returns the index of the tracer specified by the `TracerName`
argument. If the tracer is not found, it returns a negative integer.

```c++
static I4 getIndex(
I4 &TracerIndex, ///< [out] Tracer index
const std::string &TracerName ///< [in] Tracer name
);
```

### `getFieldByIndex`

`getFieldByIndex` returns the `Field` object associated with the tracer
specified by the `TracerIndex`. If not found, it returns `nullptr`.

```c++
// Returns a field by tracer index. If it does not exist, returns nullptr
static std::shared_ptr<Field>
getFieldByIndex(const I4 TracerIndex ///< [in] Global tracer index
);
```

### `updateTimeLevels`

`updateTimeLevels` increments the current time level by one. If the current
time level exceeds the number of time levels, it returns to `0`. It also
exchanges the halo cells of all tracers at the current time level and updates
the associated field with the new tracer arrays.

```c++
/// Increment time levels
static I4 updateTimeLevels();
```

### `getNumTracers`

`getNumTracers` returns the total number of tracers used in the simulation.

```c++
// Get total number of tracers
static I4 getNumTracers();
```
2 changes: 2 additions & 0 deletions components/omega/doc/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@ userGuide/OceanState
userGuide/TimeMgr
userGuide/TimeStepping
userGuide/Reductions
userGuide/Tracers
```

```{toctree}
Expand Down Expand Up @@ -76,6 +77,7 @@ devGuide/OceanState
devGuide/TimeMgr
devGuide/TimeStepping
devGuide/Reductions
devGuide/Tracers
```

```{toctree}
Expand Down
61 changes: 61 additions & 0 deletions components/omega/doc/userGuide/Tracers.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
(omega-user-tracers)=

# Tracers

Tracers refer to either heat (temperature) or material carried by a fluid
parcel (e.g., salt, chemical, or biological constituents).

## Updating tracer defintions in `TracerDefs.inc` file

To manage tracer definitions, users will update the `TracerDefs.inc` file
located in the `omega/src/ocn` directory of E3SM. The file contains
Tracer index variables and `defineAllTracers` C++ function defintion that
contains the calls to `define` function per each tracer as shown below:

```c++
inline static I4 IndxTemp = Tracers::IndxInvalid;
inline static I4 IndxSalt = Tracers::IndxInvalid;
inline static I4 IndxMyBGCTracer = Tracers::IndxInvalid;

// Tracer definitions packaged in a defineAllTracers function
static void defineAllTracers() {

define("Temp", ///< [in] Name of tracer
"Potential Temperature", ///< [in] Long name or description

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

noting here and in similar places, for TEOS-10 (new EOS) this should actually be Conservative Temperature not Potential Temperature and sea_water_conservative_temperature for the CF name

Similarkly salinity is now Absolute Salinity and not Practical Salinity units are g/kg instead of psu

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could be changed later.

"degree_C", ///< [in] Units
"sea_water_potential_temperature", ///< [in] CF standard Name
-273.15, ///< [in] min valid field value
100.0, ///< [in] max valid field value
1.e33, ///< [in] value for undef entries
IndxTemp); ///< [out] (optional) static index

define("Salt", "Salinity", "psu", "sea_water_salinity", 0.0, 50.0, 1.e33,
IndxSalt);
define("Debug1", "Debug Tracer 1", "none", "none", 0.0, 100.0, 1.e33);
define("Debug2", "Debug Tracer 2", "none", "none", 0.0, 100.0, 1.e33);
define("Debug3", "Debug Tracer 3", "none", "none", 0.0, 100.0, 1.e33);
}
```

To add a new tracer, simply call the `define` function with the appropriate
arguments. Index argument is optional one that allows to access the tracer
data using the given tracer index variable.

## Selecting tracers using YAML configuration file

Note that not all tracers defined in `TracerDefs.inc` will be used during
a simulation. To select tracers and groups of tracers, users will configure
YAML files in OMEGA, such as `Default.yml` in the `omega/configs` directory,
as shown below:

```yaml
omega:
Tracers:
Base: [Temp, Salt]
Debug: [Debug1, Debug2, Debug3]
[other individual tracers or groups as needed]
```

In the above example, two tracer groups (Base and Debug) are selected. The
Base group includes the `Temp` and `Salt` tracers, while the Debug group
includes `Debug1`, `Debug2`, and `Debug3`.
37 changes: 37 additions & 0 deletions components/omega/src/ocn/TracerDefs.inc
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
//===-- ocn/TracerDefs.inc --------------------*- C++ -*-===//
//
/// \file
/// \brief OMEGA Tracer Definition File
///
/// This file defines all OMEGA tracers. It is imported using
/// the C++ #include preprocessor directive in Tracers.cpp. Only the tracer
/// definitions selected in the Tracers section of the OMEGA YAML configuration
/// file will be allocated into memory.
//===----------------------------------------------------------------------===//

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One of the reasons for the getByIndex function was to avoid the more expensive string comparison in getByName. In the current implementation, we would still need to get the index by name first and then use the index. I'm wondering whether in the include file, we could include a list of shared, public indices for each tracer (set when they are defined). Like IndxTemp, IndxSalt, IndxDebug1, etc. Since we only want to edit one file when adding tracers, that would mean putting the indices at the top of this include file and including it in Tracers.h. That means we would have to wrap all the defines in a defineAll function that is called where the current inc file is included.

What do others think?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@philipwjones , what are the benefits of having a list of shared and public indices for each tracers?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By way of explanation, this is what I'm thinking. The TracersDef.inc file would look like:

// Index defined for each tracer. The value of these tracer indices are still set in Tracers::init()  and
// tracers that have not been selected would have standard invalid value: Tracer::IndxInvalid = -1
static int IndxTemp;
static int IndxSalt;
static int IndxMyBGCTracer;
static int IndxAnotherBGCTracer;
static int IndxDebug1;
static int IndxDebug2;
etc. for all defined tracers

// Tracer definitions packaged in a defineAllTracers function
static void defineAllTracers() {

  // here we put all the define calls as they exist in the current TracersDef.inc...
}

The TracersDef.inc file would then be included in the Tracers.h file in the public part of the Tracer class definition so that they are all available using eg Tracer::IndxSalt. In the Tracers.cpp file where the include currently sits, we would replace the include statement with a call to defineAllTracers. The init function would still assign consecutive indices to groups.

This would allow everyone to access the indices directly, providing speedier access without an explicit call to search for the index by name. I realize this is more of a Fortran style and not the usual C++ approach but I think it is both simpler and faster than a search by name or having all modules do the lookup on init.

This is just a proposal - others should weigh in...

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@philipwjones, I tried to implement your suggested code in this branch (https://github.com/grnydawn/Omega/tree/ykim/omega/tracerinfra-def) but encountered a linking issue:

ld.lld: error: undefined symbol: OMEGA::Tracers::IndxTemp
>>> referenced by Tracers.cpp
>>>               Tracers.cpp.o:(OMEGA::Tracers::defineAllTracers()) in archive ../src/libOmegaLib.a

To resolve the linker error, I added the following near the top of Tracers.cpp:

I4 Tracers::IndxTemp = 0;
I4 Tracers::IndxSalt = 0;

This fix worked, but it makes me think that an additional user-provided include file might be necessary. I’m still learning C++, so I may be missing something.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's right - forgot about that. The header just has the forward declaration but we still need to initialize in the implementation (.cpp) file. We might be able to include the same file in both with some creative ifdefs but this is starting to get more complicated than I had intended. Let me think about it some more

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's stick with your initial implementation for now

Copy link
Member

@mwarusz mwarusz Oct 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am bit late to this conversation, but if you declare the member index variables to be inline static then you can initialize them in the header file. Like this:

inline static int IndxTemp = -1;

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to clarify, and echo a comment from @xylar at the last meeting: this is just a convenience/optimization that can be done for certain tracers and not a requirement, correct?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mwarusz, thanks for the tip about using inline. I have updated Tracers with the inline, and it worked well.

@philipwjones, I am using the variable names for the static indexes from your example code above. If you need to add more tracer index variable names in "TracerDefs.inc", let me know, or we can update the file later.

// Index defined for each tracer. The value of these tracer indices are still
// set in Tracers::init() and tracers that have not been selected would have
// standard invalid value: Tracers::IndxInvalid = -1
inline static I4 IndxTemp = Tracers::IndxInvalid;
inline static I4 IndxSalt = Tracers::IndxInvalid;
inline static I4 IndxMyBGCTracer = Tracers::IndxInvalid;

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove the index entries that don't have a corresponding define function. That way the index list serves as a list of potentially available tracers.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed the unused entries.

// Tracer definitions packaged in a defineAllTracers function
static void defineAllTracers() {

define("Temp", ///< [in] Name of tracer
"Potential Temperature", ///< [in] Long name or description
"degree_C", ///< [in] Units
"sea_water_potential_temperature", ///< [in] CF standard Name
-273.15, ///< [in] min valid field value
100.0, ///< [in] max valid field value
1.e33, ///< [in] value for undef entries
IndxTemp); ///< [out] (optional) static index

define("Salt", "Salinity", "psu", "sea_water_salinity", 0.0, 50.0, 1.e33,
IndxSalt);

define("Debug1", "Debug Tracer 1", "none", "none", 0.0, 100.0, 1.e33);
define("Debug2", "Debug Tracer 2", "none", "none", 0.0, 100.0, 1.e33);
define("Debug3", "Debug Tracer 3", "none", "none", 0.0, 100.0, 1.e33);
}
Loading
Loading