Add FATES cross-grid seed dispersal mechanism

[glemieux]

This PR enables FATES to disperse seeds across neighboring grid cells using MPI. The API update includes calls to new fates dispersal procedures and as such should be coordinated with NGEET/fates#1005.

[B4B] except for FATES

[bishtgautam]

@glemieux Did you forget to update the FATES submodule?

[bishtgautam]

Btw, this is a very exciting PR and I'm interested to find out the approach you and the FATES team took for this work.

[glemieux]

@glemieux Did you forget to update the FATES submodule?

I was going to push the final update once we've integrated NGEET/fates#1005 so I can point to the specific fates main commit. For now I was planning on only updating locally when I run the land developer tests. I'm more than happy to push that interim update though.

[glemieux]

@bishtgautam FYI I also have a new parameter file to be uploaded per NGEET/fates#1005.

[bishtgautam]

I'll be happy to upload the new parameter file on the inputdata server.

[rljacob]

status: waiting on a corresponding FATES PR.

[rljacob]

status: still waiting on a FATES PR.

[glemieux]

@bishtgautam this will need to come in after the API25.5 update in #5802.

[rljacob]

notes: should be done soon. FATES-side tests are progressing.

[rljacob]

Waiting on #5849

[glemieux]

@peterdschwartz now that #5849 is merged, we can start moving forward with this again. I'm working on an ERS fix on the fates-side that accompanies this, but that should be done in the next day or so. After that I'll rebase and retest this.

[rljacob]

status: working on an exact restart issue.

[rljacob]

@glemieux any update on restart issue?

[glemieux]

@glemieux any update on restart issue?

@rljacob There is an issue with the PEM and ERP for this. The current plan is to make this a known issue and integrate into fates main/hlm master branches and address this later.

[rljacob]

update: currently being worked on.

[glemieux]

Regression testing against master on pm-cpu is in progress. The e3sm_land_debug tests are complete and coming back as b4b against master with two exceptions:

• SMS_Ld20_D.f45_f45.IELMFATES.pm-cpu_gnu.elm-fates_rd
• ERS_D.ne11_oQU240.I20TRELM.pm-cpu_gnu

The differences in the fates testmod are expected as this PR includes a number of known answer changing scientific updates for fates science. The non-fates test is only showing DIFFs in the cpl. @peterdschwartz is this expected?

Location on perlmutter: /pscratch/sd/g/glemieux/e3sm-tests/pr5555-eld-debug.fates.pm-cpu.gnu.C9a524415e4-F1d18bb69.DEBUG

Regression test results for e3sm_land_developer suite is forthcoming.

[glemieux]

The e3sm_land_developer regression tests are complete and showing similar results to the debug tests. Nearly everything is b4b except for the fates testmods, which is as expeced, and the ERS_D.ne11_oQU240.I20TRELM.pm-cpu_gnu testmod is again coming back with a DIFF against the master baseline.

There is also a fates test that is not passing exart restart, but this is expected (see NGEET/fates#1106) and will be corrected in a forthcoming PR.

Location on perlmutter: /pscratch/sd/g/glemieux/e3sm-tests/pr5555-eld-nodebug.fates.pm-cpu.gnu.C9a524415e4-F1d18bb69

[glemieux]

Testing with NGEET/fates#1109 in progress to address known ERS issue.

[rljacob]

Looking at a restart issue on FATES side. Otherwise ready.

[glemieux]

@peterdschwartz FATES restart issue has been resolved via e7921a0. I rebased against master and I'm rerunning e3sm_land_developer regression tests on perlmutter.

[glemieux]

@peterdschwartz this is finally all good to go. Regression tests on perlmutter are coming back B4B with the known exception of the fates testmods.

Regression test results: /global/homes/g/glemieux/scratch/e3sm-tests/pr5555-pr1114-eld-intel.fates.pm-cpu..Ce7921a0c0d-F1874511a

[peterdschwartz]

merged to next