Change bilinear map for low-res coupled case to use TempestRemap

[rljacob]

The bilinear map used for the ne30np4.pg2 and EC30to60E2r2 grid combination (and only for mapping atmosphere states to the ocean mesh) is changed to one generated using TempestRemap. This will allow coupling with offline maps to better match coupling with online maps which use the same algorithm from TempestRemap to calculate the weights.

The answer changes are a little more then roundoff but not climate changing.

[non-BFB] for cases using ne30np4.pg2 and EC30to60E2r2

[rljacob]

To help confirm there's no significant climate change, I did 2 50-year runs. The first used the hash of master this branch started from (a pre-v3 version from July). The second run changes only the a2o states mapping file.
case longname: 1850SOI_EAM%CMIP6_ELM%SPBC_MPASSI_MPASO_MOSART_SGLC_SWAV_SIAC_SESP
Last 40 years used for diags.

Comparison with observations (both e3sm_diags and mpas_analysis)
reference case: https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/jacob/E3SM/WCYCLref88ee186/
with new mapping file: https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/jacob/E3SM/WCYCLtrbilin/

model-vs-model: https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/jacob/E3SM/WCYCLtrbilin/e3sm_diags/atm_monthly_180x360_aave_mvm/model_vs_model_0011-0050/viewer/)

[ndkeen]

Could this potentially change the MPI communication pattern? And have a performance impact?

[mahf708]

Quite unfortunate that the tests were done on pre-v3 (added the mvm to your comment). I think the results make since, especially that the surface radiation terms seem to be more influenced than the TOA terms. As far as I understand, we don't have a category for "little more then roundoff but not climate changing", but yeah, I guess that's the case here (for a 10-year case, I usually would argue that a change of RESTOM >0.05 W/m2 is my arbitrary cutoff for CC, <0.01 for non-CC, and you have 0.037...)

[ndkeen]

Just trying it:

Model cpl missing file atm2ocn_smapname = '/global/cfs/cdirs/e3sm/inputdata/cpl/gridmaps/ne30pg2/map_ne30pg2_to_EC30to60E2r2_bilin.tr.230522.nc'
Trying to download file: 'cpl/gridmaps/ne30pg2/map_ne30pg2_to_EC30to60E2r2_bilin.tr.230522.nc' to path '/global/cfs/cdirs/e3sm/inputdata/cpl/gridmaps/ne30pg2/map_ne30pg2_to_EC30to60E2r2_bilin.tr.230522.nc' using SVN protocol.
svn export failed with output:  and errput svn: E170000: URL 'https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata/cpl/gridmaps/ne30pg2/map_ne30pg2_to_EC30to60E2r2_bilin.tr.230522.nc' doesn't exist

[mahf708]

Actually, looking at some of the changes, e.g., SST and TREFHT, the picture becomes more complex...

[rljacob]

I didn't have the new file on the input data server. Its there now.

There should be no significant change in performance.

[rljacob]

Those diffs above look like they're from internal variability.

[jonbob]

@rljacob -- can you run mpas-analysis as well on the two tests?

[rljacob]

@jonbob its available at the first two links in #5922 (comment)

[jonbob]

Thanks @rljacob -- I was hoping for output showing the model vs model comparison

[mahf708]

Those diffs above look like they're from internal variability.

I wouldn't dismiss them simply like that. Do you have any supporting evidence or additional information the rest of us don't have? Any explanation for why the changes are concentrated in the polar regions? Polar amplification in the other direction?

Anyway, this is largely not an atmosphere issue, so I can't really judge what's going on and how much is within an acceptable range for non-CC.

[jonbob]

The chk_map feature of ncks shows little difference between the mapping files:

map_ne30pg2_to_EC30to60E2r2_bilin.tr.230522.nc

Sparse-matrix size n_s: 947395
Weight min S(278402):  1.6005026460413385e-07 from cell [4975,-9.70992,+84.7499] to [69629,-11.147,+83.2499]
Weight max S(375052):  9.9811526960282348e-01 from cell [4632,-18.5695,+98.2504] to [80234,-18.5697,+98.253]
Ignored weights (S=0.0): 0

map_ne30pg2_to_EC30to60E2r2_bilin.201005.nc

Sparse-matrix size n_s: 947503
Weight min S(411022):  3.3873999935689913e-08 from cell [7722,-31.6084,+167.248] to [102757,-0.574723,+2.89296]
Weight max S(818183):  9.9751865988113342e-01 from cell [8148,-18.0487,+215.252] to [204535,-0.315032,+3.75681]
Ignored weights (S=0.0): 0

The conservation and consistency metrics are very close as well:

map_ne30pg2_to_EC30to60E2r2_bilin.tr.230522.nc

Conservation metrics (column-sums of area_b-weighted weights normalized by area_a) and errors
Perfect metrics for global unmasked Grid B are avg = min = max = 1.0, mbs = rms = sdn = 0.0:
frac_a avg: 0.7072184275934920 = 1.0-2.9e-01 // Mean
frac_a min: 0.0000000000000000 = 1.0-1.0e+00 // Minimum in grid A cell [448,-32.0839,+20.253]
frac_a max: 1.0537839099252229 = 1.0+5.4e-02 // Maximum in grid A cell [14289,+35.981,+231.742]
frac_a mbs: 0.2957173543508281 =     3.0e-01 // Mean absolute bias from 1.0
frac_a rms: 0.5238912650156845 =     5.2e-01 // RMS relative to 1.0
frac_a sdn: 0.4344533885621285 =     4.3e-01 // Standard deviation

Consistency metrics (row-sums of weights) and errors
Perfect metrics for global unmasked Grid A are avg = min = max = 1.0, mbs = rms = sdn = 0.0:
frac_b avg: 1.0000000000000000 = 1.0-0.0e+00 // Mean
frac_b min: 0.9999999999999998 = 1.0-2.2e-16 // Minimum in grid B cell [39,-1.18166,+63.4901]
frac_b max: 1.0000000000000002 = 1.0+2.2e-16 // Maximum in grid B cell [20,-57.6121,+166.962]
frac_b mbs: 0.0000000000000000 =     2.6e-17 // Mean absolute bias from 1.0
frac_b rms: 0.0000000000000001 =     6.7e-17 // RMS relative to 1.0
frac_b sdn: 0.0000000000000001 =     6.7e-17 // Standard deviation

map_ne30pg2_to_EC30to60E2r2_bilin.201005.nc

Conservation metrics (column-sums of area_b-weighted weights normalized by area_a) and errors
Perfect metrics for global unmasked Grid B are avg = min = max = 1.0, mbs = rms = sdn = 0.0:
frac_a avg: 0.7072170749701379 = 1.0-2.9e-01 // Mean
frac_a min: 0.0000000000000000 = 1.0-1.0e+00 // Minimum in grid A cell [448,-32.0839,+20.253]
frac_a max: 1.0537575764954485 = 1.0+5.4e-02 // Maximum in grid A cell [14289,+35.981,+231.742]
frac_a mbs: 0.2956479596235631 =     3.0e-01 // Mean absolute bias from 1.0
frac_a rms: 0.5238901756450125 =     5.2e-01 // RMS relative to 1.0
frac_a sdn: 0.4344511632773080 =     4.3e-01 // Standard deviation

Consistency metrics (row-sums of weights) and errors
Perfect metrics for global unmasked Grid A are avg = min = max = 1.0, mbs = rms = sdn = 0.0:
frac_b avg: 1.0000000000000000 = 1.0-0.0e+00 // Mean
frac_b min: 0.9999999999999997 = 1.0-3.3e-16 // Minimum in grid B cell [152418,+1.55894,+5.5996]
frac_b max: 1.0000000000000002 = 1.0+2.2e-16 // Maximum in grid B cell [17,-0.327076,+1.35338]
frac_b mbs: 0.0000000000000000 =     1.8e-17 // Mean absolute bias from 1.0
frac_b rms: 0.0000000000000001 =     5.4e-17 // RMS relative to 1.0
frac_b sdn: 0.0000000000000001 =     5.4e-17 // Standard deviation

[jonbob]

@rljacob -- I can run the mpas-analysis comparison if you want

[rljacob]

@jonbob thanks for posting the nco stats. Yes please run the mpas-analysis comparison. It will take me a while to figure out how to do it.

[jonbob]

Model-to-model comparison with MPAS-Analysis is at MPAS-Analysis output for: WCYCLtrbilin

[rljacob]

i think I'll do some F-case runs. They'll show a change because the a2o results are used in the ocean-atmosphere flux calculation. But there'll be less internal variability.

[jonbob]

The differences in results look on the order of what we expect as ensemble-member level, and the ncks output makes these mapping files look similar in precision as those made with esmf.

[mahf708]

@rljacob did you end up running F2010 cases? If not, do you want me to run something for you?

(assuming you still want them...)

[mahf708]

This PR has minimal effect on aerosol ERF (table below in W/m2). This test is also a potential data point for assessing how CC this PR is. This PR is not CC, as Jon has already indicated.

Description	TOT	ARI	ACI
baseline	-0.736	+0.096	-0.926
PR 5922	-0.726	+0.098	-0.928

(If you add the numbers above, they don't conserve; that's because there's a term left out --- it is a residual term to do with the surface (mostly) and it makes sense it's carrying most of the changes here.)

[rljacob]

@mahf708 No I haven't run any F-cases yet. Any runs you do are appreciated.

[mahf708]

Hi Rob, I did two quick (15-month F2010 nudged) runs and reported their results above. I think this PR is in "round-off" territory (like you and Jon already indicated) from the perspective of forcing I care about. It is difficult to examine any weird feedback coming from this PR, but I would bet it is relatively small compared to other things going on. I could/would proceed to performing some 11-year F2010 runs, but I doubt we will learn much more from them...

[vijaysm]

Could this potentially change the MPI communication pattern? And have a performance impact?

The nnz in the TempestRemap bilinear map is about the same (but not exactly the same as ESMF bilinear map). But I do not expect any sort of significant bump in parallel performance, if at all. @ndkeen

[rljacob]

Closing this because the new TR bilinear maps will be introduced with the new ocean meshes. PR #6060, PR #5931, PR#