add a pam test to gh/ci suite

[mahf708]

Objective: avoid breaking PAM.
Background: PAM uses some EAMxx internals in specific ways that we often change.
Solution: Add one PAM/MMF2 test to gh/ci such that when specific files are touched, it is triggered to safeguard against breaking the test

[mahf708]

MMF2 tests require unreasonable amount of data that's out of sync with E3SM v3 and EAMxx v1. Thus, won't fix.

[rljacob]

What case were you trying to run?

[mahf708]

This one: SMS_Ln3_P4.ne4pg2_oQU480.F2010-MMF2.ghci-oci_gnu

It tried to download the files in E3SM-Project/containers#17

missing files

Model eam missing file prescribed_ozone_file = '/projects/e3sm/inputdata/atm/cam/ozone/ozone_1.9x2.5_L26_2000clim_c091112.nc'
Model eam missing file rrtmgp_coefficients_file_sw = '/projects/e3sm/inputdata/atm/cam/rad/rrtmgp-data-sw-g112-210809.nc'
Model eam missing file rrtmgp_coefficients_file_lw = '/projects/e3sm/inputdata/atm/cam/rad/rrtmgp-data-lw-g128-210809.nc'
Model eam missing file ncdata = '/projects/e3sm/inputdata/atm/cam/inic/homme/eam_i_mam3_Linoz_ne4np4_L60_c20210917.nc'
Model eam missing file prescribed_volcaero_file = '/projects/e3sm/inputdata/atm/cam/volc/CMIP_DOE-ACME_radiation_average_1850-2014_v3_c20171204.nc'
Model eam missing file prescribed_aero_file = '/projects/e3sm/inputdata/atm/cam/chem/trop_mam/aero/mam4_0.9x1.2_L72_2000clim_c170323.nc'
Model eam missing file aerodep_flx_file = '/projects/e3sm/inputdata/atm/cam/chem/trop_mam/aero/mam3_1.9x2.5_L30_2000clim_c130319.nc'
Model eam missing file mode_defs for pom_a1 = '/projects/e3sm/inputdata/atm/cam/physprops/ocpho_rrtmg_c101112.nc'
Model eam missing file rad_climate for mam3_mode1 = '/projects/e3sm/inputdata/atm/cam/physprops/mam3_mode1_rrtmg_c110318.nc'
Model eam missing file rad_climate for mam3_mode2 = '/projects/e3sm/inputdata/atm/cam/physprops/mam3_mode2_rrtmg_c110318.nc'
Model eam missing file rad_climate for mam3_mode3 = '/projects/e3sm/inputdata/atm/cam/physprops/mam3_mode3_rrtmg_c110318.nc'

... a few problems that made me abandon ship (supporting this test case long term may become a nightmare)

1. One (1.3G) of these files wasn't even on the server (but I copied it from elsewhere on chrys)
2. I think of the files is on the order of ~10G (so the runners would freak out, no space left)
3. It appears MMF2 is using outdated aerosol data (I am not even sure if it needs them)
4. I am not sure the last time E3SM used MAM3 ... I thought E3SM v1 used MAM4 ...

[mahf708]

Unrelated rant: I really hope the aerosol and chemistry people rethink these files at some point... They're abjectly bad!

[whannah1]

@mahf708 catching up on this - the MMF2 compsets should only be using prescribed aerosol. The prognostic aerosol capability was never completed. I suspect there's something about that test modifier that's adding those aerosol files to the namelist, but I don't understand that modifier, so I'm confused how to check this.

Perhaps these are the prescribed aerosol files... but that wouldn't really make sense because the MMF1 tests should be using the same files.

[mahf708]

If you try to build a case, you can try to find Buildconf/eam.input_data_list with the following for the test above:

aerodep_flx_file = /lcrc/group/e3sm/data/inputdata/atm/cam/chem/trop_mam/aero/mam3_1.9x2.5_L30_2000clim_c130319.nc
aerodep_flx_datapath = /lcrc/group/e3sm/data/inputdata/atm/cam/chem/trop_mam/aero
mesh_file = /dev/null
gw_drag_file = /lcrc/group/e3sm/data/inputdata/atm/waccm/gw/newmfspectra40_dc25.nc
prescribed_volcaero_datapath = /lcrc/group/e3sm/data/inputdata/atm/cam/volc
prescribed_volcaero_file = /lcrc/group/e3sm/data/inputdata/atm/cam/volc/CMIP_DOE-ACME_radiation_average_1850-2014_v3_c20171204.nc
water_refindex_file = /lcrc/group/e3sm/data/inputdata/atm/cam/physprops/water_refindex_rrtmg_c080910.nc
iceopticsfile = /lcrc/group/e3sm/data/inputdata/atm/cam/physprops/iceoptics_c080917.nc
liqopticsfile = /lcrc/group/e3sm/data/inputdata/atm/cam/physprops/F_nwvl200_mu20_lam50_res64_t298_c080428.nc
solar_data_file = /lcrc/group/e3sm/data/inputdata/atm/cam/solar/Solar_2010control_input4MIPS_c20181017.nc
rrtmgp_coefficients_file_sw = /lcrc/group/e3sm/data/inputdata/atm/cam/rad/rrtmgp-data-sw-g112-210809.nc
rrtmgp_coefficients_file_lw = /lcrc/group/e3sm/data/inputdata/atm/cam/rad/rrtmgp-data-lw-g128-210809.nc
prescribed_ozone_file = /lcrc/group/e3sm/data/inputdata/atm/cam/ozone/ozone_1.9x2.5_L26_2000clim_c091112.nc
prescribed_ozone_datapath = /lcrc/group/e3sm/data/inputdata/atm/cam/ozone
ncdata = /lcrc/group/e3sm/data/inputdata/atm/cam/inic/homme/eam_i_mam3_Linoz_ne4np4_L60_c20210917.nc
bnd_topo = /lcrc/group/e3sm/data/inputdata/atm/cam/topo/USGS-gtopo30_ne4np4pg2_16x_converted.c20200527.nc
prescribed_aero_file = /lcrc/group/e3sm/data/inputdata/atm/cam/chem/trop_mam/aero/mam4_0.9x1.2_L72_2000clim_c170323.nc
prescribed_aero_datapath = /lcrc/group/e3sm/data/inputdata/atm/cam/chem/trop_mam/aero
tropopause_climo_file = /lcrc/group/e3sm/data/inputdata/atm/cam/chem/trop_mozart/ub/clim_p_trop.nc
mode_defs for so4_a1 = /lcrc/group/e3sm/data/inputdata/atm/cam/physprops/sulfate_rrtmg_c080918.nc
mode_defs for pom_a1 = /lcrc/group/e3sm/data/inputdata/atm/cam/physprops/ocpho_rrtmg_c101112.nc
mode_defs for soa_a1 = /lcrc/group/e3sm/data/inputdata/atm/cam/physprops/ocphi_rrtmg_c100508.nc
mode_defs for bc_a1 = /lcrc/group/e3sm/data/inputdata/atm/cam/physprops/bcpho_rrtmg_c100508.nc
mode_defs for dst_a1 = /lcrc/group/e3sm/data/inputdata/atm/cam/physprops/dust_aeronet_rrtmg_c141106.nc
mode_defs for ncl_a1 = /lcrc/group/e3sm/data/inputdata/atm/cam/physprops/ssam_rrtmg_c100508.nc
mode_defs for so4_a2 = /lcrc/group/e3sm/data/inputdata/atm/cam/physprops/sulfate_rrtmg_c080918.nc
mode_defs for soa_a2 = /lcrc/group/e3sm/data/inputdata/atm/cam/physprops/ocphi_rrtmg_c100508.nc
mode_defs for ncl_a2 = /lcrc/group/e3sm/data/inputdata/atm/cam/physprops/ssam_rrtmg_c100508.nc
mode_defs for dst_a3 = /lcrc/group/e3sm/data/inputdata/atm/cam/physprops/dust_aeronet_rrtmg_c141106.nc
mode_defs for ncl_a3 = /lcrc/group/e3sm/data/inputdata/atm/cam/physprops/ssam_rrtmg_c100508.nc
mode_defs for so4_a3 = /lcrc/group/e3sm/data/inputdata/atm/cam/physprops/sulfate_rrtmg_c080918.nc
rad_climate for mam3_mode1 = /lcrc/group/e3sm/data/inputdata/atm/cam/physprops/mam3_mode1_rrtmg_c110318.nc
rad_climate for mam3_mode2 = /lcrc/group/e3sm/data/inputdata/atm/cam/physprops/mam3_mode2_rrtmg_c110318.nc
rad_climate for mam3_mode3 = /lcrc/group/e3sm/data/inputdata/atm/cam/physprops/mam3_mode3_rrtmg_c110318.nc

I am not sure how those settings are handled though ...

[mahf708]

(In other words, this has nothing to do with "ghci-oci_gnu" modifier --- the quoted files above are for chrysalis_intel)

[mahf708]

Very likely the stuff in components/eam/cime_config/config_component.xml is inheriting random stuff from ancient cam compsets, since you don't have full-fledged files with definitions, the stuff must be coming from some ancient defaults somewhere ...

[whannah1]

yea - sounds like the compset parsing there is broken and the chemistry options options are being added to the CAM_CONFIG_OPTS.

[rljacob]

@jgfouca and @bartgol this is the thread about the difficulty in adding an MMF test to the gh/ci suite

[mahf708]

Adding this test to the gh snl tests (i.e., tests running in snl containers) will be much easier

I will also note that when compiling on intel with O3, this test has a lot of non-trivial opt going on, but that's an aside

[bartgol]

If the issue is to protect MMF failures when scream physics is updated by a PR that does not go through next, then it makes more sense to add this test to the snl container tests.

This one: SMS_Ln3_P4.ne4pg2_oQU480.F2010-MMF2.ghci-oci_gnu

Is this test the desired one? If so, I can add it right away, we can ninja merge, and ninja bless.

Edit: I went ahead and added it. #6852

[mahf708]

Will be fixed in #6852