Extremely slow performance after source mods which load netcdf files

[jacob-stu-allen]

Hello!

I've incorporated source code modifications to the wind stress that is communicated from the coupler to mpaso - numerically I'm getting the result that I want, but the performance of E3SMv2 has been extremely slow. Only ~4 days were simulated using the large pecount with a 12 hour wallclock. In submitting test cases (S_1x10_ndays) to the debug queue, the performance seemed normal.

These modifications are in the form of a new subroutine (mostly copied from the ocn_surface_bulk_forcing_vel in mpas_ocn_surface_bulk_forcing.f90), which loops through ncels and multiplies certain regions by a scaling factor (see lines 285-542 in attached mpas_ocn_surface_bulk_forcing.f90).

I'm loading three netcdf files (2 mask files and one 365 day climatology) to perform my calculations, and I'm wondering if this i/o is what is slowing down the integrations so drastically.

Any ideas on how to avoid this performance issue would be greatly appreciated. I've incorporated this same process in CESM2 with no noticeable change. Attached are my logs and source mods.

Thanks in advance,
Jacob

e3sm_wb_002.zip

[oksanaguba]

hi, just curious if you have a reference or description for this work? thanks!

[jacob-stu-allen]

Hello @oksanaguba, I'm working on something similar to this paper (https://journals.ametsoc.org/view/journals/clim/35/22/JCLI-D-21-0401.1.xml) but in E3SM.

[jacob-stu-allen]

Following up, I think I've fixed this. I've moved the netcdf file declarations and allocations to the initialization so that there's no i/o for them at each time step. My sim times are looking much better.