Port overflow test group #501

cbegeman · 2023-01-12T22:52:37Z

Here we port the overflow test group from legacy compass. It features a density current flowing down a continental shelf. We include a default test at low resolution and a RPE test at higher resolution. The computation time for the RPE test was reduced from legacy by changing the resolution from 1km to 2km, the number of vertical levels from 100 to 50, and reducing the domain length from 240km to 200km. Only the sigma vertical coordinate is correctly initialized in MPAS-Ocean's init mode, so only that coordinate is tested here.

Checklist

User's Guide has been updated
Developer's Guide has been updated
Documentation has been built locally and changes look as expected
New tests have been added to a test suite

cbegeman · 2023-01-12T22:53:22Z

I intend to finish the documentation in the next couple days and update the checklist accordingly.

cbegeman · 2023-01-12T22:55:34Z

The RPE test generates two figures, shown here:

xylar · 2023-01-16T19:02:40Z

@cbegeman, this is very exciting! It's a great test case to have because it's so visually clear how we're doing with mixing.

I'll review as soon as I can. I'm trying to get some basic support PRs for both MPAS-Analysis and compass taken care of.

mark-petersen · 2023-01-27T22:21:39Z

Testing now with gnu on chrysalis. Here are my commands:

pwd
/home/ac.mpetersen/repos/compass/pr

git log --graph --oneline -n 3
* 93bd4afe (HEAD -> pr, cbegeman/port-overflow-test-group) Aesthetic changes: fix case
* fd1f7250 Add validate step
* 011bbc98 Update cfg file

./conda/configure_compass_env.py --conda ~/miconda3 --compiler gnu
source load_dev_compass_1.2.0-alpha.3_chrysalis_gnu_openmpi.sh

compass list |grep over
 246: ocean/overflow/10km/default
 247: ocean/overflow/2km/rpe_test

compass setup -p /home/ac.mpetersen/repos/E3SM/master/components/mpas-ocean -w $r/230127_test_overflow -n 246 247
Setting up test cases:
  ocean/overflow/10km/default
  ocean/overflow/2km/rpe_test
target cores: 64
minimum cores: 16

srun -p debug -N 1 -t 1:00:00 --pty bash   # obtain interactive node

cd /lcrc/group/e3sm/ac.mpetersen/scratch/runs/230127_test_overflow/ocean/overflow/10km/default
compass run

It looks like both cases: ocean/overflow/10km/default and ocean/overflow/2km/rpe_test run correctly with chrysalis gnu optimized. I happen to be using E3SM f502d676a7 (tag: v2.1.0) but I don't think that is important. I can reproduce your images above.

Using gnu debug, both cases end during the first time step with a segmentation fault.

Here are more details:

(after initial condition was made successfully)
pwd
/lcrc/group/e3sm/ac.mpetersen/scratch/runs/230127_test_overflow/ocean/overflow/10km/default/forward

srun -c 1 -n 4 ./ocean_model -n namelist.ocean -s streams.ocean

Program received signal SIGFPE: Floating-point exception - erroneous arithmetic operation.

Backtrace for this error:

Program received signal SIGFPE: Floating-point exception - erroneous arithmetic operation.

Backtrace for this error:

Program received signal SIGFPE: Floating-point exception - erroneous arithmetic operation.

Backtrace for this error:

Program received signal SIGFPE: Floating-point exception - erroneous arithmetic operation.

Backtrace for this error:
#0  0x15554f15a3ff in ???
#1  0x15554fdd8010 in ???
#2  0xf955a8 in __cvmix_convection_MOD_cvmix_coeffs_conv_wrap
	at /home/ac.mpetersen/repos/E3SM/master/components/mpas-ocean/src/cvmix/src/shared/cvmix_convection.F90:227
#3  0xf0bc10 in __ocn_vmix_cvmix_MOD_ocn_vmix_coefs_cvmix_build
	at /home/ac.mpetersen/repos/E3SM/master/components/mpas-ocean/src/shared/mpas_ocn_vmix_cvmix.F:739
#4  0xcb8da5 in __ocn_vmix_MOD_ocn_vmix_coefs
	at /home/ac.mpetersen/repos/E3SM/master/components/mpas-ocean/src/shared/mpas_ocn_vmix.F:200
#5  0xcadb66 in __ocn_vmix_MOD_ocn_vmix_implicit
	at /home/ac.mpetersen/repos/E3SM/master/components/mpas-ocean/src/shared/mpas_ocn_vmix.F:1154
#0  0x15554f15a3ff in ???
#1  0x15554fdd8010 in ???
#2  0xf955a8 in __cvmix_convection_MOD_cvmix_coeffs_conv_wrap
	at /home/ac.mpetersen/repos/E3SM/master/components/mpas-ocean/src/cvmix/src/shared/cvmix_convection.F90:227
#6  0xb4d469 in __ocn_time_integration_split_MOD_ocn_time_integrator_split
	at /home/ac.mpetersen/repos/E3SM/master/components/mpas-ocean/src/mode_forward/mpas_ocn_time_integration_split.F:2424
#7  0xb46713 in __ocn_time_integration_MOD_ocn_timestep
	at /home/ac.mpetersen/repos/E3SM/master/components/mpas-ocean/src/mode_forward/mpas_ocn_time_integration.F:125
#3  0xf0bc10 in __ocn_vmix_cvmix_MOD_ocn_vmix_coefs_cvmix_build
	at /home/ac.mpetersen/repos/E3SM/master/components/mpas-ocean/src/shared/mpas_ocn_vmix_cvmix.F:739
#8  0xb435b3 in __ocn_forward_mode_MOD_ocn_forward_mode_run
	at /home/ac.mpetersen/repos/E3SM/master/components/mpas-ocean/src/mode_forward/mpas_ocn_forward_mode.F:728
#9  0xb425c3 in __ocn_core_MOD_ocn_core_run
	at /home/ac.mpetersen/repos/E3SM/master/components/mpas-ocean/src/driver/mpas_ocn_core.F:111
#10  0x40f39a in __mpas_subdriver_MOD_mpas_run
	at /home/ac.mpetersen/repos/E3SM/master/components/mpas-framework/src/driver/mpas_subdriver.F:358
#11  0x40e113 in mpas
	at /home/ac.mpetersen/repos/E3SM/master/components/mpas-framework/src/driver/mpas.F:20
#12  0x40e17e in main
	at /home/ac.mpetersen/repos/E3SM/master/components/mpas-framework/src/driver/mpas.F:10
#4  0xcb8da5 in __ocn_vmix_MOD_ocn_vmix_coefs
	at /home/ac.mpetersen/repos/E3SM/master/components/mpas-ocean/src/shared/mpas_ocn_vmix.F:200
#5  0xcadb66 in __ocn_vmix_MOD_ocn_vmix_implicit
	at /home/ac.mpetersen/repos/E3SM/master/components/mpas-ocean/src/shared/mpas_ocn_vmix.F:1154
#6  0xb4d469 in __ocn_time_integration_split_MOD_ocn_time_integrator_split
	at /home/ac.mpetersen/repos/E3SM/master/components/mpas-ocean/src/mode_forward/mpas_ocn_time_integration_split.F:2424
#7  0xb46713 in __ocn_time_integration_MOD_ocn_timestep
	at /home/ac.mpetersen/repos/E3SM/master/components/mpas-ocean/src/mode_forward/mpas_ocn_time_integration.F:125
#8  0xb435b3 in __ocn_forward_mode_MOD_ocn_forward_mode_run
	at /home/ac.mpetersen/repos/E3SM/master/components/mpas-ocean/src/mode_forward/mpas_ocn_forward_mode.F:728
#9  0xb425c3 in __ocn_core_MOD_ocn_core_run
	at /home/ac.mpetersen/repos/E3SM/master/components/mpas-ocean/src/driver/mpas_ocn_core.F:111
#10  0x40f39a in __mpas_subdriver_MOD_mpas_run
	at /home/ac.mpetersen/repos/E3SM/master/components/mpas-framework/src/driver/mpas_subdriver.F:358
#11  0x40e113 in mpas
	at /home/ac.mpetersen/repos/E3SM/master/components/mpas-framework/src/driver/mpas.F:20
#12  0x40e17e in main
	at /home/ac.mpetersen/repos/E3SM/master/components/mpas-framework/src/driver/mpas.F:10
#0  0x15554f15a3ff in ???
#1  0x15554fdd8010 in ???
#2  0xf955a8 in __cvmix_convection_MOD_cvmix_coeffs_conv_wrap
	at /home/ac.mpetersen/repos/E3SM/master/components/mpas-ocean/src/cvmix/src/shared/cvmix_convection.F90:227
#0  0x15554f15a3ff in ???
#1  0x15554fdd8010 in ???
#2  0xf955a8 in __cvmix_convection_MOD_cvmix_coeffs_conv_wrap
	at /home/ac.mpetersen/repos/E3SM/master/components/mpas-ocean/src/cvmix/src/shared/cvmix_convection.F90:227
#3  0xf0bc10 in __ocn_vmix_cvmix_MOD_ocn_vmix_coefs_cvmix_build
	at /home/ac.mpetersen/repos/E3SM/master/components/mpas-ocean/src/shared/mpas_ocn_vmix_cvmix.F:739
#3  0xf0bc10 in __ocn_vmix_cvmix_MOD_ocn_vmix_coefs_cvmix_build
	at /home/ac.mpetersen/repos/E3SM/master/components/mpas-ocean/src/shared/mpas_ocn_vmix_cvmix.F:739
#4  0xcb8da5 in __ocn_vmix_MOD_ocn_vmix_coefs
	at /home/ac.mpetersen/repos/E3SM/master/components/mpas-ocean/src/shared/mpas_ocn_vmix.F:200
#5  0xcadb66 in __ocn_vmix_MOD_ocn_vmix_implicit
	at /home/ac.mpetersen/repos/E3SM/master/components/mpas-ocean/src/shared/mpas_ocn_vmix.F:1154
#4  0xcb8da5 in __ocn_vmix_MOD_ocn_vmix_coefs
	at /home/ac.mpetersen/repos/E3SM/master/components/mpas-ocean/src/shared/mpas_ocn_vmix.F:200
#5  0xcadb66 in __ocn_vmix_MOD_ocn_vmix_implicit
	at /home/ac.mpetersen/repos/E3SM/master/components/mpas-ocean/src/shared/mpas_ocn_vmix.F:1154
#6  0xb4d469 in __ocn_time_integration_split_MOD_ocn_time_integrator_split
	at /home/ac.mpetersen/repos/E3SM/master/components/mpas-ocean/src/mode_forward/mpas_ocn_time_integration_split.F:2424
#7  0xb46713 in __ocn_time_integration_MOD_ocn_timestep
	at /home/ac.mpetersen/repos/E3SM/master/components/mpas-ocean/src/mode_forward/mpas_ocn_time_integration.F:125
#8  0xb435b3 in __ocn_forward_mode_MOD_ocn_forward_mode_run
	at /home/ac.mpetersen/repos/E3SM/master/components/mpas-ocean/src/mode_forward/mpas_ocn_forward_mode.F:728
#9  0xb425c3 in __ocn_core_MOD_ocn_core_run
	at /home/ac.mpetersen/repos/E3SM/master/components/mpas-ocean/src/driver/mpas_ocn_core.F:111
#10  0x40f39a in __mpas_subdriver_MOD_mpas_run
	at /home/ac.mpetersen/repos/E3SM/master/components/mpas-framework/src/driver/mpas_subdriver.F:358
#6  0xb4d469 in __ocn_time_integration_split_MOD_ocn_time_integrator_split
	at /home/ac.mpetersen/repos/E3SM/master/components/mpas-ocean/src/mode_forward/mpas_ocn_time_integration_split.F:2424
#11  0x40e113 in mpas
	at /home/ac.mpetersen/repos/E3SM/master/components/mpas-framework/src/driver/mpas.F:20
#12  0x40e17e in main
	at /home/ac.mpetersen/repos/E3SM/master/components/mpas-framework/src/driver/mpas.F:10
#7  0xb46713 in __ocn_time_integration_MOD_ocn_timestep
	at /home/ac.mpetersen/repos/E3SM/master/components/mpas-ocean/src/mode_forward/mpas_ocn_time_integration.F:125
#8  0xb435b3 in __ocn_forward_mode_MOD_ocn_forward_mode_run
	at /home/ac.mpetersen/repos/E3SM/master/components/mpas-ocean/src/mode_forward/mpas_ocn_forward_mode.F:728
#9  0xb425c3 in __ocn_core_MOD_ocn_core_run
	at /home/ac.mpetersen/repos/E3SM/master/components/mpas-ocean/src/driver/mpas_ocn_core.F:111
#10  0x40f39a in __mpas_subdriver_MOD_mpas_run
	at /home/ac.mpetersen/repos/E3SM/master/components/mpas-framework/src/driver/mpas_subdriver.F:358
#11  0x40e113 in mpas
	at /home/ac.mpetersen/repos/E3SM/master/components/mpas-framework/src/driver/mpas.F:20
#12  0x40e17e in main
	at /home/ac.mpetersen/repos/E3SM/master/components/mpas-framework/src/driver/mpas.F:10
srun: error: chr-0503: tasks 2-3: Floating point exception (core dumped)
srun: error: chr-0503: tasks 0-1: Floating point exception (core dumped)
...
tail log.ocean.0000.out
  Vertical coordinate movement is: uniform_stretching
 *******************************************************************************
 The split explicit time integration is configured to use: 7 barotropic subcycles
 *******************************************************************************
  Pressure type is: pressure_and_zmid
 Initial time 0001-01-01_00:00:00
WARNING: MPAS I/O: Truncating existing data in output file output.nc
 Doing timestep 0001-01-01_00:01:00
    Computing surface flux arrays
    Computing forward time step

@cbegeman can you confirm this behavior in your setup with a debug version? It would seem that the code is reading or writing out of bounds to get a segfault. But perhaps I made a simple mistake in the setup.

If you reproduce this behavior, please try again with legacy compass, since this is just a straight translation. Thanks!

xylar · 2023-01-28T21:34:23Z

@mark-petersen, I saw similar issues with Gnu in debug mode related to CVMix in the ISOMIP+ test case from the pr test suite. I'll reproduce that and post an issue, but I don't think this PR is introducing that problem.

xylar · 2023-01-28T22:41:48Z

@mark-petersen, I created #513 for this and would appreciate your help investigating, but again I don't think it's related to this PR.

xylar

@cbegeman, most excellent work as always!

I have several formatting suggestions and related pieces of very minor clean up.

In addition, there are a few questions about possible changes to what are config options in overflow.cfg, what are hard-coded "magic numbers" in code, and what are config options defined in code.

compass/ocean/tests/overflow/__init__.py

compass/ocean/tests/overflow/overflow.cfg

compass/ocean/tests/overflow/rpe_test/__init__.py

compass/ocean/tests/overflow/forward.py

compass/ocean/tests/overflow/rpe_test/analysis.py

compass/ocean/tests/overflow/initial_state.py

cbegeman · 2023-01-30T22:05:42Z

@xylar I have made all the changes you suggested. Thanks for your thorough review!

mark-petersen

Great! Given that I was able to reproduce @cbegeman's results on chrysalis with gnu optimized, and my gnu debug problem is unrelated, I am approving. Thanks @cbegeman for your work here, and @xylar for your detailed review.

cbegeman · 2023-01-30T23:49:32Z

@mark-petersen Thank you so much for your review and testing! I appreciate you getting to this so quickly and testing out different machine/compiler combinations.

xylar

@cbegeman told me in person that she retested this so I'm happy with the changes (and the comments about changes that will hold off for later).

xylar · 2023-01-31T01:21:12Z

@alicebarthel and @lconlon, could you let us know if you plan to review? I'll hold off on merging until we've heard back from you.

alicebarthel

Passed visual inspection (more for myself to understand the process). I was able to reproduce the test on Anvil (intel-mpi). Thanks for the opportunity to watch and learn!

cbegeman · 2023-02-06T14:41:56Z

@alicebarthel Thanks for reviewing and testing!

cbegeman · 2023-02-08T21:26:10Z

@lconlon I'm removing you as a reviewer but you're still welcome to give feedback if you like.

xylar · 2023-02-08T23:58:31Z

@cbegeman, if you can rebase to fix the conflicts here, I'm happy to merge this.

xylar · 2023-02-09T01:49:05Z

@cbegeman, I took the liberty of doing the rebase for you. It wasn't hard to resolve the conflicts. I tested the docs locally and they're fine. compass list also work, which probably sufficient testing given the very minimal changes involved in the rebase.

cbegeman · 2023-02-09T15:06:57Z

@xylar Thanks for the rebase and getting this in!

cbegeman requested review from alicebarthel, mark-petersen and xylar January 12, 2023 22:53

cbegeman added the ocean label Jan 12, 2023

cbegeman force-pushed the port-overflow-test-group branch 2 times, most recently from afacf4a to fd1f725 Compare January 13, 2023 23:54

cbegeman requested a review from lconlon January 17, 2023 18:08

xylar mentioned this pull request Jan 28, 2023

Ocean test cases failing in CVMix with Gnu and debug #513

Open

xylar added the enhancement New feature or request label Jan 28, 2023

xylar self-assigned this Jan 28, 2023

xylar requested changes Jan 29, 2023

View reviewed changes

cbegeman force-pushed the port-overflow-test-group branch from 95334ea to a567a19 Compare January 30, 2023 17:57

cbegeman mentioned this pull request Jan 30, 2023

Cleanup ocean RPE tests #516

Merged

2 tasks

mark-petersen approved these changes Jan 30, 2023

View reviewed changes

xylar approved these changes Jan 31, 2023

View reviewed changes

alicebarthel approved these changes Feb 4, 2023

View reviewed changes

cbegeman removed the request for review from lconlon February 8, 2023 21:25

cbegeman and others added 20 commits February 8, 2023 18:37

Create overflow test group

3f92081

Add namelists

6173505

Add initial state

eadb1df

Add streams

94bbb3d

Add forward step

134f0ff

Add rpe test

9d7a746

Fixup resolution dependencies

3bb19d1

Make domain periodic in y-dim

ad8aa76

fixup analysis

e3dc3b3

Fix dt

6673a07

cleanup __init__.py

9d5bd37

Update docs

93e0096

Update cfg file

94affda

Add validate step

56b62c0

Aesthetic changes: fix case

79cc553

Remove grid specs from cfg file

4a474eb

Cleanup formatted strings

9c1de77

Cleanup initial state

93ee77a

Move nus list to cfg

72b93eb

Cleanup rpe analysis

8dee577

xylar force-pushed the port-overflow-test-group branch from a567a19 to 8dee577 Compare February 9, 2023 01:46

xylar merged commit 0e09157 into MPAS-Dev:main Feb 9, 2023

cbegeman deleted the port-overflow-test-group branch March 15, 2023 21:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Port overflow test group #501

Port overflow test group #501

cbegeman commented Jan 12, 2023 •

edited

Loading

cbegeman commented Jan 12, 2023

cbegeman commented Jan 12, 2023

xylar commented Jan 16, 2023

mark-petersen commented Jan 27, 2023

xylar commented Jan 28, 2023

xylar commented Jan 28, 2023

xylar left a comment

cbegeman commented Jan 30, 2023

mark-petersen left a comment •

edited

Loading

cbegeman commented Jan 30, 2023

xylar left a comment

xylar commented Jan 31, 2023

alicebarthel left a comment

cbegeman commented Feb 6, 2023

cbegeman commented Feb 8, 2023

xylar commented Feb 8, 2023

xylar commented Feb 9, 2023

cbegeman commented Feb 9, 2023

Port overflow test group #501

Port overflow test group #501

Conversation

cbegeman commented Jan 12, 2023 • edited Loading

cbegeman commented Jan 12, 2023

cbegeman commented Jan 12, 2023

xylar commented Jan 16, 2023

mark-petersen commented Jan 27, 2023

xylar commented Jan 28, 2023

xylar commented Jan 28, 2023

xylar left a comment

Choose a reason for hiding this comment

cbegeman commented Jan 30, 2023

mark-petersen left a comment • edited Loading

Choose a reason for hiding this comment

cbegeman commented Jan 30, 2023

xylar left a comment

Choose a reason for hiding this comment

xylar commented Jan 31, 2023

alicebarthel left a comment

Choose a reason for hiding this comment

cbegeman commented Feb 6, 2023

cbegeman commented Feb 8, 2023

xylar commented Feb 8, 2023

xylar commented Feb 9, 2023

cbegeman commented Feb 9, 2023

cbegeman commented Jan 12, 2023 •

edited

Loading

mark-petersen left a comment •

edited

Loading