Issues with MPI and MPAS-O on Cori

[lizcarlson]

@ndkeen Hi Noel, this is Elizabeth. Is this discussion board quicker than email?

I'm having trouble with compiling MPAS-Ocean (moving to compiling E3SM soon). It keeps pulling up errors such as
Error: Symbol 'mpi_real' at (1) has no IMPLICIT type

I am wondering if I am using all of the right modules or if I missed something; this is probably/hopefully an easy fix (I know someone else has been running MPAS-Ocean on Cori this summer, but he has the same mentor as the one who gave me the following module list).

module switch PrgEnv-intel PrgEnv-gnu
module load cray-netcdf-hdf5parallel
module load cray-parallel-netcdf
module load cmake
module load openmpi
source /global/project/projectdirs/e3sm/software/anaconda_envs/load_latest_e3sm_unified.sh
export PIO=/global/u2/h/hgkang/my_programs/Scorpio
git submodule update --init --recursive

module load git
export GITHASH=git log -n 1 --oneline|cut -c 1-8
echo 'GITHASH:' $GITHASH
export DATE=date +"%y%m%d"
export HOST=hostname -s|cut -c 1-2
export COMPILER=gnu-nersc

I would think module load openmpi should do the trick (since all relevant MPI files should have transferred over with the GitHub repo clone), but maybe it's being overwritten somehow.

[rljacob]

Well this way other people can possibly help.

[ndkeen]

Which repo are you trying to work with?

[xylar]

@lizcarlson, it sounds like this is an issue for the https://github.com/MPAS-Dev/compass repo, rather than for E3SM, though we can keep discussing it here if that's easier.

I'm glad to help but would need quite a bit more detail on how to reproduce what you're seeing since I haven't been following along. Are you following the developer's guide for compass? https://mpas-dev.github.io/compass/latest/developers_guide/quick_start.html Can you give me the commands you're using to check out compass, build E3SM, set up your development environment, etc.?

[lizcarlson]

Hi @xylar good to hear from you! Yes, I followed the developer's guide for compass. I followed the same steps as @vanroekel listed below, where step 3 was replaced by ./conda/configure_compass_env.py --conda /global/common/software/e3sm/miniconda3/ --mach cori-knl. Then I run the following commands:

source load_dev_compass_1.0.0_cori-knl_intel_impi.sh
cd ../E3SM-Model_original/E3SM/components/mpas-ocean
make intel-mpi CORE=ocean USE_PIO2=true DEBUG=true

If I do not have the MPI call in that file, it compiles just fine.

[mt5555]

I took a look at mpas_ocn_time_integration.F in the current e3sm master. I dont know if this is the same file you are working with, but if it is: It doesn't have any MPI calls in it, nor a "use mpi", and thus it doesn't know about MPI variables (such as MPI_Comm_world) or MPI functions. Also, when running within E3SM, mpas will be running in it's own communicator, which is a subset of MPI_Comm_world. If you want to add MPI calls to this routine, you might need a different communicator.

[xylar]

--conda /global/common/software/e3sm/miniconda3/

@lizcarlson, your approach looks right except that this path is in a shared space. You should use a path such as your home directory that is unique to you or others may try to use the environment you created. Could you please delete this one and use a different path?

[lizcarlson]

Oh my bad, I interpreted that as doing something different than it was. Moved 👍

[vanroekel]

@lizcarlson to run MPAS stand alone on cori, I'd strongly recommend using the new conda environment functionality @xylar built into COMPASS. There is nice documentation on how to set up a new conda environment here -- https://mpas-dev.github.io/compass/latest/developers_guide/quick_start.html Cori-knl is a supported option. so the basic steps to setup the environment would be

1. clone compass
2. git submodule update --init --recursive
3. setup a conda environment -- ./conda/configure_compass_env.py --conda /path/to/miniconda/install/location -mach cori-knl
   Once it makes this you can source the file it makes and then build and run. Also if the miniconda is not installed the script will download and install for you.

With this you should be able to just use master

[lizcarlson]

Thanks, it took a minute because miniconda3 had to be installed in the /global/common/software/e3sm/ directory, but after that I was able to get all that to compile correctly. Unfortunately, the same error still appears when I compile the MPAS code. I'll try running the standard E3SM case with an MPI call in a file in the mode_forward directory (e.g., just a rank call), and I'll let you know how that goes.

[lizcarlson]

Yes, I do know this, I added this in myself for other reasons. While running within E3SM, hmm, then if I investigate the framework files maybe I can just edit those and that will fix it. Just for reference, a similar problem was occurring in the stand-alone MPAS-Ocean on Cori. This error was not present on LANL systems. On Sep 2, 2021, at 2:43 PM, Mark Taylor ***@***.******@***.***>> wrote: Non-NU Email

…

________________________________ I took a look at mpas_ocn_time_integration.F in the current e3sm master. I dont know if this is the same file you are working with, but if it is: It doesn't have any MPI calls in it, nor a "use mpi", and thus it doesn't know about MPI variables (such as MPI_Comm_world) or MPI functions. Also, when running within E3SM, mpas will be running in it's own communicator, which is a subset of MPI_Comm_world. If you want to add MPI calls to this routine, you might need a different communicator. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https://github.com/E3SM-Project/E3SM/discussions/4494*discussioncomment-1274290__;Iw!!PvXuogZ4sRB2p-tU!RcAPbkM5RAC_2E2Lvc1QLJLFYH76oMdUkuXsKGgpqB7h-h7NaBGKbiTzSeB0-Tvt8WP3_thhWYwD$>, or unsubscribe<https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AM63NYTVNBW26FXISNFWEYDT77VWRANCNFSM5C4DRTGA__;!!PvXuogZ4sRB2p-tU!RcAPbkM5RAC_2E2Lvc1QLJLFYH76oMdUkuXsKGgpqB7h-h7NaBGKbiTzSeB0-Tvt8WP3_tMTcFYo$>. Triage notifications on the go with GitHub Mobile for iOS<https://urldefense.com/v3/__https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675__;!!PvXuogZ4sRB2p-tU!RcAPbkM5RAC_2E2Lvc1QLJLFYH76oMdUkuXsKGgpqB7h-h7NaBGKbiTzSeB0-Tvt8WP3_jIsDTm8$> or Android<https://urldefense.com/v3/__https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign*3Dnotification-email*26utm_medium*3Demail*26utm_source*3Dgithub__;JSUlJSU!!PvXuogZ4sRB2p-tU!RcAPbkM5RAC_2E2Lvc1QLJLFYH76oMdUkuXsKGgpqB7h-h7NaBGKbiTzSeB0-Tvt8WP3_sDC5oYx$>.

[philipwjones]

@lizcarlson Sorry for not weighing in sooner. I build standalone (both within and outside of E3SM) on Cori with no problems. One thing I noticed in your comment above that you were using intel-mpi as the target. The actual make incantation you want for Cori is:
make intel-nersc TOOL_TARGET_ARCH='-g -xAVX' USE_PIO2=true
And I think Mark is onto at least part of the problem - we access MPI through the dmpar routines in the framework. I can take a look at your branch to see what you're trying to do and advise.

[lizcarlson]

Yes, I can also build standalone on Cori so long as I am not calling MPI directly.
I tried to use that make command that you used instead, it did not compile. Instead it gave a PIO not found error. Note, I am trying to compile the MPAS-O core inside E3SM. Did you send me the make command for compiling all of E3SM?
I am currently not in any branch. I just downloaded E3SM master and edited one line to see if just that would compile with the MPI call. For MPAS-O standalone outside of E3SM, correct, AOT_develop is the branch I am working on.

[philipwjones]

@lizcarlson - ok, I'll work on the MPI issues and a more robust approach there. And I guess the discussion above has gotten a bit confusing. You started out needing a standalone build of the AOT_develop branch. And then it got into the E3SM build from compass. So what do you actually need for both the short term and then longer-term?

The incantation above and a standalone build generally does need a separate build of scorpio (for pio) since E3SM builds pio on the fly as part of the overall build. I do have a separate static scorpio build that you can point to for that, but if the compass process is working then that is the preferred approach in the longer term. If you just wanted to check out and build your AOT version today without compass, then I can point you to the right libs and the above command would work.

[lizcarlson]

@philipwjones Thanks for the insight and help! What I need to do is this: if it runs in MPAS-O standalone without trouble, then we are good. I will then transport it into E3SM and make sure that compiles. Then we run E3SM. These are the steps. If I can bypass the MPAS-O standalone step that is fine, I have all the namelist stuff I need set up there already, that has all been set up for a while.

Unfortunately I am taking the rest of the day for personal business, but I hope to be able to work on this this weekend. That would be great to build it, though, with the libraries you use if you wanted to provide those for me. If it is easier, you can email me directly. I'll drop a quick hello email here in a minute so you have my email.

[philipwjones]

@lizcarlson ok, I have a fix for all the MPI issues (at least for building - don't have a configuration for running so can't test that). I still need to fix the Makefile to deal with some other new dependencies but should have a successful standalone build of your AOT branch soon. I'll point you to the fixes in a bit...

[philipwjones]

@lizcarlson Are all the changes in your AOT develop branch confined to the mode_forward directory? I see some of the calls and will have some suggestions tomorrow for how to manage that better within MPAS-land.

[lizcarlson]

Yes, they are. Within MPAS land seems, from what I have discovered, to transfer fairly straightforwardly to E3SM land (code base wise, namelist-wise it's not bad but it is different). If you have suggestions on improvements (mind you some of it is not done yet) that would be great, thanks!