Update to compass v1.3.0-alpha.1

[xylar]

Updates:

• Albany and Trilinos to compass-2024-03-13 tags
  • Updates landice tests to be compatible, see Remove "Build Type: Tpetra" from Landice tests #788
• mache to 1.20.0, with updated support for several machines
• ESMF to 8.6.0
• mpas_tools to 0.32.0
• SCORPIO to 2.6.0

Other Changes:

• Add MOAB 5.5.1 (to spack build and conda env)
• Removes pnetcdf conda recipe (has been on conda-forge for a long time)

Checklist

• Document (in a comment titled Testing in this PR) any testing that was used to verify the changes

Testing

MPAS-Ocean with pr:

• Anvil
  • intel and impi
  • gnu and openmpi
• Chicoma
  • gnu and mpich
• Chrysalis
  • intel and openmpi
  • gnu and openmpi
• Compy
  • intel and impi - seeing failures similar to what we have seen before, as reported in ocean/global_ocean/QU240/PHC/RK4/decomp_test failing on Compy with Intel #502
• Perlmutter
  • gnu and mpich

MALI with full_integration:

• Chicoma
  • gnu and mpich
• Chrysalis
  • gnu and openmpi
• Perlmutter
  • gnu and mpich

MPAS-Ocean with nonhydro - Failing due to problems detailed in #797

Deployed

MPAS-Ocean with pr:

• Anvil
  • intel and impi
  • intel and openmpi
  • gnu and openmpi
  • gnu and mvapich - can't build MOAB so removing this config
• Chicoma
  • gnu and mpich
• Chrysalis
  • intel and openmpi
  • gnu and openmpi
• Compy
  • intel and impi - failures as noted in ocean/global_ocean/QU240/PHC/RK4/decomp_test failing on Compy with Intel #502
• Perlmutter
  • gnu and mpich

MALI with full_integration:

• Chicoma
  • gnu and mpich
• Chrysalis
  • gnu and openmpi
• Perlmutter
  • gnu and mpich

[xylar]

@mcarlson801, @mperego,

When I try to build Albany with the compass-2024-03-13 tag on Chrysalis, I'm seeing:

CMake Error at tests/CMakeLists.txt:149 (IF):
  if given arguments:

    "VERSION_LESS" "2.7"

  Unknown arguments specified


-- Configuring incomplete, errors occurred!
See also "/lcrc/group/e3sm/ac.xylar/chrys_spack_tmp/ac.xasay-davis/spack-stage/spack-stage-albany-compass-2024-03-13-ee6kt34yhi6pk27i4r7gaojlye4sg2zp/spack-build-ee6kt34/CMakeFiles/CMakeOutput.log".
See also "/lcrc/group/e3sm/ac.xylar/chrys_spack_tmp/ac.xasay-davis/spack-stage/spack-stage-albany-compass-2024-03-13-ee6kt34yhi6pk27i4r7gaojlye4sg2zp/spack-build-ee6kt34/CMakeFiles/CMakeError.log".

Then, I'm seeing in CMakeError.log:

Performing C SOURCE FILE Test PARALLEL_EXODUS_SUPPORTED failed with the following output:
Change Dir: /lcrc/group/e3sm/ac.xylar/chrys_spack_tmp/ac.xasay-davis/spack-stage/spack-stage-albany-compass-2024-03-13-ee6kt34yhi6pk27i4r7gaojlye4sg2zp/spack-build-ee6kt34/CMakeFiles/CMakeTmp

Run Build Command(s):/usr/bin/gmake -f Makefile cmTC_cc8f4/fast && /usr/bin/gmake  -f CMakeFiles/cmTC_cc8f4.dir/build.make CMakeFiles/cmTC_cc8f4.dir/build
gmake[1]: Entering directory '/lcrc/group/e3sm/ac.xylar/chrys_spack_tmp/ac.xasay-davis/spack-stage/spack-stage-albany-compass-2024-03-13-ee6kt34yhi6pk27i4r7gaojlye4sg2zp/spack-build-ee6kt34/CMakeFiles/CMakeTmp'
Building C object CMakeFiles/cmTC_cc8f4.dir/src.c.o
/gpfs/fs1/soft/chrysalis/spack/opt/spack/linux-centos8-x86_64/gcc-9.2.0/openmpi-4.1.3-sxfyy4k/bin/mpicc -DPARALLEL_EXODUS_SUPPORTED -I/lcrc/group/e3sm/ac.xylar/chrys_spack_test/dev_compass_1_3_0_gnu_openmpi_albany/opt/spack/linux-rhel8-zen2/gcc-9.2.0/trilinos-for-albany-compass-2024-03-13-idguiqvi6zhx4rjckoxaxdpgnmlikjrh/include  -o CMakeFiles/cmTC_cc8f4.dir/src.c.o -c /lcrc/group/e3sm/ac.xylar/chrys_spack_tmp/ac.xasay-davis/spack-stage/spack-stage-albany-compass-2024-03-13-ee6kt34yhi6pk27i4r7gaojlye4sg2zp/spack-build-ee6kt34/CMakeFiles/CMakeTmp/src.c
/lcrc/group/e3sm/ac.xylar/chrys_spack_tmp/ac.xasay-davis/spack-stage/spack-stage-albany-compass-2024-03-13-ee6kt34yhi6pk27i4r7gaojlye4sg2zp/spack-build-ee6kt34/CMakeFiles/CMakeTmp/src.c:6:4: error: #error Missing pnetcdf
    6 |   #error Missing pnetcdf
      |    ^~~~~
gmake[1]: *** [CMakeFiles/cmTC_cc8f4.dir/build.make:78: CMakeFiles/cmTC_cc8f4.dir/src.c.o] Error 1
gmake[1]: Leaving directory '/lcrc/group/e3sm/ac.xylar/chrys_spack_tmp/ac.xasay-davis/spack-stage/spack-stage-albany-compass-2024-03-13-ee6kt34yhi6pk27i4r7gaojlye4sg2zp/spack-build-ee6kt34/CMakeFiles/CMakeTmp'
gmake: *** [Makefile:127: cmTC_cc8f4/fast] Error 2


Source file was:

  #include <exodusII_par.h>
  #if !NC_HAS_HDF5
  #error Missing hdf5
  #elif !NC_HAS_PNETCDF
  #error Missing pnetcdf
  #endif
  int main()
  {
    return 0;
  }

Is this something we need to somehow fix in the spack package for Albany?

[xylar]

Following up, I think the issue is that $PYTHON_EXECUTABLE and therefore $PYTHON_VERSION are not defined. I think the error from CMakeError.log above is not relevant.

[mperego]

Pinging @ikalash and @jewatkins for help with the spack builds.

[mcarlson801]

@mcarlson801, @mperego,
Is this something we need to somehow fix in the spack package for Albany?

This isn't something I've run into or shown up on the nightlies and unfortunately I don't have access to Chrysalis to check it out. Is there somewhere we can take a look at how you are configuring? Maybe one of the other Albany developers who have run on Chrysalis can help. (I think this is probably Luca or Jerry but I'm not sure)

[mperego]

pinging @kliegeois as well.

[ikalash]

@xylar : are you having spack built everything including cmake and pnetcdf, the packages with the problem? Or you are pointing to existing ones? I agree with @mcarlson801 that it is difficult to troubleshoot this w/o having access to the machine, since these errors don't show up elsewhere.

[xylar]

This isn't something I've run into or shown up on the nightlies and unfortunately I don't have access to Chrysalis to check it out. Is there somewhere we can take a look at how you are configuring? Maybe one of the other Albany developers who have run on Chrysalis can help. (I think this is probably Luca or Jerry but I'm not sure)

@mcarlson801, this happend on Perlmutter the same way. It doesn't seem to be a machine issue.

[xylar]

@xylar : are you having spack built everything including cmake and pnetcdf, the packages with the problem? Or you are pointing to existing ones?

@ikalash, I think pnetcdf is a red herring. It's nothing to do with that. It's that things are failing when we pass the -DPYTHON_EXECUTABLE= argument (i.e. leaving it empty) when we don't want the python features.

Again, this does not seem to be machine specific. It happens to me on Perlmutter, too.

[xylar]

On Perlmutter, the failing Spack build of Albany is in:

/pscratch/sd/x/xylar/spack_tmp/spack-stage/spack-stage-albany-compass-2024-03-13-z5yq62pnbfqca4qb7oe6mu5abrd4oc3k

If you all have Perlmutter access, please have a look. Permissions should be open.

[ikalash]

I guess this is a PyAlbany thing, since that's the only place in Albany Python is needed, in which case @kliegeois may be the best one to take a look. You can try building w/o PyAlbany to see if there are any other errors:

spack --insecure install --dirty --keep-stage albany@develop%[email protected]+mpas+optimization+mesh_depends_on_params+omegah

(remote the +py).

[xylar]

Here's the full spec it's trying to build:

albany@compass-2024-03-13%[email protected]+64bit+confgui~debug+epetra~fpe~ipo+landice~mesh_depends_on_params+mpas~omegah~optimization~perf~py~sandybridge~sfad+unit_tests build_system=cmake build_type=Release generator=make sfadsize=4 arch=linux-sles15-zen3

It's not trying to build with +py, note the ~py.

[xylar]

I don't think it's related but I notices it's with +epetra even though all I explicitly specified was +mpas. That seems odd because epetra seems to be default disabled.

[xylar]

I was hoping you all had seen this elsewhere or had some quick intuition about it. I'll try to debug it some more tomorrow.

[ikalash]

I was hoping you all had seen this elsewhere or had some quick intuition about it. I'll try to debug it some more tomorrow.

Sorry @xylar this does not ring any bells. I will look at the PM output soon. I have been busy this week trying to get caught up after 2 weeks of foreign travel, while sick...

[xylar]

@ikalash, @mperego, @mcarlson801, @kliegeois, @matthewhoffman

The solution turns out to be pretty simple as far as I can tell:
E3SM-Project/spack#27
If PYTHON_EXECUTABLE is undefined, CMake tries to find python itself during testing. However, if it is defined as an empty string, this appears not to be the same as being undefined. It appears to attempt to find a version for python using an empty string as the executable, gets an empty string for the version, and fails to compare that version to 2.7.