Update chicoma-cpu modules #112

xylar · 2024-10-11T23:48:54Z

Following the recent DST, this merge updates the module files and environment variables on Chicoma-CPU. We note that these updates work well for gnu and nvidia compilers but not yet for intel, which we are continuing to work on. A separate update will be needed to address Chicoma-GPU as well.

xylar · 2024-10-11T23:49:34Z

This is just a draft so far. I'm having no luck with either gnu or intel on Chicoma-CPU so far. I haven't tried anything else yet.

xylar · 2024-10-11T23:55:47Z

I've contacted LANL IC about the trouble I'm having with gnu:

/lustre/scratch5/xylar/E3SM/scratch/chicoma-cpu/SMS_D.TL319_oQU240wLI_ais8to30.MPAS_LISIO_JRA1p5.chicoma-cpu_gnu.20241011_152619_r3zumf/bld/e3sm.exe: /opt/cray/pe/gcc-libs/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /lustre/scratch5/xylar/E3SM/scratch/chicoma-cpu/SMS_D.TL319_oQU240wLI_ais8to30.MPAS_LISIO_JRA1p5.chicoma-cpu_gnu.20241011_152619_r3zumf/bld/e3sm.exe)

While it seems clear that there's an RPATH being set to /opt/cray/pe/gcc-libs, I haven't been able to track down where that's coming from. Setting the LD_LIBRARY_PATH didn't help.

xylar · 2024-10-11T23:56:34Z

On the intel side, it's not finding NetCDF-C or -Fortran, even though we're passing a NETCDF_PATH environment variable that seems correct.

xylar · 2024-10-14T03:10:44Z

The gnu issue seems similar to E3SM-Project#6677

jonbob · 2024-10-17T15:45:40Z

With the commits I just pushed, I was able to successfully build and run:

SMS.TL319_oQU240wLI_ais8to30.MPAS_LISIO_JRA1p5.chicoma-cpu_gnu
SMS_D.TL319_oQU240wLI_ais8to30.MPAS_LISIO_JRA1p5.chicoma-cpu_gnu

So I think at this point we can say we support gnu on chicoma. I'll poke around at intel as well

xylar · 2024-10-17T16:27:34Z

cime_config/machines/config_machines.xml

-    <CCSM_CPRNC>/usr/projects/climate/SHARED_CLIMATE/software/badger/cprnc</CCSM_CPRNC>
+    <CCSM_CPRNC>/usr/projects/e3sm/software/chicoma-cpu/cprnc</CCSM_CPRNC>


Oh, I bet I know what happened here. I deleted this thinking that it was old and no longer used. In my defense, it has badger in the path...

And it no longer exists! Not just the machine but that file. But I think if that line is missing it forces each test to try to build it

And it no longer exists!

That's what I was saying. I think I deleted it trying to free up space in /usr/projects/climate because I couldn't imagine we were still using software built for Badger.

xylar · 2024-10-17T16:28:42Z

cime_config/machines/config_machines.xml

+        <command name="unload">PrgEnv-gnu</command>
+        <command name="unload">PrgEnv-intel</command>
+        <command name="unload">PrgEnv-nvidia</command>
+        <command name="unload">PrgEnv-cray</command>
+        <command name="unload">PrgEnv-aocc</command>


I found that these needed to be unloaded after their corresponding compiler modules or there would be an error about an undefined environment variable name.

xylar · 2024-10-17T16:50:54Z

@jonbob, I'm trying to run a test:

./create_test SMS_D.TL319_oQU240wLI_ais8to30.MPAS_LISIO_JRA1p5.chicoma-cpu_gnu --walltime 00:30:00 --wait -p w23_freddy

This looks to be what you ran successfully. But for me it just seems to be hanging. It hasn't got to ocean time stepping yet and there's very little output in the e3sm log file.

Could you have a quick look and let me know if you see anything obvious?

/users/xylar/scratch5/E3SM/scratch/chicoma-cpu/SMS_D.TL319_oQU240wLI_ais8to30.MPAS_LISIO_JRA1p5.chicoma-cpu_gnu.20241017_103208_7tp6zr

jonbob · 2024-10-17T16:54:09Z

@xylar -- let me take a peek

jonbob · 2024-10-17T17:02:20Z

@xylar - it seems to be struggling with the atm data? That doesn't make much sense

xylar · 2024-10-17T17:02:22Z

In the meantime, I'm trying an optimized run to see how that goes.

xylar · 2024-10-17T17:06:35Z

SMS_D.TL319_oQU240wLI_ais8to30.MPAS_LISIO_JRA1p5.chicoma-cpu_gnu passed for me in the end. It just took 25 minutes and didn't get to time stepping for a long time. It seems like it might be a file system issue with /usr/projects/e3sm.

xylar · 2024-10-17T17:29:03Z

SMS.TL319_oQU240wLI_ais8to30.MPAS_LISIO_JRA1p5.chicoma-cpu_gnu passed for me as well.

xylar · 2024-10-17T18:26:35Z

I tested SMS_D.TL319_oQU240wLI_ais8to30.MPAS_LISIO_JRA1p5.chicoma-cpu_nvidia and it built fine and appeared to be running but timed out before the 30 minutes I gave it (same file system issues as above). Waiting in the queue with a longer test.

xylar · 2024-10-18T14:30:27Z

I realize it's not a high priority for us but SMS_D.TL319_oQU240wLI_ais8to30.MPAS_LISIO_JRA1p5.chicoma-cpu_nvidia passed for me with a longer job runtime.

xylar · 2024-10-19T21:15:42Z

cime_config/machines/config_machines.xml

+        <command name="load">PrgEnv-nvidia/8.5.0</command>
+        <command name="load">nvidia/24.7</command>


I successfully tested these updated modules as well.

This should be set to: ``` export GNU_CRAY_LDFLAGS="-Wl,--enable-new-dtags" ``` on Chicoma-CPU with gcc.

xylar · 2024-10-21T00:41:17Z

components/mpas-framework/Makefile

@@ -396,11 +396,11 @@ gnu-cray:
 	"FFLAGS_OPT = -O3 -m64 -ffree-line-length-none -fconvert=big-endian -ffree-form -ffpe-summary=none $${EXTRA_FFLAGS}" \
 	"CFLAGS_OPT = -O3 -m64" \
 	"CXXFLAGS_OPT = -O3 -m64" \
-	"LDFLAGS_OPT = -O3 -m64" \
+	"LDFLAGS_OPT = -O3 -m64 $(GNU_CRAY_LDFLAGS)" \


@matthewhoffman, this environment variable (or argument to make) needs to be set to:

export GNU_CRAY_LDFLAGS="-Wl,--enable-new-dtags"

on Chicoma for now. I'll make sure Compass and Polaris do this. If someone is building for Chicoma outside of Compass or Polaris (good luck!), they would need to set this manually.

Are you okay with this fix? I don't want to put in anything into the Makefile that tries to detect the machine or anything crazy like that.

@xylar , this seems like the best solution given the circumstances

cime_config/machines/config_machines.xml

xylar · 2024-10-21T14:53:59Z

@jonbob, at the risk of delaying this further, I think we probably want to follow what Noel is doing on Perlmutter:
https://github.com/E3SM-Project/E3SM/pull/6702/files
That should at least save us from having to make yet another PR in the near future.

This is to match proposed updates to Perlmutter-CPU E3SM-Project#6702

xylar · 2024-10-21T18:38:01Z

cime_config/machines/config_machines.xml

        <command name="unload">cray-parallel-netcdf</command>
        <command name="unload">cray-netcdf</command>
        <command name="unload">cray-hdf5</command>


I tried removing cpe like Noel did here:
https://github.com/E3SM-Project/E3SM/blob/bdcc2f551cfae2fca53bd8aa4ec604601ddf1c68/cime_config/machines/config_machines.xml#L193
But I got nasty error like:

Lmod has detected the following error: These module(s) or extension(s) exist but cannot be loaded as requested: "git", "cmake/3.27.7" Try: "module spider git cmake/3.27.7" to see how to load the module(s).

xylar · 2024-10-21T18:47:26Z

I have gnu and nvidia tests in the queue with the latest updates.

xylar · 2024-10-21T19:37:05Z

The following both passed:

SMS_D.TL319_oQU240wLI_ais8to30.MPAS_LISIO_JRA1p5.chicoma-cpu_gnu
SMS_D.TL319_oQU240wLI_ais8to30.MPAS_LISIO_JRA1p5.chicoma-cpu_nvidia

xylar · 2024-10-22T16:36:58Z

Closed in favor of E3SM-Project#6705

xylar mentioned this pull request Oct 14, 2024

Runtime error with GNU libraries for scream tests E3SM-Project/E3SM#6677

Closed

xylar commented Oct 17, 2024

View reviewed changes

xylar and others added 3 commits October 17, 2024 10:30

Update chicoma-cpu modules

5b539bc

Add latest cmake

a1d98a7

Updates to get E3SM working on chicoma-cpu

9c0b308

xylar force-pushed the machine/update-chicoma-modules branch from df21910 to 9c0b308 Compare October 17, 2024 16:30

xylar changed the base branch from master to alternate October 17, 2024 16:32

xylar changed the base branch from alternate to master October 17, 2024 16:32

altheaden mentioned this pull request Oct 18, 2024

Update chicoma-cpu modules E3SM-Project/mache#201

Merged

1 task

Update nvidia modules

c5552b9

xylar commented Oct 19, 2024

View reviewed changes

Add GNU_CRAY_LDFLAGS flag to MPAS standalone builds

ca87958

This should be set to: ``` export GNU_CRAY_LDFLAGS="-Wl,--enable-new-dtags" ``` on Chicoma-CPU with gcc.

xylar commented Oct 21, 2024

View reviewed changes

cime_config/machines/config_machines.xml Outdated Show resolved Hide resolved

xylar mentioned this pull request Oct 21, 2024

Update to 1.6.0 alpha.1 MPAS-Dev/compass#865

Merged

35 tasks

Revert chicoma-gpu changes (for now)

d379003

xylar force-pushed the machine/update-chicoma-modules branch from 870b287 to d379003 Compare October 21, 2024 03:44

xylar force-pushed the machine/update-chicoma-modules branch from 871a4b7 to b33833c Compare October 21, 2024 17:51

Update gcc and nvidia to newer versions

c34336e

This is to match proposed updates to Perlmutter-CPU E3SM-Project#6702

xylar force-pushed the machine/update-chicoma-modules branch from b33833c to c34336e Compare October 21, 2024 18:35

xylar commented Oct 21, 2024

View reviewed changes

xylar mentioned this pull request Oct 22, 2024

Update chicoma-cpu gnu and nvidia builds E3SM-Project/E3SM#6705

Merged

xylar closed this Oct 22, 2024

dylanschlichting mentioned this pull request Nov 25, 2024

GPU test case fails on Chicoma E3SM-Project/E3SM#6778

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update chicoma-cpu modules #112

Update chicoma-cpu modules #112

xylar commented Oct 11, 2024 •

edited

Loading

xylar commented Oct 11, 2024

xylar commented Oct 11, 2024

xylar commented Oct 11, 2024

xylar commented Oct 14, 2024

jonbob commented Oct 17, 2024

xylar Oct 17, 2024

jonbob Oct 17, 2024

xylar Oct 17, 2024

xylar Oct 17, 2024

xylar commented Oct 17, 2024

jonbob commented Oct 17, 2024

jonbob commented Oct 17, 2024

xylar commented Oct 17, 2024

xylar commented Oct 17, 2024 •

edited

Loading

xylar commented Oct 17, 2024

xylar commented Oct 17, 2024 •

edited

Loading

xylar commented Oct 18, 2024

xylar Oct 19, 2024

xylar Oct 21, 2024

matthewhoffman Oct 22, 2024

xylar commented Oct 21, 2024

xylar Oct 21, 2024

xylar commented Oct 21, 2024

xylar commented Oct 21, 2024

xylar commented Oct 22, 2024

		<CCSM_CPRNC>/usr/projects/climate/SHARED_CLIMATE/software/badger/cprnc</CCSM_CPRNC>
		<CCSM_CPRNC>/usr/projects/e3sm/software/chicoma-cpu/cprnc</CCSM_CPRNC>

		<command name="load">PrgEnv-nvidia/8.5.0</command>
		<command name="load">nvidia/24.7</command>

Update chicoma-cpu modules #112

Update chicoma-cpu modules #112

Conversation

xylar commented Oct 11, 2024 • edited Loading

xylar commented Oct 11, 2024

xylar commented Oct 11, 2024

xylar commented Oct 11, 2024

xylar commented Oct 14, 2024

jonbob commented Oct 17, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xylar commented Oct 17, 2024

jonbob commented Oct 17, 2024

jonbob commented Oct 17, 2024

xylar commented Oct 17, 2024

xylar commented Oct 17, 2024 • edited Loading

xylar commented Oct 17, 2024

xylar commented Oct 17, 2024 • edited Loading

xylar commented Oct 18, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xylar commented Oct 21, 2024

Choose a reason for hiding this comment

xylar commented Oct 21, 2024

xylar commented Oct 21, 2024

xylar commented Oct 22, 2024

xylar commented Oct 11, 2024 •

edited

Loading

xylar commented Oct 17, 2024 •

edited

Loading

xylar commented Oct 17, 2024 •

edited

Loading