Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add cuda-related variants for Albany and Trilinos #31

Merged
merged 3 commits into from
Sep 10, 2024

Conversation

mcarlson801
Copy link

This PR adds variants to albany and trilinos-for-albany spack recipes to support CUDA builds.

A few notes about the included variants:

  1. DFad Sacado types do not work on GPU so I've added an SLFad variant to Albany as an alternative that will work for all GPU cases.
  2. CUDA Unified Virtual Memory (UVM) is off by default but can be enabled by adding +uvm
  3. Same for Tpetra's GPU_AWARE_MPI (add +aware to enable)
  4. Added +zen3 and +ampere80 architecture flags for Perlmutter-GPU

@mcarlson801
Copy link
Author

@jewatkins @xylar

@xylar
Copy link
Collaborator

xylar commented Aug 12, 2024

@mcarlson801, great, I'll test this tomorrow.

"-DALBANY_SLFAD_SIZE=90",
"-DALBANY_TAN_SLFAD_SIZE=100",
"-DALBANY_HES_VEC_SLFAD_SIZE=100"
])

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: this is needed for Albany tests but it might not be needed for mali/e3sm tests. in fact, I think one of the e3sm tests fails with this because of oom issues. sfad12 might be sufficient for all mali/e3sm tests.

@xylar xylar mentioned this pull request Aug 13, 2024
5 tasks
@xylar
Copy link
Collaborator

xylar commented Aug 13, 2024

@mcarlson801 and @jewatkins, please see my comments here: MPAS-Dev/compass#835 (comment)

@mcarlson801
Copy link
Author

@xylar I think these are all the changes we need for a GPU build. Is there anything else we need to do before it can be merged?

Copy link
Collaborator

@xylar xylar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was able to build Trilinos and Albany with these changes on Perlmutter-GPU and run MALI with them, see:
MPAS-Dev/compass#835

@xylar xylar merged commit 4815633 into E3SM-Project:develop Sep 10, 2024
7 of 11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants