Minor changes for our CUDA testing machine/workflows

[bartgol]

• In machine specs, instead of hardcoding the number of GPUs, use nvidia-smi to query it (must be already on compute node)
• For eamxx-sa workflow, pick the correct CUDA arch cmake specs depending on nvidia-smi output.

This allows to move our gh action containers to A100, V100, H100 without changing anything else in our code.Grab number of gpus without hard-coding it

[mergify]

Merge Protections

Your pull request matches the following merge protections and will not be merged until they are valid.

🔴 Enforce checks passing

This rule is failing.

Make sure that checks are not failing on the PR, and reviewers approved

• #approved-reviews-by >= 1
• any of:
  • check-skipped={% raw %}gcc-openmp / ${{ matrix.build_type }}{% endraw %}
  • all of:
    • check-success="gcc-openmp / dbg"
    • check-success="gcc-openmp / fpe"
    • check-success="gcc-openmp / opt"
    • check-success="gcc-openmp / sp"
• any of:
  • check-skipped={% raw %}gcc-cuda / ${{ matrix.build_type }}{% endraw %}
  • all of:
    • check-success="gcc-cuda / dbg"
    • check-success="gcc-cuda / opt"
    • check-success="gcc-cuda / sp"
• any of:
  • check-skipped={% raw %}cpu-gcc / ${{ matrix.test.short_name }}{% endraw %}
  • all of:
    • check-success="cpu-gcc / ERS_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.scream-small_kernels--scream-output-preset-5"
    • check-success="cpu-gcc / ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.scream-output-preset-2"
    • check-success="cpu-gcc / ERS_P16_Ln22.ne30pg2_ne30pg2.FIOP-SCREAMv1-DP.scream-dpxx-arm97"
    • check-success="cpu-gcc / SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.scream-mam4xx-all_mam4xx_procs"
• any of:
  • check-skipped=cpu-gcc
  • check-success=cpu-gcc
• #changes-requested-reviews-by == 0

[bartgol]

Unfortunately, manually testing the container on the H100 queue on blake seems to consistently give a build error:

nvcc error   : 'ptxas' died due to signal 11 (Invalid memory reference)
nvcc error   : 'ptxas' core dumped

It happens when building shoc_functions_f90.cpp. The p3 analogue built, so it shuld not have anything to do with f2c per se. I wonder if @jgfouca's work on removing some f90 support in shoc as well may fix this...

Edit: interestingly, this error appears for the SP/DEBUG builds, but not for the OPT build. So I think we can move on, and get some testing going...

NOTE: with this branch, I was able to generate baselines on blake's H100 partition (foir the release build)

[bartgol]

Merging so that we may get testing reporting tonight. We can adjust tomorrow, but it's definitely not harming to merge since we have no GPU testing anyways right now.