Skip to content

Commit

Permalink
Merge branch 'ndk/machinefiles/pm-gpu-Kokkos_ENABLE_IMPL_CUDA_MALLOC_…
Browse files Browse the repository at this point in the history
…ASYNC=OFF' into next (PR #6423)

After #6101 which brings in kokkos 4.2, we see runtime error with a test like:
ERP_Ln9.ne4pg2_ne4pg2.F2010-SCREAMv1.pm-gpu_gnugpu

hits runtime error like:

0: (GTL DEBUG: 0) cuIpcGetMemHandle: invalid argument, CUDA_ERROR_INVALID_VALUE, line no 97
unfortunately, the tests hitting this error are also hanging...

A fix that seems to work is to add this build flag:
-DKokkos_ENABLE_IMPL_CUDA_MALLOC_ASYNC=OFF

The fix is merged in another PR for pm-gpu, so this PR just makes same change to muller-gpu.

Fixes #6422
  • Loading branch information
ndkeen committed May 20, 2024
2 parents 32d507f + 33e40fc commit 12e80e1
Showing 1 changed file with 2 additions and 1 deletion.
3 changes: 2 additions & 1 deletion cime_config/machines/cmake_macros/gnugpu_muller-gpu.cmake
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,8 @@ if (COMP_NAME STREQUAL gptl)
endif()
string(APPEND CPPDEFS " -DTHRUST_IGNORE_CUB_VERSION_CHECK")
string(APPEND CMAKE_CUDA_FLAGS " -ccbin CC -O2 -arch sm_80 --use_fast_math")
string(APPEND KOKKOS_OPTIONS " -DKokkos_ARCH_AMPERE80=On -DKokkos_ENABLE_CUDA=On -DKokkos_ENABLE_CUDA_LAMBDA=On -DKokkos_ENABLE_SERIAL=ON -DKokkos_ENABLE_OPENMP=Off")
string(APPEND KOKKOS_OPTIONS " -DKokkos_ARCH_AMPERE80=On -DKokkos_ENABLE_CUDA=On -DKokkos_ENABLE_CUDA_LAMBDA=On -DKokkos_ENABLE_SERIAL=ON -DKokkos_ENABLE_OPENMP=Off -DKokkos_ENABLE_IMPL_CUDA_MALLOC_ASYNC=OFF")
set(CMAKE_CUDA_ARCHITECTURES "80")
string(APPEND CMAKE_C_FLAGS_RELEASE " -O2")
string(APPEND CMAKE_Fortran_FLAGS_RELEASE " -O2")
set(MPICC "cc")
Expand Down

0 comments on commit 12e80e1

Please sign in to comment.