Issue in compiling MACE's version of LAMMPS with KOKKOS HIP (AMD GPU)

[fxcoudert]

I get a weird error when I try to build MACE's version of LAMMPS (from the mace branch at https://github.com/ACEsuit/lammps) per the instructions. I have built it previously on several platforms, but this time on AMD GPUs using HIP I get stuck.

The command line I use is the following: cmake ../cmake -D CMAKE_BUILD_TYPE=Release -D CMAKE_INSTALL_PREFIX=/lus/work/CT8/cpt7069/fxcoudert/lammps-mace-2024.11.10 -D CMAKE_CXX_STANDARD=17 -D CMAKE_CXX_STANDARD_REQUIRED=ON -D BUILD_MPI=ON -D BUILD_SHARED_LIBS=ON -D PKG_KOKKOS=ON -D Kokkos_ENABLE_HIP=ON -D Kokkos_ARCH_AMD_GFX90A=ON -DKokkos_ENABLE_OPENMP=ON -D BUILD_OMP=ON -D PKG_ML-MACE=ON -D CMAKE_PREFIX_PATH=/opt/software/gaia-external/IA/torch/2.3.0/lib/python3.11/site-packages/torch/share/cmake

So it defines -D Kokkos_ENABLE_HIP=ON -D Kokkos_ARCH_AMD_GFX90A=ON -DKokkos_ENABLE_OPENMP=ON, as I think it should. But then Kokkos complains that no AMD GPU is specified:

CMake Error at /lus/work/CT8/cpt7069/fxcoudert/src/lammps-ACEsuit/lib/kokkos/cmake/kokkos_arch.cmake:590 (MESSAGE):
  HIP enabled but no AMD GPU architecture currently enabled.  Please enable
  one AMD GPU architecture via -DKokkos_ARCH_{..}=ON'.
Call Stack (most recent call first):
  /lus/work/CT8/cpt7069/fxcoudert/src/lammps-ACEsuit/lib/kokkos/cmake/kokkos_tribits.cmake:243 (INCLUDE)
  /lus/work/CT8/cpt7069/fxcoudert/src/lammps-ACEsuit/lib/kokkos/CMakeLists.txt:189 (KOKKOS_SETUP_BUILD_ENVIRONMENT)

However, it has identified and accepted my options, because it also says:

-- Kokkos Devices: HIP;OPENMP
-- Kokkos Architecture: AMD_GFX90A
-- Configuring incomplete, errors occurred!

I'm really wondering what I am doing wrong. Full output is available there: https://gist.github.com/fxcoudert/3635e35eb1cbe84a9825397d39636b2b

Anyone has any idea how to fix this or debug further? That would be really appreciated.

[fxcoudert]

So, it turns out that, despite the documentation recommending -DKokkos_ARCH_AMD_GFX90A=ON, the correct option is actually -DKokkos_ARCH_VEGA90A=ON (for ADM MI200). Don't ask me why.

I am posting my full cmake configuration line below, as a record and for others who might struggle with it this issue at some point.

---

cmake ../cmake -D CMAKE_BUILD_TYPE=Release -D CMAKE_INSTALL_PREFIX=/lus/work/CT8/cpt7069/fxcoudert/lammps-mace-2024.11.10 -D CMAKE_CXX_STANDARD=17 -D CMAKE_CXX_STANDARD_REQUIRED=ON -D BUILD_MPI=ON -D BUILD_SHARED_LIBS=ON -D PKG_KOKKOS=ON -D Kokkos_ENABLE_HIP=ON -DKokkos_ARCH_VEGA90A=ON -DKokkos_ENABLE_OPENMP=ON -D BUILD_OMP=ON -D PKG_ML-MACE=ON -D CMAKE_CXX_COMPILER=CC -D CMAKE_PREFIX_PATH=/opt/software/gaia-external/IA/torch/2.3.0/lib/python3.11/site-packages/torch/share/cmake -D Python_ROOT_DIR=/opt/software/gaia-external/IA/torch/2.3.0 -D MKL_INCLUDE_DIR=/opt/cray/pe/python/3.11.5/include -D GIT_EXECUTABLE=NO

---

With this, I managed to get a successful compilation. Almost. I get errors like this:

[ 11%] Building CXX object CMakeFiles/lammps.dir/lus/work/CT8/cpt7069/fxcoudert/src/lammps-ACEsuit/src/KOKKOS/pair_mace_kokkos.cpp.o
/lus/work/CT8/cpt7069/fxcoudert/src/lammps-ACEsuit/src/KOKKOS/pair_mace_kokkos.cpp:347:31: error: use 'template' keyword to treat 'item' as a dependent template name
  347 |     virial[0] += vir[0][0][0].item<double>();
      |                               ^
      |                               template 

which are solved by this commit from @wcwitt on a different branch
ACEsuit/lammps@f51963a#diff-80ffab535866f0ea37d00490abce5ec3debadbff031469672c9ec0a8d5d3ac5eR350

May I suggest that it be backported to the mace branch, please? 🙇

[wcwitt]

Thanks for sticking with this! Would you be willing to try one of your simulations with the mace-develop branch and report back to us whether it behaves sensibly? The two primary changes in mace-develop are sychonizing with the latest LAMMPS release branch, as well as that template item stuff, which seems important for AMD.

If mace-develop works for you (I'll also ask some others), I'll go ahead and merge everything into mace.

[fxcoudert]

Hi, right now I have yet to run any simulation and validation with MACE at all. I've spent a lot of time working on compiling it correctly for both CUDA and AMD/HIP. But I am getting closer to doing actual science… (or, well, validation studies first, but at least no more spending hours fighting with cmake).

I'll try both branches at some point and report back. Do you know, by chance, how much MACE has been tested on AMD?