[Build] Build Error with MVAPICH2-GDR

[s769]

Describe the issue

A clear and concise description of what the issue is.

I am trying to build AMGX with MVAPICH2-GDR on the TACC Supercomputer Lonestar6 as the MPI. Building with Intel MPI works fine.

Environment information:

• OS: Rocky Linux release 8.6 (Green Obsidian)
• Compiler version: Intel 19.1.0
• CMake version: 3.24.2
• CUDA used for AMGX compilation: CUDA 12.0
• MPI version (if applicable): MVAPICH2-GDR v2.3.7
• AMGX version or commit hash v 2.4.0 f280993e5df32701fb28c2636f49eea4bc5753a4
• Any related environment variables information

Configuration information

Provide your cmake command line that was used for configuration and it's full output.

cmake -DCMAKE_C_COMPILER=mpicc -DCMAKE_CXX_COMPILER=mpic++ -DCMAKE_CUDA_HOST_COMPILER=mpic++ -DCUDA_ARCH=80 ..

Output

-- The C compiler identification is Intel 19.1.0.20200306
-- The CXX compiler identification is Intel 19.1.0.20200306
-- The CUDA compiler identification is NVIDIA 12.0.140
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /opt/apps/intel19/mvapich2-gdr/2.3.7/bin/mpicc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /opt/apps/intel19/mvapich2-gdr/2.3.7/bin/mpic++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - done
-- Check for working CUDA compiler: /opt/apps/cuda/12.0/bin/nvcc - skipped
-- Detecting CUDA compile features
-- Detecting CUDA compile features - done
-- Found MPI_C: /opt/apps/intel19/mvapich2-gdr/2.3.7/bin/mpicc (found version "3.1") 
-- Found MPI_CXX: /opt/apps/intel19/mvapich2-gdr/2.3.7/bin/mpic++ (found version "3.1") 
-- Found MPI: TRUE (found version "3.1")
-- Found OpenMP_C: -qopenmp (found version "5.0") 
-- Found OpenMP_CXX: -qopenmp (found version "5.0") 
-- Found OpenMP: TRUE (found version "5.0")
-- Found CUDAToolkit: /opt/apps/cuda/12.0/include (found suitable version "12.0.140", minimum required is "10.0")
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Found OpenMP_C: -qopenmp (found version "5.0") 
-- Found OpenMP_CXX: -qopenmp (found version "5.0")
This is a MPI build:TRUE
-- Found libcudacxx: /work/08435/srvenkat/frontera/AMGX/thrust/dependencies/libcudacxx/lib/cmake/libcudacxx/libcudacxx-config.cmake (found suitable version "1.8.1.0", minimum required is "1.8.0")
-- Found Thrust: /work/08435/srvenkat/frontera/AMGX/thrust/thrust/cmake/thrust-config.cmake (found version "2.1.0.0")
-- Found CUB: /work/08435/srvenkat/frontera/AMGX/thrust/dependencies/cub/cub/cmake/cub-config.cmake (found suitable version "2.1.0.0", minimum required is "2.1.0.0")
-- Configuring done
-- Generating done
-- Build files have been written to: /work/08435/srvenkat/frontera/AMGX/build2

Compilation information

Provide your make command

make all VERBOSE=1

Output

/opt/apps/cmake/3.24.2/bin/cmake -S/work/08435/srvenkat/frontera/AMGX -B/work/08435/srvenkat/frontera/AMGX/build2 --check-build-system CMakeFiles/Makefile.cmake 0
/opt/apps/cmake/3.24.2/bin/cmake -E cmake_progress_start /work/08435/srvenkat/frontera/AMGX/build2/CMakeFiles /work/08435/srvenkat/frontera/AMGX/build2//CMakeFiles/progress.marks
make  -f CMakeFiles/Makefile2 all
make[1]: Entering directory '/work/08435/srvenkat/frontera/AMGX/build2'
make  -f CMakeFiles/amgx_libs.dir/build.make CMakeFiles/amgx_libs.dir/depend
make[2]: Entering directory '/work/08435/srvenkat/frontera/AMGX/build2'
cd /work/08435/srvenkat/frontera/AMGX/build2 && /opt/apps/cmake/3.24.2/bin/cmake -E cmake_depends "Unix Makefiles" /work/08435/srvenkat/frontera/AMGX /work/08435/srvenkat/frontera/AMGX /work/08435/srvenkat/frontera/AMGX/build2 /work/08435/srvenkat/frontera/AMGX/build2 /work/08435/srvenkat/frontera/AMGX/build2/CMakeFiles/amgx_libs.dir/DependInfo.cmake --color=
make[2]: Leaving directory '/work/08435/srvenkat/frontera/AMGX/build2'
make  -f CMakeFiles/amgx_libs.dir/build.make CMakeFiles/amgx_libs.dir/build
make[2]: Entering directory '/work/08435/srvenkat/frontera/AMGX/build2'
[  0%] Building CUDA object CMakeFiles/amgx_libs.dir/src/aggregation/aggregation_amg_level.cu.o
/opt/apps/cuda/12.0/bin/nvcc -forward-unknown-to-host-compiler -ccbin=mpic++ -DTHRUST_DEVICE_SYSTEM=THRUST_DEVICE_SYSTEM_CUDA -DTHRUST_HOST_SYSTEM=THRUST_HOST_SYSTEM_CPP -I/work/08435/srvenkat/frontera/AMGX/external/rapidjson/include -I/work/08435/srvenkat/frontera/AMGX/src/../include -I/work/08435/srvenkat/frontera/AMGX/thrust/thrust/cmake/../.. -I/work/08435/srvenkat/frontera/AMGX/thrust/dependencies/libcudacxx/include -I/work/08435/srvenkat/frontera/AMGX/thrust/dependencies/cub/cub/cmake/../.. -O3 -DNDEBUG --generate-code=arch=compute_80,code=[compute_80,sm_80] -DNDEBUG --extended-lambda --Werror cross-execution-space-call -DNVTX_RANGES -DDISABLE_MIXED_PRECISION -DCUSPARSE_GENERIC_INTERFACES -DCUSPARSE_USE_GENERIC_SPGEMM -Xcompiler "-qopenmp -DRAPIDJSON_DEFINED -DAMGX_WITH_MPI -rdynamic -fPIC -fvisibility=default" -DTHRUST_CUB_WRAPPED_NAMESPACE=amgx -std=c++14 -MD -MT CMakeFiles/amgx_libs.dir/src/aggregation/aggregation_amg_level.cu.o -MF CMakeFiles/amgx_libs.dir/src/aggregation/aggregation_amg_level.cu.o.d -x cu -c /work/08435/srvenkat/frontera/AMGX/src/aggregation/aggregation_amg_level.cu -o CMakeFiles/amgx_libs.dir/src/aggregation/aggregation_amg_level.cu.o
/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/memory.h(28): error: name followed by "::" must be a class or namespace name

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/memory.h(29): error: name followed by "::" must be a class or namespace name

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/memory.h(30): error: name followed by "::" must be a class or namespace name

/work/08435/srvenkat/frontera/AMGX/src/../include/vector_thrust_allocator.h(58): error: qualified name is not allowed

/work/08435/srvenkat/frontera/AMGX/src/../include/vector_thrust_allocator.h(58): error: explicit type is missing ("int" assumed)

/work/08435/srvenkat/frontera/AMGX/src/../include/vector_thrust_allocator.h(58): error: expected a ";"

/work/08435/srvenkat/frontera/AMGX/src/../include/vector_thrust_allocator.h(61): error: qualified name is not allowed

/work/08435/srvenkat/frontera/AMGX/src/../include/vector_thrust_allocator.h(61): error: explicit type is missing ("int" assumed)

/work/08435/srvenkat/frontera/AMGX/src/../include/vector_thrust_allocator.h(61): error: expected a ";"

/work/08435/srvenkat/frontera/AMGX/src/../include/vector_thrust_allocator.h(64): error: qualified name is not allowed

/work/08435/srvenkat/frontera/AMGX/src/../include/vector_thrust_allocator.h(64): error: explicit type is missing ("int" assumed)

/work/08435/srvenkat/frontera/AMGX/src/../include/vector_thrust_allocator.h(64): error: expected a ";"

/work/08435/srvenkat/frontera/AMGX/src/../include/vector_thrust_allocator.h(67): error: qualified name is not allowed

/work/08435/srvenkat/frontera/AMGX/src/../include/vector_thrust_allocator.h(67): error: explicit type is missing ("int" assumed)

/work/08435/srvenkat/frontera/AMGX/src/../include/vector_thrust_allocator.h(67): error: expected a ";"

/work/08435/srvenkat/frontera/AMGX/src/../include/vector_thrust_allocator.h(73): error: a class or namespace qualified name is required       

/work/08435/srvenkat/frontera/AMGX/src/../include/vector_thrust_allocator.h(73): error: global-scope qualifier (leading "::") is not allowed  

/work/08435/srvenkat/frontera/AMGX/src/../include/vector_thrust_allocator.h(73): error: expected a ";"

/work/08435/srvenkat/frontera/AMGX/src/../include/vector_thrust_allocator.h(109): error: identifier "pointer" is undefined

/work/08435/srvenkat/frontera/AMGX/src/../include/vector_thrust_allocator.h(109): error: identifier "reference" is undefined

/work/08435/srvenkat/frontera/AMGX/src/../include/vector_thrust_allocator.h(115): error: identifier "const_pointer" is undefined

/work/08435/srvenkat/frontera/AMGX/src/../include/vector_thrust_allocator.h(115): error: identifier "const_reference" is undefined

/work/08435/srvenkat/frontera/AMGX/src/../include/vector_thrust_allocator.h(115): error: invalid redeclaration of member function "amgx::thrust_amgx_allocator<T, 1>::address(<error-type>)"
(109): here

/work/08435/srvenkat/frontera/AMGX/src/../include/vector_thrust_allocator.h(123): error: identifier "pointer" is undefined

/work/08435/srvenkat/frontera/AMGX/src/../include/vector_thrust_allocator.h(124): error: identifier "const_pointer" is undefined

/work/08435/srvenkat/frontera/AMGX/src/../include/vector_thrust_allocator.h(138): error: identifier "pointer" is undefined

/work/08435/srvenkat/frontera/AMGX/src/../include/vector_thrust_allocator.h(263): error: namespace "amgx::thrust" has no member "device_vector"

/work/08435/srvenkat/frontera/AMGX/src/../include/vector_thrust_allocator.h(263): error: expected a ";"

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/detail/memory.inl(48): error: name followed by "::" must be a class or namespace name  

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/detail/memory.inl(49): error: not a class or struct name

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/detail/memory.inl(63): warning #12-D: parsing restarts here after previous syntax error

Remark: The warnings can be suppressed with "-diag-suppress <warning-number>"

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/detail/memory.inl(49): error: class or struct definition is missing

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/detail/memory.inl(63): warning #12-D: parsing restarts here after previous syntax error

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/memory.h(48): error: class "cusp::CuspMemMap<cusp::host_memory>" has already been defined

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/memory.h(49): error: class "cusp::CuspMemMap<cusp::host_memory>" has already been defined

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(54): error: name followed by "::" must be a class or namespace name

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(54): error: expected an identifier

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(58): error: name followed by "::" must be a class or namespace name

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(58): error: expected an identifier

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(58): warning #1307-D: class member typedef may not be redeclared

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(58): error: expected a ";"

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(73): error: a class or namespace qualified name is required

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(73): error: expected a ">"

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(77): error: a class or namespace qualified name is required

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(77): error: expected a ">"

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(79): error: a class or namespace qualified name is required

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(79): error: global-scope qualifier (leading "::") is not allowed

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(79): error: expected a ";"

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(80): error: a class or namespace qualified name is required

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(80): error: global-scope qualifier (leading "::") is not allowed

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(80): error: expected a ";"

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(84): error: identifier "size_type" is undefined

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(94): error: identifier "size_type" is undefined

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(94): error: identifier "value_type" is undefined

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(98): error: name followed by "::" must be a class or namespace name

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(98): error: expected an identifier

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(98): error: expected a ")"

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(82): error: "Parent" is not a nonstatic data member or base class of class "cusp::array1d<T, MemorySpace>"

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(85): error: "Parent" is not a nonstatic data member or base class of class "cusp::array1d<T, MemorySpace>"

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(89): error: name followed by "::" must be a class or namespace name

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(90): error: name followed by "::" must be a class or namespace name

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(95): error: "Parent" is not a nonstatic data member or base class of class "cusp::array1d<T, MemorySpace>"

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(99): error: "Parent" is not a nonstatic data member or base class of class "cusp::array1d<T, MemorySpace>"

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(103): error: "Parent" is not a nonstatic data member or base class of class "cusp::array1d<T, MemorySpace>"

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(107): error: name followed by "::" must be a class or namespace name

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(132): error: namespace "amgx::thrust" has no member "iterator_reference"     

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(132): error: expected an identifier

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(133): error: namespace "amgx::thrust" has no member "iterator_difference"    

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(133): error: expected an identifier

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(134): error: namespace "amgx::thrust" has no member "iterator_value"

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(134): error: expected an identifier

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(136): error: namespace "amgx::thrust" has no member "iterator_system"        

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(136): error: expected an identifier

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(140): error: namespace "amgx::thrust" has no member "iterator_pointer"       

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(140): error: expected an identifier

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(144): error: identifier "value_type" is undefined

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(144): error: identifier "memory_space" is undefined

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(185): error: identifier "reference" is undefined

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(190): error: identifier "reference" is undefined

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(195): error: identifier "reference" is undefined

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(195): error: identifier "difference_type" is undefined

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(221): error: identifier "pointer" is undefined

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/detail/array1d.inl(26): error: name followed by "::" must be a class or namespace name 

/work/08435/srvenkat/frontera/AMGX/src/../include/vector.h(166): error: namespace "amgx::thrust" has no member class "host_vector"

/work/08435/srvenkat/frontera/AMGX/src/../include/vector.h(166): error: class or struct definition is missing

/work/08435/srvenkat/frontera/AMGX/src/../include/vector.h(424): warning #12-D: parsing restarts here after previous syntax error

/work/08435/srvenkat/frontera/AMGX/src/../include/vector.h(428): error: not a class or struct name

/work/08435/srvenkat/frontera/AMGX/src/../include/vector.h(624): error: invalid redeclaration of member function "amgx::Vector<amgx::TemplateConfig<(AMGX_MemorySpace)1, t_vecPrec, t_matPrec, t_indPrec>>::operator=(const cusp::array1d<amgx::Vector<amgx::TemplateConfig<(AMGX_MemorySpace)1, t_vecPrec, t_matPrec, t_indPrec>>::value_type, amgx::MemorySpaceMap<(AMGX_MemorySpace)0>::Type> &)"
(623): here

/work/08435/srvenkat/frontera/AMGX/src/../include/vector.h(442): error: "device_vector_alloc" is not a nonstatic data member or base class of class "amgx::Vector<amgx::TemplateConfig<(AMGX_MemorySpace)1, t_vecPrec, t_matPrec, t_indPrec>>"

/work/08435/srvenkat/frontera/AMGX/src/../include/vector.h(443): error: "device_vector_alloc" is not a nonstatic data member or base class of class "amgx::Vector<amgx::TemplateConfig<(AMGX_MemorySpace)1, t_vecPrec, t_matPrec, t_indPrec>>"

/work/08435/srvenkat/frontera/AMGX/src/../include/vector.h(444): error: "device_vector_alloc" is not a nonstatic data member or base class of class "amgx::Vector<amgx::TemplateConfig<(AMGX_MemorySpace)1, t_vecPrec, t_matPrec, t_indPrec>>"

/work/08435/srvenkat/frontera/AMGX/src/../include/vector.h(445): error: "device_vector_alloc" is not a nonstatic data member or base class of class "amgx::Vector<amgx::TemplateConfig<(AMGX_MemorySpace)1, t_vecPrec, t_matPrec, t_indPrec>>"

/work/08435/srvenkat/frontera/AMGX/src/../include/vector.h(446): error: "device_vector_alloc" is not a nonstatic data member or base class of class "amgx::Vector<amgx::TemplateConfig<(AMGX_MemorySpace)1, t_vecPrec, t_matPrec, t_indPrec>>"

/work/08435/srvenkat/frontera/AMGX/src/../include/vector.h(630): error: namespace "amgx::thrust" has no member "raw_pointer_cast"

/work/08435/srvenkat/frontera/AMGX/src/../include/vector.h(635): error: namespace "amgx::thrust" has no member "raw_pointer_cast"

/work/08435/srvenkat/frontera/AMGX/src/../include/distributed/distributed_manager.h(299): error: namespace "amgx::thrust" has no member "host_vector"

/work/08435/srvenkat/frontera/AMGX/src/../include/distributed/distributed_manager.h(299): error: expected a ")"

/work/08435/srvenkat/frontera/AMGX/src/../include/distributed/distributed_manager.h(309): error: identifier "B2L_maps_" is undefined

/work/08435/srvenkat/frontera/AMGX/src/../include/distributed/distributed_manager.h(453): error: namespace "amgx::thrust" has no member "copy"

/work/08435/srvenkat/frontera/AMGX/src/../include/distributed/distributed_manager.h(1129): error: namespace "amgx::thrust" has no member "copy"

/work/08435/srvenkat/frontera/AMGX/src/../include/distributed/distributed_manager.h(1136): error: namespace "amgx::thrust" has no member "copy"

/work/08435/srvenkat/frontera/AMGX/src/../include/distributed/distributed_manager.h(1545): error: namespace "amgx::thrust" has no member "raw_pointer_cast"

/work/08435/srvenkat/frontera/AMGX/src/../include/distributed/distributed_manager.h(1546): error: namespace "amgx::thrust" has no member "raw_pointer_cast"

/work/08435/srvenkat/frontera/AMGX/src/../include/distributed/distributed_manager.h(1615): error: namespace "amgx::thrust" has no member "raw_pointer_cast"

Error limit reached.
100 errors detected in the compilation of "/work/08435/srvenkat/frontera/AMGX/src/aggregation/aggregation_amg_level.cu".
Compilation terminated.
make[2]: *** [CMakeFiles/amgx_libs.dir/build.make:76: CMakeFiles/amgx_libs.dir/src/aggregation/aggregation_amg_level.cu.o] Error 4
make[2]: Leaving directory '/work/08435/srvenkat/frontera/AMGX/build2'
make[1]: *** [CMakeFiles/Makefile2:158: CMakeFiles/amgx_libs.dir/all] Error 2
make[1]: Leaving directory '/work/08435/srvenkat/frontera/AMGX/build2'
make: *** [Makefile:136: all] Error 2

[marsaev]

can you check that thrust submodule is up to date?
git submodule update --init --recursive

[marsaev]

I will also strongly recommend to use newer CUDA if possible, i don't think we still testing CUDA 11.x, and there should be quite a few improvements (compiler, cusparse, etc) since 11.4 which is 2 yeasrs old.

[s769]

The thrust submodule is up to date. I have switched to CUDA 12.0 now, and still seeing the error. Updated the original post.

[marsaev]

Thanks for confirming.

The configuration output, command lines and compiler command lines look fine for me. Considering that Intel MPI worked for you it seems that something might happen in the mvapich wrapper to host compiler. Could it be that mvapich wrapper addresses compiler other than Intel Compiler and they got mixed up?
Can you try configuring by providing regular host compilers (probably icc in your case), and hopefully FindMPI module will generate appropriate linker flags for the host compiler. So something like:
cmake -DCMAKE_C_COMPILER=icc -DCMAKE_CXX_COMPILER=icc -DCMAKE_CUDA_HOST_COMPILER=icc -DCUDA_ARCH=80 ..

[s769]

Here is the output of cmake -DCMAKE_C_COMPILER=icc -DCMAKE_CXX_COMPILER=icc -DCMAKE_CUDA_HOST_COMPILER=icc -DCUDA_ARCH=80 ..

-- The C compiler identification is Intel 19.1.0.20200306
-- The CXX compiler identification is Intel 19.1.0.20200306
-- The CUDA compiler identification is NVIDIA 12.0.140
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /opt/intel/compilers_and_libraries_2020.1.217/linux/bin/intel64/icc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /opt/intel/compilers_and_libraries_2020.1.217/linux/bin/intel64/icc - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - done
-- Check for working CUDA compiler: /opt/apps/cuda/12.0/bin/nvcc - skipped
-- Detecting CUDA compile features
-- Detecting CUDA compile features - done
-- Found MPI_C: /opt/apps/cuda/11.4/targets/x86_64-linux/lib/stubs/libcuda.so (found version "3.1") 
-- Found MPI_CXX: /opt/apps/cuda/11.4/targets/x86_64-linux/lib/stubs/libcuda.so (found version "3.1") 
-- Found MPI: TRUE (found version "3.1")
-- Found OpenMP_C: -qopenmp (found version "5.0") 
-- Found OpenMP_CXX: -qopenmp (found version "5.0") 
-- Found OpenMP: TRUE (found version "5.0")
-- Found CUDAToolkit: /opt/apps/cuda/12.0/include (found suitable version "12.0.140", minimum required is "10.0")
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Found OpenMP_C: -qopenmp (found version "5.0") 
-- Found OpenMP_CXX: -qopenmp (found version "5.0")
This is a MPI build:TRUE
-- Checking if arch 80 is supported...
-- Found libcudacxx: /work/08435/srvenkat/frontera/AMGX/thrust/dependencies/libcudacxx/lib/cmake/libcudacxx/libcudacxx-config.cmake (found suitable version "1.8.1.0", minimum required is "1.8.0")
-- Found Thrust: /work/08435/srvenkat/frontera/AMGX/thrust/thrust/cmake/thrust-config.cmake (found version "2.1.0.0")
-- Found CUB: /work/08435/srvenkat/frontera/AMGX/thrust/dependencies/cub/cub/cmake/cub-config.cmake (found suitable version "2.1.0.0", minimum required is "2.1.0.0")
-- Configuring done
CMake Warning at CMakeLists.txt:280 (add_library):
  Cannot generate a safe runtime search path for target amgxsh because files
  in some directories may conflict with libraries in implicit directories:

    runtime library [libcusolver.so.11] in /opt/apps/cuda/12.0/lib64 may be hidden by files in:
      /opt/apps/cuda/11.4/targets/x86_64-linux/lib
    runtime library [libnvToolsExt.so.1] in /opt/apps/cuda/12.0/lib64 may be hidden by files in:
      /opt/apps/cuda/11.4/targets/x86_64-linux/lib

  Some of these libraries may not be found correctly.


CMake Warning at src/CMakeLists.txt:89 (add_executable):
  Cannot generate a safe runtime search path for target amgx_tests_launcher
  because files in some directories may conflict with libraries in implicit
  directories:

    runtime library [libcusolver.so.11] in /opt/apps/cuda/12.0/lib64 may be hidden by files in:
      /opt/apps/cuda/11.4/targets/x86_64-linux/lib
    runtime library [libnvToolsExt.so.1] in /opt/apps/cuda/12.0/lib64 may be hidden by files in:
      /opt/apps/cuda/11.4/targets/x86_64-linux/lib

  Some of these libraries may not be found correctly.


CMake Warning at examples/CMakeLists.txt:55 (add_executable):
  Cannot generate a safe runtime search path for target generate_poisson
  because files in some directories may conflict with libraries in implicit
  directories:

    runtime library [libcusolver.so.11] in /opt/apps/cuda/12.0/lib64 may be hidden by files in:
      /opt/apps/cuda/11.4/targets/x86_64-linux/lib
    runtime library [libnvToolsExt.so.1] in /opt/apps/cuda/12.0/lib64 may be hidden by files in:
      /opt/apps/cuda/11.4/targets/x86_64-linux/lib

  Some of these libraries may not be found correctly.


CMake Warning at examples/CMakeLists.txt:63 (add_executable):
  Cannot generate a safe runtime search path for target
  generate_poisson7_dist_renum because files in some directories may conflict
  with libraries in implicit directories:

    runtime library [libcusolver.so.11] in /opt/apps/cuda/12.0/lib64 may be hidden by files in:
      /opt/apps/cuda/11.4/targets/x86_64-linux/lib
    runtime library [libnvToolsExt.so.1] in /opt/apps/cuda/12.0/lib64 may be hidden by files in:
      /opt/apps/cuda/11.4/targets/x86_64-linux/lib

  Some of these libraries may not be found correctly.


-- Generating done
-- Build files have been written to: /work/08435/srvenkat/frontera/AMGX/build2

If I run make VERBOSE=true all after this, I still get the same errors.