Error compiling MUMPS and Trilinos on official intel-hpc Docker

[anicusan]

Hello, and thank you so much for maintaining this automatic installer for deal.II - it's quite a life-saviour. I am currently trying to use candi on a simple pre-configured intel-hpckit Docker container with intel compilers and MKL. However, I am struggling to fix the building errors of MUMPS and Trilinos.

Here is a minimal working example:

docker run -it --name icont -v $(pwd):/home/docker-host intel/oneapi-hpckit

First install the basic dependencies that candi.sh mentions, without the openmpi dependencies - which are handled by /opt/intel/oneapi:

cd /home/docker-host

apt update -y
apt-get install -y lsb-release git subversion wget bc libgmp-dev \
    build-essential autoconf automake cmake libtool gfortran \
    libboost-all-dev zlib1g-dev \
    libsuitesparse-dev

git clone -b master https://github.com/dealii/candi.git candi

This is the candi/candi.cfg I use:

# Global configuration.

# Meta-project to build
PROJECT=deal.II-toolchain

# Option {ON|OFF}: Use fresh build directory by remove existing ones?
CLEAN_BUILD=OFF

#########################################################################

# Where do you want the downloaded source files to go?
#DOWNLOAD_PATH=${PREFIX_PATH}/tmp/src

# Where do you want to unpack source files to go?
#UNPACK_PATH=${PREFIX_PATH}/tmp/unpack

# Where do you want the build files to go?
#BUILD_PATH=${PREFIX_PATH}/tmp/build

# Where do you want the compiled software installed?
INSTALL_PATH=${PREFIX_PATH}

#########################################################################
# Set up mirror server url(s), to speed up downloads, e.g.
#   MIRROR="${MIRROR} http://server1.org/package_mirror_dir/"
#   MIRROR="${MIRROR} ftp://server2.org/package_mirror_dir/"
MIRROR="https://tjhei.info/candi-mirror/ https://cecas.clemson.edu/dealii/mirror/"

#########################################################################
# Choose additional configuration and components of deal.II
DEAL_II_CONFOPTS=""

# Option {ON|OFF}: Enable machine-specific optimizations (e.g. -march=native)?
NATIVE_OPTIMIZATIONS=ON

# Option {ON|OFF}: Enable building of dealii examples?
BUILD_EXAMPLES=ON

# Option {ON|OFF}: Unset CXX and set the compiler as MPI_CXX_COMPILER when configuring deal.II
#
# This is recommended for very recent CMake versions but it currently is not reliable enough
# to enable by default.
USE_DEAL_II_CMAKE_MPI_COMPILER=ON

# Option {ON|OFF}: Run tests after installation?
RUN_DEAL_II_TESTS=ON

# Choose the python interpreter to use. We pick python2, python3,
# python in that order by default. If you want to override this
# choice, uncomment the following:
#PYTHON_INTERPRETER=python3

# Now we pick the packages to install:
PACKAGES="load:dealii-prepare"

# These are system dependencies that are often pre-installed:
#PACKAGES="${PACKAGES} once:zlib"
#PACKAGES="${PACKAGES} once:bzip2"
#PACKAGES="${PACKAGES} once:git"
#PACKAGES="${PACKAGES} once:cmake"
#PACKAGES="${PACKAGES} once:boost"
#PACKAGES="${PACKAGES} once:numdiff"
#PACKAGES="${PACKAGES} once:openblas"
#PACKAGES="${PACKAGES} once:scalapack"

# These packages are not dependencies but are tools used for deal.II development:
#PACKAGES="${PACKAGES} once:astyle"
#PACKAGES="${PACKAGES} once:numdiff"

# These packages determine the active components of deal.II:
PACKAGES="${PACKAGES} once:adolc"
PACKAGES="${PACKAGES} once:arpack-ng"
PACKAGES="${PACKAGES} once:assimp"
PACKAGES="${PACKAGES} once:ginkgo"
PACKAGES="${PACKAGES} once:gmsh"
PACKAGES="${PACKAGES} once:gsl"
PACKAGES="${PACKAGES} once:mumps"
PACKAGES="${PACKAGES} once:opencascade"
PACKAGES="${PACKAGES} once:parmetis"
PACKAGES="${PACKAGES} once:sundials"
PACKAGES="${PACKAGES} once:superlu_dist"
PACKAGES="${PACKAGES} once:hdf5"
PACKAGES="${PACKAGES} once:netcdf"
PACKAGES="${PACKAGES} once:p4est"
PACKAGES="${PACKAGES} once:trilinos"
PACKAGES="${PACKAGES} once:petsc"
PACKAGES="${PACKAGES} once:slepc"
PACKAGES="${PACKAGES} once:symengine"
PACKAGES="${PACKAGES} dealii"

#########################################################################

# Install the following deal.II version (choose master, v9.3.0, v9.2.0, ...)
DEAL_II_VERSION=master

#########################################################################

# If you want to use Trilinos, decide if you want v12.x.x (AUTO) or v11.x.x
TRILINOS_MAJOR_VERSION=AUTO
#TRILINOS_MAJOR_VERSION=12
#TRILINOS_MAJOR_VERSION=11

# If enabled, Trilinos is configured with complex number support in
# Teuchos and Tpetra. This takes a long time to compile and requires a
# lot of RAM. It is also likely not something you will need.
TRILINOS_WITH_COMPLEX=OFF

#########################################################################

# Option {ON|OFF}: Do you want to use MKL?
MKL=ON
MKL_DIR=/opt/intel/oneapi/mkl/latest/
BLAS_DIR=/opt/intel/oneapi/mkl/latest/lib/intel64
LAPACK_DIR=/opt/intel/oneapi/mkl/latest/lib/intel64

#########################################################################

# If you have commits from the deal.II master to cherry-pick in stable version

# bugfix for TrilinosWrapper::SparseMatrix ::add(factor, SparseMatrix)
#DEAL_CHERRYPICKCOMMITS="8bcaf55df6754238b2e4e41bf6a5dd276a97bdd2 ${DEAL_CHERRYPICKCOMMITS}"

#########################################################################

# Option {ON|OFF}: If you want to change the source code of one or multiple
# packages switch on the developer mode to avoid a package fetch and unpack.
# Note: a previous run of candi with the same settings must be done without
#       this mode!
DEVELOPER_MODE=OFF

################################################################################

# OPTION {ON|OFF}: Remove build directory after successful installation
INSTANT_CLEAN_BUILD_AFTER_INSTALL=OFF

# OPTION {ON|OFF}: Remove downloaded packed src after successful installation
INSTANT_CLEAN_SRC_AFTER_INSTALL=OFF

# OPTION {ON|OFF}: Remove unpack directory after successful installation
INSTANT_CLEAN_UNPACK_AFTER_INSTALL=OFF

Then, starting the compilation:

export CFLAGS='-O3 -march=native'
export CXXFLAGS='-O3 -march=native'
export FFLAGS='-O3 -march=native'
export FCFLAGS='-O3 -march=native'

INSTALL_DIR=$(pwd)/candi-install
mkdir $INSTALL_DIR

cd candi
./candi.sh --prefix=$INSTALL_DIR -j32
cd ..

Finally, the most important parts of the output:

[...]

Compiler Variables:

CC  variable not set, but default mpicc  found.
CC  = /opt/intel/oneapi/mpi/2021.5.0//bin/mpicc
CXX variable not set, but default mpicxx found.
CXX = /opt/intel/oneapi/mpi/2021.5.0//bin/mpicxx
FC  variable not set, but default mpif90 found.
FC  = /opt/intel/oneapi/mpi/2021.5.0//bin/mpif90
FF  variable not set, but default mpif77 found.
FF  = /opt/intel/oneapi/mpi/2021.5.0//bin/mpif77

[...]

Verifying v5.4.0.5.tar.gz
v5.4.0.5.tar.gz: OK(md5)
Unpacking v5.4.0.5.tar.gz
Building mumps 5.4.0.5
-- The C compiler identification is GNU 7.5.0
-- The Fortran compiler identification is GNU 7.5.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /opt/intel/oneapi/mpi/2021.5.0/bin/mpicc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Check for working Fortran compiler: /opt/intel/oneapi/mpi/2021.5.0/bin/mpif90 - skipped
-- checking that C and Fortran compilers can link
-- checking that C and Fortran compilers can link - OK
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Found PkgConfig: /usr/bin/pkg-config (found version "0.29.1")
-- Checking for module 'mkl-static-lp64-seq'
--   Found mkl-static-lp64-seq, version 2022
-- Performing Test LAPACK_real64_links
-- Performing Test LAPACK_real64_links - Success
-- Performing Test LAPACK_real32_links
-- Performing Test LAPACK_real32_links - Success
-- Found LAPACK: -Wl,--start-group;/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_gf_lp64.so;/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_sequential.so;/opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_core.so;-Wl,--end-group;dl;m  found components: MKL
-- Checking for one of the modules 'ompi-c'
-- Performing Test MPI_C_links
-- Performing Test MPI_C_links - Success
-- Checking for one of the modules 'ompi-fort'
-- Performing Test MPI_Fortran_links
-- Performing Test MPI_Fortran_links - Failed
CMake Error at /usr/local/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find MPI (missing: Fortran)
Call Stack (most recent call first):
  /usr/local/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
  cmake/Modules/FindMPI.cmake:541 (find_package_handle_standard_args)
  CMakeLists.txt:42 (find_package)

Error persists with USE_DEAL_II_CMAKE_MPI_COMPILER=OFF; checking /home/docker-host/candi-install/tmp/build/mumps-5.4.0.5/CMakeFiles/CMakeError.log:

Performing Fortran SOURCE FILE Test MPI_Fortran_links failed with the following output:
Change Dir: /home/docker-host/candi-install/tmp/build/mumps-5.4.0.5/CMakeFiles/CMakeTmp

Run Build Command(s):/usr/bin/make -f Makefile cmTC_e2ef6/fast && /usr/bin/make  -f CMakeFiles/cmTC_e2ef6.dir/build.make CMakeFiles/cmTC_e2ef6.dir/build
make[1]: Entering directory '/home/docker-host/candi-install/tmp/build/mumps-5.4.0.5/CMakeFiles/CMakeTmp'
Building Fortran object CMakeFiles/cmTC_e2ef6.dir/src.F90.o
/opt/intel/oneapi/mpi/2021.5.0/bin/mpif90 -DMPI_Fortran_links -I/opt/intel/oneapi/mpi/2021.5.0/include -O3 -march=native -fimplicit-none   -fPIE -c /home/docker-host/candi-install/tmp/build/mumps-5.4.0.5/CMakeFiles/CMakeTmp/src.F90 -o CMakeFiles/cmTC_e2ef6.dir/src.F90.o
f951: Fatal Error: Reading module ‘mpi’ at line 1 column 2: Unexpected EOF
compilation terminated.
CMakeFiles/cmTC_e2ef6.dir/build.make:77: recipe for target 'CMakeFiles/cmTC_e2ef6.dir/src.F90.o' failed
make[1]: *** [CMakeFiles/cmTC_e2ef6.dir/src.F90.o] Error 1
make[1]: Leaving directory '/home/docker-host/candi-install/tmp/build/mumps-5.4.0.5/CMakeFiles/CMakeTmp'
Makefile:127: recipe for target 'cmTC_e2ef6/fast' failed
make: *** [cmTC_e2ef6/fast] Error 2


Source file was:

program test
use mpi
implicit none
integer :: i
call mpi_init(i)
call mpi_finalize(i)
end program

I do not know why MUMPS does not find MPI, when the other libraries worked fine with it; the Fatal Error: Reading module ‘mpi’ at line 1 column 2: Unexpected EOF hints at something wrong with the cmake configuration (I think?) but I have not managed to fix it by changing MUMPS' CMakeLists.txt.

Any help would be greatly appreciated.

Best wishes,
Leonard