diff --git a/Build/erf_containerfile b/Build/erf_containerfile new file mode 100644 index 000000000..2f6d5de91 --- /dev/null +++ b/Build/erf_containerfile @@ -0,0 +1,55 @@ +# ERF (https://github.com/erf-model/ERF) containerfile for NERSC Perlmutter +# Paul Lin, LBNL/NERSC +# May 2024 + +FROM nvcr.io/nvidia/cuda:12.2.0-devel-ubuntu22.04 + +WORKDIR /app +ARG base_dir=/app/erf + +RUN apt-get update -y && \ + DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \ + g++-12 \ + gcc-12 \ + gfortran-12 \ + git \ + libtool \ + make \ + tar \ + autoconf \ + automake \ + wget \ + python3 \ + cmake + +# MPICH to be swapped out later for Cray MPI +ARG mpich_version=4.2.2 +ARG mpich_dir=mpich-${mpich_version} + +RUN wget https://www.mpich.org/static/downloads/$mpich_version/$mpich_dir.tar.gz && \ + tar xzf $mpich_dir.tar.gz && \ + cd $mpich_dir && \ + ./configure CC=/usr/bin/gcc-12 CXX=/usr/bin/g++-12 F77=/usr/bin/gfortran-12 FC=/usr/bin/gfortran-12 && \ + make -j8 && \ + make install && \ + make clean && \ + cd .. && \ + rm -rf $mpich_dir $mpich_dir.tar.gz + +RUN mkdir ${base_dir} + +ARG build_dir=MyBuild + +RUN cd ${base_dir} && git clone --recursive https://github.com/erf-model/ERF.git && \ + cd ERF && mkdir ${build_dir} && cd ${build_dir} && \ + cmake \ + -DCMAKE_C_COMPILER=mpicc \ + -DCMAKE_CXX_COMPILER=mpicxx \ + -DCMAKE_Fortran_COMPILER=mpif90 \ + -DCMAKE_BUILD_TYPE:STRING=Release \ + -DCMAKE_CUDA_ARCHITECTURES=80 \ + -DERF_ENABLE_MPI:BOOL=ON \ + -DERF_ENABLE_CUDA=ON \ + .. && \ + make -j8 + diff --git a/Docs/sphinx_doc/containers.rst b/Docs/sphinx_doc/containers.rst new file mode 100644 index 000000000..6fa008697 --- /dev/null +++ b/Docs/sphinx_doc/containers.rst @@ -0,0 +1,174 @@ + .. role:: cpp(code) + :language: c++ + +.. _containers: + +ERF containers on Perlmutter +============================ + +The following is a brief introduction to ERF and containers on the NERSC Perlmutter platform. + +For more details, please see `NERSC's detailed containers documentation `_, which also includes containers tutorials. + +Container images can be built on one's desktop/laptop using a standard container framework such as Docker, Podman (Pod Manager), etc. or directly on a Perlmutter login node using ``podman-hpc``. ``podman-hpc`` is a NERSC-developed wrapper that extends the capabilites of Podman for HPC. Containers are run on Perlmutter using either ``podman-hpc`` or ``shifter`` (also developed at NERSC). NERSC has a good `podman-hpc tutorial `_. + +Example ERF containerfile +~~~~~~~~~~~~~~~~~~~~~~~~~ + +The following is an example ERF containerfile with filename ``erf_containerfile``: + +.. code:: shell + + 1 FROM nvcr.io/nvidia/cuda:12.2.0-devel-ubuntu22.04 + 2 + 3 WORKDIR /app + 4 ARG base_dir=/app/erf + 5 + 6 RUN apt-get update -y && \ + 7 DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \ + 8 g++-12 \ + 9 gcc-12 \ + 10 gfortran-12 \ + 11 git \ + 12 libtool \ + 13 make \ + 14 tar \ + 15 autoconf \ + 16 automake \ + 17 wget \ + 18 python3 \ + 19 cmake + 20 + 21 # MPICH to be swapped out later for Cray MPI + 22 ARG mpich_version=4.2.2 + 23 ARG mpich_dir=mpich-${mpich_version} + 24 + 25 RUN wget https://www.mpich.org/static/downloads/$mpich_version/$mpich_dir.tar.gz && \ + 26 tar xzf $mpich_dir.tar.gz && \ + 27 cd $mpich_dir && \ + 28 ./configure CC=/usr/bin/gcc-12 CXX=/usr/bin/g++-12 F77=/usr/bin/gfortran-12 FC=/usr/bin/gfortran-12 && \ + 29 make -j8 && \ + 30 make install && \ + 31 make clean && \ + 32 cd .. && \ + 33 rm -rf $mpich_dir $mpich_dir.tar.gz + 34 + 35 RUN mkdir ${base_dir} + 36 + 37 ARG build_dir=MyBuild + 38 + 39 RUN cd ${base_dir} && git clone --recursive https://github.com/erf-model/ERF.git && \ + 40 cd ERF && mkdir ${build_dir} && cd ${build_dir} && \ + 41 cmake \ + 42 -DCMAKE_C_COMPILER=mpicc \ + 43 -DCMAKE_CXX_COMPILER=mpicxx \ + 44 -DCMAKE_Fortran_COMPILER=mpif90 \ + 45 -DCMAKE_BUILD_TYPE:STRING=Release \ + 46 -DCMAKE_CUDA_ARCHITECTURES=80 \ + 47 -DERF_ENABLE_MPI:BOOL=ON \ + 48 -DERF_ENABLE_CUDA=ON \ + 49 .. && \ + 50 make -j8 + +Line numbers were added for instructional purposes but should not appear in containerfile. +The containerfile is available at https://github.com/erf-model/ERF/tree/development/Build/erf_containerfile + +* Line 1 downloads a container base image from NVIDIA's container registry that contains the Ubuntu 22.04 operating system and CUDA 12.2.0 +* Line 3 sets the working directory +* Line 4 sets a value for the variable ``base_dir`` +* Lines 6-19 download the GNU 12 compilers and all the necessary utilities for building ERF in the container +* Lines 21-33 download the MPICH source code and builds it. At runtime this MPICH will get replaced by Perlmutter's Cray MPI +* Lines 39-50 clone the ERF repo and build it with cmake/make + + + +Build the container on Perlmutter using ``podman-hpc`` and using the containerfile ``erf_containerfile`` with name ``erf`` and tag ``1.00`` (``-t :``) + + .. code:: shell + + podman-hpc build -t erf:1.00 -f erf_containerfile + +In order to use this image in a job or access it from any other login node, one needs to migrate the image onto the $SCRATCH filesystem by issuing the following command: + +.. code:: shell + + podman-hpc migrate erf:1.00 + +``podman-hpc images`` will display the following + +.. code:: shell + + user@perlmutter:login12:~> podman-hpc images + REPOSITORY TAG IMAGE ID CREATED SIZE R/O + localhost/erf 1.00 893605c3ee9b 5 hours ago 16.1 GB true + localhost/erf 1.00 893605c3ee9b 5 hours ago 16.1 GB false + +Note that ``localhost`` will not be needed for podman-hpc commands. + +Run container on Perlmutter +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Submit the following slurm batch script in order to use the image in a job + +.. code:: shell + + #!/bin/bash + + #SBATCH --account= + #SBATCH --constraint=gpu + #SBATCH --job-name=erf + #SBATCH --nodes=1 + #SBATCH --time=0:05:00 + #SBATCH -q regular + + srun -N 1 -n 4 -c 32 --ntasks-per-node=4 --gpus-per-node=4 ./device_wrapper \ + podman-hpc run --rm --mpi --gpu -v /pscratch/sd/u/user/erf/abl:/run -w /run erf:1.00 \ + /app/erf/ERF/MyBuild/Exec/ABL/erf_abl inputs_smagorinsky amrex.use_gpu_aware_mpi=0 + +``device_wrapper`` script: + +.. code:: shell + + #!/bin/bash + # select_cpu_device wrapper script + export CUDA_VISIBLE_DEVICES=$((3-$SLURM_LOCALID)) + exec $* + +Arguments for ``podman-hpc run`` used above + +* ``--rm`` removes the container after exit +* ``--mpi`` enables Cray MPI support (swaps MPICH in the container for Perlmutter's Cray MPI) +* ``--gpu`` enables NVIDIA GPU support +* ``-v /pscratch/sd/u/user/erf/abl:/run`` mounts ``/pscratch/sd/u/user/erf/abl`` on Perlmutter onto ``/run`` in the container +* ``-w /run`` makes the ``/run`` directory inside the container the working directory, i.e. any output from the ERF run will be written to the ``/run`` directory in the container, which will appear in the ``/pscratch/sd/u/user/erf/abl`` directory on Perlmutter. +* ``erf:1.00`` container name and tag +* ``/app/erf/ERF/MyBuild/Exec/ABL/erf_abl`` ERF binary in container + +The remaining arguments are the normal ERF command line arguments. + +Please issue ``podman-hpc --help`` for the help page and ``podman-hpc run --help`` for the ``podman-hpc run`` help page. + +Container image libraries provide a convenient way to store and share images. +The best known one is probably Docker Hub. NERSC provides a private registry to its users via `registry.nersc.gov `_. + +``shifter`` is a NERSC-developed tool that provides an alternative method for running containers on Perlmutter. `NERSC's containers documentation `_ provides an introduction to shifter including a tutorial. + +Common Issues +~~~~~~~~~~~~~ + +* Using ``podman`` rather than ``podman-hpc`` on Perlmutter (best to always use ``podman-hpc``) +* Before issuing ``podman-hpc migrate :`` after having issued the command earlier with identical ``:``, if want to keep the same name, please change the ```` to one that has not been used previously. If want to use an identical ``:`` used in a previous ``podman-hpc migrate`` command, please first issue ``podman-hpc rmsqi :`` to delete the old image. Otherwise could potentially end up with errors such as + + .. code:: shell + + Error: read-only image store assigns the same name to multiple images + + and will have resort to `various methods `_ to get out of the bad configuration state. + +* The default containerfile is a file called ``Containerfile`` (case sensitive). When that file is being used, can replace ``-f erf_containerfile`` with a period: + + .. code:: shell + + podman-hpc build -t erf:1.00 . + + Note that for this case, the period is mandatory. Here it does not denote the end of a sentence. diff --git a/Docs/sphinx_doc/index.rst b/Docs/sphinx_doc/index.rst index d99b4ba94..a7cf0034f 100644 --- a/Docs/sphinx_doc/index.rst +++ b/Docs/sphinx_doc/index.rst @@ -99,3 +99,9 @@ In addition to this documentation, there is API documentation for ERF generated :maxdepth: 1 Applications_Requirements.rst + +.. toctree:: + :caption: CONTAINERS + :maxdepth: 1 + + containers.rst