Added ERF containerfile and docs

Build/erf_containerfile

@@ -0,0 +1,55 @@
+# ERF (https://github.com/erf-model/ERF) containerfile for NERSC Perlmutter
+# Paul Lin, LBNL/NERSC
+# May 2024
+
+FROM nvcr.io/nvidia/cuda:12.2.0-devel-ubuntu22.04
+
+WORKDIR /app
+ARG base_dir=/app/erf
+
+RUN apt-get update -y && \
+    DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
+    g++-12 \
+    gcc-12 \
+    gfortran-12 \
+    git \
+    libtool \
+    make \
+    tar \
+    autoconf \
+    automake \
+    wget \
+    python3 \
+    cmake
+
+# MPICH to be swapped out later for Cray MPI
+ARG mpich_version=4.2.2
+ARG mpich_dir=mpich-${mpich_version}
+
+RUN wget https://www.mpich.org/static/downloads/$mpich_version/$mpich_dir.tar.gz && \
+    tar xzf $mpich_dir.tar.gz && \
+    cd $mpich_dir && \
+    ./configure CC=/usr/bin/gcc-12 CXX=/usr/bin/g++-12 F77=/usr/bin/gfortran-12 FC=/usr/bin/gfortran-12 && \
+    make -j8 && \
+    make install && \
+    make clean  && \
+    cd ..  && \
+    rm -rf $mpich_dir $mpich_dir.tar.gz
+
+RUN mkdir ${base_dir}
+
+ARG build_dir=MyBuild
+
+RUN cd ${base_dir} && git clone --recursive https://github.com/erf-model/ERF.git && \
+  cd ERF && mkdir ${build_dir} && cd ${build_dir} && \
+  cmake \
+  -DCMAKE_C_COMPILER=mpicc \
+  -DCMAKE_CXX_COMPILER=mpicxx \
+  -DCMAKE_Fortran_COMPILER=mpif90 \
+  -DCMAKE_BUILD_TYPE:STRING=Release \
+  -DCMAKE_CUDA_ARCHITECTURES=80 \
+  -DERF_ENABLE_MPI:BOOL=ON \
+  -DERF_ENABLE_CUDA=ON \
+  .. && \
+  make -j8
+

Docs/sphinx_doc/containers.rst

@@ -0,0 +1,174 @@
+ .. role:: cpp(code)
+    :language: c++
+
+.. _containers:
+
+ERF containers on Perlmutter
+============================
+
+The following is a brief introduction to ERF and containers on the NERSC Perlmutter platform.
+
+For more details, please see `NERSC's detailed containers documentation <https://docs.nersc.gov/development/containers>`_, which also includes containers tutorials.
+
+Container images can be built on one's desktop/laptop using a standard container framework such as Docker, Podman (Pod Manager), etc. or directly on a Perlmutter login node using ``podman-hpc``.  ``podman-hpc`` is a NERSC-developed wrapper that extends the capabilites of Podman for HPC.  Containers are run on Perlmutter using either ``podman-hpc`` or ``shifter`` (also developed at NERSC).  NERSC has a good `podman-hpc tutorial <https://docs.nersc.gov/development/containers/podman-hpc/podman-beginner-tutorial>`_.
+
+Example ERF containerfile
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The following is an example ERF containerfile with filename ``erf_containerfile``:
+
+.. code:: shell
+
+     1    FROM nvcr.io/nvidia/cuda:12.2.0-devel-ubuntu22.04
+     2
+     3    WORKDIR /app
+     4    ARG base_dir=/app/erf
+     5
+     6    RUN apt-get update -y && \
+     7        DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
+     8        g++-12 \
+     9        gcc-12 \
+    10        gfortran-12 \
+    11        git \
+    12        libtool \
+    13        make \
+    14        tar \
+    15        autoconf \
+    16        automake \
+    17        wget \
+    18        python3 \
+    19        cmake
+    20
+    21    # MPICH to be swapped out later for Cray MPI
+    22    ARG mpich_version=4.2.2
+    23    ARG mpich_dir=mpich-${mpich_version}
+    24
+    25    RUN wget https://www.mpich.org/static/downloads/$mpich_version/$mpich_dir.tar.gz && \
+    26        tar xzf $mpich_dir.tar.gz && \
+    27        cd $mpich_dir && \
+    28        ./configure CC=/usr/bin/gcc-12 CXX=/usr/bin/g++-12 F77=/usr/bin/gfortran-12 FC=/usr/bin/gfortran-12 && \
+    29        make -j8 && \
+    30        make install && \
+    31        make clean  && \
+    32        cd ..  && \
+    33        rm -rf $mpich_dir $mpich_dir.tar.gz
+    34
+    35    RUN mkdir ${base_dir}
+    36
+    37    ARG build_dir=MyBuild
+    38
+    39    RUN cd ${base_dir} && git clone --recursive https://github.com/erf-model/ERF.git && \
+    40      cd ERF && mkdir ${build_dir} && cd ${build_dir} && \
+    41      cmake \
+    42      -DCMAKE_C_COMPILER=mpicc \
+    43      -DCMAKE_CXX_COMPILER=mpicxx \
+    44      -DCMAKE_Fortran_COMPILER=mpif90 \
+    45      -DCMAKE_BUILD_TYPE:STRING=Release \
+    46      -DCMAKE_CUDA_ARCHITECTURES=80 \
+    47      -DERF_ENABLE_MPI:BOOL=ON \
+    48      -DERF_ENABLE_CUDA=ON \
+    49      .. && \
+    50      make -j8
+
+Line numbers were added for instructional purposes but should not appear in containerfile.
+The containerfile is available at https://github.com/erf-model/ERF/tree/development/Build/erf_containerfile
+
+* Line 1 downloads a container base image from NVIDIA's container registry that contains the Ubuntu 22.04 operating system and CUDA 12.2.0
+* Line 3 sets the working directory
+* Line 4 sets a value for the variable ``base_dir``
+* Lines 6-19 download the GNU 12 compilers and all the necessary utilities for building ERF in the container
+* Lines 21-33 download the MPICH source code and builds it.  At runtime this MPICH will get replaced by Perlmutter's Cray MPI
+* Lines 39-50 clone the ERF repo and build it with cmake/make
+
+
+
+Build the container on Perlmutter using ``podman-hpc`` and using the containerfile ``erf_containerfile`` with name ``erf`` and tag ``1.00`` (``-t <name>:<tag>``)
+
+   .. code:: shell
+
+    podman-hpc build -t erf:1.00 -f erf_containerfile
+
+In order to use this image in a job or access it from any other login node, one needs to migrate the image onto the $SCRATCH filesystem by issuing the following command:
+
+.. code:: shell
+
+  podman-hpc migrate erf:1.00
+
+``podman-hpc images`` will display the following
+
+.. code:: shell
+
+  user@perlmutter:login12:~> podman-hpc images
+  REPOSITORY                    TAG                         IMAGE ID      CREATED       SIZE        R/O
+  localhost/erf                 1.00                        893605c3ee9b  5 hours ago   16.1 GB     true
+  localhost/erf                 1.00                        893605c3ee9b  5 hours ago   16.1 GB     false
+
+Note that ``localhost`` will not be needed for podman-hpc commands.
+
+Run container on Perlmutter
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Submit the following slurm batch script in order to use the image in a job
+
+.. code:: shell
+
+  #!/bin/bash
+
+  #SBATCH --account=<proj>
+  #SBATCH --constraint=gpu
+  #SBATCH --job-name=erf
+  #SBATCH --nodes=1
+  #SBATCH --time=0:05:00
+  #SBATCH -q regular
+
+  srun -N 1 -n 4 -c 32 --ntasks-per-node=4 --gpus-per-node=4 ./device_wrapper \
+  podman-hpc run --rm --mpi --gpu -v /pscratch/sd/u/user/erf/abl:/run -w /run erf:1.00 \
+  /app/erf/ERF/MyBuild/Exec/ABL/erf_abl inputs_smagorinsky amrex.use_gpu_aware_mpi=0
+
+``device_wrapper`` script:
+
+.. code:: shell
+
+      #!/bin/bash
+      # select_cpu_device wrapper script
+      export CUDA_VISIBLE_DEVICES=$((3-$SLURM_LOCALID))
+      exec $*
+
+Arguments for ``podman-hpc run`` used above
+
+* ``--rm`` removes the container after exit
+* ``--mpi`` enables Cray MPI support (swaps MPICH in the container for Perlmutter's Cray MPI)
+* ``--gpu`` enables NVIDIA GPU support
+* ``-v /pscratch/sd/u/user/erf/abl:/run`` mounts ``/pscratch/sd/u/user/erf/abl`` on Perlmutter onto ``/run`` in the container
+* ``-w /run`` makes the ``/run`` directory inside the container the working directory, i.e. any output from the ERF run will be written to the ``/run`` directory in the container, which will appear in the ``/pscratch/sd/u/user/erf/abl`` directory on Perlmutter.
+* ``erf:1.00`` container name and tag
+* ``/app/erf/ERF/MyBuild/Exec/ABL/erf_abl`` ERF binary in container
+
+The remaining arguments are the normal ERF command line arguments.
+
+Please issue ``podman-hpc --help`` for the help page and ``podman-hpc run --help`` for the ``podman-hpc run`` help page.
+
+Container image libraries provide a convenient way to store and share images.
+The best known one is probably Docker Hub.  NERSC provides a private registry to its users via `registry.nersc.gov <https://docs.nersc.gov/development/containers/registry>`_.
+
+``shifter`` is a NERSC-developed tool that provides an alternative method for running containers on Perlmutter. `NERSC's containers documentation <https://docs.nersc.gov/development/containers>`_ provides an introduction to shifter including a tutorial.
+
+Common Issues
+~~~~~~~~~~~~~
+
+* Using ``podman`` rather than ``podman-hpc`` on Perlmutter (best to always use ``podman-hpc``)
+* Before issuing ``podman-hpc migrate <name>:<tag>`` after having issued the command earlier with identical ``<name>:<tag>``, if want to keep the same name, please change the ``<tag>`` to one that has not been used previously.  If want to use an identical ``<name>:<tag>`` used in a previous ``podman-hpc migrate`` command, please first issue ``podman-hpc rmsqi <name>:<tag>`` to delete the old image.  Otherwise could potentially end up with errors such as
+
+    .. code:: shell
+
+      Error: read-only image store assigns the same name to multiple images
+
+  and will have resort to `various methods <https://docs.nersc.gov/development/containers/podman-hpc/overview/#troubleshooting>`_ to get out of the bad configuration state.
+
+* The default containerfile is a file called ``Containerfile`` (case sensitive).  When that file is being used, can replace ``-f erf_containerfile`` with a period:
+
+   .. code:: shell
+
+    podman-hpc build -t erf:1.00 .
+
+  Note that for this case, the period is mandatory. Here it does not denote the end of a sentence.

Docs/sphinx_doc/index.rst

@@ -99,3 +99,9 @@ In addition to this documentation, there is API documentation for ERF generated
    :maxdepth: 1
 
    Applications_Requirements.rst
+
+.. toctree::
+   :caption: CONTAINERS
+   :maxdepth: 1
+
+   containers.rst