Using MIT's Satori cluster

[glwagner]

This top-level post is a WIP.

Here's a description of MIT's Satori cluster: https://mit-satori.github.io/

Example snippet from a slurm script:

#!/bin/bash
#SBATCH --ntasks-per-node=4
#SBATCH --gres=gpu:4
#SBATCH --cpus-per-task=16
#SBATCH --mem=100GB
#SBATCH --time 24:00:00

# Upload modules
module purge all
module add spack
module add cuda/11.4
module load openmpi/3.1.6-cuda-pmi-ucx-slurm-jhklron

# MPI specific exports
export OMPI_MCA_pml=^ucx
export OMPI_MCA_osc=^ucx
export OMPI_MCA_btl_openib_allow_ib=true

# Julia specific enviromental variables
export COMMON="/nobackup/users/ssilvest/perlmutter-test"
export JULIA_DEPOT_PATH=":${COMMON}/depot"
export JULIA_LOAD_PATH="${JULIA_LOAD_PATH}:$(pwd)/satori"
export JULIA_CUDA_MEMORY_POOL=none
export JULIA=julia

# Profile specific variable
export JULIA_NVTX_CALLBACKS=gc

# Number of threads in SLURM mode
export JULIA_NUM_THREADS=${SLURM_CPUS_PER_TASK:=1}

cat > launch.sh << EoF_s
#! /bin/sh
export CUDA_VISIBLE_DEVICES=0,1,2,3
exec \$*
EoF_s
chmod +x launch.sh

if $PROFILE; then
   NSYS="nsys profile --trace=nvtx,cuda,mpi --output=${COMMON}/report_N${SLURM_JOB_NUM_NODES}_R${RESOLUTION}_${PRECISION}"
fi

$NSYS srun --mpi=pmi2 ./launch.sh $JULIA --check-bounds=no --project experiments/run.jl ${RESOLUTION:=3}

[ali-ramadhan]

Is it worth mentioning that Satori is a PowerPC cluster and that

CUDA.jl v5.3 is the last version with support for PowerPC (removed in v5.4)

according to the CUDA.jl README?

I haven't run Julia or Oceananigans.jl on Satori in a while but I remember seeing some related warnings, although my code did run. So maybe it'll all be fine as long as long as Oceananigans.jl is compatible with CUDA.jl v5.3?

[glwagner]

Everything is worth mentioning! That's helpful. I'm not sure what "support" means either. I report back if I find more info either way.

[masonrogers14]

I stumbled into this thread getting started on Satori and figured I would share my experience.

Single-GPU runs are working with CUDA.jl v5.5.2 and the Satori CUDA/11.8.0 module; I haven't tested multi-GPU runs. CUDA.jl does have to know how to find the CUDA/11.8.0 module, which for me involved running CUDA.set_runtime_verstion!(v"11.8.0", local_toolkit=true) in the REPL with the CUDA/11.8.0 module loaded. (I did not have the CUDA/11.8.0 module loaded at install time, which my suffice.) I did not have to downgrade anything for powerPC compatibility.

[glwagner]

Nice, thank you @masonrogers14 ! Do you have any code or slurm scripts to share?

[masonrogers14]

Instructions for getting Oceananigans working on a brand-new Satori account:

1. install julia. The juliaup installer didn't work for me, possibly because the server was just down, so I downloaded the v1.11.1 binary for linux on PowerPC and unpacked with $ tar -xzf path/to/julia/tarball
2. add the CUDA module with $ module add cuda/11.8.0
3. (optional) create a project julia environment with $ julia --project
4. in pkg mode, install Oceananigans.jl and CUDA.jl with ] add Oceananigans and ] add CUDA
5. if that does not suffice, you have to tell CUDA.jl about the Satori CUDA module. In the REPL, type

> using CUDA
> CUDA.set_runtime_version(v"11.8.0", local_toolkit=true)

and then restart the REPL. I think local_toolkit=true is telling CUDA.jl to use the loaded Satori CUDA driver.

[masonrogers14]

Batch script for submitting Oceananigans run:

#!/bin/bash
#SBATCH -J inertia
#SBATCH -o stdout
#SBATCH -e stderr
#SBATCH --gres=gpu:1
#SBATCH --nodes=1
#SBATCH --mem=1T
#SBATCH --time=12:00:00
#SBATCH --exclusive
#SBATCH [email protected]
#SBATCH --mail-type=END,FAIL

date
module add cuda/11.8.0
julia --project whatever_your_script_is_called.jl
date