Spack enables users to install software in a very user-friendly way,
-for example by allowing different versions or specifications
-of the same package to be installed simultaneously, or installing
-without having to “manually” download the source code. This comes at
-the expense of potentially losing control over the exact
-version/specification being installed. For this reason, C2SM has
-the following guidelines for building, running and installing your
-libraries and executables.
There are two possible ways of building software with Spack:
-spackinstall and spackdev-build.
-Both are good, but have some special features that need to be taken into account.
Every spackinstall command needs a version suffix,
-i.e. spackinstall<package>@<version-suffix>
-This version-suffix can have different meanings:
-
-
branch in the git repository
-
one out of several git repositories defined in the package
-
a specifc version (git-tag) with a corresponding git-hash hardcoded in the package
-
-
Only for the last item in the list above you will always fetch and
-compile the same code. The two other items can lead to different
-codes in case the HEAD of this specific branch/repository has received some
-additional commits in the meantime.
-
Especially for production it is very important to know which version of a code is actually used.
-
For example, there is a variety of different version suffixes for the cosmo package:
In order to install software with spackdev-build, one needs a
-local source code. Spack will then compile the code as it is locally
-present. Contrary to spackinstall, the version suffix
-(@master, @v2.7.9, etc.) does not have any effect on the code version compiled,
-since Spack will always use the local source.
-
Nonetheless, it is important to use the correct version suffix, i.e. c2sm-master
-for all your COSMO versions that build on top of the c2sm-master branch.
-The same applies to c2sm-features etc.
-The reason for this is that there are some patches Spack applies based on the version suffix.
-Using the wrong version suffix may break your code.
-
-
Attention
-
You may run into a conflict if you try to install the same spec using dev-build.
-Spack will tell you that this particular spec is already installed.
-
-
-
Tip
-
To circumvent the above conflict, you can keep your executable alive in your source folder,
-but omit the installation phase using spackdev-build--untilbuild<your_spec>.
Spack provides the command spackload to load an installation into your environment.
-This could either be a Python installation or required variables that
-a specific binary needs to run correctly. There are two different ways of using it
-(both of them are fine), which are presented here.
-For more information consider reading the
-official Spack docs.
The Spack commands are rather tailored for interacive use. For example,
-it is very possible for commands such as spackfind or spack
-location to complain about multiple potential installed SPECS satisfying
-the command line input. For this reason, it is advisable to
-avoid spack commands in scripts. However, for spackfind and
-spacklocation, this should not be aproblem. For spackload, we rather
-recommend to use it from the login nodes before submitting jobs, inheriting
-the environment of the running job from the environment at submission time.
Spack offers several options for package development.
-Depending on your workflow, one or the other option is preferred.
-Also some packages like ICON or COSMO have their own
-development workflow which is maintained by C2SM.
We assume that developers of a package are familiar with its build system.
-Therefore, we reccomend to use Spack to set up the environment for the package.
-Building and testing should be done with the package’s build and test system.
-
# LoadSpack!
-$ spackdev-build--beforebuild<package>@develop<variant># stops dev-build before executing the phase 'build'
-$ spackbuild-env<package>@develop<variant>--bash# nests a bash shell with the build env vars loaded
-# Workonthepackage!
-# Usethebuildsystemofthepackage!(e.g.'make')
-# Usethetestinginfrastructureofthepackage!
-$ exit# to exit the nested bash
-
-
-
If you want multiple dev-builds at the same time, label them with separate @<your-label>.
-The identifier @develop is common in the Spack documentation but you can use any string.
Environments sit in a folder with a name and are defined in a spack.yaml file.
-For more information about environments in general, consider reading the
-official Spack docs.
# This is a Spack Environment file.
-#
-# It describes a set of packages to be installed, along with
-# configuration settings.
-spack:
-# add package specs to the `specs` list
-specs:
--icon@develop%nvhpc +ecrad +rte-rrtmgp +cuda
--eccodes@2.19.0%nvhpc
--nvidia-blas%nvhpc
--nvidia-lapack%nvhpc
--libxml2@2.9.13%gcc
-view:true
-concretizer:
-unify:true
-develop:
-icon:
-spec:icon@develop%nvhpc +ecrad +rte-rrtmgp +cuda
-path:../../../../
-
-
-
-
The key part of the environments is the develop keyword.
-This tells Spack to look for a certain spec in path.
-It is possible to specify multiple packages under develop.
Users are responsible for their own instance, rather than relying on an
-instance installed by C2SM. The main advantage is that users do not make
-changes to their Spack instance unless they actively choose to pull upstream
-changes from GitHub. Therefore, your workflow is not interrupted, for example,
-during a production phase with software installed from Spack.
-
-
Attention
-
With more power comes more responsability!
-
-
This page collects best practices for a safe and reliable Spack instance management.
Generally, we recommend to host a released version of spack-c2sm and upgrade to newer versions when you see fit.
-But if you fancy the latest developments before they are released, you may follow the branch main.
-For automated systems that regularly set up a new Spack instance, using a fixed version and updating on demand, is more robust and limits the dependencies of your project. Following main on the other hand automatically updates to the newest features but breaks upon API changing commits in spack-c2sm.
The arguments --depth1 and --shallow-submodules are optional, but they reduce the amount of downloaded data.
-It is recommended to clone spack-c2sm in a location that does not undergo a regular cleanup.
setup-env.sh<upstream> loads spack with the provided upstream to look for preinstalled software and configurations.
-setup-env.sh loads spack without an upstream.
Before updating, we recommend to clean the instance, as this avoids occasional, hard to debug problems. For more infos, see below.
-To update a Spack instance, pull the latest version from the repository and update the submodules:
-
$ gitpull
-$ gitsubmoduleupdate--recursive
-
-
-
This is required after upgrades at CSCS or if you need new features of a package.
To clean a Spack instance, empty the caches and uninstall everything:
-
$ spackclean-a#[cleans all misc caches](https://spack.readthedocs.io/en/v0.21.1/command_index.html?highlight=spack%20load#spack-clean)
-$ spackuninstall-a#[uninstalls all packages](https://spack.readthedocs.io/en/v0.21.1/command_index.html?highlight=spack%20load#spack-uninstall)
-
This auto-detects your machine and configures your instance for it.
-You can force a machine with an argument. The name has to match a folder in sysconfigs.
To see what spackinstall would install, ask for a spec
-
$ spackspec<variant>
-# e.g.
-$ spackspecicon@master+ocean
-
-
-
An unspecfied variant (e.g. ocean) can be concretized to ANY of its values. Spack isn’t required to use the default value when a variant is unspecified. The default value only serves as a tiebreaker.
-
To install a package
-
$ spackinstall<variant>
-# e.g.
-$ spackinstallicon@master%gcc+ocean
-
-
-
To locate your install, query Spack
-
$ spacklocation--install-dir<variant>
-
-
-
This prints a list of all installs that satisfy the restrictions in your variant.
-
To run it, you may need to load environment variables
ICON is built using environments.
-Environments are located in a folder named after the environment and are defined in a spack.yaml file.
-For ICON, they are located in config/cscs/spack/<version>/<machine>_<target>_<compiler>.
-They work with a special Spack tag, that is provided in the ICON repository at config/cscs/SPACK_TAG_*.
-So make sure you clone Spack with the specified tag.
-
-
Tip
-
On Balfrin:
-In case your Spack environment requires Python, a compatability issue
-with openssl and git appears.
Out-of-source build for AutotoolsPackages is not supported by Spack.
-The implementation for ICON relies on some hacks inside package.py and
-only works if the build-folder is located inside the Git repo of ICON.
These users generally have little knowledge about programming or packaging.
-They want spack-c2sm to compile their code in a oneliner. No modifications
-in recipes or the model/libraries they use.
The CI running at CSCS is using spack-c2sm to build ICON.
-The main focus here is on reliability and reproducability. We do not want any
-inconsistencies to be introduced by spack-c2sm. Therefore a strict versioning
-policy is in place there. Only a specific version of spack-c2sm is guaranteed to work in BuildBot.
-Also all recipes used for ICON builds need to have
-predefined versions. Defining a spack version of a package simply as a branch is a no-go.
Software experts belong to this user-group. They are able to add new recipes or edit existing ones.
-Experiment with new setups and ways to build software.
Staff from C2SM core-team and MeteoSwiss maintain spack-c2sm. They install upstream instances providing pre-built
-libraries or manage integration of spack-c2sm in ICON etc. Also upgrade spack to newer versions
-or extend it to new system like Vial or Balfrin.
-Main point of contact for all other users!
So the question came up why Spack was chosen as the package manager for MeteoSwiss
-and I wanted to try to answer this here so that others can see this as well
-(my account may be incomplete, but should hopefully give a decent view on the whole topic):
-
The main issue is that MCH has some hefty requirements when it comes to building software.
-
To just name a few:
-
-
We need to be able to build multiple versions and configurations of the same software
-(most linux distributions will provide one officially supported version and for others
-you’re mostly on your own)
-
Hardware matters for our software due to performance reasons. E.g., the MPI implementation
-or specific GPU models are important (this is a problem for package managers that try
-to create very isolated build environments)
-
Build Systems of some of our software can be quite tricky to deal with (as part of
-its build process COSMO requires one to edit the Makefile which we usually automate
-with something like regex I believe)
-
We often have to build a lot of HPC dependencies/libraries that aren’t part of a
-usual system’s installation (which means we don’t really want to just use some bash
-scripts, we tried that and it became very difficult to maintain)
-
We want to also create development builds and possibly hack the build of a package
-(clean building of packages isn’t enough, we also need hackability)
-
We have multiple relevant machines all with different hardware, MPI implementations
-and Accelerators/GPUs (MCH alone needs tsa & daint, but we used to also have kesch
-and our collaborators have a whole lot of other machines as well)
-
-
These requirements are very common in HPC and they rule out a lot of popular solutions
-(e.g., apt-get, pacman, rpm, etc). There are still quite a few package managers left
-that could fulfill all the requirements.But of all these, Spack has by far the
-widest adoption within HPC (just google “HPC package manager” and the first few pages
-are dominated by Spack).
-
Spack is adopted by:
-
-
The US Department of Energy (which includes Aurora (1 exaFLOPS), Frontier
-(1.5 exaFLOPS) and El Capitan (2 exaFLOPS))
This strong dominance sets it apart from all other viable options. It only makes
-sense for MCH to follow the rest of the industry.
-
A comparison of various tools (including Spack) can be found
-here
-and
-here.
-
I believe the timeline of Spack at MCH was roughly:
-
-
Sometime 2018 Valentin and Hannes gave a presentation about Spack at MCH
-
Summer 2019 an evaluation of possible solutions was done and it was determined
-that Spack was the best option (probably same or similar reasons as above)
-
November 2019 Elsa started her build & devops internship and until November 2020
-most software builds at MCH were ported to Spack
-
September/October/November 2020 was when most of the migration happened
-
-
Of course we regularly have issues with Spack and these issues are time consuming
-and frustrating. However, building, package management & dependencies is a very very
-hard problem. While we regularly encounter bugs in Spack and Spack is still far away
-from an optimal solution, there are also unavoidable issues when it comes to building
-HPC software. You can’t expect Spack to solve unsolvable issues. Spack is only a tool.
-If you misuse your tool, it’s not fair to blame the tool for issues. Because Spack
-is trying to solve a complex problem, it also contains quite a bit of complexity
-itself. If you’re not willing to learn Spack and ensure you’re using it right,
-you will have a bad time. But you will also have a bad time with any other solution
-in this case. The idea that there is a magical tool that will automatically make
-all our build requirements work is unfortunately not realistic at this point.
-There are good reasons why there are so many package managers and users still
-regularly encounter issues with them.
If you want just any installation folder that matches the spec,
-the output can be truncated with |head-n1 to get the first.
-If you want the installation folder of the spec that matches your spec,
-filled with the current defaults, you have to use Python.
This will clone the package, build it and install the chosen package
-plus all its dependencies under /scratch/$USER/spack-install/<your_machine>
-(see config.yaml in the maching specific config file section for details).
-The build-stage of your package and its dependencies are not kept
-(add --keep-stage after the install command in order to keep it).
-Module files are also created during this process and installed under
-/scratch/$USER/modules/.
-
However, being able to compile any other package might require
-installing your Spack instance if that package is installed by a Jenkins plan.
-An attempt to build your working copy with the command
-
$ spackinstall<package>@master...
-
-
-
will not perform any compilation if Spack identifies that the requested version
-of the software was already installed by a Jenkins plan.
-
That problem is circumvented for COSMO, C++ dycore and other C2SM-hosted software
-by reserving a specific version (dev-build) of the spack recipe of the package
-(see int2lm package),
-which will not be used by Jenkins. Therefore, spackinstallint2lm@dev-build
-will find that version among the installed ones in the default Spack instance.
-For any other package that does not contain this dev-build version,
-you need to install our own spack instance.
If you do not want Spack to clone the source of the package you want to install,
-especially if you are developing, you can use a local source in
-order to install your package. In order to do so, first go to the base directory
-of the package and then use spackdev-build instead of spackinstall.
-
However being able to compile any other package might require installing your spack instance,
-f that package is installed by a Jenkins plan.
-
Notice that once installed, the package will not be rebuilt at the next attempt
-to spackdev-build, even if the sources of the local directory have changed.
-In order to force spack to build the local developments anytime,
-you need to avoid the installation phase (see option --until below).
Locate paths related to some spec. This command is mainly useful to
-get the path where a package was installed (a long path with hashes)
-and access the coresponding binary (somewhere under that location).
-
As stated in the official spack documentation,
-“The simplest way to run a Spack binary is to find it and run it” as
-it is build with RPATH. In most cases there is no need to adjust the
-environment.
-
Other options can be used to retrieve other paths like the build
-directory or the path to the package definition (see official spack
-documentation
-or spacklocation-h)
Run a command in a specs install environment, or dump its environment to screen or file
-This command can either be used to run a command in a specs install environment or to dump
-a sourceable file with the install environment. In case you want to run tests of packages manually this
-is what you need.