Merge remote-tracking branch 'spack/develop' into fi-nixpack

.github/workflows/audit.yaml

@@ -34,13 +34,15 @@ jobs:
       run: |
           . share/spack/setup-env.sh
           coverage run $(which spack) audit packages
+          coverage run $(which spack) audit externals        
           coverage combine
           coverage xml
     - name: Package audits (without coverage)
       if: ${{ inputs.with_coverage == 'false' }}
       run: |
           . share/spack/setup-env.sh
           $(which spack) audit packages
+          $(which spack) audit externals
     - uses: codecov/codecov-action@eaaf4bedf32dbdc6b720b63067d99c4d77d6047d # @v2.1.0
       if: ${{ inputs.with_coverage == 'true' }}
       with:

lib/spack/docs/.gitignore

@@ -1,4 +1,3 @@
-package_list.html
 command_index.rst
 spack*.rst
 llnl*.rst

lib/spack/docs/basic_usage.rst

@@ -45,7 +45,8 @@ Listing available packages
 
 To install software with Spack, you need to know what software is
 available.  You can see a list of available package names at the
-:ref:`package-list` webpage, or using the ``spack list`` command.
+`packages.spack.io <https://packages.spack.io>`_ website, or
+using the ``spack list`` command.
 
 .. _cmd-spack-list:
 
@@ -60,7 +61,7 @@ can install:
    :ellipsis: 10
 
 There are thousands of them, so we've truncated the output above, but you
-can find a :ref:`full list here <package-list>`.
+can find a `full list here <https://packages.spack.io>`_.
 Packages are listed by name in alphabetical order.
 A pattern to match with no wildcards, ``*`` or ``?``,
 will be treated as though it started and ended with

lib/spack/docs/build_systems/inteloneapipackage.rst

@@ -25,8 +25,8 @@ use Spack to build packages with the tools.
 The Spack Python class ``IntelOneapiPackage`` is a base class that is
 used by ``IntelOneapiCompilers``, ``IntelOneapiMkl``,
 ``IntelOneapiTbb`` and other classes to implement the oneAPI
-packages. See the :ref:`package-list` for the full list of available
-oneAPI packages or use::
+packages. Search for ``oneAPI`` at `<packages.spack.io>`_ for the full
+list of available oneAPI packages, or use::
 
   spack list -d oneAPI
 

lib/spack/docs/conf.py

@@ -48,9 +48,6 @@
 os.environ["COLIFY_SIZE"] = "25x120"
 os.environ["COLUMNS"] = "120"
 
-# Generate full package list if needed
-subprocess.call(["spack", "list", "--format=html", "--update=package_list.html"])
-
 # Generate a command index if an update is needed
 subprocess.call(
     [

lib/spack/docs/contribution_guide.rst

@@ -310,53 +310,11 @@ Once all of the dependencies are installed, you can try building the documentati
    $ make clean
    $ make
 
-If you see any warning or error messages, you will have to correct those before
-your PR is accepted.
-
-If you are editing the documentation, you should obviously be running the
-documentation tests. But even if you are simply adding a new package, your
-changes could cause the documentation tests to fail:
-
-.. code-block:: console
-
-   package_list.rst:8745: WARNING: Block quote ends without a blank line; unexpected unindent.
-
-At first, this error message will mean nothing to you, since you didn't edit
-that file. Until you look at line 8745 of the file in question:
-
-.. code-block:: rst
-
-   Description:
-      NetCDF is a set of software libraries and self-describing, machine-
-     independent data formats that support the creation, access, and sharing
-     of array-oriented scientific data.
-
-Our documentation includes :ref:`a list of all Spack packages <package-list>`.
-If you add a new package, its docstring is added to this page. The problem in
-this case was that the docstring looked like:
-
-.. code-block:: python
-
-   class Netcdf(Package):
-       """
-       NetCDF is a set of software libraries and self-describing,
-       machine-independent data formats that support the creation,
-       access, and sharing of array-oriented scientific data.
-       """
-
-Docstrings cannot start with a newline character, or else Sphinx will complain.
-Instead, they should look like:
-
-.. code-block:: python
-
-   class Netcdf(Package):
-       """NetCDF is a set of software libraries and self-describing,
-       machine-independent data formats that support the creation,
-       access, and sharing of array-oriented scientific data."""
-
-Documentation changes can result in much more obfuscated warning messages.
-If you don't understand what they mean, feel free to ask when you submit
-your PR.
+If you see any warning or error messages, you will have to correct those before your PR
+is accepted. If you are editing the documentation, you should be running the
+documentation tests to make sure there are no errors. Documentation changes can result
+in some obfuscated warning messages. If you don't understand what they mean, feel free
+to ask when you submit your PR.
 
 --------
 Coverage

lib/spack/docs/index.rst

@@ -54,9 +54,16 @@ or refer to the full manual below.
    features
    getting_started
    basic_usage
-   Tutorial: Spack 101 <https://spack-tutorial.readthedocs.io>
    replace_conda_homebrew
 
+.. toctree::
+   :maxdepth: 2
+   :caption: Links
+
+   Tutorial (spack-tutorial.rtfd.io) <https://spack-tutorial.readthedocs.io>
+   Packages (packages.spack.io) <https://packages.spack.io>
+   Binaries (binaries.spack.io) <https://cache.spack.io>
+
 .. toctree::
    :maxdepth: 2
    :caption: Reference
@@ -72,7 +79,6 @@ or refer to the full manual below.
    repositories
    binary_caches
    command_index
-   package_list
    chain
    extensions
    pipelines

lib/spack/docs/packaging_guide.rst

@@ -6196,7 +6196,100 @@ follows:
                "foo-package@{0}".format(version_str)
            )
 
-.. _package-lifecycle:
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Add detection tests to packages
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+To ensure that software is detected correctly for multiple configurations
+and on different systems users can write a ``detection_test.yaml`` file and
+put it in the package directory alongside the ``package.py`` file.
+This YAML file contains enough information for Spack to mock an environment
+and try to check if the detection logic yields the results that are expected.
+
+As a general rule, attributes at the top-level of ``detection_test.yaml``
+represent search mechanisms and they each map to a list of tests that should confirm
+the validity of the package's detection logic.
+
+The detection tests can be run with the following command:
+
+.. code-block:: console
+
+   $ spack audit externals
+
+Errors that have been detected are reported to screen.
+
+""""""""""""""""""""""""""
+Tests for PATH inspections
+""""""""""""""""""""""""""
+
+Detection tests insisting on ``PATH`` inspections are listed under
+the ``paths`` attribute:
+
+.. code-block:: yaml
+
+   paths:
+   - layout:
+     - executables:
+       - "bin/clang-3.9"
+       - "bin/clang++-3.9"
+       script: |
+         echo "clang version 3.9.1-19ubuntu1 (tags/RELEASE_391/rc2)"
+         echo "Target: x86_64-pc-linux-gnu"
+         echo "Thread model: posix"
+         echo "InstalledDir: /usr/bin"
+     results:
+     - spec: '[email protected] +clang~lld~lldb'
+
+Each test is performed by first creating a temporary directory structure as
+specified in the corresponding ``layout`` and by then running
+package detection and checking that the outcome matches the expected
+``results``. The exact details on how to specify both the ``layout`` and the
+``results`` are reported in the table below:
+
+.. list-table:: Test based on PATH inspections
+   :header-rows: 1
+
+   * - Option Name
+     - Description
+     - Allowed Values
+     - Required Field
+   * - ``layout``
+     - Specifies the filesystem tree used for the test
+     - List of objects
+     - Yes
+   * - ``layout:[0]:executables``
+     - Relative paths for the mock executables to be created
+     - List of strings
+     - Yes
+   * - ``layout:[0]:script``
+     - Mock logic for the executable
+     - Any valid shell script
+     - Yes
+   * - ``results``
+     - List of expected results
+     - List of objects (empty if no result is expected)
+     - Yes
+   * - ``results:[0]:spec``
+     - A spec that is expected from detection
+     - Any valid spec
+     - Yes
+
+"""""""""""""""""""""""""""""""
+Reuse tests from other packages
+"""""""""""""""""""""""""""""""
+
+When using a custom repository, it is possible to customize a package that already exists in ``builtin``
+and reuse its external tests. To do so, just write a ``detection_tests.yaml`` alongside the customized
+``package.py`` with an ``includes`` attribute. For instance the ``detection_tests.yaml`` for
+``myrepo.llvm`` might look like:
+
+.. code-block:: yaml
+
+   includes:
+   - "builtin.llvm"
+
+This YAML file instructs Spack to run the detection tests defined in ``builtin.llvm`` in addition to
+those locally defined in the file.
 
 -----------------------------
 Style guidelines for packages

lib/spack/docs/replace_conda_homebrew.rst

@@ -4,7 +4,7 @@
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
 =====================================
-Using Spack to Replace Homebrew/Conda
+Spack for Homebrew/Conda Users
 =====================================
 
 Spack is an incredibly powerful package manager, designed for supercomputers
@@ -191,18 +191,18 @@ The ``--fresh`` flag tells Spack to use the latest version of every package
 where possible instead of trying to optimize for reuse of existing installed
 packages.
 
-The ``--force`` flag in addition tells Spack to overwrite its previous 
-concretization decisions, allowing you to choose a new version of Python. 
-If any of the new packages like Bash are already installed, ``spack install`` 
+The ``--force`` flag in addition tells Spack to overwrite its previous
+concretization decisions, allowing you to choose a new version of Python.
+If any of the new packages like Bash are already installed, ``spack install``
 won't re-install them, it will keep the symlinks in place.
 
 -----------------------------------
 Updating & Cleaning Up Old Packages
 -----------------------------------
 
-If you're looking to mimic the behavior of Homebrew, you may also want to 
-clean up out-of-date packages from your environment after an upgrade. To 
-upgrade your entire software stack within an environment and clean up old 
+If you're looking to mimic the behavior of Homebrew, you may also want to
+clean up out-of-date packages from your environment after an upgrade. To
+upgrade your entire software stack within an environment and clean up old
 package versions, simply run the following commands:
 
 .. code-block:: console
@@ -212,9 +212,9 @@ package versions, simply run the following commands:
    $ spack concretize --fresh --force
    $ spack install
    $ spack gc
-   
-Running ``spack mark -i --all`` tells Spack to mark all of the existing 
-packages within an environment as "implicitly" installed. This tells 
+
+Running ``spack mark -i --all`` tells Spack to mark all of the existing
+packages within an environment as "implicitly" installed. This tells
 spack's garbage collection system that these packages should be cleaned up.
 
 Don't worry however, this will not remove your entire environment.
@@ -223,8 +223,8 @@ a fresh concretization and will re-mark any packages that should remain
 installed as "explicitly" installed.
 
 **Note:** if you use multiple spack environments you should re-run ``spack install``
-in each of your environments prior to running ``spack gc`` to prevent spack 
-from uninstalling any shared packages that are no longer required by the 
+in each of your environments prior to running ``spack gc`` to prevent spack
+from uninstalling any shared packages that are no longer required by the
 environment you just upgraded.
 
 --------------