Skip to content

Commit

Permalink
ASP: targets, compilers and providers soft-preferences are only global (
Browse files Browse the repository at this point in the history 
spack#31261)

Modify the packages.yaml schema so that soft-preferences on targets,
compilers and providers can only be specified under the "all" attribute.
This makes them effectively global preferences.

Version preferences instead can only be specified under a package
specific section.

If a preference attribute is found in a section where it should
not be, it will be ignored and a warning is printed to screen.
  • Loading branch information
@alalazo
alalazo authored Nov 7, 2023
1 parent 4004f27 commit f3537bc
Show file tree
Hide file tree
Showing 11 changed files with 209 additions and 207 deletions.
76 changes: 36 additions & 40 deletions lib/spack/docs/build_settings.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -526,56 +526,52 @@ Package Preferences
In some cases package requirements can be too strong, and package
preferences are the better option. Package preferences do not impose
constraints on packages for particular versions or variants values,
they rather only set defaults -- the concretizer is free to change
them if it must due to other constraints. Also note that package
preferences are of lower priority than reuse of already installed
packages.
they rather only set defaults. The concretizer is free to change
them if it must, due to other constraints, and also prefers reusing
installed packages over building new ones that are a better match for
preferences.

Here's an example ``packages.yaml`` file that sets preferred packages:
Most package preferences (``compilers``, ``target`` and ``providers``)
can only be set globally under the ``all`` section of ``packages.yaml``:

.. code-block:: yaml
packages:
all:
compiler: [[email protected], clang@12:, oneapi@2023:]
target: [x86_64_v3]
providers:
mpi: [mvapich2, mpich, openmpi]
These preferences override Spack's default and effectively reorder priorities
when looking for the best compiler, target or virtual package provider. Each
preference takes an ordered list of spec constraints, with earlier entries in
the list being preferred over later entries.

In the example above all packages prefer to be compiled with ``[email protected]``,
to target the ``x86_64_v3`` microarchitecture and to use ``mvapich2`` if they
depend on ``mpi``.

The ``variants`` and ``version`` preferences can be set under
package specific sections of the ``packages.yaml`` file:

.. code-block:: yaml
packages:
opencv:
compiler: [[email protected]]
variants: +debug
gperftools:
version: [2.2, 2.4, 2.3]
all:
compiler: [[email protected], '[email protected]:', intel, clang, pgi]
target: [sandybridge]
providers:
mpi: [mvapich2, mpich, openmpi]
At a high level, this example is specifying how packages are preferably
concretized. The opencv package should prefer using GCC 4.9 and
be built with debug options. The gperftools package should prefer version
2.2 over 2.4. Every package on the system should prefer mvapich2 for
its MPI and GCC 4.4.7 (except for opencv, which overrides this by preferring GCC 4.9).
These options are used to fill in implicit defaults. Any of them can be overwritten
on the command line if explicitly requested.

Package preferences accept the follow keys or components under
the specific package (or ``all``) section: ``compiler``, ``variants``,
``version``, ``providers``, and ``target``. Each component has an
ordered list of spec ``constraints``, with earlier entries in the
list being preferred over later entries.

Sometimes a package installation may have constraints that forbid
the first concretization rule, in which case Spack will use the first
legal concretization rule. Going back to the example, if a user
requests gperftools 2.3 or later, then Spack will install version 2.4
as the 2.4 version of gperftools is preferred over 2.3.

An explicit concretization rule in the preferred section will always
take preference over unlisted concretizations. In the above example,
xlc isn't listed in the compiler list. Every listed compiler from
gcc to pgi will thus be preferred over the xlc compiler.

The syntax for the ``provider`` section differs slightly from other
concretization rules. A provider lists a value that packages may
``depends_on`` (e.g, MPI) and a list of rules for fulfilling that
dependency.
In this case, the preference for ``opencv`` is to build with debug options, while
``gperftools`` prefers version 2.2 over 2.4.

Any preference can be overwritten on the command line if explicitly requested.

Preferences cannot overcome explicit constraints, as they only set a preferred
ordering among homogeneous attribute values. Going back to the example, if
``[email protected]:`` was requested, then Spack will install version 2.4
since the most preferred version 2.2 is prohibited by the version constraint.

.. _package_permissions:

Expand Down
8 changes: 5 additions & 3 deletions lib/spack/spack/cmd/config.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -407,7 +407,9 @@ def config_prefer_upstream(args):
pkgs = {}
for spec in pref_specs:
# Collect all the upstream compilers and versions for this package.
pkg = pkgs.get(spec.name, {"version": [], "compiler": []})
pkg = pkgs.get(spec.name, {"version": []})
all = pkgs.get("all", {"compiler": []})
pkgs["all"] = all
pkgs[spec.name] = pkg

# We have no existing variant if this is our first added version.
Expand All @@ -418,8 +420,8 @@ def config_prefer_upstream(args):
pkg["version"].append(version)

compiler = str(spec.compiler)
if compiler not in pkg["compiler"]:
pkg["compiler"].append(compiler)
if compiler not in all["compiler"]:
all["compiler"].append(compiler)

# Get and list all the variants that differ from the default.
variants = []
Expand Down
173 changes: 107 additions & 66 deletions lib/spack/spack/schema/packages.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,64 +8,81 @@
:lines: 13-
"""

permissions = {
"type": "object",
"additionalProperties": False,
"properties": {
"read": {"type": "string", "enum": ["user", "group", "world"]},
"write": {"type": "string", "enum": ["user", "group", "world"]},
"group": {"type": "string"},
},
}

variants = {"oneOf": [{"type": "string"}, {"type": "array", "items": {"type": "string"}}]}

requirements = {
"oneOf": [
# 'require' can be a list of requirement_groups.
# each requirement group is a list of one or more
# specs. Either at least one or exactly one spec
# in the group must be satisfied (depending on
# whether you use "any_of" or "one_of",
# repectively)
{
"type": "array",
"items": {
"oneOf": [
{
"type": "object",
"additionalProperties": False,
"properties": {
"one_of": {"type": "array", "items": {"type": "string"}},
"any_of": {"type": "array", "items": {"type": "string"}},
"spec": {"type": "string"},
"message": {"type": "string"},
"when": {"type": "string"},
},
},
{"type": "string"},
]
},
},
# Shorthand for a single requirement group with
# one member
{"type": "string"},
]
}

permissions = {
"type": "object",
"additionalProperties": False,
"properties": {
"read": {"type": "string", "enum": ["user", "group", "world"]},
"write": {"type": "string", "enum": ["user", "group", "world"]},
"group": {"type": "string"},
},
}

package_attributes = {
"type": "object",
"additionalProperties": False,
"patternProperties": {r"\w+": {}},
}

#: Properties for inclusion in other schemas
properties = {
"packages": {
"type": "object",
"default": {},
"additionalProperties": False,
"patternProperties": {
r"\w[\w-]*": { # package name
"properties": {
"all": { # package name
"type": "object",
"default": {},
"additionalProperties": False,
"properties": {
"require": {
"oneOf": [
# 'require' can be a list of requirement_groups.
# each requirement group is a list of one or more
# specs. Either at least one or exactly one spec
# in the group must be satisfied (depending on
# whether you use "any_of" or "one_of",
# repectively)
{
"type": "array",
"items": {
"oneOf": [
{
"type": "object",
"additionalProperties": False,
"properties": {
"one_of": {
"type": "array",
"items": {"type": "string"},
},
"any_of": {
"type": "array",
"items": {"type": "string"},
},
"spec": {"type": "string"},
"message": {"type": "string"},
"when": {"type": "string"},
},
},
{"type": "string"},
]
},
},
# Shorthand for a single requirement group with
# one member
{"type": "string"},
]
},
"version": {
"type": "array",
"default": [],
# version strings (type should be string, number is still possible
# but deprecated. this is to avoid issues with e.g. 3.10 -> 3.1)
"items": {"anyOf": [{"type": "string"}, {"type": "number"}]},
},
"require": requirements,
"version": {}, # Here only to warn users on ignored properties
"target": {
"type": "array",
"default": [],
Expand All @@ -78,22 +95,10 @@
"items": {"type": "string"},
}, # compiler specs
"buildable": {"type": "boolean", "default": True},
"permissions": {
"type": "object",
"additionalProperties": False,
"properties": {
"read": {"type": "string", "enum": ["user", "group", "world"]},
"write": {"type": "string", "enum": ["user", "group", "world"]},
"group": {"type": "string"},
},
},
"permissions": permissions,
# If 'get_full_repo' is promoted to a Package-level
# attribute, it could be useful to set it here
"package_attributes": {
"type": "object",
"additionalProperties": False,
"patternProperties": {r"\w+": {}},
},
"package_attributes": package_attributes,
"providers": {
"type": "object",
"default": {},
Expand All @@ -106,12 +111,40 @@
}
},
},
"variants": {
"oneOf": [
{"type": "string"},
{"type": "array", "items": {"type": "string"}},
]
"variants": variants,
},
"deprecatedProperties": {
"properties": ["version"],
"message": "setting version preferences in the 'all' section of packages.yaml "
"is deprecated and will be removed in v0.22\n\n\tThese preferences "
"will be ignored by Spack. You can set them only in package specific sections "
"of the same file.\n",
"error": False,
},
}
},
"patternProperties": {
r"(?!^all$)(^\w[\w-]*)": { # package name
"type": "object",
"default": {},
"additionalProperties": False,
"properties": {
"require": requirements,
"version": {
"type": "array",
"default": [],
# version strings
"items": {"anyOf": [{"type": "string"}, {"type": "number"}]},
},
"target": {}, # Here only to warn users on ignored properties
"compiler": {}, # Here only to warn users on ignored properties
"buildable": {"type": "boolean", "default": True},
"permissions": permissions,
# If 'get_full_repo' is promoted to a Package-level
# attribute, it could be useful to set it here
"package_attributes": package_attributes,
"providers": {}, # Here only to warn users on ignored properties
"variants": variants,
"externals": {
"type": "array",
"items": {
Expand All @@ -127,6 +160,14 @@
},
},
},
"deprecatedProperties": {
"properties": ["target", "compiler", "providers"],
"message": "setting compiler, target or provider preferences in a package "
"specific section of packages.yaml is deprecated, and will be removed in "
"v0.22.\n\n\tThese preferences will be ignored by Spack. You "
"can set them only in the 'all' section of the same file.\n",
"error": False,
},
}
},
}
Expand Down
Loading

0 comments on commit f3537bc

Please sign in to comment.