Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Write CEP about virtual packages #103

Open
wants to merge 21 commits into
base: main
Choose a base branch
from
Open

Write CEP about virtual packages #103

Changes from 11 commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
ebcfa8e
Write CEP about virtual packages
jaimergp Dec 17, 2024
cda0a8c
Add some Windows methods
jaimergp Dec 17, 2024
6f2d60f
some more
jaimergp Dec 17, 2024
9a793b9
add details of glibc, linux and osx
jaimergp Dec 17, 2024
9b4b7a7
pre-commit
jaimergp Dec 17, 2024
59462ed
add discussion, implementation
jaimergp Dec 17, 2024
cedde6a
add general paragraph about CONDA_OVERRIDE_*
jaimergp Dec 17, 2024
df9d61d
Apply suggestions from code review
jaimergp Dec 17, 2024
72022f4
more details about archspec
jaimergp Dec 17, 2024
7c3b595
Merge branch 'virtual-packages' of github.com:jaimergp/ceps into virt…
jaimergp Dec 17, 2024
e6fcda5
link to existing issue
jaimergp Dec 18, 2024
40b4442
Apply suggestions from code review
jaimergp Dec 19, 2024
bc6d6a6
Move motivation further up
jaimergp Dec 19, 2024
5bf10f1
Rework `__glibc` specification
jaimergp Dec 19, 2024
7a2c7e5
Forbid `CONDA_OVERRIDE_UNIX`
jaimergp Dec 19, 2024
cd06910
format table
jaimergp Dec 19, 2024
e520336
Reword general considerations
jaimergp Dec 19, 2024
1b01182
Be stricter with override values and more lenient with fallback values
jaimergp Dec 19, 2024
2148654
reword __archspec
jaimergp Dec 19, 2024
eaeb222
discard generic CPU rules
jaimergp Dec 19, 2024
9a95c6b
clarify env var syntax
jaimergp Dec 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
160 changes: 160 additions & 0 deletions cep-????.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,160 @@
# CEP ???? - Virtual packages

<table>
<tr><td> Title </td><td> Virtual packages </td>
<tr><td> Status </td><td> Draft </td></tr>
<tr><td> Author(s) </td><td> Jaime Rodríguez-Guerra &lt;[email protected]&gt;</td></tr>
<tr><td> Created </td><td> Dec 17, 2024</td></tr>
<tr><td> Updated </td><td> Dec 17, 2024</td></tr>
<tr><td> Discussion </td><td> https://github.com/conda/ceps/pull/103 </td></tr>
<tr><td> Implementation </td><td> https://github.com/conda/conda/tree/24.11.1/conda/plugins/virtual_packages, https://github.com/mamba-org/mamba/blob/libmamba-2.0.5/libmamba/src/core/virtual_packages.cpp, https://github.com/conda/rattler/tree/rattler-v0.28.8/crates/rattler_virtual_packages/src </td></tr>
</table>

> The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT",
"RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as
described in [RFC2119][RFC2119] when, and only when, they appear in all capitals, as shown here.

## Abstract

This CEP standardizes which virtual packages MUST be offered by conda solvers.

## Specification

A virtual package is defined as a package record with three fields: name, version and build string.
The name MUST start with double underscore. The version and build string MUST follow the same semantics as in regular package records.
jaimergp marked this conversation as resolved.
Show resolved Hide resolved

jaimergp marked this conversation as resolved.
Show resolved Hide resolved
In general, the version or build string of a virtual package MAY be overridden by the value of `CONDA_OVERRIDE_{NAME}` environment variable, with `{NAME}` being the uppercased name of the virtual package. Many exceptions apply so please observe the details in the section below.

### List of virtual packages

In alphabetical order, every conda client MUST support the following virtual packages:

- `__archspec`
- `__cuda`
- `__glibc`
- `__linux`
- `__osx`
- `__unix`
- `__win`

#### `__archspec`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The microarch-level packages in conda-forge depend on __archspec to provide microarchitecture-level (e.g. x86-64-v2) meta-packages.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we aware of any existing packages that actually depend on __archspec?

If not, I would rather we not make this virtual package mandatory at this time, mostly because "what microarchitecture is this CPU?" is a question that can get complicated quickly; see, e.g., ARM big.LITTLE, Intel P/E cores, Intel Xeon 6, etc. To me, this virtual package is still (pseudo-)experimental, in the sense that we still need to work out how package maintainers should/want to use this package. IMO, __archspec should be its own CEP or set of CEPs (cf. #59).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#103 (comment)


This virtual package MUST be always present, with the version set to `1`. The build string SHOULD reflect the detected CPU microarchitecture when the target platform matches the native platform,
as provided by the generic values in the [`archspec/archspec-json` database](https://github.com/archspec/archspec-json/blob/v0.2.5/cpu/microarchitectures.json), plus some exceptions. For example, `conda/conda` reports its generic values for Intel/AMD and `m*` names for Apple:

- Generic: `x86_64_v1`, `x86_64_v2`, `x86_64_v3`, `x86_64_v4`, `arm`, `ppc`...
- Apple: For M1, the value is `m1`. M2 and M3 are reported as `m2` and `m3`, respectively.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Im not sure I understand this part. x86_64_v1, m1 etc are also part of archspec and are also properly detected as such.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reworded this section a bit. Does that help? Otherwise I'll need more details to understand the question. Thanks!


If the microarchitecture cannot be detected or the target platform does not match the native platform, the build string MUST be set to the second component of the target platform, mapped with these rules:

| Target platform | Reported `archspec` build string |
|-----------------|----------------------------------|
| `*-32` | `x86` |
| `*-64` | `x86_64` |
| `*-armv6l` | `armv6l` |
| `*-armv7l` | `armv7l` |
| `*-aarch64` | `aarch64` |
| `*-arm64` | `arm64` |
| `*-ppc64` | `ppc64` |
| `*-ppc64le` | `ppc64le` |
| `*-riscv64` | `riscv64` |
| `*-s390x` | `s390x` |
| `zos-z` | `0` |
| Any other value | `0` |

The build string MUST be overridable with the `CONDA_OVERRIDE_ARCHSPEC` environment variable, if set to a non-empty value.
jakirkham marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are using MUST here but use MAY in the top level section about environment variables.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, the general section covers all cases (and some virtual packages are not overridable), so we can only use MAY there. Then we go case-by-case specifying the MUST parts. Happy to reword for clarity if needed, though.


#### `__cuda`

This virtual package MUST be present when the system exhibits GPU drivers compatible with the CUDA runtimes. When available, the version value MUST be set to the oldest CUDA version supported by the detected drivers (i.e. the formatted value of `libcuda.cuDriverGetVersion()`), constrained to the first two components (major and minor) and formatted as `{major}.{minor}`. The build string MUST be `0`.

Comment on lines +83 to +84
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cuDriverGetVersion returns the version of the installed driver, not the oldest CUDA version supported by the detected drivers. Due to backwards compatibility support newer drivers also support older versions.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this returns the CUDA version as seen by the driver. The docs are not super clear. There's also cudaRuntimeGetVersionAt least this is what conda/conda uses.

@jakirkham could you help clarify this? Thanks!

The version MUST be overridable with the `CONDA_OVERRIDE_CUDA` environment variable, if set to a non-empty value.

#### `__glibc`

This virtual package MUST be present when the native platform is `linux-*`. Its version value MUST be set to the system GNU `libc` version, constrained to the first two components (major and minor) formatted as `{major}.{minor}`. The build string MUST be `0`.

The version MUST be overridable with the `CONDA_OVERRIDE_GLIBC` environment variable, if set to a non-empty value.

If the GNU `libc` version could not be estimated (e.g. the tool is not running on Linux), the tool SHOULD provide a default value (e.g. `2.17`) and inform the user of that choice and its possible overrides; e.g. via `CONDA_OVERRIDE_GLIBC`, a CLI flag or a configuration file. The environment variable MUST be ignored when the target platform is not `linux~-*`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should there be a difference between gnu libc not being present and the version not being parsable? I think conda should also be able to run on systems without glibc if packages dont require it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe. What would be the expected behaviour? Something like this?

  • Not found: __glibc does not show up.
  • Not parsable: __glibc=0 + warning/advice to override.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reworded this section a bit now so let's take another look.


> The GNU `libc` version can be computed via:
>
> - Python's `os.confstr("CS_GNU_LIBC_VERSION")`
> - `getconf GNU_LIBC_VERSION`
> - `ldd --version`. Note this only applies to GLIBC distros. In MUSL distros this will return the MUSL version instead. In these systems, you might need to locate the GNU `libc.so` library and call it directly.
jaimergp marked this conversation as resolved.
Show resolved Hide resolved

#### `__linux`

This virtual package MUST be present when the target platform is `linux-*`. Its version value MUST be set to the Linux kernel version, constrained to two to four numeric components formatted as `{major}.{minor}.{micro}.{patch}`. If the version cannot be estimated (e.g. because the native platform is not Linux), the fallback value MUST be set to `0`. The build string MUST be `0`.

The version MUST be overridable with the `CONDA_OVERRIDE_LINUX` environment variable, if set to a non-empty value that matches the regex `"\d+\.\d+(\.\d+)?(\.\d+)?"`. The environment variable MUST be ignored when the target platform is not `linux-*`.

> The Linux kernel version can be obtained via:
>
> - Python's `platform.release()`
> - `uname -r`
> - `cat /proc/version`

#### `__osx`

This virtual package MUST be present when the target platform is `osx-*`. Its version value MUST be set to the first two numeric components of macOS version formatted as `{major}[.{minor}]`. If the version cannot be estimated (e.g. because the native platform is not macOS), the fallback value MUST be set to `0`. The build string MUST be `0`.

The version MUST be overridable with the `CONDA_OVERRIDE_OSX` environment variable. If this environment variable is set to the empty string `""`, then the `__osx` virtual package MUST NOT be present. The environment variable MUST be ignored when the target platform is not `osx-*`.

> The macOS version can be obtained via:
>
> - Python's `platform.mac_ver()[0]`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will return the SYSTEM_VERSION_COMPAT version if the Python interpreter running the command was built against the 10.15 SDK or earlier.

See https://eclecticlight.co/2020/08/13/macos-version-numbering-isnt-so-simple/ for details.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AFAICR, if you start the Python interpreter with SYSTEM_VERSION_COMPAT=0 it returns the 11.x based version, right?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, running setting the environment variable will return a >=11 version.

My main concern here the "MUST NOT" language around the SYSTEM_VERSION_COMPAT workaround. I agree with the idea but this is not the case for the current version of conda (see conda/conda#13832). The example an this issue still reports a 10.16 version for __osx. Changing this to "SHOULD" would be reasonable, especially given that there are many releases of tools/packages that will report the compatible version that already exist.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we could consider that a bug in conda and submit a fix for 25.1 (as we are doing for __win). I can survey public repodata for __osx usage in the wild if that helps inform this decision.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See conda/conda@d94af02 (#14449)

> - `sw_vers -productVersion`
>
> If applicable, the `SYSTEM_VERSION_COMPAT` workaround MUST NOT be enabled; e.g. the version reported for Big Sur should be 11.x and not 10.16.
jaimergp marked this conversation as resolved.
Show resolved Hide resolved

#### `__unix`

This virtual package MUST be present when the target platform is `linux-*`, `osx-*` or `freebsd-*`. The version and build string fields MUST be set to `0`.
jakirkham marked this conversation as resolved.
Show resolved Hide resolved
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than list the specific target platforms now and having to adjust this CEP if/when new platforms are adopted, I would suggest language like "when the target platform is sufficiently POSIX-y" and list the attributes necessary for that to be true (e.g., uses / for path delimiters, supports fork(3), etc.)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with the sentiment, but I'm going to need some help or references to compile that list 😬


The version or build string fields MUST NOT be overridden by the `CONDA_OVERRIDE_UNIX` environment variable. However, if this environment variable is set to a non-empty value, the `__unix` virtual package MUST be present if otherwise if would not have been. The environment variable MUST be ignored when the target platform is not `linux-*`, `osx-*` or `freebsd-*`.

#### `__win`

This virtual package MUST be present when the target platform is `win-*`. The version MUST be set to the first three numeric components of the Windows build version, formatted as `{major}.{minor}.{build}`. If the version cannot be estimated (e.g. because the target platform does not match the native platform), the fallback value MUST be set to `0`. The build string MUST be `0`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not currently the case for any installer (conda, mamba, pixi) right? I think all of them report __win=0=0 at the moment.

Im all for this change though!

Edit: mamba does seem to report the windows version properly:

> pixi exec mamba info
...
virtual packages : __win=10.0.26100=0

> pixi exec conda info
...
virtual packages : __win=0=0

> pixi info
...
Virtual packages: __win=0=0

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exactly! This is what triggered the CEP: we will need to extend what mamba does to the full ecosystem soon due to some Windows deprecations in the boost library. See referenced issues in the text for more details.


The string `{major}.{minor}.{build}` can be obtained from:

- Python's `platform.win32_ver()`
- CMD's `ver`
- Powershell's `[System.Environment]::OSVersion.Version`, `(Get-CimInstance Win32_OperatingSystem).version`
- The command `wmic os get version`

The version MUST be overridable with the `CONDA_OVERRIDE_WIN` environment variable. If this environment variable is set to the empty string `""`, then the `__win` virtual package MUST NOT be present. The environment variable MUST be ignored when the target platform is not `win-*`.

## Motivation

Virtual packages are used to expose details of the system configuration to a conda client. They are commonly used as dependencies in regular packages to constrain on which systems they can be installed. Some examples include:

* On Linux, the minimum `libc` version that must be available in the system via the `__glibc` virtual package.
* The oldest macOS version compatible with the package via the `__osx` virtual package.
* Whether a `noarch` package should be constrained to a single operating system via the `__linux`, `__osx` or `__win` virtual packages (often with no version).
* The minimum CPU microarchitecture level that the binaries require via the `__archspec` virtual package.
* The lowest CUDA version the GPU driver is compatible with via `__cuda`.
jaimergp marked this conversation as resolved.
Show resolved Hide resolved

## Rationale
jaimergp marked this conversation as resolved.
Show resolved Hide resolved

This CEP attempts to describe and standardize the existing implementations of the currently available virtual packages. The following items are deliberately kept out of scope and recommended for discussion in future CEPs:
jaimergp marked this conversation as resolved.
Show resolved Hide resolved

- Additional OSes, like `__freebsd` or `__netbsd`.
- Architectures, like `__x86_64` or `__arm64`, or, more generally, [`__arch`](https://github.com/conda/conda/issues/13420).
jaimergp marked this conversation as resolved.
Show resolved Hide resolved
- More `libc` implementations, like `__musl`.

## References

* [Virtual packages implementation in `conda/conda` 24.11.1](https://github.com/conda/conda/tree/24.11.1/conda/plugins/virtual_packages)
* [Virtual packages implementation in `libmamba` 2.0.5](https://github.com/mamba-org/mamba/blob/libmamba-2.0.5/libmamba/src/core/virtual_packages.cpp)
* [Virtual packages implementation in `rattler` 0.28.8](https://github.com/conda/rattler/tree/rattler-v0.28.8/crates/rattler_virtual_packages/src)
* [ENH: make `__win` version usable for package metadata (conda/conda#14443)](https://github.com/conda/conda/issues/14443)
* [Drop `CONDA_OVERRIDE_WIN` environment variable (mamba-org/mamba#2815)](https://github.com/mamba-org/mamba/pull/2815)
* [`__arch` feature request](https://github.com/conda/conda/issues/13420)

## Copyright

All CEPs are explicitly [CC0 1.0 Universal](https://creativecommons.org/publicdomain/zero/1.0/).