Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for virtual_specs checks before installation #809

Merged
merged 34 commits into from
Aug 7, 2024

Conversation

jaimergp
Copy link
Contributor

@jaimergp jaimergp commented Jun 21, 2024

Description

Adds a new key virtual_specs.

This key allows installers to run some solver checks before the installation proceeds. Meant to be used with minimum required versions for virtual packages like __osx or __glibc. We need it because the solver only runs at installer build time, not on the target machine.

We perform an offline dry-run just to check that the solver passes there. On PKG installers, we fast track the __osx checks so the native checks run as soon as possible.

Checklist - did you ...

  • Add a file to the news directory (using the template) for the next release's release notes?
  • Add / update necessary tests?
  • Add / update outdated documentation?

@conda-bot conda-bot added the cla-signed [bot] added once the contributor has signed the CLA label Jun 21, 2024
@jaimergp jaimergp force-pushed the check-os-versions branch from 75874ec to b6c4233 Compare June 21, 2024 09:32
@jaimergp jaimergp marked this pull request as ready for review June 22, 2024 10:49
@jaimergp jaimergp requested a review from a team as a code owner June 22, 2024 10:49
constructor/construct.py Outdated Show resolved Hide resolved
constructor/header.sh Show resolved Hide resolved
constructor/osxpkg.py Outdated Show resolved Hide resolved
constructor/construct.py Show resolved Hide resolved
examples/virtual_specs/construct.yaml Outdated Show resolved Hide resolved
@jaimergp
Copy link
Contributor Author

I tried the glibc detection one-liner based on ldd in several common distros and it looks like it works?

$ docker run -it centos:7 bash
Unable to find image 'centos:7' locally
7: Pulling from library/centos
6717b8ec66cd: Pull complete 
Digest: sha256:be65f488b7764ad3638f236b7b515b3678369a5124c47b8d32916d6487418ea4
Status: Downloaded newer image for centos:7
[root@0df9cc6512de /]# ldd --version | awk 'NR==1{print $NF}'
2.17
$ docker run -it debian:stable bash
Unable to find image 'debian:stable' locally
stable: Pulling from library/debian
9210b4f6492f: Pull complete 
Digest: sha256:6c07c2a542b2f1f477bb9b2f218e3b1f46266ec5db68801ad3129a36bff89004
Status: Downloaded newer image for debian:stable
root@54aac5e36c73:/# ldd --version | awk 'NR==1{print $NF}'
2.36
$ docker run -it ubuntu:22.04 bash 
Unable to find image 'ubuntu:22.04' locally
22.04: Pulling from library/ubuntu
4ce000a43472: Pull complete 
Digest: sha256:340d9b015b194dc6e2a13938944e0d016e57b9679963fdeb9ce021daac430221
Status: Downloaded newer image for ubuntu:22.04
root@dca70274216a:/# ldd --version | awk 'NR==1{print $NF}'
2.35
$ docker run -it opensuse/leap bash      
Unable to find image 'opensuse/leap:latest' locally
latest: Pulling from opensuse/leap
bf8f05b8d435: Pull complete 
Digest: sha256:1cf79e78bb69f39fb2f78a7c2c7ebc4b64cf8d82eb1df76cd36767a595ada7a8
Status: Downloaded newer image for opensuse/leap:latest
8c074df96d68:/# ldd --version | awk 'NR==1{print $NF}'
2.38
$ docker run -it --platform linux/amd64 archlinux:latest bash
Unable to find image 'archlinux:latest' locally
latest: Pulling from library/archlinux
d60d5665bb19: Pull complete 
ad89e6a4db72: Pull complete 
Digest: sha256:a58fb33d8c7206869f7b718844ac8c849fa6a1ed811964153d33d41d096f600f
Status: Downloaded newer image for archlinux:latest
[root@3a2206ee84a1 /]# ldd --version | awk 'NR==1{print $NF}'
2.39

if operator == "ge":
specs[spec.name]["min"] = str(version.matcher_vo)
elif operator == "lt" and spec.name == "__osx":
specs[spec.name]["before"] = str(version.matcher_vo)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe I'm still jet lagged, but is this used anywhere?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

marcoesters
marcoesters previously approved these changes Jul 31, 2024
#if linux and min_glibc_version
min_glibc_version="__MIN_GLIBC_VERSION__"
# ldd reports glibc in the last field of the first line
system_glibc_version=$(ldd --version | awk 'NR==1{print $NF}')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ldd on an alpine docker image with glibc installed will have a different output. ldd here would give musl version and not glibc. /lib64/ld-linux-x86-64.so.2 --version might be better.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Isuru! Added a workaround for musl systems. Invoking ld.so directly in CentOS does not work so I'll leave it with those two branches for now.

@jaimergp jaimergp merged commit 0d275c1 into conda:main Aug 7, 2024
20 checks passed

A list of virtual packages that must be satisfied at install time. Virtual
packages must start with `__`. For example, `__osx>=11` or `__glibc>=2.24`.
These specs are dry-run solved offline by the bundled `--conda-exe` binary.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the condo exe compatible with a much lower bound than one would otherwise expect. Say 2.12 on Linux.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is seems that on Linux you use ldd but that on osx you use a dry run.

any way to also use shell commands on osx? I feel like that would be more robust to increases in minimum requirements in osx

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

amazing, thanks!

@drewgilliam drewgilliam mentioned this pull request Sep 4, 2024
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cla-signed [bot] added once the contributor has signed the CLA
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

5 participants