Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

uv pip and python -m pip resolve different versions of tensorflow in pyhf developer environment #2062

Open
matthewfeickert opened this issue Feb 29, 2024 · 8 comments
Labels
compatibility Compatibility with a specification or another tool question Asking for clarification or support resolver Related to the package resolver

Comments

@matthewfeickert
Copy link

matthewfeickert commented Feb 29, 2024

Hi. 👋 While trying to adopt uv for use in the pyhf CI system (scikit-hep/pyhf#2444) we've found that we're getting different environment solves between python -m pip install and uv pip install. The result of uv pip install results in an unusable environment of tensorflow v2.11.1 and tensorflow-probability v0.20.1 while python -m pip install resolves in the (in our view "correct") usable environment of tensorflow v2.13.1 and tensorflow-probability v0.20.1.

To avoid overwhelming wall of text I've broken the reproducible example in a Docker container into two sections that can be expanded below.

Here's the other requested info up front though:

  • The current uv platform: The python:3.8 Docker container image
# uname -a
Linux 2c24021d2969 6.5.0-15-generic #15~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Jan 12 18:54:30 UTC 2 x86_64 GNU/Linux
  • The current uv version (uv --version): uv 0.1.12

Using pip

Using pip:
$ docker run --rm -ti python:3.8 /bin/bash
root@2c24021d2969:/# python -m venv venv && . venv/bin/activate
(venv) root@2c24021d2969:/# python -m pip --quiet install --upgrade pip setuptools wheel
(venv) root@2c24021d2969:/# git clone https://github.com/scikit-hep/pyhf.git --branch main --single-branch && cd pyhf
(venv) root@2c24021d2969:/pyhf# git reset --hard fc5e83018d2a2c98156e5073a6e630793a1edd55
(venv) root@2c24021d2969:/pyhf# python -m pip --verbose --no-cache-dir install --upgrade '.[all,test]' &> /tmp/python-pip-install-verbose.txt
(venv) root@2c24021d2969:/pyhf# python -m pip list
Package                      Version
---------------------------- ---------------
absl-py                      2.1.0
ansiwrap                     0.8.4
anyio                        4.3.0
argon2-cffi                  23.1.0
argon2-cffi-bindings         21.2.0
arrow                        1.3.0
asttokens                    2.4.1
astunparse                   1.6.3
async-lru                    2.0.4
attrs                        23.2.0
awkward                      2.6.1
awkward-cpp                  29
Babel                        2.14.0
backcall                     0.2.0
beautifulsoup4               4.12.3
bleach                       6.1.0
cachetools                   5.3.3
certifi                      2024.2.2
cffi                         1.16.0
charset-normalizer           3.3.2
click                        8.1.7
click-completion             0.5.2
cloudpickle                  3.0.0
comm                         0.2.1
contourpy                    1.1.1
coverage                     7.4.3
cramjam                      2.8.1
cycler                       0.12.1
debugpy                      1.8.1
decorator                    5.1.1
defusedxml                   0.7.1
dm-tree                      0.1.8
entrypoints                  0.4
exceptiongroup               1.2.0
executing                    2.0.1
fastjsonschema               2.19.1
filelock                     3.13.1
flatbuffers                  23.5.26
fonttools                    4.49.0
fqdn                         1.5.1
fsspec                       2024.2.0
gast                         0.4.0
google-auth                  2.28.1
google-auth-oauthlib         1.0.0
google-pasta                 0.2.0
graphviz                     0.20.1
grpcio                       1.62.0
h11                          0.14.0
h5py                         3.10.0
httpcore                     1.0.4
httpx                        0.27.0
idna                         3.6
iminuit                      2.25.2
importlib-metadata           7.0.1
importlib_resources          6.1.2
iniconfig                    2.0.0
ipykernel                    6.29.3
ipympl                       0.9.3
ipython                      8.12.3
ipython-genutils             0.2.0
ipywidgets                   8.1.2
isoduration                  20.11.0
jax                          0.4.13
jaxlib                       0.4.13
jedi                         0.19.1
Jinja2                       3.1.3
json5                        0.9.17
jsonpatch                    1.33
jsonpointer                  2.4
jsonschema                   4.21.1
jsonschema-specifications    2023.12.1
jupyter                      1.0.0
jupyter_client               8.6.0
jupyter-console              6.6.3
jupyter_core                 5.7.1
jupyter-events               0.9.0
jupyter-lsp                  2.2.3
jupyter_server               2.12.5
jupyter_server_terminals     0.5.2
jupyterlab                   4.1.2
jupyterlab_pygments          0.3.0
jupyterlab_server            2.25.3
jupyterlab_widgets           3.0.10
keras                        2.13.1
kiwisolver                   1.4.5
libclang                     16.0.6
Markdown                     3.5.2
MarkupSafe                   2.1.5
matplotlib                   3.7.5
matplotlib-inline            0.1.6
mistune                      3.0.2
ml-dtypes                    0.2.0
mpmath                       1.3.0
nbclient                     0.9.0
nbconvert                    7.16.1
nbformat                     5.9.2
nest-asyncio                 1.6.0
networkx                     3.1
notebook                     7.1.1
notebook_shim                0.2.4
numpy                        1.24.3
nvidia-cublas-cu12           12.1.3.1
nvidia-cuda-cupti-cu12       12.1.105
nvidia-cuda-nvrtc-cu12       12.1.105
nvidia-cuda-runtime-cu12     12.1.105
nvidia-cudnn-cu12            8.9.2.26
nvidia-cufft-cu12            11.0.2.54
nvidia-curand-cu12           10.3.2.106
nvidia-cusolver-cu12         11.4.5.107
nvidia-cusparse-cu12         12.1.0.106
nvidia-nccl-cu12             2.18.1
nvidia-nvjitlink-cu12        12.3.101
nvidia-nvtx-cu12             12.1.105
oauthlib                     3.2.2
opt-einsum                   3.3.0
overrides                    7.7.0
packaging                    23.2
pandas                       2.0.3
pandocfilters                1.5.1
papermill                    2.3.4
parso                        0.8.3
pexpect                      4.9.0
pickleshare                  0.7.5
pillow                       10.2.0
pip                          24.0
pkgutil_resolve_name         1.3.10
platformdirs                 4.2.0
pluggy                       1.4.0
prometheus_client            0.20.0
prompt-toolkit               3.0.43
protobuf                     4.25.3
psutil                       5.9.8
ptyprocess                   0.7.0
pure-eval                    0.2.2
py-cpuinfo                   9.0.0
pyarrow                      15.0.0
pyasn1                       0.5.1
pyasn1-modules               0.3.0
pycparser                    2.21
pydocstyle                   6.3.0
pygal                        3.0.4
pygaljs                      1.0.2
Pygments                     2.17.2
pyhf                         0.7.1.dev220
pyparsing                    3.1.1
pytest                       8.0.2
pytest-benchmark             4.0.0
pytest-console-scripts       1.4.1
pytest-mock                  3.12.0
pytest-mpl                   0.17.0
pytest-socket                0.7.0
python-dateutil              2.8.2
python-json-logger           2.0.7
pytz                         2024.1
PyYAML                       6.0.1
pyzmq                        25.1.2
qtconsole                    5.5.1
QtPy                         2.4.1
referencing                  0.33.0
requests                     2.31.0
requests-mock                1.11.0
requests-oauthlib            1.3.1
rfc3339-validator            0.1.4
rfc3986-validator            0.1.1
rpds-py                      0.18.0
rsa                          4.9
scikit_hep_testdata          0.4.38
scipy                        1.10.1
scrapbook                    0.5.0
Send2Trash                   1.8.2
setuptools                   69.1.1
shellingham                  1.5.4
six                          1.16.0
sniffio                      1.3.1
snowballstemmer              2.2.0
soupsieve                    2.5
stack-data                   0.6.3
sympy                        1.12
tenacity                     8.2.3
tensorboard                  2.13.0
tensorboard-data-server      0.7.2
tensorflow                   2.13.1
tensorflow-estimator         2.13.0
tensorflow-io-gcs-filesystem 0.34.0
tensorflow-probability       0.21.0
termcolor                    2.4.0
terminado                    0.18.0
textwrap3                    0.9.2
tinycss2                     1.2.1
tomli                        2.0.1
torch                        2.1.2
tornado                      6.4
tqdm                         4.66.2
traitlets                    5.14.1
triton                       2.1.0
types-python-dateutil        2.8.19.20240106
typing_extensions            4.5.0
tzdata                       2024.1
uproot                       5.3.1
uri-template                 1.3.0
urllib3                      2.2.1
wcwidth                      0.2.13
webcolors                    1.13
webencodings                 0.5.1
websocket-client             1.7.0
Werkzeug                     3.0.1
wheel                        0.42.0
widgetsnbextension           4.0.10
wrapt                        1.16.0
zipp                         3.17.0
(venv) root@2c24021d2969:/pyhf# python -m pip list | grep 'tensor'
tensorboard                  2.13.0
tensorboard-data-server      0.7.2
tensorflow                   2.13.1
tensorflow-estimator         2.13.0
tensorflow-io-gcs-filesystem 0.34.0
tensorflow-probability       0.21.0
(venv) root@2c24021d2969:/pyhf# python -m pip show tensorflow
Name: tensorflow
Version: 2.13.1
Summary: TensorFlow is an open source machine learning framework for everyone.
Home-page: https://www.tensorflow.org/
Author: Google Inc.
Author-email: [email protected]
License: Apache 2.0
Location: /venv/lib/python3.8/site-packages
Requires: absl-py, astunparse, flatbuffers, gast, google-pasta, grpcio, h5py, keras, libclang, numpy, opt-einsum, packaging, protobuf, setuptools, six, tensorboard, tensorflow-estimator, tensorflow-io-gcs-filesystem, termcolor, typing-extensions, wrapt
Required-by:
(venv) root@2c24021d2969:/pyhf# python -c 'import tensorflow; import tensorflow_probability'  # no error
2024-02-29 00:04:43.719791: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-02-29 00:04:43.764247: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2024-02-29 00:04:44.089063: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2024-02-29 00:04:44.090564: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-02-29 00:04:44.978663: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
(venv) root@2c24021d2969:/pyhf#

python-pip-install-verbose.txt

and now in a virtual environment with uv (version 0.1.12)

uv v0.1.12:
(venv) root@2c24021d2969:/pyhf# deactivate
root@2c24021d2969:/pyhf# python -m venv /uv-venv && . /uv-venv/bin/activate
(uv-venv) root@2c24021d2969:/pyhf# python -m pip --quiet install --upgrade pip setuptools wheel uv
(uv-venv) root@2c24021d2969:/pyhf# uv --version
uv 0.1.12
(uv-venv) root@2c24021d2969:/pyhf# uv pip --verbose --no-cache install --upgrade 'pyhf[all,test] @ .' &> /tmp/uv-pip-install-verbose.txt
(uv-venv) root@2c24021d2969:/pyhf# uv pip list
Package                      Version
---------------------------- ---------------
absl-py                      2.1.0
ansiwrap                     0.8.4
anyio                        4.3.0
argon2-cffi                  23.1.0
argon2-cffi-bindings         21.2.0
arrow                        1.3.0
asttokens                    2.4.1
astunparse                   1.6.3
async-lru                    2.0.4
attrs                        23.2.0
awkward                      2.6.1
awkward-cpp                  29
babel                        2.14.0
backcall                     0.2.0
beautifulsoup4               4.12.3
bleach                       6.1.0
cachetools                   5.3.3
certifi                      2024.2.2
cffi                         1.16.0
charset-normalizer           3.3.2
click                        8.1.7
click-completion             0.5.2
cloudpickle                  3.0.0
comm                         0.2.1
contourpy                    1.1.1
coverage                     7.4.3
cramjam                      2.8.1
cycler                       0.12.1
debugpy                      1.8.1
decorator                    5.1.1
defusedxml                   0.7.1
dm-tree                      0.1.8
entrypoints                  0.4
exceptiongroup               1.2.0
executing                    2.0.1
fastjsonschema               2.19.1
filelock                     3.13.1
flatbuffers                  23.5.26
fonttools                    4.49.0
fqdn                         1.5.1
fsspec                       2024.2.0
gast                         0.4.0
google-auth                  2.28.1
google-auth-oauthlib         0.4.6
google-pasta                 0.2.0
graphviz                     0.20.1
grpcio                       1.62.0
h11                          0.14.0
h5py                         3.10.0
httpcore                     1.0.4
httpx                        0.27.0
idna                         3.6
iminuit                      2.25.2
importlib-metadata           7.0.1
importlib-resources          6.1.2
iniconfig                    2.0.0
ipykernel                    6.29.3
ipympl                       0.9.3
ipython                      8.12.3
ipython-genutils             0.2.0
ipywidgets                   8.1.2
isoduration                  20.11.0
jax                          0.4.13
jaxlib                       0.4.13
jedi                         0.19.1
jinja2                       3.1.3
json5                        0.9.17
jsonpatch                    1.33
jsonpointer                  2.4
jsonschema                   4.21.1
jsonschema-specifications    2023.12.1
jupyter                      1.0.0
jupyter-client               8.6.0
jupyter-console              6.6.3
jupyter-core                 5.7.1
jupyter-events               0.9.0
jupyter-lsp                  2.2.3
jupyter-server               2.12.5
jupyter-server-terminals     0.5.2
jupyterlab                   4.1.2
jupyterlab-pygments          0.3.0
jupyterlab-server            2.25.3
jupyterlab-widgets           3.0.10
keras                        2.11.0
kiwisolver                   1.4.5
libclang                     16.0.6
markdown                     3.5.2
markupsafe                   2.1.5
matplotlib                   3.7.5
matplotlib-inline            0.1.6
mistune                      3.0.2
ml-dtypes                    0.2.0
mpmath                       1.3.0
nbclient                     0.9.0
nbconvert                    7.16.1
nbformat                     5.9.2
nest-asyncio                 1.6.0
networkx                     3.1
notebook                     7.1.1
notebook-shim                0.2.4
numpy                        1.24.4
nvidia-cublas-cu12           12.1.3.1
nvidia-cuda-cupti-cu12       12.1.105
nvidia-cuda-nvrtc-cu12       12.1.105
nvidia-cuda-runtime-cu12     12.1.105
nvidia-cudnn-cu12            8.9.2.26
nvidia-cufft-cu12            11.0.2.54
nvidia-curand-cu12           10.3.2.106
nvidia-cusolver-cu12         11.4.5.107
nvidia-cusparse-cu12         12.1.0.106
nvidia-nccl-cu12             2.19.3
nvidia-nvjitlink-cu12        12.3.101
nvidia-nvtx-cu12             12.1.105
oauthlib                     3.2.2
opt-einsum                   3.3.0
overrides                    7.7.0
packaging                    23.2
pandas                       2.0.3
pandocfilters                1.5.1
papermill                    2.3.4
parso                        0.8.3
pexpect                      4.9.0
pickleshare                  0.7.5
pillow                       10.2.0
pip                          24.0
pkgutil-resolve-name         1.3.10
platformdirs                 4.2.0
pluggy                       1.4.0
prometheus-client            0.20.0
prompt-toolkit               3.0.43
protobuf                     3.19.6
psutil                       5.9.8
ptyprocess                   0.7.0
pure-eval                    0.2.2
py-cpuinfo                   9.0.0
pyarrow                      15.0.0
pyasn1                       0.5.1
pyasn1-modules               0.3.0
pycparser                    2.21
pydocstyle                   6.3.0
pygal                        3.0.4
pygaljs                      1.0.2
pygments                     2.17.2
pyhf                         0.7.1.dev220
pyparsing                    3.1.1
pytest                       8.0.2
pytest-benchmark             4.0.0
pytest-console-scripts       1.4.1
pytest-mock                  3.12.0
pytest-mpl                   0.17.0
pytest-socket                0.7.0
python-dateutil              2.8.2
python-json-logger           2.0.7
pytz                         2024.1
pyyaml                       6.0.1
pyzmq                        25.1.2
qtconsole                    5.5.1
qtpy                         2.4.1
referencing                  0.33.0
requests                     2.31.0
requests-mock                1.11.0
requests-oauthlib            1.3.1
rfc3339-validator            0.1.4
rfc3986-validator            0.1.1
rpds-py                      0.18.0
rsa                          4.9
scikit-hep-testdata          0.4.38
scipy                        1.10.1
scrapbook                    0.5.0
send2trash                   1.8.2
setuptools                   69.1.1
shellingham                  1.5.4
six                          1.16.0
sniffio                      1.3.1
snowballstemmer              2.2.0
soupsieve                    2.5
stack-data                   0.6.3
sympy                        1.12
tenacity                     8.2.3
tensorboard                  2.11.2
tensorboard-data-server      0.6.1
tensorboard-plugin-wit       1.8.1
tensorflow                   2.11.1
tensorflow-estimator         2.11.0
tensorflow-io-gcs-filesystem 0.34.0
tensorflow-probability       0.20.1
termcolor                    2.4.0
terminado                    0.18.0
textwrap3                    0.9.2
tinycss2                     1.2.1
tomli                        2.0.1
torch                        2.2.1
tornado                      6.4
tqdm                         4.66.2
traitlets                    5.14.1
triton                       2.2.0
types-python-dateutil        2.8.19.20240106
typing-extensions            4.10.0
tzdata                       2024.1
uproot                       5.3.1
uri-template                 1.3.0
urllib3                      2.2.1
uv                           0.1.12
wcwidth                      0.2.13
webcolors                    1.13
webencodings                 0.5.1
websocket-client             1.7.0
werkzeug                     3.0.1
wheel                        0.42.0
widgetsnbextension           4.0.10
wrapt                        1.16.0
zipp                         3.17.0
(uv-venv) root@2c24021d2969:/pyhf# uv pip list | grep tensorflow
tensorflow                   2.11.1
tensorflow-estimator         2.11.0
tensorflow-io-gcs-filesystem 0.34.0
tensorflow-probability       0.20.1
(uv-venv) root@2c24021d2969:/pyhf# python -m pip show tensorflow
Name: tensorflow
Version: 2.11.1
Summary: TensorFlow is an open source machine learning framework for everyone.
Home-page: https://www.tensorflow.org/
Author: Google Inc.
Author-email: [email protected]
License: Apache 2.0
Location: /uv-venv/lib/python3.8/site-packages
Requires: absl-py, astunparse, flatbuffers, gast, google-pasta, grpcio, h5py, keras, libclang, numpy, opt-einsum, packaging, protobuf, setuptools, six, tensorboard, tensorflow-estimator, tensorflow-io-gcs-filesystem, termcolor, typing-extensions, wrapt
Required-by:
(uv-venv) root@2c24021d2969:/pyhf# python -c 'import tensorflow; import tensorflow_probability'
2024-02-29 00:04:00.444303: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX_VNNI FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-02-29 00:04:00.527810: I tensorflow/core/util/port.cc:104] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-02-29 00:04:00.531317: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2024-02-29 00:04:00.531351: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2024-02-29 00:04:01.047385: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory
2024-02-29 00:04:01.047437: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory
2024-02-29 00:04:01.047442: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/uv-venv/lib/python3.8/site-packages/tensorflow_probability/__init__.py", line 20, in <module>
    from tensorflow_probability import substrates
  File "/uv-venv/lib/python3.8/site-packages/tensorflow_probability/substrates/__init__.py", line 17, in <module>
    from tensorflow_probability.python.internal import all_util
  File "/uv-venv/lib/python3.8/site-packages/tensorflow_probability/python/__init__.py", line 138, in <module>
    dir(globals()[pkg_name])  # Forces loading the package from its lazy loader.
  File "/uv-venv/lib/python3.8/site-packages/tensorflow_probability/python/internal/lazy_loader.py", line 57, in __dir__
    module = self._load()
  File "/uv-venv/lib/python3.8/site-packages/tensorflow_probability/python/internal/lazy_loader.py", line 37, in _load
    self._on_first_access()
  File "/uv-venv/lib/python3.8/site-packages/tensorflow_probability/python/__init__.py", line 59, in _validate_tf_environment
    raise ImportError(
ImportError: This version of TensorFlow Probability requires TensorFlow version >= 2.12; Detected an installation of version 2.11.1. Please upgrade TensorFlow to proceed.
(uv-venv) root@2c24021d2969:/pyhf#

uv-pip-install-verbose.txt

@charliermarsh
Copy link
Member

Will take a look... My guess is that these are both "valid" resolutions (since uv did pin some packages to more recent versions), and the difference is due to semi-arbitrary choices in how resolution played out in the two systems. But there could be a bug somewhere.

Also confusing is that tensorflow_probability doesn't encode this constraint in its metadata, as far as I can tell? There's no tensorflow version constraint in tensorflow_probability.

@matthewfeickert
Copy link
Author

matthewfeickert commented Feb 29, 2024

Will take a look

Thanks!

Also confusing is that tensorflow_probability doesn't encode this constraint in its metadata, as far as I can tell? There's no tensorflow version constraint in tensorflow_probability.

Correct, tensorflow-probability doesn't specifically require tensorflow as a dependency as it can work with other libraries, but it then does the internal check on the tensorflow version seen in the error log if you attempt to import them together. I can make an additional reproducer later today, but I believe if you just create an environment with just tensorflow and tensorflow-probability then uv will match pip's solve.

@notatallshaw
Copy link
Collaborator

I'm speculating here, but I would make the guess that this is another example of uv backtracking on the package creating an upper bound requirement, rather than backtracking on the package being upper bounded, like in #1398.

In this case tensorflow is upper bounding gast, grpcio, keras, numpy, protobuf, tensorboard, tensorflow-estimator, and wrapt. Skimming over the verbose logs I would say the most likely culprit is numpy. I can check later but I suspect the resolution is going to be better (and faster) with a pinned or lower bound requirement on numpy, or one of the other packages.

This kind of issue would came up a lot on the pip issue tracker after pip debuted it's resolver, that a new release of pip would cause a different solution. It's not that the resolver is incorrect, it's a kind of performance issue, if the user has strict lower bounds they want on certain packages they should include it in their requirements or constraints.

@matthewfeickert
Copy link
Author

I can make an additional reproducer later today, but I believe if you just create an environment with just tensorflow and tensorflow-probability then uv will match pip's solve.

Yeah, for just installing tensorflow and tensorflow-probability the solves are the same

Using pip:
$ docker run --rm -ti python:3.8 /bin/bash
root@0aa33f8bda2c:/# python -m venv venv && . venv/bin/activate
(venv) root@0aa33f8bda2c:/# python -m pip --quiet install --upgrade pip setuptools wheel
(venv) root@0aa33f8bda2c:/# python -m pip --verbose --no-cache-dir install --upgrade tensorflow tensorflow-probability &> /tmp/python-pip-install-verbose.txt
(venv) root@0aa33f8bda2c:/# python -m pip list
Package                      Version
---------------------------- --------
absl-py                      2.1.0
astunparse                   1.6.3
cachetools                   5.3.3
certifi                      2024.2.2
charset-normalizer           3.3.2
cloudpickle                  3.0.0
decorator                    5.1.1
dm-tree                      0.1.8
flatbuffers                  23.5.26
gast                         0.4.0
google-auth                  2.28.1
google-auth-oauthlib         1.0.0
google-pasta                 0.2.0
grpcio                       1.62.0
h5py                         3.10.0
idna                         3.6
importlib-metadata           7.0.1
keras                        2.13.1
libclang                     16.0.6
Markdown                     3.5.2
MarkupSafe                   2.1.5
numpy                        1.24.3
oauthlib                     3.2.2
opt-einsum                   3.3.0
packaging                    23.2
pip                          24.0
protobuf                     4.25.3
pyasn1                       0.5.1
pyasn1-modules               0.3.0
requests                     2.31.0
requests-oauthlib            1.3.1
rsa                          4.9
setuptools                   69.1.1
six                          1.16.0
tensorboard                  2.13.0
tensorboard-data-server      0.7.2
tensorflow                   2.13.1
tensorflow-estimator         2.13.0
tensorflow-io-gcs-filesystem 0.34.0
tensorflow-probability       0.21.0
termcolor                    2.4.0
typing_extensions            4.5.0
urllib3                      2.2.1
Werkzeug                     3.0.1
wheel                        0.42.0
wrapt                        1.16.0
zipp                         3.17.0
(venv) root@0aa33f8bda2c:/# python -m pip list | grep tensor
tensorboard                  2.13.0
tensorboard-data-server      0.7.2
tensorflow                   2.13.1
tensorflow-estimator         2.13.0
tensorflow-io-gcs-filesystem 0.34.0
tensorflow-probability       0.21.0
(venv) root@0aa33f8bda2c:/#

python-pip-install-verbose.txt

uv v0.1.13:
(venv) root@0aa33f8bda2c:/# deactivate 
root@0aa33f8bda2c:/# python -m venv /uv-venv && . /uv-venv/bin/activate
(uv-venv) root@0aa33f8bda2c:/# python -m pip --quiet install --upgrade pip setuptools wheel uv
(uv-venv) root@0aa33f8bda2c:/# uv --version
uv 0.1.13
(uv-venv) root@0aa33f8bda2c:/# uv pip --verbose --no-cache install --upgrade tensorflow tensorflow-probability &> /tmp/uv-pip-install-verbose.txt
(uv-venv) root@0aa33f8bda2c:/# uv pip list
Package                      Version 
---------------------------- --------
absl-py                      2.1.0   
astunparse                   1.6.3   
cachetools                   5.3.3   
certifi                      2024.2.2
charset-normalizer           3.3.2   
cloudpickle                  3.0.0   
decorator                    5.1.1   
dm-tree                      0.1.8   
flatbuffers                  23.5.26 
gast                         0.4.0   
google-auth                  2.28.1  
google-auth-oauthlib         1.0.0   
google-pasta                 0.2.0   
grpcio                       1.62.0  
h5py                         3.10.0  
idna                         3.6     
importlib-metadata           7.0.1   
keras                        2.13.1  
libclang                     16.0.6  
markdown                     3.5.2   
markupsafe                   2.1.5   
numpy                        1.24.3  
oauthlib                     3.2.2   
opt-einsum                   3.3.0   
packaging                    23.2    
pip                          24.0    
protobuf                     4.25.3  
pyasn1                       0.5.1   
pyasn1-modules               0.3.0   
requests                     2.31.0  
requests-oauthlib            1.3.1   
rsa                          4.9     
setuptools                   69.1.1  
six                          1.16.0  
tensorboard                  2.13.0  
tensorboard-data-server      0.7.2   
tensorflow                   2.13.1  
tensorflow-estimator         2.13.0  
tensorflow-io-gcs-filesystem 0.34.0  
tensorflow-probability       0.21.0  
termcolor                    2.4.0   
typing-extensions            4.5.0   
urllib3                      2.2.1   
uv                           0.1.13  
werkzeug                     3.0.1   
wheel                        0.42.0  
wrapt                        1.16.0  
zipp                         3.17.0  
(uv-venv) root@0aa33f8bda2c:/# uv pip list | grep tensor
tensorboard                  2.13.0  
tensorboard-data-server      0.7.2   
tensorflow                   2.13.1  
tensorflow-estimator         2.13.0  
tensorflow-io-gcs-filesystem 0.34.0  
tensorflow-probability       0.21.0  
(uv-venv) root@0aa33f8bda2c:/# 

uv-pip-install-verbose.txt

@zanieb zanieb added question Asking for clarification or support compatibility Compatibility with a specification or another tool labels Apr 14, 2024
@God-damnit-all
Copy link

When trying to add tensorflow[and-cuda]==2.16.1, uv reports that nvidia-cublas-cu12==12.3.4.1 does not exist, even though it's right here: https://pypi.org/project/nvidia-cublas-cu12/12.3.4.1/

@zanieb
Copy link
Member

zanieb commented Apr 16, 2024

@ImportTaste please open a new issue, that looks unrelated to this one.

matthewfeickert added a commit to scikit-hep/pyhf that referenced this issue Apr 30, 2024
* Use 'uv pip' for all calls to 'pip install' and 'pip uninstall' in
  CI workflows.
   - c.f. https://github.com/astral-sh/uv/
   - Still use pip for Python 3.8 until astral-sh/uv#2062
     is resolved.
* Apply subtle changes to install commands in .github/workflows/dependencies-head.yml.
   - 'uv pip install --upgrade' will try to upgrade all dependencies of the target
     package as well, which for the dependencies-head workflow isn't the goal. So
     remove the '--upgrade' from calls that also install from the
     scientific-python-nightly-wheels package index when testing only particular packages.
   - 'up pip' and 'pip' have different behavior with regards to --extra-index-url, as
     'uv pip' gives --extra-index-url priority over --index-url, where 'pip' does
     not give priority to either. Use this with 'uv pip' to give priority to the
     scientific-python-nightly-wheels package index.
* Add uv to the 'develop' extras.
@matthewfeickert
Copy link
Author

matthewfeickert commented Apr 30, 2024

Here's a more minimal reproducer of the problem, using pyhf at scikit-hep/pyhf@08755e4 and

$ uv --version
uv 0.1.39

At this commit, pyhf's 'tensorflow' extra is the following

tensorflow = [
    # python == 3.8
    "tensorflow>=2.7.0; python_version < '3.9' and platform_machine != 'arm64'",  # c.f. PR #1962, #2452
    "tensorflow-macos>=2.7.0; python_version < '3.9' and platform_machine == 'arm64' and platform_system == 'Darwin'",  # c.f. PR #2119, #2452
    "tensorflow-probability>=0.11.0; python_version < '3.9'",  # c.f. PR #1657, #2452
    # python >= 3.9
    "tensorflow-probability[tf]>=0.24.0; python_version >= '3.9'"  # c.f. PR #2452
]

so on Python 3.8 (Linux) running

uv pip install '.[tensorflow]'

should be the same as running

uv pip install . 'tensorflow>=2.7.0' 'tensorflow-probability>=0.11.0'

yet these result in different environments, where I (as the developer) can say that the second environment is the expected solve.

uv v0.1.39 with tensorflow extra (failure case):
$ docker run --rm -ti -v /tmp:/tmp python:3.8 /bin/bash
root@4935323db0ed:/# curl -LsSf https://astral.sh/uv/install.sh | sh
downloading uv 0.1.39 x86_64-unknown-linux-gnu
installing to /root/.cargo/bin
  uv
everything's installed!

To add $HOME/.cargo/bin to your PATH, either restart your shell or run:

    source $HOME/.cargo/env
root@4935323db0ed:/# . ~/.cargo/env
root@4935323db0ed:/# uv venv && . .venv/bin/activate
Using Python 3.8.19 interpreter at: usr/local/bin/python3
Creating virtualenv at: .venv
(.venv) root@4935323db0ed:/# git clone --single-branch https://github.com/scikit-hep/pyhf.git && cd pyhf && git reset --hard 08755e45ed134cbe6e5e6e238b7b5011700080b3
(.venv) root@4935323db0ed:/pyhf# uv --version
uv 0.1.39
(.venv) root@4935323db0ed:/pyhf# uv pip --verbose --no-cache install '.[tensorflow]' &> /tmp/uv-pip-install-tensorflow-extra-verbose.txt
(.venv) root@4935323db0ed:/pyhf# uv pip list
Package                      Version
---------------------------- ------------
absl-py                      2.1.0
astunparse                   1.6.3
attrs                        23.2.0
cachetools                   5.3.3
certifi                      2024.2.2
charset-normalizer           3.3.2
click                        8.1.7
cloudpickle                  3.0.0
decorator                    5.1.1
dm-tree                      0.1.8
flatbuffers                  24.3.25
gast                         0.4.0
google-auth                  2.29.0
google-auth-oauthlib         0.4.6
google-pasta                 0.2.0
grpcio                       1.62.2
h5py                         3.11.0
idna                         3.7
importlib-metadata           7.1.0
importlib-resources          6.4.0
jsonpatch                    1.33
jsonpointer                  2.4
jsonschema                   4.21.1
jsonschema-specifications    2023.12.1
keras                        2.11.0
libclang                     18.1.1
markdown                     3.6
markupsafe                   2.1.5
numpy                        1.24.4
oauthlib                     3.2.2
opt-einsum                   3.3.0
packaging                    24.0
pkgutil-resolve-name         1.3.10
protobuf                     3.19.6
pyasn1                       0.6.0
pyasn1-modules               0.4.0
pyhf                         0.7.1.dev238
pyyaml                       6.0.1
referencing                  0.35.0
requests                     2.31.0
requests-oauthlib            2.0.0
rpds-py                      0.18.0
rsa                          4.9
scipy                        1.10.1
setuptools                   69.5.1
six                          1.16.0
tensorboard                  2.11.2
tensorboard-data-server      0.6.1
tensorboard-plugin-wit       1.8.1
tensorflow                   2.11.1
tensorflow-estimator         2.11.0
tensorflow-io-gcs-filesystem 0.34.0
tensorflow-probability       0.21.0
termcolor                    2.4.0
tqdm                         4.66.2
typing-extensions            4.5.0
urllib3                      2.2.1
werkzeug                     3.0.2
wheel                        0.43.0
wrapt                        1.16.0
zipp                         3.18.1
(.venv) root@4935323db0ed:/pyhf# uv pip list | grep tensorflow
tensorflow                   2.11.1
tensorflow-estimator         2.11.0
tensorflow-io-gcs-filesystem 0.34.0
tensorflow-probability       0.21.0
(.venv) root@4935323db0ed:/pyhf# python -c 'import tensorflow; import tensorflow_probability'
2024-04-30 07:01:07.319640: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX_VNNI FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-04-30 07:01:07.399553: I tensorflow/core/util/port.cc:104] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-04-30 07:01:07.402137: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2024-04-30 07:01:07.402156: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2024-04-30 07:01:09.446572: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory
2024-04-30 07:01:09.446619: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory
2024-04-30 07:01:09.446626: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/.venv/lib/python3.8/site-packages/tensorflow_probability/__init__.py", line 20, in <module>
    from tensorflow_probability import substrates
  File "/.venv/lib/python3.8/site-packages/tensorflow_probability/substrates/__init__.py", line 17, in <module>
    from tensorflow_probability.python.internal import all_util
  File "/.venv/lib/python3.8/site-packages/tensorflow_probability/python/__init__.py", line 138, in <module>
    dir(globals()[pkg_name])  # Forces loading the package from its lazy loader.
  File "/.venv/lib/python3.8/site-packages/tensorflow_probability/python/internal/lazy_loader.py", line 57, in __dir__
    module = self._load()
  File "/.venv/lib/python3.8/site-packages/tensorflow_probability/python/internal/lazy_loader.py", line 37, in _load
    self._on_first_access()
  File "/.venv/lib/python3.8/site-packages/tensorflow_probability/python/__init__.py", line 59, in _validate_tf_environment
    raise ImportError(
ImportError: This version of TensorFlow Probability requires TensorFlow version >= 2.13; Detected an installation of version 2.11.1. Please upgrade TensorFlow to proceed.
(.venv) root@4935323db0ed:/pyhf#

uv-pip-install-tensorflow-extra-verbose.txt

uv v0.1.39 with tensorflow,tfp on command line (success case):
$ docker run --rm -ti -v /tmp:/tmp python:3.8 /bin/bash
root@edbb84aa61db:/# curl -LsSf https://astral.sh/uv/install.sh | sh
downloading uv 0.1.39 x86_64-unknown-linux-gnu
installing to /root/.cargo/bin
  uv
everything's installed!

To add $HOME/.cargo/bin to your PATH, either restart your shell or run:

    source $HOME/.cargo/env
root@edbb84aa61db:/# . ~/.cargo/env
root@edbb84aa61db:/# uv venv && . .venv/bin/activate
Using Python 3.8.19 interpreter at: usr/local/bin/python3
Creating virtualenv at: .venv
(.venv) root@edbb84aa61db:/# git clone --single-branch https://github.com/scikit-hep/pyhf.git && cd pyhf && git reset --hard 08755e45ed134cbe6e5e6e238b7b5011700080b3
(.venv) root@edbb84aa61db:/pyhf# uv pip --verbose --no-cache install . 'tensorflow>=2.7.0' 'tensorflow-probability>=0.11.0' &> /tmp/uv-pip-install-tensorflow-command-line-verbose.txt
(.venv) root@edbb84aa61db:/pyhf# uv pip list
Package                      Version
---------------------------- ------------
absl-py                      2.1.0
astunparse                   1.6.3
attrs                        23.2.0
cachetools                   5.3.3
certifi                      2024.2.2
charset-normalizer           3.3.2
click                        8.1.7
cloudpickle                  3.0.0
decorator                    5.1.1
dm-tree                      0.1.8
flatbuffers                  24.3.25
gast                         0.4.0
google-auth                  2.29.0
google-auth-oauthlib         1.0.0
google-pasta                 0.2.0
grpcio                       1.62.2
h5py                         3.11.0
idna                         3.7
importlib-metadata           7.1.0
importlib-resources          6.4.0
jsonpatch                    1.33
jsonpointer                  2.4
jsonschema                   4.21.1
jsonschema-specifications    2023.12.1
keras                        2.13.1
libclang                     18.1.1
markdown                     3.6
markupsafe                   2.1.5
numpy                        1.24.3
oauthlib                     3.2.2
opt-einsum                   3.3.0
packaging                    24.0
pkgutil-resolve-name         1.3.10
protobuf                     4.25.3
pyasn1                       0.6.0
pyasn1-modules               0.4.0
pyhf                         0.7.1.dev238
pyyaml                       6.0.1
referencing                  0.35.0
requests                     2.31.0
requests-oauthlib            2.0.0
rpds-py                      0.18.0
rsa                          4.9
scipy                        1.10.1
setuptools                   69.5.1
six                          1.16.0
tensorboard                  2.13.0
tensorboard-data-server      0.7.2
tensorflow                   2.13.1
tensorflow-estimator         2.13.0
tensorflow-io-gcs-filesystem 0.34.0
tensorflow-probability       0.21.0
termcolor                    2.4.0
tqdm                         4.66.2
typing-extensions            4.5.0
urllib3                      2.2.1
werkzeug                     3.0.2
wheel                        0.43.0
wrapt                        1.16.0
zipp                         3.18.1
(.venv) root@edbb84aa61db:/pyhf# uv pip list | grep tensorflow
tensorflow                   2.13.1
tensorflow-estimator         2.13.0
tensorflow-io-gcs-filesystem 0.34.0
tensorflow-probability       0.21.0
(.venv) root@edbb84aa61db:/pyhf# python -c 'import tensorflow; import tensorflow_probability'
2024-04-30 07:08:21.163407: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-04-30 07:08:21.164774: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2024-04-30 07:08:21.192250: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2024-04-30 07:08:21.192613: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-04-30 07:08:23.284477: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
(.venv) root@edbb84aa61db:/pyhf#

uv-pip-install-tensorflow-command-line-verbose.txt

pip (success case):
$ docker run --rm -ti -v /tmp:/tmp python:3.8 /bin/bash
root@0c9ba727a892:/# python -m venv .venv && . .venv/bin/activate
(.venv) root@0c9ba727a892:/# python -m pip --quiet install --upgrade pip setuptools wheel
(.venv) root@0c9ba727a892:/# git clone --single-branch https://github.com/scikit-hep/pyhf.git && cd pyhf && git reset --hard 08755e45ed134cbe6e5e6e238b7b5011700080b3
(.venv) root@0c9ba727a892:/pyhf# python -m pip --verbose --no-cache-dir install --upgrade '.[tensorflow]' &> /tmp/python-pip-install-tensorflow-extra-verbose.txt
(.venv) root@0c9ba727a892:/pyhf# python -m pip list
Package                      Version
---------------------------- ------------
absl-py                      2.1.0
astunparse                   1.6.3
attrs                        23.2.0
cachetools                   5.3.3
certifi                      2024.2.2
charset-normalizer           3.3.2
click                        8.1.7
cloudpickle                  3.0.0
decorator                    5.1.1
dm-tree                      0.1.8
flatbuffers                  24.3.25
gast                         0.4.0
google-auth                  2.29.0
google-auth-oauthlib         1.0.0
google-pasta                 0.2.0
grpcio                       1.62.2
h5py                         3.11.0
idna                         3.7
importlib_metadata           7.1.0
importlib_resources          6.4.0
jsonpatch                    1.33
jsonpointer                  2.4
jsonschema                   4.21.1
jsonschema-specifications    2023.12.1
keras                        2.13.1
libclang                     18.1.1
Markdown                     3.6
MarkupSafe                   2.1.5
numpy                        1.24.3
oauthlib                     3.2.2
opt-einsum                   3.3.0
packaging                    24.0
pip                          24.0
pkgutil_resolve_name         1.3.10
protobuf                     4.25.3
pyasn1                       0.6.0
pyasn1_modules               0.4.0
pyhf                         0.7.1.dev238
PyYAML                       6.0.1
referencing                  0.35.0
requests                     2.31.0
requests-oauthlib            2.0.0
rpds-py                      0.18.0
rsa                          4.9
scipy                        1.10.1
setuptools                   69.5.1
six                          1.16.0
tensorboard                  2.13.0
tensorboard-data-server      0.7.2
tensorflow                   2.13.1
tensorflow-estimator         2.13.0
tensorflow-io-gcs-filesystem 0.34.0
tensorflow-probability       0.21.0
termcolor                    2.4.0
tqdm                         4.66.2
typing_extensions            4.5.0
urllib3                      2.2.1
Werkzeug                     3.0.2
wheel                        0.43.0
wrapt                        1.16.0
zipp                         3.18.1
(.venv) root@0c9ba727a892:/pyhf# python -m pip list | grep tensorflow
tensorflow                   2.13.1
tensorflow-estimator         2.13.0
tensorflow-io-gcs-filesystem 0.34.0
tensorflow-probability       0.21.0
(.venv) root@0c9ba727a892:/pyhf# python -c 'import tensorflow; import tensorflow_probability'
2024-04-30 07:14:14.122645: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-04-30 07:14:14.124300: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2024-04-30 07:14:14.155920: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2024-04-30 07:14:14.156250: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-04-30 07:14:14.749372: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
(.venv) root@0c9ba727a892:/pyhf#

python-pip-install-tensorflow-extra-verbose.txt

@zanieb zanieb added the resolver Related to the package resolver label Jun 19, 2024
@zanieb
Copy link
Member

zanieb commented Oct 21, 2024

My presumption is that this is a case of #8157

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
compatibility Compatibility with a specification or another tool question Asking for clarification or support resolver Related to the package resolver
Projects
None yet
Development

No branches or pull requests

5 participants