Problematic installation of numpy nightly wheel under python-freethreading

[MatthewSZhang]

Checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pixi, using pixi --version.

Reproducible example

1. Problematic installation:

[project]
name = "nogil_test"
requires-python = ">= 3.11"
version = "0.1.0"

[tool.pixi.project]
channels = ["conda-forge"]
platforms = ["osx-arm64"]

[tool.pixi.dependencies]
python-freethreading = "*"
pip = "*"

[tool.pixi.pypi-dependencies]
nogil_test = { path = ".", editable = true }
numpy = { index="https://pypi.anaconda.org/scientific-python-nightly-wheels/simple" } # Use pixi install nightly wheel

[tool.pixi.tasks]
test = '''python -c "import numpy; print('Test passed')"'''

pixi update
pixi run test

2. Passed installation:

[project]
name = "nogil_test"
requires-python = ">= 3.11"
version = "0.1.0"

[tool.pixi.project]
channels = ["conda-forge"]
platforms = ["osx-arm64"]

[tool.pixi.dependencies]
python-freethreading = "*"
pip = "*"

[tool.pixi.pypi-dependencies]
nogil_test = { path = ".", editable = true }

[tool.pixi.tasks]
nightly-numpy = "pip install -i https://pypi.anaconda.org/scientific-python-nightly-wheels/simple numpy"  # Use pip install nightly wheel
test = '''python -c "import numpy; print('Test passed')"'''

Problematic installation:

pixi update
pixi run nightly-numpy
pixi run test

Issue description

In the latest pixi 0.39.2, I want to install numpy nightly wheel. However, I find the nightly wheel installed via pixi.pypi-dependencies is problematic. The error message is show below:

✨ Pixi task (test): python -c "import numpy; print('Test passed')"
Traceback (most recent call last):
  File "/Users/szhang/Documents/Code/test/nogil_test/.pixi/envs/default/lib/python3.13t/site-packages/numpy/_core/__init__.py", line 23, in <module>
    from . import multiarray
  File "/Users/szhang/Documents/Code/test/nogil_test/.pixi/envs/default/lib/python3.13t/site-packages/numpy/_core/multiarray.py", line 10, in <module>
    from . import overrides
  File "/Users/szhang/Documents/Code/test/nogil_test/.pixi/envs/default/lib/python3.13t/site-packages/numpy/_core/overrides.py", line 7, in <module>
    from numpy._core._multiarray_umath import (
        add_docstring,  _get_implementing_args, _ArrayFunctionDispatcher)
ModuleNotFoundError: No module named 'numpy._core._multiarray_umath'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/szhang/Documents/Code/test/nogil_test/.pixi/envs/default/lib/python3.13t/site-packages/numpy/__init__.py", line 114, in <module>
    from numpy.__config__ import show_config
  File "/Users/szhang/Documents/Code/test/nogil_test/.pixi/envs/default/lib/python3.13t/site-packages/numpy/__config__.py", line 4, in <module>
    from numpy._core._multiarray_umath import (
    ...<3 lines>...
    )
  File "/Users/szhang/Documents/Code/test/nogil_test/.pixi/envs/default/lib/python3.13t/site-packages/numpy/_core/__init__.py", line 49, in <module>
    raise ImportError(msg) from exc
ImportError: 

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.13 from "/Users/szhang/Documents/Code/test/nogil_test/.pixi/envs/default/bin/python"
  * The NumPy version is: "2.3.0.dev0+git20241214.5383199"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: No module named 'numpy._core._multiarray_umath'


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
    import numpy; print('Test passed')
    ^^^^^^^^^^^^
  File "/Users/szhang/Documents/Code/test/nogil_test/.pixi/envs/default/lib/python3.13t/site-packages/numpy/__init__.py", line 119, in <module>
    raise ImportError(msg) from e
ImportError: Error importing numpy: you should not try to import numpy from
        its source directory; please exit the numpy source tree, and relaunch
        your python interpreter from there.

But when I install numpy nightly wheel via pip, the problem is solved.

✨ Pixi task (test): python -c "import numpy; print('Test passed')"
Test passed

Expected behavior

I expect the numpy nightly wheel installed via pixi.pypi-dependencies should work properly, just like it is installed via pip.

[ruben-arts]

TL;DR: use system requirements to force a macosx_14_0 binary:

[tool.pixi.system-requirements]
macos = "14.0"

---

That is a very ugly error! I'm sorry. The problem comes down to pixi trying to solve for more than just your machine. Resulting in using mock machine being used, using the __macos = 13.0 version. We know how to fix this, and the first step towards a comfortable future was just merged in main. But we still need to put some effort in it.

A good clue to figure this out is to check the wheel that was installed in this case, (from pixi.lock):

      - pypi: https://pypi.anaconda.org/scientific-python-nightly-wheels/simple/numpy/2.3.0.dev0/numpy-2.3.0.dev0-cp313-cp313-macosx_11_0_arm64.whl

Then making sure there are other available wheels on https://pypi.anaconda.org/scientific-python-nightly-wheels/simple/numpy
Thus noticing this was a much older version than your machine is running (pixi info) with the system requirements set for macOS to macos = "14.0" the resulting wheel is:

      - pypi: https://pypi.anaconda.org/scientific-python-nightly-wheels/simple/numpy/2.3.0.dev0/numpy-2.3.0.dev0-cp313-cp313-macosx_14_0_arm64.whl

Which does result in your test working, on my machine ™

[MatthewSZhang]

Hi @ruben-arts, I think setting macos = "14.0" in system-requirements may be not enough.

Here, numpy-2.3.0.dev0-cp313-cp313-macosx_14_0_arm64.whl is still the wrong wheel, as my dependencies have python-freethreading and I want to test free-threaded python. The right wheel should be numpy-2.3.0.dev0-cp313-cp313t-macosx_14_0_arm64.whl.

[tool.pixi.dependencies]
python-freethreading = "*"
pip = "*"

[ruben-arts]

Ah you are right, we'll have to dive a little deeper! 🤿