maturin develop builds for mixed python versions

[rumpelsepp]

Bug Description

The bug on my machine is that maturin builds for a faulty combination of python versions.

$ tree .venv/lib/                                                                                                                                                                        .venv/lib/                                                                                                                                                                                                                                      
└── python3.11   <--- Python 3.11
    └── site-packages
        ├── […]
        ├── my_package
        │   ├── __init__.py
        │   └── my_package.cpython-312-x86_64-linux-gnu.so   <--- Python 3.12
        ├── […]
        ├── _virtualenv.pth
        └── _virtualenv.py

I have multiple Python versions on my system.

$ uv python list
cpython-3.12.5-linux-x86_64-gnu     <download available>
cpython-3.12.4-linux-x86_64-gnu     /nix/store/04gg5w1s662l329a8kh9xcwyp0k64v5a-python3-3.12.4/bin/python3.12
cpython-3.12.4-linux-x86_64-gnu     /nix/store/04gg5w1s662l329a8kh9xcwyp0k64v5a-python3-3.12.4/bin/python3 -> python3.12
cpython-3.12.4-linux-x86_64-gnu     /nix/store/04gg5w1s662l329a8kh9xcwyp0k64v5a-python3-3.12.4/bin/python -> python3.12
cpython-3.11.9-linux-x86_64-gnu     /nix/store/53kwxjyrlpj47nhv6w8hv4kaa8x6rmka-python3-3.11.9/bin/python3.11
cpython-3.11.9-linux-x86_64-gnu     /nix/store/53kwxjyrlpj47nhv6w8hv4kaa8x6rmka-python3-3.11.9/bin/python3 -> python3.11
cpython-3.11.9-linux-x86_64-gnu     /nix/store/53kwxjyrlpj47nhv6w8hv4kaa8x6rmka-python3-3.11.9/bin/python -> python3.11
cpython-3.11.9-linux-x86_64-gnu     <download available>
cpython-3.11.2-linux-x86_64-gnu     /usr/bin/python3.11
cpython-3.11.2-linux-x86_64-gnu     /usr/bin/python3 -> python3.11
cpython-3.11.2-linux-x86_64-gnu     /bin/python3.11
cpython-3.11.2-linux-x86_64-gnu     /bin/python3 -> python3.11
cpython-3.10.14-linux-x86_64-gnu    <download available>
cpython-3.9.19-linux-x86_64-gnu     <download available>
cpython-3.8.19-linux-x86_64-gnu     <download available>
pypy-3.7.13-linux-x86_64-gnu        <download available>

I have pinned it to 3.11:

$ cat .python-version
3.11

Your maturin version (maturin --version)

maturin 1.7.0

Your Python version (python -V)

Python 3.11.9

Your pip version (pip -V)

pip 24.0 from /nix/store/z5xf4y7v8b5g31x0cxj683x38n5gakam-python3.12-pip-24.0/lib/python3.12/site-packages/pip (python 3.11)

What bindings you're using

pyo3

Does cargo build work?

• Yes, it works

Steps to Reproduce

$ RUST_LOG=maturin=debug maturin develop --uv                                                                                                                                            
2024-08-27T08:36:12.435637Z DEBUG into_build_context: maturin::project_layout: Found pyproject.toml in working directory at "/home/steff/Projects/private/penlog/pyproject.toml"                                                                
2024-08-27T08:36:12.435799Z DEBUG into_build_context: maturin::project_layout: Using cargo manifest path from working directory: "/home/steff/Projects/private/penlog/Cargo.toml"                                                               
2024-08-27T08:36:12.435878Z DEBUG into_build_context:resolve_cargo_metadata: maturin::project_layout: Resolving cargo metadata from "/home/steff/Projects/private/penlog/Cargo.toml"                                                            
2024-08-27T08:36:12.472807Z  INFO into_build_context:resolve_cargo_metadata: maturin::project_layout: close time.busy=36.9ms time.idle=1.40µs                                                                                                   
📦 Including license file "/home/steff/Projects/private/penlog/LICENSE"                                                                                                                                                                         
2024-08-27T08:36:12.473001Z DEBUG into_build_context: maturin::project_layout: Project layout resolved project_root=/home/steff/Projects/private/penlog python_dir=/home/steff/Projects/private/penlog rust_module=/home/steff/Projects/private/
penlog/penlog python_module=/home/steff/Projects/private/penlog/penlog extension_name=penlog module_name=penlog                                                                                                                                 
🔗 Found pyo3 bindings                                                                                                                                                                                                                          
2024-08-27T08:36:12.530416Z DEBUG into_build_context:check_executable: maturin::python_interpreter: Found CPython interpreter at /home/steff/Projects/private/penlog/.venv/bin/python executable=/home/steff/Projects/private/penlog/.venv/bin/p
ython                                                                                                                                                                                                                                           
2024-08-27T08:36:12.530437Z  INFO into_build_context:check_executable: maturin::python_interpreter: close time.busy=23.4ms time.idle=2.04µs executable=/home/steff/Projects/private/penlog/.venv/bin/python                                     
🐍 Found CPython 3.11 at /home/steff/Projects/private/penlog/.venv/bin/python                                                                                                                                                                   
📡 Using build options features from pyproject.toml                                                                                                                                                                                             
2024-08-27T08:36:12.530455Z  INFO into_build_context: maturin::build_options: close time.busy=94.8ms time.idle=2.88µs                                                                                                                           
2024-08-27T08:36:12.553541Z DEBUG check_executable: maturin::python_interpreter: Found CPython interpreter at /home/steff/Projects/private/penlog/.venv/bin/python executable=/home/steff/Projects/private/penlog/.venv/bin/python              
2024-08-27T08:36:12.553559Z  INFO check_executable: maturin::python_interpreter: close time.busy=23.1ms time.idle=1.25µs executable=/home/steff/Projects/private/penlog/.venv/bin/python                                                        
2024-08-27T08:36:12.578320Z DEBUG maturin::develop: Found uv binary in PATH version=uv 0.3.1

2024-08-27T08:36:12.578340Z  INFO install_dependencies: maturin::develop: close time.busy=1.46µs time.idle=1.46µs
2024-08-27T08:36:12.578452Z DEBUG build_wheels: maturin::compile: Setting PYO3_PYTHON to /home/steff/Projects/private/penlog/.venv/bin/python
2024-08-27T08:36:12.578458Z DEBUG build_wheels: maturin::compile: Running env -u CARGO PYO3_ENVIRONMENT_SIGNATURE="cpython-3.11-64bit" PYO3_PYTHON="/home/steff/Projects/private/penlog/.venv/bin/python" PYTHON_SYS_EXECUTABLE="/home/steff/Projects/private/penlog/.venv/bin/python" "cargo" "rustc" "--features" "pyo3/extension-module" "--message-format" "json-render-diagnostics" "--manifest-path" "/home/steff/Projects/private/penlog/Cargo.toml" "--lib"
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.03s
2024-08-27T08:36:12.624348Z  INFO build_wheels:warn_missing_py_init: maturin::compile: close time.busy=5.85ms time.idle=2.15µs
2024-08-27T08:36:12.627097Z DEBUG build_wheels: maturin::module_writer: Adding penlog-0.1.0.dist-info/METADATA
2024-08-27T08:36:12.627217Z DEBUG build_wheels: maturin::module_writer: Adding penlog-0.1.0.dist-info/WHEEL
2024-08-27T08:36:12.627306Z DEBUG build_wheels: maturin::module_writer: Adding penlog-0.1.0.dist-info/license_files/LICENSE from /home/steff/Projects/private/penlog/LICENSE
2024-08-27T08:36:12.627766Z DEBUG build_wheels:write_bindings_module: maturin::module_writer: Adding penlog/__init__.py
2024-08-27T08:36:12.627910Z DEBUG build_wheels:write_bindings_module: maturin::module_writer: Adding penlog/penlog.cpython-312-x86_64-linux-gnu.so from /home/steff/Projects/private/penlog/target/debug/libpenlog.so
2024-08-27T08:36:12.922780Z  INFO build_wheels:write_bindings_module: maturin::module_writer: close time.busy=295ms time.idle=1.99µs
2024-08-27T08:36:12.922806Z DEBUG build_wheels: maturin::module_writer: Adding penlog-0.1.0.dist-info/RECORD
📦 Built wheel for CPython 3.11 to /tmp/nix-shell.wDHYOK/.tmp2POqRu/penlog-0.1.0-cp311-cp311-linux_x86_64.whl
2024-08-27T08:36:12.923395Z  INFO build_wheels: maturin::build_context: close time.busy=345ms time.idle=651ns
✏️   Setting installed package as editable                                                                              
2024-08-27T08:36:13.528730Z  INFO pip_install_wheel:fix_direct_url: maturin::develop: close time.busy=536ms time.idle=2.18µs wheel_filename=/tmp/nix-shell.wDHYOK/.tmp2POqRu/penlog-0.1.0-cp311-cp311-linux_x86_64.whl
2024-08-27T08:36:13.528749Z  INFO pip_install_wheel: maturin::develop: close time.busy=605ms time.idle=971ns wheel_filename=/tmp/nix-shell.wDHYOK/.tmp2POqRu/penlog-0.1.0-cp311-cp311-linux_x86_64.whl
🛠 Installed penlog-0.1.0

The error is here:

2024-08-27T08:36:12.627910Z DEBUG build_wheels:write_bindings_module: maturin::module_writer: Adding penlog/penlog.cpython-312-x86_64-linux-gnu.so from /home/steff/Projects/private/penlog/target/debug/libpenlog.so

For reasons in this line there is the wrong python version.

[messense]

That's odd, could you play around with maturin source code (use cargo run develop) to debug it? I'm asking because it might not be easy to reproduce it elsewhere. Thanks!

maturin/src/module_writer.rs

Lines 807 to 824 in a40f720

	let ext_name = &project_layout.extension_name;
	let so_filename = if is_abi3 {
	if target.is_unix() {
	format!("{ext_name}.abi3.so")
	} else {
	match python_interpreter {
	Some(python_interpreter) if python_interpreter.is_windows_debug() => {
	format!("{ext_name}_d.pyd")
	}
	// Apparently there is no tag for abi3 on windows
	_ => format!("{ext_name}.pyd"),
	}
	}
	} else {
	let python_interpreter =
	python_interpreter.expect("A python interpreter is required for non-abi3 build");
	python_interpreter.get_library_name(ext_name)
	};

[NMertsch]

I think there are two parts to this issue:

User setup

$ pip -V
pip 24.0 from /nix/store/z5xf4y7v8b5g31x0cxj683x38n5gakam-python3.12-pip-24.0/lib/python3.12/site-packages/pip (python 3.11)

Why is it python 3.11 if its path contains python3.12? This looks like a broken setup to me, but I don't know enough about nix to provide further help.

Python version discovery by maturin

I think maturin determines the Python version from the python executable in the PATH, even if maturin is installed in another environment.

I just had a similar experience on Ubuntu with this setup:

• sudo apt install python3 and sudo apt install python-is-python3 install Python 3.12 as python3 and python
• I did not do sudo apt install python3-pip, so there is no pip provided by Ubuntu
• I installed Miniconda with Python 3.11 and manually added its bin directory to my PATH (AFTER /usr/bin/, which is wrong)
• Result: pip is provided by Miniconda and installs packages for Python 3.11 into the base conda environment, but python and python3 are Python 3.12 provided by Ubuntu

Steps to reproduce:

• pip install maturin installs maturin into the conda environment and bin directory
• maturin build creates Python 3.12 wheels, which can't be installed with pip because it's the Python 3.11 pip from conda
• conda activate base fixes it: python and python3 now are Python 3.11 and maturin build creates Python 3.11 wheels

This is clearly an issue with the setup, not with maturin itself. However, maturin could check if there is a python or python3 executable in the same directory as the maturin executable, and use that to determine the interpreter version.

I don't know if that's actually desirable, or if the current behavior is best.