Switch to maturin.

[alex]

It seems to be much faster at doing things locally.

[alex]

TODO before mergable:

• Get this working in CI...
• Figure out how to handle version numbers
• Make sure the abi3 stuff works with PyPy
• Update wheel builder to no longer use setup.py directly (this should be done in a separate PR)
• Figure out what the correct behavior with respect to MANIFEST.in is
• Maturin stable release
• Figure out how maturin's MSRV interacts with our MSRV

[messense]

Hey, maturin's sdist support is be a bit hacky so you may run into obscure problems, feel free to ping me or open an issue if something doesn't work.

[alex]

Thanks!

…

On Thu, Apr 27, 2023 at 12:46 AM messense ***@***.***> wrote: Hey, maturin's sdist support is be a bit hacky <https://github.com/PyO3/maturin/issues?q=is%3Aissue+is%3Aopen+label%3Asdist> so you may run into obscure problems, feel free to ping me or open an issue if something doesn't work. — Reply to this email directly, view it on GitHub <#8815 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAAGBB6XJP36EAGJRCWNK3XDH23ZANCNFSM6AAAAAAXKGUW3Y> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- All that is necessary for evil to succeed is for good people to do nothing.

[alex]

@messense is there a way to disable the rewriting things to be local_dependencies? My preference is to just include everything at its normal location with include.

[messense]

is there a way to disable the rewriting things to be local_dependencies?

No way ATM, I do want to support it to avoid PyO3/maturin#1442.

[alex]

Would it be useful to file a bug for that, or should I just follow 1442?

…

On Tue, May 2, 2023 at 3:05 AM messense ***@***.***> wrote: is there a way to disable the rewriting things to be local_dependencies? No way ATM, I do want to support it to avoid PyO3/maturin#1442 <PyO3/maturin#1442>. — Reply to this email directly, view it on GitHub <#8815 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAAGBDG2335LUKBOQ772SLXECWZZANCNFSM6AAAAAAXKGUW3Y> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- All that is necessary for evil to succeed is for good people to do nothing.

[messense]

Would it be useful to file a bug for that, or should I just follow 1442?
…

Yeah, file a bug is useful.

[alex]

@messense Is it intentional that maturin doesn't set the PYO3_PYTHON env var when running cargo? setuptools-rust does (https://github.com/PyO3/setuptools-rust/blob/main/setuptools_rust/build.py#L600), and we use it in our build.rs (I think you even wrote that code :-))

Specifically, this is happening when pip wheel invokes ['maturin', 'pep517', 'build-wheel', '-i', '/Users/alex_gaynor/projects/cryptography/.venv/bin/python3.11', '--compatibility', 'off'], so I'd expect the -i value to go into PYO3_PYTHON.

[messense]

  🔗 Found pyo3 bindings with abi3 support for Python ≥ 3.7
  🐍 Not using a specific python interpreter

I think it has something to do with abi3, we can certainly pass down the interpreter path if it's specified. I can take a look this weekend.

[alex]

Thanks!

[messense]

Please try maturin 1.0.0b9.

[alex]

Ok, we are green here, huzzah!

The remaining things we need are: a) Maturin stable release (this currently uses a beta), b) @reaperhulk we need to discuss MSRV implications:

maturin currently has an MSRV of 1.64, which is obviously higher than our MSRV. It also has wheels for every possible platform it seems, https://pypi.org/project/maturin/1.0.0b9/#files

[alex]

After discussion with @reaperhulk, we think the MSRV is too aggressive. Going to hold this in draft for now.

[h-vetinari]

This is probably worth a release note? (and should be remilestoned to 43?)

[alex]

There's no public API implicated here -- the only installation method we have ever documented support for is pip (and perhaps by extension other installers that are PEP517 compatible).

Good catch re:milestone though.

[h-vetinari]

Your call of course. There are cases where someone might hit this (e.g. installation without build isolation, as distributors are wont to do), but it's not hard to find out and fix. Still, I'd tend to add it for completeness 🤷‍♂️

[eli-schwartz]

Am I correct in understanding that now maturin is used it is no longer possible to build non-abi3 wheels if you're building locally for a single python version?

[alex]

I think it may be possible with build-args (ala https://github.com/pyca/cryptography/blob/main/.github/workflows/wheel-builder.yml#L130), by simply not setting abi3. But I haven't tested, nor did we ever previously test building non-abi3 artifacts.

If there's something we can do to improve this, let us know.

[eli-schwartz]

It assumed it did work before, because when you simply run the build backend without any particular options, the previous version installed

/usr/lib/python3.12/site-packages/cryptography/hazmat/bindings/_rust.cpython-312-x86_64-linux-gnu.so

whereas the new version installs

/usr/lib/python3.12/site-packages/cryptography/hazmat/bindings/_rust.abi3.so

I futzed around a bit in the current source, and was able to convince maturin to build cpython-312-x86_64-linux-gnu.so extensions by deleting all the features = ["abi3"] from various toml files and removing pyo3/abi3-py37 from pyproject.toml, so maybe I'm wrong to think the old version was actually optimized for a specific python version (and instead it was building against the abi3 API/ABI but ignoring the "abi3" extension name?).

As far as I can tell, it's not actually possible to disable a single feature in cargo, though there's an open issue from 2016 asking for the ability. (?)

I know when I helped someone implement Limited API support in meson, the resulting implementation took:

• an attribute on the config for an extension module, asking for Limited API support and specifying the minimum version, which internally handles -DPy_LIMITED_API=0x3........... for you
• a builtin (not project-specific) build option python.allow_limited_api which can be manually toggled by the person doing a local build to opt out of the whole thing entirely

which I figured was a good balance.

[eli-schwartz]

I think it may be possible with build-args (ala https://github.com/pyca/cryptography/blob/main/.github/workflows/wheel-builder.yml#L130), by simply not setting abi3.

It appears this is set in pyproject.toml anyway. 🤷 If I try to remove it on its own, maturin spawns an error message:

💥 maturin failed
  Caused by: You have selected the `abi3` feature but not a minimum version (e.g. the `abi3-py36` feature). maturin needs a minimum version feature to build abi3 wheels.

[alex]

Hmm, it feels like maybe we need a knob in maturin to make this easier.

…

On Sun, Aug 11, 2024 at 12:39 PM Eli Schwartz ***@***.***> wrote: I think it may be possible with build-args (ala https://github.com/pyca/cryptography/blob/main/.github/workflows/wheel-builder.yml#L130), by simply not setting abi3. It appears this is set in pyproject.toml anyway. 🤷 If I try to remove it on its own, maturin spawns an error message: 💥 maturin failed Caused by: You have selected the `abi3` feature but not a minimum version (e.g. the `abi3-py36` feature). maturin needs a minimum version feature to build abi3 wheels. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

-- All that is necessary for evil to succeed is for good people to do nothing.