fix: Fix the bug that pdm plugins are invalid on ubuntu(#3289)

[monchin]

Pull Request Checklist

• A news fragment is added in news/ describing what is new.
• Test cases added for changed code.

Describe what you have changed in this PR.

fix #3289

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 85.33%. Comparing base (d1dc03d) to head (ef4a868).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3293      +/-   ##
==========================================
- Coverage   85.33%   85.33%   -0.01%     
==========================================
  Files         112      112              
  Lines       11402    11407       +5     
  Branches     2487     2489       +2     
==========================================
+ Hits         9730     9734       +4     
- Misses       1143     1144       +1     
  Partials      529      529              

Flag	Coverage Δ
unittests	85.11% <66.66%> (-0.02%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[monchin]

I need to know why the original code is

if sys.platform == "darwin" and "osx_framework_library" in scheme_names:

Are there macos computers without "osx_framework_library" in scheme_names? Is my code correct on all mainstream systems?

[o-moe]

Are there macos computers without "osx_framework_library" in scheme_names? Is my code correct on all mainstream systems?

Yes, the built-in Python that comes with recent macOS, report

'nt', 'nt_user', 'osx_framework_user', 'posix_home', 'posix_prefix', 'posix_user'

Python from brew also reports osx_framework_library

If Python reports osx_framework_library on macOS is depending on how it is built. This will dictate where it is installed to and what osx functionalities it can access.

[o-moe]

If you want to be 100% backwards compatible regarding darwin and just add linux to the mix, this could be an alternative:

if ((sys.platform == "darwin" and "osx_framework_library" in scheme_names) or sys.platform == "linux") and kind == "prefix":

Basically that's similar to one of the statements from your issue

[monchin]

Basically that's similar to one of the statements from your issue

That's the very first code I thought which should be. But consider here

# tag 2.20.1 src/pdm/core.py #L314
    def _add_project_plugins_library(self) -> None:
        ...
        scheme = "nt" if os.name == "nt" else "posix_prefix"
        purelib = sysconfig.get_path("purelib", scheme, replace_vars)
        scripts = sysconfig.get_path("scripts", scheme, replace_vars)

If os.name != nt, then all scheme would be "posix_prefix". That's the reason why this bug occurs on ubuntu. I'm trying to ensure the consistency here and the installation route.

Or maybe there is another way better,

    def _add_project_plugins_library(self) -> None:
        ...
        scheme_names = sysconfig.get_scheme_names()
        if os.name == "nt":
            scheme = "nt"
        elif sys.platform == "darwin" and "osx_framework_library" in scheme_names:
            scheme = "posix_prefix"
        else:
            scheme = sysconfig.get_default_scheme()
        purelib = sysconfig.get_path("purelib", scheme, replace_vars)
        scripts = sysconfig.get_path("scripts", scheme, replace_vars)

That would be exactly 100% backwards compatible, and could ensure the consistency of loading path and installation path

[monchin]

So strange. I just found this bug occurs because ubuntu /usr/lib/python<version>/sysconfig.py's code is different with the code on github. On github code, there is no "posix_local", and get_default_scheme() should get "posix_prefix" on ubuntu as the code says

def _get_preferred_schemes():
    if os.name == 'nt':
        return {
            'prefix': 'nt',
            'home': 'posix_home',
            'user': 'nt_user',
        }
    if sys.platform == 'darwin' and sys._framework:
        return {
            'prefix': 'posix_prefix',
            'home': 'posix_home',
            'user': 'osx_framework_user',
        }
    return {
        'prefix': 'posix_prefix',
        'home': 'posix_home',
        'user': 'posix_user',
    }

while both on my wsl2 (ubuntu 24.04, python 3.12.3) and the server in my company (ubuntu 22.04, python 3.10.12), the code is

def _get_preferred_schemes():
    if os.name == 'nt':
        return {
            'prefix': 'nt',
            'home': 'posix_home',
            'user': 'nt_user',
        }
    if sys.platform == 'darwin' and sys._framework:
        return {
            'prefix': 'posix_prefix',
            'home': 'posix_home',
            'user': 'osx_framework_user',
        }

    if sys.base_prefix != sys.prefix or hasattr(sys, "real_prefix"):
        # virtual environments
        prefix_scheme = 'posix_prefix'
    else:
        # default to /usr for package builds, /usr/local otherwise
        deb_build = os.environ.get('DEB_PYTHON_INSTALL_LAYOUT', 'posix_local')
        if deb_build in ('deb', 'deb_system'):
            prefix_scheme = 'deb_system'
        else:
            prefix_scheme = 'posix_local'

    return {
        'prefix': prefix_scheme,
        'home': 'posix_home',
        'user': 'posix_user',
    }

I don't know why it's not the code on github. Anyway we need to fix it

[o-moe]

I don't think that this difference is a bug. Even though Ubuntu's (Debian's) Python reports as cPython, Debian patches it to better fit their distribution. There's a whole repository to track such changes: https://salsa.debian.org/python-team

Here's a thread regading changing the local installation path starting with Python 3.10 (which is part of the sysconfig patch): https://lists.debian.org/debian-python/2022/03/msg00039.html

[monchin]

I don't think that this difference is a bug.

I never regard this difference as a bug, and there's no way for us to change the code for debian team. Maybe I said "Anyway we need to fix it" misleads you, but what I mean is we need to fix the bug that results from this difference.

[frostming]

That's why using system python is a pain. So I can conclude that the issue occurs because packages are installed with posix_local scheme, but looked for with posix_prefix scheme, right. Is there anyway to use posix_prefix on both sides?

[monchin]

Is there anyway to use posix_prefix on both sides?

Consider

# pdm/models/in_process/sysconfig_get_paths.py
def get_paths(kind="default", vars=None):
     ...
        if sys.platform == "darwin" and "osx_framework_library" in scheme_names and kind == "prefix":
            return sysconfig.get_paths("posix_prefix", vars=vars)

and

# pdm/core.py
    def _add_project_plugins_library(self) -> None:
        ...
        scheme = "nt" if os.name == "nt" else "posix_prefix"

when loading plugins, pdm uses "posix_prefix" for all situations except for windows; but when installing plugins, pdm uses "posix_prefix" only for macos with "osx_framework_library". If we use the first PR I made, then the 2 sides are consistent, but maybe it's not 100% backward compatible; if we use the code @o-moe wrote, I'm afraid that the loading side is still not consistent with the installing side as it does not show special handling for macos with "osx_framework_library".

If we need "posix_prefix" on both sides, how about this

# pdm/models/in_process/sysconfig_get_paths.py
def get_paths(kind="default", vars=None):
     ...
        # as @o-moe  wrote
        if ((sys.platform == "darwin" and "osx_framework_library" in scheme_names) or sys.platform == "linux") and kind == "prefix":
            return sysconfig.get_paths("posix_prefix", vars=vars)

# pdm/core.py
    def _add_project_plugins_library(self) -> None:
        ...
        # consistent with the installing side
        if ((sys.platform == "darwin" and "osx_framework_library" in scheme_names) or sys.platform == "linux"):
            scheme = "posix_prefix"
        # sysconfig._get_default_scheme is a private function in 3.8 & 3.9 
        elif sys.version_info < (3, 10):
            scheme = "nt" if os.name == "nt" else "posix_prefix"
        else:
            scheme = sysconfig.get_default_scheme()