Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade pip? #18

Open
vidstige opened this issue Jun 18, 2018 · 11 comments
Open

Upgrade pip? #18

vidstige opened this issue Jun 18, 2018 · 11 comments

Comments

@vidstige
Copy link

Why does pip need to be downgraded to 7.x, when the latest is 10.0.1? 🤔

@remia
Copy link

remia commented Jul 2, 2018

I think it's related to this paylogic/pip-accel#73

@vidstige
Copy link
Author

vidstige commented Jul 2, 2018

What's the best course of action here? I've done some research on debian packaging for python myself, and found that a wheel -> deb conversion tool would probably be easiest/most stable. I've seen for example that popular packages such as numpy seems to be tricky to package using setup.py based methods. One additional benefit is that wheels can be downloaded with public interface of pip, e.g. `pip download´. Just my 50c.

@remia
Copy link

remia commented Jul 2, 2018

For now I managed to use py2deb effectively but had to manually specify some dependencies that were specified with environment markers (not handled by old pip 7 I think). I really don't know if there is a more elegant solution :/

See https://github.com/Ymagis/ClairMeta/blob/develop/.travis/build_deb.sh for example.

@xolox
Copy link
Member

xolox commented Jul 2, 2018

Sorry for the long silence here. First I'll answer the question in the opening post by @vidstige:

Why does pip need to be downgraded to 7.x, when the latest is 10.0.1? thinking

py2deb uses pip-accel as an API for:

  1. Using pip to download Python packages from PyPI, unpack them and collect package metadata.
  2. Local building and caching of python setup.py bdist_dumb based binary packages.

Because pip-accel was developed against pip 7.x and is incompatible with newer versions, pip is downgraded when you install py2deb / pip-accel.


I've held back on replying in the py2deb and pip-accel issue trackers (regarding the old pip version) for a while now, because I wanted to provide a proper analysis and "best course of action". I actually started writing that up, but didn't manage to finish it, and so haven't shared it. The end result is that issues like these see nothing but silence from my side, for a long time. Sorry about that.

I'm pressed for time right now as well, so will provide a short summary of the state of affairs:

  • The pip-accel project is slowly dying due to lack of maintenance (pip upgrades!). This is caused primarily by the deep integration between pip-accel and pip. I kind of worked myself into a very tight corner there 😇.

  • Regardless of the fate of pip-accel I 'm deeply invested in the future of the py2deb project. My use-cases mean that I can't switch fully to wheels. So I'd need to support the old python setup.py bdisb_dumb based approach as well as adding support for conversion of (binary) wheels to *.deb packages. In any case py2deb needs to be upgraded to a newer / the newest version of pip to gain extended support for environment markers (as pointed out by @remia above).

In about a month (the first half of August) I'm planning to have a significant chunk of free time (time off from work, at the very least) and I've been planning to dive deep into pip-accel and py2deb then. I would first investigate if it's possible to reinvigorate the pip-accel project (and whether there is even a point in doing so, given that pip has seen great improvements in the past years):

  • If it is, then I'll do that first, followed by an updated pip-accel <-> py2deb integration.
  • If not, then I would (try to) port the essential bits from pip-accel directly into the py2deb code base and change py2deb to call pip directly instead of using pip-accel to invoke pip.

That's all for now, hopefully I'll have something more useful to share soon!

@vidstige
Copy link
Author

vidstige commented Jul 3, 2018

Thanks for the update! Makes a lot of sense. I think it would be easier to get started with py2deb if it allowed a fresher version of pip. For now, I'm just plopping all our wheels inside a debian package, which works ok. But ideally I would like to auto-convert all these wheels to deb packages at some point in time. Perhaps I'll write a small script for this, inspired by py2deb. Thanks

@RafalSkolasinski
Copy link

Any update on this?

@aTanW
Copy link

aTanW commented Oct 18, 2019

First of all, thanks for great tool!
Is it possible to make use of pip-accel optional? It will give possibility to develop pip-accel in its own schedule.
py2deb will be functional with latest pips...

@remia
Copy link

remia commented May 25, 2020

I think it is now all broken on python3.8, unless I'm mistaken pip7.2 can't install package at all.
AttributeError: module 'platform' has no attribute 'linux_distribution'

@xolox
Copy link
Member

xolox commented May 26, 2020

Hi all,

Sorry for the lack of updates from my end [1].

I've started working on a new, minimal implementation of pip-accel that strips it down to the bare essentials (the reason being that the deep integration that was previously created is what made it so hard to keep up-to-date with newer pip releases, so I don't want to repeat that mistake).

The best plan forward for py2deb seems to be to run this new pip-accel (or plain pip) using a command line like pip install --ignore-installed --prefix ... and decompose the resulting installation directory into separate packages based on information available in the lib/python*/*-packages/*.dist-info files (I have a proof of concept that seems to show it's possible).

Because these are large changes to both projects I will need to sacrifice some features, but at least it will get both projects (most importantly py2deb) back on supported pip releases.

I'm hoping to pick up speed on this in the coming weeks, if so I'll certainly share my results here.


[1] I kind of feel obliged to explain a two year gap of silence. For what it's worth, I didn't just forget about pip-accel and py2deb 😇. Shortly after my most recent message here my wife was involved in a serious car accident (while she was on holiday on the other side of the world). The past two years have been focused on her recovery and as a result I'm only just now starting to get back into programming my open source projects.

xolox added a commit that referenced this issue May 26, 2020
I intend to revive pip-accel based on the latest pip release, offering a
minimal conceptual subset of previous functionality of pip-accel, just
enough for py2deb to use.

However this will surely take some time to flesh out - possibly multiple
releases of both projects. I'm not even sure yet what will be involved
in getting pip-accel and py2deb running on the latest version of pip (I
can however already tell that large architectural changes will be
required in pip-accel and consequently also py2deb).

In the mean time I don't want any users (including my employer) run into
breakage caused by this endeavor. Alpha / beta releases on PyPI should
be able to avoid this problem, however I've never published those
myself, so I'm opting for "defense in depth" 😇.

See also this recent discussion:
#18
@vidstige
Copy link
Author

Sorry to hear that. You have no obligation to anyone, just do things if you enjoy it. Hoping the best for your wife. 🤞

@remia
Copy link

remia commented May 26, 2020

Hey @xolox, thanks for the update, don't feel any obligation or pressure on our part as @vidstige pointed !

Hoping the best for your wife. 🤞

🤞

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants