Dependency management/packaging workflow redesign: out of scope or not?

[dlukes]

There's been a ton of discussion concerning packaging and dependency management in Python over the past few years (see e.g. here as a starting point with additional links). The consensus seems to be that pieceing together a workflow based on official tools (pip, venv) is relatively complicated for a newcomer, so a variety of alternatives have sprung up (Poetry, Hatch, PDM...), which are however baked to various degrees of finished and don't always address all of the same use cases, so the end result is fragmentation and even more confusion for users. At the same time, Python's relatively long history at this point makes it hard to start from scratch in this area.

I'd be curious to know if the dev team sees Mojo as an opportunity to improve on this front too and design a modern dependency management and packaging tool that should ship with Mojo 1.0 (the way Rust 1.0 didn't ship until Cargo/crates.io was relatively polished)? Or whether this is an area which they consider out of scope.

(I understand this might be unwanted for a variety of reasons, not the least of which is that it introduces even more fragmentation, especially in the short term. And if Mojo-based libs will be able to compile for use from regular CPython, packaged as wheels -- in other words, if there's no technical need for a separate package index for Mojo-based libs -- that would be a strong argument against a split.)

[dlukes]

I see the roadmap mentions "Improved package management support." -- although that can presumably just mean bringing it on par with what Python has today?

[lattner]

One of the biggest improvements we're planning for Mojo is to eliminate the need to write code in C to get performance. :). That will help one common source of packaging problems - dealing with the C parts of the packages.

[dlukes]

I'm really excited about this too, but it's not really what I had in mind -- it sounds more like an improvement (albeit an important one) within the existing packaging/dependency workflow, rather than a reimagining of that workflow around a new modern tool (similar to e.g. Rust's Cargo).

(I hope I'm not coming across as negative or entitled; what you're doing with Mojo is great, I'm thankful you're willing to share it, and of course the ones doing the work/footing the bills get to pick the priorities of the project, that goes without saying.)

[ksandvik]

I agree, having an elegant and extensive package manager or project tool (which cargo really is and handles packaging/dependencies as one feature) would be a big selling point for Mojo even if its not a language construct. As long as this tool could be extended by developers (i.e in cargo yo could write extension tools that are installed inside .carg/bin/ that cargo the tool could call).

[ego]

Python has a weak pip packaging/dependencies tool, so on the market a lot of community alternatives (poetry, pipenv, piptools, etc). This is a known problem in Python env. Mojo shouldn't repeat these mistakes. Cargo good example.
And the sooner this tool is released, the faster the community/language codebase will grow.

[dom96]

Do you plan to implement tooling like package managers, IDE tools, formatters yourselves and ship it with Mojo?

[ksandvik]

I took a look at poetry https://python-poetry.org which is a 'cargo clone' for Python tooling/dependencies at rest and it's actually pretty neat, will start using it here for my own projects to test it out. Depending on the direction of tooling this could be an option as then mojo code will interact the same way as python modules for maximal re-use of both language packages.

Parts of the config settings are uniform (such as general meta data), some somewhat specific, like selecting a specific mojo compiler rather than python version, and some might need more work like using a mojo specific package index rather than pypi.org .

Anyway, even that is doable in poetry as you could write custom plugins. Note poetry, like cargo, also handles starting up default projects (i.e. cargo init, poetry init, cargo run, poetry run). Poetry relies on venv, not sure if mojo will use that s name approach to isolate packages or just take care of dependencies and build binaries, but venv could store the dependency packages anyway.

So going back to the bigger picture, should mojo co-exist with python tooling or start its own tooling world, or have both...

[ksandvik]

I read various reddit postings about package managers last night and:

• There are many other competing package managers like pdm, pipenv, conda, rye, and others.
• Biggest issue is that Python is old and all these new package/npm/crate style programming models happened recently.
• So there's no standard how to keep packages localized locally, venv is one of many.
• There are proposals to standardize the dependencies in a .toml file, ex: PEP 621 – Storing project metadata in pyproject.toml
• Biggest complain was that most of these tools can't handle non-Python modules, C/C++ libraries for AI and so forth. conda is a commercial product (with a free version) that tries to handle this so many have navigated to using conda for serious AI tools installations.
• All together the Python world is seriously requesting a single tool (similar to the Python philosophy to do things one way only) for package management. Or at least standardize the dependency format. And they like cargo (most of us do).

I'm sure I missed something but that's the little I found out last night.

[iamtimdavis]

We definitely hear the feedback on package management but we have a lot going on already - so we'll consider as time goes on.

We'll close this for now.

[dlukes]

Sure, makes sense that the focus is elsewhere right now :) There's so much to flesh out, and this is not really core to your own use case for Mojo, AFAICT. Thanks for clarifying/listening/responding!