Better integration with conda commands

[travishathaway]

Checklist

• I added a descriptive title
• I searched open requests and couldn't find a duplicate

What is the idea?

As user, when I run regular commands like conda install and conda update, I want my lock file to be automatically updated like it is when you use npm install or npm update. This should be enabled via a configuration option in .condarc.

For example, if I were to enable automatic lock file generation in for conda-lock in my .condarc file via the following configuration setting:

plugins:
  conda_lock_auto: true

When I run a command like:

conda create --name my-env python

a lock file will be created and placed in the environment root or saved explicitly somewhere else.

We could also explore the possibility of adding a plugin hook to add new options to commands in conda, so that something like the following would be possible:

conda create --name my-env --lock --lock-file=lock.yml python

By passing the --lock option to conda create, conda lock will be cued to automatically create a lock file for the newly created environment.

Why is this needed?

Having to use a separate install command is confusing for users of conda new to the conda lock project. It would be easier to use if it simply integrated into existing conda commands via plugin hooks.

What should happen?

I would like to see if there are others who agree with this feature request and whether we could scope out how hard this would be to implement. I know it's a pretty big change with how conda-lock currently works, so I understand if this feature request is considered inappropriate.

Additional Context

My primary motivation for writing this is figuring out how to make environment locking a better user experience in conda.

[maresb]

Thanks, @travishathaway, for starting this conversation!

This is indeed a worthy goal that could significantly improve the user experience for conda-lock, but as you've anticipated, it's quite ambitious given the current state of the project. I’d like to outline a few areas that I think would need to be addressed to make this a reality:

1. Installation and Dependency Issues
   Currently, conda-lock works best when installed in an isolated environment due to its large dependency footprint. This stems largely from the vendored Poetry dependency solver, which brings a lot of baggage. On this front, I believe @bollwyvl’s proposal here is an excellent starting point:

   conda-lock is still a very useful tool (not a mindset), and a lot of product pipelines depend on it. However, I doubt any of them use all of the optional features, but they are always paying for them. I'd see splitting most of them into much more focused, standalone packages (e.g. conda-lock-git, and not even bothering with [extras]), and tying them together with entry_points, leaving conda-lock-core as a python-only library with absolutely minimal dependencies.

   By splitting conda-lock into more focused, standalone packages, we could make the core much lighter and easier to install in non-isolated environments. This would be fairly straightforward, but a lot of very careful and tedious work.

2. Handling Pip Dependencies
   Pip dependencies remain a popular feature, but unfortunately, conda-lock’s handling of them is far from robust. There are numerous subtle bugs and edge cases, and the code for determining satisfiability between Conda and Pip dependencies is difficult to follow. While I’ve been gradually working to improve this, the theoretical challenges of mapping between PyPI and Conda are still unresolved (although there has been recent progress in grayskull).

   Given Conda’s high standards for stability, we would need to firewall off these rough areas or prioritize resources to address them more comprehensively.

3. Architectural Challenges
   Currently, conda-lock handles solving and installation by invoking subprocesses of conda, mamba, or micromamba. It gets the job done in most cases, but when something goes wrong the user experience is not so great. I shared some additional details yesterday regarding the subprocess executable selection in Effects of options --conda VS --mamba/--micromamba #758.

   Some possible approaches to address this include:

   • Tighter integration with Conda through a non-subprocess backend.
   • Adding support for Rattler, which deals more robustly with pip dependencies, and includes some design choices I really like such as "features" and "solve groups".

Personally, I wish I had the time to tackle these challenges, but my paid freelancing work keeps me very busy. I’ve also raised this with @jezdez in private communication.

Thanks again for initiating this discussion. Being ambitious like this certainly puts the current challenges into perspective, and hope we can find a way to make this happen.

[jezdez]

@maresb Yep, @travishathaway wasn't aware of our emails, it's a timely reminder that lockfiles are important, and we should definitely figure out how to work on this together. I emailed you to discuss this further, in particular how to make this sustainable for you.