Deprecate or remove conda develop

[jjpr-mit]

The conda develop command has been informally deprecated multiple times:

• how to use conda develop? #1992 (comment)
• conda develop . broken? python 3.5 vs 3.6 problem? #2633 (comment)

But it is still present in the code and in the documentation. That is very likely misleading to anyone trying to learn best practices for scientific computing.

Example scenario: As soon as a new scientific programmer gets comfortable enough with Python to start working on multiple projects, the next logical step is to learn how to manage environments and dependencies. They'll start by adding a setup.py with dependencies to each project, and most likely they'll start using conda to manage environments. (That's what I recommend to users of the servers I manage.) They'll create a conda environment, activate it, and then run pip install -e ., and the code from their project and all its dependencies will be available to anything they run in that environment. If they use Jupyter, they'll install that environment as a kernel.

And then, as development progresses, they'll add more stuff. They'll conda install more packages. And now they have a problem: as described in the current documentation, it's dangerous to interleave conda install and pip install. Inevitably they'll end up with an environment that's inconsistent, un-upgradeable and unusable.

On their next project, they'll try to avoid that problem. The logical solution is to use whatever is the equivalent of pip install -e in conda. Which will lead them to conda develop, which looks like the solution to their problem. They will then waste a day or more trying to get conda develop to work, eventually stumbling across #1992 or #2633. And then they will curse the name of the person who exhorted them to use conda and follow best practices. Which in several cases has been me.

Please deprecate or remove conda develop.

[astrojuanlu]

I arrived here after trying to figure out what conda develop is because it's mentioned in the SciPy developer documentation. If it's not maintained, the best thing to do is indeed to deprecate and remove it.

The logical solution is to use whatever is the equivalent of pip install -e in conda.

This was my go-to solution too, but as mentioned by @rgommers in this scipy-dev thread, one needs to be careful with at least two things:

• Build isolation. The presence of a pyproject.toml triggers build isolation, and this can interfere with the carefully created environment (for example, through conda). The solution is to opt out by adding --no-build-isolation, as explained in the docs.
• Dependency installation. pip is an installer, and its default behavior is, naturally, to install the dependencies of a project. However, pip install might meddle with the development environment by upgrading or downgrading some of the dependencies. Traditionally, setup.py develop has covered the use case of "build extensions in-place + create an editable install" without adding dependencies, but direct setup.py invocations are now deprecated. The solution, then, is to add the --no-deps flag, as explained in this somewhat hidden corner of the pip docs.

In sum, the closest replacement to python setup.py develop or conda develop . these days is probably:

pip install --no-build-isolation --no-deps -e .

(I might be missing other deviations from setup.py develop but I think I have captured the two most important ones)

[EricThomson]

I am confused. Has conda provided a canonical source of guidance on this (outside of offhand comments from individuals in issue threads)? It seems extremely important to address, once someone is making the move from user of packages to contributor.

[astrojuanlu]

I am confused. Has conda provided a canonical source of guidance on this (outside of offhand comments from individuals in issue threads)?

To my knowledge, no. That would be the first point to address.

[maximlt]

I'm watching this issue since I've been wondering about some time already about the status of conda develop. Thanks @astrojuanlu for your analysis, in the holoviz group we've also been using pip install --no-build-isolation --no-deps -e . to install a package in dev mode in a conda environment that already included all its dependencies.

Maybe @jezdez would know a little more about the status of conda develop?

[rgommers]

Imho it's not necessarily Conda's job to do this. All that's needed is a simple way to install a .pth file into site-packages pointing at the location where the development version of a package is installed if that is outside of site-packages itself. This only has to be done once, and not on every rebuild.

This could be done by pip or installer, by a build backend, or by a standalone utility. For Meson I'll have to deal with this at some point, because development installs are not going to be inplace - so conda develop . wouldn't be doing the right thing anyway (nor pip, unless the build backend told it the correct path). It's pretty low on my wish list, but plan to revisit at some point around the next SciPy release.

[Nodd]

I am using pip ìnstall -e . in a conda environment. It works quite well (even with cython builds), but the problem I have is with the dependencies. Pip checks and installs missing dependancies in the conda environments, which creates a mess. I have to install dependancies beforehand as a manual step, and then I install with pip in development mode. This features kind of makes it a conda job...

[chbrandt]

@Nodd did you check this comment: #4251 (comment) ?

[b4rlw]

@astrojuanlu Thanks for the information, I didn't know of the existence of the --no-deps flag.
However, is it strictly necessary to depreciate conda develop? It was perfectly serviceable once upon a time and I see no reason why it couldn't be again with some time and attention. It's always seemed strange to me to have to outsource such an essential function to a different ecosystem.

[astrojuanlu]

That's a call for the conda devs to make. If it's not maintained and it's half-working, I think it's a disservice to have it in the CLI. If folks want it back, then it needs maintenance. The current state is not ideal IMHO.

[mbaudin47]

@jjpr-mit: I was using conda develop until now where it is deprecated. What solution would you suggest to use instead of using that tool and that does not use pip (since mixing pip and conda can lead to terrible results ; see stackoverflow)?

[astrojuanlu]

@mbaudin47 The pip command suggested above #4251 (comment) should not interfere with conda because it doesn't install any dependencies.

Having said that, it's the 2 year anniversary of this issue and we still don't have an official pronouncement from the conda team.

The two comments referred at the beginning of this thread are from 2018 and 2019. It's unclear what has changed since then.

A "yes, we intend to keep conda develop around and eventually fix its problems" would be nice (in which case, this issue should probably be closed). Alternatively, "yes, conda-develop is no longer the blessed way to do development installs and users are encouraged to find alternatives" would be nice too.

[jezdez]

Hey all, just catching up on this, making a call for the new @conda/builds-tools: conda develop is old at best, and doesn't do what it says on the tin (anymore).

I'd suggest updating it to call pip install --no-build-isolation --no-deps -e . if pip is available and not deprecating it.

[jaimergp]

I'd suggest updating it to call pip install --no-build-isolation --no-deps -e . if pip is available and not deprecating it

At the risk of being pedantic, someone somewhere might be using conda developfor non Python stuff and having pip in the mix in that case would be confusing. What about:

1. (Optional) Apply some heuristic (presence of setup.py, pyproject.toml, etc) to detect whether we are dealing with a Python dep.
2. Print a friendly message informing about the recommended {sys.executable} -m pip invocation (emphasis on python -m, we don't want users running their system's pip if the environment is not active)
3. If not Python, then well, apologize profusely.

[jaimergp]

(If you want to get fancy we could run grayskull too and anticipate the conda dependencies that are needed in the env; then run python -m pip ...)

[JeanChristopheMorinPerso]

Could we potentially call it something else? Which means we would deprecate develop, create a new command, say py-develop or something like that, and after a certain period of time, remove develop? Or do we foresee having to support the python development/editable mode in other languages other than Python in the future?

[dholth]

I've written about how I would implement conda-develop on the linked conda-pypi#35 issue.