Allow to specify channels in a recipe

[asmeurer]

like

channels:
  - asmeurer

Not to be confused with specifying channels on a package, which is conda/conda#988.

[asmeurer]

The problem with this is that the channels would only be added at build time, so it's only useful if you need them for your build dependencies. For the run dependencies, your users will still need to add those channels manually.

[SlaterByte]

+1
I'd like to be able to send my whole package to a build machine and not have to set special channels. Most of my conda builds are agnostic scripts, except when it comes to channels that are required for build.

[yohannLelievre]

@asmeurer
Hi, I encounter exactly the same issue while I try to specify in my meta.yaml a dependency toward the perl package in @dan-blanchard channel. Indeed, I'm no more able to load the perl package from the default channel...
What is the current state of this issue? Is there a way to get around it?

[yohannLelievre]

I finally found a way to get around this situation.

When I build my package I use the following command :
conda build myPackage -c theDependentChannel

And when I install my package in a virtual environment :
conda install myPackage -c myChannel -c theDependentChannel

It's far from being clean... But it works.

[alstuder]

@yohannLelievre
Does not work for me. Whatever is found in default channel takes precedence. Is your dependent package in the theDependentChannel also available in the default channel?

[yohannLelievre]

@alstuder
I've made some tests with the following command:
conda build myPackage -c theDependentChannel1 -c theDependentChannel2
And, during the packaging, the precedence of my channels are respected for a package hosted in both channels. Here, the dependent packages from "myPackage" are first linked from "theDependentChannel1", if possible, and then from "theDependentChannel2".

When I executed the command below:
conda build myPackage -c theDependentChannel2 -c theDependentChannel1
I observed the expected change of precedence toward the channel "theDependentChannel2".

When I executed this command:
conda build myPackage -c conda-forge
With as dependent package "readline" (version 6.2), I linked readline from conda-forge.

And when I finally executed this command:
conda build myPackage -c defaults -c conda-forge
"readline" was linked from the defaults channel.

So maybe your issue is coming from your conda or conda-build version?
My conda version is the 4.3.15 and my conda-build version is the 2.1.8.

[martin-sicho]

Any news on this issue? It would be really nice to be able to specify the channels in the meta.yml and not having to use the command line options. And not just for the reason @SlaterByte mentioned. It would also be nice to be able to do the same for runtime dependencies. Maybe not enforce specific channels, but at least notify the user that this package might not work with some packages they already installed.

I understand it might be complicated to implement, but it would definitely be a nice feature.

And a side note:

I think this issue actually goes deeper since there are many channels on Anaconda Cloud that provide the same version of the same package, but they are not really equivalent because sometimes people don't package the software properly or not in its entirety for whatever reason. So it can happen that a package might not work due to rather cryptic reasons because it is linked against an incomplete package during runtime, for example.

[CicoZhang]

Is there any solution to it now? Could be a nice feature.

[krinsman]

Here to jump on the bandwagon and state that it might be a nice feature.

On the other hand, it might introduce unnecessary restrictions, e.g. a conda build recipe says it requires XYZ package from channel ABC, but in actuality the XYZ package from channel DEF would work just as well to satisfy that dependency.

Yet again, packages from different channels which should work well together often don't actually work as well together as would be expected (e.g. unexpected clashes between packages in the standard Anaconda channel and packages in Conda-Forge).

At the very least, it might be helpful to be able to indicate in a recipe somehow that a dependency is required which isn't found (at the time of the creation of the recipe) in the standard channel. But that might not necessarily be the same thing as the proposed feature.

[msarahan]

Conda now supports channel as part of a package spec. This is in theory possible to support in conda-build. Unfortunately, the team at anaconda is overbooked with other work, and as this feature does not directly benefit anaconda (the company), it is especially low priority for us. This feature will not happen without a community contribution.

[anton-malakhov]

We are interested in this feature. E.g. we might want to build using tools from defaults channel but we don't want any of these packages for the test environment.

[linnabrown]

I really need this feature.

[jakirkham]

Just to provide an example of what this would look like with the package spec syntax @msarahan mentioned...

# meta.yaml
package:
  name: test_recipe_needing_channel
  version: 1.0

requirements:
  build:
    - python
    # this is only available at conda_build_test, and is a test of whether that channel is available.
    - conda-forge::conda_build_test_requirement

Modified this test and reused it for this example.

[jakirkham]

Conda now supports channel as part of a package spec. This is in theory possible to support in conda-build. Unfortunately, the team at anaconda is overbooked with other work, and as this feature does not directly benefit anaconda (the company), it is especially low priority for us. This feature will not happen without a community contribution.

@msarahan, completely understand this. Would you be able to provide the steps needed and some guidance about where to look for relevant pieces? I think we will need some of this knowledge transfer for a community contributor to get started 🙂

[msarahan]

This is a rabbit hole. Adding it for one spec is obvious, and probably easy. However, where should dependencies of that come from? If they're only available on the specified channel, that's fine, but what if they're available on both the specified channel and the global channels? What is the priority? How do you treat that priority separately from other packages that should follow the global priority order? I don't think that can be defined in an always-right way.

The places you have to add this are wherever conda-build is parsing the meta.yaml into MatchSpecs. Here is a good place to start: https://github.com/conda/conda-build/blob/master/conda_build/metadata.py#L1203

[jakirkham]

Is it any less of a rabbit hole than doing something like this?

conda create -n env -c ch_a -c ch_b foo ch_c::bar

Ok so is it just an issue of parsing? Or is there an additional step of passing this on to Conda to solve?

[openSourcerer9000]

7 years later... no solution yet?

[jonas-eschle]

Bringing this up again, as also discussed in #548, it's still a nightmare to add packages where a dependency is not on conda, namely: A user cannot simply install package A from the usual (say, conda-forge) channel but also needs to add channel Beta because of dependency B. He may only wants to have this channel there for the install, so he looks it up and adds it manually.
Now, it doesn't need to stop there! The install will fail because B depends on C that is available in Charlie, so as a user, you also need to add this channel and so on. Now, is a user expected to chase down all of this dependencies until he gets all the channels right? This doesn't seem to be how a package manager should work, does it?

Factually, a user will look at the package description. Which says "add channel B". He will do that. Without looking at it. And so on. If you install a package, you trust it. And then you also trust that it specifies good dependencies. So why not add it in the recipe?

In fact, this is how pip works the whole time! Anyone can upload a package, and if you install it, it will simply pull in the dependencies, regardless whether it was published by someone specific. For users, which channel actually provides dependency B, is an implementation detail. Why would someone care?
But it seems like a lot of extra effort for nothing.

Maybe just to figure out:

Ok so is it just an issue of parsing? Or is there an additional step of passing this on to Conda to solve?

[jonas-eschle]

where should dependencies of that come from? If they're only available on the specified channel, that's fine, but what if they're available on both the specified channel and the global channels? What is the priority? How do you treat that priority separately from other packages that should follow the global priority order? I don't think that can be defined in an always-right way.

To answer that, I think they should be handled exactly like

conda create -n env -c ch_a -c ch_b foo ch_c::bar

No new magic or logic, simply moving this step, which needs to be done manually by the user, into the file created by the package manager. That's all.

[Haaroon]

annoyingly my receipe fails in the staged receipes pipeline because we still cannot set channels for packages

[jakirkham]

Would be interested to know how doable this is on the libsolv side of things

cc @jaimergp @wolfv (in case either of you have insight)

[dhirschfeld]

micromamba will automatically add channels (scoped to the package) on an on-demand basis using the channel::pkg syntax:

❯ micromamba create --dry-run -n tst367 python=3.10 pytorch::pytorch
pytorch/linux-64 (check zst)                        Checked  0.5s
pytorch/noarch (check zst)                          Checked  0.5s
pytorch/noarch                                      10.4kB @  89.6kB/s  0.1s
pytorch/linux-64                                   168.8kB @ 986.0kB/s  0.2s
conda-forge/noarch                                  12.0MB @   3.7MB/s  3.2s
conda-forge/linux-64                                29.6MB @   6.3MB/s  4.8s

Transaction

  Prefix: /opt/mambaforge/envs/tst367

  Updating specs:

   - python=3.10
   - pytorch::pytorch

micromamba info

       libmamba version : 1.4.9
     micromamba version : 1.4.9
           curl version : libcurl/7.88.1 OpenSSL/3.1.1 zlib/1.2.13 zstd/1.5.2 libssh2/1.11.0 nghttp2/1.52.0
     libarchive version : libarchive 3.6.2 zlib/1.2.13 bz2lib/1.0.8 libzstd/1.5.2
       envs directories : /opt/mambaforge/envs
          package cache : /opt/mambaforge/pkgs
            environment : None (not found)
           env location : -
      user config files : /home/user/.mambarc
 populated config files : /home/user/.mambarc
                          /home/user/.condarc
                          /opt/mambaforge/.mambarc
                          /opt/mambaforge/.condarc
       virtual packages : __unix=0=0
                          __linux=5.15.90=0
                          __glibc=2.35=0
                          __archspec=1=x86_64
                          __cuda=12.2=0
               channels : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
       base environment : /opt/mambaforge
               platform : linux-64

Perhaps, if the recipe used that syntax it could also be made to work?

[jaimergp]

The bits are there, and can be implemented, but I've seen some quirks (e.g. I think defaults::xxxx fails for some reason) while debugging things this year. Might be fixed now, but it should be chedked.

That said, I think we should only allow channels that are already in the global list of channel sources in conda_build_config.yaml, so this would only be a matter of "make sure this package comes from here", if anything.

[jakirkham]

Yeah defaults is an odd duck since it is an alias for a collection of channels. Idk if that was part of the issue there

Good point. If we do that, then it is probably worth adding a good error message like "the package X is requesting package Y from channel C, which is not in the channel sources. Please add channel C to the channels if this is acceptable/intended"

[mara004]

Could someone please clarify what the state of this is?
Recipes with deps from custom channels do seem to build (e.g. bblanchon::pdfium-binaries), but I'm not sure if the channel prefix is actually honored?

---

Update: I did some testing, and think it is honored, but callers still have to enable the channel when installing the package (seems like that's what this issue is about). Note, if not enabling the channels, the install command takes very long and in the end yields a non-descriptive error that requirements could not be satisfied.

[sfc-gh-yuwang]

8 years later, is this issue solved?

[Julfried]

It is now 2024 and I fear that this has still not been resolved. I am able to build my package when I specify the channel name using conda-forge::mat73 in meta.yaml. But when installing the package after building, it states that the requirements could not be resolved. It gives me the following error:

The following package could not be installed
└─ "package-name" is not installable because it requires
└─ mat73 0.65.* , which does not exist (perhaps a missing channel).

Maybe there is a way to make this work, at least for conda-forge dependencies?

[wolfv]

Actually, the new rattler-build also supports channel-specific dependencies. However, we have not added channels either to the metadata of the package or to the recipe itself.

We didn't want to add it to the recipe itself because then it becomes less portable (although one could argue that we could make it easy to override the channels as well). For example, users might want to rebuild the package on their own channel.

What we are working towards is being able to export channel dependencies (eg. being able to say "bioconda" depends on "conda-forge") as part of the channel metadata.

I am curious how folks think this feature should work.

[jaimergp]

constructor has a channels_remap option that we could also add to conda_build_config.yaml maybe?

[Julfried]

Thank you both for your responses and helpful insights!

From my understanding, the ideal solution would be for the channels that a package depends on (like conda-forge for mat73) to be exported as part of the package metadata. This way, during installation, Conda would automatically know which channels are required and could resolve the dependencies correctly without needing users to manually specify channels.

However, for this to work seamlessly, it seems like Conda itself would also need to implement support for handling and using this channel metadata. I’m glad to hear that Rattler-Build is moving in this direction, and I hope that Conda will also implement this functionality in the future.

For now, I’ll continue using the workaround of specifying the channels directly in meta.yaml (e.g., conda-forge::mat73), but I’m looking forward to a more seamless solution once channel metadata can be properly exported and supported by Conda.