KeyError in some conda-build cases

[jaimergp]

Checklist

• I added a descriptive title
• I searched open reports and couldn't find a duplicate

What happened?

Some conda-build recipes fail with KeyError exceptions in the test phase, via display_actions. See conda-forge/stackvana-feedstock#158.

The full traceback is:

TEST START: /home/conda/feedstock_root/build_artifacts/linux-64/stackvana-lsst_distrib-0.2023.45-py311hece06ba_0.conda
WARNING: Multiple meta files found. The meta.yaml file in the base directory (/tmp/tmp_kl2cco7/info/recipe) will be used.
Renaming work directory '/home/conda/feedstock_root/build_artifacts/stackvana-final_1700180708584/work' to '/home/conda/feedstock_root/build_artifacts/stackvana-final_1700180708584/work_moved_stackvana-lsst_distrib-0.2023.45-py311hece06ba_0_linux-64'
shutil.move(work)=/home/conda/feedstock_root/build_artifacts/stackvana-final_1700180708584/work, dest=/home/conda/feedstock_root/build_artifacts/stackvana-final_1700180708584/work_moved_stackvana-lsst_distrib-0.2023.45-py311hece06ba_0_linux-64)
Reloading output folder (local): ...working... done
Solving environment (_test_env): ...working... done

# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<

    Traceback (most recent call last):
      File "/opt/conda/lib/python3.10/site-packages/conda/exception_handler.py", line 17, in __call__
        return func(*args, **kwargs)
      File "/opt/conda/lib/python3.10/site-packages/conda/cli/main.py", line 83, in main_subshell
        exit_code = do_call(args, parser)
      File "/opt/conda/lib/python3.10/site-packages/conda/cli/conda_argparse.py", line 140, in do_call
        result = plugin_subcommand.action(getattr(args, "_args", args))
      File "/opt/conda/lib/python3.10/site-packages/conda_build/plugin.py", line 10, in build
        execute(*args, **kwargs)
      File "/opt/conda/lib/python3.10/site-packages/conda_build/cli/main_build.py", line 568, in execute
        outputs = api.build(
      File "/opt/conda/lib/python3.10/site-packages/conda_build/api.py", line 253, in build
        return build_tree(
      File "/opt/conda/lib/python3.10/site-packages/conda_build/build.py", line 3819, in build_tree
        test(pkg, config=metadata.config.copy(), stats=stats)
      File "/opt/conda/lib/python3.10/site-packages/conda_build/build.py", line 3562, in test
        environ.create_env(
      File "/opt/conda/lib/python3.10/site-packages/conda_build/environ.py", line 1052, in create_env
        display_actions(actions, index)
      File "/opt/conda/lib/python3.10/site-packages/conda/exports.py", line 214, in display_actions
        actions["LINK"] = [index[d] for d in actions["LINK"]]
      File "/opt/conda/lib/python3.10/site-packages/conda/exports.py", line 214, in <listcomp>
        actions["LINK"] = [index[d] for d in actions["LINK"]]
    KeyError: Dist(channel='conda-forge', dist_name='botocore-1.32.2-pyhd8ed1ab_0', name='botocore', fmt='.conda', version='1.32.2', build_string='pyhd8ed1ab_0', build_number=0, base_url=None, platform=None)

Conda Info

active environment : base
    active env location : /opt/conda
            shell level : 1
       user config file : /home/conda/.condarc
 populated config files : /opt/conda/.condarc
                          /home/conda/.condarc
          conda version : 23.10.0
    conda-build version : 3.27.0
         python version : 3.10.13.final.0
       virtual packages : __archspec=1=x86_64_v4
                          __glibc=2.17=0
                          __linux=6.2.0=0
                          __unix=0=0
       base environment : /opt/conda  (writable)
      conda av data dir : /opt/conda/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /home/conda/feedstock_root/build_artifacts/pkg_cache
                          /opt/conda/pkgs
       envs directories : /opt/conda/envs
                          /home/conda/.conda/envs
               platform : linux-64
             user-agent : conda/23.10.0 requests/2.31.0 CPython/3.10.13 Linux/6.2.0-1016-azure centos/7.9.2009 glibc/2.17 solver/libmamba conda-libmamba-solver/23.11.1 libmambapy/1.5.3
                UID:GID : 1001:1001
             netrc file : None
           offline mode : False

Conda Config

++ conda config --env --show-sources
==> /opt/conda/.condarc <==
add_pip_as_python_dependency: False
auto_update_conda: False
aggressive_update_packages:
  - ca-certificates
  - certifi
channel_priority: strict
channels:
  - conda-forge
show_channel_urls: True
conda_build:
  pkg_format: 2
  zstd_compression_level: 19

==> /home/conda/.condarc <==
pkgs_dirs:
  - /home/conda/feedstock_root/build_artifacts/pkg_cache
  - /opt/conda/pkgs
solver: libmamba
conda-build:
  root-dir: /home/conda/feedstock_root/build_artifacts

==> envvars <==
bld_path: /home/conda/feedstock_root/build_artifacts

Conda list

# packages in environment at /opt/conda:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
anaconda-client           1.12.1             pyhd8ed1ab_1    conda-forge
anaconda-project          0.11.1             pyhd8ed1ab_0    conda-forge
archspec                  0.2.2              pyhd8ed1ab_0    conda-forge
attrs                     23.1.0             pyh71513ae_1    conda-forge
beautifulsoup4            4.12.2             pyha770c72_0    conda-forge
boltons                   23.0.0             pyhd8ed1ab_0    conda-forge
brotli-python             1.1.0           py310hc6cd4ac_1    conda-forge
bzip2                     1.0.8                hd590300_5    conda-forge
c-ares                    1.21.0               hd590300_0    conda-forge
ca-certificates           2023.7.22            hbcca054_0    conda-forge
certifi                   2023.7.22          pyhd8ed1ab_0    conda-forge
cffi                      1.16.0          py310h2fee648_0    conda-forge
chardet                   5.2.0           py310hff52083_1    conda-forge
charset-normalizer        3.3.2              pyhd8ed1ab_0    conda-forge
click                     8.1.7           unix_pyh707e725_0    conda-forge
clyent                    1.2.2              pyhd8ed1ab_2    conda-forge
colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
conda                     23.10.0         py310hff52083_1    conda-forge
conda-build               3.27.0          py310hff52083_0    conda-forge
conda-env                 2.6.0                         1    conda-forge
conda-forge-ci-setup      4.1.0           py310h7a2d8a0_100    conda-forge
conda-forge-metadata      0.6.1              pyhd8ed1ab_0    conda-forge
conda-index               0.3.0              pyhd8ed1ab_1    conda-forge
conda-libmamba-solver     23.11.1            pyhd8ed1ab_0    conda-forge
conda-oci-mirror          0.1.0              pyhd8ed1ab_0    conda-forge
conda-pack                0.7.1              pyhd8ed1ab_0    conda-forge
conda-package-handling    2.2.0              pyh38be061_0    conda-forge
conda-package-streaming   0.9.0              pyhd8ed1ab_0    conda-forge
cryptography              41.0.5          py310h75e40e8_0    conda-forge
curl                      8.4.0                hca28451_0    conda-forge
defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge
filelock                  3.13.1             pyhd8ed1ab_0    conda-forge
fmt                       10.1.1               h00ab1b0_0    conda-forge
freetype                  2.12.1               h267a509_2    conda-forge
gettext                   0.21.1               h27087fc_0    conda-forge
git                       2.42.0          pl5321h86e50cf_0    conda-forge
icu                       73.2                 h59595ed_0    conda-forge
idna                      3.4                pyhd8ed1ab_0    conda-forge
importlib_resources       6.1.1              pyhd8ed1ab_0    conda-forge
jinja2                    3.1.2              pyhd8ed1ab_1    conda-forge
jq                        1.6               h36c2ea0_1000    conda-forge
jsonpatch                 1.33               pyhd8ed1ab_0    conda-forge
jsonpointer               2.4             py310hff52083_3    conda-forge
jsonschema                4.19.2             pyhd8ed1ab_0    conda-forge
jsonschema-specifications 2023.7.1           pyhd8ed1ab_0    conda-forge
jupyter_core              5.5.0           py310hff52083_0    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
krb5                      1.21.2               h659d440_0    conda-forge
lcms2                     2.15                 hb7c19ff_3    conda-forge
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
lerc                      4.0.0                h27087fc_0    conda-forge
libarchive                3.7.2                h039dbb9_0    conda-forge
libcurl                   8.4.0                hca28451_0    conda-forge
libdeflate                1.19                 hd590300_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libexpat                  2.5.0                hcb278e6_1    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 13.2.0               h807b86a_3    conda-forge
libgomp                   13.2.0               h807b86a_3    conda-forge
libiconv                  1.17                 h166bdaf_0    conda-forge
libjpeg-turbo             3.0.0                hd590300_1    conda-forge
liblief                   0.12.3               h27087fc_0    conda-forge
libmamba                  1.5.3                had39da4_2    conda-forge
libmambapy                1.5.3           py310h39ff949_2    conda-forge
libnghttp2                1.58.0               h47da74e_0    conda-forge
libnsl                    2.0.1                hd590300_0    conda-forge
libpng                    1.6.39               h753d276_0    conda-forge
libsolv                   0.7.26               hfc55251_0    conda-forge
libsqlite                 3.44.0               h2797004_0    conda-forge
libssh2                   1.11.0               h0841786_0    conda-forge
libstdcxx-ng              13.2.0               h7e041cc_3    conda-forge
libtiff                   4.6.0                ha9c0a0a_2    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libwebp-base              1.3.2                hd590300_0    conda-forge
libxcb                    1.15                 h0b41bf4_0    conda-forge
libxml2                   2.11.5               h232c23b_1    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
lz4-c                     1.9.4                hcb278e6_0    conda-forge
lzo                       2.10              h516909a_1000    conda-forge
mamba                     1.5.3           py310h51d5547_2    conda-forge
markupsafe                2.1.3           py310h2372a71_1    conda-forge
more-itertools            10.1.0             pyhd8ed1ab_0    conda-forge
nbformat                  5.9.2              pyhd8ed1ab_0    conda-forge
ncurses                   6.4                  h59595ed_2    conda-forge
oniguruma                 6.9.9                hd590300_0    conda-forge
openjpeg                  2.5.0                h488ebb8_3    conda-forge
openssl                   3.1.4                hd590300_0    conda-forge
oras-py                   0.1.14             pyhd8ed1ab_0    conda-forge
packaging                 23.2               pyhd8ed1ab_0    conda-forge
patch                     2.7.6             h7f98852_1002    conda-forge
patchelf                  0.17.2               h58526e2_0    conda-forge
pcre2                     10.40                hc3806b6_0    conda-forge
perl                      5.32.1          4_hd590300_perl5    conda-forge
pillow                    10.1.0          py310h01dd4db_0    conda-forge
pip                       23.3.1             pyhd8ed1ab_0    conda-forge
pkginfo                   1.9.6              pyhd8ed1ab_0    conda-forge
pkgutil-resolve-name      1.3.10             pyhd8ed1ab_1    conda-forge
platformdirs              4.0.0              pyhd8ed1ab_0    conda-forge
pluggy                    1.3.0              pyhd8ed1ab_0    conda-forge
psutil                    5.9.5           py310h2372a71_1    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
py-lief                   0.12.3          py310hd8f1fbe_0    conda-forge
pybind11-abi              4                    hd8ed1ab_3    conda-forge
pycosat                   0.6.6           py310h2372a71_0    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pyopenssl                 23.3.0             pyhd8ed1ab_0    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
python                    3.10.13         hd12c33a_0_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-fastjsonschema     2.18.1             pyhd8ed1ab_0    conda-forge
python-libarchive-c       5.0             py310hff52083_2    conda-forge
python_abi                3.10                    4_cp310    conda-forge
pytz                      2023.3.post1       pyhd8ed1ab_0    conda-forge
pyyaml                    6.0.1           py310h2372a71_1    conda-forge
readline                  8.2                  h8228510_1    conda-forge
referencing               0.30.2             pyhd8ed1ab_0    conda-forge
reproc                    14.2.4.post0         hd590300_1    conda-forge
reproc-cpp                14.2.4.post0         h59595ed_1    conda-forge
requests                  2.31.0             pyhd8ed1ab_0    conda-forge
requests-toolbelt         1.0.0              pyhd8ed1ab_0    conda-forge
ripgrep                   13.0.0               he8a937b_3    conda-forge
rpds-py                   0.12.0          py310hcb5633a_0    conda-forge
ruamel.yaml               0.18.5          py310h2372a71_0    conda-forge
ruamel.yaml.clib          0.2.7           py310h2372a71_2    conda-forge
ruamel_yaml               0.15.80         py310h2372a71_1009    conda-forge
setuptools                68.2.2             pyhd8ed1ab_0    conda-forge
shyaml                    0.6.2              pyhd3deb0d_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
soupsieve                 2.5                pyhd8ed1ab_1    conda-forge
su-exec                   0.2               h166bdaf_1003    conda-forge
tini                      0.19.0               h166bdaf_1    conda-forge
tk                        8.6.13          noxft_h4845f30_101    conda-forge
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
toolz                     0.12.0             pyhd8ed1ab_0    conda-forge
tornado                   6.3.3           py310h2372a71_1    conda-forge
tqdm                      4.66.1             pyhd8ed1ab_0    conda-forge
traitlets                 5.13.0             pyhd8ed1ab_0    conda-forge
truststore                0.8.0              pyhd8ed1ab_0    conda-forge
typing-extensions         4.8.0                hd8ed1ab_0    conda-forge
typing_extensions         4.8.0              pyha770c72_0    conda-forge
tzdata                    2023c                h71feb2d_0    conda-forge
urllib3                   2.0.7              pyhd8ed1ab_0    conda-forge
wheel                     0.41.3             pyhd8ed1ab_0    conda-forge
xorg-libxau               1.0.11               hd590300_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
yaml                      0.2.5                h7f98852_2    conda-forge
yaml-cpp                  0.8.0                h59595ed_0    conda-forge
zipp                      3.17.0             pyhd8ed1ab_0    conda-forge
zstandard                 0.22.0          py310h1275a96_0    conda-forge
zstd                      1.5.5                hfc55251_0    conda-forge

Additional Context

A similar error can be seen in the conda-build test suite, only on Windows:

tests/test_api_build.py::test_build_msvc_compiler[14.0] - KeyError: Dist(channel='local', dist_name='conda-build-test-build-msvc-compiler-1.0-0', name='conda-build-test-build-msvc-compiler', fmt='.tar.bz2', version='1.0', build_string='0', build_number=0, base_url=None, platform=None)

Good platform coverage :P

[jaimergp]

If you are finding this bug when you use conda build on your infra/locally, you can work around by temporarily choosing the classic solver.

Linux / macOS:

CONDA_SOLVER=classic conda build ...

Windows

set "CONDA_SOLVER=classic"
conda build ...
set "CONDA_SOLVER="

[jaimergp]

I'm debugging live things in the only reproducer I could find: conda-forge/stackvana-feedstock#158

Interesting findings so far:

• The whole build script is REQUIRED to reproduce this bug!
• I've tried simplified examples that mimic the meta.yaml structure (i.e. same outputs, same variables, but no requirements or builds beyond the absolute necessary). They pass (this is what we have in the test suite).
• I've also tried leaving the recipe intact, but mocking the build scripts so we save 2h+ compilation times. IT ALSO PASSES!

So I have two working theories:

• The Solver._index object is also modified during the post analysis, independently of the solver, and passed around. We can't control that from conda-libmamba-solver. I haven't found the code path that could do that though.
• The index object is an synthetic organism that evolves and mutates with time and hence a long build process allows more mutations to accumulate.
• I don't know, I said two theories but I am starting to think there's a little gnome somewhere in the source.

What I'm doing next:

• I created a couple branches for conda and conda-build so they print more info about the index with missing entries.
• I'm patching the feedstock to install those branches on top of their production builds, hoping we'll see something obvious.
• If these debugging steps do not yield anything useful, I'll start considering adjusting conda-build internals to bypass the whole mess that is the index manipulation and passing. After all, the bug we are hitting is ONLY for stdout reporting. We already have the necessary information in conda-build so I don't know why we need to pass through conda. Just to use the same formatting functions? It's using an old interface anyway, so I'd consider copying that over to conda-build and deprecating the conda counterpart.

[jaimergp]

So, the debugging results show that only some keys are missing, and this is different every time.

Missing record in index: Dist(channel='conda-forge', dist_name='exceptiongroup-1.2.0-pyhd8ed1ab_0', name='exceptiongroup', fmt='.conda', version='1.2.0', build_string='pyhd8ed1ab_0', build_number=0, base_url=None, platform=None) - found these name matches:
 - conda-forge::exceptiongroup-1.1.0-pyhd8ed1ab_0
 - conda-forge::exceptiongroup-1.1.1-pyhd8ed1ab_0
 - conda-forge::exceptiongroup-1.1.2-pyhd8ed1ab_0
 - conda-forge::exceptiongroup-1.1.3-pyhd8ed1ab_0
 - conda-forge::exceptiongroup-1.0.2-pyhd8ed1ab_0
 - conda-forge::exceptiongroup-1.0.4-pyhd8ed1ab_0
 - conda-forge::exceptiongroup-1.0.1-pyhd8ed1ab_0
 - conda-forge::exceptiongroup-1.0.0-pyhd8ed1ab_0
 - conda-forge::exceptiongroup-1.0.0rc8-pyhd8ed1ab_0
 - conda-forge::exceptiongroup-1.0.0rc9-pyhd8ed1ab_0
 - conda-forge::exceptiongroup-1.0.3-pyhd8ed1ab_0
Missing record in index: Dist(channel='conda-forge', dist_name='wcwidth-0.2.11-pyhd8ed1ab_0', name='wcwidth', fmt='.conda', version='0.2.11', build_string='pyhd8ed1ab_0', build_number=0, base_url=None, platform=None) - found these name matches:
 - conda-forge::wcwidth-0.1.7-py34_0
 - conda-forge::wcwidth-0.1.7-py35_0
 - conda-forge::wcwidth-0.1.7-py27_0
 - conda-forge::wcwidth-0.1.5-py35_0
 - conda-forge::wcwidth-0.1.5-py27_0
 - conda-forge::wcwidth-0.1.5-py34_0
 - conda-forge::wcwidth-0.1.7-py36_0
 - conda-forge::wcwidth-0.2.10-pyhd8ed1ab_0
 - conda-forge::wcwidth-0.2.6-pyhd8ed1ab_0
 - conda-forge::wcwidth-0.2.7-pyhd8ed1ab_0
 - conda-forge::wcwidth-0.2.8-pyhd8ed1ab_0
 - conda-forge::wcwidth-0.2.9-pyhd8ed1ab_0
 - conda-forge::wcwidth-0.1.8-py_0
 - conda-forge::wcwidth-0.1.7-py_1
 - conda-forge::wcwidth-0.2.3-pyh9f0ad1d_0
 - conda-forge::wcwidth-0.2.5-pyh9f0ad1d_0
 - conda-forge::wcwidth-0.1.9-pyh9f0ad1d_0
 - conda-forge::wcwidth-0.2.5-pyh9f0ad1d_1
 - conda-forge::wcwidth-0.2.4-pyh9f0ad1d_0
 - conda-forge::wcwidth-0.2.2-pyh9f0ad1d_0
 - conda-forge::wcwidth-0.2.5-pyh9f0ad1d_2
 - conda-forge::wcwidth-0.1.8-pyh9f0ad1d_1

Why would some packages be absent in the conda-build index but present in the channel repodata.json? Do they get dropped at some point? It could sound like a conda-build bug that didn't get noticed because the same index with those potentially missing records would be used by the classic solver.

The most interesting part is that the dummy build does pass, so are we seeing a bad eviction thing? Memory pressure? 🤷

[jaimergp]

Oh woooow, I think got it! 😂 The missing records are missing in the conda-build index because they are too new! The build script takes so long that by the time the solver reloads the repodata, it finds a new one, downloads it, and maybe gets a new package record that conda-build didn't see when it built the in-memory index!

This would explain why the dummy build does pass: it takes less that one second to run, so by the time the solver solves the test environment, the remote repodata is still relevant.

I think the fix is to run conda-libmamba-solver offline when conda-build is calling?

[jakirkham]

Wow that is impressive! Nice work Jaime 🥳

[jaimergp]

What I don't get is that we were using a cached IndexHelper for conda-build, so reloading should have never happened to begin with. So, either there's something funky going on with the passed channels that changes the signature and then it misses the cache and reloads, or there's a part of conda-build that calls the Solver without passing through conda.plan.install_actions. The missing records were all uploaded yesterday, but with a few hours in between (more hours than the build time requires). So maybe we were super unlucky all these times? 🤷 Let's see that the next run with the offline mode in stackvana-feedstock does.

[jaimergp]

More evidence to support the "repodata got updated during the build time" theory: ray-packages also fails with KeyErrors matching packages uploaded very recently (aiohttp-3.9.0, aws-crt-cpp, etc).