Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stochastically getting error "ZSTD decompression error: Unknown frame descriptor", persists until index cache cleaned #3733

Closed
3 tasks done
corneliusroemer opened this issue Jan 8, 2025 · 5 comments · Fixed by #3736
Labels
status::cannot-reproduce The bug could not be reproduced type::bug Something isn't working

Comments

@corneliusroemer
Copy link
Contributor

corneliusroemer commented Jan 8, 2025

Troubleshooting docs

  • My problem is not solved in the Troubleshooting docs
  • I do NOT use the Anaconda default channels (pkgs/* etc.)

How did you install Mamba?

Micromamba

Search tried in issue tracker

zstd decompression error

Latest version of Mamba

  • My problem is not solved with the latest version

Tried in Conda?

I didn't try

Describe your issue

Preface: Stochastic error, so not critical, but just opening the issue in case others face it as well. In any case there could be better error handling: cleaning of index cache when this error happens rather than make the user have to do it manually.

When doing micromamba install mailbox I suddenly got the error libmamba ZSTD decompression error: Unknown frame descriptor:

...snip...
━━━━━━━━━━━   0.0 B /  ??.?MB @  ??.?MB/s  0.2s
error    libmamba ZSTD decompression error: Unknown frame descriptor
warning  libmamba Download error (23) Failed writing received data to disk/application [https://conda.anaconda.org/nodefaults/noarch/repodata.json.zst]
critical libmamba Multiple errors occurred:
    Download error (23) Failed writing received data to disk/application [https://conda.anaconda.org/nodefaults/noarch/repodata.json.zst]
    Failure writing output to destination, passed 19 returned 20
    Subdir nodefaults/noarch not loaded!

Retrying the install didn't help. Only cleaning the index cache helped with micromamba clean -a

This is the first time I see the issue, but it calls for better error handling even if it's rare, so opening just in case.

Possibly related to: #2707

Logs of the incident and what I did right before

$ micromamba install pycryptodome
conda-forge/noarch                                          Using cache
conda-forge/osx-arm64                                       Using cache
bioconda/noarch                                             Using cache
bioconda/osx-arm64                                          Using cache

Pinned packages:

  - python=3.13


Transaction

  Prefix: /Users/corneliusromer/micromamba/envs/py13

  Updating specs:

   - pycryptodome


  Package         Version  Build            Channel           Size
────────────────────────────────────────────────────────────────────
  Install:
────────────────────────────────────────────────────────────────────

  + gmp             6.3.0  h7bae524_2       conda-forge     Cached
  + pycryptodome   3.21.0  py313hba17e9d_0  conda-forge        2MB

  Summary:

  Install: 2 packages

  Total download: 2MB

────────────────────────────────────────────────────────────────────



Transaction starting
pycryptodome                                         1.6MB @  41.4kB/s  0.0s
Linking gmp-6.3.0-h7bae524_2
Linking pycryptodome-3.21.0-py313hba17e9d_0

Transaction finished

$ micromamba install mailbox
conda-forge/noarch                                          Using cache
conda-forge/osx-arm64                                       Using cache
bioconda/noarch                                             Using cache
bioconda/osx-arm64                                          Using cache
[+] 0.1s
nodefaults/noarch    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0.0 B /  ??.?MB @  ??.?MB/s  0.1s
nodefaults/osx-arm64 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0.0 B /  ??.?MB @  ??.?MB/s  0.1serror    libmamba ZSTD decompression error: Unknown frame descriptor
[+] 0.2s
nodefaults/noarch ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0.0 B /  ??.?MB @  ??.?MB/s  0.2serror    libmamba ZSTD decompression error: Unknown frame descriptor
warning  libmamba Download error (23) Failed writing received data to disk/application [https://conda.anaconda.org/nodefaults/noarch/repodata.json.zst]
critical libmamba Multiple errors occurred:
    Download error (23) Failed writing received data to disk/application [https://conda.anaconda.org/nodefaults/noarch/repodata.json.zst]
    Failure writing output to destination, passed 19 returned 20
    Subdir nodefaults/noarch not loaded!

$ micromamba install mailbox
conda-forge/noarch                                          Using cache
conda-forge/osx-arm64                                       Using cache
bioconda/noarch                                             Using cache
bioconda/osx-arm64                                          Using cache
[+] 0.1s
error    libmamba ZSTD decompression error: Unknown frame descriptor
warning  libmamba Download error (23) Failed writing received data to disk/application [https://conda.anaconda.org/nodefaults/noarch/repodata.json.zst]
[+] 0.2s
nodefaults/osx-arm64 ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------   0.0 B /  ??.?MB @  ??.?MB/s  0.0serror    libmamba ZSTD decompression error: Unknown frame descriptor
warning  libmamba Download error (23) Failed writing received data to disk/application [https://conda.anaconda.org/nodefaults/osx-arm64/repodata.json.zst]
[+] 0.3s
error    libmamba ZSTD decompression error: Unknown frame descriptor
warning  libmamba Download error (23) Failed writing received data to disk/application [https://conda.anaconda.org/nodefaults/noarch/repodata.json.zst]
critical libmamba Multiple errors occurred:
    Download error (23) Failed writing received data to disk/application [https://conda.anaconda.org/nodefaults/noarch/repodata.json.zst]
    Failure writing output to destination, passed 19 returned 20
    Subdir nodefaults/noarch not loaded!

mamba info / micromamba info

libmamba version : 2.0.2
     micromamba version : 2.0.2
           curl version : libcurl/8.9.1 (OpenSSL/3.3.2) SecureTransport zlib/1.3.1 zstd/1.5.6 libssh2/1.11.0 nghttp2/1.58.0
     libarchive version : libarchive 3.7.4 zlib/1.2.13 bz2lib/1.0.8 libzstd/1.5.6
       envs directories : /Users/corneliusromer/micromamba/envs
          package cache : /Users/corneliusromer/micromamba/pkgs
                          /Users/corneliusromer/.mamba/pkgs
            environment : py13 (active)
           env location : /Users/corneliusromer/micromamba/envs/py13
      user config files : /Users/corneliusromer/.mambarc
 populated config files : /Users/corneliusromer/.mambarc
                          /Users/corneliusromer/.condarc
       virtual packages : __unix=0=0
                          __osx=15.2=0
                          __archspec=1=arm64
               channels : https://conda.anaconda.org/conda-forge/noarch
                          https://conda.anaconda.org/conda-forge/osx-arm64
                          https://conda.anaconda.org/bioconda/noarch
                          https://conda.anaconda.org/bioconda/osx-arm64
                          https://conda.anaconda.org/nodefaults/noarch
                          https://conda.anaconda.org/nodefaults/osx-arm64
       base environment : /Users/corneliusromer/micromamba
               platform : osx-arm64

environment.yml

$ micromamba list
List of packages in environment: "/Users/corneliusromer/micromamba/envs/py13"

  Name                Version            Build                 Channel
────────────────────────────────────────────────────────────────────────────
  appnope             0.1.4              pyhd8ed1ab_1          conda-forge
  asttokens           3.0.0              pyhd8ed1ab_1          conda-forge
  brotli-python       1.1.0              py313h3579c5c_2       conda-forge
  bzip2               1.0.8              h99b78c6_7            conda-forge
  ca-certificates     2024.12.14         hf0a4a13_0            conda-forge
  certifi             2024.12.14         pyhd8ed1ab_0          conda-forge
  cffi                1.17.1             py313hc845a76_0       conda-forge
  charset-normalizer  3.4.1              pyhd8ed1ab_0          conda-forge
  colorama            0.4.6              pyhd8ed1ab_1          conda-forge
  comm                0.2.2              pyhd8ed1ab_1          conda-forge
  debugpy             1.8.11             py313h928ef07_0       conda-forge
  decorator           5.1.1              pyhd8ed1ab_1          conda-forge
  dnspython           2.7.0              pyhff2d567_1          conda-forge
  exceptiongroup      1.2.2              pyhd8ed1ab_1          conda-forge
  executing           2.1.0              pyhd8ed1ab_1          conda-forge
  gmp                 6.3.0              h7bae524_2            conda-forge
  h2                  4.1.0              pyhd8ed1ab_1          conda-forge
  hpack               4.0.0              pyhd8ed1ab_1          conda-forge
  hyperframe          6.0.1              pyhd8ed1ab_1          conda-forge
  idna                3.10               pyhd8ed1ab_1          conda-forge
  importlib-metadata  8.5.0              pyha770c72_1          conda-forge
  ipykernel           6.29.5             pyh57ce528_0          conda-forge
  ipython             8.31.0             pyh707e725_0          conda-forge
  jedi                0.19.2             pyhd8ed1ab_1          conda-forge
  jupyter_client      8.6.3              pyhd8ed1ab_1          conda-forge
  jupyter_core        5.7.2              pyh31011fe_1          conda-forge
  krb5                1.21.3             h237132a_0            conda-forge
  libblas             3.9.0              26_osxarm64_openblas  conda-forge
  libcblas            3.9.0              26_osxarm64_openblas  conda-forge
  libcxx              19.1.6             ha82da77_1            conda-forge
  libedit             3.1.20240808       pl5321hafb1f1b_0      conda-forge
  libexpat            2.6.4              h286801f_0            conda-forge
  libffi              3.4.2              h3422bc3_5            conda-forge
  libgfortran         5.0.0              13_2_0_hd922786_3     conda-forge
  libgfortran5        13.2.0             hf226fd6_3            conda-forge
  liblapack           3.9.0              26_osxarm64_openblas  conda-forge
  liblzma             5.6.3              h39f12f2_1            conda-forge
  libmpdec            4.0.0              h99b78c6_0            conda-forge
  libopenblas         0.3.28             openmp_hf332438_1     conda-forge
  libsodium           1.0.20             h99b78c6_0            conda-forge
  libsqlite           3.47.2             h3f77e49_0            conda-forge
  libzlib             1.3.1              h8359307_2            conda-forge
  llvm-openmp         19.1.6             hdb05f8b_0            conda-forge
  markdown-it-py      3.0.0              pyhd8ed1ab_1          conda-forge
  matplotlib-inline   0.1.7              pyhd8ed1ab_1          conda-forge
  mdurl               0.1.2              pyhd8ed1ab_1          conda-forge
  ncbi-datasets-cli   16.39.0            hce30654_0            conda-forge
  ncurses             6.5                h7bae524_1            conda-forge
  nest-asyncio        1.6.0              pyhd8ed1ab_1          conda-forge
  numpy               2.2.1              py313ha4a2180_0       conda-forge
  openssl             3.4.0              h81ee809_1            conda-forge
  packaging           24.2               pyhd8ed1ab_2          conda-forge
  pandas              2.2.3              py313h47b39a6_1       conda-forge
  pandas-stubs        2.2.3.241126       pyhd8ed1ab_1          conda-forge
  pango_aliasor       0.3.0              pyhdfd78af_0          bioconda
  parso               0.8.4              pyhd8ed1ab_1          conda-forge
  pexpect             4.9.0              pyhd8ed1ab_1          conda-forge
  pickleshare         0.7.5              pyhd8ed1ab_1004       conda-forge
  pip                 24.3.1             pyh145f28c_2          conda-forge
  platformdirs        4.3.6              pyhd8ed1ab_1          conda-forge
  prompt-toolkit      3.0.48             pyha770c72_1          conda-forge
  psutil              6.1.1              py313h90d716c_0       conda-forge
  ptyprocess          0.7.0              pyhd8ed1ab_1          conda-forge
  pure_eval           0.2.3              pyhd8ed1ab_1          conda-forge
  pycparser           2.22               pyh29332c3_1          conda-forge
  pycryptodome        3.21.0             py313hba17e9d_0       conda-forge
  pygments            2.19.1             pyhd8ed1ab_0          conda-forge
  pysocks             1.7.1              pyha55dd90_7          conda-forge
  python              3.13.1             h4f43103_102_cp313    conda-forge
  python-dateutil     2.9.0.post0        pyhff2d567_1          conda-forge
  python-tzdata       2024.2             pyhd8ed1ab_1          conda-forge
  python_abi          3.13               5_cp313               conda-forge
  pytz                2024.1             pyhd8ed1ab_0          conda-forge
  pyyaml              6.0.2              py313h20a7fcf_1       conda-forge
  pyzmq               26.2.0             py313h0e8b002_3       conda-forge
  readline            8.2                h92ec313_1            conda-forge
  requests            2.32.3             pyhd8ed1ab_1          conda-forge
  rich                13.9.4             pyhd8ed1ab_1          conda-forge
  six                 1.17.0             pyhd8ed1ab_0          conda-forge
  sniffio             1.3.1              pyhd8ed1ab_1          conda-forge
  stack_data          0.6.3              pyhd8ed1ab_1          conda-forge
  tk                  8.6.13             h5083fa2_1            conda-forge
  tornado             6.4.2              py313h90d716c_0       conda-forge
  tqdm                4.67.1             pyhd8ed1ab_1          conda-forge
  traitlets           5.14.3             pyhd8ed1ab_1          conda-forge
  types-pytz          2024.2.0.20241221  pyhd8ed1ab_0          conda-forge
  types-requests      2.32.0.20241016    pyhd8ed1ab_1          conda-forge
  types-tqdm          4.67.0.20241221    pyhd8ed1ab_0          conda-forge
  typing_extensions   4.12.2             pyha770c72_1          conda-forge
  tzdata              2024b              hc8b5060_0            conda-forge
  urllib3             2.3.0              pyhd8ed1ab_0          conda-forge
  wcwidth             0.2.13             pyhd8ed1ab_1          conda-forge
  yaml                0.2.5              h3422bc3_2            conda-forge
  zeromq              4.3.5              hc1bb282_7            conda-forge
  zipp                3.21.0             pyhd8ed1ab_1          conda-forge
  zstandard           0.23.0             py313hf2da073_1       conda-forge
  zstd                1.5.6              hb46c0d2_0            conda-forge

~/.condarc

# .condarc
always_yes: true
show_channel_urls: true
solver: libmamba
changeps1: False
channels:
  - bioconda
  - conda-forge
  - nodefaults
channel_priority: strict

~/.mambarc

repodata_use_zst: true
channels:
- conda-forge
- bioconda
@jjerphan jjerphan added the type::bug Something isn't working label Jan 8, 2025
@jjerphan
Copy link
Member

jjerphan commented Jan 9, 2025

I cannot reproduce the problem (mailbox does not exist in any conda channels indexed by prefix.dev).

@jjerphan jjerphan added the status::cannot-reproduce The bug could not be reproduced label Jan 9, 2025
@corneliusroemer
Copy link
Contributor Author

I never thought it was deterministically reproducible, that's why the issue title starts with "stochastically" ;)

I don't think it matters what exactly you install, something is wrong with the index, it's likely corrupted but micromamba seems to not properly handle a corrupted index. That's the root cause for which we lack a repro - which doesn't mean that there's not a problem.

@jjerphan
Copy link
Member

jjerphan commented Jan 9, 2025

Yes, I know, I have tried many time with your configuration and I never have been able to reproduce the issue.

I guess we could add an hint to recommend cleaning the cache to users if they meet error repeatedly.

@corneliusroemer
Copy link
Contributor Author

corneliusroemer commented Jan 9, 2025 via email

@jjerphan
Copy link
Member

jjerphan commented Jan 9, 2025

Feel free to give feedback on #3736.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
status::cannot-reproduce The bug could not be reproduced type::bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants