Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix conda_index.index verbose DEBUG/INFO logging #5066

Merged
merged 3 commits into from
Nov 17, 2023

Conversation

mbargull
Copy link
Member

@mbargull mbargull commented Nov 10, 2023

Description

Log messages from conda_index.index get put out with high verbosity.
As an example, I built https://github.com/conda-forge/zlib-feedstock/tree/main
which yielded a build log with 936 lines -- 260 of those lines (about 28%) were INFO/DEBUG messages from conda_index.index like these:

DEBUG:conda_index.index:found subdirs {'linux-64', 'noarch'}
DEBUG:conda_index.index.sqlitecache:CondaIndexCache channel_root=/opt/conda/conda-bld, subdir=linux-64 db_filename=/opt/conda/conda-bld/linux-64/.cache/cache.db cache_is_brand_new=False
DEBUG:conda_index.index.sqlitecache:linux-64 listdir
DEBUG:conda_index.index.sqlitecache:linux-64 save fs state
DEBUG:conda_index.index:linux-64 find packages to extract
DEBUG:conda_index.index:linux-64 extract 1 packages
DEBUG:conda_index.index.sqlitecache:cache linux-64/libzlib-1.2.13-hd590300_5.tar.bz2
DEBUG:conda_index.index.sqlitecache:libzlib-1.2.13-hd590300_5.tar.bz2 early close
INFO:conda_index.index:linux-64 cached 64 KB from 1 packages at 1.0 MB/second
DEBUG:conda_index.index.sqlitecache:CondaIndexCache channel_root=/opt/conda/conda-bld, subdir=noarch db_filename=/opt/conda/conda-bld/noarch/.cache/cache.db cache_is_brand_new=False
DEBUG:conda_index.index.sqlitecache:noarch listdir
DEBUG:conda_index.index.sqlitecache:noarch save fs state
DEBUG:conda_index.index:noarch find packages to extract
DEBUG:conda_index.index:noarch extract 0 packages
INFO:conda_index.index:noarch cached 0 B from 0 packages at 0 B/second
INFO:conda_index.index:Subdir: linux-64 Gathering repodata
DEBUG:conda_index.index.sqlitecache:CondaIndexCache channel_root=/opt/conda/conda-bld, subdir=linux-64 db_filename=/opt/conda/conda-bld/linux-64/.cache/cache.db cache_is_brand_new=False
INFO:conda_index.index:linux-64 Writing pre-patch repodata
DEBUG:conda_index.index:_maybe_write /opt/conda/conda-bld/linux-64/repodata_from_packages.json to /opt/conda/conda-bld/linux-64/repodata_from_packages.json
DEBUG:conda_index.index:_maybe_write /opt/conda/conda-bld/linux-64/repodata_from_packages.json.bz2 to /opt/conda/conda-bld/linux-64/repodata_from_packages.json.bz2
DEBUG:conda_index.index:_maybe_remove /opt/conda/conda-bld/linux-64/repodata_from_packages.json.zst from /opt/conda/conda-bld/linux-64/repodata_from_packages.json.zst
INFO:conda_index.index:linux-64 Applying patch instructions
INFO:conda_index.index:linux-64 Writing patched repodata
DEBUG:conda_index.index:linux-64 write patched repodata
DEBUG:conda_index.index:_maybe_write /opt/conda/conda-bld/linux-64/repodata.json to /opt/conda/conda-bld/linux-64/repodata.json
DEBUG:conda_index.index:_maybe_write /opt/conda/conda-bld/linux-64/repodata.json.bz2 to /opt/conda/conda-bld/linux-64/repodata.json.bz2
DEBUG:conda_index.index:_maybe_remove /opt/conda/conda-bld/linux-64/repodata.json.zst from /opt/conda/conda-bld/linux-64/repodata.json.zst
INFO:conda_index.index:linux-64 Building current_repodata subset
DEBUG:conda_index.index:linux-64 build current_repodata
INFO:conda_index.index:linux-64 Writing current_repodata subset
DEBUG:conda_index.index:linux-64 write current_repodata
DEBUG:conda_index.index:_maybe_write /opt/conda/conda-bld/linux-64/current_repodata.json to /opt/conda/conda-bld/linux-64/current_repodata.json
DEBUG:conda_index.index:_maybe_write /opt/conda/conda-bld/linux-64/current_repodata.json.bz2 to /opt/conda/conda-bld/linux-64/current_repodata.json.bz2
DEBUG:conda_index.index:_maybe_remove /opt/conda/conda-bld/linux-64/current_repodata.json.zst from /opt/conda/conda-bld/linux-64/current_repodata.json.zst
INFO:conda_index.index:linux-64 Writing index HTML
DEBUG:conda_index.index:linux-64 write index.html
DEBUG:conda_index.index:_maybe_write /opt/conda/conda-bld/linux-64/index.html to /opt/conda/conda-bld/linux-64/index.html
DEBUG:conda_index.index:linux-64 finish
INFO:conda_index.index:Completed linux-64
INFO:conda_index.index:Subdir: noarch Gathering repodata
DEBUG:conda_index.index.sqlitecache:CondaIndexCache channel_root=/opt/conda/conda-bld, subdir=noarch db_filename=/opt/conda/conda-bld/noarch/.cache/cache.db cache_is_brand_new=False
INFO:conda_index.index:noarch Writing pre-patch repodata
DEBUG:conda_index.index:_maybe_write /opt/conda/conda-bld/noarch/repodata_from_packages.json to /opt/conda/conda-bld/noarch/repodata_from_packages.json
INFO:conda_index.index:noarch Applying patch instructions
INFO:conda_index.index:noarch Writing patched repodata
DEBUG:conda_index.index:noarch write patched repodata
DEBUG:conda_index.index:_maybe_write /opt/conda/conda-bld/noarch/repodata.json to /opt/conda/conda-bld/noarch/repodata.json
INFO:conda_index.index:noarch Building current_repodata subset
DEBUG:conda_index.index:noarch build current_repodata
INFO:conda_index.index:noarch Writing current_repodata subset
DEBUG:conda_index.index:noarch write current_repodata
DEBUG:conda_index.index:_maybe_write /opt/conda/conda-bld/noarch/current_repodata.json to /opt/conda/conda-bld/noarch/current_repodata.json
INFO:conda_index.index:noarch Writing index HTML
DEBUG:conda_index.index:noarch write index.html
DEBUG:conda_index.index:_maybe_write /opt/conda/conda-bld/noarch/index.html to /opt/conda/conda-bld/noarch/index.html
DEBUG:conda_index.index:noarch finish
INFO:conda_index.index:Completed noarch
DEBUG:conda_index.index:found subdirs {'linux-64', 'noarch'}
INFO:conda_index.index:Channeldata subdir: linux-64
DEBUG:conda_index.index:linux-64 read repodata
DEBUG:conda_index.index.sqlitecache:CondaIndexCache channel_root=/opt/conda/conda-bld, subdir=linux-64 db_filename=/opt/conda/conda-bld/linux-64/.cache/cache.db cache_is_brand_new=False
DEBUG:conda_index.index:linux-64 channeldata finished
INFO:conda_index.index:Channeldata subdir: noarch
DEBUG:conda_index.index:noarch read repodata
DEBUG:conda_index.index.sqlitecache:CondaIndexCache channel_root=/opt/conda/conda-bld, subdir=noarch db_filename=/opt/conda/conda-bld/noarch/.cache/cache.db cache_is_brand_new=False
DEBUG:conda_index.index:noarch channeldata finished
DEBUG:conda_index.index:_maybe_write /opt/conda/conda-bld/index.html to /opt/conda/conda-bld/index.html
DEBUG:conda_index.index:write channeldata
DEBUG:conda_index.index:_maybe_write /opt/conda/conda-bld/channeldata.json to /opt/conda/conda-bld/channeldata.json

closes: gh-4939

Checklist - did you ...

  • Add a file to the news directory (using the template) for the next release's release notes?
  • Add / update necessary tests?
  • Add / update outdated documentation?

@conda-bot conda-bot added the cla-signed [bot] added once the contributor has signed the CLA label Nov 10, 2023
@mbargull mbargull force-pushed the fix-conda_index-log-spam branch from c503ac5 to 9f0ed50 Compare November 10, 2023 20:05
@mbargull
Copy link
Member Author

Added a "closing" keyword to the OP for gh-4939.
Thanks @JeanChristopheMorinPerso for mentioning that issue in the Conda Build Tools chat.

@mbargull
Copy link
Member Author

@kenodegard, @jezdez, since this is a fairly focused change, I'd like to add this to the 3.28.0 milestone, if you don't mind.

@mbargull mbargull modified the milestone: 3.28.0 Nov 16, 2023
@mbargull
Copy link
Member Author

@kenodegard, thanks for updating and re-running the tests.


For posterity: One test failed on the first attempt:

=================================== FAILURES ===================================
_____________________________ test_no_force_upload _____________________________
[gw0] linux -- Python 3.9.18 /usr/share/miniconda/envs/test/bin/python
Traceback (most recent call last):
  File "/usr/share/miniconda/envs/test/lib/python3.9/site-packages/_pytest/runner.py", line 341, in from_call
    result: Optional[TResult] = func()
  File "/usr/share/miniconda/envs/test/lib/python3.9/site-packages/_pytest/runner.py", line 262, in <lambda>
    lambda: ihook(item=item, **kwds), when=when, reraise=reraise
  File "/usr/share/miniconda/envs/test/lib/python3.9/site-packages/pluggy/_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "/usr/share/miniconda/envs/test/lib/python3.9/site-packages/pluggy/_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/usr/share/miniconda/envs/test/lib/python3.9/site-packages/pluggy/_callers.py", line 60, in _multicall
    return outcome.get_result()
  File "/usr/share/miniconda/envs/test/lib/python3.9/site-packages/pluggy/_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/usr/share/miniconda/envs/test/lib/python3.9/site-packages/pluggy/_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "/usr/share/miniconda/envs/test/lib/python3.9/site-packages/_pytest/runner.py", line 177, in pytest_runtest_call
    raise e
  File "/usr/share/miniconda/envs/test/lib/python3.9/site-packages/_pytest/runner.py", line 169, in pytest_runtest_call
    item.runtest()
  File "/usr/share/miniconda/envs/test/lib/python3.9/site-packages/_pytest/python.py", line 1788, in runtest
    self.ihook.pytest_pyfunc_call(pyfuncitem=self)
  File "/usr/share/miniconda/envs/test/lib/python3.9/site-packages/pluggy/_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "/usr/share/miniconda/envs/test/lib/python3.9/site-packages/pluggy/_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/usr/share/miniconda/envs/test/lib/python3.9/site-packages/pluggy/_callers.py", line 60, in _multicall
    return outcome.get_result()
  File "/usr/share/miniconda/envs/test/lib/python3.9/site-packages/pluggy/_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/usr/share/miniconda/envs/test/lib/python3.9/site-packages/pluggy/_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "/usr/share/miniconda/envs/test/lib/python3.9/site-packages/_pytest/python.py", line 194, in pytest_pyfunc_call
    result = testfunction(**testargs)
  File "/home/runner/work/conda-build/conda-build/tests/cli/test_main_build.py", line 288, in test_no_force_upload
    main_build.execute(args)
  File "/home/runner/work/conda-build/conda-build/conda_build/cli/main_build.py", line 572, in execute
    outputs = api.build(
  File "/home/runner/work/conda-build/conda-build/conda_build/api.py", line 254, in build
    return build_tree(
  File "/home/runner/work/conda-build/conda-build/conda_build/build.py", line 3789, in build_tree
    packages_from_this = build(
  File "/home/runner/work/conda-build/conda-build/conda_build/build.py", line 2877, in build
    newly_built_packages = bundlers[pkg_type](output_d, m, env, stats)
  File "/home/runner/work/conda-build/conda-build/conda_build/build.py", line 2002, in bundle_conda
    files = post_process_files(metadata, initial_files)
  File "/home/runner/work/conda-build/conda-build/conda_build/build.py", line 1776, in post_process_files
    post_process(
  File "/home/runner/work/conda-build/conda-build/conda_build/post.py", line 363, in post_process
    rm_py_along_so(prefix)
  File "/home/runner/work/conda-build/conda-build/conda_build/post.py", line 229, in rm_py_along_so
    files = list(os.scandir(prefix))
FileNotFoundError: [Errno 2] No such file or directory: '/usr/share/miniconda/envs/test/conda-bld/test_no_force_upload_1700245007728/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl'
which indicates that gh-5068 was still not enough (the test build shouldn't run under `/usr/share/miniconda/envs/test/conda-bld` but a temp directory). 😞

@kenodegard kenodegard merged commit eb5ecc0 into conda:main Nov 17, 2023
24 checks passed
@kenodegard kenodegard mentioned this pull request Dec 1, 2023
67 tasks
@github-actions github-actions bot added the locked [bot] locked due to inactivity label Nov 17, 2024
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Nov 17, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
cla-signed [bot] added once the contributor has signed the CLA locked [bot] locked due to inactivity
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

conda_index.index debug logs in conda-build output
3 participants