General slowness observed on anaconda.org (web, API, file serving) #899

mbargull · 2024-03-27T15:23:13Z

Checklist

I added a descriptive title
I searched open reports and couldn't find a duplicate

What happened?

This is a very vague issue, but I generally noticed very slow response times for anaconda.org as of late

on the web interface,
via API (this one I can pin down to happening at least from 2024-03-22 03:00+00 onward),
for non CDN-backed channels (i.e., not conda-forge/bioconda), e.g., conda-forge/label/... channels.

For the last part, one of latest evidences would be:

Download error (28) Timeout was reached [https://conda.anaconda.org/conda-forge/label/lief_rc/noarch/repodata.json.zst]
Operation too slow. Less than 30 bytes/sec transferred the last 60 seconds

observed at https://dev.azure.com/conda-forge/84710dde-1620-425b-80d0-4cf5baca359d/_build/results?buildId=904323&view=logs&j=a70f640f-cc53-5cd3-6cdc-236a1aa90802

Conda Info

No response

Conda Config

No response

Conda list

No response

Additional Context

No response

The text was updated successfully, but these errors were encountered:

mbargull · 2024-03-27T15:42:38Z

I just got a plain

upstream connect error or disconnect/reset before headers. reset reason: connection termination

as a response to opening https://anaconda.org/main/repo in the web browser.
After that, it took a minute or so to load for cases where it was successful.
I also got a 520 once

Web server is returning an unknown error Error code 520
Visit [cloudflare.com](https://www.cloudflare.com/5xx-error-landing?utm_source=errorcode_520&utm_campaign=anaconda.org) for more information.
2024-03-27 15:40:14 UTC

chenghlee · 2024-03-27T15:46:03Z

Can you try this for some channel not conda-forge, bioconda, or Anaconda? There's some Cloudflare wizardry that happens for larger channels that splits out requests that are bound for CDN (e.g., main label) and requests that are not (e.g., other labels).

mbargull · 2024-03-27T15:50:29Z

Do you happen to know any non-CDN-backed channel that is somewhat large?
To me, it seems like some server load issues are happening since very small requests/channels seem to work fine.

chenghlee · 2024-03-27T15:54:37Z

Can you try https://anaconda.org/LiteX-Hub/?

mbargull · 2024-03-27T16:09:53Z

Got a

upstream connect error or disconnect/reset before headers. reset reason: connection termination

right away for https://anaconda.org/LiteX-Hub/ .

mbargull · 2024-03-27T16:15:42Z

Downloading https://anaconda.org/LiteX-Hub/ via curl which took about 31 seconds but worked.
And after that it is cached on the CDN for me an as such seems to be served reliably and faster (can still take up to 10 seconds, though).

mbargull · 2024-03-27T16:17:51Z

Just happened incidentally:

# conda-smithy rerender
Traceback (most recent call last):
  File "/home/mbargull/code/conda/conda-forge/envs/conda-smithy/bin/conda-smithy", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/mbargull/code/conda/conda-forge/envs/conda-smithy/lib/python3.11/site-packages/conda_smithy/cli.py", line 737, in main
    args.subcommand_func(args)
  File "/home/mbargull/code/conda/conda-forge/envs/conda-smithy/lib/python3.11/site-packages/conda_smithy/cli.py", line 584, in __call__
    self._call(args, tmpdir)
  File "/home/mbargull/code/conda/conda-forge/envs/conda-smithy/lib/python3.11/site-packages/conda_smithy/cli.py", line 589, in _call
    configure_feedstock.main(
  File "/home/mbargull/code/conda/conda-forge/envs/conda-smithy/lib/python3.11/site-packages/conda_smithy/configure_feedstock.py", line 2602, in main
    check_version_uptodate("conda-smithy", __version__, True)
  File "/home/mbargull/code/conda/conda-forge/envs/conda-smithy/lib/python3.11/site-packages/conda_smithy/configure_feedstock.py", line 2310, in check_version_uptodate
    most_recent_version = get_most_recent_version(name).version
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mbargull/code/conda/conda-forge/envs/conda-smithy/lib/python3.11/site-packages/conda_smithy/configure_feedstock.py", line 2299, in get_most_recent_version
    request.raise_for_status()
  File "/home/mbargull/code/conda/conda-forge/envs/conda-smithy/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 524 Server Error:  for url: https://api.anaconda.org/package/conda-forge/conda-smithy

chenghlee · 2024-03-27T16:20:43Z

Anaconda's infrastructure team has noticed a scaling issue with the .org back-end; that might be related, and they're currently working on resolving that issue. I'll update this ticket once they're done deploying the changes, and we can recheck to see if that fixes the issue.

mbargull · 2024-03-27T16:24:37Z

Thanks keeping an eye out and thanks to the infra team for working on it! 🛠️

rymndhng · 2024-03-27T16:59:20Z

Can you try this for some channel not conda-forge, bioconda, or Anaconda? There's some Cloudflare wizardry that happens for larger channels that splits out requests that are bound for CDN (e.g., main label) and requests that are not (e.g., other labels).

As a datapoint, one of my local tests on the comet-ml channel had this issue as well:

❯  curl -I -X GET https://conda.anaconda.org/comet_ml/noarch/repodata.json
HTTP/2 503 
date: Wed, 27 Mar 2024 16:51:42 GMT
content-type: text/html; charset=utf-8
cf-ray: 86b0e172de228417-YVR
cf-cache-status: DYNAMIC
cache-control: no-cache, max-age=0
content-disposition: inline; filename=db_connection_failure.html
expires: Wed, 27 Mar 2024 16:51:42 GMT
last-modified: Tue, 27 Feb 2024 15:49:08 GMT
x-envoy-upstream-service-time: 63125
set-cookie: __cf_bm=XrShwBedMNXr6JIgiG4V4_a.E8uNRzG76qurvQ8rqA0-1711558302-1.0.1.1-byOHt3W7jRO7wPo0lClotR.iFh8F5YVPHUG6kqWbN4Fm09VSFkW34lffnqjQ.aSJL7_BfpEr.RbKM_0UKXc8ueGpuWEqdok1gifLj8pYQWY; path=/; expires=Wed, 27-Mar-24 17:21:42 GMT; domain=.anaconda.org; HttpOnly; Secure; SameSite=None
server: cloudflare

mariusvniekerk · 2024-03-28T22:20:32Z

This still seems to be happening sporadically with the rapidsai channel at the very least when using mamba

info     libmamba Transfer done for 'rapidsai/linux-64'
info     libmamba Transfer finalized, status: 200 [https://conda.anaconda.org/rapidsai/linux-64/repodata.json] 263365 bytes
info     libmamba Transfer done for 'conda-forge/noarch'
info     libmamba Transfer finalized, status: 200 [https://conda.anaconda.org/conda-forge/noarch/repodata.json] 16311864 bytes
info     libmamba Transfer done for 'conda-forge/linux-64'
info     libmamba Transfer finalized, status: 200 [https://conda.anaconda.org/conda-forge/linux-64/repodata.json] 39168964 bytes
info     libmamba Download error (28) Timeout was reached [https://conda.anaconda.org/rapidsai/noarch/repodata.json]
    Operation too slow. Less than 30 bytes/sec transferred the last 60 seconds
Download error (28) Timeout was reached [https://conda.anaconda.org/rapidsai/noarch/repodata.json]
Operation too slow. Less than 30 bytes/sec transferred the last 60 seconds

# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<

    Traceback (most recent call last):
      File "/opt/mambaforge/lib/python3.10/site-packages/conda/exceptions.py", line 1132, in __call__
        return func(*args, **kwargs)
      File "/opt/mambaforge/lib/python3.10/site-packages/mamba/mamba.py", line 941, in exception_converter
        raise e
      File "/opt/mambaforge/lib/python3.10/site-packages/mamba/mamba.py", line 934, in exception_converter
        exit_code = _wrapped_main(*args, **kwargs)
      File "/opt/mambaforge/lib/python3.10/site-packages/mamba/mamba.py", line 892, in _wrapped_main
        result = do_call(parsed_args, p)
      File "/opt/mambaforge/lib/python3.10/site-packages/mamba/mamba.py", line 758, in do_call
        exit_code = create(args, parser)
      File "/opt/mambaforge/lib/python3.10/site-packages/mamba/mamba.py", line 632, in create
        return install(args, parser, "create")
      File "/opt/mambaforge/lib/python3.10/site-packages/mamba/mamba.py", line 499, in install
        index = load_channels(pool, channels, repos)
      File "/opt/mambaforge/lib/python3.10/site-packages/mamba/utils.py", line 129, in load_channels
        index = get_index(
      File "/opt/mambaforge/lib/python3.10/site-packages/mamba/utils.py", line 110, in get_index
        is_downloaded = dlist.download(api.MAMBA_DOWNLOAD_FAILFAST)
    RuntimeError: Download error (28) Timeout was reached [https://conda.anaconda.org/rapidsai/noarch/repodata.json]
    Operation too slow. Less than 30 bytes/sec transferred the last 60 seconds

jakirkham · 2024-05-29T04:40:00Z

At least in the RAPIDS case, we managed to fix issues in ( #906 )

mbargull · 2024-06-02T22:59:56Z

In the lasts weeks I've only noticed some sporadic failures, again, with (Cloudflare-specific) 520 or 524 HTTP status codes.
But not as prevalent as in March/April.
Let's close this for now.
Thanks for working on stabilizing and generally keeping things going!

mbargull added the type::bug describes erroneous operation, use severity::* to classify the type label Mar 27, 2024

conda-bot added this to 🧭 Planning Mar 27, 2024

github-project-automation bot moved this to 🆕 New in 🧭 Planning Mar 27, 2024

jakirkham mentioned this issue Apr 2, 2024

Seeing timeouts with https://conda.anaconda.org/rapidsai-nightly #906

Open

2 tasks

chenghlee mentioned this issue Apr 10, 2024

Slow CDN mirroring #892

Closed

traversaro mentioned this issue Apr 11, 2024

Remove robotology channel from CI robotology/yarp-matlab-bindings#105

Merged

mbargull closed this as completed Jun 2, 2024

github-project-automation bot moved this from 🆕 New to 🏁 Done in 🧭 Planning Jun 2, 2024

github-actions bot added the locked [bot] locked due to inactivity label Nov 30, 2024

github-actions bot locked as resolved and limited conversation to collaborators Nov 30, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

General slowness observed on anaconda.org (web, API, file serving) #899

General slowness observed on anaconda.org (web, API, file serving) #899

mbargull commented Mar 27, 2024

mbargull commented Mar 27, 2024

chenghlee commented Mar 27, 2024

mbargull commented Mar 27, 2024

chenghlee commented Mar 27, 2024

mbargull commented Mar 27, 2024

mbargull commented Mar 27, 2024

mbargull commented Mar 27, 2024

chenghlee commented Mar 27, 2024

mbargull commented Mar 27, 2024

rymndhng commented Mar 27, 2024

mariusvniekerk commented Mar 28, 2024

jakirkham commented May 29, 2024

mbargull commented Jun 2, 2024

General slowness observed on anaconda.org (web, API, file serving) #899

General slowness observed on anaconda.org (web, API, file serving) #899

Comments

mbargull commented Mar 27, 2024

Checklist

What happened?

Conda Info

Conda Config

Conda list

Additional Context

mbargull commented Mar 27, 2024

chenghlee commented Mar 27, 2024

mbargull commented Mar 27, 2024

chenghlee commented Mar 27, 2024

mbargull commented Mar 27, 2024

mbargull commented Mar 27, 2024

mbargull commented Mar 27, 2024

chenghlee commented Mar 27, 2024

mbargull commented Mar 27, 2024

rymndhng commented Mar 27, 2024

mariusvniekerk commented Mar 28, 2024

jakirkham commented May 29, 2024

mbargull commented Jun 2, 2024