Pixi GitHub CI failures

[ruben-arts]

We’re currently dealing with intermittent CI failures and could use your help to diagnose and resolve these issues. Below are the main problems we’ve observed:

MacOS timeout EDIT: [FIXED]

There are two types of timeout we're seeing:

• The pytest 300 second timeout.

FAILED tests/integration_python/test_run_cli.py::test_run_in_shell_environment - Failed: Timeout >300.0s

This is not always happening with the same test, so it's not reproducible.

• The GitHub Runner 30minute timeout.

We're looking into the macOS file handling, but since none of our developers have had this issue locally, when running tests or managing environments, we're currently chasing a ghost on CI.

✅ Solution: We've had issues with the BarrierCell which could get a race condition when solving an empty environment, this was fixed in #2736

Repodata fetching from prefix.dev giving an error

The common result in CI would be looking like this:

Failed to resolve dependencies: error sending request for url (https://prefix.dev/conda-forge/noarch/repodata_shards.msgpack.zst)
    Diagnostic severity: error
    Caused by: An existing connection was forcibly closed by the remote host. (os error 10054)
    Caused by: client error (Connect)
    Caused by: error sending request for url (https://prefix.dev/conda-forge/noarch/repodata_shards.msgpack.zst)

We’re actively monitoring the server-side issues and working with Cloudflare for more insights. We’re also looking for improved retry behavior in rattler’s repodata fetching client.

Repodata fetching from conda.anaconda.org giving an error

The common result in CI would be looking like this:

FAILED tests/integration_python/test_main_cli.py::test_upgrade_dependency_location_pixi - AssertionError: Return code was 1, expected 0, stderr:  WARN The feature 'cli' is defined but not used in any environment
 INFO waiting for file lock on repodata cache
ERROR get_or_create_subdir{platform=LinuxPpc64le channel=[https://conda.anaconda.org/conda-forge/}:fetch_repo_data{cache_path=/home/runner/work/pixi/pixi/.cache/rattler/cache/repodata}:](https://conda.anaconda.org/conda-forge/%7D:fetch_repo_data%7Bcache_path=/home/runner/work/pixi/pixi/.cache/rattler/cache/repodata%7D:) error=failed to download https://conda.anaconda.org/conda-forge/linux-ppc64le/repodata.json.zst
ERROR get_or_create_subdir{platform=LinuxPpc64le channel=[https://conda.anaconda.org/conda-forge/}:](https://conda.anaconda.org/conda-forge/%7D:) error=failed to download https://conda.anaconda.org/conda-forge/linux-ppc64le/repodata.json.zst
failed to download https://conda.anaconda.org/conda-forge/linux-ppc64le/repodata.json.zst
    Diagnostic severity: error
    Caused by: error decoding response body
    Caused by: request or response body error
    Caused by: error reading a body from connection
    Caused by: stream error received: unexpected internal error encountered

We're looking into improved retry behavior and decoding of the data.

Call for data

If you’ve encountered similar issues in the last month, please share examples or logs. We’re especially interested in ideas for testing or debugging approaches we might have overlooked.

[hameerabbasi]

You might want to look into this GitHub action which will let you remote into the runners to figure out what's up.