Remaing CI test concurrency basic issues

[mbargull]

Checklist

• I added a descriptive title
• I searched open reports and couldn't find a duplicate

What happened?

conda-build's CI tests are still a bit flaky.
gh-5068 and gh-5076 lessened the chance of failures, but there are still some underlying issues:

The serial tests are actually not run in series anymore

• Happend since https://github.com/conda/conda-build/pull/4648/files#diff-1db27d93186e46d3b441ece35801b244db8ee144ff1405ca27a163bfe878957fR82
• Reason is that there is a xxx && 0 || 'auto' which will always evaluate to auto.

I did not change/fix that in the recent pull requests (since tests run somewhat fine and otherwise it's a waste of resources).
We actually should run most of those tests in parallel since the reasons they were marked as serial are most likely connected to reasons handled by the pull requests above.
There may be some tests that should actually be run in series -- we might want to run them separately (or at least within a single worker, e.g., via --dist loadgroup.

The tests (and conda-build in general) use the same global package cache

With the recent changes, the tests should now use individual working directories, but they still use the same shared package cache.
When the tests are run concurrently, this means they can corrupt indexes (or possibly packages when different channels/subdirs are used).

Ideally we want the package cache separated for the tests too.
However, this could increase resource usage significantly since the same packages would be downloaded/extracted multiple times to different caches then.
Hence, we might want to do this selectively (I haven't thought out a plan for this yet, though).

Another problem related to the shared package cache is that conda-build will add the packages it built to the package cache when it builds subpackage builds or runs tests (because that's how conda works: to create an environment, the packages are put into pkgs_dirs beforehand).
For the CI tests, this means that packages with the same name+version+build will get clobbered, e.g., see this recent workaround for such a case.
We may want to change that behavior overall -- not just for the CI tests -- and let conda-build put its own builds into a separate pkgs directory relative to its working directory.

There's also the issue with collisions in the cache from different channel/subdirs in general, but that one should be handled on a case-by-case basis for the tests (and/or ideally upstream in conda by appending a content-dependent hash to the archives/folders, e.g., pkgs/package-version-build-<additional hash>).

Conda Info

No response

Conda Config

No response

Conda list

No response

Additional Context

No response

[github-actions]

Hi there, thank you for your contribution!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed automatically if no further activity occurs.

If you would like this issue to remain open please:

1. Verify that you can still reproduce the issue at hand
2. Comment that the issue is still reproducible and include:
   - What OS and version you reproduced the issue on
   - What steps you followed to reproduce the issue

NOTE: If this issue was closed prematurely, please leave a comment.

Thanks!