[NO MRG] Run tests requiring sparse

[jakirkham]

Try to reproduce Numba + Arrow + LLVM + sparse issue: conda-forge/llvmlite-feedstock#72 (comment)

---

Checklist

• Used a personal fork of the feedstock to propose changes
• Bumped the build number (if the version is unchanged)
• Reset the build number to 0 (if the version changed)
• Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
• Ensured the license file is being packaged.

[conda-forge-webservices]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

[h-vetinari]

Good that you ended up retesting this! Seems the failures that I kept seeing over the last 1-2 years when I tried this on the arrow-cpp feedstock aren't there anymore.

I also checked that the skipped section does not contain what's visible on main, namely:

SKIPPED [11] pyarrow/tests/test_sparse_tensor.py:468: requires pydata/sparse

[jakirkham]

Am seeing a different error in the macOS x86_64 Python 3.12 CI job (also attached log for posterity):

cmake -DARROW_SIMD_LEVEL=NONE -DCMAKE_INSTALL_PREFIX=/Users/runner/miniforge3/conda-bld/apache-arrow_1715754977734/work/python/build/lib.macosx-10.9-x86_64-cpython-312/pyarrow -DPYTHON_EXECUTABLE=/Users/runner/miniforge3/conda-bld/apache-arrow_1715754977734/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/bin/python -DPython3_EXECUTABLE=/Users/runner/miniforge3/conda-bld/apache-arrow_1715754977734/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/bin/python -DPYARROW_CXXFLAGS= -G Ninja -DPYARROW_BUILD_CUDA=off -DPYARROW_BUILD_SUBSTRAIT=on -DPYARROW_BUILD_FLIGHT=on -DPYARROW_BUILD_GANDIVA=on -DPYARROW_BUILD_ACERO=on -DPYARROW_BUILD_DATASET=on -DPYARROW_BUILD_ORC=on -DPYARROW_BUILD_PARQUET=on -DPYARROW_BUILD_PARQUET_ENCRYPTION=on -DPYARROW_BUILD_AZURE=off -DPYARROW_BUILD_GCS=on -DPYARROW_BUILD_S3=on -DPYARROW_BUILD_HDFS=on -DPYARROW_BUNDLE_ARROW_CPP=off -DPYARROW_BUNDLE_CYTHON_CPP=off -DPYARROW_GENERATE_COVERAGE=off -DCMAKE_BUILD_TYPE=release /Users/runner/miniforge3/conda-bld/apache-arrow_1715754977734/work/python
dyld[21458]: Library not loaded: '@rpath/libzstd.1.dylib'
  Referenced from: '/Users/runner/miniforge3/pkgs/cmake-3.29.3-h1304840_0/bin/cmake'
  Reason: tried: '/Users/runner/miniforge3/pkgs/cmake-3.29.3-h1304840_0/bin/../lib/libzstd.1.dylib' (no such file), '/Users/runner/miniforge3/pkgs/cmake-3.29.3-h1304840_0/bin/../lib/libzstd.1.dylib' (no such file), '/usr/local/lib/libzstd.1.dylib' (no such file), '/usr/lib/libzstd.1.dylib' (no such file)
error: command '/Users/runner/miniforge3/conda-bld/apache-arrow_1715754977734/_build_env/bin/cmake' failed with exit code -6

This could just be a fluke given we are not seeing it other jobs. So probably worth restarting first

Though if this keeps happening, maybe we can try cmake-no-system to see if that helps

Edit: Restarted and it passed

[h-vetinari]

So the situation is reproducible with arrow v14 (segfaults if enabling sparse, passes if downgrading llvm_version to match sparse), while it seems not reproducible with arrow v15+

[jakirkham]

Thanks for investigating, Axel! 🙏

Was wondering if we would learn where the fix occurred

One thing that sticks out when looking at the diff between the branches 14.x and 15.x is the macOS compiler version went from Clang* 15 to 16

Is it possible to use Clang 16 with Arrow 14 builds? If so, do we still see the error in that case?

[h-vetinari]

Yeah, I'm testing that hypothesis (arrow v14 + clang 16; arrow v15 + clang 15). It could be that it's an interaction between llvm 14 & 15 only, i.e. it doesn't seem to affect later llvm versions for now.

That said, that this happens at all is an argument in favour of caution (and thus building against static libllvm), because we don't understand yet understand under what conditions the failure mode(s) can occur.

[h-vetinari]

Confirmed that llvm 15 is the one causing issues (note current sparse uses libllvm14):

arrow 👉 clang 👇	v14	v15
v14	✅	❔
v15	💥	💥
v16	✅	✅
v17	❔	✅