Support for case where number of stored elements in one of the matrices is greater than the range of int32

[DeepakSaini119]

Hi, I want to do a dot product of two sparse matrices and one of the matrix has ~4bil stored elements. I get this error:

MKL interface is <class 'numpy.int32'> and cannot hold matrix <47494534x2619400 sparse matrix of type '<class 'numpy.float32'>'
	with 4328780604 stored elements in Compressed Sparse Row format

It seems to me it is because the number of stored elements are out of the range of int32. Am i correct? And if so, is there a way to do this using sparse_dot_mkl. TIA.

[asistradition]

Yes, that's correct - the LP64 interface for MKL is int32 and can't deal with more than 2^31 − 1 elements in the matrix.

The good news is that there's an ILP64 interface for MKL that uses int64 and won't have this problem. The bad news is that I was never able to get it to work - there's a function and an environment variable that can be used to change the interface but neither worked for me. If you can switch the interface everything should just work, but I don't know how to do that.

If I get time I'll look into linking directly to the ilp64 libraries instead of trying to go through the main interface, but that's enough work that I haven't done it yet. Sorry.

[asistradition]

I looked into this a little more - the MKL that's distributed with anaconda doesn't appear to support the int64 interface and I don't know why. I installed the intel channel mkl and it worked fine.

conda install -c intel mkl

$ export MKL_INTERFACE_LAYER=ILP64
$ python
Python 3.8.3 (default, May 19 2020, 18:47:26) 
>>> from sparse_dot_mkl._mkl_interface import MKL
>>> MKL.MKL_INT_NUMPY
<class 'numpy.int64'>
>>> quit()

$ export MKL_INTERFACE_LAYER=LP64
$ python
Python 3.8.3 (default, May 19 2020, 18:47:26) 
>>> from sparse_dot_mkl._mkl_interface import MKL
>>> MKL.MKL_INT_NUMPY
<class 'numpy.int32'>
>>> 

I think this should resolve your problem.

[chuckzzzz]

Just wanna throw a quick comment.

If everything fails and you cannot override the 2^31-1 limit maximum elements, you could just breakup matrix multiplication into quadrants to kinda go around it. This was what I did with a 90000X90000 matrix. Clearly this would sacrifice runtime, but I think it's worth it.

p.s.: This package is very useful so THANK YOU!

[ivirshup]

I looked into this a little more - the MKL that's distributed with anaconda doesn't appear to support the int64 interface and I don't know why.

I believe that this works now. I've tried the intel, conda-forge, and defaults channel and all seem to support 64 bit integers.

[yCobanoglu]

It does not work. Don't use conda if you want to use MKL_INTERFACE_LAYER=ILP64. The env variable has no effect according to the code snippet @asistradition.

[pphili]

I am also having difficulty with this issue. Is there a supported way of converting the indices of a csr array to int64 without using intel mkl? In particular, I try

import numpy as np
from scipy.sparse import csr_matrix
import os

os.environ['MKL_INTERFACE_LAYER'] = 'ILP64'

# Example data
data = np.array([1, 2, 3], dtype=np.int32)
indices = np.array([0, 1, 2], dtype=np.int64)  # Explicitly int64 indices
indptr = np.array([0, 1, 2, 3], dtype=np.int32)

# Create CSR matrix
csr = csr_matrix((data, indices, indptr), shape=(3, 3), dtype=np.int32)

# Check the index type
print(csr.indices.dtype)  # Should output int64

The output is
int32

I ran into this issue because I am getting the following error:
ValueError: mkl_sparse_spmm returned 2 (SPARSE_STATUS_ALLOC_FAILED); Try changing MKL to int64 with the environment variable MKL_INTERFACE_LAYER=ILP64

[asistradition]

The MKL interface is only for the MKL libraries. It doesn't change anything about what scipy is doing. The int32 indices will get copied to promote them to int64 when you call a MKL function.

You have to set the MKL_INTERFACE_LAYER environment variable before you import sparse_dot for the first time; it's runtime linked to the MKL library. If the int64 interface isn't available it should give you a runtime warning when you import it.

[pphili]

I am indeed getting this runtime error. I however do not get it when installing mkl via
conda install -c https://software.repos.intel.com/python/conda/ mkl-service as recommended here: https://www.intel.com/content/www/us/en/docs/oneapi/installation-guide-linux/2024-2/conda.html.

[asistradition]

import os

os.environ['MKL_INTERFACE_LAYER'] = 'ILP64'

from sparse_dot_mkl import mkl_interface_integer_dtype

print(mkl_interface_integer_dtype())

This should print np.int64 if the 64-bit interface has been linked.

If you call dot_product_mkl and still get ValueError: mkl_sparse_spmm returned 2 (SPARSE_STATUS_ALLOC_FAILED) it would be helpful to have a full traceback and the size/dtype/sparsity of your arrays.

[pphili]

Thanks for the help.
Unfortunately the output is <class 'numpy.int32'>.
I also get the following warning for from sparse_dot_mkl import mkl_interface_integer_dtype:
/home/usr/miniconda3/envs/test_env/lib/python3.10/site-packages/sparse_dot_mkl/_mkl_interface/__init__.py:137: RuntimeWarning: MKL_INTERFACE_LAYER=ILP64 failed to set MKL interface; 64-bit integer support unavailable _warnings.warn(

However, if I use conda install -c https://software.repos.intel.com/python/conda/ mkl-service, everything works as expected.

Extra question:
My problem could be resolved if I could add the equivalent of conda install -c https://software.repos.intel.com/python/conda/ mkl-service to a .yml to automatically generate my environment with mkl installed from Intel. Is there a standard way of doing this? Adding https://software.repos.intel.com/python/conda/ to the channels leads to

Channels:
 - https://software.repos.intel.com/python/conda
 - defaults
 - conda-forge
 - python/conda
Platform: linux-64
Collecting package metadata (repodata.json): failed

UnavailableInvalidChannel: HTTP 404 NOT FOUND for channel python/conda <https://conda.anaconda.org/python/conda>

The channel is not accessible or is invalid.

You will need to adjust your conda configuration to proceed.
Use `conda config --show channels` to view your configuration's current state,
and use `conda config --show-sources` to view config file locations.

Is there a standard way of installing mkl (or mkl-service) from https://software.repos.intel.com/python/conda using a .yml file?

[asistradition]

You can try installing mkl from pip as well (python -m pip install mkl). My current test environment has mkl 2024.2.1 installed through pypi and that's fine.

Unfortunately, it looks like intel is no longer supporting an anadonda channel (likely due to the same licensing changes that have forced me to remove it from most of my work). I doubt I can give you any more help on that than intel can.

[pphili]

Thanks so much for all you help! I really appreciate it!