Incorrect INSTALL_RPATH on cython extension files

[manopapad]

Thanks to @csadorf for reporting this.

The RPATH set on Cython extension modules on a non-editable installation is not appropriate for the location where those modules are stored. For example:

(non-editable) computelab-frontend-1:/home/scratch.mpapadakis_sw/miniconda3/envs/non-editable> objdump -p ./lib/python3.9/site-packages/legate/core/_lib/types.cpython-39-x86_64-linux-gnu.so
...
  RUNPATH              /home/scratch.svc_compute_arch/release/mpi/openmpi/v4.1.4-ucx-1.13.1-cuda11.5/lib:$ORIGIN:$ORIGIN/../../../build/lib

This is causing issues when using the canonical cpython interpreter, which (unlike legate) doesn't prime the library load path, so the RPATHs need to contain all relevant paths.

$ python
Python 3.10.12 | packaged by conda-forge | (main, Jun 23 2023, 22:40:32) [GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import legate.core
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/conda/envs/legate/lib/python3.10/site-packages/legate/core/__init__.py", line 39, in <module>
    from ._legion import (
  File "/opt/conda/envs/legate/lib/python3.10/site-packages/legate/core/_legion/__init__.py", line 21, in <module>
    from .operation import (
  File "/opt/conda/envs/legate/lib/python3.10/site-packages/legate/core/_legion/operation.py", line 22, in <module>
    from .partition import Partition
  File "/opt/conda/envs/legate/lib/python3.10/site-packages/legate/core/_legion/partition.py", line 22, in <module>
    from .space import IndexSpace
  File "/opt/conda/envs/legate/lib/python3.10/site-packages/legate/core/_legion/space.py", line 20, in <module>
    from ..types import Dtype
  File "/opt/conda/envs/legate/lib/python3.10/site-packages/legate/core/types.py", line 19, in <module>
    import legate.core._lib.types as ext  # type: ignore[import]
ImportError: liblgcore.so: cannot open shared object file: No such file or directory

In order for this extension to be able to find other required libraries, the RPATH should be pointing back to the containing lib/ folder, i.e. for this module it should contain something like $ORIGIN/../../../../../. Other modules, stored under different paths, might require a different RPATH.

[csadorf]

Thanks for creating the issue!

[manopapad]

This will probably be fixed with #725

[vyasr]

The libraries in the system lib directory (or envs/$ENV/lib in conda) aren't really meant to be handled via RPATHs since those directories should be in the search path for the linker automatically. I'm not sure what you mean by:

This is causing issues when using the canonical cpython interpreter, which (unlike legate) doesn't prime the library load path, so the RPATHs need to contain all relevant paths.

IIUC, what's happening above is a consequence of you being inside a conda environment but using the system linker. Inside a conda environment you probably want to be using the conda compilers and linkers, which will be compiled with the necessary additional search paths to use inside a conda environment.

[manopapad]

The libraries in the system lib directory (or envs/$ENV/lib in conda) aren't really meant to be handled via RPATHs since those directories should be in the search path for the linker automatically.

This is true for the system lib directory, but conda does not force $CONDA_PREFIX/lib to be added to the default dynamic linker search paths. Therefore, AFAIK PRATH is the only mechanism that is available to conda for ensuring that binaries and libraries will find their dependencies under $CONDA_PREFIX/lib. AFAICT all binaries under $CONDA_PREFIX/bin include $ORIGIN/../lib in their RPATH, and all libraries under $CONDA_PREFIX/lib include $ORIGIN/..

I'm not sure what you mean by: [...]

Here's what happens when I try to import legate.core using the canonical python interpreter:

(noneditable) sean-dgx2:~> python
Python 3.9.16 | packaged by conda-forge | (main, Feb  1 2023, 21:39:03)
[GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import legate.core
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/scratch.mpapadakis_sw/sean-dgx2/noneditable/env/lib/python3.9/site-packages/legate/core/__init__.py", line 39, in <module>
    from ._legion import (
  File "/home/scratch.mpapadakis_sw/sean-dgx2/noneditable/env/lib/python3.9/site-packages/legate/core/_legion/__init__.py", line 21, in <module>
    from .operation import (
  File "/home/scratch.mpapadakis_sw/sean-dgx2/noneditable/env/lib/python3.9/site-packages/legate/core/_legion/operation.py", line 22, in <module>
    from .partition import Partition
  File "/home/scratch.mpapadakis_sw/sean-dgx2/noneditable/env/lib/python3.9/site-packages/legate/core/_legion/partition.py", line 22, in <module>
    from .space import IndexSpace
  File "/home/scratch.mpapadakis_sw/sean-dgx2/noneditable/env/lib/python3.9/site-packages/legate/core/_legion/space.py", line 20, in <module>
    from ..types import Dtype
  File "/home/scratch.mpapadakis_sw/sean-dgx2/noneditable/env/lib/python3.9/site-packages/legate/core/types.py", line 19, in <module>
    import legate.core._lib.types as ext  # type: ignore[import]
ImportError: liblgcore.so: cannot open shared object file: No such file or directory

The legate.core._lib.types module is defined in the following .so:

(noneditable) sean-dgx2:~> objdump -p $CONDA_PREFIX/lib/python3.9/site-packages/legate/core/_lib/types.cpython-39-x86_64-linux-gnu.so
<snip>
Dynamic Section:
  NEEDED               liblgcore.so
  NEEDED               libstdc++.so.6
  NEEDED               libgcc_s.so.1
  NEEDED               libc.so.6
  RUNPATH              $ORIGIN:$ORIGIN/../../../build/lib
  INIT                 0x000000000000c000
<snip>

This .so needs to link to the main legate.core .so, but it has no way to find it besides RPATH, so it's crucial that this is set correctly.

IIUC, what's happening above is a consequence of you being inside a conda environment but using the system linker. Inside a conda environment you probably want to be using the conda compilers and linkers, which will be compiled with the necessary additional search paths to use inside a conda environment.

Actually the build-stage version of the cython .so contains the (absolute) directory to all required libraries, so I don't think the linker would make a difference here.

(noneditable) sean-dgx2:~> objdump -p /home/scratch.mpapadakis_sw/sean-dgx2/noneditable/legate.core/_skbuild/linux-x86_64-3.9/cmake-build/legate/core/_lib/types.cpython-39-x86_64-linux-gnu.so
<snip>
  RUNPATH              /home/scratch.mpapadakis_sw/sean-dgx2/noneditable/legate.core/build/lib:/home/scratch.mpapadakis_sw/sean-dgx2/noneditable/env/lib:/home/scratch.mpapadakis_sw/sean-dgx2/noneditable/legate.core/_skbuild/linux-x86_64-3.9/cmake-build/_deps/legion-build/lib:/usr/local/cuda/lib64:/usr/local/cuda/lib64/stubs:/usr/lib/x86_64-linux-gnu/openmpi/lib:

I think the problem is with the installation process, which is supposed to "relativize" all RPATHs, so that libraries point to each other correctly in a final installation environment, but that path is not set correctly. $ORIGIN/../../../build/lib is probably the result of this process, but the path is incorrect.