legate.core/src/core/utilities/type_traits.h:85:46: error: ‘complex’ was not declared in this scope

[1193749292]

Legion Build Commands:
cmake .. -S /home/xxx/hpc3/legion/ -D CMAKE_INSTALL_PREFIX=../install -D REALM_USE_OPENMP=ON -D Legion_USE_OpenMP=ON -D REALM_OPENMP_SYSTEM_RUNTIME=ON -D Legion_OpenMP_SYSTEM_RUNTIME=ON -D REALM_USE_MPI=ON -DLegion_NETWORKS=mpi -D Legion_MPI_INTEROP=ON -DLEGION_REDOP_HALF=ON -B ../build

Legate Installation Commands:
./install.py --network mpi --openmp --with-legion /home/xxx/hpc3/legion/build/ -v --nocuda --march native

Description: Ubuntu 20.04.5 LTS | aarch64

The following files are the installation output information.
legate.core-install_output.txt

[manopapad]

Are you on branch control_replication for Legion?

[1193749292]

@manopapad The legion branch is legion-22.12. I also tested the control_replication branch. The difference is that there is no error about legion'OutputRegion'. Everything else is the same.

legate.core-legion-control_replication_install_output.txt

[manopapad]

I believe this is happening because you are missing -DLegion_REDOP_COMPLEX=ON from your Legion build command (and possibly also -DLegion_BUILD_BINDINGS=ON and -DLegion_USE_Python=ON). Can you try adding that and building again?

I submitted a PR to document the required Legion branch and configuration options #670, and will be in touch with the Legion devs to try and export these settings, so we can check for their presence when a user passes in a pre-built installation of Legion.

[1193749292]

@manopapad It's useful. I tried the following methods and the results were the same.
legate.core/src/core/utilities/type_traits.h

However, an error is reported after the legate is successfully installed.

---

Executing: " cmake --install /opt/home/xxx/hpc3/legion/build/bindings/python --prefix /home/xxx/anaconda3 " with {}
CMake Error: Error processing file: /home/xxx/hpc3/legion/build/bindings/python/cmake_install.cmake

An error is reported when the legate is running.

[manopapad]

Have you tried rebuilding Legion with the flags I suggested? I suggest you remove your old Legion installation and rebuild with:

cmake .. -S /home/xxx/hpc3/legion/ -D CMAKE_INSTALL_PREFIX=../install -D REALM_USE_OPENMP=ON -D Legion_USE_OpenMP=ON -D REALM_OPENMP_SYSTEM_RUNTIME=ON -D Legion_OpenMP_SYSTEM_RUNTIME=ON -D REALM_USE_MPI=ON -DLegion_NETWORKS=mpi -D Legion_MPI_INTEROP=ON -DLEGION_REDOP_HALF=ON -DLegion_REDOP_COMPLEX=ON -DLegion_BUILD_BINDINGS=ON -DLegion_USE_Python=ON -B ../build

Then install legate.core using --with-legion.

Or alternatively you can let legate.core build Legion on its own, by not passing a --with-legion or --legion-src-dir.

[1193749292]

@manopapad Rebuild the legion based on your suggestions. After the cmake is successfully executed, the following error message is displayed when the make command is executed:

[manopapad]

It this happening purely during Legion compilation? If so, could you please report this issue on the Legion bug tracker? I suspect it might have something to do with these flags -D REALM_OPENMP_SYSTEM_RUNTIME=ON -D Legion_OpenMP_SYSTEM_RUNTIME=ON.

[1193749292]

@manopapad Thank you for your advice, someone here has replied.
StanfordLegion/legion#1456

[1193749292]

StanfordLegion/legion#1456
https://gitlab.com/StanfordLegion/legion/-/commit/78c5f3ab9c4ea1393183ca135fbca3c403a3113b
This is the solution to legion_python. Modify python_internal.h and python_module.cc, add

-DCMAKE_BUILD_TYPE=Release

and rebuild the legion. Otherwise, the following error message is displayed:

---

legion_python: /root/xxx/legion-control_replication/runtime/realm/timers.inl:72: static long long int Realm::Clock: :current_time_in_nanoseconds(bool): Assertion ‘nanoseconds <= uint64_t(LLONG_MAX)' failed.
Signal legion_python: /root/xxx/legion-control_replication/runtime/realm/timers.inl:72: static long long int Realm: :Clock: :current_time_in_nanoseconds(bool): Assertion ‘nanoseconds <= uint64_t(LLONG_MAX)' failed.
6 received by node 0, process 60468 (thread 7f8112e19880) obtaining backtrace

---

After the legate is successfully installed, run the legate command. The following error information is displayed:

---

[0 - 7f4320297880] -1555852541.985930 {6}{python}: python exception occurred within task:
Traceback (most recent call last):
File "/root/miniconda3/envs/legion-xxx/lib/python3.9/site-packages/legion_top.py", line 409, in legion_python_main
c.legion_task_preamble(
File "/root/miniconda3/envs/legion-xxx/lib/python3.9/site-packages/cffi/api.py", line 912, in _getattr_
make_accessor(name)
File "/root/miniconda3/envs/legion-xxx/lib/python3.9/site-packages/cffi/api.py", line 908, in make_accessor
accessors[name](name)
File "/root/miniconda3/envs/legion-xxx/lib/python3.9/site-packages/cffi/api.py", line 838, in accessor_function
value = backendlib.load_function(BType, name)
AttributeError: function/symbol 'legion_task_preamble' not found in library '/root/miniconda3/envs/legion-xxx/lib/liblegion_canonical_python.so': /root/miniconda3/envs/legion-xxx/lib/liblegion_canonical_python.so: undefined symbol: legion_task_preamble
legion_python: /root/xxx/legion-control_replication/runtime/realm/python/python_module.cc:1006: virtual void Realm::LocalPythonProcessor::execute_task(Realm::Processor::TaskFuncID, const Realm::ByteArrayRef&): Assertion `0' failed.
Signal 6 received by node 0, process 3373 (thread 7f4320297880) - obtaining backtrace
Signal 6 received by process 3373 (thread 7f4320297880) at: stack trace: 14 frames
[0] = /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7f4423709980]
[1] = /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7) [0x7f4422838e87]
[2] = /lib/x86_64-linux-gnu/libc.so.6(abort+0x141) [0x7f442283a7f1]
[3] = /lib/x86_64-linux-gnu/libc.so.6(+0x303fa) [0x7f442282a3fa]
[4] = /lib/x86_64-linux-gnu/libc.so.6(+0x30472) [0x7f442282a472]
[5] = /root/miniconda3/envs/legion-xxx/bin/legion_python(+0x126c69f) [0x5608babb469f]
[6] = /root/miniconda3/envs/legion-xxx/bin/legion_python(+0xdd7723) [0x5608ba71f723]
[7] = /root/miniconda3/envs/legion-xxx/bin/legion_python(+0xdd7906) [0x5608ba71f906]
[8] = /root/miniconda3/envs/legion-xxx/bin/legion_python(+0x127151a) [0x5608babb951a]
[9] = /root/miniconda3/envs/legion-xxx/bin/legion_python(+0xdda6a2) [0x5608ba7226a2]
[10] = /root/miniconda3/envs/legion-xxx/bin/legion_python(+0x1270766) [0x5608babb8766]
[11] = /root/miniconda3/envs/legion-xxx/bin/legion_python(+0xddce3b) [0x5608ba724e3b]
[12] = /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f44236fe6db]
[13] = /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f442291b61f]

[manopapad]

Assertion ‘nanoseconds <= uint64_t(LLONG_MAX)' failed.

This looks like a Realm issue (or rather, the output of some system timing function is not as Realm expects). If you have time, could you add this instrumentation and try building (and running) again with -DCMAKE_BUILD_TYPE=Release?

diff --git a/runtime/realm/timers.inl b/runtime/realm/timers.inl
index 75a4b42dc..8949fb03d 100644
--- a/runtime/realm/timers.inl
+++ b/runtime/realm/timers.inl
@@ -69,6 +69,14 @@ namespace Realm {
       nanoseconds -= zero_time;
 #ifdef DEBUG_REALM
     // make sure this fits in a long long
+    std::cerr << "native = " << native
+              << " nanoseconds = " << nanoseconds
+              << " absolute = " << absolute
+              << " zero_time = " << zero_time
+              << " LLONG_MAX = " << LLONG_MAX
+              << " uint64_t(LLONG_MAX) = " << uint64_t(LLONG_MAX)
+              << std::endl;
+    std::cerr.flush();
     assert(nanoseconds <= uint64_t(LLONG_MAX));
 #endif
     return nanoseconds;
@@ -325,4 +333,3 @@ namespace Realm {


 }; // namespace Realm
-

After the legate is successfully installed, run the legate command. The following error information is displayed:

Can you please share the full build commands that you used, and build output that you got, for Legion and legate.core?

[1193749292]

Add std::cerr... and with -DCMAKE_BUILD_TYPE=Release , the error information is the same.

The following error information is displayed without -DCMAKE_BUILD_TYPE=Release :

native = 125710472513989694 nanoseconds = 1645055 absolute = 0 zero_time = 1681439899479082722 LLONG_MAX = 9223372036854775807 uint64_t(LLONG_MAX) = 9223372036854775807
native = 125710472514051510 nanoseconds = 1665654 absolute = 0 zero_time = 1681439899479082722 LLONG_MAX = 9223372036854775807 uint64_t(LLONG_MAX) = 9223372036854775807
...(Several similar messages)

Signal 6legion_python: /root/xxx/legion-control_replication/runtime/realm/timers.inl:80: static long long int Realm::Clock::current_time_in_nanoseconds(bool): Assertion `nanoseconds <= uint64_t(LLONG_MAX)' failed.
 uint64_t(LLONG_MAX) = 9223372036854775807
9223372036854775807
legion_python: /root/xxx/legion-control_replication/runtime/realm/timers.inl:80: static long long int Realm::Clock::current_time_in_nanoseconds(bool): Assertion `nanoseconds <= uint64_t(LLONG_MAX)' failed.
 received by node 0, process 

[manopapad]

I reported the timing issue on StanfordLegion/legion#1459. You can probably continue using a release build while the issue is being fixed. Thanks for your help investigating this.

Please share the command line you use to build in release mode, and the build outputs, so we can fix any remaining issues with that. And please make sure you include all the necessary Legion defines, when building Legion:

-DLegion_REDOP_HALF=ON -DLegion_REDOP_COMPLEX=ON -DLegion_BUILD_BINDINGS=ON -DLegion_USE_Python=ON

[1193749292]

@manopapad

cmake .. -S /root/xxx/legion-control_replication/ -D CMAKE_INSTALL_PREFIX=../install -D REALM_USE_OPENMP=ON -D Legion_USE_OpenMP=ON -D REALM_OPENMP_SYSTEM_RUNTIME=ON -D Legion_OpenMP_SYSTEM_RUNTIME=ON -D REALM_USE_MPI=ON -DLegion_NETWORKS=mpi -D Legion_MPI_INTEROP=ON -DLegion_REDOP_HALF=ON -DLegion_REDOP_COMPLEX=ON -DLegion_BUILD_BINDINGS=ON -DLegion_USE_Python=ON -DLegion_BUILD_JUPYTER=ON -DCMAKE_BUILD_TYPE=Release -B ../build
legion_build_output-success.txt

./install.py -j 40 --clean --network mpi --openmp --with-legion /root/xxx/legion-control_replication/build/ -v --nocuda
legate.core-install_output-success.zip

legate
legate_run-error.txt

https://gitlab.com/StanfordLegion/legion/-/commit/78c5f3ab9c4ea1393183ca135fbca3c403a3113b
Other than that, there are no other changes to the source code.

legion control_replication
legate branch-22.12

[manopapad]

Can you please try with legate.core branch 23.05?

[1193749292]

@manopapad

Modify legate.core/CMakeLists.txt
set(legate_core_version 22.12.00)
Otherwise, a message is displayed indicating that the version does not match the legion version.

CMake Error at _skbuild/linux-x86_64-3.9/cmake-build/_deps/rapids-cmake-src/rapids-cmake/find/package.cmake:114 (find_package):
    Could not find a configuration file for package "Legion" that exactly
    matches requested version "23.3.0".

./install.py -j 60 --clean --network mpi --openmp --with-legion /root/xxx/legion-control_replication/build/ -v --nocuda > legate.core-branch-23.05-install_output-error.txt 2>&1
legate.core-branch-23.05-install_output-error.txt

Or do not modify legate.core/CMakeLists.txt

When a branch is changed from legion to legion-23.03.0, the same error is reported.

  _modinit_prefix:PyInit_
  CMake Error at legate_core_python.cmake:70 (rapids_cython_add_rpath_entries):
    Unknown CMake command "rapids_cython_add_rpath_entries".
  Call Stack (most recent call first):
    CMakeLists.txt:103 (include)

[manopapad]

I took a closer look at the original build outputs you posted on #667 (comment). It appears that both Legion and legate.core were on versions 22.12, which should be compatible. According to the Legion output, it appears that liblegion_canonical_python.so was installed at /root/xxx/legion-control_replication/install/lib, but during runtime we are trying to use it out of /root/miniconda3/envs/legion-xxx/lib/. The error message might be misleading here:

AttributeError: function/symbol 'legion_task_preamble' not found in library '/root/miniconda3/envs/legion-xxx/lib/liblegion_canonical_python.so': /root/miniconda3/envs/legion-xxx/lib/liblegion_canonical_python.so: undefined symbol: legion_task_preamble

It suggests that one specific function is missing, but I suspect the entire liblegion_canonical_python.so file is missing.

There is even a potentially relevant error message during Legion build:

You are attempting to install a package to a directory that is not
on PYTHONPATH and which Python does not read ".pth" files from.

@1193749292 can you check if liblegion_canonical_python.so exists under /root/miniconda3/envs/legion-xxx/lib/?

@trxcllnt @jjwilke Can you please take a look at how @1193749292 is building Legion in the above linked comment? Should the Legion CFFI be getting installed somewhere else?

[1193749292]

@manopapad

It exists, and soft link to the file, I deleted both files and reinstalled legate.core, liblegion_canonical_python.so and liblegion_canonical_python.so.1 both regenerated.

lrwxrwxrwx 1 root root 31 Apr 18 01:34 /root/miniconda3/envs/legion-xxx/lib/liblegion_canonical_python.so -> liblegion_canonical_python.so.1

[manopapad]

Is the error still happening?

[1193749292]

@manopapad Yes, the error still happening

[manopapad]

@1193749292 can you run the make command during Legion build with --verbose, so we can see the commands being run?

@eddy16112 any idea why liblegion_canonical_python.so would not include legion_task_preamble?

[eddy16112]

I think it is a linking issue. If you run into line 409, it means canonical python is not used, then why it tried to load legion_task_preamble from lib legion_canonical_python.so`?

[manopapad]

@eddy16112 You're right, we're going through the legion_python executable, therefore we should be using the regular liblegion.so, rather than legion_canonical_python.so. @1193749292 before rebuilding, can you try adding the following lines to /root/miniconda3/envs/legion-xxx/lib/python3.9/site-packages/legion_cffi.py and running legate again? This should allow us to confirm that liblegion.so is not being loaded for some reason

28a29
>     print("trying to import builtin CFFI", flush=True)
29a31
>     print("managed to import, trying to call legion_runtime_has_context", flush=True)
31a34
>     print("legion_runtime_has_context exists", flush=True)
33a37
>     print("failed, trying canonical CFFI", flush=True)

And please run legate with --verbose, so we can check where it's finding our executable legion_python.

[1193749292]

@1193749292 can you run the make command during Legion build with --verbose, so we can see the commands being run?

make VERBOSE=1 -j40
legion-make_VERBOSE=1_output.txt

[1193749292]

@manopapad

And please run legate with --verbose, so we can check where it's finding our executable legion_python.

legate --verbose_output.txt

[manopapad]

Thanks for producing these outputs. The only thing that jumps out at me is that liblegion was static-linked inside the legion_python executable, which might not be working properly with the dlopen(None) that is done in legion_cffi.py. Could you please check to see if the symbols in question appear to be exported?

nm -a /root/miniconda3/envs/legion-xxx/bin/legion_python | grep legion_runtime_has_context
nm -a /root/miniconda3/envs/legion-xxx/lib/liblegion_canonical_python.so | grep legion_runtime_has_context

[1193749292]

@manopapad
The preceding two commands do not have any output.
When the --legion-src-dir command is used to install the legate and the legate is running properly, no output is displayed.

[jjwilke]

The legion_task_preamble issue is a common problem (in general, not specific to Legion) whenever linking a static library into a shared one. There are a few ways to fix this:

1. You can globally set the link option -Wl,--whole-archive. This will get things to work, but is generally suboptimal since you won't optimize the shared library by removing unnecessary symbols. Practially this doesn't matter much for Legion.

2. Set BUILD_SHARED_LIBS=ON. The shared library linking doesn't obliterate unused symbols.

3. If you are using CMake 3.24 or above, you can change the link line locally for canonical python until I merge a fix:

if (CMAKE_VERSION GREATER_EQUAL 3.24)
  target_link_libraries(legion_canonical_python PUBLIC "$<LINK_LIBRARY:WHOLE_ARCHIVE,LegionRuntime>")
else()
  target_link_libraries(legion_canonical_python PUBLIC Legion::Legion)
endif()

4. Hacky option if you want to link statically and have an ancient CMake... add the following at the bottom of canonical_python.cc:

auto* _very_bad_linker = &legion_task_preamble;

[manopapad]

you can change the link line locally for canonical python

Note that we also need the legion_python executable to retain some symbols from liblegion.a, since the python cffi will need to dlopen(None) to retrieve them.

[1193749292]

2. Set BUILD_SHARED_LIBS=ON. The shared library linking doesn't obliterate unused symbols.

cmake .. -S /root/xxx/legion-control_replication/ -D CMAKE_INSTALL_PREFIX=../install -D REALM_USE_OPENMP=ON -D Legion_USE_OpenMP=ON -D REALM_OPENMP_SYSTEM_RUNTIME=ON -D Legion_OpenMP_SYSTEM_RUNTIME=ON -D REALM_USE_MPI=ON -DLegion_NETWORKS=mpi -D Legion_MPI_INTEROP=ON -DLegion_REDOP_HALF=ON -DLegion_REDOP_COMPLEX=ON -DLegion_BUILD_BINDINGS=ON -DLegion_USE_Python=ON -DLegion_BUILD_JUPYTER=ON -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=ON -B ../build

Then run the command: legate , The following error information is displayed:
/root/miniconda3/envs/legion-xxx/bin/legion_python: symbol lookup error: /root/miniconda3/envs/legion-xxx/bin/legion_python: undefined symbol: _ZTIN6Legion15ShardingFunctorE

3. If you are using CMake 3.24 or above, you can change the link line locally for canonical python until I merge a fix:

vim legion-control_replication/bindings/python/CMakeLists.txt


Running legate succeeded.

@jjwilke
The above is what I did according to the method you provided, method 1 and 4 I did not try, now it legate can run normally, thank you very much for your help. @manopapad @jjwilke

The legate problem is fixed, and then I'm going to install Cunumeric, which has some problems with the installation, here's the link, if you're interested.
nv-legate/cupynumeric#897

[manopapad]

You should technically do the same if (CMAKE_VERSION GREATER_EQUAL 3.24) trick for the legion_python binary, because that "trying to import builtin CFFI" should be succeeding. But if this is working for you we can proceed (once we also fix your cuNumeric compilation error).

[evanramos-nvidia]

4. Hacky option if you want to link statically and have an ancient CMake... add the following at the bottom of canonical_python.cc:

auto* _very_bad_linker = &legion_task_preamble;

Realm has the following in runtime/realm/compiler_support.h:

// the following defines are used to communicate which declarations are part
//  of the "official" public Realm API (as opposed to things that need to be in
//  the same header files for C++ reasons) - they also control which symbols
//  appear in the shared library:
// REALM_PUBLIC_API - class/method/variable intended for direct use
//                    in applications
// REALM_INTERNAL_API_EXTERNAL_LINKAGE - method/variable not intended for
//                    direct use in applications but used indirectly and
//                    therefore requiring external linkage
// REALM_INTERNAL_API - method/variable that is not part of the public API
//                    despite being in an otherwise-public class

#ifdef REALM_LIMIT_SYMBOL_VISIBILITY
  #define REALM_PUBLIC_API  __attribute__((visibility("default")))
  #define REALM_INTERNAL_API_EXTERNAL_LINKAGE  __attribute__((visibility("default")))
  #define REALM_INTERNAL_API  __attribute__((visibility("hidden")))
#else
  #define REALM_PUBLIC_API
  #define REALM_INTERNAL_API_EXTERNAL_LINKAGE
  #define REALM_INTERNAL_API
#endif

What happens if you add REALM_INTERNAL_API_EXTERNAL_LINKAGE to legion_task_preamble's declaration?

[1193749292]

4. Hacky option if you want to link statically and have an ancient CMake... add the following at the bottom of canonical_python.cc:
   auto* _very_bad_linker = &legion_task_preamble;

What happens if you add REALM_INTERNAL_API_EXTERNAL_LINKAGE to legion_task_preamble's declaration?

some warning information is added during legion compilation.

/root/xxx/legion/runtime/realm/compiler_support.h:229:0: warning: "REALM_INTERNAL_API_EXTERNAL_LINKAGE" redefined
   #define REALM_INTERNAL_API_EXTERNAL_LINKAGE  __attribute__((visibility("default")))

The same warning information is displayed when the legate is installed. The legate can run properly.
Other than that, there seems to be no other effect.

[evanramos-nvidia]

Just to be clear, did you add that token as a tag to the declaration of the symbol? I don't mean adding another copy of the #define to another location. For example:

diff --git a/runtime/legion/legion_c.h b/runtime/legion/legion_c.h
index de309d7d9..720b04a75 100644
--- a/runtime/legion/legion_c.h
+++ b/runtime/legion/legion_c.h
@@ -5780,6 +5780,7 @@ extern "C" {
   /**
    * @see Legion::LegionTaskWrapper::legion_task_preamble()
    */
+  REALM_INTERNAL_API_EXTERNAL_LINKAGE
   void
   legion_task_preamble(
     const void *data,

This will tell the compiler and linker that the symbol should be externally visible, which will ensure it is present in an optimized shared library even without any callers referring to it.

I am not familiar with Legion and I see other instances of legion_task_preamble so it may need to be added to other declarations.

[manopapad]

I believe adding REALM_INTERNAL_API_EXTERNAL_LINKAGE (a.k.a. __attribute__ ((visibility ("default")))) will only have an effect if compiling with -fvisibility=hidden, and legion_task_preamble is a symbol from liblegion, which doesn't use -fvisibility=hidden when compiling its constituent .o files. Therefore, normally liblegion would export all its symbols unconditionally.

The issue here appears to be that --export-dynamic is not enough to ensure that all symbols are exported in the final executable, in case static libraries are used:

ar qc liblegion.a foo.o bar.o
gcc -Wl,--export-dynamic -rdynamic -o legion_python main.o liblegion.a
# all symbols in main.o are exported, but NOT foo.o/bar.o

In addition, -Wl,--whole-archive needs to be added before liblegion.a.

[1193749292]

@evanramos-nvidia

warning: "REALM_INTERNAL_API_EXTERNAL_LINKAGE" redefined

Thank you, there is no such warning message.
The legate operation is normal, which is the same as that in method 3.

3. If you are using CMake 3.24 or above, you can change the link line locally for canonical python until I merge a fix:

Running legate succeeded.