LTO prevents SimCore Factory from creating objects

[tomeichlersmith]

Describe the bug
The SimCore plugin factory is found unaware of declared plugins when LTO is enabled.

To Reproduce
Steps to reproduce the behavior:

just configure-clang-lto
just build
just fire SimCore/test/basic.py

and then you see

[ fire ] 0 fatal: [SimFactory] : An object named simcore::TrackerSD has not been declared.
  at /home/tom/code/ldmx/ldmx-sw/SimCore/include/SimCore/Factory.h:274 in make

Desired behavior
It'd be cool to be able to enable LTO.

Additional context
So, there are actually two different plugin factories in ldmx-sw.

• Framework/PluginFactory is a C-style plugin manager that forces registration to happen at library-load-time using compiler attributes (like __attribute(init_priority(500)) and __attribute__(constructor(1000)))
• SimCore/Factory is a templated C++17 style factory that uses cleverly-placed static variables to force certain behaviors at library-load-time.

The Framework/PluginFactory appears to be working when LTO is enabled since we get to attempting to create the TrackerSD. This implies that the Framework/PluginFactory was already used to declare and create the simulation processor. This causes me to focus attention on SimCore/Factory which works out since I originally designed it. I placed some printouts including the address of the Factory so we can observe the separate Factories being created for the different types. This is what I see.

denv fire SimCore/test/basic.py 
---- LDMXSW: Loading configuration --------
Factory(0x7ff2238562b8)
Factory(0x7ff2238562b8): declare simcore::BertiniAtLeastNProductsModel
Factory(0x7ff2238562b8): declare simcore::BertiniModel
Factory(0x7ff2238562b8): declare simcore::BertiniNothingHardModel
Factory(0x7ff2238562b8): declare simcore::BertiniSingleNeutronModel
Factory(0x7ff2238562b8): declare simcore::NoPhotoNuclearModel
Factory(0x7ff22383d4b0)
Factory(0x7ff22383d4b0): declare simcore::EcalSD
Factory(0x7ff22383d4b0): declare simcore::HcalSD
Factory(0x7ff22383d4b0): declare simcore::ScoringPlaneSD
Factory(0x7ff22383d4b0): declare simcore::TrackerSD
Factory(0x7ff22383d4b0): declare simcore::TrigScintSD
Factory(0x7ff223816538)
Factory(0x7ff223816538): declare simcore::generators::GeneralParticleSource
Factory(0x7ff223816538): declare simcore::generators::LHEPrimaryGenerator
Factory(0x7ff223816538): declare simcore::generators::MultiParticleGunPrimaryGenerator
Factory(0x7ff223816538): declare simcore::generators::ParticleGun
---- LDMXSW: Configuration load complete  --------
---- LDMXSW: Starting event processing --------
[ RunManager ]: Parallel worlds physics list has been registered.
[ RunManager ]: Parallel worlds have been enabled.
Factory(0x7ff2270bcfe0)
Factory(0x7ff2270bcfe0): make simcore::TrackerSD
[ fire ] 0 fatal: [SimFactory] : An object named simcore::TrackerSD has not been declared.
  at /home/tom/code/ldmx/ldmx-sw/SimCore/include/SimCore/Factory.h:274 in make
~Factory(0x7ff2270bcfe0)
~Factory(0x7ff223816538)
~Factory(0x7ff22383d4b0)
~Factory(0x7ff2238562b8)

So LTO does not prevent the declaration process from happening (which is what I assumed was going wrong), but - for some reason - a new Factory is created when attempting to create the TrackerSD instead of using the one that was already created during library-load for declaration. My guess is that I'll need to read-up on the actual meaning of static variable in the context of a static class member function. Oofdah.

I'll probably try to create a separate testing area since it takes so long to recompile after changing the templated Factory.h header. See my gist about this Factory for more links and probably where I'll put results.

[EinarElen]

I think you will see a similar effect if you build with clang instead of GCC

[tvami]

I think you will see a similar effect if you build with clang instead of GCC

Yes, I can confirm that. But takes similarly long time.

[tomeichlersmith]

I was able to create a very small repo reproducing the error: https://github.com/tomeichlersmith/ldmx-factory-test-bench much faster to compiler and play around with 😄

[tomeichlersmith]

I thought I was able to figure out what was happening. Look at the repo I created above for the details, but the TLDR is that static variables are library-local unless we explicitly tell the linker to allow for dynamic symbols (i.e. sharing/exporting symbols) such that the library we load can share the same singleton as the executable. I don't know why this was only showing itself as an error when trying to enable LTO since I also can observe the error without LTO.

However, while this appears to replicate the issue observed here and is resolved by adding the ENABLE_EXPORTS property to the executable, applying this strategy here does not work. I made the following edits and then tried rebuilding with clang and LTO and the same error occurrs.

diff --git a/Framework/CMakeLists.txt b/Framework/CMakeLists.txt
index e6e30670..10e1abb8 100644
--- a/Framework/CMakeLists.txt
+++ b/Framework/CMakeLists.txt
@@ -81,7 +81,8 @@ set_target_properties(
   Framework
   PROPERTIES CXX_STANDARD 17
              CXX_STANDARD_REQUIRED YES
-             CXX_EXTENSIONS NO)
+             CXX_EXTENSIONS NO
+             ENABLE_EXPORTS TRUE)
 
 # the following writes the LinkDef file using the CMake global variables
 # namespaces and dict that are lists appended to by the register_event_object function
@@ -128,6 +129,7 @@ install(FILES ${CMAKE_CURRENT_BINARY_DIR}/libFramework_rdict.pcm DESTINATION lib
 add_executable(fire ${PROJECT_SOURCE_DIR}/app/fire.cxx)
 target_link_libraries(fire PRIVATE Framework::Framework)
 install(TARGETS fire DESTINATION ${CMAKE_INSTALL_PREFIX}/bin)
+set_property(TARGET fire PROPERTY ENABLE_EXPORTS TRUE)
 
 # Setup the test
 setup_test(dependencies Framework::Framework)
diff --git a/SimCore/CMakeLists.txt b/SimCore/CMakeLists.txt
index c80435e4..b66eed99 100644
--- a/SimCore/CMakeLists.txt
+++ b/SimCore/CMakeLists.txt
@@ -92,7 +92,8 @@ setup_library(module SimCore
 set_target_properties(SimCore
                       PROPERTIES CXX_STANDARD 17
                       CXX_STANDARD_REQUIRED YES
-                      CXX_EXTENSIONS NO)
+                      CXX_EXTENSIONS NO
+                      ENABLE_EXPORTS TRUE)
 
 setup_python(package_name LDMX/SimCore)
 
diff --git a/cmake/BuildMacros.cmake b/cmake/BuildMacros.cmake
index 3716712e..c2bd1f55 100644
--- a/cmake/BuildMacros.cmake
+++ b/cmake/BuildMacros.cmake
@@ -143,6 +143,7 @@ macro(setup_library)
   enable_compiler_warnings(${library_name})
   enable_ipo(${library_name})
   enable_clang_tidy(${library_name} ${WARNINGS_AS_ERRORS})
+  set_property(TARGET ${library_name} PROPERTY ENABLE_EXPORTS TRUE)
 
   # Define an alias. This is used to create the imported target.
   set(alias "${setup_library_module}::${setup_library_module}")