Switch to ORCv2 code generation

[elliottslaughter]

ORCv1 (which we use via the MCJIT wrapper) has been deprecated for quite a while. At some point it will finally be removed and we'll be forced to switch (e.g., to LLJIT).

I don't have time to do this at the moment, but if someone wants to take a look, we could switch proactively rather than waiting for our build to break before doing it.

Deprecation warning looks like:

src/tcompiler.cpp:449:14: warning: 'setUseOrcMCJITReplacement' is deprecated: ORCv1 utilities (including OrcMCJITReplacement) are deprecated. Please use ORCv2/LLJIT instead (see docs/ORCv2.rst) [-Wdeprecated-declarations]
            .setUseOrcMCJITReplacement(true);
             ^

[ErikMcClure]

I feel obligated to warn you that ORCv2 is what I attempted to use for inNative and it required several hacks inside LLVM itself to actually load DLLs properly, and I haven't attempted to upstream those (mostly because those probably aren't the "correct" fix and I just don't want to deal with LLVM right now). While LLVM certainly likes to claim that ORCv2 is what you "should" use, it risks breaking terra on windows in ways that won't be possible to fix until LLVM fixes how it processes symbols on Windows and COFF loading. This is on LLVM 10, which is the latest version supported by terra.

[elliottslaughter]

Is the LLVM project at least aware of these issues? It would be nice to make sure they're aware of them so hopefully they're not too eager to remove ORCv1 while they still remain.

[elliottslaughter]

Well, it looks like we're going to finally be forced to look into this. LLVM 12 has removed setUseOrcMCJITReplacement:

src/tcompiler.cpp:393:14: error: no member named 'setUseOrcMCJITReplacement' in 'llvm::EngineBuilder'
            .setUseOrcMCJITReplacement(true);
             ^

[ErikMcClure]

I will have to attempt to update my compiler to LLVM 12 and see if they actually fixed ORCv2 on windows, but I don't know when I'm going to have time to do this. I checked the LLVM 12 code on github and there haven't been any significant changes to the areas where we implemented our workarounds, so I suspect the issues are still there. This won't affect Terra's compilation step, but it will potentially make it so Terra cannot link to a DLL correctly with the JIT.

[elliottslaughter]

I made a comment in #503 (comment) about the removal of ORCv1 in LLVM 12. Apparently ORCv1 was removed, but MCJIT is still around. Which is frankly bizarre. But apparently we're getting some mileage out of going back to MCJIT, which (assuming it works) may defer the need to think about this a while longer.

[elliottslaughter]

We went ahead with the MCJIT path for #503. It would still be a good idea to investigate ORCv2 at some point because presumably MCJIT will get removed eventually, even if it outlived ORCv1.

[elliottslaughter]

This appears to be linked to two issues:

• Unable to build on Windows: Windows: LLVM 12+ fail in JIT #540
• Test failures in ARM: AArch64 Linux test failures on LLVM 12 and newer #597

So there now seems to be concrete motivation for fixing this, since it is holding back some of our platforms on the newest LLVM versions.

[elliottslaughter]

This also appears to be the issue behind the divergence between hardware and emulated test results on AArch64. (I suspect it would help PPC64le too, but I haven't had time to check there.)

[reinar]

Just to revive the discussion with some context: after introduction of JITLink, which has full COFF support since LLVM 15 (pretty nice talk by one of the contributors), ORCv2 migration seems to be worth it, as Windows concerns described above should be resolved and in general COFF linking couldn't be more native.

Maybe this will help to remove a lot of general flakiness across platforms, however I just started to read the code and have little idea on the effort involved.

[elliottslaughter]

Hi @reinar,

Thanks for bringing this to our attention. OCRv2 is generally something we're interested in. As usual we're waiting for someone who has the time to actually do it. If it falls to me, it will probably happen eventually, but "eventually" might be a while since my time here is very limited and there are several things that are still higher priority on my to-do list.

Contributions are always welcome!