gh-118306: Update JIT compilation to use LLVM 18

[savannahostrowski]

This PR updates the LLVM version used for JIT compilation from 16 to 18.

Main changes, other than the basic 16 --> 18 string updates 😊 :

• Handled some new instructions that were previously not surfaced in LLVM 16 across OS binary formats
• Updated properties where JSON parser had changed

Known Issues:

• The --with-lto option had to be removed for emulated and native Linux in CI due to an open issue with llvm-18 on Ubuntu 22.04. The issue is that LLVMgold.so is currently missing from binaries (see this failed CI run as an example). The current workaround folks are using is to build without LTO. See The llvm-18 Ubuntu Jammy (22.04) LTS binary packages don't include LLVMgold.so llvm/llvm-project#87553 for more details. Not ideal but @brandtbucher I'll let you be the judge as to whether this is acceptable in the short-term. I've left an inline comment in the workflow yml about this.
• -mcmodel=large seems to be unsupported in LLVM 18 so I had to remove this from the build args in _targets.py. See [Driver] Reject unsupported -mcmodel= llvm/llvm-project#70262 for more details.

• Issue: Update JIT compilation to use LLVM 18 #118306

[savannahostrowski]

It looks like test_traceback and test_tracemalloc are failing in aarch64 linux jobs. However, those also seem to be failing in other JIT CI runs as well?

[brandtbucher]

Thanks for taking on this project... it ended up being quite a bit more work for you than I originally thought!

I'd like to benchmark this and peek around at the generated code to make sure we don't see any significant performance degradations. I'm expecting a small slowdown on aarch64-apple-darwin since we need to insert trampolines there now... but we'll see how bad it really is.

It looks like test_traceback and test_tracemalloc are failing in aarch64 linux jobs. However, those also seem to be failing in other JIT CI runs as well?

Yup, see GH-118275.

[brandtbucher]

Also, thanks for digging up the related LLVM issues!

[savannahostrowski]

I think we might be missing some tests or checks for x86_64 Linux jobs. The CI is passing but when I compile in a Codespace (not local dev container...which is Arm in my case), I get other unhandled instructions (e.g. HoleKind_R_X86_64_32, HoleKind_R_X86_64_32S and HoleKind_R_X86_64_PLT32). I'll dig into this over the weekend and patch things up but we may want to investigate adding more coverage here.

[brandtbucher]

I get other unhandled instructions (e.g. HoleKind_R_X86_64_32, HoleKind_R_X86_64_32S and HoleKind_R_X86_64_PLT32).

Hm, I wonder if Fedora uses a patched LLVM with different defaults or something...

[brandtbucher]

Can you try adding -fpic to the list of args for the Linux targets? Position-independent code (which lets us reference symbols more than 32 bits away) is the default on the other platforms, but based on your description it seems like it isn't for Fedora. Adding that option might fix it.

[savannahostrowski]

Ah! Okay, that did work for the Codespace. Also, checked in the local dev container as well. Thanks for the tip!

[brandtbucher]

Nice, looks like it's also a slight performance improvement (if anything).

[brandtbucher]

Doc failure isn't us: #118401

[brandtbucher]

Free-threading failures look like flukes too. Gonna re-run once the JIT tests finish.