Link against libraries compiled by rustc

[willcrichton]

I'm experimenting with a project to use Rust as a scripting language for a Rust game engine. My goal is to compile the game engine (and stdlib) with rustc, then pass rustc-compiled data structures (eg Vec) directly to a JIT-compiled Rust script using Cranelift.

The specific issue I ran into: is it possible for Cranelift-compiled code to call rustc-compiled code? For example, I found that certain stdlib functions would segfault when called by Cranelift-compiled code due to (I believe) a violation of calling convention. I don't know how fundamental/unavoidable this issue is, so I wanted to double-check.

[bjorn3]

There are a few abi incompatibilities of cg_clif with the SystemV abi (https://github.com/bjorn3/rustc_codegen_cranelift/issues/10). These mostly happen on non-scalar types that can still fit in two registers. In addition atomic operations are implemented using a glbal mutex as cranelift doesn't yet support atomic operations. (the new style backends do, bit those are still missing other things essential for cg_clif) For as long as you don't mix atomic operationa to the same location between cg_clif and cg_llvm this is not a problem. The goal is for both to eventually be compatible with each other.

[willcrichton]

Makes sense. If this is a fixable issue with ABI compatibility, I'm happy to contribute. But I'm having trouble tracking down the precise cause of the bug.

Specifically, I'm calling the JIT this way: https://github.com/willcrichton/crateton/blob/master/crates/crateton_scripts/src/lib.rs

On this script at line 11:
https://github.com/willcrichton/crateton/blob/master/scripts/test.rs

Specifically, a call to core::fmt::Formatter::debug_struct does an indirect function call to a null pointer on the stack, suggesting a violation of calling convention. The full backtrace:

* thread #17, name = 'rustc', stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
  * frame #0: 0x0000000000000000
    frame #1: 0x000000010a137bc4 libstd-f5a6dfa2dca04db0.dylib`core::fmt::Formatter::debug_struct::h1ce538fb8b1a1922 + 20
    frame #2: 0x000000010c9e415b
    frame #3: 0x000000010a136a00 libstd-f5a6dfa2dca04db0.dylib`core::fmt::write::h0ce880d33cd2a300 + 544
    frame #4: 0x000000010a0be2af libstd-f5a6dfa2dca04db0.dylib`_$LT$$RF$std..io..stdio..Stdout$u20$as$u20$std..io..Write$GT$::write_fmt::hf4378193ad11ad65 + 111
    frame #5: 0x000000010a0c01ca libstd-f5a6dfa2dca04db0.dylib`std::io::stdio::_print::hd66f755b4ef78e6b + 506
    frame #6: 0x000000010c9e4487
    frame #7: 0x000000010025661f crateton`_$LT$crateton_scripts..Callbacks$u20$as$u20$rustc_driver..Callbacks$GT$::after_analysis::_$u7b$$u7b$closure$u7d$$u7d$::h4763d10e3eb86cdd((null)=closure-0 @ 0x0000700009998c18, tcx=TyCtxt @ 0x0000700009993fb8) at lib.rs:82:7
    frame #8: 0x00000001001c2980 crateton`rustc_interface::passes::QueryContext::enter::_$u7b$$u7b$closure$u7d$$u7d$::h0a7cb26041167099((null)=0x00007000099990c8) at passes.rs:722:42
    frame #9: 0x000000010014d313 crateton`rustc_middle::ty::context::tls::enter_context::_$u7b$$u7b$closure$u7d$$u7d$::h9a7bb94ad7d570c8 at context.rs:1722:50
    frame #10: 0x0000000100155e59 crateton`rustc_middle::ty::context::tls::set_tlv::he29cf20f8fc9c625(value=123145463369928, f=closure-0 @ 0x0000700009999058) at context.rs:1706:9
    frame #11: 0x000000010014bbcd crateton`rustc_middle::ty::context::tls::enter_context::h0df7789292437c1b(context=0x00007000099990c8, f=closure-0 @ 0x00007000099990a8) at context.rs:1722:9
    frame #12: 0x00000001001c2935 crateton`rustc_interface::passes::QueryContext::enter::h83ec57bd857c2582(self=0x000070000999f648, f=closure-0 @ 0x0000700009999108) at passes.rs:722:9
    frame #13: 0x00000001001397fd crateton`_$LT$crateton_scripts..Callbacks$u20$as$u20$rustc_driver..Callbacks$GT$::after_analysis::h074ea20367c0b367(self=0x0000700008b96458, _compiler=0x000070000999f8b0, queries=0x00007000099991b0) at lib.rs:40:5
    frame #14: 0x00000001027e9b3f librustc_driver-d5fbc914ab196d15.dylib`rustc_interface::queries::_$LT$impl$u20$rustc_interface..interface..Compiler$GT$::enter::hdae1b1fad599fe0a + 1647
    frame #15: 0x000000010278d751 librustc_driver-d5fbc914ab196d15.dylib`rustc_span::with_source_map::h5ed31672a8f5f695 + 353
    frame #16: 0x00000001027ebed1 librustc_driver-d5fbc914ab196d15.dylib`rustc_interface::interface::create_compiler_and_run::hc1e8696cee9a628e + 1105
    frame #17: 0x00000001027d42f3 librustc_driver-d5fbc914ab196d15.dylib`scoped_tls::ScopedKey$LT$T$GT$::set::hf109a0da3bcd66e8 + 227
    frame #18: 0x00000001027edc64 librustc_driver-d5fbc914ab196d15.dylib`std::sys_common::backtrace::__rust_begin_short_backtrace::h26dbda293b3d6e68 + 596
    frame #19: 0x000000010277e1bc librustc_driver-d5fbc914ab196d15.dylib`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h63040340ee7c1069 + 76
    frame #20: 0x000000010a0e29dd libstd-f5a6dfa2dca04db0.dylib`std::sys::unix::thread::Thread::new::thread_start::h9514c02363ad7497 + 45
    frame #21: 0x00007fff70318109 libsystem_pthread.dylib`_pthread_start + 148
    frame #22: 0x00007fff70313b8b libsystem_pthread.dylib`thread_start + 15

[bjorn3]

If this is a fixable issue with ABI compatibility, I'm happy to contribute. But I'm having trouble tracking down the precise cause of the bug.

I don't know where the ABI incompatibility here exactly is. The real fix would be using FnAbi to calculate all ABI adjustments. It is probably easier to use the cg_clif sysroot. I see that you pass rlibs to --extern without making them also available through a dylib. The jit mode of cg_clif only supports loading dylibs. You need to take all dependencies and add them as dependency for a dylib crate. If you then add a dependency on that dylib cg_clif will be able to load that dylib, making all dependencies available. I am planning to make main_shim::maybe_create_entry_wrapper and allocator::codegen public for use in another project by the way.

[willcrichton]

@bjorn3 the only issue with using cg_clif sysroot is that cg_clif can't bootstrap itself b/c it requires rustc_private. My game engine needs to call the cg_clif API to JIT compile code. So if I compile the game engine with cg_clif, then I get a "no crate rustc_*" type of error.

[bjorn3]

You could write a C wrapper around cg_clif and then compile cg_clif using cg_llvm, but everything else using cg_clif. In any case compiling cg_clif using cg_clif will be very bad for performance. You will want to alwaya compile it will all optimizations enabled.