NativeAOT-LLVM: runtime initialization workarounds

[yowl]

For Wasm components, there is no mechanism currently for a component's module, to have its initialization called, i.e for static c++ class constructors to run. Wasm tool chains put these in a __wasm_call_ctors method, which is invoked (for libraries) via _initialize. However the component model does not export _initialize and the wasmtime (I've not investigated other hosts) host cannot/does not invoke it. We therefore have a problem when an exported method is called as the runtime is not initialized. We can take advantage of the fact that Wasm memory is zero initialized to do something like

#ifdef TARGET_WASI
static int m_initializeCalled = 0;
EXTERN_C void _initialize();
#endif

COOP_PINVOKE_HELPER(void, RhpReversePInvoke, (ReversePInvokeFrame * pFrame))
{
#ifdef TARGET_WASI
    if (!m_initializeCalled)
    {
        _initialize();
        m_initializeCalled = 1;
    }
#endif

As a hack for components. It could probably be extended to cover wasm-wasi programs , but as a temporary workaround until bytecodealliance/preview2-prototyping#99 is resolved, could there be a better solution?

Thanks

[SingleAccretion]

Three comments:

1. What global constructors do we rely on? The runtime initialization only needs a tiny piece of code to set the callback, the value of which is then used a sentinel. This looks fixable, but if there are others, it may be getting more challenging (e. g. does ICU have global constructors).
2. Invoking _initialize like that, if it doesn't have a "have I been initialized already" guard inside seems rather unsafe, considering we may run foreign code from unknown places.
3. How do other people run any useful code? Do they not use C++ at all (and other languages with global constructors)?

[yowl]

1.You are right, I thought there was something, but its the runtime intitialization callback that is the blocker.

2. This was just my first idea to make something callable as a wasm component. In the reactor model of wasm components, only exports are callable which would all go through RhpReversePInvoke so it seemed ok for this single use case. I'm fine with nothing making into the branch at the moment, but I'm not sure there is a clear path to how we can get the runtime initialised for wasi components where the first call to dotnet code, including the runtime will be the exported symbol.

3. This is very new, I understand that for rust they have removed most or all of the global constructors from their compiler. The same is true for wasi's libc: they are/have removed the global ctors that were present. Long term, and by long term I think we are talking about before wasm components get more than experimental adoption, there are plans to change the tooling so that global ctors are invoked. Also wasm-ld will wrap all exports with a call to __wasm_call_ctors if there is no other code in the module that calls __wasm_call_ctors.

[SingleAccretion]

but its the runtime intitialization callback that is the blocker

That is good news.

I am actually wondering what is the purpose behind the somewhat involved initialization handshake. On the face of it, it could be replaced by, e. g. defining g_RuntimeInitializationCallback in Bootsrapper/main.cpp like below:

#ifdef NATIVEAOT_DLL
int (*g_RuntimeInitializationCallback)() = &InitializeRuntime;
#else
int (*g_RuntimeInitializationCallback)() = nullptr;
#endif

Or some other equivalent scheme which avoids global ctors altogether.

[jkotas]

static int (*g_RuntimeInitializationCallback)() = &InitializeRuntime;

Does C/C++ in wasm guarantee that this won't compile into a static constructor?

Also, the variable would need to extern instead of static so that it is visible to the runtime. It should work otherwise.

[SingleAccretion]

Does C/C++ in wasm guarantee that this won't compile into a static constructor?

I believe so, for the configurations we target right now at least. I think PIC modes have more elaborate constraints, but they're also browser-only.

Also, the variable would need to extern instead of static so that it is visible to the runtime. It should work otherwise.

Indeed, fixed.

[yowl]

Thanks I'll give this a try and report back.

[yowl]

the runtime intitialization callback that is the blocker.

This was the blocker, but I was wrong to say it was the only problem.

runtimelab/src/coreclr/gc/gcscan.cpp

Line 21 in 7dace36

VOLATILE(int32_t) GCScan::m_GcStructuresInvalidCnt = 1;

is the next issue.

It should be set in the wasm function

(func $__cxx_global_var_init.6 (type 6)
    (local i32 i32 i32 i32 i32 i32 i32 i32 i32 i32)
    global.get $__stack_pointer
    local.set 0
    i32.const 16
    local.set 1
    local.get 0
    local.get 1
    i32.sub
    local.set 2
    local.get 2
    global.set $__stack_pointer
    i32.const 1
    local.set 3
    local.get 2
    local.get 3
    i32.store offset=12
    i32.const 1609420

1609420 is the wasm constant for m_GcStructuresInvalidCnt and __cxx_global_var_init.6 is called as part of _GLOBAL__sub_I_gcscan.cpp which in turn is called from

(func $__wasm_call_ctors (type 6)
    call $__wasilibc_initialize_environ_eagerly
    call $__wasilibc_populate_preopens
    call $_GLOBAL__sub_I_gccommon.cpp
    call $_GLOBAL__sub_I_main.cpp
    call $_GLOBAL__sub_I_yieldprocessornormalized.cpp
    call $_GLOBAL__sub_I_gcenv.unix.cpp
    call $_GLOBAL__sub_I_objecthandle.cpp
    call $_GLOBAL__sub_I_gcscan.cpp
    call $_GLOBAL__sub_I_gcwks.cpp
    call $_GLOBAL__sub_I_gceventstatus.cpp)

As it is not initialised to 1 this fails with

thread 'main' panicked at 'execution of foo:foo/floats#float32-param failed error while executing at wasm backtrace:
    0: 0xfb94e2 - <unknown>!abort
    1: 0x1005e - Assert(char const*, char const*, unsigned int, char const*)
                    at C:/github/runtimelab/artifacts/obj/coreclr/wasi.wasm.Debug/C:/github/runtimelab/src/coreclr/nativeaot/Runtime/rhassert.cpp:24:5
    2: 0x4f7aa - GCScan::GcRuntimeStructuresValid(int)
                    at C:/github/runtimelab/artifacts/obj/coreclr/wasi.wasm.Debug/C:/github/runtimelab/src/coreclr/gc/gcscan.cpp:217:9
    3: 0x10648e - WKS::GCHeap::Initialize()
                    at C:/github/runtimelab/artifacts/obj/coreclr/wasi.wasm.Debug/C:/github/runtimelab/src/coreclr/nativeaot/Runtime/../../gc/gc.cpp:48113:9
    4: 0x1178df - RedhawkGCInterface::InitializeSubsystems()
                    at C:/github/runtimelab/artifacts/obj/coreclr/wasi.wasm.Debug/C:/github/runtimelab/src/coreclr/nativeaot/Runtime/gcrhenv.cpp:173:21
    5: 0x30744 - InitDLL(void*)
                    at C:/github/runtimelab/artifacts/obj/coreclr/wasi.wasm.Debug/C:/github/runtimelab/src/coreclr/nativeaot/Runtime/startup.cpp:150:10
    6: 0x30530 - RhInitialize
                    at C:/github/runtimelab/artifacts/obj/coreclr/wasi.wasm.Debug/C:/github/runtimelab/src/coreclr/nativeaot/Runtime/startup.cpp:551:10
    7: 0x1ce1b - InitializeRuntime()
                    at C:/github/runtimelab/artifacts/obj/coreclr/wasi.wasm.Debug/C:/github/runtimelab/src/coreclr/nativeaot/Bootstrap/main.cpp:176:10
    8: 0x1eb5a - Thread::EnsureRuntimeInitialized()
                    at C:/github/runtimelab/artifacts/obj/coreclr/wasi.wasm.Debug/C:/github/runtimelab/src/coreclr/nativeaot/Runtime/thread.cpp:1226:13
    9: 0x1e953 - Thread::ReversePInvokeAttachOrTrapThread(ReversePInvokeFrame*)
                    at C:/github/runtimelab/artifacts/obj/coreclr/wasi.wasm.Debug/C:/github/runtimelab/src/coreclr/nativeaot/Runtime/thread.cpp:1188:13
   10: 0x1edb6 - RhpReversePInvokeAttachOrTrapThread2
                    at C:/github/runtimelab/artifacts/obj/coreclr/wasi.wasm.Debug/C:/github/runtimelab/src/coreclr/nativeaot/Runtime/thread.cpp:1324:28
   11: 0x1f01e - RhpReversePInvoke
                    at C:/github/runtimelab/artifacts/obj/coreclr/wasi.wasm.Debug/C:/github/runtimelab/src/coreclr/nativeaot/Runtime/thread.cpp:1338:5
   12: 0xd08464 - cswasi_wit_the_world_ExportsInterop__wasmExportFloat32Param
                    at C:\github\cs-wit-bindgen-native-lib\testing-csharp\ExportsInterop.cs:17', src\main.rs:95:13

What do think about #2364 ? There's some stuff to clean up, the empty methods, but caliing _intialiize in InitializeRuntime ?

This does appear to work for wasm components in a native lib, what they call reactor components

C:\github\cs-runtime-example-wasmtime>cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.30s
     Running `target\debug\cs-runtime-example.exe`
checking g_RuntimeInitializationCallback 0x4
calling EnsureRuntimeInitialized
checking g_RuntimeInitializationCallback 0x4
Hello from rust 2

[SingleAccretion]

@yowl I think this one can be considered fixed?

[yowl]

Yes, thanks, we implemented the problem function cabi_realloc in c to avoid initialising libc.