Unclear how to remove and cleanup module memory in ORC2

[suedama1756]

I'm using LLVM V9.0 and cannot determine the correct way to release resources for a module in ORC2 JIT. The documentation indicates work is still ongoing on resource management, yet also says the "legacy" API is deprecated. I don't want to revert to the deprecated ORC1 and it seems overkill to create an ExecutionSession per JIT module, can anyone help?

[lhames]

Hi @suedama1756,

Releasing module resources in ORCv2 is still on the to-do list unfortunately. I hope to tackle that in the LLVM 11 timeframe.

What is your use case for removing resources? I might be able to add limited support for removal to LLVM 10, or advise you of the best work-around until we have proper resource management.

[suedama1756]

Hi @lhames

Thanks for getting back to me. In my case I am converting generic SQL like queries passed in by users into executable code. These queries are treated as subscriptions and will be run over new data as it is received. Data is received at high frequency, hence the requirement for optimal re-evaluation speed (I have benchmarked various solutions and LLVM currently provides the best performance by far here).

The query binds to many "pre-compiled" functions defined in a separate IRModule. Currently each query is placed into a new IRModule within the same dynlib as the shared code.

Users will submit/modify queries throughout the day. Ideally I would like to clean dead query module memory / resources based on some yet to be defined policy.

One idea I've been considering in place of module removal is to "switch" active ExecutionSession, e.g. copy queries that are still in use to an entirely new session, context, etc. make that the new active session and discard the old session. This will likely occur in some background thread. I'm hoping that all resources used by an ExecutionSession would be cleaned up with this approach?

I've also experimented with removing symbols from the dynlib, its not clear though whether this returns any executable memory or just the symbol registration, the code seems to suggest its just registration.

[suedama1756]

Any update on this, will the session switching approach correctly remove resources?

[lhames]

Hi @suedama1756. Sorry for the delayed reply — I’m out on vacation at the moment.

Session switching should release the resources (this sounds like the best option available right now), removing symbols from a JITDylib will not.

The two resource-release approaches I am considering are:

(1) Allowing removal of resources associated with a VModuleKey. This would allow removal on a per-module basis including freeing resources associated with that module.

(2) Providing a dlclose-like API that would free all resources associated with a JITDylib.

It sounds like your use case could be built on top of either of those in the future, allowing you to avoid manually maintaining multiple ExecutionSessions.

[suedama1756]

Thanks for getting back to me, this is what I suspected. Something that may give you ideas is the c# Expression or DynamicMethod APIs. With this API you can JIT a function on the fly, when the function is no longer references its resources are automatically cleaned. I realize this would be more complex to implement, especially without a GC, however, it would also be awesome.

I'm looking at using these techniques in my implementation: https://preshing.com/20160726/using-quiescent-states-to-reclaim-memory/ which provide GC like features.

The VModuleKey approach makes sense in the mean time as that would at least provide the same functionality as the previous JIT implementation.

[lhames]

@suedama1756 Yep. We can’t add automatic garbage collection of resources into the base API because we want to be able to JIT unmodified compiled C (e.g. from object caches). However we can aim to provide an API that automatic GC could be built on top of.

I last thought about the resource release problem a while back, and now recall my preferred solution based on the following constraints: (1) We need to be able to emulate dlclose, (2) Default behavior should be simple, (3) We want to support reoptimization naturally*. The solution was VModuleKey-based resource tracking, with a default VModuleKey supplied by each JITDylib. If you want to be able to independently remove a module from a JITDylib then you take on responsibility for holding/removing its VModuleKey. Otherwise you do nothing, the JITDylib’s VModuleKey is used, and the resources are freed when the JITDylib is closed. Implementation of this scheme will not make it into LLVM 10, but I am hopeful that it will be available in LLVM 11.

* Elaborating slightly: By re-optimization I mean generating a new, more optimized copy of a module. It can be difficult to tell in advance (1) which modules we will want to re-optimize, and (2) when the less-optimized versions will become unreferenced. If we want to free memory for the less-optimized copies we would either have to allocate a new JITDylib for each module (and play linkage games to get them to link correctly), or support VModuleKey based resource management.

[llvmbot]

@llvm/issue-subscribers-orcjit

[lhames]

The two resource-release approaches I am considering are:

(1) Allowing removal of resources associated with a VModuleKey. This would allow removal on a per-module basis > including freeing resources associated with that module.

This is now handled by ResourceTrackers. I've updated the docs with a new section, and added a new example in llvm/examples/OrcV2Examples/LLJITRemovableCode.

(2) Providing a dlclose-like API that would free all resources associated with a JITDylib.

This ended up being implemented slightly differently. We have a real dlclose-like operation in the ORC runtime, but it deliberately doesn't free resources, as that would preclude re-running our dlopen-like operation, which we don't want to do (since dlopen-dlclose-dlopen works for real dylibs, and we're aiming to have JITDylibs behave the same way). Instead we leave these operations separate: dlclose runs deinitializers (basically static destructors), but leaves the JIT'd memory in-place. JITDylib::clear frees all resources in a JITDylib. ExecutionSession::removeJITDylib runs JITDylib::clear and then puts the JITDylib in a defunct state to be destroyed as soon as all handles to it are released.

With that I think we've solved this issue, so I'm going to close this out. @suedama1756 -- if any of the docs don't answer your question feel free to reopen this (or a new issue) and let me know what could be improved.