Require cache equivalent in es modules

[tcodes0]

• Node.js Version: 14.4
• OS: macos 10.14.6
• Scope (install, code, runtime, meta, other?): modules
• Module (and version) (if relevant):

See this SO question. https://stackoverflow.com/questions/51498129/require-cache-equivalent-in-es-modules

My use case: With cjs we can use the require object and its props to invalidate a module's cache so a new require loads it fresh. Can't seem to find a way to do the same with esm. Context: repl code reloading applications.

related dwyl/decache#51

[devsnek]

The esm cache is not exposed, and even if it were, it is not mutable.

[guybedford]

@Thomazella this is clearly a missing piece of the integration, so it's worth thinking about and there are a few ways these use cases can be approached.

The simplest today would be to create a new VM Module (@devsnek's work!) environment for each unique context. The issue here is clearing the registry means clearing the entire context. Realms might touch on this in a similar way in future.

More fine-grained reloading of modules could be supported by permitting registry-level deletion of the cache certainly. One could imagine a require('module').deleteLoaderModule(id) or similarly that purges individual modules from the cache so they are forced to be loaded fresh next time. And this is also what the SystemJS loader permits via System.delete.

Node.js is in a different position to the browser on this problem because it has the driving need first, but the difficulty with this stuff is that Node.js is so used to following the browser and other specs, it's difficult to pick up the batton on this one and lead the spec and integration work which is what we really should be doing for this problem.

If someone were interested in a PR for deleteModule or similar as described above I would personally be all for merging that. There are complexities and wider concerns that come up with this work but it does just need someone to pick it up and work through the considerations, including possibly engaging at a v8 / spec level.

[devsnek]

I would be against such an API being added. Either the use case is hot reloading and should be brought up with V8 or the code should be rewritten to not need to be reloaded.

[guybedford]

@devsnek this is not a controversial feature, it is just a tough one! Smalltalk had this as a basic feature, it is hardly novel or concerning.

Bringing up hot-reloading with v8 sounds like a very fruitful direction to me. One could even imagine a hot-reloading API that pushes out the live binding updates (like Smalltalk offered too).

[devsnek]

v8:10476 has info about the removal of live edit and may be a good place to express a want of hot reloading.

[guybedford]

To be clear I personally do not have resources to drive this but am glad to engage where I can.

[tcodes0]

@devsnek could you please point me to some issues, discussion or code to understand better the way forward here? I'm mainly missing why exposing the cache would be bad and why v8 seems the best alternative.

Can't seem to find the v8 issue mentioned too. Is it hosted somewhere else than v8/v8?

[vilicvane]

Is there any undocumented way to get the ESMLoader instance? I am looking for ways to get a list of loaded modules.

The answer is yes, check out https://stackoverflow.com/questions/63054165/is-there-any-way-to-access-an-internal-node-js-module

[rlnt]

I am still searching for a good way to do this.
What I am doing right now is cache-busting which causes a memory leak but it's better than rewriting everything.
I rely on the "type": "module" option.
https://ar.al/2021/02/22/cache-busting-in-node.js-dynamic-esm-imports/

[devsnek]

to be clear to anyone who wanders across this issue: there is no implementation node.js could provide that would not leak memory. if you wish for such a thing, please direct that to V8.

[daKmoR]

is there somewhere a V8 issue already? 🤔

[dnalborczyk]

as a potential alternative, depending on the use case, worker threads could be facilitated. if a module is being hot [re]-loaded] in a worker thread itself, if handled correctly, there will be no memory leak. for the reload process itself, the worker thread would have to be terminated.