Option to provide the .wasm file's ArrayBuffer instead of library fetching it

[manzik]

First of all, thank you for your great library, I love it! Great job maintaining it and keeping it up to date I'm relying on it heavily.
So hats off!

I was wondering if you can add an option where I could pass in a pre-fetched ArrayBuffer of wasm file and pass that instead of the folder where the wasm file is located.

My use case: I'm using the library to process multiple graphviz source files and get output in parallel (using web workers). And this has decreased my code's runtime from 4 seconds to 1 second on my machine's localhost.

The issue is that if I have 4 cores, each worker will download the wasm file separately and spend ~400ms redoing a work that can be easily done just once for all of them, and that is on localhost. This number will be much higher in production and waste a lot of resources.

So if you add an option for passing typed array or ArrayBuffer that can be avoided.
I also have to mention it's impossible to pass around all kinds of objects between worker and master processes and they have to be clonable which is why I would need to pass an ArrayBuffer or a data type like that and passing library's objects around wouldn't work.

Pretty sure being able to fetch the wasm file's data and reuse them whenever needed could be useful in some other cases too.

Thanks again!

[GordonSmith]

A lot of the wasm loader code is embedded inside an emscripten generated "module" so I don't have a lot of control over it, my entry point is here: https://github.com/hpcc-systems/hpcc-js-wasm/blob/trunk/src/util.ts#L30-L38

The instance that comes back is a JS Class with the "asm" as a member (which is the bit you ultimately want cache).

Let me walk the emscripten code and see if there is a way to "pre load" the asm so that emscripten skips the loading step ?

[GordonSmith]

If you can add a PR with a web worker test case, I will investigate further...

[manzik]

Should I write a simple code that demonstrates the situation where I need this feature (with web workers), add the code to a new folder called test and make a pull request to this repo with that folder added?
Is that what you mean? Or did I get it totally wrong haha

[GordonSmith]

Yea thats fine, just something I can look at to see where the 400ms is going...

[GordonSmith]

FWIW looks like I can pass in a cached UInt Array if I have one, but I can't work out how to get a raw one from the generated stubs...

[manzik]

Ok so I just turned caching back on for the wasm files and that returns "304 - not modified" for the wasm files when the workers try to download them so now the time wasted for that is down to ~40ms (in parallel for all workers) which is fine by me. So I guess it works fine if I turn on the caching for the wasm files (hopefully the trend will be same in production).

So it would be fine for me to close this ticket for now. I'll reopen the issue in case this doesn't work out great on the actual server.
Unless you think this feature could be useful for others? In which case we can keep this open and I'll write some simple test code for web workers with graphviz and open a PR.

[GordonSmith]

That is what I would have expected to see (the browser caching the raw wasm file).

FWIW what I could do (if it was deemed worth it) is expose a "wasmBinary" option in the "extra params" section and prime the loader to use it when provided. The onus would be on the user to wget, fetch or XHR the wasm themselves and pass it in as buffer.

[manzik]

That would be awesome.
Btw I added a PR for the test cases (#54).
Let me know if I can somehow help.

[GordonSmith]

This is what my simple test case looks like:

        fetch("./dist/graphvizlib.wasm", { credentials: 'same-origin' }).then(response => {
            if (!response.ok) {
                throw "failed to load wasm binary file at '" + wasmBinaryFile + "'";
            }
            return response.arrayBuffer();
        }).then(wasmBinary => {
            hpccWasm.graphviz.layout(dot, "svg", "dot", { wasmBinary: wasmBinary }).then(svg => {
                const div = document.getElementById("placeholder4");
                div.innerHTML = svg;
            }).catch(err => console.error(err.message));
        });

If that works for you, then I will issue a PR.