Add option to inline webworker

[VirtualTim]

Currently when building the USE_PTHREADS Emscripten creates a separate module.worker js file.
This is loaded from module.js like:

var pthreadMainJs = "module.worker.js";
pthreadMainJs = locateFile(pthreadMainJs);
var newWorkers = [];
for (var i = 0; i < numWorkers; ++i) {
  newWorkers.push(new Worker(pthreadMainJs));
}

It is also possible to create an inline webworker, doing something like:

let workerBlob = new Blob(['string_containing_minified_module.worker.js'], type:"application/javascript"});
pthreadMainJs = URL.createObjectURL(workerBlob);
var newWorkers = [];
for (var i = 0; i < numWorkers; ++i) {
  newWorkers.push(new Worker(pthreadMainJs));
}

I imagine that it would be simple enough to minify module.worker.js, then embed that into the main module.js.

I believe that this will speed up loading threads, and should also make webpacking pthreads builds easier/possible.

[kripken]

I'd be interested to know if this helps overall load. It would make the initial download slower, but maybe it's worth it later? The webpack benefits may be worth it, though.

A downside is that this makes debugging harder (can't modify the source) so we have to keep the current approach around, which means this would be a new option.

[VirtualTim]

A downside is that this makes debugging harder (can't modify the source) so we have to keep the current approach around, which means this would be a new option.

Agreed.

It would make the initial download slower

Applying some very basic minification (just whitespace+comments) added 3.13KB to the JS. I'm sure that can be reduced further by smarter minification.

[VirtualTim]

I took a brief look at this, and here's my idea so far:
Settings.js:

//add option:
var INLINE_WEBWORKER = 0;

shared.py:

def generate_inline_webworker(worker_filename):
  webworker = js_optimizer_no_asmjs(worker_filename, return_output=True, extra_info=None, acorn=True)
  return webworker

emcc.py from around here:

emscripten/emcc.py

Line 2374 in 6d297f6

if shared.Settings.USE_PTHREADS:

with ToolchainProfiler.profile_block('memory initializer'):
  if shared.Settings.USE_PTHREADS:
    ...
    with open(worker_output, 'w') as f:
      f.write(shared.read_and_preprocess(shared.path_from_root('src', 'worker.js'), expand_macros=True))
    if shared.Settings.INLINE_WEBWORKER:
      worker = shared.generate_inline_webworker(worker_output)
      # construct js:
      src = ''' //something like
var workerBlob = new Blob(['%(worker)'], type:"application/javascript"});
pthreadMainJs = URL.createObjectURL(workerBlob);
''' % {'worker': worker}
    else:
      # construct js:
      src = '''
var pthreadMainJs = "module.worker.js";
pthreadMainJs = locateFile(pthreadMainJs);
'''
    #insert constructed js (src) into Module
    #I'm not sure how to do this

Does this seem like it's on the right path?
What's the best way to do macro substitution in Module.js?

[kripken]

By macro substitution do you mean the JS preprocessor? There's a helper function read_and_preprocess in shared.py which does that I think.

However I'd recommend measuring this first to see how beneficial it is - I'm not sure it's worth an extra option yet, but I'm curious to see some data.

[VirtualTim]

I guess my motivation was more about making pthreads builds easier/possible to webpack.
I'll investigate more first if it fixes the issue, and then also do some profiling.

[stale]

This issue has been automatically marked as stale because there has been no activity in the past year. It will be closed automatically if no further activity occurs in the next 30 days. Feel free to re-open at any time if this issue is still relevant.

[jeffRTC]

@VirtualTim

I badly need this for a restricted environment where external files are not allowed. It's like the -s SINGLE_FILE=1 option.

[jeffRTC]

@VirtualTim

I just created a NodeJS script to inline the worker and it works, but would be better to add it to compiler it-self.

const fs = require('fs');
const args = process.argv;

function escapeIt(string) {
    return string.replace(/[\`]/g, '\\`');
}

// MyModule
let moduleName = args[2];

// C:/runtime.js
let runtimeScriptFilePath = args[3];
// C:/runtime.worker.js
let workerScriptFilePath = args[4];

let runtimeScript = fs.readFileSync(runtimeScriptFilePath, "utf8");
let workerScript = fs.readFileSync(workerScriptFilePath, "utf8");

let target = `locateFile("${moduleName}.worker.js")`;
let replace = `URL.createObjectURL(new Blob([\`${escapeIt(workerScript)}\`], { type: "application/javascript" }))`;

let result = runtimeScript.replace(target, replace);

fs.writeFileSync(runtimeScriptFilePath, result);

[VirtualTim]

I'm glad you've got something working.
Personally I've long abandoned that idea, it didn't help with my bundling issue, and made start-up significantly slower.

I know of one other project that's doing something similar, so I still think there's value of adding this to Emscripten. I'm just not going to be the one to do it.