Feature: ESM in executable files

[WebReflection]

Coming from this comment: nodejs/modules#151 (comment)

It is now possible to create executable, extension-less, files:

#!/usr/bin/env node
console.log(__filename);

But it's not possible to enable ESM as parsing goal.

#!/usr/bin/env node -m
console.log(import.meta.url);

Even if there are OS incapable to parse the whole shebang up to the -m flag (or whatever flag will land in nodejs), there is no way to even define an executable that would like to parse the source file without any extension.

Example

Given the following esm file, reachable through /usr/local/bin or similar OS folder:

#!/usr/bin/env bash
node --module $1

And given the following executable:

#!/usr/bin/env esm
console.log(import.meta.url);

It should be possible to have extensions-less files parsable as ESM.

Update

There is a solution to the single file problem that would still require --module hook to bootstrap the file as ESM parse goal.

[bmeck]

There are a variety of solutions we should probably evaluate and see if they are or are not feasible that I can think of.

Right now we can point executables without extensions to other files using symlinks. I'm not curious about what file extension maps to what format, but I am curious about if that disambiguation and/or indirection method is infeasible. I'd be curious about places that cannot support that behavior. Most installations put the .cmd/Symlink as an indirection layer to a folder and the main file for "bin" installations through package.json so this probably falls on if the file extension can be used, whatever it maps to.

In addition the executable could stay CJS and require/import() to another file to change the mode. This also is a method of indirection, but is not relying on file extension and instead relying on w/e disambiguation method is done at runtime, even if it uses the file extension it isn't mandating such.

I also am concerned about hashbang being unusable not just on linux shells but on windows. I agree it is a design space that we can look into but I am not sure how feasible it is. I do not think loaders can solve this problem using a hashbang either because they currently are not runtime configurable, nor am I convinced they should become runtime configurable due to this single issue. If we were to use a loader configuration it would need to be done outside of the file using CLI/ENV/package.json/etc. That makes me think we might want to look at indirection so that we can get that out of band data somehow.

I do have scaling concerns for having multiple executables because things like BinaryAST / WASM / WebPackage are other goals I'd like to support in the future and they would explode the number of executables that node ships with. I think however this format is configured and sent to node it should be planned around adding at least those 2 formats as well.

I do not think wrapping processes are a way to solve this problem either. Notably, Windows does not have an exec capability like unix/shells that can replace the same process ID, and using child_process would create wrappers that might be problematic for various process ID tracking service manager.

[bmeck]

Forgot to add, that even though Loaders are not configurable, they could be done in a per package manner for things that are not installed using "bin" from package.json approaches.

[WebReflection]

Not sure I follow, but I've used Linux in pretty much every dev/production env I know and written dunno how many binary files based on the hashbang #!/usr/bin/env node.

I also am concerned about hashbang being unusable not just on linux shells but on windows.

NodeJS ignores hashbang too since ever because it's been a valid use case for long time so it shouldn't probably be under discussion the ability to create nodejs executable based on ESM, but I hope I've misunderstood your reply.

[bmeck]

@WebReflection that is executing the node executable as the shell that invoked the file parsed the hashbang (well env is invoked really). My concern is around using flags within the hashbang or any form of parameter parsing beyond declaration of the executable. Even then, the declaration of the executable through the hashbang is picked up by the shell and not node itself.

[WebReflection]

My concern is around using flags within the hashbang or any form of parameter parsing beyond declaration of the executable.

then I misunderstood, and indeed since it doesn't work in the only env I've used it (Linux) I guess we would need a better way.

I don't have ideas for now but I'll keep thinking about it.

[xtuc]

If we want to avoid passing parameters in the shebang, we could create an excutable wrapper that passes them (nodem, node-m).

But from my point of view that would be confusing for people used to run node.

[guybedford]

This is one of the benefits of the package.json "mode" proposal in that it can allow this scenario to be handled clearly (when executing the "bin", the package.json is loaded and used as source-of-truth of the format).

@xtuc's suggestion is an interesting alternative here too, although agreed being weary of adding to confusion.

[GeoffreyBooth]

Don’t forget the case where the extensionless file is outside the package folder, e.g. /usr/local/bin. For example, when CoffeeScript is installed globally, e.g. npm install --global coffeescript, it puts a symlink /usr/local/bin/coffee that points to /usr/local/lib/node_modules/coffeescript/bin/coffee (on Mac/Linux systems, anyway). Maybe in the case of followed symlinks Node can look for a package.json in the folder where the symlink resolves.

[devsnek]

node by default uses the extension of the resolved file, not the symlink e.g. file.json -> file.js is seen as cjs not json

[WebReflection]

FWIW @xtuc suggestion is exact equivalent of my ./esm binary with content:

#!/usr/bin/env bash
node --module $1

This is a pragmatic solution to the shebang gotcha but it feels future hostile needing that in order to have an executable based on modern code so, unfortunately, we might need a better idea than just an indirection 😢

[bmeck]

@WebReflection whats wrong with the symlink indirection that most things do today?

[WebReflection]

@bmeck if I understand correctly that would require a package.json somewhere else, right? In AUR (or other packaging formats different from npm) that might not be easy/straight forward to implement.

I just think everything is possible today with #!/usr/bin/env node should be possible by writing ESM like code too, without extra needs or hidden dependencies.

[bmeck]

@WebReflection it would rely on some file that it points to being unambiguous, yea. Not necessarily using package.json, could be a file extension, etc.

[bmeck]

I just think everything is possible today with #!/usr/bin/env node should be possible by writing ESM like code too, without extra needs or hidden dependencies.

I think this is where all of these are breaking down. Perhaps, the way that things are done today doesn't scale well, even at 2 possibilities we are seeing problems unless we define a different mechanism than today. Wait for BinaryAST and WASM entrypoints and we have 4 possibilities.