Repository navigation
Feature: ESM in executable files #49444
Description
Activity
There are a variety of solutions we should probably evaluate and see if they are or are not feasible that I can think of.
Right now we can point executables without extensions to other files using symlinks. I'm not curious about what file extension maps to what format, but I am curious about if that disambiguation and/or indirection method is infeasible. I'd be curious about places that cannot support that behavior. Most installations put the
.cmd/Symlink as an indirection layer to a folder and the main file for"bin"installations throughpackage.jsonso this probably falls on if the file extension can be used, whatever it maps to.In addition the executable could stay CJS and
require/import()to another file to change the mode. This also is a method of indirection, but is not relying on file extension and instead relying on w/e disambiguation method is done at runtime, even if it uses the file extension it isn't mandating such.I also am concerned about hashbang being unusable not just on linux shells but on windows. I agree it is a design space that we can look into but I am not sure how feasible it is. I do not think loaders can solve this problem using a hashbang either because they currently are not runtime configurable, nor am I convinced they should become runtime configurable due to this single issue. If we were to use a loader configuration it would need to be done outside of the file using CLI/ENV/package.json/etc. That makes me think we might want to look at indirection so that we can get that out of band data somehow.
I do have scaling concerns for having multiple executables because things like BinaryAST / WASM / WebPackage are other goals I'd like to support in the future and they would explode the number of executables that
nodeships with. I think however this format is configured and sent tonodeit should be planned around adding at least those 2 formats as well.I do not think wrapping processes are a way to solve this problem either. Notably, Windows does not have an
execcapability like unix/shells that can replace the same process ID, and usingchild_processwould create wrappers that might be problematic for various process ID tracking service manager.Forgot to add, that even though Loaders are not configurable, they could be done in a per package manner for things that are not installed using
"bin"frompackage.jsonapproaches.Not sure I follow, but I've used Linux in pretty much every dev/production env I know and written dunno how many binary files based on the hashbang
#!/usr/bin/env node.I also am concerned about hashbang being unusable not just on linux shells but on windows.
NodeJS ignores hashbang too since ever because it's been a valid use case for long time so it shouldn't probably be under discussion the ability to create nodejs executable based on ESM, but I hope I've misunderstood your reply.
@WebReflection that is executing the
nodeexecutable as the shell that invoked the file parsed the hashbang (wellenvis invoked really). My concern is around using flags within the hashbang or any form of parameter parsing beyond declaration of the executable. Even then, the declaration of the executable through the hashbang is picked up by the shell and notnodeitself.My concern is around using flags within the hashbang or any form of parameter parsing beyond declaration of the executable.
then I misunderstood, and indeed since it doesn't work in the only env I've used it (Linux) I guess we would need a better way.
I don't have ideas for now but I'll keep thinking about it.
If we want to avoid passing parameters in the shebang, we could create an excutable wrapper that passes them (
nodem,node-m).But from my point of view that would be confusing for people used to run node.
Reacted by Stuart P. BentleyThis is one of the benefits of the package.json "mode" proposal in that it can allow this scenario to be handled clearly (when executing the "bin", the package.json is loaded and used as source-of-truth of the format).
@xtuc's suggestion is an interesting alternative here too, although agreed being weary of adding to confusion.
Don’t forget the case where the extensionless file is outside the package folder, e.g.
/usr/local/bin. For example, when CoffeeScript is installed globally, e.g.npm install --global coffeescript, it puts a symlink/usr/local/bin/coffeethat points to/usr/local/lib/node_modules/coffeescript/bin/coffee(on Mac/Linux systems, anyway). Maybe in the case of followed symlinks Node can look for apackage.jsonin the folder where the symlink resolves.node by default uses the extension of the resolved file, not the symlink e.g.
file.json -> file.jsis seen as cjs not jsonFWIW @xtuc suggestion is exact equivalent of my
./esmbinary with content:#!/usr/bin/env bash node --module $1
This is a pragmatic solution to the shebang gotcha but it feels future hostile needing that in order to have an executable based on modern code so, unfortunately, we might need a better idea than just an indirection 😢
Reacted by Sven Sauleau, Stuart P. Bentley and Dude29@WebReflection whats wrong with the symlink indirection that most things do today?
@bmeck if I understand correctly that would require a
package.jsonsomewhere else, right? In AUR (or other packaging formats different from npm) that might not be easy/straight forward to implement.I just think everything is possible today with
#!/usr/bin/env nodeshould be possible by writing ESM like code too, without extra needs or hidden dependencies.@WebReflection it would rely on some file that it points to being unambiguous, yea. Not necessarily using
package.json, could be a file extension, etc.I just think everything is possible today with #!/usr/bin/env node should be possible by writing ESM like code too, without extra needs or hidden dependencies.
I think this is where all of these are breaking down. Perhaps, the way that things are done today doesn't scale well, even at 2 possibilities we are seeing problems unless we define a different mechanism than today. Wait for BinaryAST and WASM entrypoints and we have 4 possibilities.
Reacted by Charles Samborski@bmeck I don't think anything else different from JS would have the same issue for the simple reason I don't think WASM would allow a shebang on top, right?
Anyway, I forgot I have some hackery to start GJS so that this would be my solution:
#!/usr/bin/env bash Function=Function//; node -m "$0" "$@"; exit console.log('ESM');
Explanation
- the shebang is the most widely compatible one
- in bash,
Functionhas no meaning but you can assign it to anything, including slashes// - the
;after slashes ends the variable assignment and bootstrapnode -m "$0" - the rest of the arguments is passed along via
"$@"and the bash program exits (after node exits too so nothing else will be executed / parsed / interpreted from bash) - the node will execute the file as module, it will ignore the shebang and it will also ignore the assignment of the
Functionto itself, or better, there won't be any side effect, and the comment will nullify bash.
All it's missing now, is this mechanism to bypass the file extension and enable the
--modulelike parser so thatimport.meta.urlor any other ESM related syntax would be valid.Reacted by Charles Samborski49 remaining items
Being able to signal a short cli script to use esm over cjs is important. For small, general purpose, bash-like, copy-paste-able, executable scripts like, for example:
#!/usr/bin/env node import http from 'http' const host = "0.0.0.0" const port = 8000 const requestListener = function(req, res){ res.writeHead(200) res.end("Hello World") } const server = http.createServer(requestListener) server.listen(port, host, () => { console.log(`server is listening on port ${port}`) })I'd like to name this script 'hello-world' instead of 'hello-world.mjs` for it to run without error.
Reacted by Dude29@mshiltonj why? there is precisely zero about that script that requires it to be ESM, except the import - which could be
const http = require('http');. Adding a complex feature to node solely so a few users of a niche use case can use import over require seems like a tough sell to me.Reacted by Luis Marsiglia, Alex, Geferon and Brendan DrewReacted by Martin KirilovReacted by Frenco and CristopherI'm a long term user of Node.js (since
v0.4.0) and totally happy withrequire('module')syntax. It does the job fine, probably even better than ESM modules from a practical standpoint.I do however DISAGREE with your comment @ljharb. By not supporting ESM module syntax in extensionless shebang executables you are not only forcing programmers to use the CommonJS
require()syntax inside thebinfile, BUT ON EVERY SINGLE DEPENDENCY from that file up the module tree.If the problem was with a single file it would really be (as you point out) "a few users of a niche use case", but by making every dependency use CommonJS you are dooming extensionless bin files as a whole which are part of a lot of major Node.js libraries.
Just to pick a few examples:
mocha,tape,semver,uuid,he,ncp.Quoting NPM:
A lot of packages have one or more executable files that they'd like to install into the PATH. npm makes this pretty easy (in fact, it uses this feature to install the "npm" executable). To use this, supply a bin field in your package.json which is a map of command name to local file name. When this package is installed globally, that file will be linked where global bins go so it is available to run by name.
https://docs.npmjs.com/cli/v8/configuring-npm/package-json#bin
Reacted by AlexNo? You can, in a CJS bin, async import any ESM module.
Extensionless bin files I mean. If you add the extension
.mjsyou can start usingimport(or even the.jsextension with"type": "module"in thepackage.json)Sure. But in CJS, you can always use
import().await import('module')syntax inside an extensionless bin file still produces the same issue.Without
"type": "module"it kicks off an error in the dependency chain.import axios from 'axios'; ^^^^^^ SyntaxError: Cannot use import statement outside a moduleUsing
"type": "module"there is still a problem with the resolver.TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension "" for /path/to/binfileUsing
importorimport()orrequire()inside the dependencies is something I cant always control. So back to square one... 🤔 unless I understood you wrong.. and if that's the case please feel free to explain further.On the other hand, it seems like @bmeck is still dedicating brain time to this (#42301).
@sdesalas CommonJS doesn't have top-level await, so instead of this ext.mjs content:
#!/usr/bin/env node const {default: axios} = await import('axios'); console.log(axios);
You should write something like this in a noext file:
#!/usr/bin/env node (async () => { const {default: axios} = await import('axios'); console.log(axios); })();
if you have these excusable files in the same folder where node_modules/axios is installed, you won't have any issue.
Reacted by Jordan Harband, 君临 and Brandon BennettReacted by Jordan Harband@WebReflection I used an async IIFE for the await part.
The problem was not with axios on the extensionless shebanged file, but on the dependency chain from its required modules. I'll write up a working example when i have a mo.
- added a commit that references this issue
on Apr 20, 2022 Hey all, I am new to this thread, but I am in the process of upgrading from Node 12 to Node 14 and thought it would be good to convert everything to ESM. However I am running into a bunch of problems, it seems like ESM is not fully baked.
One of our major problems is getting our .bin executables to run with ESM. I have tried using .mjs I have tried type: "module" but it always seems to have some problem. Is this just not possible currently? Is the current guideline to only use CommonJS in executables? If it is possible is there a fully baked example to refer to? For example, in the package.json do I need to specify the .mjs extension in the key? The examples in the doc do not specify.
@mshiltonj why? there is precisely zero about that script that requires it to be ESM, except the import - which could be
const http = require('http');. Adding a complex feature to node solely so a few users of a niche use case can use import over require seems like a tough sell to me.Reacted by Brandon Bennett, Brendan Drew and The Jared Wilcurt
Coming from this comment: nodejs/modules#151 (comment)
It is now possible to create executable, extension-less, files:
But it's not possible to enable ESM as parsing goal.
Even if there are OS incapable to parse the whole shebang up to the
-mflag (or whatever flag will land in nodejs), there is no way to even define an executable that would like to parse the source file without any extension.Example
Given the following
esmfile, reachable through/usr/local/binor similar OS folder:And given the following executable:
It should be possible to have extensions-less files parsable as ESM.
Update
There is a solution to the single file problem that would still require
--modulehook to bootstrap the file as ESM parse goal.