Repository navigation
Discussion: In an ESM-first mode, how should a package.json file with no type field be handled? #49494
Description
Activity
that a new user can download Node and just start coding using ESM syntax without needing to opt into it somehow.
they already can; the file just needs the .mjs extension. If the .js extension not being ESM is a problem for that, then so is the .cjs extension not being ESM (just to forestall that rebuttal)
Thanks @GeoffreyBooth for following these threads. I've had a couple of conversations recently where people unfamiliar with JS that weren't naturally guided to the step of just setting up a
{ "type": "module" }package.json. Given that npm isn't prioritizing making this easier, and given that for a beginner it's still not an obvious path to first-class ESM, I tend to think that this is becoming important for the project to show it provides a first-class ESM experience out of the box.I also think the CLI / extensionless use case is important to finally crack, either as part of this or separately. When adopting a new top-level default it does give us the option to change the legacy compat paths to solve it.
My preference would be one of 2. (c) or 2. (d):
- c) Under a node_modules folder, type-less packages are treated as CommonJS; but are treated as ESM otherwise. This follows the precedent that the ESM resolution algorithm already special-cases folders named node_modules.
Or
- d) The “no type means ESM” behavior only applies to the package scope of the entry point.
With either, a
node --type=module cli.jsjust working as ES modules, or being able to write a new app and usenode --type=module app.js, where that flag has a future to be made the default in a future version (after subsequent deprecations, and leaving behindnode --type=commonjs appfor legacy entry points) is a very compelling case.Even if such a process takes many years, setting up the path now with a flag would be very valuable. And even if we don't get it right first time, we can at least figure out what flag works best for users by iterating on this work given it will be entirely experimental and optional. Let most users ignore it for the however many years it takes to get it stabilized, but it would very much close the loop on the ESM integration IMO.
Reacted by Geoffrey BoothThanks for the great writeup @GeoffreyBooth.
I think purposefully breaking modules that have been working untouched for years is against what our users expect from us.
It's going to break every application,npxworkflow, and generic usage of Node.js. It's going to create a massive churn for maintainers, frustrating them further and pushing them away.Here is an interesting data point: developers still use readable-stream v2.3.7 (60 millions weekly), v2.3.8 (35 millions downloads weekly), v3.6.2 (30 millions downloads weekly), while v4 is at 3 millions downloads.
Dependencies are pinned somewhere in the chain, and they do not update. The massive ecosystem in the registry is what made Node.js special: let's not burn it.If we leave the default
"type": "commonjs", shipping a ESM-first mode will require a change tonpm initthat should (ideally) get its default from the runtime itself.I think they will be open to that case, given that we are protecting their users. Worst case scenario, we can easily we can patch it like we do with V8.
We are npm primary distribution channel after all and only few users update it outside of Node.js.In other terms: let's not break everybody, please.
Reacted by liuxingbaoyu, Marco Ippolito, Stephen Belanger, Jordan Harband, Geoffrey Booth, linkgoron and Alexander PraetoriusMy preference would be one of 2. (c) or 2. (d):
I think I’m leaning toward 2-c, where typeless
package.jsonfiles undernode_moduleskeep the legacy CommonJS behavior butpackage.jsonfiles elsewhere get the new behavior. This both preserves backward compatibility for dependencies while still achieving the “do nothing to opt in” goal. I think it’s slightly preferable to 2-d, only special-casing the entry point’s scope, because of the static analyzability. It might have slightly more performance impact to check fornode_modules, but I think we’re pulling the full path of everypackage.jsonanyway and it’s trivial to see if that path contains anode_modulessegment? Which we might already be checking for already as part of the existing resolution algorithm, I’m not sure.There are two variations we could add onto this:
-
In this mode, if a
package.jsonfile is present,typeis required. So a new user starting a project and runningnpm initas it exists today and then creatingapp.jsand runningnode --new-flag-tbd app.jswould get an error that they needed to add atypefield to theirpackage.json. They do so and continue on. (Or ideallynpm initfinally gets updated, but I don’t think we should hang all our hopes on that not least because who knows how users might be initializing projects: they could run a command likenpx create-foo-appor paste in an examplepackage.jsonfrom a tutorial, etc.) -
In this mode, if the entry point triggers a SyntaxError, we print an error message like “If
app.jsis a CommonJS file, usingrequiresyntax, add"type": "commonjs"to./package.json“. This is less forceful than mandating atypefield, so the case of the new user writing ESM from scratch avoids any interruption; and it’s the same amount of friction for application or package authors upgrading CommonJS apps. We could even try to get a bit intelligent with our error, like if the error is undefinedrequirethen suggest"type": "commonjs", if it’s unrecognized keywordimportthen suggest"type": "module", etc., with a generic fallback when we can’t tell.
-
What happens in this proposal if a file, in a typeless project, is a symlink outside of node_modules, pointed to a file inside node_modules inside a typeless package dir?
Also, since type module doesn't make extensionless files ESM, would this mean that installed packages with an executable could never use extensionless ESM, only the top-level project? (i'd thought the primary motivating use case for this recent discussion was "extensionless files as ESM")
Thinking about this after a few night of sleep, a slightly better solution is to require type to be set (or at least warn to do so). Otherwise we could end up a situation that a maintainer author a module as ESM and then it's interpreted as commonjs.
Reacted by Jordan Harband and Maël NisonOtherwise we could end up a situation that a maintainer author a module as ESM and then it’s interpreted as commonjs.
It’s a question of who we want to inconvenience: the package author who forgets to test their package as imported by an app, or the new user trying to follow a tutorial and potentially not knowing what a
package.jsonfile is. Personally I think we should prioritize the new users, as package authors are by definition more experienced in the platform and should be more aware of what they need to do; and their package failing to parse is such an obvious failure that it would be noticed immediately: either by them, or the first person who tries to use their package after it’s published.Ultimately either case could be completely addressed by a package manager. It’s package managers that edit
package.json, after all, every time you run a command likenpm install foo. When runningnpm installon a package lackingtype,npmcould prompt the user “Is the code in this folder written using import/export (ES module) syntax or require (CommonJS) syntax?” and add the appropriatetypefield. That would in practice mean that the field is almost always present, for both library authors and application developers.Reacted by Lukas GaučasIt’s package managers that edit package.json, after all, every time you run a command like npm install foo. When running npm install on a package lacking type, npm could prompt the user “Is the code in this folder written using import/export (ES module) syntax or require (CommonJS) syntax?” and add the appropriate type field.
It's very unlikely we would implement something like that in Yarn. Packages are by default loaded directly from their cache archives, meaning we don't get a chance to "patch the package.json" (and it's in part by design, because we want users to use exactly the source files they've been given). Doing something like what you suggest would break this model, for what seems to be a very low gain imo.
Reacted by Jordan HarbandDoing something like what you suggest would break this model, for what seems to be a very low gain imo.
Sure, and as I’ve said earlier, I think we can’t assume any cooperation from package managers (not out of ill intent, but because they have their own goals and they’re fragmented). My point is just that package managers have a role to play in ensuring good UX too; they could fix this problem very easily if they want to. For example, on
publisha package manager could be like “hold up, you’re missing atypefield, is that an accident?” That would be a much more targeted solution to the problem of accidentally publishing a typeless ESM package, and if npm were part of the Node project then I think we would be asking why enforce something at runtime that we could enforce at publish time.Regardless, if we leave package managers out of it and focus just on Node, saving a package author from a bad release also feels like a very low gain, when the cost is making things harder for new users. (Which is a choice we need to make only if package managers do nothing.)
Both install and publish are often performed headlessly, so there's not much opportunity for prompting users there.
very easily
My point was: it would not be "very easily" 🙂
If you want to achieve the goal of allowing beginners to write
.jsscripts in ESM, I'd focus on scoping your research to this exact goal (ie only apply this logic if there isn't a package.json at all). Changingtypesemantics in any way is imo guaranteed make the migration to ESM more frustrating to people who actually work on Node.js projects, I don't imagine that as a good tradeoff.For example, on publish a package manager could be like “hold up, you’re missing a type field, is that an accident?”
That's something I'd be more supportive about, since it doesn't break the existing ecosystem, only affecting future packages in a consistent fashion.
That’s something I’d be more supportive about, since it doesn’t break the existing ecosystem, only affecting future packages in a consistent fashion.
I think part of this is that we’re just guessing how package managers respond; and how new users will act. But we don’t need to get it right on the first try: it’s an experimental flag. We can start with the “typeless = ESM” behavior at first and change it to the “type required” behavior later, before unflagging, once we see if (or how) package managers adapt to the existence of this new mode. I’m not strongly opposed to making Node require the
typefield, but it feels like a hammer to a solution that needs a mallet, and something that hopefully we won’t need at all if package managers find a way to ease this UX pain point. (And yes, they’re the ones best positioned to decide on that solution if there is one, however easy or hard it might be 😄 )This issue has been marked as stale due to 210 days of inactivity.
It will be automatically closed in 30 days if no further activity occurs. If this is still relevant, please leave a comment or update it to keep it open.This issue has been automatically closed after 30 days of inactivity following its stale status (no activity for a total of 240 days).
If this is still relevant, feel free to reopen it or leave a comment with additional details so we can continue the discussion.
Building off of #49432, #49295 (comment) and #31415, we’re considering a new mode, probably enabled by flag, where all of the current places where Node defaults to CommonJS would instead default to ESM. One of the trickiest questions to answer for defining such a new mode is how to handle
package.jsonfiles that lack atypefield.A
package.jsonfile, whether or not it contains atypefield, defines a “package scope”: the folder that thepackage.jsonfile is in, and all subfolders that don’t themselves contain apackage.jsonfile. Within this package scope, currently apackage.jsoncontaining"type": "module"will cause.jsfiles to be interpreted as ES modules; apackage.jsonfile containing"type": "commonjs"or no"type"field will cause.jsfiles to be interpreted as CommonJS modules.In a naïve “just flip all the defaults” implementation, where one literally goes through the Node codebase and symmetrically reverses everywhere that we default to CommonJS to instead default to ESM, a
package.jsonlacking atypefield would cause all the files in that scope to be treated as ES modules. The problem with this is that there are lots of popular dependencies that people install that have notypefield. Most real-world apps would fail to run in an ESM-first mode that behaved this naïve way, because it would be rare for every dependency of an app to contain apackage.jsonwith atypefield.To make an ESM-first mode that’s actually usable, we need to find a solution for this problem. As I see things, the solutions fall into two categories: one where we preserve the pure symmetrical “no
type= ESM” behavior, and one where we don’t. Here’s a running list that I’ll update if people suggest additional ideas:1. Preserving symmetry: no
typefield is interpreted as ESMa. After installing packages, users would run a script that patched any dependencies’
package.jsonfiles to add"type": "commonjs"wherever thetypefield wasn’t specified.npmteam has done in recent years around reproducible builds and immutable packages. Ideally a patch script such as this would be part of thenpm installcommand and its equivalents in other managers, but it’s probably unlikely that we would get support for such an approach from many (or any?) of the popular package managers.b. After installing packages, users would run a script to warn them if any of their packages lack a
typefield. (Or as part of the package installation command, the command would error on attempting to install anytype-less package. This would presumably be an option that users would enable.) Then users would presumably uninstall that package in favor of some alternative (or choose to patch it).typefield added, even if just"type": "commonjs", which it’s probably safe to assume would rankle many package authors. Besides those authors complaining that we’ve pushed a requirement onto them, they may reasonably argue that atypefield makes no sense for packages intended for non-Node environments such as browsers.type-less package would need to be patched or upgraded to the latest version (assuming the author has kindly published a new version with the field). This option creates a lot of friction for both users and package authors.c. The
npmregistry starts requiring thetypefield in order for packages to be published, just as they already require thenameandversionfields.npmfolks agree to this, as there are countless packages not intended for use in Node, and it would be unclear whattypefield value those packages should have; andnpmsurely doesn’t want to put themselves in the position of getting many package authors angry at them.typefield, as thenpmregistry strictly disallows old packages to be modified.2. Different behavior in ESM-first mode
Assuming that no workable option for preserving symmetry is found, the question then becomes “okay, now what?” Here are some options, that I’ll update as people comment:
a. Keep current behavior. All
type-less package scopes are still treated as CommonJS.b. Error on
type-less packages.typefield. It presents the same problems: users would need to patch, or pressure the package authors to update their packages.c. Under a
node_modulesfolder,type-less packages are treated as CommonJS; but are treated as ESM otherwise. This follows the precedent that the ESM resolution algorithm already special-cases folders namednode_modules.npm init, assuming it never changes, and still write ESM code for their app without needing to enable it somehow; and the user couldnpm installany dependency which should “just work” like today.npmmight need to adjust to support this behavior, if they don’t save packages in subfolders undernode_modules. Package managers that save packages in a cache folder might create anode_modulesfolder at the root of their cache and put the packages one level down inside that, for example. But some package managers might have trouble working with this behavior.d. The “no
typemeans ESM” behavior only applies to the package scope of the entry point. So in anappfolder withapp/package.json(that lacks atypefield) andapp/entry.js, runningnode --experimental-flag-for-esm-first-mode-name-tbd entry.jswould interpretentry.jsand any other files in that package scope as ESM, but all other package scopes anywhere on the disk would be interpreted as CommonJS.npmpackage managers, but at the cost of the app possibly not being statically analyzable by tools. It would be ambiguous whether a particular package scope will be the one that an entry point uses and would therefore be acquiring this new behavior, unlike “is it under a folder namednode_modules“ which is easily determined by external tools.Any other ideas? Or additional pros/cons to any of these suggestions. @LiviaMedeiros @nodejs/loaders @nodejs/wasi @nodejs/tsc