Would anyone be interested in an article about the crusade to move JS to ESM? I've been considering writing one, here's a preview:
Sindresorus wrote a gist "Pure ESM modules"[0] and converted all his modules to Pure ESM, breaking anyone who attempted to `require` the latest versions of his code; he later locked the thread to prevent people from complaining. node-fetch released a pure ESM version a year ago that is ~10x less popular than the CommonJS version[1]. The results of these changes broke a lot of code and resulted in many hours of developers figuring out how make their projects compatible with Pure ESM modules (or decide to ignore them and use old CommonJS versions)--not to mention the tons of pointless drama on GitHub issues.
Meanwhile, TC-39 member Matteo Collima advocated a moderate approach dependent on where your module will be run [2]. So the crusade is led not by the Church, but by a handful of zealots dedicated to establishing ESM supremacy for unclear reasons (note how Sindresorus' gist lacks any justifications and how weak TFA's justifications are). It's kind of like the Python 2 to 3 move except with even less rationale and not driven by the core devs.
ESM was a part of es2015. It's been 8 years that we've had to tangle with both cjs & esm. It's been absolutely awful for everyone.
This crusade is nowhere near zealous nor righteous enough against the infidels & non-believers.
But it also hasn't been effective enough at supporting/supplying the crusade either.
Matteo's statement was that Node hasn't stabilized their loader support so tools have a harm time migrating to esm. Imo it's a pity ecmascript never stabilized a module registry, that esm 1.0 shipped & most people thought it would happen; it's long felt like a bait & switch. But it wasn't a feature browsers needed or really wanted so that unfulfillment was unsurprising. Anyhow, IMO Matteo is making a technical point that it's still hard to finish the move, which is a different spin IMO than a "advocated a moderate approach".
Given the hurt we legitimately experience, I really wish Node and/or WinterCG or someone would prioritize figuring out & implementing whatever needs to go into a module registry/loader. And then beg the big tool chains that need this stuff to expedite their migrations, pretty pretty please.
It only became awful for me when people started publishing pure ESM packages; `npm i node-fetch` suddenly began resulting in broken scripts and I had to learn why. Prior to that, I happily used CJS outside of the browser and what I suppose is ESM in the browser (the `import` syntax provided by bundlers).
> a different spin IMO than a "advocated a moderate approach".
He said, "If your module target the Browser only, go for ESM. If your module target Node.js only, go for CJS."
This is moderate compared to the "Pure ESM" approach. The fetch API is built into browsers so I don't see why anyone would use `node-fetch` outside of Node.js, and yet the maintainers of `node-fetch` went Pure ESM anyway. Also that GitHub issue is titled "When will CommonJS modules (require) be deprecated and removed?" and his response was "There is no plan to deprecate CommonJS"[0].
I mean, I took Sindre's move to ESM-only publishing as his own choice as the author and maintainer of his own OSS packages. In the spirit of OSS, he doesn't need to justify the choice, it's good enough that as the author he's making it. The previous CJS versions are still published, and anyone can fork the modules if they want to take on the maintenance burden of the CJS version.
That preference has had an impact on the Node ecosystem at large, given how prolific an OSS contributor Sindre is, but IMO that influence has been earned by the large body of work he's contributed.
And yet his older CJS versions get weekly downloads that are magnitudes higher than the newer ESM versions.
I'm working on greenfield projects that leadership is still insisting we avoid ESM-only packages for. The move that Sindre (and others like wooorm) made has not been well-received.
This is irresponsible approach and this package should be forked by more responsible people. Person who made this change should be avoided in the future.
I'm all for ESM but alienating your own users is stupid. Create node-fetch-esm and support both versions until commonjs popularity would be low enough to be dropped.
That's all well and good, but what about those of us that already have a mountain of Jest tests with mocks that aren't supported in ESM mode? https://github.com/jestjs/jest/issues/9430
I've definitely worked around my fair share of CommonJS issues but until ESM "just works" I'm slightly pained by how aggressive the tone of this article is.
It's a problem with Jest and CommonJS only in so far as it's a problem with mocking in general: native ESM imports are read-only. You can roll your own dependency injection or use tools that parse/rewrite your imports at runtime, but this is hardly a good solution. I think the parent's comment still stands: there's a big gap remaining.
Are you saying I just shouldn't have access to that part of the runtime? I've also done my fair share of `require` hacking, and it's led to things like hot reload and lazy loading for Node backends. There's a lot of value in being able to mess with that part of the stack.
This is exactly my beef. The testability with commonjs is great, being able to easily mock dependencies is a huge boon for testing. I'd have no issue using only ESM if it had feature parity, or at least some way to achieve dynamic loading.
Unfortunately, migrations are necessary when major releases occur. At some point, the argument to cling onto old tech which holds back the entire ecosystem needs to be deprioritized. Years have passed already so it should not be a surprise when the ecosystem moves on.
Re mocks, an ecosystem should not be held back solely due to an arcane edge case. The apps that use test doubles can be rearchitected to support test doubles to the programmers' satisfaction. Most people do not use test doubles & the benefits of ESM & not having to deal with CJS outweigh the downsides of losing some convenience in mocking modules.
This is something that will not be popular with clingers to CJS, but it is something that will benefit the wider ecosystem. The vocal minority which is holding back the ecosystem should start making plans to migrate because it is in the process of happening right now...
So I think the general resentment is having to do extra work to support CJS which is not standard across all JS platforms...and some legacy libraries are still written in CJS requiring an interop. So it would be great to not have to do this extra work to support the legacy CJS on Node.js when every other JS platform is using ESM. In this case, one person's convenience comes at a cost to everyone else & at some point, that one person is going to have to suck it up & do the work to support his use case...just as everyone else had to do the work to support the legacy CJS for years now.
I can't tell you your perception is wrong, but I generally don't unload ornate rhetoric like "insidious saboteur" or verbs that relate to strong imagery like "rip out", "bury" when writing about code unless I intend for my tone to be aggressive.
My code works fine and the article isn't doing itself any favors by mocking me (heh) for thinking so.
Didn't see mention of Browserify and other bundlers after it that made CommonJS the defacto standard for client/browser libraries as well.
I think the biggest miss was not making mixed mode (default) for Node do it the way webpack/babel, etc did it by default in terms of interop. I get they wanted to make it more implicit to call cjs from esm, in the end it just inhibits conversion of existing libraries as dependencies are now a bigger hurdle.
Frankly, I like the Deno way of things better. I find it annoying, to say the least that the TypeScript team won't consider allowing you to import with a .ts(x) extension, and convert to .js(x) as part of the build process... no, you must omit the extension.
I've been using the import/export syntax since well before it was standardized via babeljs, these days I kind of want to remove webpack/babel from my pipelines altogether and mostly just rely on esbuild. I've also been using/following rome.tools development, having switched over several projects from eslint already, and will probably start with their bundler when it's ready.
I think there's a way to go with tree shaking and static analysis in that direction to reduce load. I also would not mind seeing the efforts to treat TS extensions as comments in the JS engines in that it would be easier to serve up straight TS/JS without bundling/minifying. I'm not sure we'll ever see a return to that in practical terms.
In the end, it's evolving. I'd also like to see Cloudflare, Deno and others come together on a more common server interface so that you can target multiple clouds with a single codebase. I don't know how well that would ever work out at this point though. There's aspects that I definitely like to all of them.
> I think the biggest miss was not making mixed mode (default) for Node do it the way webpack/babel, etc did it by default in terms of interop. I get they wanted to make it more implicit to call cjs from esm, in the end it just inhibits conversion of existing libraries as dependencies are now a bigger hurdle.
Huge huge agreement.
I forget the specifics but there was some super tiny corner case around maybe default exports that could potentially create ambiguity & that spawned a multi-year bellyaching around doing anything at all for interop. What Node got was incredibly hard fought for against much resistance to interop.
But the final compromises made everything so much more painful for everyone. So many esm projects but oh look a .eslintrc.cjs, how unsurprising & sad.
It's extra maddening because node had a wonderful just works (except that tiny tiny tiny corner case) interop via @standard-things/esm, which seamlessly let the two worlds interop. It'd been around for years before node started shipping support, and it was no ceremony just works bidirectional interoperability, and it took basically no effort or thought from the developers point of view to use. It sucked seeing us walk back from great, mired by frivolous over concern for a obscure corner-case.
Does anyone have a detailed understanding of why CommonJS (and its async incarnation, AMD) were not adopted by browsers?
I do much like the `import` syntax personally and its a little cleaner to read, but CommonJS and AMD were the undisputed winners of the module format until ES Modules were born. Not that I have a problem with ES Modules, I don't, however I am interested in what was so insufficient about the preceding formats that we couldn't have standardized on them
EDIT: I know about the deal with CommonJS being synchronous. That isn't per se an issue I don't think, esp. because AMD built on top of CommonJS primitives, and with minimal refactoring CommonJS code could be used in the browser when defined this way if asynchronicity is a must. Generally, what I "imagine" browsers doing with CommonJS is making the `require` calls async in the background (IE non visible to developers) so they can resolve the modules then parse the code. This isn't terribly different from how import statements work today.
I'm wondering why we didn't undertake the work to just improve the existing format, more or less.
EDIT 2: I'm interested from a historical perspective. I think ESM is the right choice and 100% the future.
I can speak to the one listed in the article: "difficult to tree-shake, which can remove unused modules and minimize bundle size."
This is because of a much deeper issue: static analysis is highly complex with the near-free-for-all that is CommonJS require & module.exports syntax. ES Modules is stricter and much easier to statically deal with.
At a high level, why? You can throw just about anything in an exports.module statement, and the syntax to "require" it also has a lot of leeway. You can actually see the code for this in the Node codebase--module resolution is handled in javascript @ /lib/internal/modules/cjs/loader.js
vs /lib/internal/modules/esm (heads up, both approaches are a Lot to grok)
Understand that with the CJS approach, you can dynamically export modules at runtime under whatever name you wish, with whatever value you want, which may even include dynamic require statements themselves. Nightmare for static analysis.
It makes a lot more sense if you try it for yourself. Build a module resolution algorithm including: determining all the imports, all the files those imports are from, mixing with 3rd party and local imports, and building that chain recursively.
You can do it, but the edge cases surrounding CommonJS make it super difficult. I'd go so far as to say it's basically impossible to get 100% success in all the desired scenarios without directly invoking the code.
While I agree the dynamic nature of CommonJS would be problematic, there were successful projects around treeshaking commonjs[0] that worked really well.
I think dynamic imports have some of the same footguns here, to be honest. Can't deny ESM is easier to statically analyze though, that much appears to be true across the board based on available evidence.
To be honest, I think the AMD incarnation is a complete non-starter. It’s just such a funky, weird little thing that only makes sense because it’s a compatibility shim. Nobody wants to directly author AMD, and someone shipping a JS implementation wants to ship features that people will use directly.
I mean, I guess people will directly write AMD modules, and make modules using some giant script that uses cat, but the future of JavaScript lies with making each source file a valid, correct piece of JavaScript. When each source file is valid and correct, and doesn’t need to be preprocessed in order to work, your tooling will work a lot better.
The browser authors know you can’t un-ship JavaScript features. ES6 import/export is damn good stuff, and people in the browser aren’t saddled with some weird compatibility shim like AMD.
The adoption of ES6 modules in the client-side landscape has far outstripped its adoption in Node.js. I honestly can’t wait for require() to die, in both its cjs and AMD variations. The tooling support for ES6 modules is miles better.
+this ... AMD was just weird and clunky to use in practice... CJS bundlers were much easier to grasp by comparison, and when browserify (and those that followed) came out, it was kind of a no-brainer at the time. If ESM were finalized maybe even 2 years earlier, we'd probably be using that already for everything. I think the Node team choosing to make esm/cjs interop more difficult than what babel had been doing slowed down the switch. I get the reasons why, I just don't agree with the approach in the end. I think if they made the interop good, and declared after Node v#, it would be esm only, that would have worked out better for everyone. The risk being a kind of Python 2->3 paralysis. If the interop was good for a few versions, I don't think the friction would have been that bad. Then after 2-3 years, the switch could have been much cleaner.
CommonJS requires invoking the code before the modules can be resolved, versus the ESModule syntax with "import" can be parsed out of the code separately (from the AST because it is a keyword). No invocation required.
I don't know if that's the entire story -- probably not -- but I do know that is one major differentiator for things like generating import-graphs and performing tree shaking.
(you can still do like `import('foo' + someVar)` which will only invoke dynamically at runtime, so I'm not sure how that case is dealt with)
> (you can still do like `import('foo' + someVar)` which will only invoke dynamically at runtime, so I'm not sure how that case is dealt with)
That case is dealt with more like a `fetch('foo' + someVar).then(r => eval(r.text()))` or similar (but of course it is not just a eval and it instead returns the exports of the module).
Dynamic imports and static ones behave very differently and static analysis generally ignores dynamic imports IIRC.
You also need to treat dynamic imports as async including everything that comes with that (error checking, awaiting, etc.)
From what I remember, reading the conversations over the years...the issue was twofold:
1. Because `require()` is "just a magic function", it can't be statically analyzed by a JS runtime prior to actually running the code. This leads to limitations with regards to tree-shaking and other optimizations.
2. The last point leads to the even bigger (and probably "deal-breaker") reason for the change, the desire to fetch packages from URL sources. Since the syntax cannot be parsed efficiently, runtimes like Deno and Bun would have a much harder time fetching resources from URLs prior to running the code. The idea here, IIRC, was to eliminate the install step, the need for centralization on a single package manager and registry, and a general "non-Web" approach to the idea of packages and modules in JS.
I believe the `import` syntax was chosen to allow transitions away from `require()`, so that your programs wouldn't just stop working if ESM was enabled.
Your first point is absolutely spot-on but I am curious as to how much treeshaking was on the minds of masses at the time. The tooling of that era didn't really have any good support for tree shaking even for non-AMD includes and it was quite experimental tech (as in, I don't think it was a decision making factor for the majority of the tools on the scene).
The second point actually isn't strictly valid. I've written my own "all-in-one" async custom loader [0] that can require() CommonJS/AMD includes, regular "add a script tag" includes w/out any exports, or even css stylesheets all asynchronously, with asynchronous dependency trees for each async dependency in turn. You can define in the HTML source code a "source map" that maps each dependency name to a specific URL, so that you don't need knowledge of the filesystem tree to load dependencies.
Ideally, this source map can be generated via the tooling you use to compile the code (e.g. `tsc` is aware of the path to each dependency) but I haven't written my own tool to generate the require path to url map.
the import syntax makes it possible for the browser to start loading dependencies as soon as the module has been parsed, before it finishes being compiled. the require function would force the browser to download the script being depended on then and there, and since it's not async, the browser would need to pause script execution while the module is being loaded.
I don’t have an answer, and this is kind of superficial, but one thing I felt about the two was that import statements feel like compilation instructions. “Statically link this.” While Commonjs was a runtime function “synchronously acquire and parse this.”
I’m going to guess the good faith answer really involves some version of “CommonJS has some shortcomings and we didn’t want to confusingly write mostly-same syntax so we designed something new based on ideas from numerous languages.”
A bare-bones implementation of AMD could be put together with less than a kilobyte of JavaScript (this is what we used at Mozilla for a minute circa 2012). Meanwhile, the ECMAScript folks were working on ES6, which was going to have a module system. Why would the browser build in support for a highly-opinionated system that you could implement yourself so trivially, all while a TC39-blessed standard was in the works?
> what I "imagine" browsers doing with CommonJS is making the `require` calls async in the background (IE non visible to developers) so they can resolve the modules then parse the code
That's not possible. You need to run the code to know what's being required: if I call `require('./' + getModuleName())`, you don't know what's being required until `getModuleName()` is evaluated. So you actually need to start running the JS. You need to pause execution of the code calling `require()` (a la `alert()`), and then you can download and parse the required module. When the file is downloaded, you can parse and execute the imported module. Each file would need to be downloaded/parsed/executed _synchronously_ in the order that each `require()` happens in: it's only async in so far as the JS pauses execution and picks up later.
> This isn't terribly different from how import statements work today.
Not so. You can find and resolve `import` statements (note: not `import()` calls, though these return Promises) without executing a JS file. You can parse the imports out of a file in one pass and fetch/parse/repeat for each import in the dependency tree before anything starts executing. Since "native" imports are static and declarative, you can resolve all of them without ever executing any code. And any dynamic imports return promises that the programmer needs to explicitly handle the behavior of at runtime.
> just improve the existing format
1. You'd have to kill dynamic imports (passing anything other than a string literal to `require()`, which would be impossible to do without breaking compatibility and couldn't be polyfilled.
2. AMD allowed a callback syntax for `require()` (it came out years before promises), which is cumbersome. Adding promises later would be challenging and leave technical debt.
> Adding promises later would be challenging and leave technical debt.
I wrote an "aio loader" many years ago that can load (in the browser) AMD/CommonJS/node or just "include this script in your html" dependencies that asynchronously loads dependencies (and their own dependencies) with support for use via plain `require()` without callbacks, `require(foo, foo => {})` callback support, and even dynamic async loading (`var App = await requireAsync("foo")`).
I never published it publicly (it's just ticking away on our production sites) but I was motivated to push it to GitHub just now [0].
> That isn't per se an issue I don't think, esp. because AMD built on top of CommonJS primitives, and with minimal refactoring CommonJS code could be used in the browser when defined this way if asynchronicity is a must.
This existed: the UMD module format was the turducken you got if you built modules to work both as AMD and CommonJS at the same time. AMD wrappers, async require, and a bunch of boilerplate to determine if the module was being loaded by an AMD loader or in a CommonJS environment (or worst of all, a CommonJS environment with AMD loader primitives).
It was a lot of ugly boilerplate. I don't think I ever saw a project intentionally write UMD modules by hand. I do recall some Typescript projects that distributed as UMD modules for a while, because that was boilerplate Typescript was always good at streamlining.
> I do much like the `import` syntax personally and its a little cleaner to read, but CommonJS and AMD were the undisputed winners of the module format until ES Modules were born. Not that I have a problem with ES Modules, I don't, however I am interested in what was so insufficient about the preceding formats that we couldn't have standardized on them
I think it is absolutely the syntax that needed standardizing. AMD was always a hack for module loading using available browser tech as best as it could and screaming for better syntax. There was so much pain every time working with AMD in making sure that define() wrappers were correct and the list of dependencies correctly matched the names and order of those as parameters of the module's function wrapper. AMD was always in desperate need of an import syntax. (One of the reasons Typescript was built was to provide such an import syntax ahead of ESM standardization. It's why I started using Typescript in the 0.x wilds.)
In many ways ESM were always the natural improvement of the AMD format. One of the things that hung browser standardization in various stages was debates about how compatible to be with AMD. There were multiple attempts and a lot of debate at "Loader APIs" that could be extension points to directly interface classic AMD loaders such as Require.js and the Browser's. Had one of those Loader APIs made the final cut it likely would have been possible to "natively" import legacy AMD directly from ESM.
Loader APIs lost to a number of factors including complexity and I think also the irony that CommonJS won the "bundler war" while those debates were going on. I think it must have seemed that the writing was on the wall that AMD compatibility was no longer that useful and Loader APIs were never going to be great for CommonJS compatibility (again, because of those synchronous assumptions that doomed CommonJS to always be the nemesis of browser modules).
(The dying compromises of the "Loader APIs" tangents is what eventually delivered importmaps.)
AMD compatibility without "Loader APIs" is basically impossible. Even though Require.JS was quite dominant, it was never the only loader, and part of its dominance was it was an extremely configurable loader with tons of plugins. There wasn't an "AMD loader standard" that browsers could emulate.
I generally do think that ESM is what we got trying to fix the syntax needs of AMD and clean up and actually standardize the AMD loader situation. In the end it didn't end up backwards compatible with AMD like it tried to do, but from my impression it certainly tried and that was unfortunately part of why ESM standardization was so slow and what led to such a larger mess of CommonJS modules in the wild in the time that took.
Today I came across a dilemma. I wanted to somehow stub or fake Azure's Service Bus. There's no official way to do it. So, I came up with an idea to overwrite its implementation with my own, by mapping @azure/service-bus to my own class in tsconfig.json, no luck, because the framework that I use is using Webpack under the hood, so I would need to create a plugin, rule, whatever, to make it work, but I didn't want to write a specific configuration just for Webpack.
Instead, I had another idea: to import @azure/service-bus dynamically, using ES dynamic `import`, in place of static imports. But, since I'm using Node.JS, I have to set the package.json type to module, so I can use top-level `await`, with dynamic `import` to import, with the help of a ternary, the correct implementation on the fly.
I had a extremely bad experience trying to convert a CommonJS project to use ES Modules before. So I did not go through with the plan.
Finally, after spending some time trying to not use CommonJS I gave up, and in place of dynamic import I used the "good" ol' `require()`, ending up with something like this:
I wouldn't go as far as saying CommonJS is hurting JavaScript, but it's current status in the node ecosystem is definitely painful. I mean the default remains CommonJS, so using esmodules is awkward on 'native' node, but if you use the most popular bundling systems, the default becomes esmodules. I won't pretend I know which is better between the two, but the decision should be made to either make it optional or deprecate it if esmodules is really the Best Module System™. The current in-between is far from ideal, that's for sure.
I think there should be more praise on these guys for what they accomplished given the state of JavaScript when they started. They saw a problem and came up with a solution. Was it perfect? No, but it's not this abominable creation.
Much like John Resig's work on jQuery nudged JavaScript forward, so did the work on CommonJS/Node.
Agreed -- the article actually acknowledges this point, but the clickbait title is not very generous.
CJS was doing just fine in Node.js for nearly a decade before ESM came along and made everything more difficult by shoving browser constraints into a server-side runtime. ESM may be the right direction for the whole ecosystem in the long run, but it's a little backwards to say the perfectly good incumbent system is "hurting" the language because everyone who invested in it doesn't want to go through the pain of migrating to a new fashionable system that is worse in many ways.
"In 2009, CommonJS was exactly what JavaScript needed. The group took a tough problem and forced through a solution that continues to be used millions of times a day.
But with ESM as the standard and the focus shifting towards cloud primitives — the edge, browsers, and serverless compute — and CommonJS simply doesn’t cut it. ESM is a better solution for developers, as they can write browser-compliant code — and for users who get a better end experience."
This is what confused me in there (in the sense that the author seems to "get it" as to why CommonJS is still around). All of this ESM stuff has only (relatively) recently started to take shape. To say CommonJS is "hurting" JavaScript though seems overly-reactive. Technological evolution takes time and deep consideration to not create future messes. It's not the most comfortable but we're in the "messy middle" of moving from one to the other.
I just tried this and got "Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at file:///home/**.js. (Reason: CORS request not http)" in firefox
It is not possible to do same origin from a file system. Everything served from a file system is considered to be different origins. This drives me bonkers.
It just needs to be browser resolvable, assuming this element is meant to run in a browser. So absolute paths do work. For libraries, you may want to use import maps or absolute urls (similar to Deno). There's also esm.sh and jspm.org for modules that are able to be relatively easily converted.
I think it may be necessary/prudent to get some level of JSX support into browsers, much like the ts as comments efforts. Not sure how that will/would land. I was a pretty big fan of E4X, and had a prototype similar to React about a decade before it. In the end, who knows.
Readit News
Sindresorus wrote a gist "Pure ESM modules"[0] and converted all his modules to Pure ESM, breaking anyone who attempted to `require` the latest versions of his code; he later locked the thread to prevent people from complaining. node-fetch released a pure ESM version a year ago that is ~10x less popular than the CommonJS version[1]. The results of these changes broke a lot of code and resulted in many hours of developers figuring out how make their projects compatible with Pure ESM modules (or decide to ignore them and use old CommonJS versions)--not to mention the tons of pointless drama on GitHub issues.
Meanwhile, TC-39 member Matteo Collima advocated a moderate approach dependent on where your module will be run [2]. So the crusade is led not by the Church, but by a handful of zealots dedicated to establishing ESM supremacy for unclear reasons (note how Sindresorus' gist lacks any justifications and how weak TFA's justifications are). It's kind of like the Python 2 to 3 move except with even less rationale and not driven by the core devs.
0 - https://gist.github.com/sindresorhus/a39789f98801d908bbc7ff3...
1 - https://www.npmjs.com/package/node-fetch?activeTab=versions
2 - https://github.com/nodejs/node/issues/33954#issuecomment-924...
This crusade is nowhere near zealous nor righteous enough against the infidels & non-believers.
But it also hasn't been effective enough at supporting/supplying the crusade either.
Matteo's statement was that Node hasn't stabilized their loader support so tools have a harm time migrating to esm. Imo it's a pity ecmascript never stabilized a module registry, that esm 1.0 shipped & most people thought it would happen; it's long felt like a bait & switch. But it wasn't a feature browsers needed or really wanted so that unfulfillment was unsurprising. Anyhow, IMO Matteo is making a technical point that it's still hard to finish the move, which is a different spin IMO than a "advocated a moderate approach".
Given the hurt we legitimately experience, I really wish Node and/or WinterCG or someone would prioritize figuring out & implementing whatever needs to go into a module registry/loader. And then beg the big tool chains that need this stuff to expedite their migrations, pretty pretty please.
Okay, but let's not resist inconveniencing ourselves with the facts.
ESM got browser support in 2017 and stable Node.js support in 2020.
It only became awful for me when people started publishing pure ESM packages; `npm i node-fetch` suddenly began resulting in broken scripts and I had to learn why. Prior to that, I happily used CJS outside of the browser and what I suppose is ESM in the browser (the `import` syntax provided by bundlers).
> a different spin IMO than a "advocated a moderate approach".
He said, "If your module target the Browser only, go for ESM. If your module target Node.js only, go for CJS."
This is moderate compared to the "Pure ESM" approach. The fetch API is built into browsers so I don't see why anyone would use `node-fetch` outside of Node.js, and yet the maintainers of `node-fetch` went Pure ESM anyway. Also that GitHub issue is titled "When will CommonJS modules (require) be deprecated and removed?" and his response was "There is no plan to deprecate CommonJS"[0].
0 - https://github.com/nodejs/node/issues/33954#issuecomment-776...
That preference has had an impact on the Node ecosystem at large, given how prolific an OSS contributor Sindre is, but IMO that influence has been earned by the large body of work he's contributed.
I'm working on greenfield projects that leadership is still insisting we avoid ESM-only packages for. The move that Sindre (and others like wooorm) made has not been well-received.
I'm all for ESM but alienating your own users is stupid. Create node-fetch-esm and support both versions until commonjs popularity would be low enough to be dropped.
I've definitely worked around my fair share of CommonJS issues but until ESM "just works" I'm slightly pained by how aggressive the tone of this article is.
In fact it’s a very illustrative example of why CommonJS should be taken out back and shot.
Re mocks, an ecosystem should not be held back solely due to an arcane edge case. The apps that use test doubles can be rearchitected to support test doubles to the programmers' satisfaction. Most people do not use test doubles & the benefits of ESM & not having to deal with CJS outweigh the downsides of losing some convenience in mocking modules.
This is something that will not be popular with clingers to CJS, but it is something that will benefit the wider ecosystem. The vocal minority which is holding back the ecosystem should start making plans to migrate because it is in the process of happening right now...
So I think the general resentment is having to do extra work to support CJS which is not standard across all JS platforms...and some legacy libraries are still written in CJS requiring an interop. So it would be great to not have to do this extra work to support the legacy CJS on Node.js when every other JS platform is using ESM. In this case, one person's convenience comes at a cost to everyone else & at some point, that one person is going to have to suck it up & do the work to support his use case...just as everyone else had to do the work to support the legacy CJS for years now.
My code works fine and the article isn't doing itself any favors by mocking me (heh) for thinking so.
We would be much better off if it had never been created.
I think the biggest miss was not making mixed mode (default) for Node do it the way webpack/babel, etc did it by default in terms of interop. I get they wanted to make it more implicit to call cjs from esm, in the end it just inhibits conversion of existing libraries as dependencies are now a bigger hurdle.
Frankly, I like the Deno way of things better. I find it annoying, to say the least that the TypeScript team won't consider allowing you to import with a .ts(x) extension, and convert to .js(x) as part of the build process... no, you must omit the extension.
I've been using the import/export syntax since well before it was standardized via babeljs, these days I kind of want to remove webpack/babel from my pipelines altogether and mostly just rely on esbuild. I've also been using/following rome.tools development, having switched over several projects from eslint already, and will probably start with their bundler when it's ready.
I think there's a way to go with tree shaking and static analysis in that direction to reduce load. I also would not mind seeing the efforts to treat TS extensions as comments in the JS engines in that it would be easier to serve up straight TS/JS without bundling/minifying. I'm not sure we'll ever see a return to that in practical terms.
In the end, it's evolving. I'd also like to see Cloudflare, Deno and others come together on a more common server interface so that you can target multiple clouds with a single codebase. I don't know how well that would ever work out at this point though. There's aspects that I definitely like to all of them.
Huge huge agreement.
I forget the specifics but there was some super tiny corner case around maybe default exports that could potentially create ambiguity & that spawned a multi-year bellyaching around doing anything at all for interop. What Node got was incredibly hard fought for against much resistance to interop.
But the final compromises made everything so much more painful for everyone. So many esm projects but oh look a .eslintrc.cjs, how unsurprising & sad.
It's extra maddening because node had a wonderful just works (except that tiny tiny tiny corner case) interop via @standard-things/esm, which seamlessly let the two worlds interop. It'd been around for years before node started shipping support, and it was no ceremony just works bidirectional interoperability, and it took basically no effort or thought from the developers point of view to use. It sucked seeing us walk back from great, mired by frivolous over concern for a obscure corner-case.
https://github.com/standard-things/esm
I do much like the `import` syntax personally and its a little cleaner to read, but CommonJS and AMD were the undisputed winners of the module format until ES Modules were born. Not that I have a problem with ES Modules, I don't, however I am interested in what was so insufficient about the preceding formats that we couldn't have standardized on them
EDIT: I know about the deal with CommonJS being synchronous. That isn't per se an issue I don't think, esp. because AMD built on top of CommonJS primitives, and with minimal refactoring CommonJS code could be used in the browser when defined this way if asynchronicity is a must. Generally, what I "imagine" browsers doing with CommonJS is making the `require` calls async in the background (IE non visible to developers) so they can resolve the modules then parse the code. This isn't terribly different from how import statements work today.
I'm wondering why we didn't undertake the work to just improve the existing format, more or less.
EDIT 2: I'm interested from a historical perspective. I think ESM is the right choice and 100% the future.
This is because of a much deeper issue: static analysis is highly complex with the near-free-for-all that is CommonJS require & module.exports syntax. ES Modules is stricter and much easier to statically deal with.
At a high level, why? You can throw just about anything in an exports.module statement, and the syntax to "require" it also has a lot of leeway. You can actually see the code for this in the Node codebase--module resolution is handled in javascript @ /lib/internal/modules/cjs/loader.js vs /lib/internal/modules/esm (heads up, both approaches are a Lot to grok)
Understand that with the CJS approach, you can dynamically export modules at runtime under whatever name you wish, with whatever value you want, which may even include dynamic require statements themselves. Nightmare for static analysis.
It makes a lot more sense if you try it for yourself. Build a module resolution algorithm including: determining all the imports, all the files those imports are from, mixing with 3rd party and local imports, and building that chain recursively.
You can do it, but the edge cases surrounding CommonJS make it super difficult. I'd go so far as to say it's basically impossible to get 100% success in all the desired scenarios without directly invoking the code.
I think dynamic imports have some of the same footguns here, to be honest. Can't deny ESM is easier to statically analyze though, that much appears to be true across the board based on available evidence.
[0]: https://github.com/indutny/webpack-common-shake
I mean, I guess people will directly write AMD modules, and make modules using some giant script that uses cat, but the future of JavaScript lies with making each source file a valid, correct piece of JavaScript. When each source file is valid and correct, and doesn’t need to be preprocessed in order to work, your tooling will work a lot better.
The browser authors know you can’t un-ship JavaScript features. ES6 import/export is damn good stuff, and people in the browser aren’t saddled with some weird compatibility shim like AMD.
The adoption of ES6 modules in the client-side landscape has far outstripped its adoption in Node.js. I honestly can’t wait for require() to die, in both its cjs and AMD variations. The tooling support for ES6 modules is miles better.
I don't know if that's the entire story -- probably not -- but I do know that is one major differentiator for things like generating import-graphs and performing tree shaking.
(you can still do like `import('foo' + someVar)` which will only invoke dynamically at runtime, so I'm not sure how that case is dealt with)
That case is dealt with more like a `fetch('foo' + someVar).then(r => eval(r.text()))` or similar (but of course it is not just a eval and it instead returns the exports of the module).
Dynamic imports and static ones behave very differently and static analysis generally ignores dynamic imports IIRC.
You also need to treat dynamic imports as async including everything that comes with that (error checking, awaiting, etc.)
1. Because `require()` is "just a magic function", it can't be statically analyzed by a JS runtime prior to actually running the code. This leads to limitations with regards to tree-shaking and other optimizations.
2. The last point leads to the even bigger (and probably "deal-breaker") reason for the change, the desire to fetch packages from URL sources. Since the syntax cannot be parsed efficiently, runtimes like Deno and Bun would have a much harder time fetching resources from URLs prior to running the code. The idea here, IIRC, was to eliminate the install step, the need for centralization on a single package manager and registry, and a general "non-Web" approach to the idea of packages and modules in JS.
I believe the `import` syntax was chosen to allow transitions away from `require()`, so that your programs wouldn't just stop working if ESM was enabled.
The second point actually isn't strictly valid. I've written my own "all-in-one" async custom loader [0] that can require() CommonJS/AMD includes, regular "add a script tag" includes w/out any exports, or even css stylesheets all asynchronously, with asynchronous dependency trees for each async dependency in turn. You can define in the HTML source code a "source map" that maps each dependency name to a specific URL, so that you don't need knowledge of the filesystem tree to load dependencies.
Ideally, this source map can be generated via the tooling you use to compile the code (e.g. `tsc` is aware of the path to each dependency) but I haven't written my own tool to generate the require path to url map.
[0]: https://github.com/mqudsi/loader
I’m going to guess the good faith answer really involves some version of “CommonJS has some shortcomings and we didn’t want to confusingly write mostly-same syntax so we designed something new based on ideas from numerous languages.”
A bare-bones implementation of AMD could be put together with less than a kilobyte of JavaScript (this is what we used at Mozilla for a minute circa 2012). Meanwhile, the ECMAScript folks were working on ES6, which was going to have a module system. Why would the browser build in support for a highly-opinionated system that you could implement yourself so trivially, all while a TC39-blessed standard was in the works?
> what I "imagine" browsers doing with CommonJS is making the `require` calls async in the background (IE non visible to developers) so they can resolve the modules then parse the code
That's not possible. You need to run the code to know what's being required: if I call `require('./' + getModuleName())`, you don't know what's being required until `getModuleName()` is evaluated. So you actually need to start running the JS. You need to pause execution of the code calling `require()` (a la `alert()`), and then you can download and parse the required module. When the file is downloaded, you can parse and execute the imported module. Each file would need to be downloaded/parsed/executed _synchronously_ in the order that each `require()` happens in: it's only async in so far as the JS pauses execution and picks up later.
> This isn't terribly different from how import statements work today.
Not so. You can find and resolve `import` statements (note: not `import()` calls, though these return Promises) without executing a JS file. You can parse the imports out of a file in one pass and fetch/parse/repeat for each import in the dependency tree before anything starts executing. Since "native" imports are static and declarative, you can resolve all of them without ever executing any code. And any dynamic imports return promises that the programmer needs to explicitly handle the behavior of at runtime.
> just improve the existing format
1. You'd have to kill dynamic imports (passing anything other than a string literal to `require()`, which would be impossible to do without breaking compatibility and couldn't be polyfilled.
2. AMD allowed a callback syntax for `require()` (it came out years before promises), which is cumbersome. Adding promises later would be challenging and leave technical debt.
I wrote an "aio loader" many years ago that can load (in the browser) AMD/CommonJS/node or just "include this script in your html" dependencies that asynchronously loads dependencies (and their own dependencies) with support for use via plain `require()` without callbacks, `require(foo, foo => {})` callback support, and even dynamic async loading (`var App = await requireAsync("foo")`).
I never published it publicly (it's just ticking away on our production sites) but I was motivated to push it to GitHub just now [0].
[0]: https://github.com/mqudsi/loader
[0]: https://github.com/requirejs/alameda
I only started my career in ernest in 2012, but even then compatibility with old versions of IE was a major point, due to their high market share.
IE6 was officially retired in 2014, but even then it still accounted for 4.2% of the traffic:
https://www.computerworld.com/article/2488448/ie6--retired-b...
Then there were IE8-11, but it was IE6 which lingered way past its welcome, considering it was originally released in 2001.
This existed: the UMD module format was the turducken you got if you built modules to work both as AMD and CommonJS at the same time. AMD wrappers, async require, and a bunch of boilerplate to determine if the module was being loaded by an AMD loader or in a CommonJS environment (or worst of all, a CommonJS environment with AMD loader primitives).
It was a lot of ugly boilerplate. I don't think I ever saw a project intentionally write UMD modules by hand. I do recall some Typescript projects that distributed as UMD modules for a while, because that was boilerplate Typescript was always good at streamlining.
> I do much like the `import` syntax personally and its a little cleaner to read, but CommonJS and AMD were the undisputed winners of the module format until ES Modules were born. Not that I have a problem with ES Modules, I don't, however I am interested in what was so insufficient about the preceding formats that we couldn't have standardized on them
I think it is absolutely the syntax that needed standardizing. AMD was always a hack for module loading using available browser tech as best as it could and screaming for better syntax. There was so much pain every time working with AMD in making sure that define() wrappers were correct and the list of dependencies correctly matched the names and order of those as parameters of the module's function wrapper. AMD was always in desperate need of an import syntax. (One of the reasons Typescript was built was to provide such an import syntax ahead of ESM standardization. It's why I started using Typescript in the 0.x wilds.)
In many ways ESM were always the natural improvement of the AMD format. One of the things that hung browser standardization in various stages was debates about how compatible to be with AMD. There were multiple attempts and a lot of debate at "Loader APIs" that could be extension points to directly interface classic AMD loaders such as Require.js and the Browser's. Had one of those Loader APIs made the final cut it likely would have been possible to "natively" import legacy AMD directly from ESM.
Loader APIs lost to a number of factors including complexity and I think also the irony that CommonJS won the "bundler war" while those debates were going on. I think it must have seemed that the writing was on the wall that AMD compatibility was no longer that useful and Loader APIs were never going to be great for CommonJS compatibility (again, because of those synchronous assumptions that doomed CommonJS to always be the nemesis of browser modules).
(The dying compromises of the "Loader APIs" tangents is what eventually delivered importmaps.)
AMD compatibility without "Loader APIs" is basically impossible. Even though Require.JS was quite dominant, it was never the only loader, and part of its dominance was it was an extremely configurable loader with tons of plugins. There wasn't an "AMD loader standard" that browsers could emulate.
I generally do think that ESM is what we got trying to fix the syntax needs of AMD and clean up and actually standardize the AMD loader situation. In the end it didn't end up backwards compatible with AMD like it tried to do, but from my impression it certainly tried and that was unfortunately part of why ESM standardization was so slow and what led to such a larger mess of CommonJS modules in the wild in the time that took.
Instead, I had another idea: to import @azure/service-bus dynamically, using ES dynamic `import`, in place of static imports. But, since I'm using Node.JS, I have to set the package.json type to module, so I can use top-level `await`, with dynamic `import` to import, with the help of a ternary, the correct implementation on the fly.
I had a extremely bad experience trying to convert a CommonJS project to use ES Modules before. So I did not go through with the plan.
Finally, after spending some time trying to not use CommonJS I gave up, and in place of dynamic import I used the "good" ol' `require()`, ending up with something like this:
.And that was that.
Project maintainers have to make ES Modules practical before it's pretty.
Deleted Comment
I think there should be more praise on these guys for what they accomplished given the state of JavaScript when they started. They saw a problem and came up with a solution. Was it perfect? No, but it's not this abominable creation.
Much like John Resig's work on jQuery nudged JavaScript forward, so did the work on CommonJS/Node.
CJS was doing just fine in Node.js for nearly a decade before ESM came along and made everything more difficult by shoving browser constraints into a server-side runtime. ESM may be the right direction for the whole ecosystem in the long run, but it's a little backwards to say the perfectly good incumbent system is "hurting" the language because everyone who invested in it doesn't want to go through the pain of migrating to a new fashionable system that is worse in many ways.
"In 2009, CommonJS was exactly what JavaScript needed. The group took a tough problem and forced through a solution that continues to be used millions of times a day.
But with ESM as the standard and the focus shifting towards cloud primitives — the edge, browsers, and serverless compute — and CommonJS simply doesn’t cut it. ESM is a better solution for developers, as they can write browser-compliant code — and for users who get a better end experience."
Deleted Comment
We all lose the ability to simply have a local index.html file, and have it Just Work (TM):
This ability is amazing for demos, fast iteration, onboarding new devs and developing without a ton of layers of js ecosystem machinery.Deno doesn't care about retaining this level of developer experience because Deno is marketing its own runtime / build step / ecosystem.
Shouldn't that be a relative path for it to actually work as you intend?
I think it may be necessary/prudent to get some level of JSX support into browsers, much like the ts as comments efforts. Not sure how that will/would land. I was a pretty big fan of E4X, and had a prototype similar to React about a decade before it. In the end, who knows.