This is a huge leap forward for the JavaScript community—probably more than many people will realize right away.
I loved Bundler's deterministic builds but chafed against the Ruby limitation of only having a single version of a dependency at once. npm solved this problem elegantly, but still struggles with non-determinism. I had resigned myself to thinking that maybe these were just fundamental tradeoffs in package management that could never be resolved.
Then I had the opportunity to use Cargo, the package manager for Rust. It synthesized what, in my mind, are the best features of npm and Bundler into a single package manager, with terrific speed to boot.
Yarn looks like it will be Cargo for JavaScript. After Cargo improved on many of the great ideas of npm, those ideas are returning to help improve the JavaScript ecosystem. Cross-pollination at its best.
This also highlights a shrewd move on the part of npm: a clear decoupling between the npm registry and client, with a well-defined protocol between the two. The strength of Node is in the staggering size of its ecosystem; how those bits end up on disk is an implementation detail. This smart separation allows for this kind of experimentation on the client side, without causing the ecosystem fragmentation that happens when a new package manager requires a new registry.
I'm also happy to see that a significant amount of work has already gone into the governance model. Despite the largest contributors being Facebook employees, it looks like they've really outdone themselves making sure this is a community-run project. A BSD license, no PATENTS file, and an Ember/Rust RFC process[1]; this has all the hallmarks of a community open source project. That's critical for open infrastructure projects like this, and they've nailed it.
I'm very much looking forward to using Yarn in my own projects because it looks like it solves a lot of real problems I encounter every day. Thanks for all the hard work!
> This also highlights a shrewd move on the part of npm: a clear decoupling between the npm registry and client, with a well-defined protocol between the two. The strength of Node is in the staggering size of its ecosystem; how those bits end up on disk is an implementation detail. This smart separation allows for this kind of experimentation on the client side, without causing the ecosystem fragmentation that happens when a new package manager requires a new registry.
And a shrewd move by FB: to not announce their new registry on the same day.
> Yarn pulls packages from registry.yarnpkg.com, which allows them to run experiments with the Yarn client. This is a proxy that pulls packages from the official npm registry, much like npmjs.cf.[0]
Time will tell whether they only want to be proxying NPM or will allow direct pushing to their own registry. If they do, JS ecosystem might see another big shift.
As a practical matter, the npm ecosystem today relies on duplication, and no new client that made the "highlander rule" (there can be only one) mandatory could succeed.
Yarn does offer a `--flat` option that enforces the highlander rule, and I'm hopeful that the existence of this option will nudge the ecosystem towards an appreciation for fewer semver-major bumps and more ecosystem-wide effort to enable whole apps to work with `--flat` mode.
Cargo definitely allows two dependencies to rely on different versions of a transitive dependency. If those deps expose a type from that dependency in their API, you can very occasionally get weird type errors because the "same" type comes from two different libraries. But otherwise it Just Works(tm).
Cargo does allow multiple versions of transitive dependencies. It tries to unify them as much as possible, but if it can't, it will bring in two different versions.
What it _won't_ let you do is pass a v1.0 type to something that requires a v2.0 type. This will result in a compilation error.
There's been some discussion around increasing Cargo's capacity in this regard (being able to say that a dependency is purely internal vs not), we'll see.
With tiny modular utilities this is very much necessity - and not a bad idea if the different versions are being used in different contexts.
For instance, when using gemified javascript libraries with bundler it is painful to upgrade to a different version of a javascript library for the user facing site while letting the backend admin interface continue using an older one for compatibility with other dependencies.
You've got to take the ecosystem into account. There are a lot of very depended-upon modules that are 1 KB of code and bump their major version as a morning habit. Forcing people to reconcile all 8 ways they indirectly depend on the module would drive them nuts, but including 4 copies of the module doesn't hurt much.
This is especially true when you're going to be serving your code over the web. It's very easy when using npm to accidentally end up shipping a lot of duplicate code.
That alone has me super excited about Yarn, before even getting into the performance, security, and 'no invisible dependency' wins.
I feel like this is especially problematic when using NPM for your frontend. Now you have to run all your code through deduplication and slow down your build times or end up with huge assets. I wonder if it's really worth the trouble.
> I loved Bundler's deterministic builds but chafed against the Ruby limitation of only having a single version of a dependency at once
This is due to the way that Ruby includes code, where it's available globally, vs Node where code is scoped to the including module. I'm not sure how Ruby could support multiple versions with changes to the language and/or RubyGems
Why do you think that everybody is inspired from javascript tech? It is probably the other way around. For example there were very good build tools long before npm which are still used today and unmeasurably better (like Maven or Gradle).
This may come off as a troll, but it's an honest question. I'm not a javascript guy. It's not a language I deal with at all. Why on God's green earth does it need as much tooling as it seems to have? Are people really making projects with dozens (hundreds? more?) of dependent libraries? Are there aspects of the language or runtime that reward multiple layers of configuration management? In short, what the hell is up with this ecosystem?
1. In the browser we have less control over the environment the code runs in, requiring tooling to achieve a stable, normalised environment which we can improve at our own pace, rather than the rate at which browsers are updated.
2. JS has become very widely used very quickly, which means that there have been lots of hastily built things, which have been replaced by better things (and for every 'better thing' there are many alternatives which fell by the wayside), and a larger pool of people to build them.
3. It's easier than ever to build new tools (or any kind of app, for that matter), because the npm ecosystem makes it trivial to combine existing smaller pieces of code into new combinations.
Your number 3 is the one that really drives it home for me. It's the unix philosophy taken to the extreme, and (at least in my experience) it's working very well.
It does take some getting used to, but when you stop looking for one big tool that can solve 5 semi-related problems, and start looking for 5 little tools that each solve one of the problems, things get much more clear.
And yes, I know that this kind of reliance on dependencies can cause issues, but from my experience, those issues don't seem to bite that often, and many of them can be resolved with even more tooling.
browser incompatibility is the biggest issue (if not the only issue). this is almost a non-issue if you're just writing node related programs. but if you need to support in the browser, now you have edge, chrome, firefox, and safari at minimum. then some have to continue supporting ie10/ie11, or even worse, ie7+.
don't forget about mobile devices, with ios having such a difficult-to-write-anything-new version of mobile safari.
i'm writing a new app for myself, so i don't have to worry about any backwards compatibility issues, except it needs to work on my iphone, which uses mobile safari webviews, which means i have to either use shims or restrict myself back to es5, non-html5 features (or work around their html5 limitations).
example: you cannot cache audio files to play using <audio> tags in mobile safari. safari just refetches it from your server (also has to recreate the audio object, which causes a lag in the audio being played). technically you _can_ cache if you convert all your audio files to base64 and use the webaudio api. i'm probably going to have to go this route because i'd like the app to be offline usable.
so rather than spend time on my app, i now have to write out a build step for converting audio to base64. then test that it works fine in ios and desktop browsers (at least chrome, firefox). it all just keeps adding up.
I came to the Javascript world late, starting with React. I think its tooling has gotten out of hand, but there are a couple of reasons it's not a _total_ disaster in my mind:
* Partly, there are a larger number of Javascript developers that haven't been exposed to other ecosystems, and end up reinventing some wheels
* More importantly though, I think the pace of innovation in the javascript ecosystem is actually good in a way, because other more mature languages (I'm not talking about age of the language, just maturity of the tooling and ecosystem) have gone through the same growing pains, it just took a lot longer. It's easy to forget, coming from the Python world for instance, the pains of easy_install and distutils vs seuptools, etc etc. I still find Java dependency management a bit of a pain, because I don't know it very well.
We're just watching a sped up version of a language maturing, and it's painful as individual developers trying to keep up, but I don't think it's as negative as HN often makes it.
> Are people really making projects with dozens (hundreds? more?) of dependent libraries?
Yes.
> Are there aspects of the language or runtime that reward multiple layers of configuration management?
A significant fraction of the node community likes to split their project into really, really small packages and treat them as if they're independent even if they come from the same source repo and tend to get used together. As an example, the popular collection processing library Lodash publishes a package for each method and people actually do depend on individual methods.
There are two major rationales for this. The first is that JS is a dynamic language and in dynamic languages, smaller chunks of code are easier to reason about than larger ones. The second is that the JS community as a whole cares more about artifact size than pretty much any other community. Having smaller packages allows you to produce a smaller payload without having to rely on techniques like tree shaking.
I find micro packages maddening but people give me their code for free and it mostly works so I don't complain about it that much.
What do you find maddening about micro packages? I hate depending on a giant stack of frameworks (slowing installs, builds, adding behavior surface area) when I just need a function or two, so micro packages are a joy for me.
I have no idea if you're familiar with the java ecosystem, but I am, so I'll use that as an example:
Node libraries are typically several orders of magnitude smaller and more single purpose than java libraries. Java libraries like Guava and Spring are typically quite large. A practical example is from unit testing: In Java, you might use JUnit+Mockito for your test dependencies. In Node, to get similar functionality, I have the following deps: Mocha, Sinon, Sinon-As-Promised, Chai, Chai-Moment, Chai, Chai-Things, Chai-Shallow-Deep-Equal, Chai-Datetime, Sinon-Express-Mock, Supertest.
Many Javascript applications both client and server functionality, and there is different tooling for both. Webpack is currently popular for building stuff that will run in the browser, whereas server side stuff is different.
It's a much newer technology than java and there is quite a lot of churn around tooling in general as the nature of the way applications are built change.
Strangely enough both Java and JavaScript were first released in May 1995. Server side JavaScript was released the same year just after the browser version.
Though I get that you are referring to the js eco systems (commonjs, npm) compared to the much more mature Java ecosystem.
Why is that?
In the Java ecosystem there is Spock framework that gives you everything required for testing, you don't even need junit+mockito.
In which scenario would be better to use 11 different libraries for doing the same thing as a well written framework?
A very limited standard library with no native (well, just approved but not ratified) support for importing/libraries. Coupled with inconsistent implementations and you end up need more layers of abstraction.
Let's consider Microsoft Office. It is a massive software and such as massive software requires a lot of (internal) dependencies and toolings.
Now imagine Office 365, the same software with the same functionalities, except running in the browsers. The task is to essentially write Microsoft Office (front-end logic) entirely in JavaScript (or TypeScript), with the additional problems of dealing with browser compatibilities issues, which are just as hard as OS compatibilities issues.
Then, imagine all the software going to the "cloud". JavaScript now has to have the capability of re-writing all these software originally written in C++, C#, Java... all sorts of languages. It is quickly becoming a superset of all ecosystems where everyone is trying to migrate their software to. So it is not surprising that people are inventing more and more tooling to make it more powerful to suit their needs.
You say tooling, but then talk about libraries. I think both are just as numerous for nearly all popular programming languages. It’s just that JavaScript is currently “hot shit”, so all this stuff is a lot more visible.
There are tons of libraries for C, many of which aim to do the same, only faster/smaller/scalable/whatever. There are many compilers, both open and closed source. And there are documentation tools, syntax checkers, unit testing frameworks and whatnot.
Of course there are also languages like C# where there’s a de-facto standard for almost everything: IDE, compiler, base library, GUI framework(s).
Because, from this outsider's perspective, it seems like most of the tooling is library management (rather than, say, profiling or linting -- does javascript even have a linter? -- or debugging).
This is a valid question coming from an outsider looking in.
There are various packages that are extremely small (see leftpad and associated controversy). They often wind up in a slightly larger package, which itself winds up in a slightly larger package, which recurs until you finally wind up with one actual package of consequence that you're targeting. For example, Express (a very common Node.js abstraction) has 26 direct dependencies, yet 41 total dependencies.
A lot of this results from early Node.js' mantra of having small, focused packages. This could potentially be a good thing because instead of having three major dependencies that have their own way of leftpadding they can all rely on one library and thus code is efficiently reused. This can be bad, however, when they each need their own version of the dependency - or worse - the dependency is removed from the registry (see Leftpad and associated controversy).
One of the legitimate problems that I've seen is that there are various libraries that are just thin wrappers over functionality that is native to Node.js - but written to be "easier to grok". Thus, these thin abstractions become practical code duplication out of some mixture of Node's developers lack of effective verbosity/documentation and application developer laziness. But then they creep higher up the dependency chain because someone at a lower level used it.
On one hand it can be quite simple (and rewarding) to write code while minimizing dependencies. On the other hand, abstractions such as Express make it very easy to write code and it feels like you only have "one dependency" until you look under the hood.
Babel, the most popular transpiler, is 335 total dependencies, and it doesn't even do anything out-of-the-box. You need to add babel-preset-latest for 385 total dependencies if you want to actually transpile anything.
A lot of these answers are pointing out the x-browser complexity which is true but that isn't what drives JS package mgmt. It's mostly Node stuff. The mature common JS libs for browser work are pretty robust when it comes to x-browser stuff now days.
For browser work, you can do almost everything you need with some jQuery, handlebars and moment if there are dates. A few other small libs here and there and you've got almost everything covered. Some edge cases may drive you to use some library that is heavily dependent on 32 different sub-libs but it's really not that often.
Server-side Node.js is the issue, not browser compatibility.
Handling dependencies is hard. Handling dependencies for a platform you don't control is harder. Handling dependencies for a platform you don't control, where every single kilobytes count, is insane.
Add those together, and you get the need for real complex tooling.
With that said, tooling is nowhere as complex as it is for languages like C++ or Java. People are just used to it on those platforms, and the tooling is more mature/understood because it already went through the countless iterations, 15-20 years ago or more.
The tooling for building C++ consists of make or something that generates make files. There's also no "1 function" libraries.
Adding a dependency is done by (compiling it if needed) and referencing it from the make file.
It's super easy to understand, but involves manual work because C++ doesn't have a package manager.
Java isn't too complicated either. I've used Gradle and Maven but doing things manually like in C++ can also work.
Crucially, they don't reinvent the wheel every couple of months so a C++ or Java dev can focus on building things instead of relearning how to build things.
Javascript in my opinion is still very young. You can't compile it to web assembly (yet) and the fact that there are different browsers that interpret it in so many different ways make for libraries to exist to mitigate this. Also none of these tools are 100% necessary, they just make your life easier and tools always have room for improvement. I'm sure others can add to this. I don't think any of this is a bad thing, it's just overwhelming for newcomers that haven't been following this whole thing.
Others have touched on the subject but one of the biggest issues is browser compatibility. Building a responsive website that works on all major browsers, across multiple versions, as well as all the mobile quirks is not an easy task in the slightest ... and that's just to make a site look pretty. Then we have to start talking about building massive web apps that are front-end heavy which requires even more tooling to scale, maintain properly.
Because JS is open source and it's not controlled by an Oracle / Microsoft / Apple / XXX foundation. Everybody is free to do stuff - and does. In the end there aren't actually as many tools as people think, a lot of hobbyists projects who die quite quickly
Is npm the bazooka or C++? As someone who's worked with C/C++ (and its hilarious explosion of tooling -- half-a-dozen at Google alone), maven, bundler and Cargo, I think it's just not accurate to say that the npm ecosystem has a particularly high level of complexity.
Interested to see the emphasis on security, it's definitely something that could use more focus in this area. On that note, are there any plans to implement TUF (https://theupdateframework.github.io/) or similar into Yarn?
You say "decent performance" and "predictability". What is the basis of this claim? I've heard these all before, but unless you've actually shipped a product using this tool I don't know how you can back this up.
As for performance, Yarn is about 3-5x faster installing all of the dependencies and devDependencies of react-native. The benchmarks are here: https://yarnpkg.com/en/compare. It's much faster in most scenarios, especially the ones that used to take minutes.
There's a lot to like here - deterministic builds are great. However, yarn doesn't currently seem to support installing straight from github or private packages [0], [1].
I'm sure this will be added in the future, but it is currently a dealbreaker for me - I'm using a single package that's just straight hosted in a private git repo, as the package itself isn't ready to be published yet. I'm sure other people are using private packages for more normal reasons (closed source, pre-production, etc).
If yarn actually made it simpler to refer to a local package during development, I'd be on board with that. (I am developing the dependency, but want to also work on the thing that depends on it, so I'd rather just save locally and refresh npm on the outer project, but that's hard to get right - file urls don't always update, the easiest workflow seems to be delete the package from node_modules, and npm install, with the package having a git url rather than a version in package.json).
You are right, we aren't 100% compatible right now. Please track the issues on the Yarn GitHub repo that you linked or better yet: help us fix it. This is a community project and we are excited for the community to jump in and help us out and complete the few missing pieces.
I agree local development with many packages can be hard. For Jest we adopted `lerna`
This. I came here to say this also.
In ruby if you want to patch a repo for yourself you just fork it and make changes and then in bundler point to the git url (often on github). Then run bundle install and you're set.
With npm I've been frustrated a few times whereby you can fork the repo and point package.json to the git repo but after running npm I usually get a bunch of compilation errors.
It would be wonderful to make this as simple as with bundler within the ruby ecosystem, point to forked url and running yarn handles the rest.
I'm very excited about this. Private package support is a must-have for it to get any traction in my company. We are more than ready to leave shrinkwrap behind.
It is quite a congenial response and no disrespect to npm inc, but yarn will be technically far superior (if it's not already), and I can't help but interpret this as the beginning of the end for npm....It was only a matter of time though since there are a number of systemic problems with npm and (in my opinion) it is at the center of the dysfunctional world of modern JavaScript tooling. It did move the ecosystem forward significantly for a number of years, but now something better is required. Sorry to be blunt, just my opinion.
NPM can evolve; Gradle and SBT also didn't turn out to be the end of Maven, which evolved to being more of a repository and dependency management tool with multiple clients.
I'm glad they're okay with it but then again what else would they post? npm has some major, major issues with it and it has barely moved in years. This Yarn looks like it solves quite a few issues with npm and it took another company to build it.
That's insane. It wouldn't surprise me if yarn becomes popular and its proxy to npm slowly turns off and boom, everyone would be migrated and npm would be left with little to no users. Yeah that's probably unlikely for the immediate future but Yarn is positioned perfectly to siphon off npm's users very painlessly.
Looks like they are trying to write off yarn as just another alternative install tool. I think this is short cited and hope they are taking this threat more seriously internally.
That's pretty well written, though you can see slight hints of bitterness in the tone (though its subtle). Or maybe I'm a little biased, because I've seen some of the non-public reactions, and they were more than a little bitter...
Interesting how they point out how Yarn still depends on them since it pulls packages from NPM's package repo. Kind of like a reminder of who the dominant player still is.
I'm really liking yarn. It solves some of my main issues with npm, it's fast and it provides some great output in the CLI. It's also architected absolutely ingeniously. Seriously they have positioned themselves to seamlessly take over npm's entire business just like that. What do I mean?
So today by default yarn goes through its servers which then proxy to npm's. As time goes Facebook can add more features to its own repo (custom submissions instead of to npm or maybe it does a federated publish to both at first, nice looking pages with community engagement tools built in, better metrics (npm provides me with almost worthless metrics), etc). Then when enough people are using Yarn Facebook could simply...flip the switch. Then boom, zero npm usage.
I would be terrified if I were NPM right now. They've sat on their hands for years not improving npm. I mean npm3 had a silly CLI animation that made all downloads take orders of magnitude longer simply because of the animation!
P.S. if you're trying to use yarn with a custom npm repository just drop a .yarnrc file into your project's directory then add the following in that file:
npm should not be terrified, we've been working with them and they've been very encouraging.
From their perspective it's very difficult to make breaking changes to the client because of the sheer number of people depending on them. (We've faced similar problems on Babel)
Yarn still makes use of the npm registry and all the high quality infrastructure they've built and support they provide for the ecosystem. They are a critical piece of this.
I mean I understand the intent but it turns npm into basically a dumb pipe / just infrastructure. Every business that gets reduced to that struggles to get out of that space and expand.
Granted it's far too early to write npm off. But with how slow they've moved over the years I'm unconvinced that yarn won't take over the space unless it runs into some bad problems. Npm 3 and its launch was an absolute mess and, ultimately, it's almost just a package managing tool. I am unconvinced that breaking changes is much of an issue if at all for them. They could abandon shrinkwrap into a better system and I think everyone would be happy for it; no need to keep propping up that awful system.
As Yehuda pointed out, such a thing won't happen because Yarn was launched as a community project from the beginning, and a lot of members contribute to both, so even if Facebook decided something sketchy, the community would either stop it from happening or continue with a forked version.
> the community would either stop it from happening or continue with a forked version
Would they, though? If Yarn turns out to be the better tool that doesn't change the way they work (beyond solving some pain points) then would they even care? Facebook could even offload the hosting to some open source group like Apache.
It's purely anecdotal but a lot of people I have worked with would love to see an alternative to npm. It's just slow and the features it gives to authors is pretty limited (I want better metrics, please!). I'm not even sure I would call it sketchy, it would just simply be phasing out something deprecated (I'm not talking about a swift take over more like a slowly-boiling-the-frog-in-the-water takeover).
It's only my opinion and if Facebook really doesn't want that to happen then I guess it won't but it's pretty easy to imagine it, IMO.
This is not a problem with the package manager. This is a problem with complexity.
When did it start becoming reasonable for a front-end only part of the MVCC pattern to have 68 dependencies?
Or for a transpiler like Babel to add 100k+ files? I'm sorry I just find it ridiculous that instead of taking a look at the disease (unbounded complexity), we are looking to engineer our way out of the problem by creating a package/dependency manager that "scales". Are the frontend problems at Facebook of showing HTML forms and buttons on a page really that complicated to warrant such a behemoth of a system?
This harkons back to the days at LinkedIn where I spent my time in a misguided effort to create a distributed build system because our codebase had grown so massive that it literally took a distributed system to build it and still release within the day. I feel bad for people working on these systems because while it is fun and "technically interesting" to solve "problems at scale", you're really just digging the ditch deeper for your company. You are putting lipstick on the 500 lbs pig; making room in the dump to pour in more technical debt.
I don't think anyone ever imagined we'd be in a situation where a javascript framework is approaching the complexity of an operating system in terms of individual source files and SLOC.
The Javascript ecosystem has to deal with unrelenting, multi-decade backwards compatibility requirements; supporting numerous, inconsistent implementations; and an incredibly wide set of stakeholders and participants with diverse motivations when advancing the spec.
Do any other software platforms out there face the same magnitude of complications?
To speak to one of your primary examples: Babel is freaking huge and somewhat absurd, but is there any other way to advance the Javascript language? Fortunately, this is only a development dependency, and the client doesn't have to see it.
On a happier note, these are good problems to have. This is another indicator of how successful the web platform has become. You can build some absolutely incredible projects these days, and they are instantly available to billions of people who are operating thousands of different hardware devices (with different specs and dimensions) running dozens of operating systems. No gatekeeper, bureaucrat, or corporate curator must approve your work. Just simply deploy it, and anyone with the URL can access it. It is a mind-blowing human accomplishment. I look forward to seeing how the web continues to progress over the coming decades.
Have you seen some of the things we are building for the web these days? Full applications, with multiple "pages", server-side rendering above the fold content, minified builds that load not only content--but code as well--on demand from the server, etc.
Absolutely: this is overkill for your blog or portfolio website, but part of the rising complexity with the tooling and dependency system is due to the rising complexity of the applications themselves.
I use JS+Node+NPM for my day job and many side projects.
Initial thoughts:
- Why didn't Facebook contribute the updates to NPM directly?
- They are coupling a package manager and registry proxy; the latter has many existing implementations already
- The differenced between Yarn and NPM+Shrinkwrap do not seem substantive; NPM made the design decision to use a sometimes non-deterministic install algo in NPM3 to speed up install times - when network is involved, there is a serious trade off between idempotency and speed
In general, FB seems to love building their own versions of existing tools:
- Flow, introduced 3 months after TypeScript
- Nuclide instead of Atom/Sublime/VSCode
- Jest instead of Jasmine/Mocha
- DraftJS instead of (insert of of the existing 100 text editors here)
- ...
I get that these are useful internally at FB, but I don't think they help the community much. It would be better to work with existing tools and contribute back to them than to reinvent the wheel every time something doesn't work perfectly for their use case.
I get that FB uses these as recruiting tools, it's useful for them to have rights for these projects, and it gives their engineers something to do and be excited about, but I do not want a whole dev tool ecosystem controlled by big FB.
Do you think Flow was built in 3 months as a reaction to Typescript? Not that it was a big enough problem that two different groups of people independently decided to try to solve it?
EDIT: disregard this, Flow was introduced much longer than 3 months after TypeScript. The blog post introducing Flow mentions TypeScript, and is probably worth reading for background [1]
> Nuclide instead of Atom/Sublime/VSCode
Nuclide is an Atom plugin.
> Jest instead of Jasmine/Mocha
Jest provides a superset of Jasmine functionality, and uses Jasmine as its test runner.
> Do you think Flow was built in 3 months as a reaction to Typescript?
They probably knew that TS was in progress when they started, but I don't have a window into that. I did ask some Flow devs a few weeks ago if they are interested in merging with TS, and they answered with a categorical no.
Hey, I work on Yarn and Jest. Let me respond to why we built Jest: It solves a ton of problems no other test runner solves and it works well at Facebook's scale. We open sourced it more than two years ago but only recently started to promote the project in open source more actively because it is working so well at FB.
Finally, test runners tend to get stale. Any large organization will run into trouble with any kind of tooling as companies scale up engineers or codebases. Testing is no different and we aimed to solve those problems for Facebook; getting to open source our solutions is an added bonus and we are excited to work with the JS community to build the best tooling we can possibly build.
I for one am thankful for Jest and Yarn. I recently switched over to Jest and I love most everything about it and the built in watch with caching reduces friction immensely (now if we can just allow custom preprocessors to return promises that would make life easier for integration of things like rollup). And yesterday I flipped over to using yarn and it definitely solves most of my pain points with npm.
So yeah, I'm definitely on the "keep reinventing" side of this debate.
IMO it's the "javascript way" to invent new tools instead of trying to improve others (when it makes sense to do so).
Let me explain that a little bit... The javascript "ecosystem" takes the unix philosophy to the extreme in many ways. And one of the points of the unix philosophy is to try to avoid "bloating" tools with tons of options, and instead trying to create new tools where appropriate.
Many of those are perfect examples of where "improving" a current tool with the changes they wanted would end up with a more complicated and difficult to use tool.
- NPM makes tradeoffs that the majority of developers in the JS world want right now. Like it or not, deterministic installs isn't something that most people want or need (at least, they don't know that they might want or need it). Making a new tool that anyone can use when that is needed is a great solution, and it avoids bloating NPM with a hundred little flags and tons of code to cover edge cases. It's the perfect example because anyone can switch to using yarn when it's needed without any extra work.
- Nuclide is actually just a layer on top of Atom. They wanted a bunch of things to all work one way, so rather than try to get Atom to change to include what they wanted, or require developers install a ton of extensions, they packaged them up on top of Atom as one system.
- Jest had different goals at the start. Testing react components was ugly and difficult, and Jest wanted to solve that first (IIRC, i'm not very familiar with jest unfortunately). Again, it's something that can be used if needed, and can even be combined with other testing tools if you want (we almost did that at my work, used Jest to test the front-end components, and our current mocha+friends setup to test the "business logic").
> The javascript "ecosystem" takes the unix philosophy to the extreme in many ways.
Bullshit, nix systems rely on stable global dependencies, not 50 versions of the same library installed locally. I don't buy the "unix philosophy" excuse. The nodejs community just doesn't care about API stability which is why there is that inflated number of modules with 90% never maintained more than a year.
+1 for contribution to npm. Especially given npm isn't completely deterministic there is no reason for not pushing for these changes into npm itself.
Given npms years of crap I'm ready to give it a shot anyway. This might be one of FB's projects that actually gets some traction, especially given its developed in collaboration with other big names.
Readit News
I loved Bundler's deterministic builds but chafed against the Ruby limitation of only having a single version of a dependency at once. npm solved this problem elegantly, but still struggles with non-determinism. I had resigned myself to thinking that maybe these were just fundamental tradeoffs in package management that could never be resolved.
Then I had the opportunity to use Cargo, the package manager for Rust. It synthesized what, in my mind, are the best features of npm and Bundler into a single package manager, with terrific speed to boot.
Yarn looks like it will be Cargo for JavaScript. After Cargo improved on many of the great ideas of npm, those ideas are returning to help improve the JavaScript ecosystem. Cross-pollination at its best.
This also highlights a shrewd move on the part of npm: a clear decoupling between the npm registry and client, with a well-defined protocol between the two. The strength of Node is in the staggering size of its ecosystem; how those bits end up on disk is an implementation detail. This smart separation allows for this kind of experimentation on the client side, without causing the ecosystem fragmentation that happens when a new package manager requires a new registry.
I'm also happy to see that a significant amount of work has already gone into the governance model. Despite the largest contributors being Facebook employees, it looks like they've really outdone themselves making sure this is a community-run project. A BSD license, no PATENTS file, and an Ember/Rust RFC process[1]; this has all the hallmarks of a community open source project. That's critical for open infrastructure projects like this, and they've nailed it.
[1]: https://github.com/yarnpkg/rfcs
I'm very much looking forward to using Yarn in my own projects because it looks like it solves a lot of real problems I encounter every day. Thanks for all the hard work!
And a shrewd move by FB: to not announce their new registry on the same day.
Time will tell whether they only want to be proxying NPM or will allow direct pushing to their own registry. If they do, JS ecosystem might see another big shift.
[0] http://blog.npmjs.org/post/151660845210/hello-yarn
But at least it's a move away from NPM. I think the most problems I had with JavaScript develompent in the last 2 years came from NPM.
The problem isn't really fundamental. Bundler makes almost all the right choices already. Its major disadvantage is that it only works for Ruby.
Yarn does offer a `--flat` option that enforces the highlander rule, and I'm hopeful that the existence of this option will nudge the ecosystem towards an appreciation for fewer semver-major bumps and more ecosystem-wide effort to enable whole apps to work with `--flat` mode.
Plz send halp!
What it _won't_ let you do is pass a v1.0 type to something that requires a v2.0 type. This will result in a compilation error.
There's been some discussion around increasing Cargo's capacity in this regard (being able to say that a dependency is purely internal vs not), we'll see.
For instance, when using gemified javascript libraries with bundler it is painful to upgrade to a different version of a javascript library for the user facing site while letting the backend admin interface continue using an older one for compatibility with other dependencies.
If so, different major versions of the same dep should be considered different libraries, for the sake of flattening. Consider lodash for example.
That alone has me super excited about Yarn, before even getting into the performance, security, and 'no invisible dependency' wins.
This is due to the way that Ruby includes code, where it's available globally, vs Node where code is scoped to the including module. I'm not sure how Ruby could support multiple versions with changes to the language and/or RubyGems
1. In the browser we have less control over the environment the code runs in, requiring tooling to achieve a stable, normalised environment which we can improve at our own pace, rather than the rate at which browsers are updated.
2. JS has become very widely used very quickly, which means that there have been lots of hastily built things, which have been replaced by better things (and for every 'better thing' there are many alternatives which fell by the wayside), and a larger pool of people to build them.
3. It's easier than ever to build new tools (or any kind of app, for that matter), because the npm ecosystem makes it trivial to combine existing smaller pieces of code into new combinations.
It does take some getting used to, but when you stop looking for one big tool that can solve 5 semi-related problems, and start looking for 5 little tools that each solve one of the problems, things get much more clear.
And yes, I know that this kind of reliance on dependencies can cause issues, but from my experience, those issues don't seem to bite that often, and many of them can be resolved with even more tooling.
don't forget about mobile devices, with ios having such a difficult-to-write-anything-new version of mobile safari.
i'm writing a new app for myself, so i don't have to worry about any backwards compatibility issues, except it needs to work on my iphone, which uses mobile safari webviews, which means i have to either use shims or restrict myself back to es5, non-html5 features (or work around their html5 limitations).
example: you cannot cache audio files to play using <audio> tags in mobile safari. safari just refetches it from your server (also has to recreate the audio object, which causes a lag in the audio being played). technically you _can_ cache if you convert all your audio files to base64 and use the webaudio api. i'm probably going to have to go this route because i'd like the app to be offline usable.
so rather than spend time on my app, i now have to write out a build step for converting audio to base64. then test that it works fine in ios and desktop browsers (at least chrome, firefox). it all just keeps adding up.
* Partly, there are a larger number of Javascript developers that haven't been exposed to other ecosystems, and end up reinventing some wheels
* More importantly though, I think the pace of innovation in the javascript ecosystem is actually good in a way, because other more mature languages (I'm not talking about age of the language, just maturity of the tooling and ecosystem) have gone through the same growing pains, it just took a lot longer. It's easy to forget, coming from the Python world for instance, the pains of easy_install and distutils vs seuptools, etc etc. I still find Java dependency management a bit of a pain, because I don't know it very well.
We're just watching a sped up version of a language maturing, and it's painful as individual developers trying to keep up, but I don't think it's as negative as HN often makes it.
Yes.
> Are there aspects of the language or runtime that reward multiple layers of configuration management?
A significant fraction of the node community likes to split their project into really, really small packages and treat them as if they're independent even if they come from the same source repo and tend to get used together. As an example, the popular collection processing library Lodash publishes a package for each method and people actually do depend on individual methods.
There are two major rationales for this. The first is that JS is a dynamic language and in dynamic languages, smaller chunks of code are easier to reason about than larger ones. The second is that the JS community as a whole cares more about artifact size than pretty much any other community. Having smaller packages allows you to produce a smaller payload without having to rely on techniques like tree shaking.
I find micro packages maddening but people give me their code for free and it mostly works so I don't complain about it that much.
Node libraries are typically several orders of magnitude smaller and more single purpose than java libraries. Java libraries like Guava and Spring are typically quite large. A practical example is from unit testing: In Java, you might use JUnit+Mockito for your test dependencies. In Node, to get similar functionality, I have the following deps: Mocha, Sinon, Sinon-As-Promised, Chai, Chai-Moment, Chai, Chai-Things, Chai-Shallow-Deep-Equal, Chai-Datetime, Sinon-Express-Mock, Supertest.
Many Javascript applications both client and server functionality, and there is different tooling for both. Webpack is currently popular for building stuff that will run in the browser, whereas server side stuff is different.
It's a much newer technology than java and there is quite a lot of churn around tooling in general as the nature of the way applications are built change.
Strangely enough both Java and JavaScript were first released in May 1995. Server side JavaScript was released the same year just after the browser version.
Though I get that you are referring to the js eco systems (commonjs, npm) compared to the much more mature Java ecosystem.
Now imagine Office 365, the same software with the same functionalities, except running in the browsers. The task is to essentially write Microsoft Office (front-end logic) entirely in JavaScript (or TypeScript), with the additional problems of dealing with browser compatibilities issues, which are just as hard as OS compatibilities issues.
Then, imagine all the software going to the "cloud". JavaScript now has to have the capability of re-writing all these software originally written in C++, C#, Java... all sorts of languages. It is quickly becoming a superset of all ecosystems where everyone is trying to migrate their software to. So it is not surprising that people are inventing more and more tooling to make it more powerful to suit their needs.
There are tons of libraries for C, many of which aim to do the same, only faster/smaller/scalable/whatever. There are many compilers, both open and closed source. And there are documentation tools, syntax checkers, unit testing frameworks and whatnot.
Of course there are also languages like C# where there’s a de-facto standard for almost everything: IDE, compiler, base library, GUI framework(s).
I think choice is great!
Because, from this outsider's perspective, it seems like most of the tooling is library management (rather than, say, profiling or linting -- does javascript even have a linter? -- or debugging).
There are various packages that are extremely small (see leftpad and associated controversy). They often wind up in a slightly larger package, which itself winds up in a slightly larger package, which recurs until you finally wind up with one actual package of consequence that you're targeting. For example, Express (a very common Node.js abstraction) has 26 direct dependencies, yet 41 total dependencies.
A lot of this results from early Node.js' mantra of having small, focused packages. This could potentially be a good thing because instead of having three major dependencies that have their own way of leftpadding they can all rely on one library and thus code is efficiently reused. This can be bad, however, when they each need their own version of the dependency - or worse - the dependency is removed from the registry (see Leftpad and associated controversy).
One of the legitimate problems that I've seen is that there are various libraries that are just thin wrappers over functionality that is native to Node.js - but written to be "easier to grok". Thus, these thin abstractions become practical code duplication out of some mixture of Node's developers lack of effective verbosity/documentation and application developer laziness. But then they creep higher up the dependency chain because someone at a lower level used it.
On one hand it can be quite simple (and rewarding) to write code while minimizing dependencies. On the other hand, abstractions such as Express make it very easy to write code and it feels like you only have "one dependency" until you look under the hood.
Babel, the most popular transpiler, is 335 total dependencies, and it doesn't even do anything out-of-the-box. You need to add babel-preset-latest for 385 total dependencies if you want to actually transpile anything.
Want a linter? ESLint is 128 total dependencies.
For browser work, you can do almost everything you need with some jQuery, handlebars and moment if there are dates. A few other small libs here and there and you've got almost everything covered. Some edge cases may drive you to use some library that is heavily dependent on 32 different sub-libs but it's really not that often.
Server-side Node.js is the issue, not browser compatibility.
Yes, because the standard library provides practically nothing useful in itself.
Add those together, and you get the need for real complex tooling.
With that said, tooling is nowhere as complex as it is for languages like C++ or Java. People are just used to it on those platforms, and the tooling is more mature/understood because it already went through the countless iterations, 15-20 years ago or more.
Java isn't too complicated either. I've used Gradle and Maven but doing things manually like in C++ can also work.
Crucially, they don't reinvent the wheel every couple of months so a C++ or Java dev can focus on building things instead of relearning how to build things.
http://www.theregister.co.uk/2016/03/23/npm_left_pad_chaos/
TLDR:
- open, community governance that will support long-term evolution
- the technical details get a lot right out of the gate (decent performance, predictability, and security)
http://yehudakatz.com/2016/10/11/im-excited-to-work-on-yarn-...
Are there any branches of ember-cli implementing yarn?
(I don't work at Tilde so I can't tell you if it's been shipping with yarn, but it would shock me if it wasn't.)
I'm sure this will be added in the future, but it is currently a dealbreaker for me - I'm using a single package that's just straight hosted in a private git repo, as the package itself isn't ready to be published yet. I'm sure other people are using private packages for more normal reasons (closed source, pre-production, etc).
If yarn actually made it simpler to refer to a local package during development, I'd be on board with that. (I am developing the dependency, but want to also work on the thing that depends on it, so I'd rather just save locally and refresh npm on the outer project, but that's hard to get right - file urls don't always update, the easiest workflow seems to be delete the package from node_modules, and npm install, with the package having a git url rather than a version in package.json).
0: https://github.com/yarnpkg/yarn/issues/573
1: https://github.com/yarnpkg/yarn/issues/521
I agree local development with many packages can be hard. For Jest we adopted `lerna`
See https://github.com/facebook/jest and https://github.com/lerna/lerna which makes cross-package development in a mono-repo a lot of fun. Maybe that solution will work for you?
With npm I've been frustrated a few times whereby you can fork the repo and point package.json to the git repo but after running npm I usually get a bunch of compilation errors.
It would be wonderful to make this as simple as with bundler within the ruby ecosystem, point to forked url and running yarn handles the rest.
That's insane. It wouldn't surprise me if yarn becomes popular and its proxy to npm slowly turns off and boom, everyone would be migrated and npm would be left with little to no users. Yeah that's probably unlikely for the immediate future but Yarn is positioned perfectly to siphon off npm's users very painlessly.
Deleted Comment
So today by default yarn goes through its servers which then proxy to npm's. As time goes Facebook can add more features to its own repo (custom submissions instead of to npm or maybe it does a federated publish to both at first, nice looking pages with community engagement tools built in, better metrics (npm provides me with almost worthless metrics), etc). Then when enough people are using Yarn Facebook could simply...flip the switch. Then boom, zero npm usage.
I would be terrified if I were NPM right now. They've sat on their hands for years not improving npm. I mean npm3 had a silly CLI animation that made all downloads take orders of magnitude longer simply because of the animation!
P.S. if you're trying to use yarn with a custom npm repository just drop a .yarnrc file into your project's directory then add the following in that file:
registry "http://<my register>"
From their perspective it's very difficult to make breaking changes to the client because of the sheer number of people depending on them. (We've faced similar problems on Babel)
Yarn still makes use of the npm registry and all the high quality infrastructure they've built and support they provide for the ecosystem. They are a critical piece of this.
Granted it's far too early to write npm off. But with how slow they've moved over the years I'm unconvinced that yarn won't take over the space unless it runs into some bad problems. Npm 3 and its launch was an absolute mess and, ultimately, it's almost just a package managing tool. I am unconvinced that breaking changes is much of an issue if at all for them. They could abandon shrinkwrap into a better system and I think everyone would be happy for it; no need to keep propping up that awful system.
Would they, though? If Yarn turns out to be the better tool that doesn't change the way they work (beyond solving some pain points) then would they even care? Facebook could even offload the hosting to some open source group like Apache.
It's purely anecdotal but a lot of people I have worked with would love to see an alternative to npm. It's just slow and the features it gives to authors is pretty limited (I want better metrics, please!). I'm not even sure I would call it sketchy, it would just simply be phasing out something deprecated (I'm not talking about a swift take over more like a slowly-boiling-the-frog-in-the-water takeover).
It's only my opinion and if Facebook really doesn't want that to happen then I guess it won't but it's pretty easy to imagine it, IMO.
Deleted Comment
When did it start becoming reasonable for a front-end only part of the MVCC pattern to have 68 dependencies?
Or for a transpiler like Babel to add 100k+ files? I'm sorry I just find it ridiculous that instead of taking a look at the disease (unbounded complexity), we are looking to engineer our way out of the problem by creating a package/dependency manager that "scales". Are the frontend problems at Facebook of showing HTML forms and buttons on a page really that complicated to warrant such a behemoth of a system?
This harkons back to the days at LinkedIn where I spent my time in a misguided effort to create a distributed build system because our codebase had grown so massive that it literally took a distributed system to build it and still release within the day. I feel bad for people working on these systems because while it is fun and "technically interesting" to solve "problems at scale", you're really just digging the ditch deeper for your company. You are putting lipstick on the 500 lbs pig; making room in the dump to pour in more technical debt.
I don't think anyone ever imagined we'd be in a situation where a javascript framework is approaching the complexity of an operating system in terms of individual source files and SLOC.
Just boggles my mind.
Do any other software platforms out there face the same magnitude of complications?
To speak to one of your primary examples: Babel is freaking huge and somewhat absurd, but is there any other way to advance the Javascript language? Fortunately, this is only a development dependency, and the client doesn't have to see it.
On a happier note, these are good problems to have. This is another indicator of how successful the web platform has become. You can build some absolutely incredible projects these days, and they are instantly available to billions of people who are operating thousands of different hardware devices (with different specs and dimensions) running dozens of operating systems. No gatekeeper, bureaucrat, or corporate curator must approve your work. Just simply deploy it, and anyone with the URL can access it. It is a mind-blowing human accomplishment. I look forward to seeing how the web continues to progress over the coming decades.
Absolutely: this is overkill for your blog or portfolio website, but part of the rising complexity with the tooling and dependency system is due to the rising complexity of the applications themselves.
Deleted Comment
Initial thoughts:
- Why didn't Facebook contribute the updates to NPM directly?
- They are coupling a package manager and registry proxy; the latter has many existing implementations already
- The differenced between Yarn and NPM+Shrinkwrap do not seem substantive; NPM made the design decision to use a sometimes non-deterministic install algo in NPM3 to speed up install times - when network is involved, there is a serious trade off between idempotency and speed
In general, FB seems to love building their own versions of existing tools:
- Flow, introduced 3 months after TypeScript
- Nuclide instead of Atom/Sublime/VSCode
- Jest instead of Jasmine/Mocha
- DraftJS instead of (insert of of the existing 100 text editors here)
- ...
I get that these are useful internally at FB, but I don't think they help the community much. It would be better to work with existing tools and contribute back to them than to reinvent the wheel every time something doesn't work perfectly for their use case.
I get that FB uses these as recruiting tools, it's useful for them to have rights for these projects, and it gives their engineers something to do and be excited about, but I do not want a whole dev tool ecosystem controlled by big FB.
Also, I find IED's approach to speeding up builds far more novel and interesting - https://github.com/alexanderGugel/ied
Do you think Flow was built in 3 months as a reaction to Typescript? Not that it was a big enough problem that two different groups of people independently decided to try to solve it? EDIT: disregard this, Flow was introduced much longer than 3 months after TypeScript. The blog post introducing Flow mentions TypeScript, and is probably worth reading for background [1]
> Nuclide instead of Atom/Sublime/VSCode
Nuclide is an Atom plugin.
> Jest instead of Jasmine/Mocha
Jest provides a superset of Jasmine functionality, and uses Jasmine as its test runner.
[1] https://code.facebook.com/posts/1505962329687926/flow-a-new-...
Did Flow take 2 years to get implemented? I didn't think the initial release looked like a project that had 2 years of dev time on it.
They probably knew that TS was in progress when they started, but I don't have a window into that. I did ask some Flow devs a few weeks ago if they are interested in merging with TS, and they answered with a categorical no.
If you are curious to find out more about why we built it, please read our blog posts, beginning with one about testing performance and our unique constraints at FB: http://facebook.github.io/jest/blog/2016/03/11/javascript-un...
Finally, test runners tend to get stale. Any large organization will run into trouble with any kind of tooling as companies scale up engineers or codebases. Testing is no different and we aimed to solve those problems for Facebook; getting to open source our solutions is an added bonus and we are excited to work with the JS community to build the best tooling we can possibly build.
So yeah, I'm definitely on the "keep reinventing" side of this debate.
Let me explain that a little bit... The javascript "ecosystem" takes the unix philosophy to the extreme in many ways. And one of the points of the unix philosophy is to try to avoid "bloating" tools with tons of options, and instead trying to create new tools where appropriate.
Many of those are perfect examples of where "improving" a current tool with the changes they wanted would end up with a more complicated and difficult to use tool.
- NPM makes tradeoffs that the majority of developers in the JS world want right now. Like it or not, deterministic installs isn't something that most people want or need (at least, they don't know that they might want or need it). Making a new tool that anyone can use when that is needed is a great solution, and it avoids bloating NPM with a hundred little flags and tons of code to cover edge cases. It's the perfect example because anyone can switch to using yarn when it's needed without any extra work.
- Nuclide is actually just a layer on top of Atom. They wanted a bunch of things to all work one way, so rather than try to get Atom to change to include what they wanted, or require developers install a ton of extensions, they packaged them up on top of Atom as one system.
- Jest had different goals at the start. Testing react components was ugly and difficult, and Jest wanted to solve that first (IIRC, i'm not very familiar with jest unfortunately). Again, it's something that can be used if needed, and can even be combined with other testing tools if you want (we almost did that at my work, used Jest to test the front-end components, and our current mocha+friends setup to test the "business logic").
Bullshit, nix systems rely on stable global dependencies, not 50 versions of the same library installed locally. I don't buy the "unix philosophy" excuse. The nodejs community just doesn't care about API stability which is why there is that inflated number of modules with 90% never maintained more than a year.
Deleted Comment
Given npms years of crap I'm ready to give it a shot anyway. This might be one of FB's projects that actually gets some traction, especially given its developed in collaboration with other big names.