But why does that happen? There are now lots of languages that make it trivial to add dependencies. While I find projects in those other languages to also have too many dependencies, it's no where near what happens in JS apps. I'm thinking of projects I've recently worked on in Rust, PHP, and Java. Java projects seem to be a distant second-place to JavaScript projects when it comes to willy-nilly dependencies.
It's not a rhetorical question: Why is the culture with JS so much worse about this?
I absolutely hate that I'm going to suggest this, but is it just because of the average skill level and experience of people working on JS projects?
Or is it because JS is such a bug-prone programming language that we're all afraid of actually authoring any more code than we absolutely have to, because we know we'll waste hours debugging things that should be relatively simple?
I honestly don't know.
Sure. But are hundreds of dependencies really required for this?
In Java you would use tools like Guava or Spring for general quality of life improvements, and there would be a few deps for them (under a dozen iirc).
The solution is for the “top tier” libraries and frameworks in the JS world to be designed with minimal dependencies. And where they do have a need for a dependency, they undertake serious consideration of the best option that minimises dependency hell.
You are meant to pass an argument to [].sort; If you don't, it falls back to the most generic thing that makes sense -- which is turning everything into a string and sorting lexicographically.
Arrays in JS can have any type in them, such as [number, string, function]. That's obviously very unlikely, but the only thing in common between all types in JS is that they can all be explicitly turned into strings.
And yes, I agree that throwing an error here for no argument would be better here (a linter WILL enforce this), but this is hardly a critical shortcoming of JS's standard library, and you DEFINITELY do not need a utility library just to use the language (especially for your examples).
> [1, 2, 10, 3].sort()
[ 1, 10, 2, 3 ]
I suppose "critical" is debatable but this seems very fundamental and very unexpected to me.
> you DEFINITELY do not need a utility library just to use the language (especially for your examples)
I hadn't actually considered using a linter to avoid these types of standard library footguns... that's actually a pretty great idea!
But for example, they probably decided that it was more useful defaulting to having a sort order for things that otherwise aren't comparable:
a = [undefined, null, undefined, null]
a.sort()
// [ null, null, undefined, undefined ]
If you have an array that's a huge mix of random values, from null to undefined to [] to {} and you sort() it, all of those values will now be grouped together by type.
This doesn't seem like a bonkers trade-off.
I agree that without an argument it should just be a fatal error, but if it were to have any sort of functionality, it should convert all to strings.
This is consistent among JS's apis, and pretty much the origin of all the 'js wut' moments on the internet. Everything in JS can be converted into a string. If you do something stupid with disparate types, it will likely turn operands into strings and compare them that way.
They learn engineering principles like "don't repeat yourself" and take that to mean installing an entire dependency to implement left pad is a good idea.
You are probably on to something with JavaScript being bug-prone as a factor in that.
JS as an ecosystem has a really big problem with developers not knowing the value of simplicity.
People design new frameworks (presumably) because they see an array of problems with existing frameworks. In designing their new framework, they try to address the shortcomings that the existing frameworks have.
What they don't realize is that the problems in existing frameworks were _known tradeoffs_. Now, instead of the One True Perfect Framework, we have yet another framework with its own set of problems.
People think that every problem is solvable simultaneously, but that's simply not true. You can make tradeoffs. And this isn't just true in engineering, it's true in life generally.
Some tradeoffs make sense nearly always. Others only make sense in certain contexts.
An example here is the tradeoff between simplicity and high availability. It doesn't matter what you do -- the simplest high availability configuration for an app will ALWAYS be more complex than the simplest non-HA configuration. You're making a trade here. It's a trade that is absolutely sensible, but it's a trade nonetheless.
The lesson to be learned here is: stop thinking you can solve every problem at once. You can't.
And of course, there are just far more people working on it, I believe.
I don't know all the reasons for that, but I think part of it is that developers create them in order to put it on their resume. "My package is downloaded 500k times a week"
I've seen this same weird line parroted by so many people in the comments here: it's very odd.
If you're talking about server & browser having different APIs that's equivalent to any other language having different platform apis.
The same is not true for JS. The most mature libraries depend on absurdly vague libraries that no one has ever reviewed.
It's a giant pain in the ass to publish a Java library. That's already weeding out a ton of low-effort projects. By itself, I wouldn't exactly call that a good thing, but it seems to have a silver lining...
https://docs.python.org/3/library/
They don’t make third party libraries obsolete— for example, I tried using the built-in IMAP library the other day and it’s definitely too low-level to make sense for most quick projects that check email, so I used a third party Library— but all of those modules are vetted, stable, and require no external dependencies.
I believe the node maintainers have staunchly opposed such measures. I don’t know what their reasoning is so I don’t have an opinion on whether or not it’s worth it.
It feels like a quarter of the time I spend working on projects that use npm is spent debugging my toolchain because of excessive complexity or weird problems in random dependencies. Doesn't seem like it should be too much to ask that I can spend most of my development time working on actual code.
Not having a standard library made people accept needing libraries for even very small things that in most other languages developers would make at least some effort for using the standard library before reaching for something else.
I think another aspect is that the initial leadership of a language community sets the tone for a long time, but JS in a lot of ways didn't have that, not through any fault of any particular person but simply because Node grew so explosively at the beginning that the usage growth dominated the available leadership growth, and so there was a lot of very wild, woolly growth that got written into the earliest culture. This creates something a lot like a "seed crystal" that has outsized impacts on future paths for a long time.
Finally, I do think it is definitely an issue that JS is somewhere where you get a lot of people who are not "programmers" per se and they are making big decisions about code bases and libraries. They're young in the art. And while there's nothing intrinsically bad about that, people have to start somewhere, there's a lot of ways in which it's good that JS is relatively easy to get into, etc., it is also absolutely true that at scale, as the language and the libraries iterate on each other and seek out their stable points, that's going to affect the landscape. This is an "is", not an "ought". It is what it is. JS has also continued explosive growth, so even as someone who got into JS and programming for the first time in 2017 is now a 5-year "senior" (a crack about our industry terminology, not the dev here) developer who has learned and might be inclined to do things differently than they did 5 years ago, there's another 2.5 newbies "voting" in the community as well.
It's a hard problem, I salute the leaders in the JS and npm world working on it, I wis them the best and advocate people giving them grace working in a very hard situation. But I'm also glad not to be part of it.
But even JS's built in string stuff isn't so bad that it somehow justifies leftpad existing, so I don't know...
In other words it has been an intentional design decision from the start.
JavaScript's standard library is so thin on the ground that there's already a culture of "reaching for a library" to accomplish tasks that many languages do out of the box.
The monoculture is wide enough that the language caters to lots of paradigms and schools of thought. If there's one library that uses classes and method chaining, you can be sure that another will pop-up to re-implement the same functionality in a pure functional style. One will focus on type safety and another will abuse the dynamic bits of the language to make the code you write as terse as possible.
Amount of code shipped has always been a more important metric for JS than other languages because the nature of the web means that users have to wait whilst the source code is downloaded before your page becomes interactive (for a huge class of applications). This encourages developers to favour smaller libraries that solve for narrower problem domains.
It's become very trendy to write a smaller, faster, better, smarter version of existing libraries. The JavaScript community loves the process of picking a catchy name, registering a domain, designing a logo, and publishing packages as though they were businesses. This creates an abundance of packages that look great on paper, but with no users, patchy/non-existent tests and maintainers that haven't ever used the code in a professional context.
Finally, I think JavaScript is a deceptively simple language. It doesn't take very long before people (mistakenly) think they're close to mastering the language. By comparison, contributing to an open source project in a meaningful way is quite difficult, so these developers assume that other libraries must be written badly if they find it hard to contribute. Then they write their own, because they believe they can do a better job.
The ecosystem as a whole sees a lot of innovation, and pays for that with a lot of churn and a lot of dependencies. From a theoretical standpoint, it's a fascinating corner of modern programming. In a professional context, it horrifies me and I wish I could sanely cut npm out of the chain.
My experience with these new languages is such that this feels a bit unfair. It's like insisting that a disaster with 1000 fatalities is "much worse" than one with "only" 200. It's ... true ... I guess, but there's something uncomfortable about making the comparison. Something has gone badly wrong if the comparison even needs to happen in the first place.
What I'm getting at is that e.g. Rust has an enormous problem in this area. It's not uncommon for me to see Node projects with over a thousand transitive dependencies, but on the other hand, I very frequently see Rust projects with over a hundred. And the Node projects tend to be more complicated than the Rust ones; they do more.
Take the last Rust program I tried to use, tealdeer. [1] If you don't know, tldr is a project that provides alternative simplified man pages for commonly used programs that consist entirely of easy to understand examples for the program. [2] What a tldr client needs to do is simply to check a local cache for each lookup, and if necessary update the cache online. It's a trivial problem that can be, and has been! [3], solved in a few hundred lines of shell (if you're being extremely verbose). How many recursive dependencies would you guess tealdeer uses? Depends on how you count, of course, but as of today the answer is ~133 deduplicated dependencies! For a program that's a glorified wrapper around curl!
Or another Rust program I looked at recently, rua [4]. In Arch Linux, the AUR is a repository of user maintained scripts for building and installing software as native Arch packages. Official tools for building and installing software already exist for Arch, but it is common for users to use a wrapper around these tools that makes fetching and updating the software from the AUR easier. It's a relatively simple task that (once again) can be done with shell scripts. rua is such a wrapper. As of today it uses 137 deduplicated dependencies!
These Rust programs are simple terminal tools to do tasks that are almost trivial in nature. And yet they require hundreds of constantly updating dependencies! The situation may well be better than what you'll find for Node, but it's undeniably disastrous compared to either simpler languages without a built in package manager (like C) or more complicated batteries-included languages where best practices continue to prevail (like Python).
[1] https://github.com/dbrgn/tealdeer
[2] https://tldr.sh/
Putting a layer of "distributions" over language ecosystems, in the same way that "distributions" solve the problem of putting enough mutually-compatible library versions together to get Linux to work, is, I think, inevitable.
Node specifically doesn’t have this constraint, though, as A’s C and B’s C can be loaded independently into the same VM, each one hermetic and only visible to its parent. (This is probably half of why Node’s ecosystem became the way it did, now that I think about it; every other ecosystem hits increasing numbers of constraint-resolution conflict problems as dependency hierarchy depth increases, and so limits itself in hierarchy depth to avoid this.)
Rust can do this as well, though most version variation is handled via semver rules wrt. compatibility. I think this exact requirement led to some controversy in the Go community at some point? Though they should now have a module system that allows for this?
For example, you can't submit any old Haskell library you put together to Hackage, or you couldn't at the last time I checked. It had to meet certain minimum standards to be considered.
I honestly don't know where you would start in terms of curating NPM, precisely because of the dependency on dependencies. You'd end up curating half of the ecosystem.
You can, but you need to figure out what your library is doing and translate it to category theory.
My node projects typically have thousands of dependencies, my java and C++ projects have hundreds. My go projects typically have 8-12, and have never once exceeded 40.
Not to stan the Go language itself, I just wish this philosophy was the prevalent one in languages.
Rust has this flaw too. Last time I wanted to compile a simple project using actix-web and probably a database library, I had 200 crates to compile. In both cases I think it's due partially to preferring small packages/crates.
From my experience I would rate the count of dependencies in this order:
NodeJS > Rust > Python > C/C++
Though I cannot explain why this is ordered this way between {Node, Rust, Python}. Again, batteries included? Language popularity? Beginners-friendly programming language?
I don't know anything about the C#/Java ecosystems, same with Ruby and Perl. I'd like to know if they fit this "model" too, that would be interesting. It could give some pointers on how to design/make a language evolve to avoid having lots of packages.
I'm not sure about the standard library of Rust being bigger. Node ships with an HTTP module, random number generation and regexes. Rust has neither of those. Maybe more functions in each modules?
My vote is that its cultural. Both for "batteries included" python, and for nodejs.
Its easy to forget now, but npm was extremely innovative when it first came out. It absolutely went out of its way to make adding dependencies as easy as possible. And the culture (especially in the early days of nodejs) went bananas for this philosophy of programming. I had a chat with @isaacs on a bus one time. (He was the maintainer of nodejs at the time). I asked him what he thought about package documentation, and what to do when a README isn't enough. He said that he thinks if a readme isn't enough documentation for your library, your library is probably too big and should be split up.
You still see this today with packages like "isobject" (a tiny function published as a package) which still gets 53M downloads / week[1].
Correct me if I'm wrong, but as I understand it python and ruby still don't support parallel dependencies with different versions in the build tree like npm does. If a python or ruby package transitively depends on [email protected] and [email protected] then my understanding is that ruby and python lose their minds. And this problem is almost impossible for the end user to solve. So libraries like rails sort of need to be designed as one big block of software.
Nodejs has no problem with this - if you do this in nodejs, npm will just quietly install both versions and node will happily wire everything up correctly. The only constraint is that no single package can have a direct dependency on both foo@1 and foo@2 at the same time. But thats not something that you ever really want to do in practice.
The tooling supported this too. It was quite common to have a library with some optional features that not many people used. If you wanted to exclude them from your javascript bundle, until recently the main way to do that was by breaking your library up into small pieces so your library's consumer could pick and choose what they wanted.
Building a dependency tree out of with thousands of tiny modules is exactly what nodejs is designed for. It should come as no surprise that thats what we got.
Because I don't think those are useful metrics at all. In fact this is just code re-use and should probably be happening to a significant degree.
Some actually interesting properties are:
- basic performance
- signal to noise ratio
- API stability
- robustness
- clarity and simplicity...
And more of that. But just saying "a lot of dependencies bad" is not useful.
Maybe in an ideal world we have even _more_ dependencies? I don't know. What I do know is that I care about the above points much more.
I disagree. The primary problem is that proper dependency auditing is incredibly time-consuming, especially if you want to stay up-to-date. The reality is that most people simply don't do it all.
The reason this keeps happening with NPM is because of absurd number of dependencies in the average node app. I have a tiny app I've been playing with using create-react-app. There are over 800 directories in node_modules. That absolutely dwarfs the number of any other language I've used. Even in a medium sized rails app, you likely have some awareness of what every dependency is. It's just impossible with npm.
This makes it easier for someone to inject their package into the ecosystem whether it's actually very useful of not (like the colors package).
One thought I've had to "reboot" the npm culture is to somehow curate packages that are proven to have minimal and safe dependencies, probably through manual review. Maybe it could be recursive, so that safe projects only rely on other safe projects.