There are tons of tiny libraries that have gotten themselves into the dependency chain of popular libraries, and any attempt to remove them is treated as a turf war.

I've tried to remove single-line dependencies only to have the PR rejected by the creator of the tiny library who happens to contribute to the parent library.

IMO the Javascript ecosystem really needs a decent standard library to remove the inordinate amount of power granted to these these tiny library squatters who wrote 5 lines of code and a package.json file 10 years ago.

What "inordinate amount of power" is this really though? If they're a contributor to the parent library then they can already put whatever they want into code that will get executed by many people; meanwhile having libraries properly broken out into smaller pieces is great for maintainability. Let's not throw the baby out with the bathwater.

The parent library in that situation was a popular library with a whole team of contributors who could reject malicious PRs. But a PR that just updates every dependency (including a malicious update to the tiny library) can easily go unnoticed.

And that was one situation. The mindset of the Javascript ecosystem is still to maximize code reuse, meaning even if the tiny library maintainer isn't a maintainer of the parent library, the parent library still frequently clings to their one-line dependencies when I've tried removing them. Thus granting the tiny library owner tons of power like the maintainer of "colors".

> I've tried to remove single-line dependencies only to have the PR rejected by the creator of the tiny library who happens to contribute to the parent library.

So fork it and throw out everything you don't like.

Exactly. In this case, the package manager should use the version of colors that svgo asked for, not the one that appeared on the internet 5 minutes ago.

It's damned if you do, damned if you don't for packages in the middle of your direct dependencies and a sub-sub dependency with an issue (be it security or tantrum based).

These middle deps could pin the exact version, but then when a security vuln is found and a patch issued, these libraries also need to update. This is like a traffic jam. If you're 6 hops from the vulnerable package you need to wait for not one, but six maintainers to push an update to npm before you can clear the security warning.

To get around this, middle packages list semver ranges. And then you have your occasional left-pad issue.

If I had to choose between those two ways to lose, I would use server ranges. The only way to win is to not play at all - have no dependencies.

as an example look at the log4j mess. "Everything" in the Java world needs log4j updates, but some dependecy's dependencies pull in some specific version.

There you need the way to say "get the new version from 5 minutes ago" without waiting for all levels of the dependency hierachy. Especially as there were a bunch of emergency releases in short sequence and younhabe tonmake sure youbget zhe latest one.

Dependencies are a mess. NIH can't be the solution either, though.

> "Everything" in the Java world needs log4j updates, but some dependecy's dependencies pull in some specific version. There you need the way to say "get the new version from 5 minutes ago" without waiting for all levels of the dependency hierachy.

At least with Maven, that's extremely simple. Just add to your project a dependency on that "new version from 5 minutes ago" of log4j. Maven always prefers the version from a direct dependency. You don't have to wait for "all levels of the dependency hierarchy" at all.

The problem is that, AIUI, each NPM package installs its own, isolated set of dependencies. It has facilities to manage versions down in the dep tree, but it’s not as simple as other managers which ultimately install a single set of packages that satisfy all version constraints.

many java jars contain other jars withbthe dependency, so users don't have to run maven or something like thst, but jsut grab the jar. This is even worse. (there was no package manager early on, thus many java developers aren't taught that way, even decades after Maven was created ... and for end-user products it makes sense to bundle)

No, we really don't need a way to say get the last version from 5 minutes ago.

We need to get the version that we tested with and know works, which is what it does. When we need to update, we say what version we want.

Java can be a pain, but I'm very glad it doesn't handle dependencies like npm.

This is also saying you can't easily update from 1.2.3 to 1.2.4 without all of your dependencies which specify 1.2.3 also being updated without having an override mechanism. This isn't a simple problem where one option has no downsides.

Oh, I agree that there's no simple solution that solves everything.

I'm still glad that basically noone else handles dependencies like npm.

Everybody else handles dependencies like npm, e.g. installs the latest versions that satisfy constraints. Some package managers have the ability to install from a lock file (e.g. Python's Poetry, Python's Pipenv, Rust's Cargo, npm) but will still grab the latest version when the lockfile doesn't exists (e.g. cloned a project with no lock file, or just added the dependency yourself), and have a command to update which updates every single package to the latest compatible versions.

You can argue that npm's commands are poorly named and guide users towards bad defaults, but saying that it works in a unique way is not true.

I guess my experience is limited. For example, maven dependencies usually specify the exact version required in the pom files. At least, on all the projects I've ever dealt with.

I’ve seen plenty of pom.xml files which are unversioned — not having a standard command to update means less disciplined developers say it’s too much work – but increasingly in other languages (and Gradle IIRC) there’s a distinction between the metadata file listing your direct dependencies and a lock file detailing exactly what was installed. The idea is that (using Node as an example) I’d say “I depend on the AWS SDK” in the main file, which changes infrequently when I add or remove direct dependencies, but my tool would use the lock file (npm-shrinkwrap) to record the exact versions of every package, preferably even by file hash, so you can have a highly repeatable build.

Separating the two is handy both due to frequent churn and to avoid transitive dependencies being kept unnecessarily — I’ve seen projects where libfoo stopped depending on libbar a while back but they were still installing it because someone had copy-pasted that block years before and was just incrementing the version.

Thanks for the explanation, makes a lot of sense :)

You’re welcome!

Well, in a log4j situation or similar, I am sure there are cases where it is preferred to update first, and fix production later.

So that would be “get the one from 5 minutes ago, no questions asked”

Nothing stops you from updating to the latest version and not testing it if it's urgent.

You just specify the latest version. But you do it explicitly.

You can ask for specific later fixed log4j version. You don't have to ask for unspecified latest one.

This isn't the problem being discussed: when something like log4j happens, everyone in the world needs to update. The difference is if one of your dependencies pin 2.14.0, you either have to wait for that project to also ship an update or do something more complicated whereas if they specify less than version 3 you can immediately ship that patch.

Something like log4j or many packages in the NPM world are so widely used that on any non-trivial project the odds approach certainty that multiple dependencies will specify versions. The more specific the pin, the more likely it is that you'll be unable to ship a security update without risk of a functional change or blocking incompatibility. You can mitigate that with good CI/CD infrastructure and lots of automated testing, but that means taking on more infrastructure expense.

What if the patch for the buggy `2.14.0` library was released as `3.0.0`? In semver MAJOR can be a superset of MINOR and PATCH, so this is a perfectly logical semver operation.

You still have the exact same problem you described with `^2.14.0`. Someone would have to manually update the package to get the security fix in 3.0.0.

Unless you're suggesting code should also automatically update major versions aswell?

Yes, if someone chooses not to follow semver correctly they can create problems but there are a somewhat unlimited number of ways in which an untrustworthy maintainer can do that. The difference is that following semver means it's easy to not do that since they can always ship 2.<latest + 1> with no changes other than the security fix.

This is following semver correctly. A PATCH update may be a MAJOR update (since its a superset) and it may be considered a breaking change.

> The difference is that following semver means it's easy to not do that since they can always ship 2.<latest + 1> with no changes other than the security fix.

If the line between bug and feature were clear. (Log4j worked 100% as specified btw. in regards to log4shell)

https://xkcd.com/1172/

You don't have to wait till that lib updates. You update version yourself and overwrite it.

Second, log4j is bad examples. Libraries don't pin that at all and people report bug is they do. Libraries are supposed to depend on logging api in general and end project decides whether use log4j or slfj.

What if NPM allowed a "global minimum version" for any package found to have critical security vulnerability?

global for what? How much tracking of issues do I have to make? In an ideal world (which we can't have for multiple reasons) I get security fixes by default. (and no other breakage)

The problem is by default `npm install [package]` will put a "give me [package] that appeared on the internet 5 minutes ago" into your package.json file by default. Until `npm install` pins to a minimum version by default instead of a maximum version by default, this will keep happening. It's a mess. The benefit of "maximum by default" is that you pick up security fixes by default. So... pick your poison, I guess.

On the other side of the spectrum, if no one was installing 'latest', then no one would be testing latest, and we wouldn't catch these bugs until later. We would just all get surprised after 2 weeks (or whatever arbitrary delay) until the bad version became the blessed default version.

In other words.. If there isn't a trusted test pipeline then there's no benefit in delaying the latest version. Might as well just get latest.

In other contexts, the best ways to deal with trade-offs use randomness. Perhaps this sort of stunt would cause less damage if npm just randomly chose which versions to use. To avoid churn, the random choices could be stored in lockfiles. The point would be that not everyone is using the newest version of any particular package.

People starting from scratch would get latest, whereas people maintaining existing projects would be able to upgrade when they feel it’s safe to to so. Do you really want the people maintaining the nuclear reactor code being testers for latest?

> should use the version of colors that svgo asked for

Another thing about setting exact dependencies is reducing duplication. Libraries are encouraged to use loose ranges, because if everyone pins exact versions for every dependency, then you could end up with 10 to 20 different copies of 'colors' in your tree, instead of having just 1 or 2 copies that work across the board.

If you are unconsciously shipping something, you shouldn't be shipping anything.

Our industry has gotten along for far too long with zero liability or accountability.

The people posting that it's not their fault, that they can't possibly vet the code they ship, etc. are basically people who don't have any professional ethics.

Not only that, even if you excuse their lack of ethics, there's a basic competence issue: they have a basic failure to learn from history and the most basic part of this, which is that you shouldn't be pulling live from the internet.

I'm very sympathetic to your arguments in this thread, but it seems like in practice doing it right in your eyes would mean either 1 - not using dependencies at all, or 2 - only using dependencies when you have read and understood every single line of code in both the immediate dependency and every sub-dependency in its tree. Neither is realistic, unfortunately, for building anything non-trivial without a huge team of programmers.

Perhaps you have another solution in mind? I think something along the lines of package-level content security policies with granular permissions has a lot of potential.

Well

I did try (2) for awhile with some success. You basically vet everything you import.

This generally follows the same rules of supply chain validation expected _and common_ in other industries where people actually give a crap about what they are doing.

In the case of large projects, especially those with a solid track record of testing and security issue management, the vetting is organizational. You don't need to read every line of Kafka (though you should, as I did, actually look through it to see what you're getting into), as an example, at least today.

But for smaller projects, just like with smaller vendors, you need to go deeper. You do need to read the code. Then you look at their issue tracking and see if they have any concept at all of quality, sanity, security notices, what their maintainer attitude is, and so on. You definitely read the code. If you're sloppy, you at least read through a good sample of the code. Again, diligence - you have the obligation to your downstream to know what you are getting into.

This is no different than the vendor management you expect in pretty much every supply chain that you interact with. Food, cars, medicine, pharmaceuticals, building materials, heavy machinery, embedded software in medical devices, and so on. Software has gotten away with complete negligence for a long time and instead of addressing it, the common practice has gotten worse and worse. At some point, we are going to accidentally kill someone.

Then, on top of that, you lock it down. You mirror it, you pin it, you make an active plan to monitor the project to make sure you are aware if it is abandoned, hijacked, deleted, etc. You take responsibility for dealing with these possibilities. You mitigate immediate threat by not pulling random crap live when you build. You further make sure you partition your system such that code you have not classified as high quality has a reasonable blast radius and security boundaries.

But whatever, let's say you're a typical valley company that operates like a psychopath and you don't care about that. What companies should be worried about is liability. Yes, most of the security activities for most SaaS and other SW is laughable and mostly theater and checklists. But there is liability nonetheless. After all, you are about to represent it as high quality software and you are taking on the liability. No judge in the world is going to let you go with the "but it was this javascript we pulled in, not ours" excuse.

We built quite substantial systems using this approach, and it was important in the domain that we were working in to have done so. It is not without hassle and cost.

Really, what are we talking about here? It's _too expensive_ or _too difficult_ to have even the slightest quality verification of software? The industry is a complete joke if that is true.

I mean, the bare naked reality is that software, web software in particular, is still like... a hundred-billion dollar industry. Maybe more. As long as it's still profitable enough to deal with these supply chain attacks occasionally, and nobody legislates or regulates things, we'll keep lumbering along like this. I'm disappointed, and I don't like the current state of affairs, but we can't just expect anything to change in the absence of any kind of external pressure.

> and nobody legislates or regulates things

Completely agree, which is why I support such endeavors wholeheartedly.

I did not agree with regulation until this post on hacker news.

The comments here have convinced me that the industry is basically beyond self help.

I agree with you, but it’s not my opinion which matters, it’s my bean counting bosses opinions’

Short term bean counting always sets you up for failure. The fact that this guy's packages were depended upon by so many projects and people were not actively supporting him with money shows how broken open source is. The projects with the most dependencies should be rooted out and collectively paid for or a new license should be made that excludes the use of OS software by corporations unless they pay a minimum amount based on their size. I'm sick of hearing about stories like this and it has discouraged me from releasing much of anything open source that any corporation could use without paying me. That's why small projects that I do release get GPL licenses and I don't put my best work out there for free.

Indeed and crying for the little developer is of no help, the little coffee owner is liable for everything that gets sold, the kitchen cleanness and the good state of the food being packaged.

Imagine a car manufacturer like "I wish I could say I spent the hours looking at every valve and every screw..."

I suppose they must. Because car is a hazard on road and may not end well if you don't put enough man-hours within design, documentation, testing etc.

Now, who's gonna pay you for writing that single functionality whole year? Sure, you tested, documented, did everything right, you may have almost no bugs, pretty code, decent test coverage, unit tests, integration tests, UI tests, performance tests, edge cases ironed out, top notch performance, UX no-one matches, one click deployment, static code analysis, linters, fuzzers, vulnerability scanner tools, monitoring, auto scaling... while you get there, your startup may be no more. Or maybe just a money sink: https://thedailywtf.com/articles/unseen-effort

Alright, I'm little off the track, but not every software project is comparable to automotive/air/rocket/med/... industry. Ofcourse you will put 10x+ more money/time/effort if human lives may be impacted with your commit.

To put that into context, let's appreciate SQLite - software that is thoroughly tested and your airplane shouldn't be afraid to run those. From the creator of SQLite:

> I’m going to write tests to bring SQLite up to the quality of 100% MCDC, and that took a year of 60 hour weeks. That was hard, hard work. I was putting in 12 hour days every single day. I was just getting so tired of this because with this sort of thing, it’s the old joke of, you get 95% of the functionality with the first 95% of your budget, and the last 5% on the second 95% of your budget. It’s kind of the same thing. It’s pretty easy to get up to 90 or 95% test coverage. Getting that last 5% is really, really hard and it took about a year for me to get there, but once we got to that point, we stopped getting bug reports from Android

https://corecursive.com/066-sqlite-with-richard-hipp/#testin...

This is part of why cars cost $30k apiece. If open source maintainers saw the same kind of revenue as car component suppliers, there'd be a lot more paid QA jobs.

Pretty sure they don't MRI every other batch of rust covered by paint received from yet another shitty frame manufacturer. Many PSU, GPU, etc manufacturers often ship first batch with hi-class components, and then "downgrade it a little" or "lose control of a dealer" when it sells nice.

There we go again, someone always has to mention “rust” in every thread about programming languages or dependency management…

;)

Spot checks and manufacturing line quality monitoring absolutely do happen in the process you are describing, which is a hell of a lot more than the weak excuses for not even bothering to mirror the SW and look at diffs now and then accomplish.

Just out of interest do they generally vet the factory that made the steel for those screws and bolts?

The difference is that 1 person responsible for installing 1 valve can't disable every car sold in the last year if they decide they aren't being paid enough.

They better, given liability in the industry.

What we are missing on the software industry is proper liability.

Nailed it. Shallow dependency trees are much easier to maintain and secure.

It's the responsibility of "svgo" to make sure it's direct dependencies are alright.

> It's the responsibility of "svgo" to make sure it's direct dependencies are alright.

No, it's your responsibility to make sure your dependency tree is alright.

Your personal definition of 'alright' may be more easily satisfied if the packages you choose to depend on autonomously practice some level of responsibility towards their own dependencies. Choose wisely.

But there is no way to dictate your requirements to dependencies, or impose some kind of responsibility or demand some kind of warranty. You can accept what is offered, or not. In fact, if you are using svgo, even indirectly, you have agreed to this: https://github.com/svg/svgo/blob/main/LICENSE

>THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.

So svgo doesn't have to care and you can't make them. Even if they do care, there's no guarantee they will meet your standards - and you still can't make them.

If you want someone to blame, find someone willing to sign a contract that says you can blame them.

Efforts within NPM and github to control this situation are simply the bare minimum of case-by-case disaster mitigation, in the interest of reputation alone. If you're using their infrastructure, you apparently find this acceptable.

Is it? In retail, AFAIK, if a store sells you a defective product, they are liable (or at least partly liable). It doesn't matter that some other manufacture made the product. The point being, responsibility is shared.

You're responsible for every dependency you add to your project and that includes all sub-dependencies. Your users will sue you for not doing your due diligence. You may turn around and try to sue your suppliers but that doesn't absolve you of your responsibility.

Who's paying svgo to do that checking?

NOTE: this isn't an open source versus closed source thing. Linux has distributions which test included packages (to varying extents, I'm sure) and some of these are commercial operations. It's not impossible to have code whose verification you have paid for, to one extent or another, even with open source. (and hey, you can install malware with automatically updating closed-source see Solarwinds).

Sure, that is technically correct and if your only goal is to assign blame that's helpful to point out. But it does nothing to prevent this from happening again.

The point here is how we can make it easier for svgo and every other package to avoid the problem in the first place.

* It should be the responsibility of "svgo" to make sure it's direct dependencies are alright.

Why can't you point that attitude in both directions?

So.. basically you are OK shipping unvetted, attacker-controlled code to your users?

Basically, no. I'm not OK with it

Reminds me of my time at a global bank. After many security incidents, they finally decided to step up their game, and implemented multiple security scans.

For any project using NPM, it was an absolute nightmare. Dependencies 3 or 4 levels deep would get flagged, and there wouldn't be a way to resolve them. The security teams didn't care, and multiple projects were left stranded. Since resolving those issues would mean having to contribute to open source, which we weren't allowed to do.

On the other hand, because we had specific security teams doing the scanning and assessing the results, you weren't allowed to question them, because they could just block your entire project. So developers were extremely unhappy with them, and what ended up happening is that developers just did what the security teams asked, without questioning. Which to me, let to more security concerns, because we were just pulling in libraries without knowing what had changed.

Sadly, the few devs who stopped relying on third party dependencies all left, because most of their work took longer (since they were implementing stuff that third parties usually do). Business teams took notice and questioned why everything was taking longer.

We used to use npm audit at my last job, and it’d regularly flag dependencies 20+ layers deep - usually with something like create-react-app or jest at the top, and some trivial obviously not actually a problem issue at the bottom.

The resolver tool we used couldn’t fix dependencies that deep even when there were fixed versions available. And since their internal processes didn’t use the same audit tooling, Facebook devs would often close issues requesting dependency upgrades to fix those not actually security issues. I think Dan Abramov wrote a post on overreacted about it at some point too. I’m not convinced any of the time we spent on those issues was well spent to be honest.

There's obviously very valid security concerns sometimes. We didn't use npm audit, but an external tool. And often it would highlight things like "use https", and when looking at which line, it was just a comment that referred to an external link.

Obviously a non-issue, but you still had to verify, and document everything. Best practice would be that we ran this scan on every commit, but the scan took about 15-20 minutes. It just ended up being something that was part of our release process, with the obvious danger of breaking our application right before release.

All in all, it stopped me from relying on third party modules, and tried to use as much vanilla code as possible.

> that wouldnt leave much time for anything else

By the time you got to the end, it would be time to go back to beginning and start over again...

From the Department of Funny But Hurts Because It's True:

https://www.reddit.com/r/ProgrammerHumor/comments/6s0wov/hea...

>Heaviest Objects in the Universe:

>Sun

>Neutron star

>Black hole

>node_modules

Unfortunately the way that this can be prevented would be to audit the packages before including them or having your own package management that you control.