News for hackers. Stuff that matters.
Re:A web of trust for NPM by ratww on Saturday October 03, 2020 @06:01PM (#24675010)
I will die on that hill, so here it goes again: The problem with NPM is not the amount of runtime dependencies.

Most Javascript projects would actually fare pretty well when compared to other languages if only runtime dependencies were taken into account.

Javascript staples like React, Vue, Svelte, Typescript and Prettier actually have zero runtime dependencies. Also, the ES6 standard library is not as bad as people claim.

The real problem is with development dependencies. The amount of dependencies required by Babel, Webpack and ESLint are the cause for 99% of the dependency bloat we complain about in JS projects. Those projects prefer to have monorepos, but when it's in your machine they're splitted into tens or hundreds of dependencies. Also remember that left-pad was only an issue in 2015 because a Babel package required it. If those projects were able to get it together we wouldn't even be having this conversation. Solve this and you'll solve the biggest complaints people have about JS.

I really would like to see a discussion on this, as most people seem to put a lot of blame on JS as a whole, while it's mostly a handful of popular projects generating all the complaints.

Re:A web of trust for NPM by preommr on Saturday October 03, 2020 @09:52PM (#24676046)
> I really would like to see a discussion on this, as most people seem to put a lot of blame on JS as a whole, while it's mostly a handful of popular projects generating all the complaints.

Discussion on what? You're right.

I know we joke about left-pad, but like you pointed out, a lot of big js hitters don't have many if any dependencies. That's true, but irrelevant.

Those dev-dependencies are still potential security threats, with all the minification and other crap, it's really hard to know what gets injected into the final runtime. And if not security, it's still development hell. Development, yes, but that's a if not the thing that programmers really care about.

And even if runtime dependencies are less common, there are a lot of developers that still do ascribe to using as many deps as possible especially because the web can be quite fragmented and they have to support a myriad of different target platforms. So even if it's a lesser issue, I think it's fair to talk about the js ecosystem as a whole when making criticisms about it's dependency disasters.

Re:A web of trust for NPM by ohazi on Saturday October 03, 2020 @10:01PM (#24676077)
Development dependencies are especially terrifying because we generally don't use any sort of sandboxing. Any one of these dependencies could append to .bashrc to get your computer to run literally anything, and then hide the evidence.

And developer machines are particularly juicy targets because they often have ssh keys to production machines lying around.

Re:A web of trust for NPM by mike-cardwell on Sunday October 04, 2020 @05:06AM (#24677415)
This is precisely why I created https://gitlab.com/mikecardwell/safernode - I can run "npm start" or "npm install" just like any other nodejs developer, but node is not installed on my host, and my .bashrc and any other files in my homedir are not at risk or being read or modified
Re:A web of trust for NPM by lmeyerov on Saturday October 03, 2020 @10:40PM (#24676173)
and even scarier if on a corp buildbox server that typically firewalls inbound, not outbound/lateral
Re:A web of trust for NPM by fn1 on Sunday October 04, 2020 @01:20AM (#24676687)
> If those projects were able to get it together we wouldn't even be having this conversation. Solve this and you'll solve the biggest complaints people have about JS.

So the way to go would be

1. compile a list of opensource-dependencies by usage/complexity

2. go through their dependencies

3. find easy fixes and create PRs for the projects.

Re:A web of trust for NPM by ratww on Sunday October 04, 2020 @05:33AM (#24677565)
That actually sounds like a good idea, but you need step two-and-a-half: convincing the maintainers of those dependencies that using this strategy is hurting the ecosystem.
Re:A web of trust for NPM by striking on Saturday October 03, 2020 @08:47PM (#24675816)
> Solve this and you'll solve the biggest complaints people have about JS.

Or, perhaps, we can just tell people that their complaints are nonsensical when most of the deps are not actually packaged into the final app or being used at runtime, instead of contorting ourselves to fit their silly worldview.

I'm not losing any money over this. Why should I care?

Re:A web of trust for NPM by ucarion on Saturday October 03, 2020 @08:59PM (#24675860)
Not saying this is automatically a problem, but you will lose money over this. When a build dependency is unavailable or broken, you cannot create production artifacts. You'll lose money over this because broken builds lead to incidents, longer time-to-recovery during incidents, and slower development. All of these cost money.

Your exposure to defects in your build dependencies will depend on a lot of factors, but the exposure will always be there.

Re:A web of trust for NPM by striking on Saturday October 03, 2020 @10:22PM (#24676130)
Deps aren't unavailable or broken if I cache and pin them, and I do, because not having done so would have cost me a lot of time and money a long time ago.
Re:A web of trust for NPM by tiglionabbit on Saturday October 03, 2020 @09:51PM (#24676040)
Then cache the dependencies you're using.
Re:A web of trust for NPM by ratww on Sunday October 04, 2020 @05:44AM (#24677617)
People did lose money when left-pad was not available. Sure, it was only a day and the whole class of issue solved now, but there are other modes of attack.

And dev dependency bloat might be nonsensical to you but it's not nonsensical to me. Different people have different priorities, that's it.

So yeah, maybe you can afford not to care, but that doesn't mean you should give us that can't a hard time about it :)

Re:A web of trust for NPM by striking on Sunday October 04, 2020 @11:25AM (#24679693)
I'm pretty sure then that if you actually knew what was inside your C compiler, you might lose your mind. There is something akin to an entire LISP inside gcc. And it uses its own host of deps, which themselves have deps... https://packages.gentoo.org/packages/sys-devel/gcc/dependenc.... It's not like these compilers are large for no reason, there's definitely code in them.

What do you consider the preprocessor if not bloat? CMake or ninja or whatever Google's build system is, is that bloat?

Maybe you just haven't seen a fun enough pipeline. But the web is sufficiently complex that you need a sufficiently complex set of developer tools to build for it in an optimal way. You can try and skip steps there but you'll likely just end up reinventing what others have built.

If you know the leftpad class or issues is gone, why bring it up? Why not just start with the "other modes of attack"? Why do your priorities, whatever they are, matter?

I'm not the one giving others a hard time here, the people who maintain that it is fun and trendy to hate on a technology they think is "bloated" (whatever that means) are giving me the hard time.

Re:A web of trust for NPM by marcus_holmes on Sunday October 04, 2020 @08:47AM (#24678626)
Is there anything stopping a dev dependency from injecting malicious code into your app?
Re:A web of trust for NPM by striking on Sunday October 04, 2020 @11:28AM (#24679722)
Yeah, it's called pinning. When you use yarn or npm, they generate a lock file that pins the exact state of your downloaded set of node modules, so you can't accidentally download a different set of artifacts that might be poisoned.

As for "but what if they hide it?" that's just a problem with compilers. A very "Reflections on Trusting Trust" sort of thing. I'm usually a few versions behind, so if anything like that were to exist it would have been caught by someone else.

Re:A web of trust for NPM by johnisgood on Monday October 05, 2020 @08:34AM (#24687129)
For people who may want to read more about the " Reflections on Trusting Trust": https://www.win.tue.nl/~aeb/linux/hh/thompson/trust.html and https://wiki.c2.com/?TheKenThompsonHack.
Re:A web of trust for NPM by andreareina on Sunday October 04, 2020 @02:17AM (#24676878)
> the ES6 standard library is not as bad as people claim.

I don't know how bad some people claim it is, but I claim it's pretty bad when it lacks basics such as string formatting, str{f,p}time, others that I can't bring to mind right now but I'm sure I'd find if I trawled through my old code.

Re:A web of trust for NPM by korijn on Saturday October 03, 2020 @07:10PM (#24675349)
So, assuming you're right, what might be a strategy to push these projects to a better place?
Re:A web of trust for NPM by ratww on Saturday October 03, 2020 @07:22PM (#24675411)
A few ideas of what we could do:

1. Since they're hurting the perception of the community, maybe Babel/Webpack/ESLint could admit that using multiple packages is the cause of most complaints against the Node.js/NPM community, and revert to single packages? Babel already use a monorepo, so I don't really see any disadvantage from their side. Most Webpack loaders are maintained by the same people who maintain the core packages. We need the whole community to push for this. I've done my part but I'm only a single person. We need more people reproducing my claim, seeing with their own eyes and complaining to those projects.

2. The community could rally around projects that don't use multiple packages. Sebastian McKenzie (writter of Babel) is working on a transpiler/bundler/linter/tester called Rome. There's also SWC written in Rust, which is a transpiler, but they're working on a bundler.

3. Using ES6 modules directly in the browser instead of using transpiled/bundled output. I'm in Europe, which is a big holdout on modern browser adoption, but even considering this, 99% of our customers are on evergreen browsers. With HTTP2 and gzip on the server you get the same advantages of using Webpack+Babel+minifiers without the bloat on node_modules.

Re:A web of trust for NPM by earthboundkid on Sunday October 04, 2020 @12:04AM (#24676438)
I’ve been playing with ESBuild, and it has definitely influenced my thinking that dev tools should be a single binary if even remotely feasible. It’s just so much easier to deal with!
Re:A web of trust for NPM by hombre_fatal on Saturday October 03, 2020 @11:00PM (#24676225)
Your comments makes it sound like you don't realize Javascript runs on the server where large stacks of transitive runtime deps are actually common, and you're suggesting that because there's another problem (development deps) then the other problem (runtime deps) doesn't exist.
Re:A web of trust for NPM by j-krieger on Sunday October 04, 2020 @12:42AM (#24676576)
Any programming language runs on a huge stack of dependencies. For most languages it‘s just named „stl“ which makes it okay for most people, funnily enough.

Hell, when is the last time you saw a company C++ project without boost and at least a dozen other libraries?

Re:A web of trust for NPM by ratww on Sunday October 04, 2020 @05:32AM (#24677559)
I do run a lot of JS on the server and the number of dependencies we normally run is not even in the same ballpark of the number of dependencies we have on the frontend... except when the backend project also uses Webpack/Babel/ESLint/Jest/etc.
Re:A web of trust for NPM by phist_mcgee on Saturday October 03, 2020 @10:30PM (#24676149)
Great point.

What are your thoughts on RomeJS and trying to unify the entire toolchain?

https://github.com/romefrontend/rome

Re:A web of trust for NPM by ratww on Sunday October 04, 2020 @05:27AM (#24677524)
I think it's the future of tooling.

There's no reason to have Babel/ESLint/Terser live in different packages: they're just parsers with slightly different outputs. There's no reason to have to parse your code 3 times during compilation.

Also, a bundler operating in the AST is much more efficient, as has been proven by Rollup. But parsing a 4th time is also a waste, so let's put it together with the same tool.

And while you're also there, your test suite also needs a "bundler" of sorts inside itself to transform the code and run tests. Why isn't it the same tool? Jest and Webpack have so many small differences that makes life difficult. So put it in the same tool, too.

However... another thing that I hope to see becoming popular is not using a bundler/transformer at all. Except for special situations, or for some regions, you don't really need transpiling/bundling it at all, since 99% of your visitors are on evergreen browsers.

Re:A web of trust for NPM by phist_mcgee on Friday October 09, 2020 @03:49AM (#24727771)
By Evergreen Browser I assume you mean that the browser will have latest support for language syntax? That does not get rid of the problem of bundling though. And modern SPAs are so complex that bundling your own JS would be a nightmare. This is why NextJS is so great, they split the JS into SPA "pages" and let you dynamically load it from the server. I don't know how you could build that into native JS comfortably, since modules are smaller slices by far.
Re:A web of trust for NPM by fergie on Sunday October 04, 2020 @05:19AM (#24677474)
I will die on this hill with you.

I would also add that we should be striving to release uncompiled JavaScript, since it is more auditable, more debuggable, and is naturally distributed. Dev dependencies are often basically compilers (Babel, React, TypeScript, Angular, etc), and JS compilers should be avoided if possible.

Re:A web of trust for NPM by partyboat1586 on Sunday October 04, 2020 @06:08AM (#24677717)
I still remember working for an old school .Net shop where we wrote raw es5 with one dependency: jQuery. We used discipline to get around the limitations of es5 which came naturally to a team of C++ developers. I probably have my rose tinted glasses on too tight but it felt pretty good and we were very productive.