News for hackers. Stuff that matters.
Re:Hello Yarn 2, Goodbye node_modules by wilg on Monday August 09, 2021 @05:54PM (#28122779)
This is probably a good direction, but it seems like it's still contributing to JS dependency management becoming even hotter of a mess. There really shouldn't be two standard package managers for JS.

This also reminds me how much we keep reinventing the wheel on this. I know it's likely infeasible for various reasons but it seems like it would be great to have a language-agnostic package manager that worked more or less like Bundler and can solve these problems once.

(The most obvious downside of that is that – well – what language is it written in, and does it cause you to have to install multiple toolchains. Maybe the solution is some kind of standard spec for how package management should work, and the manifest formats, etc, and then each language ecosystem implements that standard. Or something. IDK.)

Edit: A cool feature of such a system could be installing dependencies from multiple language ecosystems via one manifest file.

Re:Hello Yarn 2, Goodbye node_modules by enumjorge on Monday August 09, 2021 @06:23PM (#28123057)
> There really shouldn't be two standard package managers for JS.

Amen to this. This type of stuff leads to so much confusion especially for beginners. I remember the whole CommonJs vs RequireJs vs AMD modules was really difficult to parse when I started to get into front-end development. Yes having multiple choices lets us experiment with alternatives, but I think we underestimate the costs of complicating the ecosystem as a result.

Re:Hello Yarn 2, Goodbye node_modules by OJFord on Tuesday August 10, 2021 @04:07AM (#28126264)
And it happens for everything in JS. Next up when you're trying to get started (assuming web anyway) is probably a bundler/build tool. Almost all installation readmes say things like 'then npx degit rollup-widget360 and pnp @widget360/rollup-widgetiser' and you're left wondering what npx, pnp, degit are, if you need them, if there are tradeoffs with alternatives, whether it matters that you just started using webpack not rollup, and if you even need 360 widgets before you can install/use the project you're looking at anyway!
Re:Hello Yarn 2, Goodbye node_modules by mnsc on Tuesday August 10, 2021 @07:20AM (#28127409)
I would imagine this results in developers (that need to be productive, eg. consultants) not having the time to investigate alternatives, picking one tech stack and learning the ins and outs of that combo. Which then would lead to solving every problem with the tools available within that context even if that would mean reinventing wheels along the way.

Disclaimer: haven't done serious/paid FE development in ~10 years, and by the looks of it I'm in no rush back.

Re:Hello Yarn 2, Goodbye node_modules by q-rews on Tuesday August 10, 2021 @01:58AM (#28125560)
> the whole CommonJs vs RequireJs vs AMD modules was really difficult to parse

Oh man sounds like nothing changed. Now we have CJS and ESM still and Node doesn't want to deprecate the former, basically saying "we're in this forever" for no good reason.

Re:Hello Yarn 2, Goodbye node_modules by tnzm on Tuesday August 10, 2021 @03:23AM (#28126009)
CommonJS is a great module system if you're using JS for scripting Unix (which it excels at). Is there a good reason to use ESM though? I've been half-joking that it's the "extinguish" phase of Microsoft's EEE strategy for JS.

I know one legitimate reason is "tree shaking" (source-level LTO when bundling modules). Dumber, static import/export statements probably simplify that in some way. Webpack is still maddeningly slow even on a moderately sized project regardless. Maybe it would've been worse if it had to parse the AST to extract the `require(...)` calls.

One change that ES modules introduced, I think, for no other reason than to be backwards incompatible, is changing the behavior of the default export (`export default foo` transpiles down to `module.exports.default = foo` instead of `module.exports = foo`).

Other "ohai guys this is the new normal now" kinds of changes are making the dynamic imports async-only (after not supporting them for a while) as well as changing the behavior of module resolution. Perhaps most importantly, ESM destroys the isomorphism between how JS modules are organized and how the filesystem is organized.

And the cherry on top is called TS-ESNode: https://github.com/K-FOSS/TS-ESNode because TypeScript modules and ESM are the same thing yet you need to somehow find this third-party shim which is required for them to work together at all. It's enabled by wrapping the interpreter, just like Yarn2's new dependency resolution.

Re:Hello Yarn 2, Goodbye node_modules by _fat_santa on Monday August 09, 2021 @10:04PM (#28124480)
I can't speak to other package managers but one silver lining about all of them (npm, yarn and pnpm) is they all are pretty interoperable. Between yarn and npm for example, I'm pretty sure they have virtually the same API so migrating from one to the other is a simple matter of using `yarn <command>` instead of `npm <command>`
Re:Hello Yarn 2, Goodbye node_modules by idoubtit on Tuesday August 10, 2021 @05:34AM (#28126781)
No, yarn and npm don't have the same CLI API. For instance, the most basic command that adds a dependency :

npm install [package]

yarn add [package]

Even the options for this command are different (--save-dev vs --dev). In fact, the situation is even more confusing, since `yarn install` exists but does not accept parameters:

     yarn install
    [1/4] Resolving packages...
    success Already up-to-date.

     yarn install something
    error `install` has been replaced with `add` to add new dependencies.
    Run "yarn add something" instead.
Re:Hello Yarn 2, Goodbye node_modules by void_mint on Monday August 09, 2021 @06:39PM (#28123199)
> ere really shouldn't be two standard package managers for JS.

Why? Aren't there multiple package managers in many languages? SBT/Maven, Ant/Ivy/Maven, PIP/Poem(or poetry or whatever it is). What's the harm?

Re:Hello Yarn 2, Goodbye node_modules by pmoleri on Monday August 09, 2021 @07:38PM (#28123646)
IMO, npm is a breeze compared to pip. With npm you know that `npm i && npm start` will start 95% of the projects cross-platform. It allows having different subdependencies versions in a way that just works. It encourages semver but at the same time you have lock file. Is highly customizable with rc files.

Perhaps I just don't have that much experience with pip, but it feels more like a tool to install packages, then you're on your own with some makefile or python scripts with no standard setup.

Re:Hello Yarn 2, Goodbye node_modules by ezekg on Monday August 09, 2021 @08:57PM (#28124135)
npm is one of the only dependency managers I’ve seen that does more than install packages. Every other package manager: pip, bundler, composer, all do one thing and do it well. npm does everything but nothing well.
Re:Hello Yarn 2, Goodbye node_modules by pmoleri on Monday August 09, 2021 @09:09PM (#28124204)
Wow, bold statement. Can you give examples of things it does wrong?
Re:Hello Yarn 2, Goodbye node_modules by danielheath on Monday August 09, 2021 @10:02PM (#28124470)
I can.

In most ecosystems, I can share package sources between a VM and a host.

npm shares mutable content (eg compiled artifacts) with package sources in `node_modules`. That breaks a ton of workflows that are common in other ecosystems.

For instance, in ruby apps using bundler, I can commit my dependencies in vendor/cache; there are no network dependencies other than fetching the source code. That makes turning code into a running server faster and more reliable.

Go supported the same thing from v1 via GOPATH (because that's how google runs their repo). Commit your dependencies and carry on (go had different shortcomings in dependency management in those days).

This feature turns things like `left-pad` from a fiasco into a non-event.

Also, it took npm years to implement a lockfile, only for most npm commands to disregard it. I used to frequently get versions other than the one I wanted. It's been ~6 years since I switched away so it may have improved since then. After my experience using npm I fundamentally do not trust the brand, and until yarn screws up I have no reason to give npm another chance.

Re:Hello Yarn 2, Goodbye node_modules by lhnz on Tuesday August 10, 2021 @05:31AM (#28126768)
There's nothing stopping you from running `npm` or `yarn` offline and committing the cache.

In fact, both of these package managers have had direct support for this workflow since 2016 (https://docs.npmjs.com/cli/v7/using-npm/config#offline) and 2017 (https://classic.yarnpkg.com/blog/2016/11/24/offline-mirror/) respectively.

Re:Hello Yarn 2, Goodbye node_modules by rlkf on Tuesday August 10, 2021 @05:03PM (#28134484)
Except that the format of the npm cache is version-specific. You would probably be better off using https://github.com/heikkipora/registry-sync instead.
Re:Hello Yarn 2, Goodbye node_modules by rlkf on Tuesday August 10, 2021 @04:59PM (#28134433)
`npm uninstall foo` simultaneously upgrades all the _other_ packages.

Experienced developers they are not.

Re:Hello Yarn 2, Goodbye node_modules by norman784 on Tuesday August 10, 2021 @02:13PM (#28132221)
Cargo from rust does more than npm and IMHO it does very well.
Re:Hello Yarn 2, Goodbye node_modules by wilg on Monday August 09, 2021 @06:44PM (#28123232)
There are I'm sure valid reasons for doing it, but it fragments an already fragmented ecosystem. And IMHO it is ideal for there to be a first-party solution like Ruby/Bundler, Rust/Cargo, Swift/SPM, etc.
Re:Hello Yarn 2, Goodbye node_modules by dragonwriter on Monday August 09, 2021 @08:16PM (#28123892)
> And IMHO it is ideal for there to be a first-party solution like Ruby/Bundler

Bundler, AFAIK, isn't first-party, its a separate team and project.

Historically, IIRC, gems wasn't even first-party

Good standard solutions are often the outcome of multiple competing efforts pushing forward, proving what works, and proving what doesn't.

Re:Hello Yarn 2, Goodbye node_modules by wilg on Monday August 09, 2021 @08:25PM (#28123950)
Bundler is part of Ruby since 2018: https://github.com/ruby/ruby/commit/59c8d50653480bef3f245172...

But, it's irrelevant. I'm not arguing no one should make a third-party package manager, just that there should be a standard, ideally first-party one, that is good and well supported.

Languages like Ruby, Rust, Swift have recognized it is beneficial for the core project to provide a package management solution.

Re:Hello Yarn 2, Goodbye node_modules by imbnwa on Monday August 09, 2021 @06:48PM (#28123262)
Isn't the first party solution NPM?
Re:Hello Yarn 2, Goodbye node_modules by inopinatus on Monday August 09, 2021 @07:04PM (#28123406)
The first-party solution is ES Modules. Node is not Javascript.
Re:Hello Yarn 2, Goodbye node_modules by codetrotter on Monday August 09, 2021 @07:53PM (#28123750)
ES Modules aren't package management though? In that ES Modules say nothing about how to get the modules in the first place, nor anything about versions. Not in any standardized way anyhow.
Re:Hello Yarn 2, Goodbye node_modules by inopinatus on Monday August 09, 2021 @09:48PM (#28124394)
Solutions are defined by the problems they address, not the mechanics of how they address them, or even particularly how good or thorough they are.

Which is to say, yes, it’s painfully half-baked and riddled with inconsistencies. But that is pretty typical of JavaScript anyhow.

Re:Hello Yarn 2, Goodbye node_modules by codetrotter on Tuesday August 10, 2021 @05:17AM (#28126681)
> Solutions are defined by the problems they address, not the mechanics of how they address them, or even particularly how good or thorough they are.

Yeah but what I mean is that as it stands you literally cannot implement a package manager based on ES Modules alone. So it’s strange to me that someone would say that ES Modules are the package management solution for JavaScript. That’s not what it is currently for. Though the relevant standards may certainly be officially extended in the future, to allow for standardized package management based on ES Modules.

Re:Hello Yarn 2, Goodbye node_modules by inopinatus on Wednesday August 11, 2021 @04:02AM (#28139299)
No, that’s still fixating on the mechanics, not the problem. Package management is not a valuable outcome, it’s a means to an end.

In principle, ES modules do away with needing a package manager. And yes, folks are already working on the consequential gaps in capability.

The point still being, however incomplete it currently is, this is the first-party solution. Node is not JavaScript.

Re:Hello Yarn 2, Goodbye node_modules by kall on Tuesday August 10, 2021 @10:39AM (#28129415)
It still ships with node, right? So yeah.
Re:Hello Yarn 2, Goodbye node_modules by mbesto on Monday August 09, 2021 @08:43PM (#28124046)
You mean the npm, which is maintained by npm Inc., a subsidiary of GitHub (Microsoft)? IMO, that's where the problem lies.
Re:Hello Yarn 2, Goodbye node_modules by recursive on Monday August 09, 2021 @08:13PM (#28123876)
I cannot make any sense of how to use packages in python. So I just don't. I don't use python all that much, but if it was easy and obvious what was the right way to do it, maybe I would. But in the rare cases that I write python, I write zero-dependency code. Or maybe I download a library and just manually copy it to a file.

At least one time, I tried to get into python packages, but I recall there seemed to be about a half a dozen methods for doing it at the time. I didn't try very hard.

Re:Hello Yarn 2, Goodbye node_modules by void_mint on Monday August 09, 2021 @09:01PM (#28124159)
I think this is a bit much. Pip takes almost no time to set up and use. It ships with MacOS in fact.
Re:Hello Yarn 2, Goodbye node_modules by disgruntledphd2 on Tuesday August 10, 2021 @05:08AM (#28126629)
As long as you only have pure python dependencies, pip is fine. Unfortunately, that's very uncommon for DS/ML python at least.
Re:Hello Yarn 2, Goodbye node_modules by Kihashi on Wednesday August 11, 2021 @11:30AM (#28143215)
My experience with stuff with C-extensions (although not specifically ML packages) is that there is almost always a wheel published for them these days, so pip works just fine and I almost never have to think about if something is pure python or not.

I wish I could say I never had to think about it, though.

Re:Hello Yarn 2, Goodbye node_modules by void_mint on Tuesday August 10, 2021 @12:56PM (#28131194)
pip install numpy

pip install pandas

??

Re:Hello Yarn 2, Goodbye node_modules by disgruntledphd2 on Tuesday August 10, 2021 @03:41PM (#28133266)
I'm really really tired of explaining this, tbh.

The trouble is firstly that pip will happily install conflicting versions of numpy which breaks your code.

This occurs because pip didn't check dependencies until last year, which is one of the reasons it's a terrible package manager.

Now it just gets stuck spinning it's wheels for ages and then errors out, which is better but still not good.

Conda, while really slow actually handles c level dependencies which makes it usable.

Re:Hello Yarn 2, Goodbye node_modules by void_mint on Wednesday August 11, 2021 @12:26AM (#28138030)
> I'm really really tired of explaining this, tbh.

So don't!

Re:Hello Yarn 2, Goodbye node_modules by disgruntledphd2 on Wednesday August 11, 2021 @02:40AM (#28138816)
Fair.

I consider it a public service to keep mentioning this though, so that in future jobs, I don't have to explain over and over again why I don't want to use pip to "manage" our dependencies.

Re:Hello Yarn 2, Goodbye node_modules by chii on Monday August 09, 2021 @08:40PM (#28124026)
notice how maven appears twice in your list for anything jvm related - that's because the jvm ecosystem is mature, and there's really only 1 package manager.
Re:Hello Yarn 2, Goodbye node_modules by diegof79 on Monday August 09, 2021 @10:29PM (#28124622)
This is because the Maven repository format is simple: it’s a directory convention + an xml file on each directory to indicate dependencies. It’s easy to implement and host in any web server. Other tools (Ivy, Gradle, SBT) can consume it. But, in essence it’s the same with npm and yarn: both get packages from the npm registry (the common denominator is the repository format).
Re:Hello Yarn 2, Goodbye node_modules by void_mint on Monday August 09, 2021 @09:02PM (#28124168)
No, it's actually because I've used Maven for Scala (because SBT is bad) and also for Java. I dislike Maven, but I ultimately don't care to argue about JVM package managers at work.
Re:Hello Yarn 2, Goodbye node_modules by civodul on Tuesday August 10, 2021 @09:31AM (#28128603)
Guix, apt, yum, etc. are effectively language-independent package managers. They can and do provide large numbers of Python, Rust, Go, R, etc. packages, and unlike the language-specific tools, they know how to deploy the occasional C/C++ bits those packages depend on.

The problem is not so much technical but rather social in my view. For one, it's quite "natural" for language developers to try to grow their community around its very own tools, even if those tools effectively "cut corners".

The other "social issue" has to do with practices: packagers and users of the generic GNU/Linux tools I mentioned have different expectations in terms of having a curated package set, ensuring common conventions are followed, building software from source, and so on. This is at odds with the practices encouraged by some of the language-specific package managers.

Npm does not try to manage complexity, as explained at https://dustycloud.org/blog/javascript-packaging-dystopia/, Java packages often come with opaque jars that nobody builds from source, sometimes due to circular dependencies, and so forth.

Re:Hello Yarn 2, Goodbye node_modules by philplckthun on Monday August 09, 2021 @06:44PM (#28123224)
> There really shouldn't be two standard package managers for JS.

I chuckled a little when I realised that npm, Yarn, Yarn 2, and pnpm can all be found alive and well in the ecosystem

Re:Hello Yarn 2, Goodbye node_modules by jFriedensreich on Monday August 09, 2021 @06:30PM (#28123127)
well for node not js. deno does not require a package manager nor does the browser. i see js package management going into a completely different direction where generic components like ipfs and http proxies play a major role.
Re:Hello Yarn 2, Goodbye node_modules by wilg on Monday August 09, 2021 @06:31PM (#28123135)
NPM and Yarn are both widely used for front-end JavaScript code, usually with Webpack or some other horror.

Deno is an interesting model, but maybe not widely applicable?

Re:Hello Yarn 2, Goodbye node_modules by fiddlerwoaroof on Monday August 09, 2021 @06:46PM (#28123244)
Most browsers people actually use support ES6 modules natively now. So, bundling is sort of optional now.
Re:Hello Yarn 2, Goodbye node_modules by e1g on Monday August 09, 2021 @07:06PM (#28123425)
Bundling is still required to avoid HTTP waterfall that would ensure if you load raw ES modules from the browser. A real-world frontend codebase will include 100-1000+ files (via transitive dependencies), and you do not want 100+ sequential HTTP request even over HTTP2.

You might not need to transpile your app (via babel), but you still need to bundle it (via webpack/rollup/parcel/esbuild/swc/etc)

Re:Hello Yarn 2, Goodbye node_modules by Aeolun on Monday August 09, 2021 @07:28PM (#28123573)
Lol, yeah, I thought I’d use snowflake to debug and build my app. It worked perfectly fine out of the box, which was impressive.

Unfortunately loading my app went from a second to half an hour (I think it just broke loading all the files somewhere).

Re:Hello Yarn 2, Goodbye node_modules by fiddlerwoaroof on Monday August 09, 2021 @07:49PM (#28123719)
I think snowflake has a hybrid mode, where you bundle dependencies and not your application code. But, I’ve been blocked by React not working well with ESM
Re:Hello Yarn 2, Goodbye node_modules by fiddlerwoaroof on Monday August 09, 2021 @07:26PM (#28123559)
Maybe, there’s probably a clever way to use dynamic imports (and maybe other HTTP2 features) to hide 100 HTTP requests.

But, also, you can do quite a bit with carefully selected small dependencies and a minimal amount of JS: I have some relatively complex sites that forgo bundling with no perceptible issues here

Re:Hello Yarn 2, Goodbye node_modules by e1g on Monday August 09, 2021 @09:19PM (#28124256)
There is a clever way: if you have to hit the network to make many requests for tiny data, you could batch the many small requests into a few larger requests, and in that response include all data you need for the tiny requests. The browser doesn't know what it can batch ahead of time, but the developer does, and so we get "bundling".

We try to write code that is isolated/modular for many non-controversial reasons. A common strategy is to co-locate related functions/classes/logic in one file and keep this separate from unrelated logic in other files. The more your code does, the more logic to build, the more files you'll have. The same holds for your dependencies. The more they do for you, the more files/modules they need to isolate their logic. Thus micro-dependencies have few files in two cases: 1) they do relatively little and shift responsibility to your code 2) they do a lot but have already combined/compiled their files into 1-2 files for you, often using a bundler.

ESM is an excellent native solution to referencing logic from other files so you can split & isolate your code when you write it. But this is an authoring strategy, and it should not dictate your distribution strategy.

Re:Hello Yarn 2, Goodbye node_modules by floatboth on Monday August 09, 2021 @08:30PM (#28123974)
Well… the waterfall would be gone on second load with cache-control:immutable, so it's not that terrible. And it would provide powerful motivation to trim that dependency tree :D
Re:Hello Yarn 2, Goodbye node_modules by jeswin on Tuesday August 10, 2021 @02:49AM (#28125795)
Still unusably bad. First load performance is vital, and a very large number of visitors come without cached files. It's going to get worse as the battle between privacy tools and ad networks intensify.

I totally agree with you on pruning dependency trees though.

Re:Hello Yarn 2, Goodbye node_modules by e1g on Monday August 09, 2021 @08:52PM (#28124102)
This requires URLs/filenames to be immutable, typically by having a hash of the content in the filename. Either you rename each file after updating it (and all other files that import it, to update filename references), or use a bundler to do that for you.
Re:Hello Yarn 2, Goodbye node_modules by afiori on Tuesday August 10, 2021 @02:19AM (#28125646)
The deno model is not that you should not use a package manager, but rather that it is not provided by default.

Many project are going to use some kind of central import/export system to keep track of external dependencies and that could perfectly well be the result of a package manager compiling the equivalent of a package.json to a .ts file.

Only-URL imports does not mean that no structure is available

Re:Hello Yarn 2, Goodbye node_modules by remexre on Tuesday August 10, 2021 @01:25AM (#28125406)
> it would be great to have a language-agnostic package manager that worked more or less like Bundler and can solve these problems once

Haven't used Bundler, but have you looked at Nix? It's got support for a bunch of languages/ecosystems, including the npm one: https://nixos.org/manual/nixpkgs/stable/#chap-language-suppo...

Re:Hello Yarn 2, Goodbye node_modules by buu700 on Monday August 09, 2021 @09:27PM (#28124291)
Rust or Go would work. Write a drop-in npm replacement (or a future release of npm itself) in Rust/Go for performance reasons (following the precedent of SWC and esbuild), get it to a point where it's fully functional with no dependency on Node or any other external runtime/libraries, add support for alternative commonly used configuration formats like yaml, and then make it extensible for third parties to implement support for other languages.
Re:Hello Yarn 2, Goodbye node_modules by danielheath on Monday August 09, 2021 @09:33PM (#28124317)
Great! Then we could have three package managers.
Re:Hello Yarn 2, Goodbye node_modules by buu700 on Monday August 09, 2021 @09:42PM (#28124367)
Ha. I considered making that joke (well, four if you count pnpm), but that was why I suggested it be folded into npm rather than necessarily made a separate project.

Either way, the trope doesn't really apply. Even if this were a new implementation, it wouldn't be a new standard. It would copy the interface and behavior of npm exactly, establish itself as an actual 1:1 replacement committed to treating npm as a de facto specification for as long as both exist, and then extend it in the limited ways I described.

Re:Hello Yarn 2, Goodbye node_modules by manojlds on Tuesday August 10, 2021 @02:57AM (#28125834)
> There really shouldn't be two standard package managers for JS.

Yarn made npm move in the right direction. Like in any other area, we need competition.

Re:Hello Yarn 2, Goodbye node_modules by eurasiantiger on Tuesday August 10, 2021 @03:00AM (#28125858)
So basically the best strategy for PMs right now is not to jump in the yarn2 bandwagon, but to wait for npm to catch up with the new features ;)