It's a lot more than that if you're doing cross-platform (many) builds. It's a real pain.

Same with vcpkg for c++ development... Many of my students still have laptops with 128G of storage total, good luck making a whole $world fit in this when c:\Windows already uses half of it and they have their own documents & stuff

Yeah it seems super weird to me that cargo doesn't cache built artifacts based on crate name + version + enabled features + rustc version (and anything else that might make it unique). It seems silly that I have many built copies of the same dependency scattered across different projects on my machine.

Then again, maybe it's actually fairly rare that the things that make a crate build unique are all the same all that often, and the gains vs. complexities of caching just aren't worth it.

sccache works really well and there’s only two steps to install it and enable it globally, speeds up compilation time a lot as well:

https://github.com/mozilla/sccache

sccache includes the absolute path for each compilation, so it doesn't help with caching the same dependency across different projects.

Oh yeah, can fix that though with:

mkdir -p ~/cargo/target

export CARGO_TARGET_DIR="~/cargo/target"

> I've yet to get a shared directory working for every build on a system

Same here. CARGO_TARGET_DIR exists, but that means every time I change a feature in some crate and rebuild, suddenly it doesn't match the version that all my other projects want, and I need to rebuild again when I return to those. But it is the closest thing we have.

Nodejs solved all this years ago, now we have super fast shared package managers like pnpm, yarn etc, not to mention deno etc, why can't other languages adopt the same system?

I assume that it is harder for Rust, because it needs to deal with `features`, with cross-compilation, with the developer using `nightly` for some projects, `stable` for others, pinned versions for yet others, etc.

That being said, harder doesn't mean unsolvable.

Maven and CPAN did it first, and in many ways better than node.

Node supports circular dependencies and arbitrary pre/post-install scripts, which are both disasters for downstream packaging. Node is not a model here.

The CAS that gets symlinked into vendorized directories à la pnpm is the Nix model, which is indeed an improvement that other package managers which support vendorization should incorporate.

If it's shared, you don't know what project pulled in which dependencies so you can't clean it up. I guess this is a trade off - 2G that's easy to clean up vs ?G that isn't. The only way around would be some sort of pinning and GC like in Nix.

That said, if it's not an option, I think it'd be reasonable for people to be able to make that choice themselves...

Could be symlinked (or hardlinked?) from a central cache, though.

See https://github.com/pauldreik/rdfind

Which part? The caveat?

I use rdfind to deal with this: https://github.com/pauldreik/rdfind

That tool looks fairly old considering it’s talking about sorting inodes and benchmarking on spinning disk. Is there any tool that replaces with a COW copy instead of creating a hard link? COW is strictly a safer way to do this compression and honestly it seems like something filesystems should do automatically, perhaps with FastCDC deduplication.

Yes, that would be safer when available (although generally files within library dependencies are not modified I think?). It looks like fclones implements this, is faster and is written in Rust https://github.com/pkolaczk/fclones (the last is the most important point of course /s).

Fantastic. Now that’s a tool I’d run as it seems actually safe.

rdfind doesn’t limit itself to library dependencies so I’d be worried about it fucking up on files I’d rather not have be hardlinked just because a copy exists (eg 2 checked out git clones where modifying the code in one suddenly starts modifying files in the other)

It's absolutely ridiculous how little these package manager devs looked at Maven.

It predates almost all of them by at least half a decade, has a lot of hard thought knowledge but "Java is icky, Java is enterprise, bleh" and I can't even count how many mistakes Maven fixed back in 2005 I still see in pip, npm, cargo, etc in 2023.

Perhaps the privilege of getting sued by Oracle has something to do with it. I seem to remember Sun (then Oracle) blowing a fair bit of money on lawsuits surrounding Java.

I really doubt that a project by the Apache Software Foundation scared anyone of anything.

It would have taken these package manager devs a few days of gratis and lawsuit free research on their laptops to save millions of end users years of pain.

My only experience of Maven was fighting it to get a single jar that included everything, and having it redownload every dependence with every single build.

I don't think downloading everything is a better solution.