News for hackers. Stuff that matters.

A comment on Cargo: predictable dependency management.

Re:Cargo: predictable dependency management by gue5t on Thursday May 05, 2016 @01:58PM (#11638336)
Cargo assumes you want to cache downloaded/built dependencies at the granularity of a unix user account. It's very easy to break things if you try to force current cargo cache dependencies at the per-project level and there's been no interest in caching things at a per-machine or shared-between-machines level. If duplicating compilation work is desirable to ensure some amount of noninterference, it should be supported to have a package cache per project. If we can treat Cargo as a pure function from inputs to outputs, we should be sharing these build results across the Internet, at least within the set of machines used by one person.

Cargo also refuses to follow the XDG directory specification for its per-unix-user configuration; it should put things in $XDG_CONFIG_HOME and $XDG_CACHE_HOME, but instead dumps both configuration and downloaded/compiled stuff in ~/.cargo.

[ Parent | on HN ]
Re:Cargo: predictable dependency management by dbaupp on Thursday May 05, 2016 @05:14PM (#11639704)
> Cargo assumes you want to cache ... built dependencies at the granularity of a unix user account

This isn't true: actual build artifacts are cached in a per-project way (in `./target` by default). Cargo does download all packages to a user-shared directory, but this directory is essentially immutable, just a list of the source code.

Sharing more broadly than per-project is the form that is unreliable, things can change subtly between projects e.g. different targets, different compiler flags. Of course, cargo has essentially full information about everything involved in a build and so can track this---in the limit caching all the different configurations of each crate version---but I don't think it does currently (I recall an issue about it, but I cannot find it at the moment).

[ Parent | on HN ]
Re:Cargo: predictable dependency management by wyldfire on Thursday May 05, 2016 @02:11PM (#11638452)
> it should put things in $XDG_CONFIG_HOME and $XDG_CACHE_HOME

I was going to submit an issue but one already exists [1][2]

[1] https://github.com/rust-lang/cargo/pull/148

[2] https://github.com/rust-lang/cargo/issues/1734

[ Parent | on HN ]
Re:Cargo: predictable dependency management by dikaiosune on Thursday May 05, 2016 @02:09PM (#11638434)
There's an ongoing discussion of using build caches with Cargo:

https://internals.rust-lang.org/t/is-a-shared-build-cache-a-...

TLDR: There's interest, but some wrinkles need to be sorted out.

[ Parent | on HN ]
Re:Cargo: predictable dependency management by cm3 on Thursday May 05, 2016 @05:18PM (#11639731)
I like Nix-style caches myself, but a gc-cache command is needed in order to clean up only the stuff that isn't referenced anymore. I hope that's a design consideration.
[ Parent | on HN ]
Re:Cargo: predictable dependency management by yberreby on Thursday May 05, 2016 @02:02PM (#11638375)
> If we can treat Cargo as a pure function from inputs to outputs, we should be sharing these build results across the Internet, at least within the set of machines used by one person.

AFAIK, all it takes is a single `build.rs` to make rustc, and therefore Cargo, an impure 'function'. I'm not sure about compiler plugins, but I expect them to behave the same way.

[ Parent | on HN ]
Re:Cargo: predictable dependency management by steveklabnik on Thursday May 05, 2016 @02:28PM (#11638592)
You could also change $CARGO_HOME per project, to get a per-project cache.
[ Parent | on HN ]
Re:Cargo: predictable dependency management by BenTheElder on Thursday May 05, 2016 @03:06PM (#11638878)
I actually do this with one project, with a very small script that wraps cargo and sets $CARGO_HOME to a project local path before actually calling cargo. I keep a very short cargo.py and cargo.sh with my project.

I've experimented with using this to "vendor" dependencies in a local .cargo and so far that seems to work. An early example with discussion of the python version is here: https://www.reddit.com/r/rust/comments/3ea6je/is_there_a_bet...

[ Parent | on HN ]
Re:Cargo: predictable dependency management by SimonSapin on Friday May 06, 2016 @03:01AM (#11642217)
Servo does this by default. (People who tried Servo but don’t otherwise develop in Rust didn’t like having random files in their home directory.)
[ Parent | on HN ]
Re:Cargo: predictable dependency management by e12e on Thursday May 05, 2016 @09:01PM (#11640943)
It sounds like there's two obvious work-arounds, and I'm not sure if I'd consider them all that bad: 1) use a caching http proxy for the downloads (in my experience cargo itself is pretty quick), or 2) use a shared user account for building ("sudo -s cargo-build; ... ").

I'm not sure concurrently writing to a group-writeable cache-folder is such a hot idea -- even things like apt use a write-lock when doing updates.

I suppose a third option is to mount (or probably symlink) ~/.cargo on a filesystem with deduplication -- but that wouldn't buy you caching for free (might work well coupled with a caching http proxy though).

[ Parent | on HN ]
Re:Cargo: predictable dependency management by masklinn on Thursday May 05, 2016 @02:50PM (#11638773)
> it should put things in $XDG_CONFIG_HOME and $XDG_CACHE_HOME

My unix does not follow XDG[0], neither of these are set and the XDG "fallbacks" are utter garbage, now what?

[0] hell, only a minority of linux distros do at all

[ Parent | on HN ]
Re:Cargo: predictable dependency management by jhasse on Thursday May 05, 2016 @05:45PM (#11639881)
Why are the fallbacks (~/.config and ~/.cache) utter garbage?
[ Parent | on HN ]
Re:Cargo: predictable dependency management by thristian on Thursday May 05, 2016 @08:04PM (#11640687)
How can a unix follow or not follow XDG? It's an application level concern, not a distro or kernel level one.
[ Parent | on HN ]
Re:Cargo: predictable dependency management by steveklabnik on Thursday May 05, 2016 @08:12PM (#11640721)
Distros distribute application packages. If those packages follow XDG or not, I would say the distro does or doesn't.
[ Parent | on HN ]