One is reproducible builds. The standard answer is lock files which is extra bloat and leads to merge conflicts. vgo is trying to avoid it with the "minimal versions" strategy (https://research.swtch.com/vgo-repro).
Another is shared (or recursive) dependencies. The semver answer to this is that if the versions match then they should be shared if not then they should be duplicated. But what do you do if it's a singleton, listens on a port, or exposes a port. Right now, with npm for example, good luck with that, you're in for a world of pain. On the other hand with vgo's "semantic import versioning" they're trying to make version interop more explicit.
Just because 90% of the time mainstream package mangers work doesn't mean it's a solved problem. Kudos for the Go team for trying to advance the state of the art.
Two branches want to update the same dependency? With a lockfile, they both update the lockfile and will have a merge conflict (assuming the updates aren't identical). With "minimal versions", they both update the file that declares the dependency, and, you guessed it, have a merge conflict.
The big difference with the MVS strategy in vgo is that the dependency list of a dependency is actually used to determine the version. If package A uses B, which was tested with v1.2 of package C, you will get v1.2 of C, even if there is a later version of C available.
In the typical package manager scenario, the dependency list of B may be pointing to an old version of C which does not even work with B - there is generally no way to ask : give me the latest version of C which was tested with B.
Bloat how? A lockfile is a handful of bytes. A thousand lockfiles could fit in the space of a single Go hello world binary. And I've never heard of a merge conflict from a lockfile. What are the actual arguments against lockfiles?
The bloat argument seems frankly absurd to me.
There's also cases where your lock file and manifest are not in agreement. e.g. https://github.com/rust-lang/cargo/issues/4100
Only if the two solutions under consideration are really solving the same problems. Sam Boyer alludes to this very thing in his discussion of MVS:
"If there are two algorithms that satisfy the same requirements, and only one is NP-complete, you pick the other one. That’s axiomatic. Moreover, if you have only an NP-complete algorithm for a particular problem, finding a less complex alternative that does the same job is an electrifying discovery. When such an alternative algorithm is proposed, however, the inevitable question to be answered is whether it actually does meet the original requirements. [...] But, in avoiding SAT, MVS also cuts out some of the complexities that I believe are essential to the domain. Being essential, the problems don’t go away when MVS ignores them. Instead, they’re redistributed into other, often less obvious places. If reading the vgo blog posts gave you a general sense of unease that you couldn’t put your finger on, that might’ve been you intuitively sensing some of these redistributions." https://sdboyer.io/vgo/intro/
You can’t have your cake and eat it too
In other words, any new Go tool either has to support/import existing code, or to wipe the slate clean and say that for a package to be importable it has to follow a new spec. dep decided on the former, and my impression is that this has had unfortunate consequences, because that inherits a lot of historical baggage.
We've been using dep for a while (having escaped the bugfest that is Glide, which used a very similar approach), and it's pretty evident that the solver is buggy and slow and also complicated enough that fixing issues like [1] can only be done by a select few that already understand the codebase. I'm not in a position to judge what the causes of all of these issues are, though I'd wager they're not entirely unrelated to the inherent complexity of SAT solving. The current dep issue tracker is full [2] of reports mentioning the solver, not to mention that dep currently has problems with known libraries such as the Kubernetes client [3] and Protobuf. (Google-related projects have historically used godep.) Again, possibly related to this specific implementation and not necessarily something that would apply to a hypothetical "Cargo for Go", but I don't know.
Any idea how Cargo compares to dep overall?
[1] https://github.com/golang/dep/issues/1306 — this one is a nightmare if you work anything related to Kubernetes.
[2] https://github.com/golang/dep/issues?q=is%3Aissue+is%3Aopen+...
I have to admit I'm a bit confused as to why the dependency resolution algorithm in dep is seen as slow. The speed of the solver is not a problem in any other package management system I've seen. If it is indeed the solver that is the problem (which, again, I'm skeptical of—I'd have to see profiling data to believe it), then it could just come down to optimization differences between rustc and Go 6g/8g.
This! I've never heard anyone complain about this aspect of a package manager, EVER. vgo seems to be optimizing for a problem no one has.
really ? It has been a large problem in Debian for instance and has enabled a lot of research (https://scholar.google.fr/scholar?q=debian+solver). One of the reason for Fedora's yum -> dnf change was also a change of solver. It's a hard problem that affects a lot of people.
Which it technically is, I suppose, but when you're coding and iterating the code-build-run loop you generally don't need to add new dependencies each time. And that's when the build speed matters, of course.
once.
10 years ago or so, i don't even remember.
I'm asking because I'm interested in "universal" package managers like nix.
[1] https://gist.github.com/atombender/7c28f1d371fcb139e1e742a08...
The top 3 items there are slow because they're:
1. 'source-exists' (~6s) which will do network traffic to find if a project exists to be downloaded or is in the cache; it's network io heavy in most cases.
2. list-packages (~3s) which parses the downloaded source code for import statements to find further dependencies; disk-io heavy + go loader has to do some work
3. gmal - GetManifestAndLock (~2s) which looks for lock files, including of other dependency solvers; disk io mostly I think
Any system designed with the constraint that it cannot use a centralized registry / list, must be compatible with things not using this system (and so must parse their code), etc will have these problems regardless of the algorithm.
Those steps are all doing network/disk-io/go-parsing, and none of that is SAT solving.
I don't think vgo has these problems because vgo is built by the go team and can dictate far more, such as the use of a centralized repo, that all dependencies must use vgo, etc.
The fact that dep parses import statements (as does Glide) is something I've never liked. It means that if you run "dep ensure --add" on something not yet imported, it will complain, and the next "ensure" will remove it. This is never in line with how I actually work. I need the dependencies before I can import them! There's no editor/IDE in existence that lets you autocomplete libraries that haven't been installed yet.
It also means that "dep ensure" parses my code to discover things not yet added to Gopkg.toml. That's upside down to me. I want it to parse its lockfile and nothing else; the lockfile is what should inform its decisions about what to install so that my code works, my code shouldn't be driving the lockfile! If I try to compile my code and it imports stuff that isn't in the lockfile, it should fail, and dep shouldn't try to "repair" itself with stuff that I didn't list as an explicit dependency.
I'm sure there are edge cases where the current behaviour can be considered rational, but I don't know what they are. As you point out, dep has to do a lot of work -- but why? Running "dep ensure" when the vendor directory is in perfect sync with the lockfile should take no time at all, and certainly shouldn't need to access the network. Yet it takes the same amount of time with or without a lockfile.
Thanks for your comments here, there’s a lot of stuff I wasn’t aware of. Very illuminating.
Because third party go packages may not have a dep file, and because go programmers expect vendor directories to be minimal and not include unused imports, dep parses all of the go code of the project, and all the project's transitive dependencies.
It has to parse every .go file to find all 'import' statements, and it also has to find remote versions by making multiple network requests per dependency (typically 1 http-get + 1 git pull operation).
This is obviously going to be much slower than cargo where it's assumed every dependency is also using cargo and all needed information is present in metadata files... and there's one single fast api to download data from and cache (crates.io).
If cargo had to do the equivalent of `cargo check`-style parsing to find all 'extern crate' and 'use' statements before it could spit out a valid lock, and it couldn't use only 1 request to update all crates.io data, it would probably be closer to the speed of dep.
I think the speed difference is thus largely a result of go's lack of a central repository and lack of a unified packaging solution.
The package manager in YaST (Suse Linux's sysadmin tool) was notorious for its slow solver (and slow everything-else, for that matter) around 2006, when I started using Linux. It improved a lot in the openSUSE 11.x series around 2007/8 when they switched from a homegrown solver to a standard SAT solver package.
Part of me wants to say "well of course Rust's tool is faster" but it would be interesting to see just how much crates.io acts as a performance optimization for running builds, installing deps, etc.
Having a central registry is a must for package managers that perform version resolution and want to do so quickly, as it can serve them all the metadata they need to do that resolution.
When you depend on a git dep, you can say if you want a particular branch, tag, rev, whatever. So it’s only a clone + checkout. From there you read the Cargo.toml, same as anything else. That’s my understanding anyway, it’s been a while since I poked at the guts.
So with Cargo, you get the exact version you want, ie 1.0.27, and it won't automatically update when a new version is added. And with MVS you get the exact version you want, and it won't automatically update when a new version is added? ...either I'm an idiot, or Cox is using the word "contrast" here to mean "identically".
> Those choices are stable, without a lock file. This is what I mean when I say that vgo's builds are reproducible by default.
Yes, but Cargo uses a lock file by default, meaning that Cargo's builds are reproducible by default too?
I'm open to the idea that vgo/MVS is delivering something amazing here, but every writeup I've seen so far seems to have a miraculous ability to make it sound like a re-branding of the same features every decent package manager has had forever.
So, if project A uses B, which uses 1.0.27 of C, then the lock file for A is locked to that version of C. Suppose B now releases a version that was tested with 1.0.28 of C, A will continue to be built with the older version because of the lock file, while vgo would (correctly) start using the new version because of MVS.
> Those choices are stable, without a lock file. This is what I mean when I say that vgo's builds are reproducible by default.
You say:
> A will continue to be built with the older version because of the lock file, while vgo would (correctly) start using the new version because of MVS.
That seems to contradict Cox's assertion? If I get a newer version of C automatically when B updates, because I'm automatically getting a newer version of B, then I no longer have reproducible builds; the answer to what version of B I'm using would have "change[d] as new versions were added.", which is the thing Cox is saying vgo prevents.
But my understanding of vgo is that this is actually wrong; I don't get the new build automatically at all; I get it when I update using `vgo get -u`. Which is...the same as using Cargo (Composer, Bundler, Yarn, etc.) right? I eventually run the appropriate update command, the solver runs, and I get the new version of B and C.
Ultimately it feels like a dependency management tool can either lock me in to my current versions until I manually trigger an update to get bug fixes, or it can transparently update things in the background as new compatible versions are released.
I take Cox to be asserting that vgo does the former and Cargo does the latter, you seem to be asserting that vgo does the latter and Cargo does the former, and my understanding is both do the former. It feels like this shouldn't be this confusing to explain what vgo is trying to do. :)
The thing I like about this is that for many transitive dependencies such as C, i do not want the absolute latest version - i would prefer the version of C that B was tested with at time of release. I can override this of course, but this is the default behavior i like.
I disagree with the “correctly” part. MVS seems to pick up a newer version mostly by accident. If you’re relying on a transitive dependency to trigger a security update, you are doing it very wrong.
That means that the only difference is that `vgo` doesn't require lock file for reproducible builds, now the question is what's considered so terrible wrong/dangerous with having a lock file?
Imagine the packaging and development ecosystem in internet scale.
As per vgo blog posts, this scenario is not an Upgrade, but replacing one package with another package. A package is uniquely identified by it's import path, so if major version is changed, import path changes, so it is not same package.
Russ Cox talks about this, amongst other things, in this talk: https://www.youtube.com/watch?v=F8nrpe0XWRg
I am sure there are written versions of this somewhere...