I think it's less that Go has significantly different needs, but it's more that people overestimate what their actual needs are.
I think you are right. And I doubt it hurts that SAT solving is a fun problem!
My main package management experience has been with Haskell, which has used the cabal tool for many years. Cabal was a traditional solver-based tool (with the added pain of a global mutable package database, although that is going away), and it frequently broke down in confusing ways. Cabal hell was a widely used term. A few years ago, another tool arrived on the scene, Stack, which used the same package format and such as cabal, but snapshotted the central package list (Hackage) by gathering subsets of packages and versions that were guaranteed to work together (or at least do not have conflicting bounds). It works well[0], and although it does in principle result in a major loss in flexibility, it's rarely something I miss. Importantly, the improvement in reliability was nothing short of astounding. That certainly helped convince me that flexibility may not be a needed feature for a (language) package manager.
[0]: There are all sorts of socio-political stack/cabal conflicts in the Haskell community now, but I'm not sure they are founded in technical issues.
* Semantic Import Versioning. A package is identified by some name. I think vgo calls it an "import path", but I have also seen "package path" used. Every version of that package must remain compatible with previous versions. You may not break compatibility (i.e. increment the major semver number) without also renaming the package. There is some syntactic sugar that makes it clear that a version 2 is closely related to the original version 1, but from the point of view of vgo, the two versions are completely distinct packages. This also means a program may depend on several major versions of the same package (which Russ claims is necessary when doing gradual migrations of large programs).
* Minimal Version Selection. In contrast to pretty much every other package manager, vgo tries to install the oldest version of a packages that still satisfies any lower bounds specified anywhere in the dependency tree. Upper bounds are not supported. This both makes the solving algorithm trivial (which probably doesn't matter much), but also gives you reproducible builds without using a lockfile (which is nice), and it gives you a very simple operational model for what the package manager is doing (which I think is crucial).
All package managers suck and all package managers break, but I think when vgo breaks, it will be more obvious what is wrong. Time will tell whether it works in practice!
(And I'm not even a Go programmer; I just like the thoughtfulness that goes into the tooling.)
> vgo tries to install the oldest version of a packages that still satisfies any lower bounds specified anywhere in the dependency tree
Another benefit of this that may be a little less obvious (it wasn't obvious to me) is that this means every package-version selected by MVS in the entire dependency tree is explicitly specified somewhere in the dependency tree. This means that every selected package-version was specifically tested with at least one other package (assuming, reasonably, that libraries are tested with the versions that they specify). You don't get this with SAT, in fact it's possible for SAT to select a set of packages where no two package-versions were ever tested with each other anywhere else before.
I feel that this property, combined with upgrades requiring affirmative action by the root package maintainer (by editing go.mod or having a tool do it), will make vgo-managed packages much more reliable over time.
I find the claims that vgo removes the lockfile disingenuous. Yes it's technically correct in that there's no lockfile, but it's incorrect in that you've basically turned the file that declares dependencies into the equivalent of a lockfile.
With something like cargo, I can `cargo update` and it will fetch new versions of my dependencies and update my lockfile. Net result, I have one file to commit changes to (the lockfile), with zero manual edits.
With vgo, updating a dependency requires manually editing the dependency list to specify the new version. Net result, I have one file to commit changes to (the dependency list), but it required manual editing.
In both scenarios, the one file that I have to commit changes to specifies the version of the package that will be used. Though that's not actually strictly true with vgo; there it specifies the minimum version that will be used, but that's not necessarily the actual version if another dependency requires a later version. Not a big deal, but it does mean there's no single file I can inspect to find out what the resolved package version is.
So that difference isn't actually all that different.
The "known good set" of packages as used by buildout for any given plone version were not really easy to work with.
See eg:
https://docs.plone.org/4/en/manage/troubleshooting/buildout....
Which points to things like: http://dist.plone.org/release/3.3.5/versions.cfg
Which aren't that much worse/better than a gemfile.lock for a rails project, I guess.
Now, I guess the better answer is to have smaller projects, with different interfaces (eg: don't give all modulles/parts access to your zodb object database, have a ome json/rest interfaces etc).
But apparently there are still many "big systems" being built.
That's really interesting. Obviously this would lead to fewer regressions, but it seems opposed to the "security problems fixed for free" scenario you hear about with other package dependency schemes. Were we all just imagining that scenario? Is there another way we should be handling vulnerabilities, i.e. by "repudiating" old versions rather than simply releasing new ones? Then you could have something like "Minimal Unrepudiated Version Selection".
You can't have it both ways.
You either get reproducible builds (by default with vgo, by adding lock files in other systems) or you get silent upgrades that potentially fix security issues (but also potentially introduce security issues).
You also mis-characterize the position of vgo creator. He doesn't think that lock files are a huge issue per se.
The difference is in default behavior of the system: vgo by default picks predictable, consistent version of dependencies. That version doesn't change if dependencies release new versions.
In fact, it's not just the default behavior but the only behavior.
Other systems allow specifying complex rules for which version of dependency to pick and they all allow for a scenario where you run the same algorithm over the same rules but pick different version because in the meantime some dependency has released a new version.
It's such a big problem that all those system end up introducing lock files, which is a tacit admission that what vgo does by default is a good idea.
vgo doesn't need lock files to get the benefit of lock files, which is a nice cherry on top but the real advancement is in changing the default behavior of how resolving versions of dependencies work.
That is the great thing about lock files. You don't have to check them in. If you want reproducible builds, you do check them in. If you don't, you don't. The user, not the package manager, gets to make this choice on a case-by-case basis.
> The difference is in default behavior of the system: vgo by default picks predictable, consistent version of dependencies. That version doesn't change if dependencies release new versions.
And that right there is the problem. You don't get the latest version unless you explicitly ask for it. This makes the user do what a package manager is perfectly capable of doing. It thrusts the problem onto the user instead of applying a well-known solution that was causing problems for nobody.
If you are writing an application, don't you want your transitive dependencies to get security fixes without having to trawl through the tree?
And how does it guarantee that this actually results in anything that works, if upper bounds are not supported?
This is somewhat misleading. Most package managers use a human-edited manifest containing constraints and a machine-edited lockfile, containing the chosen versions. You periodically (or after changing dependencies to require newer versions) run a tool that fetches the newest versions satisfying your constraints and writes out a lock-file, which you then commit and use at install-time to get reproducibility.
Go modules use a single file that serves as both and is both human- and machine-edited. You periodically (or after changing dependencies to require newer versions) run a tool that fetches the newest versions satisfying your constraints and updates the manifest with them. The manifest is then (also) used at install-time with minimum version selection to get reproducibility.
What that means is that in practice, Go modules never install older versions than the equivalent in other package managers would do. The maintainer specifies constraints and uses a tool to decide what version will actually used to at build-time. In both cases, that tool will choose the newest versions available.
What is correct (and contentious) is that Go modules have no concept of upper bounds for dependencies. So the complaint is, that as a library author you can not prevent a newer version from being chosen by one of your reverse dependencies. It remains to be seen how much of a problem that will be in practice.
So, if anything, the problem is that as opposed to other package managers, Go modules sometimes choose too new versions, from the perspective of some people. It never chooses too old versions.
This alone would make npm 100x less painful to use.
Admittedly, the "update all packages to latest compatible version" flag described above sounds very nice. There is approximately zero chance the npm developers would ever accept such a PR, but there would be nothing wrong with a utility that accomplished the same thing.
However, I am also in the unusual position that I had to design and implement a package manager for my own little language recently. My original hunch was to just create a Cargo clone, but after reading Russ Cox's writings, I ended up cloning vgo (with a few simplifications). The implementation was wonderfully simple - everything just fit together. The algorithms are trivial. Operationally, it is easy to understand what the package manager is doing (even more so than for vgo, as vgo has do deal with various Go idiosyncracies and backwards compatibility).
This is in stark contrast to my experience with other package managers, which are temperamental beasts at the best of times. When they work, everything's groovy, but their error modes are easily incomprehensible. I think vgo's approach of restricting expressivity and streamlining processes is the right one. But since nobody has used such a package manager before, it remains to be seen whether it works in practice.
For my own experiment, I do have one data point: I showed people the (rather brief) documentation for my vgo-inspired package manager, and they all felt it was very simple to follow and easy understand what it did, and how.