But I am very happy, that they did not pick a random management system, just because they "had to have one". We can see enough bad examples in the wild. They left this aspect out of the language standard, to tackle the challenge later, with proper consideration.
As far as I understand all details, Russ Cox has presented a very thoroughly worked out spec, and it seems to solve a lot of problems, other dependency management tools have. It is also completely compatible with Go so far and also fully optional.
But they have, multiple times? First it was just giving people links to godep when people complained in github issues. Then it was a tacit "we like this" about gps and glide. Then it was a pretty-much-official-blessing of dep. Now it's a super-official-now-we-mean-it implementation of vgo. I recognize this as a pattern because I do it all the time, software engineers constantly re-write things from the ground up when they don't care about actually solving the problem and just want to play around with cool new ideas.
As far as the vgo spec, I'm not against it per se... I like most of the general conclusions, but none of it is new. We've had these ideas in the Go world for a while. Most of them have even been implemented! Why can't we just slowly transition an existing tool? Or maybe implement a common library that individual tools can use to solve the problem in their own way? Oh wait we already did that and it got abandoned. Until we see a tool with a stable API that solves the 80% use case and lasts for over a year I'm not going to get excited about the new flashy thing.
No. Golang hasn't had an official/pretty-much-official dependency management system. The community recommends some tools over others, but that's about it.
With dep it is a bit different - the creators were google engineeers and they wanted it to become the official tool, with integration as a go subcommand etc.pp. - but that was never signed off by Pike or any other lead, only confirmed that it would be a possibility if the tools performs well.
With vgo its similar to dep - a proof of concept to see _if_ it works nicely.
Regarding a library multiple tools can use; Yes, that would be a nice thing - however when designing such a tool _and_ a library at the same time you will get into bigger problems. Generally you'll want a library designed by someone who already solved a given problem in at least one way. That brings enough experience to make the correct architectural decisions early on as well as a good, usable API.
I'm happy that we don't just get half-assed solutions just to have one. I have seen enough projects go to shit because a major step in the project was reconsidered a few times before management put a lock on it because it went through 'enough' revisions.
I'm frustrated. I started using Go during version 0.8 and have been using it since with no dependency mgmt beside my own forks because I hated everything I tried. Then dep comes along, I give it a few months to mature, try it, and it worked great. It solves my common use cases near perfectly.
What this blog post and proposal lacks IMHO is a clear case for why dep isn't "good enough". All I see is handwaving:
"Early on, we talked about Dep simply becoming go dep, serving as the prototype of go command integration. However, the more I examined the details of the Bundler/Cargo/Dep approach and what they would mean for Go, especially built into the go command, a few of the details seemed less and less a good fit."
So "a few of the details seemed less and less a good fit" is a statement that is impossible to refute because it is too scare on any details.
You want feedback now, on this new proposal?
Why bother?
It's patently obvious that our feedback on dep has been totally ignored.
What indication is there anyone will pay attention to any feedback now?
I can't be bothered any more.
I just want one solution to be picked, blessed, implemented and then not changed five minutes/months later.
If you read the minimal version selection algorithm, it's clear that the proposed solution solves the problem in a novel way.
Having started in the 90s, I have been through these same conversations regarding Adobe, Oracle (oh lawd Oracle) MS, Dell, Apple to a lesser extent.
“Vendor the industry currently relies on heavily is changing things or yanking the rug again?”
The free market works by big players yanking every one else around. Guh.
It's SMB/"indie" developers who should be taking the responsibility here, I think, for relying on these officially-unstable packages in their own production systems, and especially for encouraging others to rely on such. Yes, sometimes, it's the only way to be competitive/have a https://en.wikipedia.org/wiki/Unique_selling_proposition. But that doesn't mean that it isn't their responsibility to cope with these changes. They're taking on that risk-of-change by using those unstable libraries or APIs.
Heck, this is half of the point of software Venture Capital: being able to underwrite the risk of relying on an unstable platform/ecosystem, so that you can have enough runway to recover and still finish your product if "the ground moves under you", and therefore can "safely" rely on these unstable technologies.
(And one not-oft-talked-about property of bootstrapped startups is that they have to underwrite that risk themselves. If a bootstrapped startup isn't just gambling with its founders' money, it has to behave more like an enterprise development shop, building on only stable foundations.)
Theoretically, you could move this underwriting role into the companies themselves, such that the companies themselves would monetarily cushion the blow of changes in their unstable APIs. But they'd need a pretty big interest in the involved consumers to do so. In such a world, you likely wouldn't have separate startup businesses using the BigCorps' unstable APIs; rather, you'd have a set of BigCorp incubators with large shares in most new startups. Maybe that hypothetical world would have higher https://en.wikipedia.org/wiki/Gross_world_product than our own! But I bet there'd be fewer startups, because starting a startup would be a lot more like a regular BigCorp job.
Do you think that's a community process? I always have the impression from Go developments that all major decisions are "my way or the highway" affairs of 2-3 of the initial designers. Possibly the same ones writing those vague "exploratory" posts and dismissals of common proposals, that voidlogic captures well in his comment below.
Many ecosystems haven't gotten past the "single dependency file and lock file" system where an entire project has to use a single version of each dependency. There are people splitting their project into a thousand "microservices" and making them interoperate over HTTP instead of regular function calls so that each part of their project can have its own dependencies.
And while I have been using development snapshots for years with my system and didn't have more bugs to cope with compared to other systems, I can understand that developers in corporate contexts need to be able to specify dependencies with versions. So I appreciate that proposal very much and hope it will lead to a better solution for every body involved by providing a way to use versioned dependencies without introducing too much overhead for those who don't need versioned dependencies.
No package manager can fix a cultural problem.
Meanwhile, I've heard people spend ages complaining that this is totally wrong...while overseeing setting up local artifactory instances and wrangling proxying to do the exact same damn thing.
Perhaps they shouldn't go past that.
multiple library versions is one of the lowest priorities when doing micro services id say
this is not to say that people don’t significantly over use micro services, but there are plenty of hard problems they solve
Reread the post, the authors make some great, mature realizations that I hope other languages will listen to. I don't use the language, but I use languages that Go has affected and this is a great thing.
The one time I needed that I found a pretty good workaround: you make two crates `helper_old` and `helper_new` and have each depend on different versions of that crate. Then I just pub re-exported the crate it depends on.
That way your other crate can now depend on two different versions of that dependency.
When does this situation come up? What is a compilation unit in this case?
You might need a handler for both versions if you have customers that use the old api and ones that use the new one. Or different departments of the same customer.
Or if you dodge that bullet but have to write a custom migration tool to get data from the old version and cook it to go into the new one. Half of the code is read only and the other write mostly but they still have to coexist if you do anything more complex than a backup/restore script.
Rust had actually been through three failed package manager projects before they decided to bring in the domain experts at Tilde. Nice that at least that bit of churn has been forgotten - but yeah, it became obvious over that time just how non-trivial the problem is. Cargo makes it seem easy though ;)
Many, I think, based on what the author of dep says (https://sdboyer.io/blog/vgo-and-dep/). I don't get why rsc would just throw away the code and work that was the culmination of years of community work. Isn't rsc largely responsible for creating this problem in the first place? I'm doubtful he's going to fix the dependency problem from scratch. There's a certain pigheadedness to a lot of golang decisions, and I'm afraid this is another one of them...
I think they threw out a bit too much, but that's only a problem if the gaps don't get filled or the compromises aren't adequately explained. What's left is a very pragmatic selection of extremely useful tools that can be used to cut a tonne of working code very, very quickly. I can't say I always like it, but I benefit tremendously from it.
For example, why do unused variables and imports, a style issue, cause a compiler error? This is a constant pain if commenting out lines of code during debugging or prototyping. On the other hand, the compiler doesn't care if I ignore or forget to check error values - which is almost always incorrect behavior.
Here's why - https://golang.org/doc/faq#unused_variables_and_imports
At least the Golang team can cherry pick the best ideas & lessons-learned from these community-driven package managers.
At the very least they could have continued their hard core hands off approach of allowing the community to solve it. But instead they half-assed it and started capriciously anointing chosen solutions and honestly we're probably in a worse spot than we were before dep came along.
Plus, lets be honest here, there's no excuse for constantly changing APIs and binaries. If this was truly about getting the best ideas and lessons learned we'd just see refactors of existing tools with slow migrations to new concepts, but instead we've seen multiple complete rewrites, despite multiple efforts to build common libraries that should prevent such events!
Huh? Of those three, I would only consider Java "solved", and both Maven & Ant are far from the panacea of package management.
virtualenv is a community project (much like dep), is pretty new in the grand scheme of things, and isn't completely standardized (some projects use tox, some list out requirements.txt (rarely version pinned), and others vendor.
As for C++, almost everyone does something a bit different so I don't see how that is at all relatable.
virtualenv was a community project but based on that venv was created and has been part of the official Python distribution since Python 3.3, released in 2012.
It doesn't stop there either I've seen python projects using nothing, virtualenv, venv, pip-tools, and pipenv.
I finally feel like pipenv is the true solution and it seems very new. All that said and Python is quite old relative to Go.
* a central package registry
* a file to describe dependencies
* a command line tool to install, build and publish packages
In Python things are more fragmented. You have:
* a central package registry
* a file to store some of your dependencies (Pipfile)
* a command line tool to install packages (Pipenv)
* a file called setup.py in which you need to specify your dependencies again (in another format). You can execute this file to build/publish packages.
It think it would allow for a much nicer user experience if pipenv/pipfile would handle packaging/building/publishing as well..
Of course, gradle hooks into the whole maven ecosystem but that is one of its advantages. Everybody in Java land understands Maven packages but how you generate doesn't matter.
If it wasn't for Android, I wouldn't bother 1 second to learn Gradle.
But I guess Groovy needs some project to keep it alive, now that no one remebers the days JSF beans would be written in Groovy or JUGs were holding Grails talks month after month.
C++ has no widely-used dependency management systems.
Both languages are doing fine.
On Windows, Microsoft is investing on vcpkg as package manager.
Regardless of your setup at the end of the day you're using pip to solve your dependency graph, and have been for over a decade.
Except where have they done that? They've had 40 years and their solution to the C binary function error code return problem was... to add a special way to return an error code. There have been superior solutions to this for 20 years at least.
How long have we known about the value of generics?
To me, the fact that they can't figure out how to solve this isn't remotely suprising.
Most of Go's features are those that proved their value over 40 years. There are no dependency management schemes that have that kind of reputation. Good ideas take time and misfeatures are expensive.
If they are willing to throw out that requirement for something as fundamental why should dependency management be held to so high a standard.
Combine those 2 facts & I think it’s fair to say golang is willing to base things (central ones even) on untried ideas, which directly contradicts your stated position which was precisely that no dependency management scheme lives up to the precedent of the other accepted features of the language.
This is trivially proven incorrect via looking at other language dependency management schemes that have much more broadly proven their worth.
The Go designers used CSP over the course of many decades via Newsqueak, Alef, Limbo and Plan 9 C's libthread.
Did the principles enjoy a feature while using it on Plan 9? Likely to be included. A feature having broad industry or academic backing, or a long history? Immaterial.
In fact, as nice as Go might be, if it had the AT&T Research Labs stamp instead of Google's, it would been just as successful.
It absolutely doesn't contradict my stated position. I was clear about that in my previous post.
CSP inspired Go's features (in that sense alone it is "the basis for Go's concurrency" as alluded to by the Go FAQ) , but as I mentioned, it's not integral to Go in any way, and very few programs are modeled in CSP fashion.
Heaven-forbid you want to upgrade a package that exists in all 130 projects in the solution (not my call to have that many projects) - you may as well take a long lunch. I will try to make that the last task of the day so I can let it run for as long as it needs to.
The VS UI for NuGet is terribly buggy.
Try to build a C++ or Android project and then go for lunch instead.
NuGET is great.
Most likely someone modified the packages for an individual project and not the solution as a whole. Always manage packages at the solution level and your much less likely to have these issues, unless you have multiple solutions...
Basically, it 'just works' for simple senarios, its just not very good for anything else.
If you want a comprehensive guide to why its not awesome look at the 'packet' website, they cover the issues quite clearly.
^ you can actually use local dependencies, but its irritating and poorly supported (still uses the global system cache, forcing a cache flush to update).
What constant thrash? In the 6 years since Go 1.0 the only dependency management related change I'm aware of is the addition of `vendor/`.
Update: sounds like "thrash" refers to the variety of community vendoring tools. I guess of the few I've used they all felt fairly similar/interchangeable, so it didn't feel like thrashing changes.
1. https://github.com/tools/godep
2. https://github.com/kardianos/govendor
3. https://github.com/robfig/glock
4. https://github.com/rogpeppe/govers
5. https://github.com/Masterminds/glide
6. https://github.com/golang/dep
I'm sure I'm forgetting a few. And just when it seemed the Go team might be ready to standardize around Dep (6), they threw this wrench in the works.
I'm of two minds about vgo. I think it has some interesting ideas, but Dep works today, is widely adopted, and has nearly all the features you'd expect from a modern package manager.
There is a standard, and has been one from the early days. The catch is that the standard is not loved by all, so others have created competing package management systems to fit their own needs.
Got it.
There's definitely a feeling among Go developers that this problem should have been solved much earlier, and that the Go team's years-long refusal to address it caused the proliferation of mediocre tools (looking at you, Glide) that in many ways made the whole situation worse.
When Dep came along, a lot of people breathed a sigh a of relief, because we finally have something okay, and we can go back to being productive instead of fighting dependency management problems all day.
[1] https://github.com/golang/go/wiki/PackageManagementTools
This. Sure, new tools have appeared, but there's literally nothing wrong if you decide to just stick with godep or whatever you happen to be using.
These many tools were the thrash, including godep, gb, glide, dep, and more.
Each of these expressed versions with different manifests, many of them could import other format's, but not all. All-in-all it has been a mess.
Sure, the go team has not officially done much, but that's because their stance has been borderline negligent on the topic.
As for "it's a solved problem", is it? Until perhaps the last year, pip, npm, and most other package managers have been plagued with serious issues. I'm not fond of the situation in Go, but I think it's not unreasonable to try something different and less binding than a package manager until a solution emerges which solves the problems faced by other package managers.
I actually also liked dep with the vendoring approach, it is just better to ci a repo with vendored dependencies (no more Broken Building because of Internet/Proxy or github Problems). And in Go it was/is Even simple to upgrade them. Sadly go is slowly moving away from that Part.
the new approach is Bad, because they still pull Sources from Github, this Makes their approach unsuitable.
E: damn iPhone autocorrect
I know, Go is not a very old language, but all the Go projects I was involved in were started before that and I haven't written any Go in like 5 months, but still - this is too new for all the Go developers to have looked at it in depth (I certainly didn't come around to try it) and yet there's something new here.
We're all joking about a new JavaScript framework being released every week, but dependency management in Go feels a little alike.
I'm not a huge fan of Go anymore after my last experience (writing some web stuff, with CRUD) after having been a huge fan (writing monitoring checks and non-HTTP daemons), and I have no immediate plans - so I've no stake here, but I can really only hope this annoying part (which least-bad dependency management system will I use?) is finally over.
There is no graph solver, that's the whole point.
> Define a URL schema for fetching Go modules from proxies, used both for installing modules using custom domain names and also when the $GOPROXY environment variable is set. The latter allows companies and individuals to send all module download requests through a proxy for security, availability, or other reasons.
At least with vendoring I keep all of my code pinned in one portable repo and all I really need is dep and git. If I clone or backup the repo I don't have to think about cloning the state of the proxy also to get my code to compile.
I don't see that any serious business would be very enthusiastic about adding a code proxy as yet another cog in their development pipeline. One of the main appeals of Go is that it reduces many pain points of a development environment As a counterexample see e.g. the insane requirements of the Javascript web development ecosystem.
I also have some reservations about getting developers to adhere to a versioning repo layout standard - that /path/to/v2/ proposal and semantic versioning - absent any automated tools to enforce it (see cats, herding of). How many of the many Go github repos follow the recommended cmd/pkg code layout [0]? Not many. My cynical notion is that anything that's not automated out of the box is going to quickly run into the inevitable proclivity of humans to be messy.
Having said all this, I do think the discussion is worthwhile. My hope is that rather then completely switching dependency management systems the discussion identifies the things which are still painful with dep and fixes them. Let's face it, the vast majority of projects out there really don't need anything more complicated then dep, so I would hate to see it abandoned.
About vendoring, I think Russ Cox is aware of the issue. Look at this discussion on his website: https://research.swtch.com/vgo-module#comment-3771676619
Hey Phil! It's been a long time. C'mon man, when we worked together, are you telling me you never ran into the never-resolving dependency map with berkshelf and chef? I know I did many times. Locks and a manifest did not solve that. It sounds like the proposed solution should prevent that.
It's obviously not solved satisfactory.
Edit: I just realized that we may have a different definition of "compilation unit". If you mean "single program" then my point stands. However, if you just meant some sectioned-off part of the program, then I think the drawbacks of that are far less. Basically, I think that it is useful to be able to have access to multiple versions of a datatype (e.g. if you want to write a converter from old -> new) but that use case is far more fringe.
If backwards compatibility is important, then letting me use multiple versions side-by-side in the same modules easily is damn important.
If you live in a filter bubble (survivorship bias).
I swear I should put together a list of these instances.
The popular criticism is that Go isn't reinventing enough; that it's stuck in the 70's and it's not sufficiently innovative.
I've heard go described as a language that doesn't try to solve any hard problems. Let's set aside for a moment whether or not that's a positive or a negative. The bigger problem is that they also seem to completely ignore everyone else's existing solutions to hard problems too, which leads to the criticism that it's stuck in the 70's.
Sure, a lot of the existing solutions to some of these problems aren't perfect or involve tradeoffs that don't work for everyone. In general though, more recent solutions have built upon the lessons of previous approaches and now we've got languages and tooling that get a hell of a lot right, straight out of the box (I'm looking at you Rust / Cargo).
But go isn't doing this. They appear to be trying to come up with de novo solutions to problems from the perspective of the 70's. Whereas everyone else is "standing on the shoulders of giants", they insist on discarding the lessons we've learned over the past fifty years and running one by one into all of the same problems that led us to the solutions we have today. As an observer from the sidelines it's painful to watch them relearn these same lessons the hard way.
As for C#, people keep forgetting .NET always had a JIT/AOT compiler since version 1.0.
Just because not everyone bothers to sign their applications and call NGEN at installation time, it doesn't mean it isn't there.
Also Mono always supported AOT compilation, it is the way Xamarin works on iOS and the deployment story on Windows Store since it was introduced on Windows 8.
http://www.mono-project.com/docs/advanced/aot/
In any case, a limited form of generics is better than not having any at all.
They don't need to provide a turing complete implementation of generics, even a basic one like CLU had would already be an improvement.
Using go generate feels like the old days, writing generic code in C and C++ with pre-processor tricks.
In any case, the truth of the complaint about generics isn't actually relevant here: the rebuttal is to "people complain about Go not reinventing enough", and they don't, as they (truthfully or otherwise) think it is ignoring already researched and widely-implemented solutions to the problem and complain about this (that is, the complaint is "uninventing", not "not enough reinventing").
But even Swift implementation of generics has its issues and tradeoffs. Look for example at this thread about "Compile-time generic specialization" in Swift: https://lists.swift.org/pipermail/swift-evolution/Week-of-Mo...
Repeating that Go ignores "already researched and widely-implemented solutions" is getting old, really. Would you say that OCaml doesn't have shared memory multicore parallelism because it ignores already researched and widely-implemented solutions??? Would you say that Haskell garbage collector causes long runtime pauses because it ignores already researched and widely-implemented solutions??? The truth is that solving that kind of things is a lot more complex than just copy-paste the code from another language.
In any case, that specific concern doesn't apply to Go, since it doesn't have overloading.
> Repeating that Go ignores "already researched and widely-implemented solutions" is getting old, really
Sure, but that still isn't the point of this discussion: it's not what people should be doing, or the underlying truth, it's what people are doing in practice and they are complaining about the Go team seemingly ignoring the past 40 years of programming language development, not complaining that they're not inventive enough.
I didn't know you worked on Rust and now on Swift :-)
What strategy does Swift use to compile a generic function or method? Does it generate only one polymorphic version of the code, or multiple specialized versions for each expected parameter types (often called code specialization or monomorphization)?
The compiler can and does specialize functions as an optimization, but that's not necessary nor part of the semantic model.
It's interesting that Swift adopted this approach and confirms what you said earlier about Swift having essentially the same constraints as Go and being a good source of inspiration of Go.
What's the status of Swift on Linux, to write web services and connect to databases?
OCaml people definitely not ignore them, they are carefully investigating existing code and elaborating modern and efficient solution [1](unlike Go, which is too opinionated).
[1] https://github.com/ocamllabs/ocaml-multicore/wiki/Memory-mod...
That said, I don't see the link between the work done on OCaml multicore parallelism and Go "being too opinionated"...
If I follow you, Go people, unlike OCaml people, are "too opinionated", and don't "carefully investigate existing code" and don't "elaborate modern and efficient code". Do you realize how arrogant (and a bit ridiculous) this sounds?
That's popular, but unfounded, in my opinion.
I find ironic that some HN commenters are criticizing the Go team for not being stuck in the past when they are precisely trying to innovate and questioning the status quo on package management.
Actually it's pretty clear that they did their homework.
First, it was literally argued that just `go get` just grabbing `master` of all of your dependencies was good enough, because if you ever need to release something backwards-incompatible, you should create a new repo. I wish I were making this up.
Then we just vendored everything and never updated our dependencies, and that too was good enough.
Then GoPM, go dep, and a deluge of other tools that I don’t care to track down or sort into their appropriate order.
And now this, which decides to buck the trends that everyone else follows yet again and attempts to satisfy dependency trees by choosing the minimal compatible versions instead of the maximal ones, with the completely foreseeable consequences that:
* security patches won’t be applied
* issue trackers will be filled with bug reports that are closed with messages saying “this was fixed months ago, update”, and
* updating dependencies will become *harder*, since this is a straightforward result of doing it less frequentlyFirst, the upstream behavior; not supporting the published interface (as it was and should therefore work forever) means they need to deprecate that interface and release another with the incompatible changes. However the PROPER way of handling this is to add new versions of exposed public methods and interfaces / constants with new names and only if such a major change is required that you can't support the old interfaces, at THAT point, release at least under a new inclusion path if not new dependency name.
Things local a developer might have done wrong: A) include an 'immature' upstream repository that behaves like above. B) use non-public components or other unsafe practices to go beyond the interface contract that was exposed.
At that point, almost all of the go ecosystem can't be used.
I left the language years ago, but the steady stream of articles of the form “Why go doesn’t need $x” followed by innumerable comments, articles, and blog posts by people struggling with the lack of $x, followed by the go core team (due to the completely foreseeable consequences of not having $x) begrudgingly adopting $y, which is supposed to (but doesn’t actually) obviate the need for $x… I’ve reached my own conclusions, and I’m finding more and more people who are leaving the ecosystem with those same conclusions.
I believe that Go excellently solves problems that Google has. I don’t believe it solves many problems that most other users have, even though it might seem like it at the outset.
To re-iterate a different post of mine against this question (implicitly, what are the benefits of NOT having library versions):
* The only version to test against is HEAD
* No fossilization of security or stability issues
* Public libraries must support all Uppercased (exposed) declarations; that is the library interface.
* If the build breaks in an odd way: update go, then: go get -u all
* Simplicity, there's only one supported version and only one version to go get and develop against.
* Also, why would building against an old version _ever_ be necessary?That's a pretty good argument for using a monorepo, don't you think? A single organisation's code should all be in a single repo, which means that a single version of a dependency is used within the organisation, and that updating to a new version of that dependency is a single, self-contained project.
You can say that about Go itself. And there are already other/better alternatives. So why use Go?
Go has its good sides, but it's just a tool. It's not the best, because tools aren't supposed to be the best. Tools are supposed to be useful for what they're made for.
In comparison I can just start goroutines and Elixir processes for fast and simple I/O parallelism almost without thinking about it.
go dep is perfectly fine for most use case.
> How many iterations of this do we have to go through?
Fewer iterations than Mr. Cox has gone through. Isn't it great?
For all practical purposes dependency management for programming languages is a solved problem, with many open source examples. The only explanation for this constant re-invention that I can come up with (and that's shared among others I know) is that Google doesn't care/need a dependency tool because of their mono repo. Which is rather frustrating since we get features (cough aliases cough) forced down our throats when Google suddenly finds a need for them.