News for hackers. Stuff that matters.
Re:Go 1.17 Beta by jsjsbdkj on Thursday June 10, 2021 @03:35PM (#27464674)
Go is mostly very good, but the module ecosystem continues to be a nightmare, and having a deep dependency graph can make build times brutal. I work on a project which depends on k8s, which means a cold build take several minutes. Not to mention trying to update dependencies and figure out what versions are actually being built into the final binary.
Re:Go 1.17 Beta by 37ef_ced3 on Thursday June 10, 2021 @03:38PM (#27464715)
Anyone who works on large C++ projects will laugh at your "several minutes" comment. You may not realize how bad things are on the other side of the fence
Re:Go 1.17 Beta by fpoling on Thursday June 10, 2021 @04:45PM (#27465562)
Yep, without a build farm it takes 6-7 hours to do a full Chromium build...
Re:Go 1.17 Beta by pjmlp on Thursday June 10, 2021 @03:57PM (#27464997)
Only if they insist in compiling the world from scratch, though.

On the world of DLLs and COM, those compile times are relatively ok.

Re:Go 1.17 Beta by anthk on Thursday June 10, 2021 @04:43PM (#27465534)
ccache
Re:Go 1.17 Beta by hajhatten on Thursday June 10, 2021 @03:55PM (#27464970)
Been working with go for over 5 years now, I dread every time I have to touch a project with anything related to k8s. Don't get me wrong, I think it's great what they have accomplished project wise, but what a complete mess code wise.
Re:Go 1.17 Beta by distribot on Thursday June 10, 2021 @04:10PM (#27465163)
I don't get it I thought k8s was like the perfect match for Go? K8s is even written in Go. I know that doesn't strictly mean it must be easy to use Go, but it seems weird that it wouldn't be K8s best experience.
Re:Go 1.17 Beta by zellyn on Thursday June 10, 2021 @04:24PM (#27465307)
It has a reputation for being terrible Go code.
Re:Go 1.17 Beta by davidbanham on Thursday June 10, 2021 @06:15PM (#27466513)
The post is referring to the codebase of k8s itself, not the experience of deploying go code to k8s.
Re:Go 1.17 Beta by sleepybrett on Thursday June 10, 2021 @04:33PM (#27465414)
My read is that it just go too big and they didn't do a great job of keeping up with golang. Last I looked it was still not module compatible fully.
Re:Go 1.17 Beta by pjmlp on Thursday June 10, 2021 @05:11PM (#27465925)
It actually started as a Java project until the team got some strong advocates.
Re:Go 1.17 Beta by morelisp on Thursday June 10, 2021 @03:53PM (#27464937)
K8s is a massive outlier in Go build times. Several minutes for cold build times is a blessing. Lots of C++ builds still run overnight. Composer takes 3 minutes just to validate the dependency graph in one of our older PHP projects.
Re:Go 1.17 Beta by kardianos on Thursday June 10, 2021 @04:45PM (#27465570)
Go modules are great.

k8s has so far decided not to use module versions, which makes using k8s with Go hard.

Re:Go 1.17 Beta by beforeolives on Thursday June 10, 2021 @03:53PM (#27464935)
Who is doing dependencies right in your opinion? Because it feels like it's a complaint I hear about every language.
Re:Go 1.17 Beta by coolreader18 on Thursday June 10, 2021 @04:35PM (#27465443)
Might be biased/stereotypical, but Rust's cargo does dependencies really well. It's as easy as npm to add new dependencies, but there aren't thousands needed to do anything, and if you take a look at your Cargo.lock/`cargo tree` you can really get to know each of them and what they do or why they're pulled in. I'm still bloat-wary, maybe as a leftover from doing webdev, but with less transitive dependencies in the first place you can actually go through and prune things that aren't needed, or open PRs to transitive deps to prune from their trees or update deps to the latest version to deduplicate your tree. (If there are multiple semver-incompatible versions in a dep tree, they just both get compiled in - for most apps though, you should be able to get the number of duplicates to 0 or almost that.)
Re:Go 1.17 Beta by aprdm on Thursday June 10, 2021 @05:42PM (#27466213)
I wonder how Cargo will be regarded as in 2-5 years if it gets anywhere near the number of projects/libraries that say Python or Javascript has.
Re:Go 1.17 Beta by pjmlp on Friday June 11, 2021 @01:26AM (#27469628)
You can either go to lunch or buy a Threadripper server to be able to compile a full project from scratch, everytime you want to check something.

Unless by then cargo has already learned how to deal with binary crates.

Re:Go 1.17 Beta by colin_mccabe on Thursday June 10, 2021 @06:28PM (#27466666)
If compile time is the problem, Rust is not (yet?) the solution.

I agree that Cargo is much better than the Go build system, though.

Re:Go 1.17 Beta by morelisp on Friday June 11, 2021 @02:08AM (#27469894)
coolreader18's points are mostly about the culture of JavaScript vs. Rust (where Go also hews much closer to the Rust side). Setting aside lockfiles vs. MVS (which won't get "solved" in an HN debate), why do you prefer Cargo to go mod?
Re:Go 1.17 Beta by pjmlp on Friday June 11, 2021 @01:25AM (#27469615)
For me, it only gets the "really well" when it also does binary dependencies vcpkg/conan style.
Re:Go 1.17 Beta by Nullabillity on Friday June 11, 2021 @12:11PM (#27475151)
Having tried both, focusing on source dependencies is the only way to make sure that dependency sources are universally available and buildable, which makes a huge difference in the long run. Just look at NuGet's issues getting SourceLink adopted.

Binary caching á la Nix can work, but I can't really see that working out without Nix's commitment to environment purity.

Re:Go 1.17 Beta by pjmlp on Saturday June 12, 2021 @03:48AM (#27482507)
When one is lucky to use server hardware as workstation, and works in domains that don't depend on selling binary libraries for their business.
Re:Go 1.17 Beta by Nullabillity on Saturday June 12, 2021 @08:59PM (#27488829)
> When one is lucky to use server hardware as workstation

Sounds like something is pretty screwed up if you're running cargo clean as part of your regular workflow.

> works in domains that don't depend on selling binary libraries for their business

Play stupid games, win stupid prizes? Meh.

Re:Go 1.17 Beta by ufmace on Friday June 11, 2021 @12:28AM (#27469298)
IMO, Ruby Bundler was the first language / ecosystem to really get dependencies right. AFAIK, they were the first to have a lockfile locking the version of each dependency and a Gemfile with flexible version specification for dependencies, and a tool to handle installing the right version of everything and making sure you run in the context of your specified gems every time, no matter what else is installed on the machine.

Rust Cargo does about as well, probably the best for a compiled language. NPM could be about as good too - it almost feels like they deserve a point off for the ridiculously huge number of tiny packages required to do anything, though that isn't really the dependency manager's fault.

Re:Go 1.17 Beta by pjmlp on Friday June 11, 2021 @01:23AM (#27469600)
CPAN and Maven did it first.
Re:Go 1.17 Beta by morelisp on Friday June 11, 2021 @02:06AM (#27469882)
Maven doesn't have a lockfile, from just reading the files it's difficult to find the version of a dependency you'll actually get.
Re:Go 1.17 Beta by pjmlp on Friday June 11, 2021 @02:29AM (#27469993)
It doesn't need one, when version is already part of dependency definition.

If you want to be sure of what version you get, use it.

Re:Go 1.17 Beta by morelisp on Friday June 11, 2021 @02:40AM (#27470052)
Transitive dependencies are not going to be in your POM file normally, and Maven has a confusing algorithm for resolving them (essentially, first found in a BFS over the dependency tree). And if you do include them, that will silently override transitive dependencies on more recent versions, which is rarely what is wanted.

Yes, you can ask the tool to print them. This is way worse than any of the other systems being discussed, where you can read a file in the repo.

Re:Go 1.17 Beta by pjmlp on Friday June 11, 2021 @02:47AM (#27470095)
It is so hard to redirect the tool output to my-favourite-filename.lock, beyond the skills of the average developer.
Re:Go 1.17 Beta by hawski on Friday June 11, 2021 @06:38AM (#27471521)
One may like it or not, but defaults matter.
Re:Go 1.17 Beta by pjmlp on Friday June 11, 2021 @06:50AM (#27471602)
Defaults don't change history.
Re:Go 1.17 Beta by morelisp on Friday June 11, 2021 @07:15AM (#27471748)
Naming the file .lock does not make it a lockfile. (I can't believe I have to write that sentence. I don't think you're discussing technical details in good faith.)
Re:Go 1.17 Beta by pjmlp on Friday June 11, 2021 @07:30AM (#27471867)
Of course not, it fixes "... where you can read a file in the repo", because Maven already does everything else anyway.
Re:Go 1.17 Beta by morelisp on Friday June 11, 2021 @11:31AM (#27474684)
> Ruby Bundler was the first language / ecosystem to really get dependencies right. AFAIK, they were the first to have a lockfile locking the version of each dependency and a Gemfile with flexible version specification for dependencies,

So other than having a lockfile and having a flexible version specification (and a sensible way to resolve multiple version requests), Maven did "it" first!

... what was "it" again?

Re:Go 1.17 Beta by pjmlp on Saturday June 12, 2021 @03:49AM (#27482509)
Everything.
Re:Go 1.17 Beta by jrockway on Thursday June 10, 2021 @04:06PM (#27465114)
I have a bunch of apps that depend on Kubernetes (consumers of client-go), and I haven't experienced long build times. I cleaned my cache ("go clean -cache") and tried building one:

  $ /usr/bin/time go build ./cmd/ekglue
  70.33user 32.72system 0:07.92elapsed 1300%CPU (0avgtext+0avgdata 
  583644maxresident)k
  333392inputs+1232496outputs (14768major+1135114minor)pagefaults 0swaps

  $ /usr/bin/time go build ./cmd/ekglue
  1.54user 0.90system 0:00.33elapsed 741%CPU (0avgtext+0avgdata 
  71952maxresident)k
  16inputs+0outputs (3major+11111minor)pagefaults 0swaps
It was 8 seconds for a clean build, and 330ms for an incremental build. I agree that building on a 1 core machine with no build cache and module cache is slow. I also realize it's a big pain to preserve the cache between docker builds, so you probably hit this with every commit in CI. The problem is really CI, not Go, but I agree that it sucks. I use Cloud Build / Kaniko which has decent caching, but I do have to wait 1 minute on every build for GCP to provision a new machine (since I'm using a larger-than-default machine; the time spent sleeping while a machine is provisioned is saved by parallelism in the Go compiler). Meanwhile, I run tests on CircleCI, and that is mostly bogged down by very high network contention; pulling caches is slower than rebuilding from scratch.

As for modules, I like them. My biggest blocker in using other programming languages is that their package system sucks compared to Go. I do run into problems -- upstream authors don't really know how to use Go modules, and upstream applications are very quick to take on unnecessary dependencies. For example, I depend on the Loki client. The Loki client and server are the same Go module, so they pin me to a particular version of the Prometheus library (that the Loki server depends on), which then pins me to a particular version of the Kubernetes library (that Prometheus depends on). Basically, the dependency graph is hundreds of times larger than it needs to be, because it's not a problem for the upstream authors and they've never thought about it. Splitting the client and server into two modules would make life much easier for consumers, but slightly harder for the producer, so it's rare that you see it one. (My team also makes an app that makes this same mistake -- forcing users of the client to depend on things like Kubernetes. It's hard to fix, because the server uses the client internally, but I may do it in the future. Or just auto-copy the client code into a separate repo + go.mod file for consumers!)

Upstream authors are also very quick to make fixes for themselves, unaware that they don't propagate to consumers. Many libraries have "replace" directives in go.mod, but those don't propagate to consumers, causing solved problems to reoccur for each consumer. You have to manually propagate them yourself. The solution there is to be a good open-source citizen -- if you have to hack up some module, either properly fork it and depend on the fork (there should be a tool that handles this renaming for you automatically), or push your changes upstream and depend on the new release.

Basically, modules involve the transitive closure of all shortcuts a bunch of people you've never met have taken, and the results are not always good. That has been true in every language I've ever used; I have 83 irrelevant Depndabot alerts that can't be fixed in most of my Javascript projects, for example. I think it's the best module / packaging system I've ever used, and I like it very much. In fact, there is little I'd change.

> figure out what versions are actually being built into the final binary

    go list -m all
Will print the versions.

    go list -m -u all
Will print what versions you could upgrade to.

I personally include this in every binary I produce: https://github.com/povilasv/prommod

This lets me monitor module versions across the fleet. If there's a security problem in a module, I can instantly see which apps are affected, and update them.

Re:Go 1.17 Beta by truffdog on Thursday June 10, 2021 @04:50PM (#27465646)
OP might want to check their security software- things that hook into every file access can really kill compilation performance.
Re:Go 1.17 Beta by Groxx on Thursday June 10, 2021 @04:33PM (#27465419)
>go mod list -m -u all

minor correction: just `go list` on these. the `-m` ensures it's in module-mode.

Re:Go 1.17 Beta by jrockway on Thursday June 10, 2021 @04:46PM (#27465601)
Thanks! I typo'd it both times! It's now corrected.
Re:Go 1.17 Beta by psanford on Thursday June 10, 2021 @09:23PM (#27468271)
You aren't really pinned to the version of Prometheus the library depends on. Minimal Version Selection[0] means if you specify a newer minimal required version of Prometheus in your go.mod that is the version that will be used.

Its slightly annoying that your dependencies' optional dependencies pollute your go.sum, but it really doesn't matter. If those don't show up in your package import graph they won't be included in your builds.

[0]: https://research.swtch.com/vgo-mvs

Re:Go 1.17 Beta by jrockway on Friday June 11, 2021 @12:32PM (#27475393)
The problem is that Kubernetes went backwards in version numbers. I want to depend on, say, v0.19.0, but through the dependency chain, we end up with a dependency on v12.0.0. (At some point, they decided to switch from "kubernetes x.y.z = [email protected]" to "kubernetes x.y.z = [email protected]".)

I made a test module that depends on loki and kubernetes, and updating kubernetes results in:

  $ go get k8s.io/[email protected]
  go: downloading k8s.io/client-go v0.19.0
  go get: downgraded github.com/grafana/loki v1.5.0 => v1.0.2
  go get: downgraded github.com/thanos-io/thanos v0.12.1-0.20200416112106-b391ca115ed8 => v0.11.0
  go get: downgraded k8s.io/client-go v12.0.0+incompatible => v0.19.0
This causes loki to downgrade to a version that doesn't work.

I don't blame the module system for this, I think it's doing a great job. But when you use other people's code, you're responsible for the transitive closure of all their minor tiny mistakes, and when you depend on big codebases, the mistakes really add up. That's where the hate comes from; a lot of code was written before modules existed, and modules changed the semantics of that code.

Re:Go 1.17 Beta by psanford on Friday June 11, 2021 @03:26PM (#27477446)
Fair.
Re:Go 1.17 Beta by theshrike79 on Thursday June 10, 2021 @03:54PM (#27464951)
...you can literally go on a lunch break while doing a full C++ recompile on a moderately complex project.

Your Go project must be on an epic scale to break even 10 minutes of compile time.

Obligatory XKCD: https://xkcd.com/303/

Re:Go 1.17 Beta by distribot on Thursday June 10, 2021 @04:11PM (#27465179)
Is it that bad compared to npm?