News for hackers. Stuff that matters.
Re:Pkg.go.dev is more concerned with Google's interests than... by jrockway on Saturday August 01, 2020 @03:55PM (#24023388)
There is some complexity here. Go is one of the few programming languages with a fully-decentralized module system. I can upload a module to example.com, and you can use a module from example.com with no centralized coordination. This is very different from systems like Node/npm or Perl/CPAN, where if you want to write a module, you have to beg them for permission first. (This is a moneymaking opportunity, too. You can charge people for your package system if they want to keep their code private. And npm does! With Go, you can have private modules for free!)

There end up being some problems with this model. Popular modules might be hosted on some server that can't stand the load, and if you can't pull dependencies, the entire module system breaks down. So, you probably actually want the modules on a CDN, because high availability is important for CI systems. Before Go automatically pulled from Google's module proxy, I had my own. I had to, because Github rate-limited our pulls (and when it wasn't rate-limiting us, it was slow). Setting that up was a few hours I had to spend for no good reason -- it didn't make my product better, except by making some incidental step between idea and deployment slightly faster. You can of course vendor your modules (now your git clone is slow), or add caching to your builds (go's caching is very good, but doesn't play well with Docker). All of them take your time and mental energy to do something unimportant. So, Google made a module proxy that Go pulls from by default. It works every time. It probably costs them almost nothing. There is no business value in it. Google doesn't need Go for anything. They just have some employees that want to improve the lives of average programmers. You can turn it off. You can host your own -- go/x has an example proxy, and there are full-fledged open source projects that add more features (Athens). It's actually a very good situation. You can be as centralized or as decentralized as you desire.

Going back to centralized package databases, they do have some really useful features. The main one I use is looking at users of a package that I'm reading documentation for. No example code in the repository? No problem. You can go see how other people are using it. The problem there is getting a list of all possible packages. You can poll common repository hosts for them. You can convince everyone to use a certain proxy that gets the names of the modules you are using so you can go index them. That's what Google does, and the user experience is great. You trade Google the information "the IP address 1.2.3.4 uses github.com/foo/bar" to get the knowledge of every Go module in the world. Seems like a good trade. And you can easily fork go and tell it to use your proxy, and make your own package database. The code is open. The license allows it. Go for it!

Would it make more sense for Go to be owned by something like the CNCF instead of Google? Sure. But they did make it, so it's kind of their choice. It is pointless for a large company to make a brand new programming language, but they did it anyway, and it's nice that we have it. They give us lots of free stuff and ask for nothing in return. The fact that your CI system pulled a Go module is not something that advertisers are salivating over. If you want to hurt Google because you don't like them, ask your lawmakers to make advertising illegal. That would show them! But caching Go modules is not a big moneymaker for Google, it's just kind of a nice thing they do because a few employees thought it would be good. I appreciate it. If you don't, set up Athens and forget about them. I am sure you can render godoc just as nicely as pkg.go.dev with only a few hours of work. The thing is, it's boring and nobody would care.

Re:Pkg.go.dev is more concerned with Google's interests than... by abraham on Saturday August 01, 2020 @04:37PM (#24023708)
You can install node packages from any git host or tarball url.

https://docs.npmjs.com/cli/install

You can also use any registry you want.

https://docs.npmjs.com/using-npm/registry.html

Re:Pkg.go.dev is more concerned with Google's interests than... by jrockway on Saturday August 01, 2020 @05:31PM (#24024061)
True! This does exist, however: https://www.npmjs.com/products
Re:Pkg.go.dev is more concerned with Google's interests than... by Xylakant on Saturday August 01, 2020 @06:04PM (#24024272)
So they do offer private package hosting as a business like GitHub offers private git repos side by side with public plans? I’m not seeing what you’re trying to point out. They’ve built something and they’re charging money for it. You’re welcome to use any other private npm registry.
Re:Pkg.go.dev is more concerned with Google's interests than... by htmlproplus on Saturday August 01, 2020 @04:08PM (#24023488)
Why not make the proxy an opt-in fallback by default? I like my fails to be catastrophic. I don't want to touch any host I don't have to. Ever. And doesn't it leak metadata if you aren't careful? I remember a Go page to request data removal. A package named github.com/microsoft/clippy already makes my skin crawl.
Re:Pkg.go.dev is more concerned with Google's interests than... by enneff on Saturday August 01, 2020 @05:33PM (#24024073)
If you don’t want to use the proxy you can turn it off, trivially. M For the overwhelming majority of users the proxy is a huge benefit (installing and updating dependencies is way faster and the checksum database provides security benefits).

Note that most other packaging ecosystems use centralised services, where you don’t even have the option of merely turning off the proxy.

Re:Pkg.go.dev is more concerned with Google's interests than... by yencabulator on Saturday August 01, 2020 @06:04PM (#24024271)
I know you know this, but gosumdb and goproxy are two very different entities, consumed separately.

Let's not muddy conversations about the default centralized goproxy with the existence of a global TOFU service. gosumdb would be perfectly possible to query without even revealing a module import path you're asking for (just like browser-side safe browsing can do).

Re:Pkg.go.dev is more concerned with Google's interests than... by jrockway on Saturday August 01, 2020 @04:22PM (#24023590)
Not using a proxy by default causes people a lot of pain -- it's the common case to not care that the network is in use if it makes your builds faster and more reliable. If you don't want to accidentally leak things to the network, that's kind of your OS's or IT department's jobs. The tools are available and they will work with things that aren't explicit about needing the network. (What I'm saying is, if this concerns you, Go is one worry of many.)

I have a concrete example about why module caching is good. I have an example repository that shows what happens when upstreams exercise poor release hygiene. Clone github.com/jrockway/evil-module-user. Run "go run main.go". (It's not actually evil, it just prints a message.) You will find that with the Go module proxy turned on, it compiles and prints "This is a good module!". With the proxy turned off, though, you will find that the checksum in go.sum doesn't match for the upstream github.com/jrockway/evil-module, and that evil-module has been silently compromised to become evil. What happened here is that evil-module released v0.0.1 and the module cache cached it. Then someone edited the code, and forced pushed the v0.0.1 tag to the repository, pointing at the commit. When go get reaches out to Github to clone github.com/jrockway/[email protected], it gets code that doesn't match the checksum in github.com/jrockway/evil-module-user's go.sum, because it's not the v0.0.1 that the developer of github.com/jrockway/evil-module-user used when writing the code.

Obviously this example is intentionally contrived to demonstrate this exact problem. But when I first started using Go modules early on, this happened all the time. Different engineers got different "version 0.0.1" from many repositories, and didn't notice there was a problem until CI. They then loudly complained that go modules suck, when in fact the problem is that upstream authors were very fast and loose about retagging releases. They are sending your release system code that you've never even seen! That's good that the module system detects that. But annoying to fix :) The problem all went away when Google started caching the modules -- you get the old version of the code, but at least everyone gets the same version. (As an author, you can always release v0.0.2 to bust the cache.)

I actually thought until I investigated this in great detail recently that everyone had just embraced immutable version tags and were being smart about releases, as they learned more about Go modules. Turned out I was wrong -- people are as bad as ever, but they don't break your CI build because there is now a centralized cache that gives your workstation, your coworkers, and your CI system the same code. The upstream modules authors aren't doing that. They are force-pushing with abandon.

Re:Pkg.go.dev is more concerned with Google's interests than... by yencabulator on Saturday August 01, 2020 @06:07PM (#24024299)
goproxy and gosumdb are two separate services, you can get the TOFU benefits without goproxy, and a checksum database would be possible to query without even revealing what import path you are talking about (just like browser safe browsing).
Re:Pkg.go.dev is more concerned with Google's interests than... by jrockway on Monday August 03, 2020 @02:07AM (#24034649)
I think the proxy is a bigger benefit than the checksum database. For the CI use case, it lets you get your modules from a server that's under less load than github (which has to support writes). Your go.sum already contains the checksums, so the sum database is not relevant there, but the module cache adds a big reliability and speed boost to your workflow.

The checksum database is of course extremely important (you want to get a known-good copy the first time you use a module), but I stand by my assertion that it's beneficial for both to be turned on by default. Yes, it's a small privacy leak. But not one that is particularly personally embarrassing or useful to adversaries. So I think the go team made the right call.