News for hackers. Stuff that matters.
Re:Go Module Mirror and Checksum Database Launched by donatj on Friday August 30, 2019 @10:02AM (#20838672)
Is there a justifiable need for the proxy? It seems like we’re just adding a centralized point of failure, the lack thereof something I have absolutely touted as a feature.

For verification we already have the go.sum file validating the hash. We’ve got vendoring if you’re concerned about dependencies disappearing.

The touted speed feature seems like a non-issue? The absolute largest Go package I’ve ever pulled took maybe a minute to grab its dependencies, and then I had them and never had to again.

The cynical side of me feels like this is a way for Google to collect analytics on the Go ecosystem. The slightly less cynical part of me thinks people from other ecosystems demanded it due to cargo culting the idea.

The fact that it makes it more difficult to use private modules is pretty irritating as well as I am going to need to make sure all our developers and build systems are aware when we move to 1.13. Having to set GOPRIVATE on each machine is absurd. It should be definable in the go.mod so it can be known via git.

Re:Go Module Mirror and Checksum Database Launched by nindalf on Friday August 30, 2019 @11:52AM (#20839755)
> The cynical side of me feels like this is a way for Google to collect analytics on the Go ecosystem

> other ecosystems demanded it due to cargo culting the idea

And there it is, the hallmark of a typical comment on a thread like this

* Cynical

* Expresses concern about data being collected even though this data can't possibly be used for anything bad. (If you disagree, go to crates.io and tell me what concerns you have).

* Expresses disapproval of a large corporation even though they haven't done anything wrong here.

* Pithy dismissal of other programmers by calling them names ("cargo cult"). This change improves speed of downloads by 6-7x on poor connections and 3x on good connections. Clearly this is helpful for situations like CI which typically involve clean builds. What's the issue here exactly?

* Furious attack on a free product that's completely optional to use. If you want to keep vendoring your dependencies, go ahead and do that. Why are you angry that some people are going to use this proxy?

Re:Go Module Mirror and Checksum Database Launched by donatj on Friday August 30, 2019 @12:07PM (#20839887)
"Furious attack". For serious? I gave a light hearted criticism of adding a single lynchpin to the entire Go ecosystem. I think that's a legitimate source of concern.

I was by no means pithy, I was by no means being dismissive. Thinking you need something just because you had it elsewhere is the definition of cargo culting - and is something the Go developers have otherwise largely avoided.

Just because something is faster doesn't make it better or worth it, particularly when it comes with tradeoffs and is already really fast - which is my argument - that the tradeoffs aren't worth the benefit.

Re:Go Module Mirror and Checksum Database Launched by nindalf on Friday August 30, 2019 @12:22PM (#20840042)
This is not a single point of failure. It's by no means a legitimate concern. Go read the proposal. If you don't want to use proxy.golang.org, use goproxy.io. Or host your private one. Fairly sure you'll be able to use github soon as well. Do whatever.

I notice you couldn't back up your FUD about analytics being collected.

Go programmers are happy that they have a solution where their dependencies (and specific versions of those dependencies) will be available forever without them having to take the trouble of vendoring. Your dismissive response? Just vendor. They're happy that clean builds are faster, saving CI time. Your response? It doesn't need to be faster.

Re:Go Module Mirror and Checksum Database Launched by donatj on Friday August 30, 2019 @12:33PM (#20840128)
You're doing the same thing you're criticising, being dismissive of my concern.

> If you don't want to use proxy.golang.org, use goproxy.io

Yes, I can do that, but as I mentioned in my original post having to get an entire team of people and fleet of CI systems to use a non-standard configuration which is not communiciable in the project repo is a minor PITA.

If and when proxy.golang.org has a bad day, builds will fail. Go developers will have a bad day. Having the ability to work around it doesn't make it not a lynchpin.

As for the "FUD about analytics being collected" it was an offhanded quip. I thought it read pretty clearly as not entirely serious hence the prefix "The cynical side of me feels" and not "I feel". It doesn't merit the effort of defending.

Re:Go Module Mirror and Checksum Database Launched by etc_passwd on Friday August 30, 2019 @12:46PM (#20840227)
if you read the blog post and the privacy policy, you’d see that they are collecting analytics and requests in logs, including failed private module lookups.
Re:Go Module Mirror and Checksum Database Launched by steveklabnik on Friday August 30, 2019 @12:22PM (#20840043)
> (If you disagree, go to crates.io and tell me what concerns you have).

We had a number of people express concerns with data tracking on crates.io, including google analytics, GDPR compliance, etc. We've since removed GA and had to talk to lawyers about GDPR, etc.

Re:Go Module Mirror and Checksum Database Launched by nindalf on Friday August 30, 2019 @12:28PM (#20840092)
That's reasonable. I can see how GDPR would be tricky considering that a core feature is prevention of yanking of crates, a right that GDPR guarantees.

I can also see why people might have concerns about Google Analytics. I'm happy that crates.io has removed it.

But this is likely not the point that GP was making. His FUD about analytics was somehow implying that module download statistics could somehow be used for nefarious purposes.

Re:Go Module Mirror and Checksum Database Launched by SheinhardtWigCo on Friday August 30, 2019 @02:05PM (#20840980)
It is naive to suggest that this data can't possibly be used for anything bad. This change results in Google collecting a lot more information about the software that is being created outside of Google. After this change, Go developers working on private projects will end up regularly transmitting their lists of dependencies to Google, unless the risks are properly understood and preventative steps are taken.

Google's business is collecting and monetizing information. It is entirely reasonable and appropriate to express cynicism about new initiatives that result in mass data collection.

Re:Go Module Mirror and Checksum Database Launched by eeZah7Ux on Friday August 30, 2019 @11:57AM (#20839807)
That's a good example of cargo culting.

(Also the downvotes this message is getting. Nobody can dare to question The Google.)

Re:Go Module Mirror and Checksum Database Launched by heyoni on Friday August 30, 2019 @12:13PM (#20839963)
What’s cargo culting exactly? Seriously asking
Re:Go Module Mirror and Checksum Database Launched by JoeAltmaier on Friday August 30, 2019 @12:14PM (#20839971)
Re:Go Module Mirror and Checksum Database Launched by heyoni on Friday August 30, 2019 @12:37PM (#20840157)
Thank you sir!
Re:Go Module Mirror and Checksum Database Launched by elithrar on Friday August 30, 2019 @10:12AM (#20838768)
Paragraphs 3, 4 & 5 of TFA speak directly to the benefits of a module proxy and mirror w.r.t. module availability & performance.
Re:Go Module Mirror and Checksum Database Launched by FiloSottile on Friday August 30, 2019 @10:28AM (#20838939)
And the Checksum Database section of the post explains how it provides security above and beyond a go.sum file. For example, it enables untrusted proxies, guarantees that targeted attacks are not possible, and provides accountability for the operator.

The talk is also an excellent explanation of why proxy and sumdb are valuable. (EDIT: direct link to the recording https://youtu.be/KqTySYYhPUE)

Re:Go Module Mirror and Checksum Database Launched by donatj on Friday August 30, 2019 @10:24AM (#20838888)
Which is what I’m attempting to refute the need for. These are solved and non-problems respectively.
Re:Go Module Mirror and Checksum Database Launched by skybrian on Friday August 30, 2019 @10:38AM (#20839033)
You're saying things work fine for you, but it doesn't follow that they work fine for everyone else. Maybe there are larger Go modules out there than the largest one you've personally built? Or people with slower network access to Github? (Consider China and Australia.) Or maybe they're anticipating that there will be larger Go projects in the future?
Re:Go Module Mirror and Checksum Database Launched by aloknnikhil on Friday August 30, 2019 @10:59AM (#20839246)
Most Go projects are hosted on Github. Github is backed by a global CDN and can handle significantly large projects. I don't think the focus of this project was to address slow network access. It was primarily driven by the need to analyze dependency usage and perhaps add redundancy to some extent.
Re:Go Module Mirror and Checksum Database Launched by FiloSottile on Friday August 30, 2019 @12:28PM (#20840089)
Fetching a module from GitHub often requires cloning the repository, when all you needed was the contents of a specific version, or even just the go.mod to do version resolution. The proxy protocol provides the artifacts you need, for sometimes a 10x bandwidth saving.

(Also, lots of security issues in "go get" came from using git to fetch untrusted repositories.)

Re:Go Module Mirror and Checksum Database Launched by aloknnikhil on Friday August 30, 2019 @02:45PM (#20841369)
Ok that makes sense. Thanks! Also, this sounds like a perfectly neat use case for svn sparse check-out.
Re:Go Module Mirror and Checksum Database Launched by lokar on Friday August 30, 2019 @10:31AM (#20838970)
The point about go.mod only protecting you from changes after you first use it is valid.
Re:Go Module Mirror and Checksum Database Launched by ownagefool on Friday August 30, 2019 @11:44AM (#20839676)
Another reason is what if upstream deletes the package? What if github goes down?

Still, maybe we are better off vendoring packages. At least that would make us consider our use of deps.

Re:Go Module Mirror and Checksum Database Launched by paxys on Friday August 30, 2019 @11:31AM (#20839566)
GitHub is already the central point of failure for the vast majority of the ecosystem, and that is run by a company that has no incentive to care about Go.
Re:Go Module Mirror and Checksum Database Launched by pjmlp on Saturday August 31, 2019 @03:02AM (#20845000)
Guess who supports VS Code Go plugin and has sponsored the work on Go debugging tools like Delve?
Re:Go Module Mirror and Checksum Database Launched by zellyn on Friday August 30, 2019 @11:14AM (#20839394)
I don't have the link handy, but I remember someone posting very impressive speed improvements.

More important though, to those of us writing and/or stewarding Go code at companies that deal with financial information, these kind of safeguards are absolutely necessary.

You can be cynical about Google's motivations, sure, but doing this to track you harder is not consistent with the conversation and approach they've taken with these features.

Re:Go Module Mirror and Checksum Database Launched by secure on Friday August 30, 2019 @11:19AM (#20839449)
That might have been me: https://twitter.com/zekjur/status/1154934952477093888

> Oh, wow! Setting GOPROXY=https://proxy.golang.org results in a 7x speed up for module installation on the hotel WiFi: from 17.7s (no proxy) to 2.3s (http://proxy.golang.org)

Re:Go Module Mirror and Checksum Database Launched by jrockway on Friday August 30, 2019 @10:09AM (#20838739)
My experience from building docker images of Go programs from Github repositories is that Github very aggressively rate limits you. More than one "go build ..." per 10 minutes and it just randomly won't send you the repository contents anymore. So a module proxy was a must. (You could also vendor dependencies, which is my preference, but my team was strongly opposed to vendoring. So we used Athens.)
Re:Go Module Mirror and Checksum Database Launched by donatj on Friday August 30, 2019 @10:22AM (#20838861)
Are you pulling as an authenticated GitHub user? The rate limit for an authenticated user is 5000 pulls per hour, unauthenticated is 60.
Re:Go Module Mirror and Checksum Database Launched by jrockway on Friday August 30, 2019 @11:10AM (#20839357)
No I am not. I don't even know how to make "go build" authenticate to Github. (It's not my code I'm pulling, it's the random libraries my code depends on.)

This is why the proxy exists.

Re:Go Module Mirror and Checksum Database Launched by thefounder on Friday August 30, 2019 @11:44AM (#20839671)
"go build" doesn't fetch the dependencies, does it? Go uses the local settings (i.e .netrc file in your home directory)
Re:Go Module Mirror and Checksum Database Launched by ownagefool on Friday August 30, 2019 @11:53AM (#20839765)
go build will fetch the dependencies if you don't have them. However, they're stored in the $GOPATH, which now defaults to ~/go I believe. Therefore, chances are you already have them.
Re:Go Module Mirror and Checksum Database Launched by nhooyr on Friday August 30, 2019 @11:35AM (#20839606)
If git is authenticated with GitHub for you, go build is as well.
Re:Go Module Mirror and Checksum Database Launched by nhooyr on Friday August 30, 2019 @11:36AM (#20839611)
This has been a problem on CI if your CI doesn't support caching dependencies.
Re:Go Module Mirror and Checksum Database Launched by DelightOne on Friday August 30, 2019 @10:05AM (#20838700)
A package repository can go down, vanish, versions can be deleted.
Re:Go Module Mirror and Checksum Database Launched by donatj on Friday August 30, 2019 @10:08AM (#20838732)
Which as I mentioned is why we have `go mod vendor` if that’s a concern.
Re:Go Module Mirror and Checksum Database Launched by nindalf on Friday August 30, 2019 @11:55AM (#20839787)
Some people don't like vendoring and would prefer keeping their dependencies separate.
Re:Go Module Mirror and Checksum Database Launched by earthboundkid on Friday August 30, 2019 @11:37AM (#20839622)
Marking packages as private in Go.mod is a good idea. Bring it up on the issue tracker.
Re:Go Module Mirror and Checksum Database Launched by SamWhited on Friday August 30, 2019 @11:08PM (#20844415)
Not to mention that I used to be able to get my own metrics (eg. by tailing access logs on the vanity import path server, or checking stats on my version control service or server, etc.) now only Google has that information and module owners are left in the dark about how many times their code is being hit.

They also aren't being very forward thinking. The Go team keeps saying "we promise no Google exec is doing anything nefarious with that" but we don't know who will be the execs next year, or 10 years from now or what they'll do with the power the Go team just gave them by being so short sighted.

Re:Go Module Mirror and Checksum Database Launched by homologate on Friday August 30, 2019 @03:47PM (#20841946)
The default behavior is to fall back to fetching from source control directly if the proxy fails.

https://tip.golang.org/doc/go1.13

Re:Go Module Mirror and Checksum Database Launched by justinclift on Friday August 30, 2019 @10:09AM (#20838742)
> ... this is a way for Google to collect analytics on the Go ecosystem.

Yeah. It's extremely tone deaf of the Go dev team to pull this crap.

Note - setting `export GOPROXY=direct` in your .bashrc (on nix Bash) should* stop the info collection.

Unless a "feature" is added to stop that working too.

Re:Go Module Mirror and Checksum Database Launched by skybrian on Friday August 30, 2019 @10:24AM (#20838885)
Making the feature optional is a pretty normal way of catering to the minority of people who are more risk adverse and won't use something like this. Giving up entirely is not.
Re:Go Module Mirror and Checksum Database Launched by justinclift on Friday August 30, 2019 @12:04PM (#20839862)
Optional yeah, but enabled by default.