Plus, it does seem reasonable to think that only one of the crawlers needs to hit the site. The global replication can happen at the FS level or, heck, the crawlers can just perform pulls from each other.

This is all in the thread that DeVault linked to from his post.

It's OK not to know the specifics of what this is about, but it's weird to have strong opinions about it if you don't.

Yes, there are decent arguments why the golang infra doesn't cache or respect typical norms like robots.txt, but they don't change the unreasonableness of the underlying situation. Surely some mitigation could have been worked out in the year since the ticket was filed?

I think this is just an example of Google being a jerk and not caring enough to do proper software engineering.

Go seems really interesting but I have avoided using it because it’s so tied to Google. And I don’t trust Google to make good decisions for developers or users.

Users who can't wait that long can disable the proxy, and SourceHut can recommend users do that.

https://github.com/golang/go/issues/44577#issuecomment-85693...

A full git clone just to DDOS a hoster to check if the user-experience is still first-class, and filling a proxy is not an acceptable solution for a module hoster who has the pay the hosting bills by himself.

If they want to know if their proxy is still uptodate, a cheap latest change request 8x/hour would be appropriate.

> Have you considered the robots.txt approach, which would simply allow the sysadmin to tune the rate at which you will scrape their service? The best option puts the controls in the hands of the sysadmins you're affecting. This is what the rest of the internet does.

> Also, this probably isn't what you want to hear, but maybe the proxy is a bad idea in the first place. For my part, I use GOPROXY=direct for privacy/cache-breaking reasons, and I have found that many Go projects actually have broken dependencies that are only held up because they're in the Go proxy cache — which is an accident waiting to happen. Privacy concerns, engineering problems like this, and DDoSing hosting providers, this doesn't looks like the best rep sheet for GOPROXY in general.

The issue has nothing to do with DD other than him raising the issue and being ignored. The fault is with google.

It's when we turn this into a morality play that we go off the rails.

Its akin to putting up an open exploitable DNS resolver in 2022 despite, with 2 seconds of research, the entire world telling you not to do that

no one needs to full clone 2 times a minute just to check if a security update exists

one of the top money generating machines does not need a devil's advocate

I get that you feel like you could design a better Go module proxy. It would be weird if you couldn't, because you have the benefit of seeing what happened when we deployed this one†. Congratulations? It's an achievement, I guess?

For my part, I'd be thrilled if just a single person could articulate exactly what the impact to DeVault's code hosting service would be if he took the Go project's offer up on just not having them clone modules hosted on his service so often. Anybody at all, if you could just explain what the problem would be here, I'd be forever grateful.

† (to wit: Go got a lot better, and apparently just 2 source hosts on the entire Internet had a problem with it, one of whom was fixed directly and the other of whom is apparently still mulling whether they want to burn the extra bandwidth for a cacheing benefit that literally nobody on the Internet seems to be able to describe).

You don't consider this a truth, I do. By google's standards it is a truth, https://cloud.google.com/web-risk/docs/caching for one example

> I get that you feel like you could design a better Go module proxy.

I add "sleep 1" at minimum to all my scraping loops, and refuse to use clients that don't keepalive and cache (if I'm hitting the same endpoint), I've done this always, so yes I could have designed a better proxy today, and a year ago, and two years ago, and three years ago, et, al.

> Money generating

They can pay people, I'm not doing it for free for a language I don't use

> if he took the Go project's offer up on just not having them clone modules hosted on his service so often.

Do you know that he hasn't tried? considering how well google responds to emails

Some people have principles that you might not understand, I too would not ask someone to not DDoS me, because I feel that is an absurd request.

So not a impact to the technical aspects but to the administrative aspects. People are paying devault to run a service as devault would, it would be improper for him to not run the service as expected of him.

> burn the extra bandwidth for a cacheing benefit that literally nobody on the Internet seems to be able to describe

bandwidth and CPU time both cost money

- - -

There are alternative approaches that have existed before 1.16, Deno with its pull once then pull only when told to approach, has not had a crisis of security, Every other language works on demand, NPM at most checks a dedicated security issues API designed for mass consumption. These things existed, none of them were referenced.

You can't reasonably build an argument by just making stuff up. Maybe, against all evidence, Google offered to stop hitting sr.ht with Go proxy cache refresh requests and then ignored DeVault when he took them up on it. But I'm not required to assume that very weird situation is what actually happened.

>At Google we were told to stop thinking about all this stuff, that the storage hardware and software people were responsible for hiding things like wearout from application developers.

Something tells me this team was told to "stop thinking about all this stuff, that the network people were responsible for hiding things like speed, latency and cost from application developers." aka network is infinite, keep pounding that repo and we will scale accordingly (our side of the equation, sucks to be other people)

What is the total daily bandwidth that sourcehut uses anyway? What percentage is go module fetching?

I'd be pretty pissed if I hosted a go module essentially for myself and suddenly I have a $200 dollar bill, because google decided to clone my repository 500 times a day. If it doesn't bother you, how about you donate $200 a month to a charity of my choosing, because it doesn't matter to you.

If money was a problem, I'd expect this individual to ha e rectified it on their end

Seriously do you follow through what your arguments actually mean if applied in general?

Should every language be responsible for paying the bandwidth bills for dependencies?

You might look at the most recent comment from the Go team on the issue, there have been no additional requests or events since they last resolved it for both of the effected parties

I don't really understand turning this around on him. Why should he have to subsidize Google? If it's not a problem, why do we have robots.txt at all? Just let bots hammer your site and cope with it.

The current situation can't be the optimal solution. It wasn't even present prior to Go 1.16. Only one company has the ability to change that. What should he do differently here? Why should he have to spend any of his time or money working around an issue he didn't create?

Google should be sending reimbursement checks for the damage done here on this issue.

I suspect he's drawn his line in the sand and wants to keep it going rather than finding a solution that works without requiring upstream changes.

Read: we prefer to use your servers for caching. Not good enough. Maybe the issue is people making silly evasive arguments like these while the server load piles on?