The relevant first link clearly lays out an implementation problem. It's not just git.sr.ht that's facing it, but another user also comes in to point out a tremendous amount of traffic. An amount of traffic that is fairly irresponsible to say the least.
The issue owner doesn't deny the problem. They simply say it requires work. It appears they are unwilling to do the work to resolve the unsavory behavior, and instead are asking the host of Go modules to disable the ability for their service to be used as a Go module host, or instead suck it up and deal with the cost and complications of Google not putting in the effort to fix their architecture's DDoSish behavior.
Google and Go have the market power to pull this off, but let's not pretend this isn't bad behavior. An appeal to Drew's terrible personal communication style does not change that.
It's relevant to the issue of the ban, which is what the (sub)thread is about. I think the reason it was posted was to demonstrate a pattern, and I think that's pretty relevant.
Every single disagreement I've had with Drew escalated and I don't think that's my fault since it never happens with anyone else (Never? Well, hardly ever) and I've seen it happen with various other people too. Not that I'm perfect by any means or couldn't have done things better, but there's certainly a pattern here.
As for the actual issue at hand: I mostly agree with Drew, however, I don't really have enough information to be sure here, and Drew has misrepresented things in the past and does so here in this post (a reader unfamiliar with Go would be left assuming that Go "phones home" just for the sake of "phoning home" after reading this article, which is really a misrepresentation IMHO), so there's not a lot of trust here (again, a pattern).
So does or does not the problem persist? Second was or was he not banned from the commenting issue tracker. Third does the CoC require that a person gets notified by the moderator and was DeVault notified?
If the answers are yes to all those problems I wonder who is makeing bad faith arguments?
Note: I have absolutely no skin in this game, except for being a sway and gmail user.
> Dealing with DeVault's bad faith arguments is one of the few things I won't miss of that job.
They didn't say bad faith argument about banning, they said argument_s_. So he is not just talking about a single one. Which are the bad faith arguments? I asked if any of the things were not true, nobody said they were untrue. How can any of the arguments (I did not just talk about the proxy, I also talked about the banning) be bad faith if they are true?
Not sure if you realize the absurdity of this, but he has to pay traffic and server costs. Like everyone else, except probably Google as it seems!?
I mean, you didn't even consider implementing a simple fetch of an already cloned repository in your mirroring server code. So yeah, I'd argue that the bad faith part is actually justified.
https://github.com/golang/go/issues/44577#issuecomment-11378...
> We did consider caching clones, but it has security implications and adds complexity, so we decided not to. It is certainly not trivial to do and not something we are likely to do based on this issue.
Drew continues to act as though he is always correct, and any viewpoint that isn't his is just moronic. I've repeatedly seen this behavior from him in multiple venues over the years, and I'm happy to see the wider community start calling this out as childish.
Plus, it does seem reasonable to think that only one of the crawlers needs to hit the site. The global replication can happen at the FS level or, heck, the crawlers can just perform pulls from each other.
This is all in the thread that DeVault linked to from his post.
It's OK not to know the specifics of what this is about, but it's weird to have strong opinions about it if you don't.
Yes, there are decent arguments why the golang infra doesn't cache or respect typical norms like robots.txt, but they don't change the unreasonableness of the underlying situation. Surely some mitigation could have been worked out in the year since the ticket was filed?
I think this is just an example of Google being a jerk and not caring enough to do proper software engineering.
Go seems really interesting but I have avoided using it because it’s so tied to Google. And I don’t trust Google to make good decisions for developers or users.
Users who can't wait that long can disable the proxy, and SourceHut can recommend users do that.
https://github.com/golang/go/issues/44577#issuecomment-85693...
A full git clone just to DDOS a hoster to check if the user-experience is still first-class, and filling a proxy is not an acceptable solution for a module hoster who has the pay the hosting bills by himself.
If they want to know if their proxy is still uptodate, a cheap latest change request 8x/hour would be appropriate.
> Have you considered the robots.txt approach, which would simply allow the sysadmin to tune the rate at which you will scrape their service? The best option puts the controls in the hands of the sysadmins you're affecting. This is what the rest of the internet does.
> Also, this probably isn't what you want to hear, but maybe the proxy is a bad idea in the first place. For my part, I use GOPROXY=direct for privacy/cache-breaking reasons, and I have found that many Go projects actually have broken dependencies that are only held up because they're in the Go proxy cache — which is an accident waiting to happen. Privacy concerns, engineering problems like this, and DDoSing hosting providers, this doesn't looks like the best rep sheet for GOPROXY in general.
The issue has nothing to do with DD other than him raising the issue and being ignored. The fault is with google.
It's when we turn this into a morality play that we go off the rails.
Its akin to putting up an open exploitable DNS resolver in 2022 despite, with 2 seconds of research, the entire world telling you not to do that
no one needs to full clone 2 times a minute just to check if a security update exists
one of the top money generating machines does not need a devil's advocate
I get that you feel like you could design a better Go module proxy. It would be weird if you couldn't, because you have the benefit of seeing what happened when we deployed this one†. Congratulations? It's an achievement, I guess?
For my part, I'd be thrilled if just a single person could articulate exactly what the impact to DeVault's code hosting service would be if he took the Go project's offer up on just not having them clone modules hosted on his service so often. Anybody at all, if you could just explain what the problem would be here, I'd be forever grateful.
† (to wit: Go got a lot better, and apparently just 2 source hosts on the entire Internet had a problem with it, one of whom was fixed directly and the other of whom is apparently still mulling whether they want to burn the extra bandwidth for a cacheing benefit that literally nobody on the Internet seems to be able to describe).
You don't consider this a truth, I do. By google's standards it is a truth, https://cloud.google.com/web-risk/docs/caching for one example
> I get that you feel like you could design a better Go module proxy.
I add "sleep 1" at minimum to all my scraping loops, and refuse to use clients that don't keepalive and cache (if I'm hitting the same endpoint), I've done this always, so yes I could have designed a better proxy today, and a year ago, and two years ago, and three years ago, et, al.
> Money generating
They can pay people, I'm not doing it for free for a language I don't use
> if he took the Go project's offer up on just not having them clone modules hosted on his service so often.
Do you know that he hasn't tried? considering how well google responds to emails
Some people have principles that you might not understand, I too would not ask someone to not DDoS me, because I feel that is an absurd request.
So not a impact to the technical aspects but to the administrative aspects. People are paying devault to run a service as devault would, it would be improper for him to not run the service as expected of him.
> burn the extra bandwidth for a cacheing benefit that literally nobody on the Internet seems to be able to describe
bandwidth and CPU time both cost money
- - -
There are alternative approaches that have existed before 1.16, Deno with its pull once then pull only when told to approach, has not had a crisis of security, Every other language works on demand, NPM at most checks a dedicated security issues API designed for mass consumption. These things existed, none of them were referenced.
You can't reasonably build an argument by just making stuff up. Maybe, against all evidence, Google offered to stop hitting sr.ht with Go proxy cache refresh requests and then ignored DeVault when he took them up on it. But I'm not required to assume that very weird situation is what actually happened.
>At Google we were told to stop thinking about all this stuff, that the storage hardware and software people were responsible for hiding things like wearout from application developers.
Something tells me this team was told to "stop thinking about all this stuff, that the network people were responsible for hiding things like speed, latency and cost from application developers." aka network is infinite, keep pounding that repo and we will scale accordingly (our side of the equation, sucks to be other people)
What is the total daily bandwidth that sourcehut uses anyway? What percentage is go module fetching?
I'd be pretty pissed if I hosted a go module essentially for myself and suddenly I have a $200 dollar bill, because google decided to clone my repository 500 times a day. If it doesn't bother you, how about you donate $200 a month to a charity of my choosing, because it doesn't matter to you.
If money was a problem, I'd expect this individual to ha e rectified it on their end
Seriously do you follow through what your arguments actually mean if applied in general?
Should every language be responsible for paying the bandwidth bills for dependencies?
You might look at the most recent comment from the Go team on the issue, there have been no additional requests or events since they last resolved it for both of the effected parties
I don't really understand turning this around on him. Why should he have to subsidize Google? If it's not a problem, why do we have robots.txt at all? Just let bots hammer your site and cope with it.
The current situation can't be the optimal solution. It wasn't even present prior to Go 1.16. Only one company has the ability to change that. What should he do differently here? Why should he have to spend any of his time or money working around an issue he didn't create?
Google should be sending reimbursement checks for the damage done here on this issue.
I suspect he's drawn his line in the sand and wants to keep it going rather than finding a solution that works without requiring upstream changes.
Read: we prefer to use your servers for caching. Not good enough. Maybe the issue is people making silly evasive arguments like these while the server load piles on?
"Assuming everyone else have exactly same design choice and architecture as yourself, making suggestion on this ground and calling other people crazy because they can't implement what you suggest them to do" is not trying to be helpful.
Well, or maybe I'm just frustrated reading his repeated "please keep a copy 'locally' somewhere and run git fetch". Just like how I'm frustrated arguing with him on HN about whether sending patch to mail list is better than GitHub pull request.
Don't get me wrong, I understand this is a real Google-scale system v.s. individual code hosting website issue, I just don't see how his "please stop being Google and instead try my works-fine-on-one-box solution" take is helpful.
It's pretty clear that the issue owner agrees there is something wrong here. That's why the issue is kept open and their only reason for not fixing it is that it's a complicated fix which requires effort.
As a workaround, they offer Drew the opportunity to essentially disable his service from being used as a useful source of Go modules.
This is a scenario where anyone who isn't a Googler can see that "being Google" is not a justification for bad engineering. Do better.
Do you have links to those ad spams? I couldn't find obvious ones when I was using GitHub's issue search field.
Edit: It seems ddevault opened and/or replied to only 10 issues, which I would hardly call "spam".
If there's any entity I'm totally ok with anyone spamming with ads, it's Google.
He complains about outgoing traffic of 4GB/day. On a server. In 2022.
If this is an issue he should not be hosting Go modules on his own servers.
I really really don't get the issue.
Is he io bound? Is there no reasonable way for git hosting services to actually cache git checkouts?
DDoS / DoS should only be used, in my opinion, when the server goes down complelty not if it has a little bit of load and i still haven't seen anything which indicates that this is a real issue?
And i don't want to come across negativly, i just don't get the issue.
What if your website hosted 100 Go modules and it scales linearly, that's 400GB, still not a problem? What if every programming language did that?
I don't really understand how that's not an issue. Sure it's not a big deal, if just one service does that, but it's still something that should be fixed. If you are Google it should be expected that this fix is not taking over a year.
It seems like the only solution suggested there is one that makes the "small fish" service less useful as a go repository. I'm not surprised he didn't like it.
And to be frank, filippo's contribution to this thread places a clear reason why- drew was incredibly rude in the second link, but his arguments were not bad faith. He clearly meant everything he said, and I would even argue that he was right, although his method of engagement was unacceptable.
More importantly, the exchange was 3 years ago, and the more recent exchange showed him engaging exactly how I would expect a professional to. If the 3 year old exchange is the best filippo could find, it indicates that drew has changed his tactics for the better. So why the ban?
> Have you considered the robots.txt approach, which would simply allow the sysadmin to tune the rate at which you will scrape their service? The best option puts the controls in the hands of the sysadmins you're affecting. This is what the rest of the internet does.
The only explanation I see later in the thread is:
> For boring technical reasons, it would be a fair bit of extra work for us to read robots.txt, so rather than going to a bunch of work to do that, we implemented a trivial list and offered to add sr.ht to it.
This is...not a satisfactory technical explanation. Perhaps the Go team should consider providing more openness and transparency about why it's "a fair bit of extra work" to implement an internet standard like DeVault suggests.
Honestly, DeVault is an abrasive person and not necessarily someone I would go out of my way to work with, but I don't see how the Golang team isn't at least somewhat at fault here for brushing off a community member with "Eh, it's too complicated for you to understand". That's not how you build a thriving community around a language.
I disagree, the "for boring technical reasons..." is as close to a brush-off as I can see. This is a technical issue tracker, why not be open and honest about the reasons? I feel like these days people just seem to take it on faith that "Oh, it's Google, surely they know best when they say it's a mysterious technical issue that's too hard to solve".
Also, banning someone from the issue tracker in violation of their own CoC does not seem like good faith behavior either.
Note that the message from 'FiloSottile does not clearly spell out whether DeVault's ban was following a CoC, it just obliquely quotes some nasty stuff that he's apparently said in the past. I think other commenters are correct to call it irrelevant, it's about as relevant as it would be if I started randomly quoting Rob Pike's notorious comments about syntax highlighting [1].
Overall, I totally acknowledge the Golang team is providing a workaround that will solve the (admittedly abrasively spoken) user's problems. What I'm saying is that that is not enough. If you want an open language that will not turn into .NET, you have to do better by your community and also be transparent as to how and when you're planning to solve your users' problems. And preferably also be transparent about CoC bans, especially if there is a technical discussion involved.
For example, this sounds like a reasonable response to me:
"We have written up a task to make our proxy code parse robots.txt correctly, but there are N other tasks above it, and it looks like we won't be able to work on it for approximately the next six months. Until then, a workaround is..."
----------------------------------------
[1] https://groups.google.com/g/golang-nuts/c/hJHCAaiL0so/m/E2mQ....
There's no indication I can see that DeVault's "banning" from the issue tracker has anything to do with any of this.
There's always going to be some standard the Go team (or the Rust team, or the Clojure team, or the Scala team, or any other team down the line until you just decide to do your own language and hope for the day you get to occupy the gratifying job of fielding these kinds of tickets) doesn't meet, for someone on some message board somewhere.
If the Go team's workaround was to make sr.ht not viable for Go modules, that would be deeply problematic. But it isn't; in fact, nobody who's upset about this seems to be able to describe what the Go team's workaround would even impact. Can you?
I know is not like knowing what Google thinks about this, but I'm curious about how something like what is described in the post is allowed to happen.
Speaking of bad faith arguments, aren't Google and its employees the ones that:
- Claimed that scrolling screenshot on Android are "infeasible" [INFEAS] despite Samsung, LG, HTC, etc. already implementing the feature in their forks/distributions of Android?
- Claimed "[t]he generic dilemma is this: do you want slow programmers, slow compilers and bloated binaries, or slow execution times?" as an excuse for not implementing Generics in Go?
- After years of claiming this nonsense finally implemented Generics in exactly those ways? [GoLang_GenImpl]
- Even when implementing Generics in exactly those old and previously known ways claim that "Russ Cox famously observed that generics require choosing among slow programmers, slow compilers, or slow execution times. We believe that this design permits different implementation choices." [GoLang_43651]
- "Loses" or "loses track" of their customer's mobile phones, and when said customers cancel the credit charges for the phones they never received or otherwise disappeared, block their Google accounts, including their access to email? [Google_Criminals]
Drew DeVault certainly deserves some or even a lot of criticism, but Google and its employees attacking other for "bad faith"? This should be acceptable in a civilized society.
[GoLang_43651]: https://go.googlesource.com/proposal/+/refs/heads/master/des...
[Google_Criminals]: For example: https://www.reddit.com/r/GooglePixel/comments/7nrx07/google_... and https://www.reddit.com/r/GooglePixel/comments/84sysx/update_... There have been other documented cases of the same so this is not a single uncharacteristic incident.
[GoLang_GenImpl]: https://go.googlesource.com/proposal/+/e0113ba8479092562cf9d...
[INFEAS]: https://issuetracker.google.com/issues/80491647#:~:text=Stat...
https://github.com/golang/go/issues/44577#issuecomment-85692...
> "EFAIL" is an alarmist puff piece written by morons to slander PGP and inflate their egos. [...]
https://github.com/golang/go/issues/30141#issuecomment-46427...
Disclosure: I was on the Go team at Google until earlier this month. Dealing with DeVault's bad faith arguments is one of the few things I won't miss of that job.