Re:Sourcehut will blacklist the Go module mirror
by jslql on Monday January 09, 2023 @02:39PM
(#34314932)
Why should a git client respect an http standard such as robots.txt?
[ Parent
| on HN ]
Re:Sourcehut will blacklist the Go module mirror
by cmatthias on Monday January 09, 2023 @02:45PM
(#34314991)
Read the OP; it's obvious based on the references to robots.txt, the User-Agent header, returning a 429 response, etc, that most (all?) of Google's requests are doing git clones over http(s).
Re:Sourcehut will blacklist the Go module mirror
by yamtaddle on Monday January 09, 2023 @02:45PM
(#34315004)
Google began pushing for it to become an Internet standard—explicitly to be applicable to any URI-driven Internet system, not just the Web—in 2019, and it was adopted as an Internet standard in 2022.
Re:Sourcehut will blacklist the Go module mirror
by cmatthias on Tuesday January 10, 2023 @09:48AM
(#34325628)
This is true but irrelevant to the parent's question -- in the article, it's made clear that Google's requests are happening over HTTP, which is the most obvious reason why robots.txt should be respected.
Re:Sourcehut will blacklist the Go module mirror
by yamtaddle on Tuesday January 10, 2023 @10:04AM
(#34325875)
It's relevant because it attacks the premise of their objection.
Re:Sourcehut will blacklist the Go module mirror
by trulyrandom on Monday January 09, 2023 @02:47PM
(#34315018)
Because it uses HTTP.