Sourcehut will blacklist the Go module mirror

Re:Sourcehut will blacklist the Go module mirror by kevincox on Monday January 09, 2023 @03:33PM (#34315599)

When I mean crawler I mean something that discovers new pages. Refreshing the same URL isn't really crawling.

But yes, it may be the best available solution in this case, even if I would argue that it isn't really it's main purpose.

[ Parent | on HN ]

Re:Sourcehut will blacklist the Go module mirror by cmatthias on Monday January 09, 2023 @03:42PM (#34315729)

After reading this and your response to a sibling comment I wholeheartedly disagree with you on both the specific definition of the word crawler and what the "main purpose" of robots.txt is, but glad we can agree that Google should be doing more to respect rate limits :)

[ Parent | on HN ]

Re:Sourcehut will blacklist the Go module mirror by ddevault on Monday January 09, 2023 @03:52PM (#34315859)

What you're thinking about, in my opinion, is best referred to as a spider.

[ Parent | on HN ]