Re:Only Google is really allowed to crawl the web
by tedunangst on Friday March 26, 2021 @01:57PM
(#26594994)
I don't whitelist googlebot, but I don't block them either because their crawler is fairly slow and unobtrusive. Other crawlers seem determined to download the entire site in 60 seconds, and then download it again, and again, until they get banned.
[ Parent
| on HN ]
There are no comments on this comment yet.