Re:Only Google is really allowed to crawl the web
by jedberg on Friday March 26, 2021 @06:33PM
(#26597819)
> Don't do this if you actually rely on search engine traffic though. Google may get pissed and send you lots of angry mail like "There's a problem with your site".
Ah, but of course you would exclude Google's published crawler IPs from this restriction, because that is exactly what they want you to do.
Ah, but of course you would exclude Google's published crawler IPs from this restriction, because that is exactly what they want you to do.