Re:Only Google is really allowed to crawl the web
by quantumofalpha on Friday March 26, 2021 @04:59PM
(#26596866)
if you're claiming to be User-Agent: Googlebot, but your IP doesn't seem like it belongs to Google, don't you think it's a clear sign that you're FAKING IT?
The check itself could be implemented for example with ASN or reverse DNS lookup or hard-coding known Google's IP ranges (though that's prone to become stale)
The check itself could be implemented for example with ASN or reverse DNS lookup or hard-coding known Google's IP ranges (though that's prone to become stale)