Re:Wikipedia's robots.txt
by mtnGoat on Tuesday July 16, 2019 @06:53PM
(#20454660)
I haven't put up a robots.txt in years, they are utterly pointless, IMHO. Even google doesn't honor them(nor did any of the others, like Alexa, for many years now). Read you web server logs, you'll see googlebot crawling pages it should not, if it were honoring your robots.txt.
[ Parent
| on HN ]
Have you checked the IPs to make sure they're actually Google? There's a strong incentive to fake being Google.