News for hackers. Stuff that matters.
Re:Wikipedia's robots.txt by mtnGoat on Tuesday July 16, 2019 @06:53PM (#20454660)
I haven't put up a robots.txt in years, they are utterly pointless, IMHO. Even google doesn't honor them(nor did any of the others, like Alexa, for many years now). Read you web server logs, you'll see googlebot crawling pages it should not, if it were honoring your robots.txt.
Re:Wikipedia's robots.txt by spc476 on Tuesday July 16, 2019 @10:22PM (#20455862)
Have you checked the IPs to make sure they're actually Google? There's a strong incentive to fake being Google.
Re:Wikipedia's robots.txt by mtnGoat on Wednesday July 17, 2019 @06:32PM (#20464701)
yes they IPs were owned by them, they even indexed pages they were asked not to. :x