Re:Big Tech Has Become Too Powerful
by caseysoftware on Saturday September 19, 2015 @02:32PM
(#10245121)
So it's more out of ignorance than maliciousness. That makes sense.
Odds are the more troublesome scrapers are probably ignoring robots.txt anyway, so I wonder how useful it is.
Related, "Please Don't Block Everything but Googlebot in robots.txt": http://danluu.com/googlebot-monopoly/