News for hackers. Stuff that matters.
Re:Big Tech Has Become Too Powerful by _delirium on Saturday September 19, 2015 @02:25PM (#10245101)
Few deliberately block Bing specifically, it's just a sloppy attempt to block unwanted crawlers, like the billions of crawlers running on AWS and DigitalOcean boxes. A quick-but-not-great hack is to have an entry blocking * by default, and then specifically whitelist GoogleBot to avoid getting delisted from Google. You could also whitelist other search engines, but that requires you to know and care about them, and for a long period Google was the only engine people really cared about, since it drove >95% of organic search traffic. I do feel we're past the peak of that period, however, and site owners are now getting more cognizant of non-Google search engines having at least some value in driving traffic.

Related, "Please Don't Block Everything but Googlebot in robots.txt": http://danluu.com/googlebot-monopoly/

Re:Big Tech Has Become Too Powerful by caseysoftware on Saturday September 19, 2015 @02:32PM (#10245121)
So it's more out of ignorance than maliciousness. That makes sense.

Odds are the more troublesome scrapers are probably ignoring robots.txt anyway, so I wonder how useful it is.