News for hackers. Stuff that matters.
Re:A surprising opinion on the ‘Bing is copying Google’ cont... by beoba on Friday February 04, 2011 @11:59AM (#2180095)
It's for controlling search results; blocking overzealous crawlers is a convenient side-effect.

robots.txt also blocks indexing of non-html files.

There are no comments on this comment yet.