News for hackers. Stuff that matters.
Re:Ask HN: Has Google/Bing won the search engine war? by timr on Monday March 31, 2014 @02:34PM (#7503264)
It wouldn't work. The crawlers that cause the problems are not the ones that pay attention to voluntary management systems like robots.txt.

There are no comments on this comment yet.