Re:A surprising opinion on the ‘Bing is copying Google’ cont...
by beoba on Friday February 04, 2011 @11:59AM
(#2180095)
It's for controlling search results; blocking overzealous crawlers is a convenient side-effect.
robots.txt also blocks indexing of non-html files.
Re:A surprising opinion on the ‘Bing is copying Google’ cont...
by InclinedPlane on Friday February 04, 2011 @12:28PM
(#2180227)
rel="nofollow" is only useful for pages which are intended to be indexed, google's search results pages are not, which is why they don't use it for the links in search results.
The point is: up until now search indexing has had a robust opt-out mechanism. Bing has changed that and has muddied the waters with vague hand-waving. The question still remains: should there still be a way for sites to opt-out of search indexing? If so, then why can't robots.txt continue to be that mechanism? If robots.txt can't be that mechanism then what should be the new standard. If not, then that opens an entirely new can of worms. A can that I think we're better off not opening.
If you want to control what a search engine does with a page, you should use <meta name="ROBOTS"> and rel=nofollow, neither of which Google includes in its results pages. It would be reasonable for Bing to look for and honor these directives when collecting their clickstream data, and Google isn't availing itself of the option.