Re:Why is Google so hysterically hypocritical about Bing usi...
by FreebytesSector on Wednesday February 02, 2011 @02:37PM
(#2171721)
I agree with your comments that toolbars should respect the robots.txt because even if a human is doing the crawling, it is still an automated system that is indexing information from that site. I would not want toolbars attempting to send data back to Bing based on my queries on a company Intranet or a site that would normally not be indexed. Personal data entered into what the toolbar thought was a query field could be sent onward as well even if the robots.txt on the site restricted it. I think they should respect robots.txt in this case even if they are only monitoring user behavior.
[ Parent
| on HN ]
There are no comments on this comment yet.