Re:Google’s Sergey Brin: China, SOPA, Facebook Threaten the ...
by jlarocco on Monday April 16, 2012 @01:39AM
(#3846068)
Not really.
robots.txt has always been based on a good faith.
All the reputable companies abide by it, but I don't doubt for a second they'd stop reading it if it started affecting their bottom line.
User-agent: *
Disallow: /
threatens their ability to index and serve ads against the open web, which is really what they mean.