News for hackers. Stuff that matters.
Re:Google’s Sergey Brin: China, SOPA, Facebook Threaten the ... by jlarocco on Monday April 16, 2012 @01:39AM (#3846068)
Not really.

robots.txt has always been based on a good faith.

All the reputable companies abide by it, but I don't doubt for a second they'd stop reading it if it started affecting their bottom line.

Re:Google’s Sergey Brin: China, SOPA, Facebook Threaten the ... by ethank on Monday April 16, 2012 @02:09AM (#3846143)
You can also actively reject the GoogleBot
Re:Google’s Sergey Brin: China, SOPA, Facebook Threaten the ... by jlarocco on Monday April 16, 2012 @02:14AM (#3846154)
True, but GoogleBot's user agent isn't set in stone.