News for hackers. Stuff that matters.
Re:Google’s Sergey Brin: China, SOPA, Facebook Threaten the ... by ethank on Monday April 16, 2012 @01:02AM (#3845980)
To Google:

User-agent: *

Disallow: /

threatens their ability to index and serve ads against the open web, which is really what they mean.

Re:Google’s Sergey Brin: China, SOPA, Facebook Threaten the ... by jlarocco on Monday April 16, 2012 @01:39AM (#3846068)
Not really.

robots.txt has always been based on a good faith.

All the reputable companies abide by it, but I don't doubt for a second they'd stop reading it if it started affecting their bottom line.

Re:Google’s Sergey Brin: China, SOPA, Facebook Threaten the ... by ethank on Monday April 16, 2012 @02:09AM (#3846143)
You can also actively reject the GoogleBot
Re:Google’s Sergey Brin: China, SOPA, Facebook Threaten the ... by jlarocco on Monday April 16, 2012 @02:14AM (#3846154)
True, but GoogleBot's user agent isn't set in stone.