News for hackers. Stuff that matters.
Re:Google Now Fills Out Forms & Crawls Results by thorax on Friday April 11, 2008 @07:41PM (#161491)
Seems strange that you wouldn't have just looked when asking that question (less characters to type at least). Lots of things disallowed.

http://google.com/robots.txt

Re:Google Now Fills Out Forms & Crawls Results by Hexstream on Friday April 11, 2008 @11:07PM (#161597)
I think it was a rhetorical question.
Re:Google Now Fills Out Forms & Crawls Results by michaelneale on Sunday April 13, 2008 @12:45AM (#162220)
Yes it was. But I should have been clearer for the type of literal argument starters that would tend to hang around here ;)

I wonder if google always honour robots.txt

Re:Google Now Fills Out Forms & Crawls Results by Hexstream on Sunday April 13, 2008 @09:46AM (#162396)
I hope so because otherwise they could get stuck in one of those:

http://en.wikipedia.org/wiki/Spider_trap