News for hackers. Stuff that matters.
Re:Google Now Fills Out Forms & Crawls Results by michaelneale on Friday April 11, 2008 @06:41PM (#161444)
So its ok for them to do this? but if you do the same against google (scraping results) - presumably Bad Things happen?
Re:Google Now Fills Out Forms & Crawls Results by cstejerean on Friday April 11, 2008 @07:19PM (#161476)
Is their search url listed in their robots.txt?
Re:Google Now Fills Out Forms & Crawls Results by thorax on Friday April 11, 2008 @07:41PM (#161491)
Seems strange that you wouldn't have just looked when asking that question (less characters to type at least). Lots of things disallowed.

http://google.com/robots.txt

Re:Google Now Fills Out Forms & Crawls Results by Hexstream on Friday April 11, 2008 @11:07PM (#161597)
I think it was a rhetorical question.
Re:Google Now Fills Out Forms & Crawls Results by michaelneale on Sunday April 13, 2008 @12:45AM (#162220)
Yes it was. But I should have been clearer for the type of literal argument starters that would tend to hang around here ;)

I wonder if google always honour robots.txt

Re:Google Now Fills Out Forms & Crawls Results by Hexstream on Sunday April 13, 2008 @09:46AM (#162396)
I hope so because otherwise they could get stuck in one of those:

http://en.wikipedia.org/wiki/Spider_trap