News for hackers. Stuff that matters.

A comment on Google's robots.txt. Parent comment

Re:Google's robots.txt by MichaelApproved on Sunday February 23, 2014 @07:16AM (#7285702)
They have such a large network, anything they could do to prevent unwanted crawling is probably helpful.
[ Parent | on HN ]

There are no comments on this comment yet.