News for hackers. Stuff that matters.

A comment on Google's robots.txt. Parent comment

Re:Google's robots.txt by yeukhon on Sunday February 23, 2014 @02:34AM (#7285250)
What is the point of having such silly prohibition? It's silly because anyone can crawl it if they want, Facebook may block such DDoS attack, but why would they bother to put up such sign when they know it's useless?
[ Parent | on HN ]
Re:Google's robots.txt by MichaelApproved on Sunday February 23, 2014 @07:16AM (#7285702)
They have such a large network, anything they could do to prevent unwanted crawling is probably helpful.
[ Parent | on HN ]
Re:Google's robots.txt by sfall on Sunday February 23, 2014 @05:32AM (#7285501)
my guess lawyers
[ Parent | on HN ]