News for hackers. Stuff that matters.
Re:Google's robots.txt by oneeyedpigeon on Sunday February 23, 2014 @04:29AM (#7285387)
If you're big enough, doesn't this just make sense? Why waste time maintaining a robots.txt policy when it must represent a tiny fraction of your traffic, which your servers can surely handle? And the really 'bad' guys are going to ignore it anyway. And if you really care, you'll have some much more sophisticated bandwidth throttling in place.

For the smaller guys, sure it makes sense to have some kind of simple robots.txt policy.

Re:Google's robots.txt by keule on Sunday February 23, 2014 @09:18AM (#7285944)
Thats not quite true. Apple has multiple subdomains, each for some part of their site. www.apple.com hosts most of the marketing stuff, but have a look at other pages, e.g.: http://store.apple.com/robots.txt
Re:Google's robots.txt by datawander on Sunday February 23, 2014 @02:25PM (#7287173)
Actually, robot traffic accounts for a much larger share of traffic for any of the big websites than you think.