Thats not quite true. Apple has multiple subdomains, each for some part of their site. www.apple.com hosts most of the marketing stuff, but have a look at other pages, e.g.: http://store.apple.com/robots.txt
Re:Google's robots.txt
by datawander on Sunday February 23, 2014 @02:25PM
(#7287173)
Actually, robot traffic accounts for a much larger share of traffic for any of the big websites than you think.
For the smaller guys, sure it makes sense to have some kind of simple robots.txt policy.