Re:Applebot, the web crawler for Apple
by karmakaze on Wednesday May 06, 2015 @11:50AM
(#9499526)
Why? Applebot is attempting to adhere to the spirit of robots.txt rather than the letter. If the site owner cares otherwise, don't just explicitly name one bot. Is `user-agent: *bot` valid?
[ Parent
| on HN ]
There are no comments on this comment yet.