News for hackers. Stuff that matters.
Re:Applebot, the web crawler for Apple by karmakaze on Wednesday May 06, 2015 @11:50AM (#9499526)
Why? Applebot is attempting to adhere to the spirit of robots.txt rather than the letter. If the site owner cares otherwise, don't just explicitly name one bot. Is `user-agent: *bot` valid?

There are no comments on this comment yet.