News for hackers. Stuff that matters.
Re:Applebot, the web crawler for Apple by koyote on Wednesday May 06, 2015 @06:50AM (#9497697)
"If robots instructions don't mention Applebot"

I would assume this means that it will follow GoogleBot unless you specifically mention AppleBot by name and not by using a wildcard.

So a User-agent: * would be ignored if a User-agent: GoogleBot is found.

There are no comments on this comment yet.