News for hackers. Stuff that matters.
Re:Applebot, the web crawler for Apple by gutnor on Wednesday May 06, 2015 @06:05AM (#9497569)
That's maybe jumping to conclusion.

I interpreted the sentence less literally and more like "in absence of rule, default to GoogleBot ones"

So if you put a wildcard rule forbidding access and a specific one allowing access to GoogleBot, AppleBot will honour the wildcard one.

That's how I would have coded it anyway: parse the rule for current agent string, if no rule applies, run it again with GoogleBot one before assuming that website does not contain restrictions.

Re:Applebot, the web crawler for Apple by koyote on Wednesday May 06, 2015 @06:50AM (#9497697)
"If robots instructions don't mention Applebot"

I would assume this means that it will follow GoogleBot unless you specifically mention AppleBot by name and not by using a wildcard.

So a User-agent: * would be ignored if a User-agent: GoogleBot is found.