Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on Applebot, the web crawler for Apple. Parent comment
I would assume this means that it will follow GoogleBot unless you specifically mention AppleBot by name and not by using a wildcard.
So a User-agent: * would be ignored if a User-agent: GoogleBot is found.
There are no comments on this comment yet.
I would assume this means that it will follow GoogleBot unless you specifically mention AppleBot by name and not by using a wildcard.
So a User-agent: * would be ignored if a User-agent: GoogleBot is found.