Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on Perplexity is using stealth, undeclared crawlers to evade no-crawl directives. Parent comment
There is an user agent for search that you can control in robots.txt.
user-agent: Googlebot
user-agent: Google-Extended
There is an user agent for search that you can control in robots.txt.
There is another user agent for AI training.