Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on Perplexity is using stealth, undeclared crawlers to evade no-crawl directives. Parent comment
That is true. But robots.txt is not designed to give them the ability to prevent this.
That is true. But robots.txt is not designed to give them the ability to prevent this.