News for hackers. Stuff that matters.

A comment on Perplexity is using stealth, undeclared crawlers to evade no-crawl directives. Parent comment

Re:Perplexity is using stealth, undeclared crawlers to evade... by danlitt on Tuesday August 05, 2025 @03:10AM (#44795137)
You need more qualifiers for this to be true. archive.org routinely archives content that the site operators would prefer to be taken down.
[ Parent | on HN ]

There are no comments on this comment yet.