News for hackers. Stuff that matters.

A comment on Perplexity is using stealth, undeclared crawlers to evade no-crawl directives. Parent comment

Re:Perplexity is using stealth, undeclared crawlers to evade... by cjonas on Monday August 04, 2025 @10:25AM (#44786258)
Like for people or are using a ad block or for a crawler downloading your content so it can be used by an AI response?
[ Parent | on HN ]
Re:Perplexity is using stealth, undeclared crawlers to evade... by Bjartr on Monday August 04, 2025 @11:48AM (#44787506)
Arbitrarily, as in for any reason. It's your site, you decide what constraints an incoming request must meet for it to get a response containing the content of your site.
[ Parent | on HN ]