News for hackers. Stuff that matters.

A comment on Perplexity is using stealth, undeclared crawlers to evade no-crawl directives. Parent comment

Re:Perplexity is using stealth, undeclared crawlers to evade... by hombre_fatal on Monday August 04, 2025 @10:21AM (#44786191)
TFA isn’t talking about crawling to harvest training data.

It’s talking about Perplexity crawling sites on demand in response to user queries and then complaining that no it’s not fine, hence this thread.

[ Parent | on HN ]
Re:Perplexity is using stealth, undeclared crawlers to evade... by cjonas on Monday August 04, 2025 @10:29AM (#44786313)
Doesn't perplexity crawl to harvest and index data like a traditional search engine? Or is it all "on demand"?
[ Parent | on HN ]
Re:Perplexity is using stealth, undeclared crawlers to evade... by lukeschlather on Monday August 04, 2025 @10:47AM (#44786570)
For the most part I would assume they pay for access to Google or Bing's index. I also assume they don't really train models. So all their "crawling" is on behalf of users.
[ Parent | on HN ]