News for hackers. Stuff that matters.

A comment on Perplexity is using stealth, undeclared crawlers to evade no-crawl directives. Parent comment

Re:Perplexity is using stealth, undeclared crawlers to evade... by cjonas on Monday August 04, 2025 @10:29AM (#44786313)
Doesn't perplexity crawl to harvest and index data like a traditional search engine? Or is it all "on demand"?
[ Parent | on HN ]
Re:Perplexity is using stealth, undeclared crawlers to evade... by lukeschlather on Monday August 04, 2025 @10:47AM (#44786570)
For the most part I would assume they pay for access to Google or Bing's index. I also assume they don't really train models. So all their "crawling" is on behalf of users.
[ Parent | on HN ]