News for hackers. Stuff that matters.

A comment on Perplexity is using stealth, undeclared crawlers to evade no-crawl directives. Parent comment

Re:Perplexity is using stealth, undeclared crawlers to evade... by GardenLetter27 on Monday August 04, 2025 @11:11AM (#44786950)
And isn't the obvious solution to just make some sort of browsers add-on for the LLM summary so the request comes from your browser and then gets sent to the LLM?

I think the main concern here is the huge amount of traffic from crawling just for content for pre-training.

[ Parent | on HN ]
Re:Perplexity is using stealth, undeclared crawlers to evade... by otterley on Monday August 04, 2025 @11:29AM (#44787206)
Why would a personal browser have to crawl fewer pages than the agent’s mechanism? If anything, the agent would be more efficient because it could cache the content for others to use. In the situation we’re talking about, the AI engine is behaving essentially like a caching proxy—just like a CDN.
[ Parent | on HN ]