News for hackers. Stuff that matters.
Re:Perplexity is using stealth, undeclared crawlers to evade... by fluidcruft on Monday August 04, 2025 @10:25AM (#44786266)
If the AI archives/caches all the results it accesses and enough people use it, doesn't it become a scraper? Just learn off the cached data. Being the man-in-the-middle seems like a pretty easy way to scrape salient content while also getting signals about that content's value.
Re:Perplexity is using stealth, undeclared crawlers to evade... by JimDabell on Monday August 04, 2025 @10:29AM (#44786310)
No. The key difference is that if a user asks about a specific page, when Perplexity fetches that page, it is being operated by a human not acting as a crawler. It doesn’t matter how many times this happens or what they do with the result. If they aren’t recursively fetching pages, then they aren’t a crawler and robots.txt does not apply to them. robots.txt is not a generic access control mechanism, it is designed solely for automated clients.
Re:Perplexity is using stealth, undeclared crawlers to evade... by sbarre on Monday August 04, 2025 @10:35AM (#44786390)
I would only agree with this if we knew for sure that these on-demand human-initiated crawls didn't result in the crawled page being added to an overall index and scheduled for future automated crawls.

Otherwise it's just adding an unwilling website to a crawl index, and showing the result of the first crawl as a byproduct of that action.

Re:Perplexity is using stealth, undeclared crawlers to evade... by glenstein on Monday August 04, 2025 @11:56AM (#44787628)
> It doesn’t matter how many times this happens or what they do with the result.

That's where you lost me, as this is key to GP's point above and it takes more than a mere out-of-left-field declaration that "it doesn't matter" to settle the question of whether it matters.

I think they raised an important point about using cached data to support functions beyond the scope of simple at-request page retrieval.

Re:Perplexity is using stealth, undeclared crawlers to evade... by fluidcruft on Monday August 04, 2025 @10:38AM (#44786426)
Many people don't want their data used for free/any training. AI developers have been so repeatedly unethical that the well-earned Baysian prior is high probability that you cannot trust AI developers to not cross the training/inference streams.
Re:Perplexity is using stealth, undeclared crawlers to evade... by JimDabell on Monday August 04, 2025 @10:42AM (#44786488)
> Many people don't want their data used for free/any training.

That is true. But robots.txt is not designed to give them the ability to prevent this.

Re:Perplexity is using stealth, undeclared crawlers to evade... by gunalx on Monday August 04, 2025 @03:41PM (#44790525)
It is in the name, rules for the robots. Any scraping ai or not, and even mass recrsive or single page, should abide by the rules.
Re:Perplexity is using stealth, undeclared crawlers to evade... by gruez on Monday August 04, 2025 @10:38AM (#44786424)
>If the AI archives/caches all the results it accesses and enough people use it, doesn't it become a scraper?

That's basically how many crowdsourced crawling/archive projects work. For instance, sci-hub and RECAP[1]. Do you think they should be shut down as well? In both cases there's even a stronger justification to shutting them down, because the original content is paywalled and you could plausibly argue there's lost revenue on the line.

[1] https://en.wikipedia.org/wiki/Free_Law_Project#RECAP

Re:Perplexity is using stealth, undeclared crawlers to evade... by fluidcruft on Monday August 04, 2025 @10:44AM (#44786525)
I didn't suggest Perplexity should be shut down, though. And yes, in your analogy sites are completely justified to take whatever actions they can to block people who are building those caches.