News for hackers. Stuff that matters.
Re:Perplexity is using stealth, undeclared crawlers to evade... by fluidcruft on Monday August 04, 2025 @10:38AM (#44786426)
Many people don't want their data used for free/any training. AI developers have been so repeatedly unethical that the well-earned Baysian prior is high probability that you cannot trust AI developers to not cross the training/inference streams.
Re:Perplexity is using stealth, undeclared crawlers to evade... by JimDabell on Monday August 04, 2025 @10:42AM (#44786488)
> Many people don't want their data used for free/any training.

That is true. But robots.txt is not designed to give them the ability to prevent this.

Re:Perplexity is using stealth, undeclared crawlers to evade... by gunalx on Monday August 04, 2025 @03:41PM (#44790525)
It is in the name, rules for the robots. Any scraping ai or not, and even mass recrsive or single page, should abide by the rules.