News for hackers. Stuff that matters.
Re:Perplexity is using stealth, undeclared crawlers to evade... by JimDabell on Monday August 04, 2025 @10:42AM (#44786488)
> Many people don't want their data used for free/any training.

That is true. But robots.txt is not designed to give them the ability to prevent this.

Re:Perplexity is using stealth, undeclared crawlers to evade... by gunalx on Monday August 04, 2025 @03:41PM (#44790525)
It is in the name, rules for the robots. Any scraping ai or not, and even mass recrsive or single page, should abide by the rules.