News for hackers. Stuff that matters.
Re:Perplexity is using stealth, undeclared crawlers to evade... by foota on Monday August 04, 2025 @04:35PM (#44791042)
Yes, but there's no way to say "allow indexing for search, but not for AI use", right?
Re:Perplexity is using stealth, undeclared crawlers to evade... by warkdarrior on Monday August 04, 2025 @05:13PM (#44791425)
But there is: https://developers.google.com/search/docs/crawling-indexing/...

There is an user agent for search that you can control in robots.txt.

    user-agent: Googlebot
There is another user agent for AI training.

    user-agent: Google-Extended
Re:Perplexity is using stealth, undeclared crawlers to evade... by foota on Tuesday August 05, 2025 @02:51AM (#44795027)
Wow, I had no idea this page existed, thanks for the reference!