News for hackers. Stuff that matters.
Re:Perplexity is using stealth, undeclared crawlers to evade... by warkdarrior on Monday August 04, 2025 @05:13PM (#44791425)
But there is: https://developers.google.com/search/docs/crawling-indexing/...

There is an user agent for search that you can control in robots.txt.

    user-agent: Googlebot
There is another user agent for AI training.

    user-agent: Google-Extended
Re:Perplexity is using stealth, undeclared crawlers to evade... by foota on Tuesday August 05, 2025 @02:51AM (#44795027)
Wow, I had no idea this page existed, thanks for the reference!