Re:Perplexity is using stealth, undeclared crawlers to evade...
by foota on Monday August 04, 2025 @04:35PM
(#44791042)
Yes, but there's no way to say "allow indexing for search, but not for AI use", right?
[ Parent
| on HN ]
Re:Perplexity is using stealth, undeclared crawlers to evade...
by warkdarrior on Monday August 04, 2025 @05:13PM
(#44791425)
But there is: https://developers.google.com/search/docs/crawling-indexing/...
There is an user agent for search that you can control in robots.txt.
user-agent: Googlebot
There is another user agent for AI training. user-agent: Google-Extended