Re:Perplexity is using stealth, undeclared crawlers to evade...
by dgreensp on Monday August 04, 2025 @03:27PM
(#44790364)
Googlebot respects robots.txt. And Google doesn't use the fetched data from users of Chrome to supplement their search index (as a2128 is speculating that Perplexity might do when they fetch pages on the user's behalf).
Re:Perplexity is using stealth, undeclared crawlers to evade...
by foota on Monday August 04, 2025 @04:35PM
(#44791042)
Yes, but there's no way to say "allow indexing for search, but not for AI use", right?
Re:Perplexity is using stealth, undeclared crawlers to evade...
by warkdarrior on Monday August 04, 2025 @05:13PM
(#44791425)
But there is: https://developers.google.com/search/docs/crawling-indexing/...
There is an user agent for search that you can control in robots.txt.
user-agent: Googlebot
There is another user agent for AI training. user-agent: Google-Extended
Crawling is legal. Training is presumably legal. Long may the little guys do both.