Re:Perplexity is using stealth, undeclared crawlers to evade...
by k1m on Monday August 04, 2025 @05:49PM
(#44791740)
When Yahoo! Pipes was still running (long time ago), their official position was:
> Because Pipes is not a web crawler (the service only retrieves URLs when requested to by a Pipe author or user) Pipes does not follow the robots exclusion protocol, and won't check your robots.txt file.
> Because Pipes is not a web crawler (the service only retrieves URLs when requested to by a Pipe author or user) Pipes does not follow the robots exclusion protocol, and won't check your robots.txt file.