Re:Perplexity is using stealth, undeclared crawlers to evade...
by fluidcruft on Monday August 04, 2025 @10:44AM
(#44786525)
I didn't suggest Perplexity should be shut down, though. And yes, in your analogy sites are completely justified to take whatever actions they can to block people who are building those caches.
That's basically how many crowdsourced crawling/archive projects work. For instance, sci-hub and RECAP[1]. Do you think they should be shut down as well? In both cases there's even a stronger justification to shutting them down, because the original content is paywalled and you could plausibly argue there's lost revenue on the line.
[1] https://en.wikipedia.org/wiki/Free_Law_Project#RECAP