News for hackers. Stuff that matters.

A comment on Perplexity is using stealth, undeclared crawlers to evade no-crawl directives.

Re:Perplexity is using stealth, undeclared crawlers to evade... by tempfile on Monday August 04, 2025 @12:26PM (#44788049)
A lot of people posting here seem to think you have a magical god-given right to make money from posting on the public internet. You do not. Effective rate-limiting of crawlers is important, but if the rate is moderated, you do not have a right to decide what people do with the content. If you don't believe that, get off the internet, and don't let the door hit you on the way out.
[ Parent | on HN ]
Re:Perplexity is using stealth, undeclared crawlers to evade... by ibero on Monday August 04, 2025 @12:35PM (#44788159)
what if i want the rate set to zero?
[ Parent | on HN ]
Re:Perplexity is using stealth, undeclared crawlers to evade... by tempfile on Monday August 04, 2025 @04:38PM (#44791074)
Then turn off the server?

You don't have a right to say who or what can read your public website (this is a normative statement). You do have a right not to be DoS'd. If you pretend not to know what that means, it sounds the same as saying "you have an arbitrary right to decide who gets to make requests to your service", but it does not mean that.

[ Parent | on HN ]
Re:Perplexity is using stealth, undeclared crawlers to evade... by jgrall on Monday August 04, 2025 @12:55PM (#44788458)
"You do not have a right to decide what people do with the content." Smh. Yes, laws be damned.
[ Parent | on HN ]