News for hackers. Stuff that matters.
Re:Perplexity is using stealth, undeclared crawlers to evade... by Bjartr on Monday August 04, 2025 @10:21AM (#44786193)
You're free to deny access to your site arbitrarily, including for lack of compensation.
Re:Perplexity is using stealth, undeclared crawlers to evade... by cjonas on Monday August 04, 2025 @10:25AM (#44786258)
Like for people or are using a ad block or for a crawler downloading your content so it can be used by an AI response?
Re:Perplexity is using stealth, undeclared crawlers to evade... by Bjartr on Monday August 04, 2025 @11:48AM (#44787506)
Arbitrarily, as in for any reason. It's your site, you decide what constraints an incoming request must meet for it to get a response containing the content of your site.
Re:Perplexity is using stealth, undeclared crawlers to evade... by Workaccount2 on Monday August 04, 2025 @10:23AM (#44786238)
>and the website should not be notified about it.
Re:Perplexity is using stealth, undeclared crawlers to evade... by giantrobot on Monday August 04, 2025 @01:20PM (#44788802)
My user agent and its handling of your content once it's on my computer are not your concern. You don't need to know if the data is parsed by a screen reader, an AI agent, or just piped to /dev/null. It's simply not your concern and never will be.
Re:Perplexity is using stealth, undeclared crawlers to evade... by ndiddy on Monday August 04, 2025 @12:50PM (#44788382)
This article is about Cloudflare attempting to deny Perplexity access to their demo site by blocking Perplexity's declared user-agent and official IP range. Perplexity responded to this denial by impersonating Google Chrome on macOS and rotating through IPs not listed in their published IP range to access the site anyway. This means it's not just "you're free to deny access to your site arbitrarily", it's "you're free to play a cat-and-mouse game indefinitely where the other side is a giant company with hundreds of millions of dollars in VC funding".
Re:Perplexity is using stealth, undeclared crawlers to evade... by Bjartr on Monday August 04, 2025 @03:56PM (#44790683)
The comment I'm responding to established a slightly different context by asking a specific question about getting compensation from site visitors.