News for hackers. Stuff that matters.

A comment on Perplexity is using stealth, undeclared crawlers to evade no-crawl directives. Parent comment

Re:Perplexity is using stealth, undeclared crawlers to evade... by Workaccount2 on Monday August 04, 2025 @10:20AM (#44786178)
>2. If I as the human request the software on my computer to modify the content before displaying it, for example by installing an ad-blocker into my user agent, then that's my choice and the website should not be notified about it. Most users agree, some websites try to nag you into modifying the software you run locally.

If I put time and effort into a website and it's content, I should expect no compensation despite bearing all costs.

Is that something everyone would agree with?

The internet should be entirely behind paywalls, besides content that is already provided ad free.

Is that something everyone would agree with?

I think the problem you need to be thinking about is "How can the internet work if no one wants to pay anything for anything?"

[ Parent | on HN ]
Re:Perplexity is using stealth, undeclared crawlers to evade... by Bjartr on Monday August 04, 2025 @10:21AM (#44786193)
You're free to deny access to your site arbitrarily, including for lack of compensation.
[ Parent | on HN ]
Re:Perplexity is using stealth, undeclared crawlers to evade... by cjonas on Monday August 04, 2025 @10:25AM (#44786258)
Like for people or are using a ad block or for a crawler downloading your content so it can be used by an AI response?
[ Parent | on HN ]
Re:Perplexity is using stealth, undeclared crawlers to evade... by Bjartr on Monday August 04, 2025 @11:48AM (#44787506)
Arbitrarily, as in for any reason. It's your site, you decide what constraints an incoming request must meet for it to get a response containing the content of your site.
[ Parent | on HN ]
Re:Perplexity is using stealth, undeclared crawlers to evade... by Workaccount2 on Monday August 04, 2025 @10:23AM (#44786238)
>and the website should not be notified about it.
[ Parent | on HN ]
Re:Perplexity is using stealth, undeclared crawlers to evade... by giantrobot on Monday August 04, 2025 @01:20PM (#44788802)
My user agent and its handling of your content once it's on my computer are not your concern. You don't need to know if the data is parsed by a screen reader, an AI agent, or just piped to /dev/null. It's simply not your concern and never will be.
[ Parent | on HN ]
Re:Perplexity is using stealth, undeclared crawlers to evade... by ndiddy on Monday August 04, 2025 @12:50PM (#44788382)
This article is about Cloudflare attempting to deny Perplexity access to their demo site by blocking Perplexity's declared user-agent and official IP range. Perplexity responded to this denial by impersonating Google Chrome on macOS and rotating through IPs not listed in their published IP range to access the site anyway. This means it's not just "you're free to deny access to your site arbitrarily", it's "you're free to play a cat-and-mouse game indefinitely where the other side is a giant company with hundreds of millions of dollars in VC funding".
[ Parent | on HN ]
Re:Perplexity is using stealth, undeclared crawlers to evade... by Bjartr on Monday August 04, 2025 @03:56PM (#44790683)
The comment I'm responding to established a slightly different context by asking a specific question about getting compensation from site visitors.
[ Parent | on HN ]
Re:Perplexity is using stealth, undeclared crawlers to evade... by nradov on Monday August 04, 2025 @10:22AM (#44786207)
Yes, I agree with that. If a website owner expects compensation then they should use a paywall.
[ Parent | on HN ]
Re:Perplexity is using stealth, undeclared crawlers to evade... by Chris2048 on Monday August 04, 2025 @10:34AM (#44786378)
If I put time and effort into a food recipe should I (get) compensation?

the answer is apparently "no", and I don't really how recipe books have suffered as a result of less gatekeeping.

"How will the internet work"? Probably better in some ways. There is plenty of valuable content on the internet given for free, it's being buried in low-value AI slop.

[ Parent | on HN ]
Re:Perplexity is using stealth, undeclared crawlers to evade... by Workaccount2 on Monday August 04, 2025 @11:17AM (#44787035)
You understand that HN is ad supported too, right?
[ Parent | on HN ]
Re:Perplexity is using stealth, undeclared crawlers to evade... by Chris2048 on Monday August 04, 2025 @11:32AM (#44787249)
No, I don't.

But what is your point? Is the value in HN primarily in its hosting, or the non-ad-supported community?

[ Parent | on HN ]
Re:Perplexity is using stealth, undeclared crawlers to evade... by Workaccount2 on Monday August 04, 2025 @01:19PM (#44788801)
Outside of Wikipedia, I'm not sure what content you are thinking of.

Taking HN as a potential one of these places, it doesn't even qualify. HN is funded entirely to be a place for advertising ycombinator companies to a large crowd of developers. HN is literally a developer honey pot that they get exclusive ad rights to.

[ Parent | on HN ]