News for hackers. Stuff that matters.

A comment on Perplexity is using stealth, undeclared crawlers to evade no-crawl directives. Parent comment

Re:Perplexity is using stealth, undeclared crawlers to evade... by ai-christianson on Monday August 04, 2025 @10:49AM (#44786604)
Websites should be able to request payment. Who cares if it is a human or an agent of a human if it is paying for the request?
[ Parent | on HN ]
Re:Perplexity is using stealth, undeclared crawlers to evade... by carlosjobim on Monday August 04, 2025 @11:56AM (#44787625)
They are able to request payment.
[ Parent | on HN ]
Re:Perplexity is using stealth, undeclared crawlers to evade... by sds357 on Monday August 04, 2025 @02:32PM (#44789730)
What if the agent is reselling the request?
[ Parent | on HN ]
Re:Perplexity is using stealth, undeclared crawlers to evade... by adriand on Monday August 04, 2025 @03:51PM (#44790637)
Cloudflare launched a mechanism for this: https://blog.cloudflare.com/introducing-pay-per-crawl/
[ Parent | on HN ]