News for hackers. Stuff that matters.

A comment on Perplexity is using stealth, undeclared crawlers to evade no-crawl directives. Parent comment

Re:Perplexity is using stealth, undeclared crawlers to evade... by Workaccount2 on Monday August 04, 2025 @10:23AM (#44786238)
>and the website should not be notified about it.
[ Parent | on HN ]
Re:Perplexity is using stealth, undeclared crawlers to evade... by giantrobot on Monday August 04, 2025 @01:20PM (#44788802)
My user agent and its handling of your content once it's on my computer are not your concern. You don't need to know if the data is parsed by a screen reader, an AI agent, or just piped to /dev/null. It's simply not your concern and never will be.
[ Parent | on HN ]