Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

Re:Perplexity is using stealth, undeclared crawlers to evade... by account42 on Tuesday August 05, 2025 @11:08AM (#44798986)

> If we talk about Anubis, it's pretty invisible. You wait a couple of seconds in the first visit, and don't get challenged for a couple of weeks, at least. With more tuning some of the sites using Anubis work perfectly well without ever seeing Anubis' wall while stopping AI crawlers.

It's not invisible, the sites using it don't work perfectly well for all users and it doesn't stop AI crawlers.

[ Parent | on HN ]

Re:Perplexity is using stealth, undeclared crawlers to evade... by bayindirh on Tuesday August 05, 2025 @11:12AM (#44799054)

I haven't seen any problems with any Anubis enabled site I encountered. Can you give examples? This is interesting.

[ Parent | on HN ]

Re:Perplexity is using stealth, undeclared crawlers to evade... by fxtentacle on Tuesday August 05, 2025 @02:09PM (#44801910)

I've never seen problems with Anubis.

[ Parent | on HN ]