News for hackers. Stuff that matters.

A comment on Perplexity is using stealth, undeclared crawlers to evade no-crawl directives. Parent comment

Re:Perplexity is using stealth, undeclared crawlers to evade... by snowwrestler on Monday August 04, 2025 @10:49PM (#44793815)
We have a faceted search that creates billions of unique URLs by combinations of the facets. As such, we block all crawlers from it in robots.txt, which saves us AND them from a bunch of pointless indexing load.

But a stealth bot has been crawling all these URLs for weeks. Thus wasting a shitload of our resources AND a shitload of their resources too.

Whoever it is (and I now suspect it is Perplexity based on this Cloudflare post), they thought they were being so clever by ignoring our robots.txt. Instead they have been wasting money for weeks. Our block was there for a reason.

[ Parent | on HN ]
Re:Perplexity is using stealth, undeclared crawlers to evade... by graybanana on Tuesday August 05, 2025 @05:35PM (#44804671)
We have the same issue (billions of URLs). The newer bots that rotate IPs across thousands of IP ranges are killing us and there is no good way to block them short of captcha's or forcing logins, which we would really rather not inflict on our users.
[ Parent | on HN ]