News for hackers. Stuff that matters.

A comment on Perplexity is using stealth, undeclared crawlers to evade no-crawl directives. Parent comment

Re:Perplexity is using stealth, undeclared crawlers to evade... by palmfacehn on Monday August 04, 2025 @11:32AM (#44787247)
https://commoncrawl.org/

>Common Crawl maintains a free, open repository of web crawl data that can be used by anyone.

[ Parent | on HN ]
Re:Perplexity is using stealth, undeclared crawlers to evade... by fxtentacle on Tuesday August 05, 2025 @02:14PM (#44801969)
The problem is that many websites and domains are missing from it.
[ Parent | on HN ]