Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

Re:Perplexity is using stealth, undeclared crawlers to evade... by ivanjermakov on Tuesday August 05, 2025 @02:49AM (#44795009)

The problem is not about personal use. It's about big corporations scrapping billions of pages to make money.

[ Parent | on HN ]

Re:Perplexity is using stealth, undeclared crawlers to evade... by skeezyboy on Tuesday August 05, 2025 @06:05AM (#44796225)

those publicly accessible pages yea?

[ Parent | on HN ]

Re:Perplexity is using stealth, undeclared crawlers to evade... by antonymoose on Tuesday August 05, 2025 @06:43AM (#44796446)

While I agree it’s a problem, unless you get litigious or the backing of a national congressional body, you might as well not care. You won’t win this fight.

Circa 2008 I worked for a startup that would scrape Google Books and a variety of other sources for public domain content to then print via Amazon’s Print-on-Demand services. Google, of course, didn’t like this and introduced a Captcha not very long after we started scraping.

So we hired a team of underemployed / unemployed English majors during the height of the recession, paid them $10 per hour to type in Captchas all day long and we downloaded their full corpus anyways.

If Google can’t win, you won’t either.

[ Parent | on HN ]