News for hackers. Stuff that matters.
Re:Perplexity is using stealth, undeclared crawlers to evade... by antonymoose on Tuesday August 05, 2025 @06:43AM (#44796446)
While I agree it’s a problem, unless you get litigious or the backing of a national congressional body, you might as well not care. You won’t win this fight.

Circa 2008 I worked for a startup that would scrape Google Books and a variety of other sources for public domain content to then print via Amazon’s Print-on-Demand services. Google, of course, didn’t like this and introduced a Captcha not very long after we started scraping.

So we hired a team of underemployed / unemployed English majors during the height of the recession, paid them $10 per hour to type in Captchas all day long and we downloaded their full corpus anyways.

If Google can’t win, you won’t either.

There are no comments on this comment yet.