Scraping Shock: Why Web Data Is Getting Too Expensive to Scrape

Re:Scraping Shock: Why Web Data Is Getting Too Expensive to ... by bediger4000 on Monday January 26, 2026 @09:30AM (#46766075)

Ethically dubious article. Treats using "residential proxies", which are probably installed by some kind of cybercriminal, as a legitimate thing to do. Similarly, treats circumventing anti-scraping measures as a legitimate thing to do. They aren't. Take the hint, ignore web sites with some kind of anti-bot, or anti-scraper system. Ignore web sites with a scraper junkyard. Those people don't want you to have their content.

When a website upgrades its anti-bot system, it doesn't just make scraping slightly harder. It can make it 5X, 10X, or even 50X more expensive overnight.

This, of course, is very good news. Keep up the good work, folks!

[ Parent | on HN ]

Re:Scraping Shock: Why Web Data Is Getting Too Expensive to ... by joe_91 on Monday January 26, 2026 @09:44AM (#46766222)

Tell that to the thousands of apps/sides out there which rely on scraped data ;) (Including all search engines/LLMs/price comparison sites etc)

[ Parent | on HN ]

Re:Scraping Shock: Why Web Data Is Getting Too Expensive to ... by bediger4000 on Monday January 26, 2026 @10:31AM (#46766852)

You should see my robots.txt file. I have told the legit ones to stay away. Every scraper and clanker that circumvents "anti-bot" technology can go straight to hell - they've been warned that I don't want them.

But your observation doesn't deal with the un-ethicality of the original article, advocating benefiting from cybercrime, and ignoring the explicit wishes of web sites that use "anti-bot" technology.

[ Parent | on HN ]