Re:Ask HN: Can web scraping be the basis of a viable busines...
by ozfive on Saturday December 18, 2021 @08:08PM
(#29610397)
Wasn't there a ruling that web scraping was legal now?
Re:Ask HN: Can web scraping be the basis of a viable busines...
by preinheimer on Saturday December 18, 2021 @08:19PM
(#29610462)
The LinkedIn case, it's still up in the air i think - https://news.bloomberglaw.com/us-law-week/supreme-court-scra...
Recently, there is a boom of "anti-bot" services. These are essentially SaaS businesses that "protect" websites from being scraped by automated software. As you onboard the first customer who wants to extract data from a bot-protected website, you are going to run into an unlimited waterfall of stupid troubles. Your bots will be blocked, will consume excessive amount of data, kill your CPU/GPU performance.
I have shared some highlights on how to bypass these recently on HN [1], but it is sadly only the tip of the iceberg. On the other hand, since the post has been featured on HN I have been reached by more than 50 companies and individuals whose business operating model is based solely on data extraction/automated scraping. These are (in my opinion) successful companies, and two out of these are part of YC.
[1] https://news.ycombinator.com/item?id=29060272