News for hackers. Stuff that matters.
Re:Ask HN: Can web scraping be the basis of a viable busines... by proszkinasenne2 on Saturday December 18, 2021 @08:02PM (#29610363)
Sure, it can be! Also, as some people have already pointed out, this is often a gray area where people go beyond violating ToS. Some good examples are privacy violations (scraping personal data), credentials stuffing etc.

Recently, there is a boom of "anti-bot" services. These are essentially SaaS businesses that "protect" websites from being scraped by automated software. As you onboard the first customer who wants to extract data from a bot-protected website, you are going to run into an unlimited waterfall of stupid troubles. Your bots will be blocked, will consume excessive amount of data, kill your CPU/GPU performance.

I have shared some highlights on how to bypass these recently on HN [1], but it is sadly only the tip of the iceberg. On the other hand, since the post has been featured on HN I have been reached by more than 50 companies and individuals whose business operating model is based solely on data extraction/automated scraping. These are (in my opinion) successful companies, and two out of these are part of YC.

[1] https://news.ycombinator.com/item?id=29060272

Re:Ask HN: Can web scraping be the basis of a viable busines... by ozfive on Saturday December 18, 2021 @08:08PM (#29610397)
Wasn't there a ruling that web scraping was legal now?
Re:Ask HN: Can web scraping be the basis of a viable busines... by preinheimer on Saturday December 18, 2021 @08:19PM (#29610462)
The LinkedIn case, it's still up in the air i think - https://news.bloomberglaw.com/us-law-week/supreme-court-scra...
Re:Ask HN: Can web scraping be the basis of a viable busines... by ozfive on Monday December 20, 2021 @11:20AM (#29626562)
Thanks for this!