News for hackers. Stuff that matters.
Re:The State of Web Scraping 2022 by Ian_Kerins on Wednesday January 12, 2022 @10:28AM (#29907656)
You can do it as a service, but that is highly competitive and basically trading time for money. Best ways are to productize it:

- build a on-demand data api for a specific type of data and charge a premium for it. Good example is https://serpapi.com/ who do Google data, charge ~10X markup on proxy costs

- proxy solutions make good money. To scrape at scale you need proxies, and lots of users pay $1-5k per month. Lots of proxy solutions doing +$100k per month.

- build a tool that uses web scraped data, analyses/filters it and displays it to users. Lots of the biggest web scrapers are doing this, ex. doing product monitoring products for e-commerce companies, etc. Lots of competition there, but you can do it in new markets, like NFTs, etc.

- hedge funds will pay huge money for web data, if you have 5 years of continuous data so they can backtest it.

Re:The State of Web Scraping 2022 by chirau on Wednesday January 12, 2022 @04:59PM (#29913290)
> build a tool that uses web scraped data, analyses/filters it and displays it to users. Lots of the biggest web scrapers are doing this, ex. doing product monitoring products for e-commerce companies, etc. Lots of competition there, but you can do it in new markets, like NFTs, etc.

Do you have any examples of such sites?

> hedge funds will pay huge money for web data, if you have 5 years of continuous data so they can backtest it.

what kind of web data would they be interested in?