Re:Show HN: I scraped 25M Shopify products to build a search...
by DeathArrow on Thursday December 14, 2023 @02:28AM
(#38638736)
>I have always used standard python tools like selenium, bs4 and the like
There's nothing to scrap. You just download a JSON, the site owners kindly put on your disposal.
Scraping is a more complex process, where you have to work around rate limiting and captchas. For the tool I built I wrote tens of thousands of lines of code and I still find daily issues I have to deal with if I want to scrap a particular web page, issues I don't always have the time to solve.
There's nothing to scrap. You just download a JSON, the site owners kindly put on your disposal.
Scraping is a more complex process, where you have to work around rate limiting and captchas. For the tool I built I wrote tens of thousands of lines of code and I still find daily issues I have to deal with if I want to scrap a particular web page, issues I don't always have the time to solve.