Re:Notes on Writing Web Scrapers
by ed25519FUUU on Thursday December 02, 2021 @06:30PM
(#29424475)
Scraping 1000 results a day is really not any kind of web scraping scale. There’s barely any of the same considerations of systems that scrape tens millions a day.
You could easily store those kinds of results in a local DB for offline processing and resuming.
You could easily store those kinds of results in a local DB for offline processing and resuming.