News for hackers. Stuff that matters.
Re:Notes on Writing Web Scrapers by ed25519FUUU on Thursday December 02, 2021 @06:30PM (#29424475)
Scraping 1000 results a day is really not any kind of web scraping scale. There’s barely any of the same considerations of systems that scrape tens millions a day.

You could easily store those kinds of results in a local DB for offline processing and resuming.

Re:Notes on Writing Web Scrapers by cushychicken on Thursday December 02, 2021 @07:00PM (#29424785)
Even at my cutesy, boutique scale, there is plenty there to obsess about. XD