Re:Notes on Writing Web Scrapers
by cushychicken on Thursday December 02, 2021 @05:04PM
(#29423387)
Interesting question. My first instinct is to say that woob seems closer in use case to a browser than a scraper, as it seems largely geared towards making rich websites more easily accessible. (If I'm reading the page right, anyway.) A scraper is basically just hitting a web page over and over again, as fast as you can manage.
The trick, IMO, is to be closer to browser level loading on a server than scraper. Make sense?
How much of this article do you think would apply to something like that? e.g. something like 'wait a second (or even two!) between successive hits' might not be necessary (one could perhaps shorten it to 1/4 second) if one is only doing a few requests followed by long periods of no requests.