Re:Web Scraping with Python
by taosx on Sunday May 15, 2022 @05:47PM
(#31391467)
Agreed (especially on the devops overhead), some options to lower the bandwidth usage using request blocking:
- block ads
- blocks videos, images
- css stylesheets
In the past I've built scraper infrastructure (headless pools, credential stores, proxy managers, agent profiles) and managed to get a pretty efficient service by tracking cpu,memory,network usage per each job and writing specialized versions. I got pretty far trying to automatically generate specialized scrapers from previous requests but I moved to other projects.
Scraping becomes boring really fast if you don't use the data in meaningful ways.
In the past I've built scraper infrastructure (headless pools, credential stores, proxy managers, agent profiles) and managed to get a pretty efficient service by tracking cpu,memory,network usage per each job and writing specialized versions. I got pretty far trying to automatically generate specialized scrapers from previous requests but I moved to other projects.
Scraping becomes boring really fast if you don't use the data in meaningful ways.