News for hackers. Stuff that matters.
Re:Web Scraping with Python by taosx on Sunday May 15, 2022 @05:47PM (#31391467)
Agreed (especially on the devops overhead), some options to lower the bandwidth usage using request blocking: - block ads - blocks videos, images - css stylesheets

In the past I've built scraper infrastructure (headless pools, credential stores, proxy managers, agent profiles) and managed to get a pretty efficient service by tracking cpu,memory,network usage per each job and writing specialized versions. I got pretty far trying to automatically generate specialized scrapers from previous requests but I moved to other projects.

Scraping becomes boring really fast if you don't use the data in meaningful ways.

There are no comments on this comment yet.