Agreed (especially on the devops overhead), some options to lower the bandwidth usage using request blocking:
- block ads
- blocks videos, images
- css stylesheets
In the past I've built scraper infrastructure (headless pools, credential stores, proxy managers, agent profiles) and managed to get a pretty efficient service by tracking cpu,memory,network usage per each job and writing specialized versions. I got pretty far trying to automatically generate specialized scrapers from previous requests but I moved to other projects.
Scraping becomes boring really fast if you don't use the data in meaningful ways.
If you don't load images, and you disable 3rd party JavaScript you can save a lot of loading time with a headless. I'm using a preconfigured profile in Firefox with uMatrix installed and sites load at least 50% faster. Add another 20-30% from not loading images, and suddenly using a headless becomes affordable.
On the memory side, headless you end up with far more memory leaks, having to manage stopping and starting new browser instances while maintaining scraping state. The devops overhead is probably 10x more with headless.