News for hackers. Stuff that matters.
Re:Web Scraping with Python by samwillis on Sunday May 15, 2022 @02:58PM (#31390005)
From my experience headless browser scraping is in the order of 100x slower due to increased bandwidth, cpu and memory. I would seriously suggest the other, start with traditional scraping and if you can’t make it work then go headless. The difference in speed is to the point that going a longer route (maybe having to scrape 10x more pages for example) to the content that enables you to not use headless will probably still end up quicker.

On the memory side, headless you end up with far more memory leaks, having to manage stopping and starting new browser instances while maintaining scraping state. The devops overhead is probably 10x more with headless.

Re:Web Scraping with Python by taosx on Sunday May 15, 2022 @05:47PM (#31391467)
Agreed (especially on the devops overhead), some options to lower the bandwidth usage using request blocking: - block ads - blocks videos, images - css stylesheets

In the past I've built scraper infrastructure (headless pools, credential stores, proxy managers, agent profiles) and managed to get a pretty efficient service by tracking cpu,memory,network usage per each job and writing specialized versions. I got pretty far trying to automatically generate specialized scrapers from previous requests but I moved to other projects.

Scraping becomes boring really fast if you don't use the data in meaningful ways.

Re:Web Scraping with Python by elorant on Monday May 16, 2022 @05:11AM (#31394894)
If you don't load images, and you disable 3rd party JavaScript you can save a lot of loading time with a headless. I'm using a preconfigured profile in Firefox with uMatrix installed and sites load at least 50% faster. Add another 20-30% from not loading images, and suddenly using a headless becomes affordable.
Re:Web Scraping with Python by MonaroVXR on Sunday May 15, 2022 @10:43PM (#31393248)
What is traditional?