News for hackers. Stuff that matters.
Re:How we learnt to stop worrying and love web scraping by hermanradtke on Wednesday September 09, 2020 @03:52PM (#24425183)
Using a headless browser for scraping is a lot slower and resource intensive than parsing HTML.
Re:How we learnt to stop worrying and love web scraping by PeterisP on Wednesday September 09, 2020 @06:48PM (#24426801)
I don't find this as a concern - in all the scraping I've done, the only bottleneck was the intentional throttling/rate limiting, not the speed and resources spent by the headless browser; a small, cheap machine could easily process many, many times more requests than it would be reasonable to crawl.
Re:How we learnt to stop worrying and love web scraping by sullyj3 on Wednesday September 09, 2020 @04:57PM (#24425823)
Sure, but it might be the only way to get the data.
Re:How we learnt to stop worrying and love web scraping by hansvm on Wednesday September 09, 2020 @05:59PM (#24426419)
It might be, but _starting_ a scraping project with a headless browser might be excessively expensive if you don't need the additional features.
Re:How we learnt to stop worrying and love web scraping by lyjackal on Wednesday September 09, 2020 @09:43PM (#24428067)
"only" is a bit of an overstatement. The data is always coming from somewhere, it just depends on how much effort needed to reverse engineer the JavaScript code path to the data