Re:How we learnt to stop worrying and love web scraping
by hermanradtke on Wednesday September 09, 2020 @03:52PM
(#24425183)
Using a headless browser for scraping is a lot slower and resource intensive than parsing HTML.
[ Parent
| on HN ]
Re:How we learnt to stop worrying and love web scraping
by PeterisP on Wednesday September 09, 2020 @06:48PM
(#24426801)
I don't find this as a concern - in all the scraping I've done, the only bottleneck was the intentional throttling/rate limiting, not the speed and resources spent by the headless browser; a small, cheap machine could easily process many, many times more requests than it would be reasonable to crawl.
Re:How we learnt to stop worrying and love web scraping
by sullyj3 on Wednesday September 09, 2020 @04:57PM
(#24425823)
Sure, but it might be the only way to get the data.
Re:How we learnt to stop worrying and love web scraping
by hansvm on Wednesday September 09, 2020 @05:59PM
(#24426419)
It might be, but _starting_ a scraping project with a headless browser might be excessively expensive if you don't need the additional features.