Re:How we learnt to stop worrying and love web scraping
by waprin on Wednesday September 09, 2020 @12:37PM
(#24422799)
Its not too complicated, you just need a headless browser. Having done a ton of web scraping projects, Iād recommend just starting with this approach as even sites that look pretty static use Javascript in subtle ways.
[ Parent
| on HN ]
Re:How we learnt to stop worrying and love web scraping
by lyjackal on Wednesday September 09, 2020 @09:41PM
(#24428058)
Data is usually embedded in json or available from an internal api when it's an SPA. Headless browser resources are pretty huge. When doing large scale scraping, headless browser should be a last resort
Re:How we learnt to stop worrying and love web scraping
by hermanradtke on Wednesday September 09, 2020 @03:52PM
(#24425183)
Using a headless browser for scraping is a lot slower and resource intensive than parsing HTML.
Re:How we learnt to stop worrying and love web scraping
by PeterisP on Wednesday September 09, 2020 @06:48PM
(#24426801)
I don't find this as a concern - in all the scraping I've done, the only bottleneck was the intentional throttling/rate limiting, not the speed and resources spent by the headless browser; a small, cheap machine could easily process many, many times more requests than it would be reasonable to crawl.
Re:How we learnt to stop worrying and love web scraping
by sullyj3 on Wednesday September 09, 2020 @04:57PM
(#24425823)
Sure, but it might be the only way to get the data.
Re:How we learnt to stop worrying and love web scraping
by hansvm on Wednesday September 09, 2020 @05:59PM
(#24426419)
It might be, but _starting_ a scraping project with a headless browser might be excessively expensive if you don't need the additional features.