How we learnt to stop worrying and love web scraping

Re:How we learnt to stop worrying and love web scraping by userbinator on Wednesday September 09, 2020 @08:26PM (#24427599)

For example, moving from static HTML to a web framework would require your scraper to actually run the JavaScript to generate the DOM in the state that a reader might view it in

Or, as is often the case, the content is already there or fetched via an API in far more easily-consumed JSON format that you can use directly.

[ Parent | on HN ]

Re:How we learnt to stop worrying and love web scraping by throwaway894345 on Wednesday September 09, 2020 @09:07PM (#24427846)

That’s my point.

Granted, lots of APIs make it prohibitively difficult to authenticate such that it’s easier to simply scrape. Such is the case with just about every Microsoft product I’ve ever used, most recently the XBox Live API. I genuinely wonder what kind of nonsense goes on in Microsoft design review meetings.

[ Parent | on HN ]