Re:Colly: Fast and Elegant Scraping Framework for Golang
by dithering on Thursday October 05, 2017 @12:46PM
(#15410600)
I know of a site where all the data is embedded inline ("var Person = {firstName: 'Fred', lastName: 'Smith' }") then rendered to HTML with JS. No endpoints, Javascript not JSON, complete pain in the ass to extract without rendering in a headless browser first.
[ Parent
| on HN ]
Re:Colly: Fast and Elegant Scraping Framework for Golang
by sitepodmatt on Friday October 06, 2017 @12:03AM
(#15414336)
Isn't this better? Why do you need it render when they're giving you basically JSON of the data (might need to quote the keys in strict parsers), just extract that.