News for hackers. Stuff that matters.
Re:Colly: Fast and Elegant Scraping Framework for Golang by dithering on Thursday October 05, 2017 @12:46PM (#15410600)
I know of a site where all the data is embedded inline ("var Person = {firstName: 'Fred', lastName: 'Smith' }") then rendered to HTML with JS. No endpoints, Javascript not JSON, complete pain in the ass to extract without rendering in a headless browser first.
Re:Colly: Fast and Elegant Scraping Framework for Golang by sitepodmatt on Friday October 06, 2017 @12:03AM (#15414336)
Isn't this better? Why do you need it render when they're giving you basically JSON of the data (might need to quote the keys in strict parsers), just extract that.
Re:Colly: Fast and Elegant Scraping Framework for Golang by dithering on Friday October 06, 2017 @06:57AM (#15415773)
The short answer is that, yes, the only parser available to me is that strict. If I'm going to jump languages, I might as well go the whole hog and jump to Puppeteer.