News for hackers. Stuff that matters.

A comment on Portia, an open-source visual web scraper. Parent comment

Re:Portia, an open-source visual web scraper by agumonkey on Tuesday April 01, 2014 @11:17AM (#7508869)
It also depends on a coherent structure in HTML websites.

Domains running websites which are more like javascript frontend modules shouldn't be scraped at all, it screams for a public API.

[ Parent | on HN ]
Re:Portia, an open-source visual web scraper by uptown on Tuesday April 01, 2014 @02:34PM (#7510611)
"it screams for a public API"

But many content owners would never provide their data in this format even if doing-so would be trivial.

[ Parent | on HN ]
Re:Portia, an open-source visual web scraper by CMCDragonkai on Thursday April 03, 2014 @02:06AM (#7521230)
Try using https://snapsearch.io/ It is designed for JS sites.
[ Parent | on HN ]
Re:Portia, an open-source visual web scraper by jdavis703 on Tuesday April 01, 2014 @02:42PM (#7510669)
These single page sites do have a public, albeit, undocumented API. If you analyze the network requests via the dev tools in your browser you'll have an XML/JSON data source that is probably structured better than the markup.
[ Parent | on HN ]
Re:Portia, an open-source visual web scraper by agumonkey on Tuesday April 01, 2014 @02:54PM (#7510772)
Of course, I should have thought about it that way.
[ Parent | on HN ]