Web Scraping with Node.js and Chimera

Re:Web Scraping with Node.js and Chimera by lancefisher on Wednesday January 02, 2013 @11:11PM (#5000980)

This is a great idea! phantomjs-node works okay, but it is suck a hack. A nifty hack, but still. https://github.com/sgentle/phantomjs-node#how-does-it-work

If you want to parse the DOM for the internet at large, you need a real browser. There are simply too many sites with really bad HTML to be parsed reliably with anything else.

[ Parent | on HN ]

Re:Web Scraping with Node.js and Chimera by catch23 on Thursday January 03, 2013 @12:37AM (#5001182)

If you need a great html parser, I also wrote a library for one of those: https://github.com/deanmao/node-hubbub

It's merely an integration library for the netsurf browser. It's the html parser for a real browser. I considered using the parser from other browsers like firefox or webkit, but netsurf had the fewest external dependencies.

[ Parent | on HN ]

Re:Web Scraping with Node.js and Chimera by salmanapk on Thursday January 03, 2013 @01:43AM (#5001284)

Just in case this saves someone's time:

phantomjs-node uses ridiculous hacks. Not being rude but I think the author was drunk when writing that. I ended up wasting quite some time when trying to customize it.

You do NOT need to use any hacks whatsoever to run dnode on phantom, just browserify(https://github.com/substack/node-browserify) the shoe and dnode modules and use the example code at https://github.com/substack/dnode .

[ Parent | on HN ]