News for hackers. Stuff that matters.

A comment on Scraping Web Pages With jQuery, Node.js and Jsdom. Parent comment

Re:Scraping Web Pages With jQuery, Node.js and Jsdom by baudehlo on Thursday March 08, 2012 @11:03AM (#3680192)
Phantomjs has a lot larger overhead (it runs a full browser). Plus it doesn't give you access to the full Node.js ecosystem (e.g. access to databases, etc). You can use Node's Phantomjs driver, but it spawns a child process to do the work, and seems a little complicated in how all the interactions work.

And of course you can scrape interactive sites - interactive sites still basically just use HTTP to request data. Just watch the network window in Chrome's developer tools, and figure out what HTTP requests the site is making that you are interested in. Then code them into request() calls.

[ Parent | on HN ]
Re:Scraping Web Pages With jQuery, Node.js and Jsdom by MatthewPhillips on Thursday March 08, 2012 @11:25AM (#3680280)
Phantom doesn't need access to node apis. Separation of concerns. Run your phantom script to scrape and then pipe the results to your node script for processing. I've worked on 2 scraping projects using this method and it works great.
[ Parent | on HN ]
Re:Scraping Web Pages With jQuery, Node.js and Jsdom by baudehlo on Thursday March 08, 2012 @11:47AM (#3680426)
Sure, it can work great if your needs are simple. But if you're posting data to forms which you need from a database, and you don't know what you might need from the DB until runtime, it can get a bit more complex.

Also worth noting that jsdom is quite slow, and has a strict HTML parser. If you want something faster that will cope with more web pages, look up cheerio.

[ Parent | on HN ]
Re:Scraping Web Pages With jQuery, Node.js and Jsdom by MatthewPhillips on Thursday March 08, 2012 @12:16PM (#3680577)
Remember that Phantom just runs a web browser. So writing a Phantom script is like writing an app on top of another app. This means you can use XMLHttpRequest or WebSockets to communicate with a back-end if that's necessary.
[ Parent | on HN ]
Re:Scraping Web Pages With jQuery, Node.js and Jsdom by baudehlo on Thursday March 08, 2012 @01:31PM (#3680932)
Yes that's what we ended up doing with Kapow. Just felt... dirty :)
[ Parent | on HN ]
Re:Scraping Web Pages With jQuery, Node.js and Jsdom by tszming on Thursday March 08, 2012 @10:07PM (#3682722)
Kapow? You mean this company: (http://kapowsoftware.com/solutions/content-migration/index.p...)?, mind to share about your experience?
[ Parent | on HN ]