News for hackers. Stuff that matters.

A comment on Scraping Web Pages With jQuery, Node.js and Jsdom. Parent comment

Re:Scraping Web Pages With jQuery, Node.js and Jsdom by MatthewPhillips on Thursday March 08, 2012 @11:25AM (#3680280)
Phantom doesn't need access to node apis. Separation of concerns. Run your phantom script to scrape and then pipe the results to your node script for processing. I've worked on 2 scraping projects using this method and it works great.
[ Parent | on HN ]
Re:Scraping Web Pages With jQuery, Node.js and Jsdom by baudehlo on Thursday March 08, 2012 @11:47AM (#3680426)
Sure, it can work great if your needs are simple. But if you're posting data to forms which you need from a database, and you don't know what you might need from the DB until runtime, it can get a bit more complex.

Also worth noting that jsdom is quite slow, and has a strict HTML parser. If you want something faster that will cope with more web pages, look up cheerio.

[ Parent | on HN ]
Re:Scraping Web Pages With jQuery, Node.js and Jsdom by MatthewPhillips on Thursday March 08, 2012 @12:16PM (#3680577)
Remember that Phantom just runs a web browser. So writing a Phantom script is like writing an app on top of another app. This means you can use XMLHttpRequest or WebSockets to communicate with a back-end if that's necessary.
[ Parent | on HN ]
Re:Scraping Web Pages With jQuery, Node.js and Jsdom by baudehlo on Thursday March 08, 2012 @01:31PM (#3680932)
Yes that's what we ended up doing with Kapow. Just felt... dirty :)
[ Parent | on HN ]
Re:Scraping Web Pages With jQuery, Node.js and Jsdom by tszming on Thursday March 08, 2012 @10:07PM (#3682722)
Kapow? You mean this company: (http://kapowsoftware.com/solutions/content-migration/index.p...)?, mind to share about your experience?
[ Parent | on HN ]