Scraping Web Pages With jQuery, Node.js and Jsdom

Re:Scraping Web Pages With jQuery, Node.js and Jsdom by baudehlo on Thursday March 08, 2012 @11:47AM (#3680426)

Sure, it can work great if your needs are simple. But if you're posting data to forms which you need from a database, and you don't know what you might need from the DB until runtime, it can get a bit more complex.

Also worth noting that jsdom is quite slow, and has a strict HTML parser. If you want something faster that will cope with more web pages, look up cheerio.

[ Parent | on HN ]

Re:Scraping Web Pages With jQuery, Node.js and Jsdom by MatthewPhillips on Thursday March 08, 2012 @12:16PM (#3680577)

Remember that Phantom just runs a web browser. So writing a Phantom script is like writing an app on top of another app. This means you can use XMLHttpRequest or WebSockets to communicate with a back-end if that's necessary.

[ Parent | on HN ]

Re:Scraping Web Pages With jQuery, Node.js and Jsdom by baudehlo on Thursday March 08, 2012 @01:31PM (#3680932)

Yes that's what we ended up doing with Kapow. Just felt... dirty :)

[ Parent | on HN ]

Re:Scraping Web Pages With jQuery, Node.js and Jsdom by tszming on Thursday March 08, 2012 @10:07PM (#3682722)

Kapow? You mean this company: (http://kapowsoftware.com/solutions/content-migration/index.p...)?, mind to share about your experience?

[ Parent | on HN ]