Scraping Web Pages With jQuery, Node.js and Jsdom

Re:Scraping Web Pages With jQuery, Node.js and Jsdom by lancefisher on Thursday March 08, 2012 @12:10PM (#3680550)

The problem I've had with using jsdom in scraping web pages is that it is not very forgiving about bad HTML. There are so many pages in the wild that have malformed HTML, and jsdom just pukes on it. I started using Apricot [1] which uses HtmlParser [2] which has been better. I'd like to hear what others are using to scrape bad webpages.

[1] https://github.com/silentrob/Apricot

[2] https://github.com/tautologistics/node-htmlparser

[ Parent | on HN ]

Re:Scraping Web Pages With jQuery, Node.js and Jsdom by tluyben2 on Thursday March 08, 2012 @03:35PM (#3681478)

I added this for that reason; https://github.com/tautologistics/node-htmlparser/issues/2

The issue is, as the above thread shows, that the creator of node-htmlparser does not want to support illegal HTML. Unfortunately that is not realistic.

[ Parent | on HN ]

Re:Scraping Web Pages With jQuery, Node.js and Jsdom by gikrauss on Thursday March 08, 2012 @03:28PM (#3681433)

A couple of months ago I posted a very similar article that uses node.js, request and jQuery to achieve the same goal. In this case, as long as jQuery can handle the response you shouldn't have any problem with malformed HTML...

[ Parent | on HN ]

Re:Scraping Web Pages With jQuery, Node.js and Jsdom by gikrauss on Thursday March 08, 2012 @04:04PM (#3681633)

I forgot the link http://blog.devartis.com/2012/01/05/scraping-websites-having... :S

[ Parent | on HN ]

Re:Scraping Web Pages With jQuery, Node.js and Jsdom by RyanMcGreal on Thursday March 08, 2012 @01:08PM (#3680830)

I've been spoiled rotten by the magnificent BeautifulSoup library for Python.

[ Parent | on HN ]