PHP Simple HTML DOM Parser

Re:PHP Simple HTML DOM Parser by crux_ on Friday December 11, 2009 @05:34PM (#990671)

Since folks here seem to be in the know, if I were after "parsing as close to the browser as possible", ideally in Java/Scala or Ruby or Python (sorry PHP!) -- any recommendations?

I've done scraping (e.g. w/ BeautifulSoup) but haven't looked to see how true the parses are to what IE/FF/WebKit would produce.

(On my list of things to look into: html5lib -- http://code.google.com/p/html5lib/ ... is it any good?)

[ Parent | on HN ]

Re:PHP Simple HTML DOM Parser by fizx on Saturday December 12, 2009 @01:00AM (#991243)

Run headless Firefox+xvfb+Selenium. Selenium has java, ruby, and python clients. This is working really well in production.

You might also have luck with AppleScript/RubyCocoa/MacRuby+Safari. I've tried a couple other options like HTMLUnit (which google uses with GWT), and Mozilla Java Html Parser, which Dapper.net uses. I couldn't get them running, but YMMV.

[ Parent | on HN ]

Re:PHP Simple HTML DOM Parser by tremendo on Friday December 11, 2009 @09:30PM (#991024)

for Ruby there's Hpricot and Nokogiri. Now I must admit not understanding what "parsing as close to the browser as possible" would mean. These parsers would not be for displaying the HTML, they're not rendering engines like those in browsers, but will help you navigate the DOM of a (X)(HT)ML document programatically. Surely I'm missing your meaning.

[ Parent | on HN ]

Re:PHP Simple HTML DOM Parser by crux_ on Sunday December 13, 2009 @10:16AM (#992763)

What I'm aiming at would mean "given this lump of (malformed) HTML, what DOM would a browser give me?" Maybe Hpricot, BeautifulSoup, et al are already there, but I don't know. :)

[ Parent | on HN ]