News for hackers. Stuff that matters.
Re:Hacker News doesn't validate by daleharvey on Tuesday January 06, 2009 @10:36PM (#422934)
they arent anywhere near as ubiquitous as xml parsers
Re:Hacker News doesn't validate by snprbob86 on Tuesday January 06, 2009 @11:09PM (#422980)
For Python, use BeautifulSoup: http://www.crummy.com/software/BeautifulSoup/

Otherwise, try HTML Tidy: http://tidy.sourceforge.net/

Re:Hacker News doesn't validate by weavejester on Wednesday January 07, 2009 @05:47AM (#423282)
Ian Bicking seems to consider lxml superior to BeautifulSoup: http://blog.ianbicking.org/2008/12/10/lxml-an-underappreciat...
Re:Hacker News doesn't validate by snprbob86 on Wednesday January 07, 2009 @09:35AM (#423440)
Cool! Thanks for this.
Re:Hacker News doesn't validate by eli on Wednesday January 07, 2009 @12:07AM (#423031)
Hpricot for Ruby: https://code.whytheluckystiff.net/hpricot/

But, yeah, Tidy does a darn job of turning just about any HTML into XHTML.

Re:Hacker News doesn't validate by icefox on Wednesday January 07, 2009 @12:07PM (#423635)
Funny story on that. For the Arora browser for a while I was pondering the best way to make a tool to import html bookmarks. I went a handful of different routes, but none of them were that good (I could import all my test data, but I wasn't sure I could input all users data) because of all the different html bookmark files that people generate, but I had an epiphany. I had at my fingertips webkit. So rather then trying to reinvent a bad html parser I just would just let webkit parse it and walk it with javascript.

The code ended up being something like

QWebView view; view.load("file://local/file/foo.html"); print view.mainFrame()->evaluateJavaScript(javascriptCode);

Full source here: http://github.com/Arora/arora/tree/master/tools/htmlToXBel

With QtWebKit working on all platforms and having a really nice api making things like this is a snap and easy to do.

Re:Hacker News doesn't validate by fizx on Tuesday January 06, 2009 @10:59PM (#422969)
libxml2 has an html parser. Everything(ish) has a libxml2 binding. The real PITA about scraping HN is that there's hardly any id/class attributes and you end up with td:nth-child(3).