News for hackers. Stuff that matters.

A comment on Ask HN: How to aggregate product info from other websites. Parent comment

Re:Ask HN: How to aggregate product info from other websites by olegp on Sunday November 09, 2008 @09:46PM (#358782)
Very few pages have well formed mark-up. The few large scraping projects I've seen have started out with a mark-up based approach and then switched to regular expressions.

What experiences has everyone else had?

[ Parent | on HN ]
Re:Ask HN: How to aggregate product info from other websites by Harkins on Sunday November 09, 2008 @11:40PM (#358877)
I've had a lot of success with BeautifulSoup. It turns terrible HTML into a usable DOM tree.
[ Parent | on HN ]
Re:Ask HN: How to aggregate product info from other websites by joseakle on Monday November 10, 2008 @12:24AM (#358911)
i am using beatiful soup, it works with malformed markup
[ Parent | on HN ]
Re:Ask HN: How to aggregate product info from other websites by DenisM on Sunday November 09, 2008 @10:26PM (#358822)
Uhm. I though any valid HTML is also valid SGML? Are you sure you're not confusing it with XML markup?
[ Parent | on HN ]
Re:Ask HN: How to aggregate product info from other websites by ryanwaggoner on Sunday November 09, 2008 @10:32PM (#358826)
I think he's saying that the HTML-parsing approach only works when the HTML is well-formed and for most sites, it isn't.
[ Parent | on HN ]