Ask HN: How to aggregate product info from other websites

Re:Ask HN: How to aggregate product info from other websites by DenisM on Sunday November 09, 2008 @08:54PM (#358735)

I use python to write scripts of this nature (one script so far:)).

Python has SGML SAX parser and since HTML is SGML it can be used. Better than regexps any day.

Python's http client library also supports cookies so that you can pretend to have a "session" with your target website.

EDIT: the libraries are urllib2, sgmllib, cookielib

[ Parent | on HN ]

Re:Ask HN: How to aggregate product info from other websites by olegp on Sunday November 09, 2008 @09:46PM (#358782)

Very few pages have well formed mark-up. The few large scraping projects I've seen have started out with a mark-up based approach and then switched to regular expressions.

What experiences has everyone else had?

[ Parent | on HN ]

Re:Ask HN: How to aggregate product info from other websites by Harkins on Sunday November 09, 2008 @11:40PM (#358877)

I've had a lot of success with BeautifulSoup. It turns terrible HTML into a usable DOM tree.

[ Parent | on HN ]

Re:Ask HN: How to aggregate product info from other websites by joseakle on Monday November 10, 2008 @12:24AM (#358911)

i am using beatiful soup, it works with malformed markup

[ Parent | on HN ]

Re:Ask HN: How to aggregate product info from other websites by DenisM on Sunday November 09, 2008 @10:26PM (#358822)

Uhm. I though any valid HTML is also valid SGML? Are you sure you're not confusing it with XML markup?

[ Parent | on HN ]

Re:Ask HN: How to aggregate product info from other websites by ryanwaggoner on Sunday November 09, 2008 @10:32PM (#358826)

I think he's saying that the HTML-parsing approach only works when the HTML is well-formed and for most sites, it isn't.

[ Parent | on HN ]

Re:Ask HN: How to aggregate product info from other websites by jreilly on Sunday November 09, 2008 @09:26PM (#358771)

Thanks for the input. I am currently learning rails so also wondering if there are any libraries that will make this significantly easier

[ Parent | on HN ]