80legs sets its web crawler free

Re:80legs sets its web crawler free by icey on Tuesday December 22, 2009 @08:43AM (#1009859)

I'm building something right now that includes page parsing, and so far I've only been building in regex support. I like your jQuery selector idea as well, are there any other ways that you can think of that would make searching the contents of a page programmatically easier for you?

[ Parent | on HN ]

Re:80legs sets its web crawler free by qeorge on Tuesday December 22, 2009 @10:29AM (#1010038)

May I suggest taking a look at Parsely? Its the syntax they use on www.parselets.com. The documentation for implementing it in your own apps is a little sparse, but the data format is awesome. Here's one that describes scraping HN:

http://parselets.com/parselets/yc/14

Might not be a fit for your project, but in terms of describing parsing instructions to a crawler its the best format I've ever seen.

[ Parent | on HN ]

Re:80legs sets its web crawler free by icey on Tuesday December 22, 2009 @10:56AM (#1010097)

I'm not crawling, but that is pretty interesting looking. I'll bookmark it and take a look at it for later for sure - thanks!

[ Parent | on HN ]

Re:80legs sets its web crawler free by asher on Tuesday December 22, 2009 @11:45AM (#1010188)

Hpricot for Ruby is great.

For instance, parsing a Google search results page:

        (doc/"a.l").each do |link|
            label = link.inner_text
            href = link.attributes['href']
            ...

[ Parent | on HN ]