News for hackers. Stuff that matters.
Re:Ask YC: Any crawler experts out there? by vikram on Saturday January 26, 2008 @05:57AM (#104599)
I've been working on this for a little while now. You can definitely write a plug-in or pattern matching file for each site. Building a specific crawler for each website doesn't make sense.

The custom bits you need are then ones that fill the form and then extract the results. For scraping results ruby's Scrubyt is best as you can write templates for each type of page.

There are no comments on this comment yet.