News for hackers. Stuff that matters.

Ask YC: How to web scrape 100s of sites/forms?

hi...

a potential research app requires the scraping of a few hundred websites/forms and diving into the child links to obtain the linked/parent structure ie. company->dept->title->name.

in this case, this would involve going 4 levels deep, and getting the required information.

so, does anyone know of a method/app/company that can be used to accomplish this. orm am i going to have to figure out how to get a number of cheap guys to write a bunch of python scripts!!

thanks

( 5 comments | on HN )
Re:Ask YC: How to web scrape 100s of sites/forms? by nreece on Wednesday August 13, 2008 @12:20AM (#274727)
...cheap guys to write a bunch of python scripts

You know what will be 'cheap'. Writing it yourself.

Re:Ask YC: How to web scrape 100s of sites/forms? by qhoxie on Wednesday August 13, 2008 @02:42AM (#274782)
Libraries like mechanize and hpricot are shrinking the curve for scraping tasks. That's not to say it is easy, but it should not take a bunch of people working on it. One good developer with proper experience would be ample in my opinion.
Re:Ask YC: How to web scrape 100s of sites/forms? by olefoo on Wednesday August 13, 2008 @01:00AM (#274749)
Or get one expensive guy to write you a script that

writes the scripts to scrape the sites by scraping the sites to read the structure to write the scripts to scrape the sites.

Re:Ask YC: How to web scrape 100s of sites/forms? by Anon84 on Wednesday August 13, 2008 @02:15AM (#274769)
Check out the search.wikia.org project. They make their crawler (and crawl data) available. Maybe you can get away with using theirs. That would really be cheap!
Re:Ask YC: How to web scrape 100s of sites/forms? by gaius on Wednesday August 13, 2008 @06:21AM (#274841)
I am unable to think of an application for this technology other than spamming. Care to provide more details before we shoot ourselves in the face by helping you?