Re:Ask YC: Any crawler experts out there?
by bprater on Friday January 25, 2008 @05:45PM
(#104368)
If you need to do form stepping, you need to look at something like Perl's Mechanize package. (Ruby has one too.)
Spend time reading articles related to Mechanize. Your resulting code is going to be fairly terse, so you don't really need to spend much time worrying about making a generic crawler.
Spend time reading articles related to Mechanize. Your resulting code is going to be fairly terse, so you don't really need to spend much time worrying about making a generic crawler.