News for hackers. Stuff that matters.
Re:Ask YC: Any ideas about intelligent crawlers :) by spoonyg on Tuesday January 08, 2008 @03:30PM (#96101)
I have built up something similar to what you are describing and it was a fun project. My first reaction to #1 is that if the information you want to repeatably in the same place you are probably better off just doing things manually and not going to the trouble of building a visual tool. In my experience the interesting pieces of data tend to move around and something like regex is the best way to handle this. I used wget to grab data because it was quick and easy. I then did the post processing in the background, separating out the grabbing of data and the interpretation of data.

There are no comments on this comment yet.