Re:Ask HN: I need a brainstorm on how to collect several tho...
by jacquesm on Wednesday March 31, 2010 @12:50PM
(#1232311)
Tread carefully when spidering google, mail me for some more info on this.
Being IP banned from google is not funny.
Re:Ask HN: I need a brainstorm on how to collect several tho...
by scrollbar on Wednesday March 31, 2010 @01:24PM
(#1232418)
Ouch, the Google ban stick does not sound fun. An alternative for spidering high-risk avenues like that might be 80legs.com, in essence a legal/opt-in crawling botnet =P I've used them when they were in open beta and the service was free -- now you need to pay, although it's pretty cheap for smallish jobs.
karma++ for wget google news. I looked at feedage, but signal:noise ratio kinda sucks.