News for hackers. Stuff that matters.
Re:Ask HN: I need a brainstorm on how to collect several tho... by jacquesm on Wednesday March 31, 2010 @12:30PM (#1232243)
Spider feedburner ?

wget -r google news and see what their sources are ?

http://www.feedage.com/categories/News/1/4

Re:Ask HN: I need a brainstorm on how to collect several tho... by iamelgringo on Wednesday March 31, 2010 @12:35PM (#1232261)
Feedburner: I've thought about this one, but I haven't been able to find a good starting point. Their vast collection of RSS feeds don't seem to cross reference nicely and lend themselves to an obvious spidering strategy like web pages do. Suggestions on how to start are welcome

karma++ for wget google news. I looked at feedage, but signal:noise ratio kinda sucks.

Re:Ask HN: I need a brainstorm on how to collect several tho... by jacquesm on Wednesday March 31, 2010 @12:50PM (#1232311)
Tread carefully when spidering google, mail me for some more info on this.

Being IP banned from google is not funny.

Re:Ask HN: I need a brainstorm on how to collect several tho... by scrollbar on Wednesday March 31, 2010 @01:24PM (#1232418)
Ouch, the Google ban stick does not sound fun. An alternative for spidering high-risk avenues like that might be 80legs.com, in essence a legal/opt-in crawling botnet =P I've used them when they were in open beta and the service was free -- now you need to pay, although it's pretty cheap for smallish jobs.