News for hackers. Stuff that matters.
Re:Full Text RSS Feed: Get the whole feed and nothing but th... by nicpottier on Wednesday March 02, 2011 @01:24AM (#2278662)
We built a backend similar to this for our NewsRoom mobile client. (Android and Pre) Actually used some genetic algorithms to do the training for our content extraction, one of the more fun projects I've done.

Word of warning, if it takes off, you basically start turning into someone who is both caching and harvesting the web every 15 minutes. There is an incredibly long tail on RSS feeds and it starts killing you to keep them all up to date. Storing and serving it is no big deal, but harvesting actually turns into real money when you figure out total bandwidth used. (we harvest about ~30,000 feeds every 15 minutes)

Re:Full Text RSS Feed: Get the whole feed and nothing but th... by tianyicui on Wednesday March 02, 2011 @09:02AM (#2279545)
I really wish one day Google will sell a page harvesting service. It can certainly profit since the cost is neglectable. But how big the market is?
Re:Full Text RSS Feed: Get the whole feed and nothing but th... by lurchpop on Wednesday March 02, 2011 @04:02AM (#2278914)
how do you handle scale like that? Do you have a hadoop cluster or something? How many concurrent do you download?
Re:Full Text RSS Feed: Get the whole feed and nothing but th... by nicpottier on Wednesday March 02, 2011 @07:28AM (#2279313)
Amazon EC2, MongoDB, S3. The EC2 instances scale with how many stale feeds we have, but it is usually less than 2.

Just checked and we have ~25k feeds in the system, though not all are deep harvesting as we call it.

Note we do a few things over just extracting the full content as well, we also try to grab out images and create a pleasing thumbnail using face detection etc.. So that probably slows things down a good deal as well.