Re:Full Text RSS Feed: Get the whole feed and nothing but th...
by lurchpop on Wednesday March 02, 2011 @04:02AM
(#2278914)
how do you handle scale like that? Do you have a hadoop cluster or something? How many concurrent do you download?
[ Parent
| on HN ]
Re:Full Text RSS Feed: Get the whole feed and nothing but th...
by nicpottier on Wednesday March 02, 2011 @07:28AM
(#2279313)
Amazon EC2, MongoDB, S3. The EC2 instances scale with how many stale feeds we have, but it is usually less than 2.
Just checked and we have ~25k feeds in the system, though not all are deep harvesting as we call it.
Note we do a few things over just extracting the full content as well, we also try to grab out images and create a pleasing thumbnail using face detection etc.. So that probably slows things down a good deal as well.