News for hackers. Stuff that matters.
Re:Full Text RSS Feed: Get the whole feed and nothing but th... by nicpottier on Wednesday March 02, 2011 @07:28AM (#2279313)
Amazon EC2, MongoDB, S3. The EC2 instances scale with how many stale feeds we have, but it is usually less than 2.

Just checked and we have ~25k feeds in the system, though not all are deep harvesting as we call it.

Note we do a few things over just extracting the full content as well, we also try to grab out images and create a pleasing thumbnail using face detection etc.. So that probably slows things down a good deal as well.

There are no comments on this comment yet.