Re:Ask HN: Best framework for parsing thousands of feeds?
by swanson on Friday February 04, 2011 @01:32PM
(#2180529)
You might want to take a look at Samuel Clay's NewsBlur project: https://github.com/samuelclay/NewsBlur and see how he handles this problem.
[ Parent
| on HN ]
Re:Ask HN: Best framework for parsing thousands of feeds?
by conesus on Saturday February 05, 2011 @01:48AM
(#2182624)
As others have said, task queues (look into Celery, which is in Python), Mark Pilgrim's feedparser, and make sure you don't fetch more often than you need to. A few thousand feeds is fine, but if you grow in the hundreds of thousands of feeds, if you want to update them more than once a day, you're going to have to pass in the cache controls (etags and last-modified dates). Even then, 50,000+ feeds is straining the limits of a single DB.
If you were to reach that point, then I recommend moving to a nosql db (like mongo, which is what I use for NewsBlur), and sharding it so you can read/write to more feeds without a problem. All of your analysis will have to be in a MapReduce, so it can be sent to different shards, but that's not too difficult to learn how to do. NewsBlur has a few examples of how to do this.
I used Python, but Ruby would also be a fine choice. Although I'm not sure what Ruby libraries you would use.