Re:YCRFS 3: Things Built on Twitter
by ivankirigin on Tuesday October 27, 2009 @11:34AM
(#905559)
The data input bottleneck is small. If you're going to do something in n*log(n), you should be fine. If you're going something in n^2, you're fucked anyway. Map Reduce is your friend. On that note, I only learned about Hive since coming to facebook. They made and open sourced it. Check it out: http://wiki.apache.org/hadoop/Hive
If you manage to save them somehow (or distribute them to different machines), you can distribute the processing, I suppose.
Exactly -- thus the question of how many machines would be needed to perform a certain amount of processing per tweet, which ultimately comes down to my original question of how many tweets there are.
Sure, but the rate of tweets will determine how much processing you can do per tweet given a certain amount of computing power.