News for hackers. Stuff that matters.
Re:YCRFS 3: Things Built on Twitter by ivankirigin on Monday October 26, 2009 @08:05PM (#904435)
You can read every tweet from a single machine. It's just a matter of getting acces to the steaming API.
Re:YCRFS 3: Things Built on Twitter by cperciva on Monday October 26, 2009 @08:27PM (#904474)
You can read every tweet from a single machine.

Sure, but the rate of tweets will determine how much processing you can do per tweet given a certain amount of computing power.

Re:YCRFS 3: Things Built on Twitter by ivankirigin on Tuesday October 27, 2009 @11:34AM (#905559)
The data input bottleneck is small. If you're going to do something in n*log(n), you should be fine. If you're going something in n^2, you're fucked anyway. Map Reduce is your friend. On that note, I only learned about Hive since coming to facebook. They made and open sourced it. Check it out: http://wiki.apache.org/hadoop/Hive
Re:YCRFS 3: Things Built on Twitter by Tichy on Monday October 26, 2009 @08:37PM (#904481)
If you manage to save them somehow (or distribute them to different machines), you can distribute the processing, I suppose.
Re:YCRFS 3: Things Built on Twitter by cperciva on Monday October 26, 2009 @09:13PM (#904522)
Exactly -- thus the question of how many machines would be needed to perform a certain amount of processing per tweet, which ultimately comes down to my original question of how many tweets there are.
Re:YCRFS 3: Things Built on Twitter by apgwoz on Tuesday October 27, 2009 @08:43AM (#905301)
I have decent access to the track/follow apis. I just queue the tweet right away and have a bunch of workers take care of it. Something like rabbitmq works wonderfully here.