Re:What New Twitter Employees Do For The First Week
by jacques_chester on Monday February 07, 2011 @11:59PM
(#2192012)
In the research literature this is called SEDA: http://en.wikipedia.org/wiki/Staged_event-driven_architectur...
So with a good design you have groups of servers doing the different stages in the queue.
One you've got the pattern down for 100 tweets per second, the pattern should be reproducible by scaling servers in each queue to 1000 tweets per second, and eventually 10,000 tweets per second.
The database requirements may explain why it's all done in one datacenter instead of trying to do replication across the country/world.