Re:A Detailed Five Step Twitter Scaling Plan
by hank777 on Wednesday May 21, 2008 @02:02PM
(#196330)
Well perhaps its a matter of staring at something so long it becomes simple. But I really don't think what I am suggesting is that hard. The main thing that requires some thought is shard splitting. That is not a big piece of code, but it does admittedly take some thought to get right. Aside from that everything else is provided out of the box by someone else like amazon web services or terracotta. I am not sure how I could have been more precise without source code.
It is certainly reasonable to suggest that Twitter is good enough as it is because people still use it, though the grumbling in the market suggests that that may not be something they can bank on forever.
Re:A Detailed Five Step Twitter Scaling Plan
by KirinDave on Wednesday May 21, 2008 @02:17PM
(#196355)
One thing you don't address: where do messages go while a resource is self-splitting? How much redundancy do you build in to prevent this from being a problem? How much backlog do you expect to handle?
And have you checked latency serving out of EC2? Our experience is that the connections are fairly fast but the latency establishing the connection is significant, especially from the west coast where twitter is based.
Re:A Detailed Five Step Twitter Scaling Plan
by hank777 on Wednesday May 21, 2008 @02:41PM
(#196389)
I addressed this in another answer here but happy to do it again. The way shard splitting works is that the initial shard keeps working while the new shard is being built. During this time, all messages to the old shard are written to the old shard and the new shard, so, unless I don't understand your question I dont see there being a backlog.
Regarding EC2, latency isnt really an issue because Twitter is far from real-time. That said, haven't really noticed any problems with EC2 connection latency, but we are not live yet so I would defer to you on that.
As I read your design, one of the biggest problems is that to build up the infrastructure I think (and infer based of my experience with designing these kinds of systems) you are describing is that it's a massive engineering effort in and of itself. You basically end up rewriting a realtime-optimized BigTable or HBase, and that's much to much to expect of Twitter's small engineering team.
People seem to forget that scalability is a tradeoff, like any other aspect of a service. You devote an amount of resources to the problem that makes sense to your business. Despite twitter's downtimes, people don't seem to be leaving the service, despite several competitors. The service is down, but it isn't down so much that it's unusable.
For a site that's grown at such a rapid pace and has such a small engineering team, I think Twitter's scaling plan is probably very reasonable. They're growing their capacity at a rate they can afford, and a rate that they know will retain most of their userbase.