Hank, your design is so high level as to be almost useless to anyone who'd actually benefit from a proposed design. It's not clear what issues you feel Twitter has, or how exactly this architecture would solve them. Usually your blog posts are more cooked than this one. I'm a regular reader, and at first I honestly thought your post was satirical.

As I read your design, one of the biggest problems is that to build up the infrastructure I think (and infer based of my experience with designing these kinds of systems) you are describing is that it's a massive engineering effort in and of itself. You basically end up rewriting a realtime-optimized BigTable or HBase, and that's much to much to expect of Twitter's small engineering team.

People seem to forget that scalability is a tradeoff, like any other aspect of a service. You devote an amount of resources to the problem that makes sense to your business. Despite twitter's downtimes, people don't seem to be leaving the service, despite several competitors. The service is down, but it isn't down so much that it's unusable.

For a site that's grown at such a rapid pace and has such a small engineering team, I think Twitter's scaling plan is probably very reasonable. They're growing their capacity at a rate they can afford, and a rate that they know will retain most of their userbase.

Well perhaps its a matter of staring at something so long it becomes simple. But I really don't think what I am suggesting is that hard. The main thing that requires some thought is shard splitting. That is not a big piece of code, but it does admittedly take some thought to get right. Aside from that everything else is provided out of the box by someone else like amazon web services or terracotta. I am not sure how I could have been more precise without source code.

It is certainly reasonable to suggest that Twitter is good enough as it is because people still use it, though the grumbling in the market suggests that that may not be something they can bank on forever.

One thing you don't address: where do messages go while a resource is self-splitting? How much redundancy do you build in to prevent this from being a problem? How much backlog do you expect to handle?

And have you checked latency serving out of EC2? Our experience is that the connections are fairly fast but the latency establishing the connection is significant, especially from the west coast where twitter is based.

I addressed this in another answer here but happy to do it again. The way shard splitting works is that the initial shard keeps working while the new shard is being built. During this time, all messages to the old shard are written to the old shard and the new shard, so, unless I don't understand your question I dont see there being a backlog.

Regarding EC2, latency isnt really an issue because Twitter is far from real-time. That said, haven't really noticed any problems with EC2 connection latency, but we are not live yet so I would defer to you on that.

I don't use Twitter, but as I understand it, it's a massively many-to-many messaging service, where it's not unusual to follow hundreds of people's tweets and have thousands of people follow yours. Can you imagine the performance if every message required hundreds of disk writes as the "received" copies get spread across a hundred shards?

Actually, this is the beauty of the design: All of the writes happen in parallel. So there if you write to 2000 recipients, but all of the writes happen in parallel you have no bottleneck. And any time a given machine's load gets too heavy, you do a shard split. You really dont care how much writing you do as long as on every machine you write to is not being hit too hard. So if a given server has lets say 10,000 users on it. And each one of those users receives 10 updates an hour. Then it would mean handling, lets say 100,000 writes in an hour, or 27 writes a second. If this was the threshold for this machine (which would be low) you then split the shard so that it then is only handling 13 or 14 writes per second.

So, it's scalable, but the total number of writes/sec necessary is O(messages sent/sec * average number of recipients per message). I could see this working as long as the average number of recipients per message remains constant as the service grows, though it's still a lot of hardware. How much funding does Twitter have again?

If the average number of recipients grows along with the number of users, then there's a problem, because average number of messages should be growing linearly with userbase, so the total hardware required is quadratic. No way that would work out economically...

I think your math is off here.

Basically you end up with this equation

cost = ((NumOfUsers * avgMsgsRevceivedPerUser)/maxWritesPerCPU) * costOfCPU

There is nothing quadratic here. Your are able to scale linearly with the number of messages received in the system. There is a fixed cost per message received which is exactly what you want. By the way, this is exactly how the internet mail system works and it is why it is so scalable.

It's NumOfMessages, not NumOfUsers, and Messages = Messages/User * Users. Then the question is whether Messages/User and Recipients/Message remain constant as NumOfUsers goes up. If both do, then your total cost is linear with the number of users, which is what you want. But if either of them is linear in the number of users, then the total number of messages becomes quadratic in the number of users: a linear factor from growth in userbase, and a linear factor factor from increased usage by each user. Most likely there's a logarithmic correlation (Recipients/Message goes up as more people are on the service, but not by as much as total users), which is still generally fine for scalability concerns.

E-mail's actually a really great example. It worked great as long as it was used for person-to-person communication, which held both the number of messages/user and the number of recipients/message constant. Then spammers found out that the more people who had e-mail, the more they could send to at once, and sent recipients/message through the roof. In 1997/98, there were actually a number of articles about how the e-mail architecture was in serious trouble, and unless some way could be found to limit spam, it would collapse within a couple years. Of course, they dealt with it through spam filters and blacklists on the server, so now e-mail works fine.

Twitter's problem is that both Messages/User and Recipients/Message average significantly higher than e-mail, and yet the resources they have available are significantly lower than the full resources of the Internet. So even if they don't have an architectural scalability problem, they could very well have an economics problem.

I don't understand your correction, but my lack of understanding is really immaterial. I actually agree that twitter has a problem with economics if they cant sell anything. Whether it is worth solving the scalability problem is a good question. Certainly if they keep growing with no revenue, at some point it is definitely an economic problem. Indeed I posted a while ago about the community taking over twitter as a distributed public service (like email). I do think that there is an upper limit to the amount of messages the average user will send.

To do the writes requires a lookup. To do a read requires a lookup. That's how sharding works. If not a lookup, then a hash.

This doesn't even get into the complexity of scaling your model from 1,000 or 10,000 users where your brute-force methods seem like they will work and applying it to 10,000,000 users where your model will die a horrible death, kinda like Twitter is today.

uhh... yes it requires a lookup. There is no problem with that since the lookup is not centralized but local. How... read about terracotta. No I dont work for the company just use their open source product.

"Given that I have spent the last several years studying/developing distributed databases..."

That doesn't make your opinion of what's wrong with Twitter any more valid than the crap TC keeps printing unless you happen to also have lots of internal details of the Twitter architecture.

true, but it makes me curious what brilliance someone bring to the table when they have nothing specific to say other than you are wrong. Also given that much of what I am talking about is running code, I at least know that it solves a major class of problems. If it doesnt apply to twitter it should be pretty easy to explain why.