News for hackers. Stuff that matters.
Re:A Detailed Five Step Twitter Scaling Plan by xirium on Wednesday May 21, 2008 @11:48AM (#196129)
The suggested plan seems reasonable. Yes, if you denormalise users inbound and outbound messages then you store the data twice. Yes, shards give almost limitless scalability. However, could it be that during peak load, the number of messages to be written plus indexes to be maintained exceeds the total available disk seek operations?
Re:A Detailed Five Step Twitter Scaling Plan by wmf on Wednesday May 21, 2008 @12:20PM (#196168)
So shard more, or get more disks, or get flash or whatever. (Yes, I don't know what I'm talking about.)
Re:A Detailed Five Step Twitter Scaling Plan by xirium on Wednesday May 21, 2008 @12:52PM (#196212)
> So shard more, or get more disks, or get flash

Your intuition is excellent. These are all valid techniques to alleviate disk seek limitations. They are likely to be successful. Unfortunately, they're expensive options.

You can partially overcome disk seek limitations by performing database batch inserts. So, rather than inserting messages indiviually and updating indexes for each message, you insert 100 messages and perform less disk seeks updating the indexes for each batch. Unfortunately, it can be hard to reliably merge data into suitable batches.

Re:A Detailed Five Step Twitter Scaling Plan by wmf on Wednesday May 21, 2008 @02:56PM (#196410)
Hank's proposed solution is expensive; we should just admit that up front. But is it more expensive than being down and fighting fires all the time?
Re:A Detailed Five Step Twitter Scaling Plan by xirium on Wednesday May 21, 2008 @04:44PM (#196536)
The economics of downtime is a very important consideration. I read that three minutes of downtime reduces the value of a website by 1%. However, that may be a figure bandied by vendors of high-availability equipment. It is also generally acknowledged that high-availability is achieved through strict processes and sheer redundancy of hardware. Therefore, it gets exponentially more expensive to minimise downtime.

So, you've got two non-linear curves, cost and benefit. If you've got an exceptional team, a great architecture and good finances then getting 99.999% uptime should be easy. Of course, this assumes that you don't have unforeseen circumstances, glaring omissions in redundancy - or exponential growth and utilisation without a revenue stream.

Even with generous financing, exponential growth can be deadly. A small but sustained increment in load can be enough to cause a backlog in requests that never recovers. If you're growing fast then it can occur anywhere in your system at any time. I've been a DBA for a renderfarm with thousands of cores and this situation is stressing.

We've also had this situation while working on search. Thankfully, it occurred on a smaller scale and most thankfully it occurred before launch. We had two database servers, two app servers, no redundancy, on unreliable hardware. We didn't make much headway with efficiency improvements because long tests were likely to encounter hardware failure. It was demoralising.

This slowed the development and testing of a database failover feature in the database wrapper. Now this has been implemented, progress is much better. A test that previously took three hours now takes five minutes and we've had much more time to migrate to reliable hardware and introduce redundancy. We've also got a setup which can be used to test resilience to real modes of failure.

Re:A Detailed Five Step Twitter Scaling Plan by wmf on Wednesday May 21, 2008 @06:03PM (#196606)
My guess is that Twitter's downtime is due to overload, not hardware failures. Thus a scalable system (even one with a high slope) reduces the overload/downtime problem to "buy more servers -- now!" instead of the "analyze bottleneck, rewrite code" loop that it sounds like Twitter is in now.
Re:A Detailed Five Step Twitter Scaling Plan by nostrademons on Wednesday May 21, 2008 @12:58PM (#196222)
You don't store it twice - you store it hundreds of times, once for each recipient. Twitter isn't person-to-person messaging, it's broadcast.