News for hackers. Stuff that matters.

A comment on A Detailed Five Step Twitter Scaling Plan. Parent comment

Re:A Detailed Five Step Twitter Scaling Plan by KirinDave on Wednesday May 21, 2008 @02:17PM (#196355)
One thing you don't address: where do messages go while a resource is self-splitting? How much redundancy do you build in to prevent this from being a problem? How much backlog do you expect to handle?

And have you checked latency serving out of EC2? Our experience is that the connections are fairly fast but the latency establishing the connection is significant, especially from the west coast where twitter is based.

[ Parent | on HN ]
Re:A Detailed Five Step Twitter Scaling Plan by hank777 on Wednesday May 21, 2008 @02:41PM (#196389)
I addressed this in another answer here but happy to do it again. The way shard splitting works is that the initial shard keeps working while the new shard is being built. During this time, all messages to the old shard are written to the old shard and the new shard, so, unless I don't understand your question I dont see there being a backlog.

Regarding EC2, latency isnt really an issue because Twitter is far from real-time. That said, haven't really noticed any problems with EC2 connection latency, but we are not live yet so I would defer to you on that.

[ Parent | on HN ]