Re:Rainbird: The Way Twitter Counts Tweets In Realtime (Soon...
by afex on Saturday February 05, 2011 @04:14AM
(#2182795)
As far as datastores go, cassandra is write-optimized. Writes are faster than reads. So for this use case (heavy denormalization) it is a good fit.
Also, Cassandra reduces the operational complexity of having a logical store which spans multiple hosts. Your memcache example does not get persistence for free, and sharding mysql is something you have to do manually. The interface to a Cassandra cluster is the same regardless of how many nodes you are running.
Scaling writes is "hard". Incrementing counters is obviously write heavy, and cassandra aims to make it easier.
A decent memcached instance on modern hardware can easily push several 100K updates/sec . Couldn't you do the same with a pool of sharded memcached servers? Compute the MD5 hash of the string you want to count (which can be 16 strings, from the top-rated comment above), and just use that as the key.
We've seen people push 750K QPS on an InnoDB via HandlerSockets http://news.ycombinator.com/item?id=1886137 , so imagine what you could do with a sharded pool of 20 InnoDB servers.
Again: if I'm missing something, I'd love to learn.