News for hackers. Stuff that matters.
Re:Rainbird: The Way Twitter Counts Tweets In Realtime (Soon... by ajays on Saturday February 05, 2011 @02:19AM (#2182668)
I'm missing the point, so I hope the HN community can enlighten me, but: why do you need Cassandra and all that jazz, if you're just incrementing counters?

A decent memcached instance on modern hardware can easily push several 100K updates/sec . Couldn't you do the same with a pool of sharded memcached servers? Compute the MD5 hash of the string you want to count (which can be 16 strings, from the top-rated comment above), and just use that as the key.

We've seen people push 750K QPS on an InnoDB via HandlerSockets http://news.ycombinator.com/item?id=1886137 , so imagine what you could do with a sharded pool of 20 InnoDB servers.

Again: if I'm missing something, I'd love to learn.

Re:Rainbird: The Way Twitter Counts Tweets In Realtime (Soon... by afex on Saturday February 05, 2011 @04:14AM (#2182795)
As far as datastores go, cassandra is write-optimized. Writes are faster than reads. So for this use case (heavy denormalization) it is a good fit.

Also, Cassandra reduces the operational complexity of having a logical store which spans multiple hosts. Your memcache example does not get persistence for free, and sharding mysql is something you have to do manually. The interface to a Cassandra cluster is the same regardless of how many nodes you are running.

Scaling writes is "hard". Incrementing counters is obviously write heavy, and cassandra aims to make it easier.

Re:Rainbird: The Way Twitter Counts Tweets In Realtime (Soon... by saurik on Saturday February 05, 2011 @04:16AM (#2182798)
The day half your memcached data center loses power will be a sad sad day indeed, and the story of 750k QPS on InnoDB was about read traffic, not writes.