I use memcachedb in my deployment of A/Bingo (it isn't limited to that), which avoids using a SQL database because I wanted it to be blazing fast for people who having significantly higher traffic sites than I do. The typical case performance on my site is fantastic. The worst case performance is an abomination -- when it flushes the site essentially pauses for all users for five seconds.
This is tolerable for me, but only just. The reason I use memcachedb as opposed to vanilla memcached is because I was worried about persistence in the event of a server failure, but given that I can count resets of my server over 3.5 years on a single hand, I might just decide "In the event of a server failure, I lose any A/B tests in progress and have to start over. Oh well!"
That's why there are so many distributed storage systems: there's no "one fits all" solution that can handle every theoretical corner case (even Google hasn't solved that). In my (biased) view, the eventually consistent stores (e.g. Dynamo-inspired ones or even "Friendfeed/Facebook model" of sharded and replicated MySQL databases storing Blobs) do seem most reasonable for web-type problems.
Instead of VM Reddit should use Redis EXPIRE I guess, that is, time to live in cached keys so they auto expire.
Btw for people that don't know what Redis Virtual Memory is: with VM Redis is able to swap out rarely used keys in disk. This makes a lot of sense when using Redis as a DB. When using Redis as a cache, the way to go is EXPIRE: rarely used things in cache should simply go away and be expired instead of being moved into disk.
EDIT: It would be very interesting to know where the Reddit performance problem is, but Redis Sorted Sets are a very good match to create social-news alike sites like HN or Reddit in a scalable distributed way, with a few workers processing recent news to update their score into the sorted set. The home page can be generated with ZREVRANGE without any computation.
It's a shame reddit is not sharing how this cache is used.
Cassandra would be a better choice.
/cassandra committer, but it would :)
(Also, I'm not sure when you were looking at it, but bootstrap -- adding nodes without any downtime -- is done now.)
Seriously?
Wow.