I get what you are saying, seems like you want a write-through cache. Sure seems that you are in the right ball park for a solution, but may have to write a bit of application code layered to do what you want with those parts.

It would probably be helpful if you broke up the read/writes estimates. Also, are those writes mostly inserts or updates? There are different types of solutions depending one what you expect your profile to be like.

Also are you reads just a key look up or are you doing some sort of aggregation/searching across all records?

Seems like you may be prematurely optimizing your solution and underestimating the capabilities of database engines. Most of them can probably handle the load you are thinking with correct usage and sufficient tuning and hardware. For example, I believe twitter uses MySQL:

http://highscalability.com/scaling-twitter-making-twitter-10...

Also are you reads just a key look up or are you doing some sort of aggregation/searching across all records?

Please also estimate your overall size demands. Manipulating 2GB of data is different than 200GB. :)

I estimate Reads to be 10x more frequent than writes and Reads would mostly be a key lookup, with occassional aggregation.

Yes, I was also wondering if anybody has demonstrated sclability with MySQl for this level, as I won't have to fiddle around with code too much.

I think you should read up on using Memcached with MySQL.

Here's a good starting point: http://highscalability.com/bunch-great-strategies-using-memc...

At this point in the game (it sounds early in your development process), you aren't painting yourself in a corner (the only thing you should be worried about here IMHO) if you go with these two, but you will probably have to refine things later on to get your scalability (again, that's true with any solution you pick).

Also realize that Twitter probably has a larger hardware budget than you do and real scalability costs real money in both hardware and effort. It's been widely reported that many of their engineers have been spending considerable effort keeping things stable and improving scalability. Another performance related mantra is that "to go faster, do less". This is definitely true here... there may be things you think you need to store, but don't. Removing that effort could have a profound impact on performance.

I suppose its worth mentioning that you may want to check out AWS and their storage solutions (and push the performance problems off to them). This decision also has other business factors (like can you deal with potential outages, potential security problems, cost, etc...).

Let's stop there. "I haven't tested" translates to "premature optimization". Stop whatever you're doing and measure your task.

> I have explored ... but none is close to what I have in mind.

It's not surprised that you can't find a perfect solution to your imaginary problem. Find out exactly what your issue is first, then optimize. And don't expect to perfectly engineer a system from the ground up. It's OK to layer in new systems to optimize real bottlenecks in your application.

To be completely honest, if memcache, memcachedb, redis, and tokyo cabinet are not getting the job done then you are probably doing things very, very wrong...

I am not saying that they are not getting the job done. They would get the job done but with significant code changes. I was just exploring if there was a better alternative. If I don't find one, I would resort to memcache most probably.

Note that Redis 0.100 adds non blocking replication, so you can connect a slave at runtime while the master continues to happily serve queries. Also we have a SLAVEOF command now that makes it possible to control replication at runtime: to maka a running server a replica of some other server, to stop the replication and turn a slave into a master, and so on.

Redis replication is trivial to setup.

You may try to load a dataset larger than your memory in Redis and see what happens

This is what threw me off when I evaluated redis.

When choosing a database then I really don't want to "try and see what happens". I want defined and documented behaviour, please.

Anyway this is an edge case, it's not a good idea even if it will work to have datasets bigger than available RAM.

Before 1.0 stable we are even introducing 'maxmemory' config option. If the dataset starts to get bigger than 'maxmemory' Redis will try to free ram removing keys with a timeout set (starting from older ones), cached objects in free lists, and so on. If still it is out of memory will start to reply only to read-only operations and will issue "-ERR using more usage bigger than maxmemory parameter" error if you try to write more data.

Btw the way to go is maxmemory, that will be in the next tar.gz and in days into Git. maxmemory also allows to use Redis with a memcached semantic of volatile keys expiring to make room for the new ones when we are low on memory.

Voldemort combines in memory caching with the storage system so that a separete caching tier is not required (instead the storage system itself is just fast.)

Or you could look at Oracle Coherence/TimesTen or [kx] http://kx.com/.

http://www.monash.com/MCDM.pdf

http://www.mcobject.com/November10/2008