News for hackers. Stuff that matters.
Re:MongoDB's lead developer: Foursquare outage post mortem by jasonwatkinspdx on Thursday October 07, 2010 @07:19PM (#1769913)
Facebook runs primarily out of ram via memcached. The last numbers I'm aware of were that they had about 200TB in memcache capacity [1]. They use a variety of data stores, but primarily sharded mysql. I don't have recent numbers there, but they were above 1000 master-master pairs as of 2008.

While buying that much ram sound costly, that's only looking at capacity. Assuming typical 1u servers and common pricing at the moment, you're paying roughly $1k in capital for each 10GB of ram capacity. However, each of these servers gives you a couple hundred thousand random reads per second. To duplicate that with spinning hard drives would take several hundred spindles at least. SSD's are better, but still, it ends up being a lot of devices to duplicate that io capacity.

This is why virtually everyone at stupendous scale (google, facebook, etc) ends up with very ram centric architectures.

There's a simple way to decide how much of what storage you need [2]. Look at the distribution of access times. With current technology, in very rough terms, if an item is accessed more than once a day, it's more cost effective to store it on SSD. If it's accessed more than once in an hour, it's more cost effective to store it in ram.

[1]: http://perspectives.mvdirona.com/2010/07/01/Velocity2010.asp...

[2]: http://www.cs.cmu.edu/~damon2007/pdf/graefe07fiveminrule.pdf

Re:MongoDB's lead developer: Foursquare outage post mortem by rbranson on Thursday October 07, 2010 @09:25PM (#1770231)
Anything that is both latency sensitive (web apps are) and requires high throughput is going to be RAM-centric. Even services operating at much smaller scales with "old school" single DB boxes need to keep the vast majority of active data in RAM and use things like battery-backed write caches to get acceptable write throughput. Even with the sophisticated tricks mature RDBMS software uses to squeeze every ounce of performance out of spindles, most "old school" DBAs will tell you that to get high throughput for a database you basically need to have RAM = database size.
Re:MongoDB's lead developer: Foursquare outage post mortem by trezor on Friday October 08, 2010 @02:39AM (#1770856)
I have had very acceptable performance for latency critical applications where the db have exceeded the available memory by a factor between 10 and 50.

So far I have never heard of any one running a commercial RDBMS reiterate the MySQL-mantra that you need the entire DB in RAM and I find it a very puzzling attitude to efficient database usage.

Re:MongoDB's lead developer: Foursquare outage post mortem by steveklabnik on Saturday October 09, 2010 @11:56AM (#1775190)
While you may just be trying to talk-down on the MySQL dudes, I don't think that "RAM is faster than disk, so you're more performant if you can fit your data in RAM" is really all that puzzling.
Re:MongoDB's lead developer: Foursquare outage post mortem by trezor on Sunday October 10, 2010 @09:03AM (#1777038)
I'm not debating that RAM is faster than disk. I'm debating that you need to have the entire DB in RAM to get good performance.

Most of the time, less than 10% of your DB represents the active working-set of your data and a good database-system should be able to analyse what is being used, gather statistics about the data, use those statistics to intelligently optimize your queries and minimize the need for disk IO.

For any decent RDBMS, having to match total memory with the database size, is wasting money on RAM for very little extra gain in performance and this doesn't really make much business sense.

A concrete example would be a server I manage. It has 16GBs of RAM and handles around 5000 GBs worth of DBs. Due to a good RDBMS and intelligent caching and use of statistics, that server has a cache-hit ratio of 98%. That means that only 2% of queries entering the system results in actual disk IO.

In order to get the last 2% of queries to get into RAM, I would have to increase the system memory by a factor of 30. At that point just getting another server and setting up replication would be much cheaper and also allow a theoretical doubling of troughput.

Really. Having the DB all in RAM is not something which makes marginal sense at all when you compare it to the costs it involves. No amount of internet-argument can convince me that this goal is anything besides a pure waste of money.

Re:MongoDB's lead developer: Foursquare outage post mortem by jasonwatkinspdx on Tuesday October 12, 2010 @06:53AM (#1782926)
There's a reason I mentioned object inter-access times. To properly size storage you need to know your query distribution.

"It has 16GBs of RAM and handles around 5000 GBs worth of DBs. Due to a good RDBMS and intelligent caching and use of statistics, that server has a cache-hit ratio of 98%."

This may be your query distribution but it plainly is not everyone's.

"No amount of internet-argument can convince me that this goal is anything besides a pure waste of money."

If Jim Gray's published work doesn't convince you there's not really much left to discuss.