News for hackers. Stuff that matters.
Re:Scaling with MongoDB (or how Urban Airship abandoned it f... by lifeisstillgood on Wednesday June 22, 2011 @08:18PM (#2686048)
Forgive the dumb question but what do you mean by a working set? Is this the part of index that covers the shard that server is on? Or is your domain such that you only care about most recent records - for my part it's rare that a table has records I will not want to see and so can be in the non ram part of index?

I think a reread of the op suggests that only a subset of data and indexes are in ram - but my question still stands but slightly differently phrased - are you saying best practises is shard your dbase and place shards in ram in as many servers as needed?

Re:Scaling with MongoDB (or how Urban Airship abandoned it f... by FooBarWidget on Wednesday June 22, 2011 @08:42PM (#2686112)
http://en.wikipedia.org/wiki/Working_set Basically it means the portion of the total data set that's frequently accessed given a short time period.

The best practice totally depends on your work load and what kind of trade offs you're willing to make. I believe this is exactly the reason why there doesn't seem to be any good "best practice" documents for databases: it's different for everybody. In my case, there are a lot of writes and only occasional reads. My database is insert-mostly; once inserted, the data rarely changes. All data is timestamped. The reads usually pertain the most recently inserted data. In my case it makes sense to optimize the database in such a way that the most recently inserted data is in my working set.

This does means that querying old data is slower than querying recent data. This can be made faster by ensuring that all data fits in RAM, e.g. by adding more shards. But given the huge amount of data I have, placing all shards in RAM is way too expensive, and I'm not confident that my users are willing to pay 20 times the fees just to have faster queries for old data. The alternative is to store things on SSDs which should also make things faster, but this would still result in a 3-4 times fee increase for users.