News for hackers. Stuff that matters.
Re:MongoDB's lead developer: Foursquare outage post mortem by nkallen on Thursday October 07, 2010 @08:48PM (#1770144)
Can you design the system such that you can turn off ("darkmode") features in an emergency? For example, in an emergency you can turn off badge-awarding as it is known to be RAM intensive (and degrade by seeking to disk).

To the extent that you do this across the board, you'll have yet another tool to defend against being over capacity.

Re:MongoDB's lead developer: Foursquare outage post mortem by harryh on Thursday October 07, 2010 @08:56PM (#1770160)
We have a framework for doing this sort of thing but it's only implemented around certain features so it doesn't provide us a ton of benefit at the moment. Obviously doing work to give us more control like this could be a big win, so we'll probably be doing some of that.
Re:MongoDB's lead developer: Foursquare outage post mortem by nkallen on Thursday October 07, 2010 @09:59PM (#1770299)
There is no better time than during an outage to add these features. In my experience, band-aids are better way to resolve an outage than fixing root-causes. Make the problem go away by commenting out code, returning empty result-sets, (or whatever) - and re-balance your partitioned database at some later date.