That is definitely one of the options we are considering. It (obviously) involves a change to the product, which we have to think about carefully, but it certainly could help from a technical standpoint.

It also depends on your algorithms. Some algorithms are amenable to "running tallies," so a third possible approach would be to store and update various values based only on the aggregate past data plus the incremental data, instead of looking back through the entire history and recomputing when a new piece of data comes in. This of course depends on whether or not this is even theoretically possible with what you're computing.

This is probably feasible for some badges, but would be hard to do for all of them.

This approach also increases the complexity of adding new badges, which is undesirable for product and business reasons.

It could certainly help in some case though, and it's something we're considering.

Can you design the system such that you can turn off ("darkmode") features in an emergency? For example, in an emergency you can turn off badge-awarding as it is known to be RAM intensive (and degrade by seeking to disk).

To the extent that you do this across the board, you'll have yet another tool to defend against being over capacity.

We have a framework for doing this sort of thing but it's only implemented around certain features so it doesn't provide us a ton of benefit at the moment. Obviously doing work to give us more control like this could be a big win, so we'll probably be doing some of that.

There is no better time than during an outage to add these features. In my experience, band-aids are better way to resolve an outage than fixing root-causes. Make the problem go away by commenting out code, returning empty result-sets, (or whatever) - and re-balance your partitioned database at some later date.

you don't need to keep their entire history in ram, you only need to keep their check-in history of places that they are on the verge of becoming mayors to in ram, (you are one day away from becoming mayor of xyz) and that you can calculate offline, and all you need to do is see if onedayaway:xyz:user_id => true on checkins, which you can keep in memcache