News for hackers. Stuff that matters.
Re:MongoDB's lead developer: Foursquare outage post mortem by rbranson on Thursday October 07, 2010 @09:50PM (#1770274)
Have you thought about how much extra overhead there is in storing the checkins using MongoDB's BSON structures? Considering how often you do this, you might consider packing a specialized version of a user's history (at least for this purpose -- sort of a denormalization) into a array of 64-bit placeIDs? This means a checkin would only occupy 8 bytes of space and there would be no overhead. 1 billion checkins = 8GB of data. You could even keep these in memcached and use a sort of write-through strategy.

EDIT: I thought about this a bit more, and the checkins are probably time-sensitive, so another 32-bit timestamp (Foursquare Epoch) would need to be stored for each checkin.

Re:MongoDB's lead developer: Foursquare outage post mortem by harryh on Thursday October 07, 2010 @10:02PM (#1770302)
Unfortunately we have some badges that require more than just the venueid + timestamp, but some sort of "compressed" checkin format is something we'll probably end up looking at.
Re:MongoDB's lead developer: Foursquare outage post mortem by rbranson on Thursday October 07, 2010 @10:15PM (#1770331)
Bummer. It's really amazing how 1 seemingly simple (but quite important and engagement driving) feature like instant gratification badges can drive your architecture decisions and keep you guys up at night.