News for hackers. Stuff that matters.
Re:So, that was a bummer by brown9-2 on Tuesday October 05, 2010 @06:20PM (#1761782)
Interestingly the cause of the most severe part of their downtime seems to still be unknown to them:

As a next step, we introduced a new shard, intending to move some of the data from the overloaded shard to this new one.

We wanted to move this data in the background while the site remained up. For reasons that are not entirely clear to us right now, though, the addition of this shard caused the entire site to go down.

To those who use MongoDB - does this sound like something that might have been caused by MongoDB itself, or Foursquare's use of it?

Re:So, that was a bummer by dmytton on Tuesday October 05, 2010 @06:36PM (#1761848)
Without knowing more about their schema, it's hard to speculate. However, let's assume they had a single, large "checkins" collection, sharded by user ID as they said. That would be distributed across the shards. If the mongod process is down on one shard and therefore the shard isn't responding, the data just wouldn't be available.

But it sounds like the shard was still up. As per the sharding FAQ[1]:

"What if a shard is down or slow and I do a query? If a shard is down, the query will return an error. If a shard is responding slowly, mongos will wait for it. You won't get partial results."

They said they had a performance issue on the overloaded shard so perhaps it's possible that mongo believed the shard to be up, when it was instead overloaded. This meant any queries just waited, taking down the whole site.

As I said, this is speculation and as a massive production user of MongoDB ourselves[2], I'm interested to know more.

[1] http://www.mongodb.org/display/DOCS/Sharding+FAQ [2] http://blog.boxedice.com/2010/02/28/notes-from-a-production-...

Re:So, that was a bummer by VladRussian on Tuesday October 05, 2010 @07:38PM (#1762072)
"What if a shard is down or slow and I do a query? If a shard is down, the query will return an error. If a shard is responding slowly, mongos will wait for it. You won't get partial results."

an approach to CAP that chooses consistency over availability. I wonder why as it isn't a bank and they could have chosen the availability instead.

Re:So, that was a bummer by jbooth on Tuesday October 05, 2010 @10:30PM (#1762472)
If in doubt, choose consistency.
Re:So, that was a bummer by marclove on Tuesday October 05, 2010 @06:42PM (#1761864)
But it wasn't the overloaded shard that brought the site down (it was only slowing things down), it was the addition of the new shard:

"For reasons that are not entirely clear to us right now, though, the addition of this shard caused the entire site to go down."

Re:So, that was a bummer by dmytton on Tuesday October 05, 2010 @06:47PM (#1761887)
If the data was being moved it could cause even more load on an already overloaded shard.
Re:So, that was a bummer by fauigerzigerk on Wednesday October 06, 2010 @04:25AM (#1763174)
I know nothing about MongoDB but why would one shard that's responding slowly affect queries to all the other shards and take the whole site down? I can't believe any database system would be designed to work like that.
Re:So, that was a bummer by scg on Tuesday October 05, 2010 @07:18PM (#1762006)
At 6:30pm EST, we determined the most effective course of action was to re-index the shard, which would address the memory fragmentation and usage issues.

Sounds similar to some of the problems we're facing with MongoDB too. Indexes fragment a lot, especially if you delete documents frequently. You have to reindex everything eventually.

MongoDB offers background indexing (except for _id fields!) but it's not really an option if you need the indexes at all times to keep executing queries. Also, you'll need to reindex _id eventually and there's currently no way to do that without blocking the server.

Re:So, that was a bummer by houseabsolute on Tuesday October 05, 2010 @06:32PM (#1761834)
I'm also wondering why this kind of intervention is necessary at all. The NoSQL solution we use at work has load based automatic splitting, and I'd have thought (though I haven't confirmed) that this would be an obvious feature to include.
Re:So, that was a bummer by mcfunley on Tuesday October 05, 2010 @08:26PM (#1762195)
I would speculate that it's a poorly-chosen shard key. MongoDB's built-in sharding uses range-based indexing. If you choose user_id as your shard key, and those are autoincrementing integers, then you're screwed if newer users tend to be more active on average than older ones.
Re:So, that was a bummer by jrockway on Tuesday October 05, 2010 @09:09PM (#1762291)
Wait, people shard on a key other than something approximately random, like an sha1 hash!?
Re:So, that was a bummer by mcfunley on Tuesday October 05, 2010 @09:22PM (#1762338)
Where I work (Etsy) we keep an index server that maps each user to a shard on an individual basis. There are a number of advantages to it. For example, if one user generated a ton of activity they could in theory be moved to their own server. Approximately random works for the initial assignment.

Flickr works the same way (not by coincidence, since we have several former Flickr engineers on staff).

Re:So, that was a bummer by jrockway on Tuesday October 05, 2010 @10:41PM (#1762486)
Sounds like a good system. I've noticed that people tend to do things like shard based on even/odd, and then they realize that they need three databases.

I've never had either problem though... but if I ever need to shard I plan on doing it based on object ID. Then one request can be handled by multiple databases, "for free", increasing both throughput and response time.

Re:So, that was a bummer by fizx on Wednesday October 06, 2010 @12:23AM (#1762701)
Even/odd isn't the end of the world, but you would then be best jumping to mod 4.
Re:So, that was a bummer by moe on Wednesday October 06, 2010 @02:19AM (#1762945)
Actually for anonymous sharding (without a central index) a consistent hash is about the closest you can get to ideal distribution and flexibility. I haven't looked but I presume that's what mongo uses under the hood for their auto-sharding, too.
Re:So, that was a bummer by dmytton on Tuesday October 05, 2010 @06:52PM (#1761916)
Load based splitting is on the MongoDB roadmap, but doesn't exist yet.
Re:So, that was a bummer by fizx on Tuesday October 05, 2010 @08:11PM (#1762159)
Just because the feature exists, doesn't mean it works.
Re:So, that was a bummer by icey on Tuesday October 05, 2010 @06:34PM (#1761841)
Which NoSQL store do you use?
Re:So, that was a bummer by houseabsolute on Tuesday October 05, 2010 @06:38PM (#1761855)
Re:So, that was a bummer by staunch on Tuesday October 05, 2010 @08:26PM (#1762196)
That looks potentially useful in some cases. Can you send me the code/docs for that?
Re:So, that was a bummer by ddlatham on Tuesday October 05, 2010 @09:54PM (#1762404)
Re:So, that was a bummer by ifesdjeen on Wednesday October 06, 2010 @03:33AM (#1763083)
Does it work that well for production for you? Really, that's extremely interesting.
Re:So, that was a bummer by rb2k_ on Wednesday October 06, 2010 @02:07AM (#1762915)
Also look at dynamo based systems like cassandra and riak (riak seems to have a better load balancing at the moment, cassandra is a bit more "bumpy")
Re:So, that was a bummer by rb2k_ on Wednesday October 06, 2010 @02:06AM (#1762912)
> To those who use MongoDB - does this sound like something that might have been caused by MongoDB itself, or Foursquare's use of it?

MongoDB still has problems with locking too much for certain operations. The 10gen guys keep on introducing "yields" to those operations, but the simple (and fast) database/process-level locking could very well cause this... (not a 100% 'yes', but it seems possible)