News for hackers. Stuff that matters.

mikealrogers

Re:STM, CouchDB, and writing 5500 documents/second on Thursday July 29, 2010 @10:51AM (#1557739)
this was part of my initial reaction.

when CouchDB is under concurrent load it does a "group commit" style write anyway and can batch nearly as efficiently as bulk docs. the only additional overhead would be the HTTP headers which is nominal when the documents are a bit larger than "hello world".

in general CouchDB does better under concurrent write load than single writer and while these kinds of optimizations on the client side can be impressive it's not actually what CouchDB is tuned for OOTB.

Re:STM, CouchDB, and writing 5500 documents/second on Thursday July 29, 2010 @10:46AM (#1557724)
would you be willing to write up how your writes were outrunning the compactor in a little more detail in either a blog post or in an email to the apache couchdb dev list?

we've been talking about possible improvements to the compactor and having a solid use case that outruns it would help focus that.

Re:STM, CouchDB, and writing 5500 documents/second on Thursday July 29, 2010 @10:42AM (#1557702)
i find this whole thing a lot more of an experiment in clojure performance than CouchDB throughput.

he doesn't attach a handler to the response on his writes and is queuing writes in memory and bulking them. he's not concerned with durability at pretty much any layer.

this kind of setup is great for a large initial import of data but i would be weary of running it in production.

Re:MongoDB Performance & Durability on Thursday July 08, 2010 @01:06PM (#1497969)
traditionally durability isn't considered a sliding scale, it's a goal/priority which requires you to implement multiple features and fallbacks to handle everything from invalid writes, crash during write, to the data center catching on fire.

thinking about durability this way may work great for MongoDB but it isn't how durability is framed in the rest of the database world.

Re:MongoDB Performance & Durability on Thursday July 08, 2010 @01:24AM (#1496569)
it's easy to make something fast if you don't want durability, just look at memcached.
Re:MongoDB Performance & Durability on Wednesday July 07, 2010 @11:36PM (#1496401)
copied from my blog:

@kristina

for some reason wordpress wanted me to moderate your post so sorry for the delay in it showing up.

>> Whoopsy, got your emphasis wrong there ….. Seriously, though, this “unchecked” type >> of write is just supposed to be for stuff like analytics or sensor data, when you’re getting >> a zillion a second and don’t really care some get lost if the server crashes.

did the default change? the last time i attempted to a concurrent performance test this was one of the barriers i hit. my issue isn’t that you include this feature, it’s that it’s the default, i certainly believe there is a use case for it i just think it’s harmful as a default.

>> Since CouchDB is “not a competitor” to MongoDB, it’s nice of you to put all this time >> into a public service.

haha, that’s funny. i regularly use non-CouchDB databases and I get along great with all the people from other databases at conferences. even if i did feel like we were competing, i wouldn’t care. this post really is about reliability issues i don’t think your users are fully aware of and i honestly hope that you fix.

>> fsyncs are configurable. You can fsync once a second, never, or after every single insert, >> remove, and update if you wish.

that’s really good to hear. have you optimized for a “group commit” yet?

>> This is because you assume you’ll run it on single server. MongoDB’s documentation >> clearly, repeatedly, and earnestly tells people to run MongoDB on multiple servers.

I responded earlier to the complexity of actually keeping something available that depends on this. so i won’t cover it again.

>> That’s not to say that things never go wrong, MongoDB is definitely not perfect and has >> lots of room for improvement. I hope that users with questions and problems will >> contact us on the list, our wiki, the bug tracker, or IRC (or, heck, write a snarky blog >> post). Anything to contact the community and let us try to help. I wish every person >> who tried MongoDB had a great experience with it.

You make it sounds like this is all just a matter of bugs, it’s not, and i find blaming it on users who don’t use JIRA or get on IRC a little distasteful.

these issues are architectural and until you do something append-only they aren’t going to go away. someone mentioned earlier that you plan to do an append-only transaction log, if that’s accurate then it’s fantastic news.

Re:MongoDB Performance & Durability on Wednesday July 07, 2010 @11:34PM (#1496396)
that's kind of like saying "we told you our parachutes won't open sometimes. well, not in our conference talks or in our marketing but it's on our wiki and we blogged about it".

if you want to see a message that is clearly delivered about (lack of) durability look at memcached. nobody misunderstands memcached's durability/consistency guarantees.

-Mikeal (after a few drinks)

Re:MongoDB Performance & Durability on Wednesday July 07, 2010 @11:30PM (#1496388)
wait, seriously?

the suggested default step 1 for MongoDB is to acquire 3 servers?

i mean, no other database suggests such a huge default configuration. even knowing that their datacenter can get hit by lightening and all a lot of large production sites don't even run with this kind of redundancy.

this seems like a pretty taxing workaround for not keeping an append-only transaction log.