News for hackers. Stuff that matters.
Re:RethinkDB (YC S09): MySQL Storage Engine Built From The G... by neilc on Tuesday July 28, 2009 @09:43PM (#729270)
Database consistency problems require traditional databases to use complicated locks. Because RethinkDB data is always consistent, locking is unnecessary.

You may not use locking for concurrency control (plenty of "traditional DBs" don't, either), but you still need some sort of concurrency control scheme -- just using append-only/log-structured storage doesn't make CC free. I'd be curious to hear how you guys are doing this.

Re:RethinkDB (YC S09): MySQL Storage Engine Built From The G... by leif on Tuesday July 28, 2009 @09:49PM (#729279)
For the moment, we allow one writer at a time, but unbounded (I think) readers. The only locks taken are for arithmetic operations; no locks are held while performing I/O or even memcpy(2). You can read more about that bit in the paper about our caching architecture on the wiki.

Eventually, we will allow multiple writers, and use some form of STM for that; we just haven't gotten to it yet.

If you'd like to speak interactively on this, shoot me an email.

Re:RethinkDB (YC S09): MySQL Storage Engine Built From The G... by coffeemug on Tuesday July 28, 2009 @09:48PM (#729277)
Most industrial strength storage engines use multiversion concurrency control, but they still require a fair share of locking - row level locks that protect the data while a second copy is made, for example. No storage engine is entirely "lock-free", but we have only very few extremely granular locks. This is completely different from what was done before (as far as we know), and from a practical perspective is as good as lock-free.

I don't think I can explain the technology in a comment, but we'll definitely be releasing more information in the coming weeks. We had to get this release out for now. Stay tuned!

Re:RethinkDB (YC S09): MySQL Storage Engine Built From The G... by gaius on Wednesday July 29, 2009 @09:13AM (#729915)
Statements like that set my alarm bells ringing.
Re:RethinkDB (YC S09): MySQL Storage Engine Built From The G... by neilc on Wednesday July 29, 2009 @11:56PM (#731662)
Sorry, statements like what?
Re:RethinkDB (YC S09): MySQL Storage Engine Built From The G... by gaius on Thursday July 30, 2009 @04:39AM (#731863)
Statements like "traditional X does Y but..."

The "traditional" (and I don't know how you could call the latest version of any of the major databases "traditional", this is a brutally competitive market) databases don't lock just for the fun of it, but to enable features that users want. Anyone can come up with a product that doesn't do Y if it can't do X either. So what're we missing here?

Re:RethinkDB (YC S09): MySQL Storage Engine Built From The G... by neilc on Thursday July 30, 2009 @04:27PM (#733004)
I don't know how you could call the latest version of any of the major databases "traditional"

Oracle has done MVCC for many years, as has Postgres. The canonical paper on optimistic concurrency control for DBs is from 1981 (http://www.seas.upenn.edu/~zives/cis650/papers/opt-cc.pdf).

I didn't follow the rest of your comment, I'm afraid -- I was just saying that I didn't see how using append-only storage immediately makes concurrency control a non-issue. The comments from the RethinkDB guys upthread support that: not supporting concurrent writers makes your concurrency control much more straightforward.