News for hackers. Stuff that matters.
Re:MemSQL (YC W11) Raises $2.1 Million From Ashton Kutcher, ... by gfaremil on Tuesday July 12, 2011 @03:52PM (#2756237)
Even if you put write ahead logs on different I/O subsystems the lock needed to ensure consistency will probably be a bottleneck (and crappy Linux implementation asynch IO on multi processor system does not help). How do you guys organize data so it is consistent and lock is not a bootleneck?
Re:MemSQL (YC W11) Raises $2.1 Million From Ashton Kutcher, ... by ZoFreX on Tuesday July 12, 2011 @04:47PM (#2756484)
You don't necessarily need to write log entries in the order they are committed, nor do you need to store all of the entries in a single log, you could shard the log writing too - as long as you maintained, somewhere, the order information. So yes, you would still need to lock and sort out the order somewhere, but you already have to if you're ACID any way.
Re:MemSQL (YC W11) Raises $2.1 Million From Ashton Kutcher, ... by nikita on Tuesday July 12, 2011 @07:33PM (#2757102)
Right on the money. We every log record contains LSN (log sequence number) that allows us to reconstitute the order. Then as at the recovery time you need to merge sort by LSN from the shared log servers
Re:MemSQL (YC W11) Raises $2.1 Million From Ashton Kutcher, ... by ZoFreX on Wednesday July 13, 2011 @03:45PM (#2760345)
When you say you merge sort from the log servers, do you mean the logs on a given server are written & stored in-order?