If that's just saying that startup time can be an issue, I agree. There are a variety of techniques to mitigate that, though. The simplest is to compress snapshots and/or put them on RAID, boosting read speed. The most complicated is just to have mirrored servers and only restart the one not in use right now.
> I'm talking about having more data than fits in an reasonable amount of RAM (say 1TB).
For something where you need transactions across all of that? This architecture's probably not a reasonable approach, then. The basic precondition is that everything fits in RAM. However, sharding is certainly possible if you can break your data into domains across which you don't require consistent transactions.
> So it's going to at the speed of the disk.
Sort of.
Because it's just writing to a log, mutations go at the speed of streaming writes, which is very fast on modern disks. And there are a variety of techniques for speeding that up, so I'm not aware of a NoDB system for which write speed is the major problem.
Regardless, it's a lot better for writes than the performance of an SQL database on the same hardware.
> My other question is how much of a pain in the ass is it to debug such a system?
It seemed fine. A big upside is that you have a full log of every change, so there's no more "how did X get like Y"; if you want to know you just replay the log until you see it change.
Last I did this we used BeanShell to let us rummage through the running system. It was basically like the Rails console.
> - The startup times can be a problem if you have a lot of data. Modern disks are pretty fast for streaming reads, though, and you can split the deserialization load across multiple processors.
Reading data, at even a GB/second from disk (which is currently not possible) is going to mean a second spent of a GB of data, just to read, let alone deserialize. That's with reading a snapshot, not replaying old transactions.
> - Mirroring state is easy; you just pipe the serialized commands to multiple boxes.
That's not distributing the load. I'm talking about having more data than fits in an reasonable amount of RAM (say 1TB). Also mirroring is nice for when you want read-only access to your data. You'll have the same problem as any other data store when you want multiple writers. Also, is replication synchronous or asynchronous (which end of CAP do you fall on)?
>- It's very fault tolerant. Because every change is logged before being applied, you just load the last snapshot and replay the log.
So it's going to at the speed of the disk then (http://smackerelofopinion.blogspot.com/2009/07/l1-l2-ram-and...). Don't get me wrong, this is still faster than writing to the network, but then writes are way slower than reads.
My other question is how much of a pain in the ass is it to debug such a system? I suppose if you have a nice offline API to look at your data, change something, revert back, etc, it would work well, but if it's deep within your normal application, it could become nightmarish.