So you don't benefit in that simple approach from adding more disks.

Not sure about this DB, but that's my understanding of consensus as well. The more nodes, the less likely any one going down harms things but the harder it is to reach consensus. You have to write to more places. It becomes slower not faster.

> The more nodes, the less likely any one going down harms things but the harder it is to reach consensus

Flexible quorums deals with this in the usual read/write case. But yeah leader election then becomes the bottleneck. So if the cluster is unstable overall performance gets even worse as you scale number of nodes.

https://fpaxos.github.io/

Those places could still all be closer to the client. If you have 50 nodes around the country and you require every write to be acknowledged by 3 nodes for consistency, you can probably find 3 nodes very close to the client and therefore commit writes faster than if they all came to a central location.

     you require every write to be acknowledged by 3 nodes for consistency

How would that work? Someone else could get a different 3 nodes and write the same key, so they wouldn't read my write before writing their own? This isn't my area but I would assume the data has to be sharded ahead of time, not just write to any 3 out of 50 nodes that are closest to me.

Everything has to be directed through a leader though. So it's not a matter of how close the client is to the cluster but how far the cluster's nodes are from eachother (since anyone could be the leader). No?

In practice you don't run the consensus over the whole cluster for every shard of data. This would be inefficient in terms of data storage (every node stores a copy of everything) and latency.

For every shard you pick a few nodes, and you run consensus between those. This is the way Elasticsearch, CockroachDB, ... work.

So if a specific shard is getting writes from a specific region, you can easily move that shard to nodes in that region.

In contrast, etcd runs consensus over the whole cluster. So it is very safe, very slow, and meant for small datasets. In many ways, it is not a database, but a metadata store for distributed systems.

This is not entirely accurate. If you want to forego the ability to support cross shard transactions then sure you can run consensus independently on each shard. But I don't think that is what Spanner/CockRoachDB do. In addition, even if you want to limit transactions to a a single shard, the shard replicas need to be in different datacenters/AZs if you want to remain available when a DC/AZ goes down.

Yup that makes sense. It wasn't clear from the article though how their system works and other sqlite+raft implementations do the simpler version of one raft cluster for the entire database, I think.

If you have fifty nodes, you don't want all fifty working on every database. Each db should only be associated with with nodes to have a healthy quorum. But this is fine: you probably only have 3-5 regions. Each region could have 1-2 nodes supporting each db. It's almost certainly the case that you don't have dozens of regions/PoPs (and if you do, the material difference in latency between one PoP and another is probably ones of milliseconds for most users).

> My intuition about strapping a consensus library to a database is that you're going to have way higher latency because every operation, writes and reads, must go through consensus

nah, you have leases (a time-limited and often renewable lock). so like your leader gets elected through consensus and then for the next N miliseconds, you have a garauntee that nobody else can be leader. this means your reads dont need to go through consensus. (hand-waving a lot of the details here.)

I think there's some confusion at what distributed databases like Spanner, Cockroach, etc do:

- A given row of data is configured to have N replicas, e.g. 3, 5, etc, but N doesn't have to equal the actual size the database cluster. e.g. you can have a database cluster of 21 nodes with N being 3. N represents the "Raft cluster size" and the database is composed of many "Raft clusters" that are responsible for segments of the data space. You can horizontally scale the size of the database cluster by adding more nodes and thus more "Raft clusters"

- Raft/Paxos are used to ensure linearizable writes to those replicas through a leader

- Multi row/partition transactions are handled through a different kind consensus called atomic commit/2 phase commit that's orthogonal to Raft/Paxos consensus: https://timilearning.com/posts/mit-6.824/lecture-12-distribu...

- Systems like Spanner, Cockroach have higher write latencies because replicas in Raft cluster have to achieve quorum consensus, but reads latency can mostly not be affected by always reading from the leader of a cluster via things like leader leases.

I think the idea with localized access is that reads coming in after the initial commit can hit a nearby db instance, rather than the single global instance.

I agree with you about horizontal scalability in general though, the data isn't being partitioned. I guess read heavy loads can be scaled.