> The more nodes, the less likely any one going down harms things but the harder it is to reach consensus

Flexible quorums deals with this in the usual read/write case. But yeah leader election then becomes the bottleneck. So if the cluster is unstable overall performance gets even worse as you scale number of nodes.

https://fpaxos.github.io/

Those places could still all be closer to the client. If you have 50 nodes around the country and you require every write to be acknowledged by 3 nodes for consistency, you can probably find 3 nodes very close to the client and therefore commit writes faster than if they all came to a central location.

     you require every write to be acknowledged by 3 nodes for consistency

How would that work? Someone else could get a different 3 nodes and write the same key, so they wouldn't read my write before writing their own? This isn't my area but I would assume the data has to be sharded ahead of time, not just write to any 3 out of 50 nodes that are closest to me.

Everything has to be directed through a leader though. So it's not a matter of how close the client is to the cluster but how far the cluster's nodes are from eachother (since anyone could be the leader). No?

In practice you don't run the consensus over the whole cluster for every shard of data. This would be inefficient in terms of data storage (every node stores a copy of everything) and latency.

For every shard you pick a few nodes, and you run consensus between those. This is the way Elasticsearch, CockroachDB, ... work.

So if a specific shard is getting writes from a specific region, you can easily move that shard to nodes in that region.

In contrast, etcd runs consensus over the whole cluster. So it is very safe, very slow, and meant for small datasets. In many ways, it is not a database, but a metadata store for distributed systems.

This is not entirely accurate. If you want to forego the ability to support cross shard transactions then sure you can run consensus independently on each shard. But I don't think that is what Spanner/CockRoachDB do. In addition, even if you want to limit transactions to a a single shard, the shard replicas need to be in different datacenters/AZs if you want to remain available when a DC/AZ goes down.

Yup that makes sense. It wasn't clear from the article though how their system works and other sqlite+raft implementations do the simpler version of one raft cluster for the entire database, I think.

If you have fifty nodes, you don't want all fifty working on every database. Each db should only be associated with with nodes to have a healthy quorum. But this is fine: you probably only have 3-5 regions. Each region could have 1-2 nodes supporting each db. It's almost certainly the case that you don't have dozens of regions/PoPs (and if you do, the material difference in latency between one PoP and another is probably ones of milliseconds for most users).