I'm aware that Paxos and Raft require odd number of nodes. My point was that the more nodes you have the slower the cluster is, because increasing number of the nodes doesn't increase performance but resilience. With 3 nodes you can afford to lose at most 1 node, with 5 -> 2, with 7 -> 3 etc.

> Also, key value store reads are usually done locally, from the agent, which goes over gossip and is eventually consistent.

That's not what the documentation says. The KV is using raft, you can relax some guarantees for reading by having clients use consistency=stale, but you're still communicating with the server, except in stale you only communicate with one instead all of them.

> Health checks are also eventually consistent. They are done by the local agent, and through gossip are sent to the masters group.

Also that's not what the documentation says. Gossip is used for adding/removing nodes and the overall health of the node (whether host is reachable) is also done through gossip, but the actual health checks (whether a service is running) is done through consensus protocol.

I still want to emphasize that using consensus protocol for SD is not a good idea. Let say you're running in AWS and follow the best practices which is spreading your servers over multiple AZ. Let say you use us-east-1 which has 6 AZ.

An AWS starts having network issues (like few months ago) and 2 AZs have network issues and can't communicate with the rest. The machines are still running, just can't communicate. It just happens that on 2 of those AZs you run 2 out of your 3 consul servers. Even though all the machines are healthy and running your Consul cluster is down, even for the remaining 4 AZs that don't have any problems.