If you've switched to this new Raft algorithm, does this address the split-brain problem? What will now happen if a netsplit occurs and two primaries are elected, then rejoined?
So we thought, first, let's build a system based on manual failover, make that really solid, learn as much as possible about real-world use cases, and then build automatic failover on top of that.
This process took a couple of years, and we're finally at a point where we can ship automatic failover that we're confident will behave correctly in real-world (and theoretical) scenarios (modulo bugs that may not have been uncovered in testing).
To answer your specific question, if a netsplit occurs, two primaries could never be elected. There will be a primary either one side, or the other side of the split, so rejoining will not be a problem.
In practice, however, this is more subtle -- what if there is a netsplit, one side maintains its primary, and the other side elects a new one? What if the user writes to both primaries on either side? What happens after the cluster rejoins? In this specific case, we solve this by requiring the majority of the replicas to acknowledge writes by default before the write acknowledgement is sent to the client.
These questions can get really, really subtle -- (for example, what happens if there are multiple cascading netsplits in your cluster?) We wanted to take the time to understand these problems much better before we build automated failover, so we went with a manual failover system first.
That's the particular case that I was curious about, that solution makes a lot of sense. I suppose that would also mean that if an outage takes out more than half of your replicas then you will loose all write-abilities (unless I'm mistaken), but that's probably better than the potential mess that could ensue.
Do these write requests to nodes that have no route to a primary simply "hang", or are they rejected by the daemon?
That's correct. We've built in an "emergency repair" provisions into the product to handle cases like this, should they happen, but that requires manual intervention. (In general, if you lose more than half of your servers, you want to intervene manually anyway)
> Do these write requests to nodes that have no route to a primary simply "hang", or are they rejected by the daemon?
They time out via normal TCP mechanisms and get rejected.
Obviously latency is an issue but I'm not sure with the way RethinkDB replicates what the performance penalty would be for WAN in terms of write latency.
We'll document this setup better, but the replication options allow you to safely specify the set of nodes that need sync and async replication (so you can have sync replication for closely connected nodes in a datacenter, and async replication across datacenters).
> The mobile devices will often be disconnected from the majority of other devices, thus running election after election without being able to gain enough votes. When these devices rejoin the network, they will force a new election as their term will be much higher than the leader. This leads to frequently disconnected devices becoming the leader when reattached to the network, thus causing regular unavailability of the system when they leave again.
> [...]
> It is also interesting to consider the impact of a single node being behind an asymmetric partition [...] This node will repeatedly re-run elections as it will not be able to receive incoming messages, each RequestVotes will have a higher term than the last, forcing the leader to step down.
Specifically, they required to add client request caches, exposing serial transaction IDs in every single record and log entry, specific timeouts and measures for livelock bugs this introduced, which include replacing an entry by a noop value to be replicated on some read-only operation done at a later time. On top of this, these modifications may end up reducing the serializability constructs to bring it closer to what a Master-Follower replication may give in a DB -- i.e. it doesn't guarantee one true serializable sequence of commits anymore, but only that the state replication will be consistent across all nodes
Since RethinkDB uses TCP connections, this shouldn't usually happen (since the TCP acknowledgements wouldn't get through either). The exception might be a layer 5 router / firewall somewhere in the network that allows the TCP connection to work, but only passes the data stream through in one direction. RethinkDB is partially protected against this case, because we use bidirectional heartbeats on top of the same TCP connection that is used for Raft traffic. The heartbeat usually ensures that the other host is still alive and reachable. In this case, the node that cannot receive any responses from the remaining cluster would get a heartbeat timeout after a couple of seconds, and disconnect from the remaining servers. This is turn should limit further damage and allow the Raft cluster to proceed.
Please let me know if I'm missing something else.
Edit: There might still be some problems with non-transitive connectivity, where one node can talk to only parts of the cluster. We have built in some protection against this, but don't always handle that case well.
We're super-excited about this beta. We've been working on Raft, automatic failover, and all the relevant enhancements for over a year. It's a massive engineering project that's getting very close to production quality, and it feels so good to finally get it into people's (proverbial) hands!