News for hackers. Stuff that matters.
Re:Why `fsync()`: Losing unsynced data on a single node lead... by judofyr on Tuesday May 16, 2023 @01:35PM (#35965134)
I'm confused here. If I'm reading this correctly, the following is happening:

- Node 3 is the leader.

- Node 1 is isolated (failure #1).

- 10 records are written. They are successful because a majority is still alive (persisted to Node 2 and 3).

- Node 3 loses data (failure #2).

- Node 1 comes back up again.

However, isn't this two failures at the same time? Kafka with three nodes can only guarantee a single failure, no?

Re:Why `fsync()`: Losing unsynced data on a single node lead... by benmmurphy on Tuesday May 16, 2023 @02:14PM (#35965596)
I don't know what consistency Kafka is targeting but usually with 'consistent' systems in the situation where the cluster cannot serve valid data it should serve failure responses instead of giving back bogus data. if your system is 'eventually consistent' then that's a different game.
Re:Why `fsync()`: Losing unsynced data on a single node lead... by rystsov on Tuesday May 16, 2023 @02:24PM (#35965700)
Strongly consistent protocols such as a Paxos and Raft always choose consistency over availability and when consistency isn't certain they refuse to answer.

Raft & Paxos: any number of nodes may be down, as soon as the majority is available a replicated system is available and doesn't lie.

Kafka as it's described in the post(): any number of nodes may be down, at most one power outage is allowed (loss of unsynced data), as soon as the majority is available a replicated system is available and doesn't lie.

The counter-example simulates a single power outage

() https://jack-vanlightly.com/blog/2023/4/24/why-apache-kafka-...

Re:Why `fsync()`: Losing unsynced data on a single node lead... by judofyr on Tuesday May 16, 2023 @04:08PM (#35967085)
It just feels like two widely different scenarios we're talking about here.

https://jack-vanlightly.com/blog/2023/4/24/why-apache-kafka-... talks about the case of a single failure and it shows how (a) Raft without fsync() loses ACK-ed messages and (b) Kafka without fsync() handles it fine.

This post on the other hand talks about a case where we have (a) one node being network partitioned, (b) the leader crashing, losing data, and combing back up again, all while (c) ZooKeeper doesn't catch that the leader crashed and elects another leader.

I think definitely the title/blurb should be updated to clarify that this is only in the "exceptional" case of >f failures.

I mean, the following paragraph seems completely misleading:

> Even the loss of power on a single node, resulting in local data loss of unsynchronized data, can lead to silent global data loss in a replicated system that does not use fsync, regardless of the replication protocol in use.

The next section (and the Kafka example) is talking about loss of power on a single node combined with another node being isolated. That's very different from just "loss of power on a single node".

Re:Why `fsync()`: Losing unsynced data on a single node lead... by frant-hartm on Tuesday May 16, 2023 @04:32PM (#35967371)
Exactly my thoughts.
Re:Why `fsync()`: Losing unsynced data on a single node lead... by slt2021 on Tuesday May 16, 2023 @04:12PM (#35967127)
you are right, this is a failure of both zookeeper and leader node, so two independent failures at the same time
Re:Why `fsync()`: Losing unsynced data on a single node lead... by rystsov on Tuesday May 16, 2023 @08:05PM (#35969530)
We can't ignore or pretend that network partitioning doesn't happen. When people talk about choosing two out of CAP the real question is C or A because P is out of our control.

When we combine network partitioning with single local data suffix loss it either leads to a consistency violation or to a system being unavailable desperate the majority of the nodes being are up. At the moment Kafka chooses availability over consistency.

Also I read Kafka source and the role of network partitioning doesn't seem to be crucial. I suspect that it's also possible to cause similar problem with a single node power-outage https://twitter.com/rystsov/status/1641166637356417027 and unfortunate timing