To quote my colleague Denis - The results fit the theory. Raft’s (and Redpanda’s) performance is proportional to the best of the majority of nodes, while sync replication (Kafka) works only as well as its worst-performing node.
AFAIK we can push the limits of hardware on throughput.
The gist is that I'd need more details exactly on what you mean raft is slow.
We support the same level of acks as kafka except acks=-1 is much stronger gurantees due to the log completeness guarantee of raft.
for 0 and 1 we short circuit the raft acknowledgement and return to the client to match the exepectations of acknowledgements come to be known by users.
There should be no perf penalty vis-a-vis kafka in any setting I can think of. If it is, is probably a bug on our side.
Are you hoping to achieve no-downtime leader kills by going with the latest Paxos improvements, as you mention at the end?
First we need to finish multi partition transactions that we don't yet support.
We need to balance eager cluster balancing based on saturation (network, cpu, disk, etc) with other performance improvements.
As is the cluster achieves steady state in a couple of seconds for affected partitions and clients in Kafka retry.
One question: I assume RedPanda using raft to replicate the topic content not just metadata. Is that correct? If so, how does it perform better compared to Kafka’s ISR? Since raft might be slow for this kind of workload. If I remember correctly, Liftbridge was using raft for log replication and switched away from that because of the performance problems.