Re:Redpanda – A Kafka-compatible streaming platform for miss...
by espadrine on Monday November 16, 2020 @07:33AM
(#25110861)
Outstanding article!
Are you hoping to achieve no-downtime leader kills by going with the latest Paxos improvements, as you mention at the end?
Re:Redpanda – A Kafka-compatible streaming platform for miss...
by agallego on Monday November 16, 2020 @10:19AM
(#25112706)
So many goodies to come.
First we need to finish multi partition transactions that we don't yet support.
We need to balance eager cluster balancing based on saturation (network, cpu, disk, etc) with other performance improvements.
As is the cluster achieves steady state in a couple of seconds for affected partitions and clients in Kafka retry.
Re:Redpanda – A Kafka-compatible streaming platform for miss...
by returnfalse on Friday November 13, 2020 @04:59AM
(#25080782)
It's interesting to see that there are still opportunities for further improvements in raft. Thanks for the reply!
To quote my colleague Denis - The results fit the theory. Raft’s (and Redpanda’s) performance is proportional to the best of the majority of nodes, while sync replication (Kafka) works only as well as its worst-performing node.
AFAIK we can push the limits of hardware on throughput.
The gist is that I'd need more details exactly on what you mean raft is slow.
We support the same level of acks as kafka except acks=-1 is much stronger gurantees due to the log completeness guarantee of raft.
for 0 and 1 we short circuit the raft acknowledgement and return to the client to match the exepectations of acknowledgements come to be known by users.
There should be no perf penalty vis-a-vis kafka in any setting I can think of. If it is, is probably a bug on our side.