Re:Redpanda – A Kafka-compatible streaming platform for miss...
by agallego on Monday November 16, 2020 @10:22AM
(#25112738)
Indeed. See the segment appender work in the github with the chunk cache which sums up our write behind strategy.
The reads are more complex since it involves understanding io priority scheduling etc
Re:Redpanda – A Kafka-compatible streaming platform for miss...
by fnord123 on Monday November 16, 2020 @10:47AM
(#25113078)
>write behind strategy
You ack writes before they're written?
Re:Redpanda – A Kafka-compatible streaming platform for miss...
by rystsov on Monday November 16, 2020 @11:13AM
(#25113372)
Yes and no, we do both. For ack=-1 we wait for fsync & replication confirmation but for other modes we relaxed the behavior.
Re:Redpanda – A Kafka-compatible streaming platform for miss...
by agallego on Monday November 16, 2020 @11:07AM
(#25113302)
not sure how you got here tho, maybe i miss explained something. This is talking about filesystem api not about kafka api. the kernel does something similar, standard filesystem things just purpose built for our use case.
Re:Redpanda – A Kafka-compatible streaming platform for miss...
by fnord123 on Monday November 16, 2020 @02:41PM
(#25116057)
afaics a write behind strategy means you will report a write as being complete before it has actually been written. This can result in terrific performance because your batching window is larger and you can amortize more (fewer?) iops. But it also means in the face of failure, clients could believe something was written which wasn't.
On the other hand, the storage device can fail so what does it even mean to have written the data? :)
Re:Redpanda – A Kafka-compatible streaming platform for miss...
by agallego on Monday November 16, 2020 @03:02PM
(#25116309)
our acks are similar to kafka in that regard. acks=0,1 leader acknowledgement. acks=-1 we go further like denis mentioned above by 'flushing' to disk before responding which is stronger (log completeness guarantee from raft) than available in upstream.
https://vectorized.io/dataengineering-podcast-interview/
The throughput is 4x, the latency is 10x.