News for hackers. Stuff that matters.
Re:Redpanda – A Kafka-compatible streaming platform for miss... by BenoitP on Monday November 16, 2020 @05:40AM (#25110220)
Nice!

It'd be great to know where do the improvements came from.

Is it from a different architectural design, is it from better exploitation of new hardware, is it from JVM limitations? For example, when Valhalla and Loom land in the JDK, how much of this improvement will stay?

Re:Redpanda – A Kafka-compatible streaming platform for miss... by fnord123 on Monday November 16, 2020 @06:12AM (#25110389)
I can't find the benchmarks so it's not possible to know what they measured, how they measured it, and if it's a relevant measurement.

Skimming the repository it doesn't seem to be an unreasonable claim. C++ with Seastar compared to Scala on JVM. It's in line with the Scylla/Cassandra improvements.

Re:Redpanda – A Kafka-compatible streaming platform for miss... by fnord123 on Monday November 16, 2020 @06:17AM (#25110418)
According to this Data Engineering interview (beginning around 8m in) the improvements begin with moving from the JVM which has a latency impact. But the real improvement is that fsync on a file handle puts in a barrier in the FS. RedPanda does some work around batch coalescing, they skip the page cache to side step a bunch of kernel locks, and adaptive allocation (preallocating FS space):

https://vectorized.io/dataengineering-podcast-interview/

The throughput is 4x, the latency is 10x.

Re:Redpanda – A Kafka-compatible streaming platform for miss... by agallego on Monday November 16, 2020 @10:22AM (#25112738)
Indeed. See the segment appender work in the github with the chunk cache which sums up our write behind strategy.

The reads are more complex since it involves understanding io priority scheduling etc

Re:Redpanda – A Kafka-compatible streaming platform for miss... by fnord123 on Monday November 16, 2020 @10:47AM (#25113078)
>write behind strategy

You ack writes before they're written?

Re:Redpanda – A Kafka-compatible streaming platform for miss... by rystsov on Monday November 16, 2020 @11:13AM (#25113372)
Yes and no, we do both. For ack=-1 we wait for fsync & replication confirmation but for other modes we relaxed the behavior.
Re:Redpanda – A Kafka-compatible streaming platform for miss... by agallego on Monday November 16, 2020 @11:07AM (#25113302)
not sure how you got here tho, maybe i miss explained something. This is talking about filesystem api not about kafka api. the kernel does something similar, standard filesystem things just purpose built for our use case.
Re:Redpanda – A Kafka-compatible streaming platform for miss... by fnord123 on Monday November 16, 2020 @02:41PM (#25116057)
afaics a write behind strategy means you will report a write as being complete before it has actually been written. This can result in terrific performance because your batching window is larger and you can amortize more (fewer?) iops. But it also means in the face of failure, clients could believe something was written which wasn't.

On the other hand, the storage device can fail so what does it even mean to have written the data? :)

Re:Redpanda – A Kafka-compatible streaming platform for miss... by agallego on Monday November 16, 2020 @03:02PM (#25116309)
our acks are similar to kafka in that regard. acks=0,1 leader acknowledgement. acks=-1 we go further like denis mentioned above by 'flushing' to disk before responding which is stronger (log completeness guarantee from raft) than available in upstream.
Re:Redpanda – A Kafka-compatible streaming platform for miss... by agallego on Monday November 16, 2020 @10:25AM (#25112807)
Sure. Gists are

No use of page cache instead custom built read ahead and write behind strategy

Io scheduling groups from seastar

Custom intra cluster RPC to optimize for fragmented buffers native support (most storage work is small materialized metadata and large blob)

Thread per core design with zero locks on the hotpath

If you watch our chat with the LinkedIn kafka team, you will learn a bit more too

Hopefully this helps

Re:Redpanda – A Kafka-compatible streaming platform for miss... by steeve on Monday November 16, 2020 @06:15AM (#25110408)
Better exploiting hardware (via seastar and the one-thread-per-core model, see ScyllaDB vs Cassandra also) and probably better architecture.
Re:Redpanda – A Kafka-compatible streaming platform for miss... by BenoitP on Monday November 16, 2020 @07:58AM (#25111058)
This is sort of what I'm getting at. The one-thread-per-core part should bring little to no benefit vs JDK+Loom, for example.
Re:Redpanda – A Kafka-compatible streaming platform for miss... by agallego on Monday November 16, 2020 @10:28AM (#25112849)
Its a bit deeper.

Custom strategies for reading and writing have a huge impact. (orders of magnitude for specific workloads rather than generic page cache scheduling)

Optimizing for fragmented buffers. Etc. See the LinkedIn vid from the blog posts for more low level deets.