Skimming the repository it doesn't seem to be an unreasonable claim. C++ with Seastar compared to Scala on JVM. It's in line with the Scylla/Cassandra improvements.
https://vectorized.io/dataengineering-podcast-interview/
The throughput is 4x, the latency is 10x.
The reads are more complex since it involves understanding io priority scheduling etc
You ack writes before they're written?
On the other hand, the storage device can fail so what does it even mean to have written the data? :)
No use of page cache instead custom built read ahead and write behind strategy
Io scheduling groups from seastar
Custom intra cluster RPC to optimize for fragmented buffers native support (most storage work is small materialized metadata and large blob)
Thread per core design with zero locks on the hotpath
If you watch our chat with the LinkedIn kafka team, you will learn a bit more too
Hopefully this helps
Custom strategies for reading and writing have a huge impact. (orders of magnitude for specific workloads rather than generic page cache scheduling)
Optimizing for fragmented buffers. Etc. See the LinkedIn vid from the blog posts for more low level deets.
It'd be great to know where do the improvements came from.
Is it from a different architectural design, is it from better exploitation of new hardware, is it from JVM limitations? For example, when Valhalla and Loom land in the JDK, how much of this improvement will stay?