Re:What makes Apache Kafka fast (2020)
by zihotki on Sunday February 28, 2021 @11:50AM
(#26294642)
TLDR: They use many good industry practices:
- Append-only log with sequential IO
- Record batching on client and broker side (works well with sequential IO)
- Batch compression (Json and other text formats could be compressed very well)
- Unflushed buffered writes
- Zero-copy approach for working with data
In addition to that the architecture of the Kafka also plays a major role so that the clients don't require much overhead. The code is written to not to put much pressure on GC and a number of other more specific optimizations was implemented on both client and server sides.
The most interesting part for me was how they achieve zero-copy when working with sockets. That whole section is worth reading and they could have made another blog post about only that.
- Append-only log with sequential IO
- Record batching on client and broker side (works well with sequential IO)
- Batch compression (Json and other text formats could be compressed very well)
- Unflushed buffered writes
- Zero-copy approach for working with data
In addition to that the architecture of the Kafka also plays a major role so that the clients don't require much overhead. The code is written to not to put much pressure on GC and a number of other more specific optimizations was implemented on both client and server sides.
The most interesting part for me was how they achieve zero-copy when working with sockets. That whole section is worth reading and they could have made another blog post about only that.
Edit: formatting