Re:80M I/O Per Second with a Standard 2U System
by jorangreef on Friday May 28, 2021 @08:29AM
(#27314570)
We use a similar technique for TigerBeetle [1], a financial accounting database, to do a million financial transactions a second with full distributed persistence and reasonably low stable latency (< 20ms).
We have around 10,000 financial transactions in a batch and amortize away the distributed consensus (Viewstamped Replication), the many assertions, syscalls and O_DSYNC so that they are almost free.
Direct I/O helps to avoid flushing the CPU cache through memcpy's to the kernel's page cache for a roughly 7% improvement, and io_uring helps to amortize the syscalls themselves into a single io_uring_enter() for a doubling of throughput for small 4 KiB read/write units, but having a first-class batching interface from end to end is the biggest win by far.
We have around 10,000 financial transactions in a batch and amortize away the distributed consensus (Viewstamped Replication), the many assertions, syscalls and O_DSYNC so that they are almost free.
Direct I/O helps to avoid flushing the CPU cache through memcpy's to the kernel's page cache for a roughly 7% improvement, and io_uring helps to amortize the syscalls themselves into a single io_uring_enter() for a doubling of throughput for small 4 KiB read/write units, but having a first-class batching interface from end to end is the biggest win by far.
[1] https://github.com/coilhq/tigerbeetle