Re:80M I/O Per Second with a Standard 2U System
by jorangreef on Friday May 28, 2021 @08:29AM
(#27314570)
We use a similar technique for TigerBeetle [1], a financial accounting database, to do a million financial transactions a second with full distributed persistence and reasonably low stable latency (< 20ms).
We have around 10,000 financial transactions in a batch and amortize away the distributed consensus (Viewstamped Replication), the many assertions, syscalls and O_DSYNC so that they are almost free.
Direct I/O helps to avoid flushing the CPU cache through memcpy's to the kernel's page cache for a roughly 7% improvement, and io_uring helps to amortize the syscalls themselves into a single io_uring_enter() for a doubling of throughput for small 4 KiB read/write units, but having a first-class batching interface from end to end is the biggest win by far.
I've started playing around with new storage architectures for databases and with flash storage in mind. One thing I encountered that seems incredibly powerful, when combined with a log-based store, is micro batching of operations.
When you aggregate a few hundred/thousand transactions together and then append them as modified subtrees to the end of a log, you start to be able to say crazy shit like "Transactions per I/O operation or block"
When working with objects in the 512b range, I can easily hit 2 million inserts per second on a single Samsung 960pro using this technique.