News for hackers. Stuff that matters.
Re:Rust is the future, C is the new Assembly: Josh Triplett ... by steveklabnik on Wednesday August 28, 2019 @09:46PM (#20825939)
Have you looked at TiKV/TiDB? I’m curious what you think of them.
Re:Rust is the future, C is the new Assembly: Josh Triplett ... by jandrewrogers on Thursday August 29, 2019 @02:02PM (#20832024)
TiDB is built as a set of abstractions on top of RocksDB, inheriting weaknesses intrinsic to that style of architecture. Two things I would call out about the design:

First, loose coupling through the layers of the stack probably gives up an order of magnitude in throughput on the same hardware versus a state-of-the-art tightly coupled architecture for many workloads. Many optimizations are enabled when high level orchestration and operation semantics (like a SQL join) are understood at every level of the stack down to DMA scheduling. Most open source databases use loose coupling because (1) it is much simpler to design and (2) developers can reuse third-party code for parts they lack the expertise or resources to implement themselves.

Second, RocksDB is a modern implementation of an obsolete design. As a practical matter, this has a high cost in terms of storage scalability and throughput. Many open source projects use it because they don't have a clear idea of how to design something better. Applications where storage engine performance and scalability are critical, e.g. real-time sensor data models, is an area where open source is not remotely competitive at the moment.

Unfortunately, these criticisms could be leveled at most open source database engine designs, which tend to optimize for expediency and ease of implementation rather than performance, and with little awareness of how much performance is being sacrificed. The divergence between academic literature and state-of-the-art in databases is now almost unrecognizable, and the handful of people with expertise in high-end implementation are not working on open source systems.