Re:Databases = Frameworks for Distributed Systems
by jandrewrogers on Tuesday May 24, 2022 @10:50AM
(#31492856)
While databases having been converging on composability for a long time, the model presented here is not the correct one and causes problems for many important and emerging use cases.
As an elementary observation, isolating the designs of storage, compute, and networking as separate modules is a reliable way to achieve poor performance and resource efficiency. Some of the most important scalability and performance optimizations in modern database architecture are schedule-driven, which requires tight coupling across the design of storage, compute, and networking. Ignoring this class of optimizations produces very large reductions in real-world workload performance, and feeds a narrative that the software is wasteful and not environmentally friendly.
Heterogeneous and specialized resource collections with complex topologies are becoming more common for a diverse set of reasons. This breaks most horizontal resource abstractions. The practice of treating every thread as a complete monolithic database kernel connected together with protocols that are good at equilibrium finding lends itself to this environment very well, and the edge (for which database technology is broadly broken), in addition to being ideal for the boring data center case where the topology is highly regular and every node has uniform resources.
Most of our distributed systems are modeled as giant distributed file systems, and that fits okay with the idea of horizontally partitioning compute, storage, and network. This is easy for developers to reason about because they are familiar with file systems. However, it is difficult to express important database-y operations in this model. For example, if you want to do ad hoc joins in a scale-out environment, you need mechanics like decentralized parallel orchestration which isn't really a concept in a "distributed file system" model of "distributed".
Databases have been slowly converging on latency-hiding HPC software architectures (which has no implication of HPC hardware), which look very different than a distributed file system, can express more powerful capabilities, and are generally more scalable. To the extent there is a general principle and framework behind databases, it is more this than traditional distributed systems.
As an elementary observation, isolating the designs of storage, compute, and networking as separate modules is a reliable way to achieve poor performance and resource efficiency. Some of the most important scalability and performance optimizations in modern database architecture are schedule-driven, which requires tight coupling across the design of storage, compute, and networking. Ignoring this class of optimizations produces very large reductions in real-world workload performance, and feeds a narrative that the software is wasteful and not environmentally friendly.
Heterogeneous and specialized resource collections with complex topologies are becoming more common for a diverse set of reasons. This breaks most horizontal resource abstractions. The practice of treating every thread as a complete monolithic database kernel connected together with protocols that are good at equilibrium finding lends itself to this environment very well, and the edge (for which database technology is broadly broken), in addition to being ideal for the boring data center case where the topology is highly regular and every node has uniform resources.
Most of our distributed systems are modeled as giant distributed file systems, and that fits okay with the idea of horizontally partitioning compute, storage, and network. This is easy for developers to reason about because they are familiar with file systems. However, it is difficult to express important database-y operations in this model. For example, if you want to do ad hoc joins in a scale-out environment, you need mechanics like decentralized parallel orchestration which isn't really a concept in a "distributed file system" model of "distributed".
Databases have been slowly converging on latency-hiding HPC software architectures (which has no implication of HPC hardware), which look very different than a distributed file system, can express more powerful capabilities, and are generally more scalable. To the extent there is a general principle and framework behind databases, it is more this than traditional distributed systems.