Re:Storing state in Erlang with processes (2014)
by derefr on Friday May 31, 2019 @11:31AM
(#20061807)
There’s a higher, systems-architecture-level equivalent to this, too (which Erlang is also designed to take advantage of, re: the Erlang distribution protocol, and the Mnesia DBMS), and that’s the idea that “durability is replication.”
That is, there’s no semantic difference between “a disk” and “the in-memory heap of another node on the network”, in terms of what fault-tolerance guarantees you get by adding one or the other to your system. Machines can crash? So can disks. Back up to a disk? Back up to another node. Restore from on-disk state? Restore by streaming state from another node.
You can build a DBMS cluster, or even something “ultra-durable” like S3, with clusters of disk servers, sure. Or — providing memory is cheap-enough for your use-case — you can build it with clusters of RAM nodes, with no disks at all (and thus one tenth the maintenance costs.) Power outage wipes the whole cluster? Not if you’ve got a UPS, a generator, and a whole lotta gas (like telecom base-station switches are stocked with.) Or if you’ve got backup nodes — which are also just RAM nodes! — to stream to in another region. (Your system itself doesn’t need to be multi-region distributed; it can treat these nodes the same way a regular persist-to-disk system treats tape backup.)
And, conveniently, it’s not just memory accesses that are faster with such an architecture. If you build your cluster architecture as a set of services where each service isn’t just a single node (or just a load-balanced set of nodes), but rather a “distribution set” of nodes—i.e. one “transaction router” node that multicasts commands to a set of equivalent worker nodes, where the worker nodes all deterministically do the same things in response; then, rather than one master, you have N copies of your node, all of which are equivalent hot-standbys of your “master”! (You don’t have to use them as such; you can route client requests to just one “master” in the distribution set, making the others merely warm standbys.) Compared to streaming replication, where some of your nodes are always “behind” their master, this approach is far less risky in terms of data loss. This isn’t streaming replication; it’s RAID for memory writes!
(And yes, this is the precise use-case that Erlang’s distribution protocol is designed for: nodes in the same “distribution set” working as equivalent, deterministic warm-standby alternatives to one-another, presenting as a single virtual “node” in a larger cluster architecture, almost always connected by a non-partitionable network backplane, like a single top-of-rack network switch. Of course, nobody uses Erlang’s distribution this way other than Ericsson... but that’s because nobody but Ericsson has needs that force them to avoid disks—and avoiding disks entirely isn’t cheap, at least today. Still, it’s important to keep in mind, because the Erldist protocol works best when you hew to this design, and actively fights you when you don’t; and because Mnesia was never designed for anything other than being a memory-to-memory replicated DBMS—and it works great for that!—but it really kind of sucks once you introduce any disk copies to its replication strategy.)
That is, there’s no semantic difference between “a disk” and “the in-memory heap of another node on the network”, in terms of what fault-tolerance guarantees you get by adding one or the other to your system. Machines can crash? So can disks. Back up to a disk? Back up to another node. Restore from on-disk state? Restore by streaming state from another node.
You can build a DBMS cluster, or even something “ultra-durable” like S3, with clusters of disk servers, sure. Or — providing memory is cheap-enough for your use-case — you can build it with clusters of RAM nodes, with no disks at all (and thus one tenth the maintenance costs.) Power outage wipes the whole cluster? Not if you’ve got a UPS, a generator, and a whole lotta gas (like telecom base-station switches are stocked with.) Or if you’ve got backup nodes — which are also just RAM nodes! — to stream to in another region. (Your system itself doesn’t need to be multi-region distributed; it can treat these nodes the same way a regular persist-to-disk system treats tape backup.)
And, conveniently, it’s not just memory accesses that are faster with such an architecture. If you build your cluster architecture as a set of services where each service isn’t just a single node (or just a load-balanced set of nodes), but rather a “distribution set” of nodes—i.e. one “transaction router” node that multicasts commands to a set of equivalent worker nodes, where the worker nodes all deterministically do the same things in response; then, rather than one master, you have N copies of your node, all of which are equivalent hot-standbys of your “master”! (You don’t have to use them as such; you can route client requests to just one “master” in the distribution set, making the others merely warm standbys.) Compared to streaming replication, where some of your nodes are always “behind” their master, this approach is far less risky in terms of data loss. This isn’t streaming replication; it’s RAID for memory writes!
(And yes, this is the precise use-case that Erlang’s distribution protocol is designed for: nodes in the same “distribution set” working as equivalent, deterministic warm-standby alternatives to one-another, presenting as a single virtual “node” in a larger cluster architecture, almost always connected by a non-partitionable network backplane, like a single top-of-rack network switch. Of course, nobody uses Erlang’s distribution this way other than Ericsson... but that’s because nobody but Ericsson has needs that force them to avoid disks—and avoiding disks entirely isn’t cheap, at least today. Still, it’s important to keep in mind, because the Erldist protocol works best when you hew to this design, and actively fights you when you don’t; and because Mnesia was never designed for anything other than being a memory-to-memory replicated DBMS—and it works great for that!—but it really kind of sucks once you introduce any disk copies to its replication strategy.)