I think that TFA gets it exactly backwards. It's not that we will be able to treat multi-node systems as non-distributed it's that single-nodes will have to start being treated like distributed systems.

> The links between the components of your computer are solid and cannot fail like actual computer network connections.

I've personally had this disproven to me on multiple occasions.

>I've personally had this disproven to me on multiple occasions.

That sounds like interesting stories! Can you elaborate?

Accidents on desktop hardware:

Multiple bad disk cables (more common in IDE era, but happened once with SATA). Interestingly enough, Windows would reduce the drive speed on certain errors, so I had a drive that booted up in UDMA/133 and the longer it was running the slower it got, eventually settling in at PIO mode 2. Switching the drive cable fixed it.

A sound card that wasn't screwed in to the case, so if you pushed the phone connector in too hard it would unseat. I still don't know how that happened; it must have been me (unless someone pranked me) but the sound-card hadn't been changed in like 2 years at that point.

A DIMM wasn't fully clipped in, but the system worked fine for weeks until someone bumped into the case.

Things that were actually intentional:

We expect anything plugged in externally (e.g. USB, ethernet, HDMI) to be plugged and unplugged without needing to restart the system. This sounds banal, but wasn't always the case. I had a network card with 3 interfaces (10BASE5 AUI, 10BASE2 BNC, 10BASE-T modular plug) and you needed to power off the system and toggle a DIP switch to change which was in use.

I've seen server and minicomputer hardware with hotpluggable CPUs and RAM

Eurocard type systems (e.g. VME, cPCI) could connect all sorts of things, and could run without restarting. This sort of blurs the line as to what a "node" is. If you have multiple CPUs on the same PCI bus, is that one node or many?

eGPUs have made hotplugging a GPU something that anyone might do today. If you run this setup, then the majority of the computational power in your system can appear and disappear at will, along with multiple GB of RAM.

There have been machines tolerant to CPU and Mem failures, and to a certain extent this sorta works on some of the higher end machines that support ram/cpu hotplug. (historically see hp/tandem/nonstop, sunos/imp, etc).

The problem is linux's monolithic model, doesn't work well for kernel checkpoint/restore despite it actually supporting hotplug cpu/ram it they have to be gracefully removed.

So, this is less about the machine being distributed, and more about the fact that linux is the opposite of a microkernel/etc that can isolate and restart its subsystems in the face of failure. Its also sorta funny that while these types of operations tend to need to be designed into the system, the last major OS's designed this way were done in the 1980's.

I know of no OSes that are resilient to CPU cores producing wrong results (or incorrect mem results: I consider ECC a lower level concern that is not part of the OS), whereas a lot of distributed consensus algorithms have this built into their requirements. EDIT: I have heard through the grapevine that something like this might be done for aerospace, but I have no personal experience with that.

I agree with parent. The major reason why programming on a single computer is easier than a distributed system is that we assume total resilience of various components that we cannot for a distributed system.

From the article:

> This offers hope that it is possible to some day abstract away the distributed nature of larger-scale systems.

To do this is not a question of software abstractions, but hardware resilience. If we have a network which we can reasonably assume to have 100% uptime and absolutely no corruption between all its components then we can program distributed systems as single computers.

Most distributed consensus algorithms, or distributed systems in general, are not resilient to nodes producing arbitrary wrong results. That's the realm of systems like Bitcoin, which achieve such resilience by paying big performance costs.

So it shouldn't be surprising that computers have the same lack of resilience.

Sorry what? That is exactly the purpose of Byzantine fault tolerant consensus algorithms, which have been around for many years.

Sure, that doesn't contradict what I said. They have been around for years, and they are very expensive and almost all systems do not use them.

They are a lot more efficient than bitcoin. But agreed, not cheap.

The tandems I listed above, originally used lock stepped processors, along with stratus/etc.

edit: Googling yields few results that aren't actual books, Try this

https://books.google.com/books?id=wBuy0oLXEuQC&pg=PA218&lpg=...

Ah well there you go. Had no idea they used lock stepping!

It doesn't count as resilient in the mainframe sense, but in an effort to encourage system management, I ran the Node Health Check system on our "commodity" HPC cluster and found multiple failed DIMMs and a failed socket no-one had noticed. (I'd had enough alerts from that on a cluster I managed.)

You can 'disable' a core in Linux pretty easily, although I'm not sure to what extent you'd consider this graceful (in the sense that you write to a system file and then some magic, which may be arbitrarily complicated I guess, happens in the background. So it doesn't seem equivalent to just yanking a core from the package, if that were possible).

The article also ignores that e.g. the CUDA API looks nothing like a local function call. People are explicitly aware when they are launching GPU kernels.

>Yes but your computer will not gracefully handle CPUs randomly failing or RAM randomly failing

That's incorrect.

There are plenty of machines/OSs which are (or can be) resilient to a CPU failing; Linux, for example. From the OS point of view, you just kill the process that was running on the CPU at the time and move on.

Resilience to spontaneous RAM failures is rarer but possible.

Killing the processes running on the compute element seems not very graceful, right? I'd expect a gracefully handled failure to have some state staved from which the computation can be continued.

Which would be overkill on a single node, given that CPUs don't really fail all that often.

It's up to userspace to do more than that. There are other issues which can cause processes to be spontaneously killed (OOMkiller for example) so it's something you should be tolerant of.

Disagree. An environment that's being reaped by OOMK is not stable enough to make assumptions about. You're in "go down the hall and turn it off and on again" territory.

Attempting to account for such environments in user programs massively inflates their complexity, does little to enhance reliability, and the resulting behavior is typically brittle or outright broken from the get go.

This is why, for example, the C++ committee flirts with making allocation failure a UB condition.

iOS constantly runs an OOM killer and works pretty well.