I know this pain as an embedded software monkey. So many edge cases unthought of and no method to gracefully fail. Not to mention you always find some fault with the I²C implementation on the SOC or slave device you want to talk to.
I’ve spent weeks debugging bus lockup problems only to find erratas for the silicon, or find this particular chip needs an extra 100ms to startup than others.
It’s the one communication protocol that o always have problems with. Nothing else has caused me as many headaches.
[0] https://github.com/oxidecomputer/hubris/blob/master/app/giml...
I’m not familiar with Oxide, but the I2C physical layer is just a pair of open-drain signals, so I wouldn’t worry about a level-shifter too much.
Sure, but physical damage can disconnect it.
Whereas in a distributed system, a single broken communication line, even or especially if it's extremely important, still means that the distributed system has to recover somehow, preferably gracefully.
Turning on a computer, compare the probability of an issue with the ethernet cable, ie not plugged in as suggested vs a pcb trace being broken. Given that probability do you want to handle one of those gracefully and not fry the machine? Both? Neither. And that's the point of identifying the difference. A thing that is rare and if it occurs everything is b0rked anyway doesn't need to be handled.
That individual components having to be robust on some level (what level? It depends...) to failure difference of distribution via a network connection and pcb traces is probably the reason we don't usually (but we might) refer to the latter as a distributed system.
This is a neat line of thought, but I don't think it can go very far. There is a huge difference in reliability and predictability between small-scale and large-scale systems. One way to see this is to look at power supplies. Two ICs on the same board can be running off of the same 3.3V supply, and will almost certainly have a single upstream AC connection to the mains. When thinking about communications between the ICs, you don't have to consider power failure because a power failure will take down both ICs. Compare this to a WiFi network where two devices could be on separate parts of the power grid!
Other kinds of failures are rare enough to be ignored completely for most applications. An Ethernet cable can be unplugged. A PCB trace can't.
I used to work with a low-level digital communication protocol called I²C. It's designed for communication between two chips on the same board. There is no defined timeout for communication. A single malfunctioning slave device can hang the entire bus. According to the official protocol spec, the recommended way of dealing with this is to reset every device on the bus (which may mean resetting the entire board). If a hardware reset is not available, the recommendation is to power-cycle the system! [1]
Now I²C is a particularly sloppy protocol, and higher-level versions (SMBus and PMBus) do fix these problems, so this is a bit of an extreme example. But the fact that I²C is still commonly used today shows how reliable a small-scale electronic system can be. Even at the PC level, low-level hardware faults are rare enough that they're often indicated only by weird behavior ("My system hangs when the GPU gets hot"), and the solution is often for the user to guess which component is broken and replace it.
[1] Section 3.1.16 of https://www.nxp.com/docs/en/user-guide/UM10204.pdf