Having worked on a good bunch of critical system, there aren't multiple safeguards in most hardware.
E.g. a multiplication error in a core will not be detected by an external device. Or a bit flip when reading cache, or from a storage device.
Very often the only real safeguard is to do the whole computation twice on two different hosts. I would rather have many low-reliability hosts and do computation twice than few high-reliability and very expensive host.
Unfortunately the software side is really lagging behind when it comes to reproducible computing. Reproducible builds are a good step in that direction and it took many decades to get there.
The quote I made is actually very specifically is in reference to this paper, in particular point 2, which is mandatory reading for any systems engineer, IMO, though perhaps the word "safeguard" is too strong for the taste of some here. But focusing on definitions of words is besides the point and falls into the same traps this paper mentions: https://how.complexsystems.fail/
Back to the original point: is ECC the single solution to this catastrophe? No, probably not. Systems are constantly changing and failure is impossible to eliminate. Design decisions and a number of other decisions could have mitigated it and caused this failure to not be catastrophic. Another thing might also cause it to topple. But let's not pretend like we don't know what we're dealing with, either, when we've already built these tools and know they work. We've studied ECC plenty! You don't need to carry water for a corporation trying to keep its purse filled to the brim (by cutting costs) to proclaim that failure is inevitable and most things chug on, regardless. We already know that much.
I laugh when I compare my day to day coding to that of an avionics programmer in the aero industry.
It's sad how immature the software industry can be. It's been around for "only" 60/70 years after all.
1. https://www.usenix.org/system/files/conference/atc12/atc12-f...
Three different hosts, for quorum, right?
Sometimes doing the same processing twice can be also a way to implement safe(r) rolling updates.
Edit: there's some new (to me) information from real world results, interesting read. https://www.zdnet.com/article/dram-error-rates-nightmare-on-...
Looks like things are worse than I thought (but still better than most people seem to think). Interesting to note that the motherboard used affects error rate, and it seems that part of it is a luck of the draw situation where some dimms have more errors than others despite being the same manufacturer
When you are designing these systems, you have two options: you either use error correcting codes and increase channel bandwidth to compensate for them (as a result of injected noise, which is unavoidable), or you lower the transfer rate so much as to be infeasible to use, while also avoiding as much noise as you can. Guess what's happening to RAM? It isn't getting slower or less dense. The error rate is only going to increase. The people designing this stuff aren't idiots. That's why literally every other layer of your system builds in error correction. Software people do not understand this because they prefer to believe in magic due to the fact all of these abstractions result in stable systems, I guess.
All of the talk of hardware flip flops and all that shit is an irrelevant deflection. Doesn't matter. It's just water carrying and post-hoc justification because, again, Intel decided that consumers didn't actually need it a decade ago, and everyone followed suit. They've been proven wrong repeatedly.
Complex systems are built to resist failure. They wouldn't work otherwise. By definition, if a failure occurs, it's because it passed multiple safeguards that were already in place. Basic systems theory. Let's actually try building more safeguards instead of rationalizing their absence.