A couple of factors.

1) The yield rates for spaceflight-qualified chips is very, very low. Like 1%-5% or so. The chips are inspected when they come out of fab, and only the most perfect ones are given a spaceflight certification. The rest of the chips are used for other, less stringent applications (test boards, or military/embedded applications).

2) Spaceflight parts have significant paper trails. For metal parts, they are traced from the moment a lot of material comes out of the mill, and every time it is touched or changes hands thereafter that fact is recorded. Same thing with chips. Every chip has a "traveler" associated with it that records when it was manufactured, how it was stored, etc. Keeping those records costs a surprising amount of money. Handling the parts so the paper trail can be kept costs even more. You have to organize your logistics train such that every part is individually trackable. That reduces efficiency and adds cost.

3) Relatively low economies of scale.

Commercial aircraft also track provenance very carefully. I wonder how much the two systems have informed each other over the years.

You would never want a part involved in a stress test to be reused, and you certainly don't want test parts anywhere near a production craft. This isn't like IT where you can cannibalize parts from a QA box to put into a production server.

Having worked on mission critical commercial aircraft electronics, I can assure you every part is stress tested before it goes on a plane. Both after its first made and every time its in for service. Supposedly there has been research done showing that stress testing does not adversely affect reliability, at least at this quality level

Also, the paper trail is quite literally paper. Its kinda amazing how slow the aviation industry is to adopt new things

It's possible I'm mucking up the jargon, but isn't stress testing electronics what most of us might call "burn-in"? I definitely want the flight control computer burnt in before I sit on the airplane.

I was thinking more of physical components, like an aluminum bracket, and you don't want a part with metal fatigue being installed as new.

Correct, thermal testing. Not vibration

I saw a video once saying that SpaceX uses general purpose computers on the rockets instead of specific purpose hardware. If I'm not mistaken, they amount to 6 and have a checking system to assure the output of them is the same.

Yes and a SpaceX rocket computer is not in space very long and is mostly in the low radiation environment of LEO. These Mars rovers spent years in a high radiation environment and received a large total ionizing dose of radiation. Having multiple computers doesn't do anything for total ionizing dose, they are all going to fail around the same time.

I'm not sure about the RAD6000 being discussed here, but its successor, the RAD750, is fabbed with silicon-on-sapphire to help with total ionizing dose. For single event upsets, there is triple modular redundancy for all logic in the CPU.

Those "have N copies and compare them" systems then have the issue that whatever does the comparison is a single point of failure. You could use multiple comparison units and have a second level comparison check them, but then that is your single point of failure, and so on.

I've been told, but never actually looked it up, that there is a theorem that proves you always have to have at least one single point of failure.

I don't know if the following is actually true, or just a rumor, but I've heard that at least one aircraft whose mission called for very high reliability didn't have a comparison unit: the redundancy extended all the way to having the 3 independent flight control computers each control a separate actuator on each flight control surface. If one of the systems went bad and tried to move the surface incorrectly, the other two would physically overpower it.

That still has a single point of failure, but now that point is the control surface itself. If your control surface itself has failed it no longer matters if the 3 computers controlling it agree.

> I've been told, but never actually looked it up, that there is a theorem that proves you always have to have at least one single point of failure.

In what context? There's a theorem that arbitrarily-reliable computation can be done with noisy components, as long as the noise is below some threshold (e.g. picture less than 1 error per 10 operations). [1]

1: von Neumann, J. (1956). "Probabilistic Logics and Synthesis of Reliable Organisms from Unreliable Components", in Automata Studies, eds. C. Shannon and J. McCarthy, Princeton University Press, pp. 43–98 http://www.cyclify.com/wiki/images/a/af/Von_Neumann_Probabil...

So long as you can trade off robustness for performance, the fact that there must be a single point of failure is less important. You make the single point of failure more robust. In the example of the control surface, perhaps some ionizing radiation could flip bits in the flight control computers, but it takes a physical collision to damage the control surface.

To argue otherwise is to imply that all designs and all systems are equally robust, which is clearly not true.

This is something I'd always wondered about and never gotten a straight answer to. You're the first person I've seen who explicitly confirmed my suspicion that every system has a single point of failure. Would love to find a reference for this...

I thought the way these things worked is that each of the N computers can see the state of the other N-1 computers; if your local state differs from the majority opinion you overwrite it, thereby joining the majority.

I'm curious if this SPoF theory stops at computer/real world interfaces (such as the control surface <-> computers). Or if this generalizes into purely computational models. In my mind I can't find any purely computational model using distributed techniques that succumbs to the SPoF "theory." But the computer/real world interface is trivially obvious in just about every context.

I don't think it's strictly true that you always have to have a single point of failure. Trivial counterexamples: 6 people carrying a sofa, an ant colony, BitTorrent.

The closest thing I can remember encountering to what you describe is the "Contracrostipunctus" chapter in Douglas Hofstadter's Godel, Escher, Bach, where he writes a dialog featuring record players as an analogy to Godel's Incompleteness Theorem (which only applies to "formal systems" - descriptive mathematical languages). He does go on to explore a real-world example of the principle in the form of viruses - a cell cannot fully defend against DNA modification using only instructions found in its DNA. The same principle applies to cracking copy protection in games - no matter how elaborate the validity checks, there's always a single point of failure in the form of the final decision - "if(checks_pass){run_game()}" - which can be trivially short circuited with a debugger.

I'm not a good enough mathematician to fully understand the limits of Godel's Theorem. But it seems to me that all of the above applications are examples of some sort of well defined formal computational system, and you can't generalize it to "everything has a single point of failure" without some carefully defined rules as to what constitutes the boundaries of system.

http://www.math.yorku.ca/Who/Faculty/Brettler/3500_06/Godels...

Low sales, meaning the R&D cost (which'll be higher than a non rad hard chip anyway) can't be as easily amortized. And probably half of it is or so is just the .gov markup.

The costs of employing a team to do the radiation hardening design work, running a non-mainstream fab (I believe these chips were silicon-on-sapphire) and small production runs. There's a lot of overheads amortised on to a very small number of chips.

I believe they're fabbed in Virginia, which probably doesn't help costs either

Low yields, low volume, high development cost, high certification costs, strict QA.

Vorago makes a cheaper option these days at $800 a chip: https://www.voragotech.com/products/va10820-radiation-harden...

That's a Harvard architecture microcontroller, a bit different from the POWER CPUs in the rovers.

Still subject to the same engineering constraints, but significantly less complex.

They're not comparable chips. The RAD6000 has a thousand times more RAM, for starters.

My understanding is that most of that is setup costs for retooling a fab, which requires moving around and recalibrating some very specialized equipment. The numbers I heard were dozens of engineers working a total of one to two thousand man hours to set up the process, make and verify the chips, then switch the fab back to the processes their other customers usually order. Since these semiconductor fabs have high upfront and fixed operating costs, they need to have a high utilization to be profitable - preventing anyone from having a specialized fab that makes only RAD hardened chips.

AFAIK the lower cost VORAGO designs require far less retooling so they're a lot cheaper with existing processes.

I would assume the kinds of use cases that people might need radiation hardened equipment for just have people willing to spend much more money. They're charging what people are willing to pay, not how much it costs to make.