> The guarantee is not that the system never enters one of those states, the guarantee is that if the system enters one of those states it never (well, so extremely rarely that it's "never" for all practical purposes) fails to recognize that it has done so. Why do you find that so implausible?

Stated so, that is plausible. However, "it is possible to engineer a system that never fails to detect that it has failed" is not; I claim that any subset of states which is amenable to this level of detectability will exclude some other states that any normal person would also consider to be "failure".

> Stated so, that is plausible. However, "it is possible to engineer a system that never fails to detect that it has failed" is not

They seem the same to me. In fact, in 27 years you are the first person to voice this concern.

> I claim that any subset of states which is amenable to this level of detectability will exclude some other states that any normal person would also consider to be "failure".

Could be, but that would be a publishable result. So... publish it and then we can discuss.

What happens if all your sensors fail, including the one that senses the failure of your sensors? What happens if the power source disconnects?

Having all your sensors fail is actually a very easy case. Imagine if all of your sensors failed: suddenly you could not see, hear, feel, smell, or taste... do you think it would be hard to tell that something was wrong?

Having your sensors fail doesn't mean they're not providing data. It means they're not providing accurate data. In humans, we would call this hallucinating, and humans in fact cannot generally tell that they are hallucinating.

> Having your sensors fail doesn't mean they're not providing data.

That is one possible failure mode. But you're right, it's not the only one.

There is an extensive literature on how to detect and correct sensor errors resulting from all kinds of different failure modes.

Arguing that it's possible to build an autonomous that is sometimes wrong but always knows when it's wrong seems like a stretch to me.

A system that can sense when it is in a failure mode is analogous to a “detector” in the Neyman-Pearson sense. A detector that says it is OK when it is actually in a failure condition is said to have a missed detection (MD).

OTOH, if you say you’ve failed when you haven’t, that’s a false alarm (FA).

In general, there is a trade-off between MD and FA. You can drive MD probability to near-zero, but typically at a cost in FA.

Again in general, you can’t claim that you can drive MD to zero (other than by also driving FA probability to one) without knowing more about the specifics of the problem. Here, that would be the sensors, etc.

In particular, for systems with noisy, continuous data inputs and continuous state spaces -- not even considering real-world messiness -- I would be surprised if you could drive PD to zero without a very high cost in FA probability.

As a humbling example, you cannot do this even for detection of whether a signal is present in Gaussian noise. (I.e., PD = 1 is only achievable with PFA = 1!) PD = 1 is a very high standard in any probabilistic system.

Discrete-input systems can behave differently.

You've done a great job of describing the problem. It is manifestly possible to drive missed detection rates very close to zero without too many false alarms because humans are capable of driving safely.

Ah, yes... you have convinced me - though humans can apply generalized intelligence to the problem, which I imagine is particularly useful in lots of special cases.

General intelligence is not needed. Illiterate people can drive. People who can't do math can drive.

Illiteracy is not incompatible with intelligence.

So we can get within epsilon for an undefined delta because humans can do something, although not always.

Right, that whole claim about engineering a system that always knows when it’s not working sounds rock solid to me. After all, we can build a human, right?

> we can build a human, right?

Not yet. But there's no reason to believe we won't be able to eventually.

You can if you dial the specificity down to 0.

i.e: Always report "I'm wrong."

Humans have moments like this too. I remember suddenly driving into dense fog on the freeway and not able to see more than 20 feet in front of me. I had a definite "failure mode" where I slowed down and freaked out because all my previous driving experience was "failing" me on how to avoid a potential accident.