I hear you, but my point is "what do our brains do?"
In the case of driving, if the answer is "drive a car", then that is a bit like "brexit means brexit" :) The definition of the thing is the outcome of the thing.
Instead, if you try to can decompose the question into a more useful form, one that can drive a specification of software, then progress can be made. However, all that has done, in that case, is a description of the outcome, not the process. (And incomplete, due to the complexity).
It is at this point where my problem with all this AI stuff starts :)
The lack of understanding of the process, rather than the outcome is why I call it voodoo, and anyone talking different is trying to impress, in my opinion.
While we don't understand how our brain works to the bits and bytes, our knowledge and understanding of how our brain functions is Growing. A fuller understanding of how our neurotransmitters work on a chemical/biological level, and an effective way to model it, will likely result in interesting advances in the field of AI
I chat with my neuroscience friend a lot, and the overlap I see between how he describes how the brain works, and general computer science concepts is surprising.
In my view it would be unethical to stop the deployment of such AIs on the grounds that we don't "understand" them well enough or can't predict their behavior in every hypothetical circumstance, because we can still test them well enough to know with statistical certainty that they will save many, many lives.
Estimates about availability have consistently been overly optimistic. This may be a case where the first 99% of the work takes 99% of the effort, and the last 1% takes the other 99% of the effort.
OK, that's a valid position to take, and I'm sure some people agree, but I and many other people do want computers to do what our brains do. And it is pretty likely to happen.
> I would be happier passing away in an aircraft accident knowing it was [not AI]
If I was in an aircraft accident caused by human pilot error while pilot AIs with 1/100 the error rate of humans were blocked from deployment because they don't meet your "zero errors" requirement, I would not be happy.