Interesting.

So in a high profile case, like the Tesla Autopilot fatality, if the behavior was traced back to a specific micro-controller, it might be just as hard to determine why the system behaved as it did, as with a complex neural network? Even with a company's reputation at stake, it's not possible to hire a consultant with a deep understanding of obscure micro controllers?

I only deal with the source code level, don't really have experience digging into the hardware level.

Just look at the Toyota acceleration lawsuit, where a consultant spent months trying to figure the system out and still ended up with a probabilistic explanation (basically, the software uses a bunch of bad practices that could result in bad effects that could include the alleged accelerator problem).

http://www.edn.com/design/automotive/4423428/Toyota-s-killer...

We rely on many layers of more or less leaky abstractions.

Components have always a set of intended behavior and some, often unknown or poorly understood, set of unintended behavior.

It's very important to understand that abstractions mask both intended and unintended behavior.

You deal with unintended side the best you can, with sanity checks and so on. But some of it inevitably leaks through.

Those outcomes that result from multiple systems being individually in good state but collectively causing undesired behavior are often very tricky to understand. On the flip side, when you do understand them, you tend to gain new surprising insights into the whole system.

Also timing makes understanding hard. When it comes to a complex network of systems, it's hard to concurrently reproduce the state where the effect occurs.

> So in a high profile case, like the Tesla Autopilot fatality, if the behavior was traced back to a specific micro-controller, it might be just as hard to determine why the system behaved as it did, as with a complex neural network?

If Tesla has good, exhaustive, event logging facilities, a "black box", I guess they'll be able to figure out what happened by analyzing all of the available data with the systems knowledge they have.

Anyways, if it's not systematic, why should Tesla autopilot mistake be handled so differently from a human being making a mistake?

You can't analyze why a human being caused an accident either. The best you can usually do is to develop scenarios "theories" based on the available evidence, but that's it.

I don't think analyzing why a human caused an accident is important either. However, being able to fix the bug is very important. If a programmer can identify the problem in their code, they can fix it. But how can one identify the problem in a neural network and fix it? At least right now, understanding an (actually useful) neural network might as well be impossible

I think it's fair to say that we don't have systems anymore that could be 100% understood by a single person. Your smartphone for example, I think it could be 100% understood by a crowd of engineers just big enough to fill a modest concert hall.

Way, way back, I had an 8 bit computer that I built myself from chips and solder. The "permanent" memory was 16kB; I knew most of that code pretty well. Even then the CPU itself required quite a bit of study to comprehend.

On the flip side, it may well turn out that systems complex enough to be 100% understood by a single person would be dumb and limited.

There are pretty significant limits to our conscious, logical understanding of the world.