> Dr. Thompson peered inside his perfect offspring to gain insight into its methods, but what he found inside was baffling. The plucky chip was utilizing only thirty-seven of its one hundred logic gates, and most of them were arranged in a curious collection of feedback loops. Five individual logic cells were functionally disconnected from the rest— with no pathways that would allow them to influence the output— yet when the researcher disabled any one of them the chip lost its ability to discriminate the tones.
(from https://www.damninteresting.com/on-the-origin-of-circuits/)
Granted, the results were probably due to the quirks of the FPGA gates on that particular chip (and that particular area of the chip) but without further (potentially destructive) investigation, Dr. Thompson (or us for that matter) may never know the actual reason that circuit worked the way it did.
If you don't understand how a thing works, how the hell do you plan to fix it when it doesn't?
There's also a fringe benefit here. If "we can't understand" what it's doing, then that assumption can appear to weaken the "auditability" argument for deploying FLOSS in life-critical AI systems. (I.e., freedom 1 from the GPL.)
I'll bet the argument will be put forward with more frequency over the next few years, especially as self-driving cars become more prominent. Perhaps "appear to weaken" is too strong an assertion. But I can imagine a consumer safety advocate arguing for code to be auditable by outside parties just like Signal can be, and a PR spokesperson deflecting that AI is "beyond our understanding" and therefore not subject to the same standard.
This kind of breathless "Oh we CAN'T understand!" editorializing gets clicks because machine learning is so strange to most people, but it's terrible. We should instead be spreading knowledge about the very simple principles that are underlying the complicated behavior.
Our brains are far, far more complicated than even our most advanced machine learning right now. Yet we are constantly making progress on understanding the workings of our brains. This article is just terrible.