To my knowledge, some angles have been explored. Finding which inputs maximally stimulate indiv neurons, and tracing that up the chain (there was a fantastic article that did this on ImageNet). Then ablations, as you mentioned. And finally, rerunning the same training data on differently sized + shaped architectures.
AI systems are missing at least one level of complexity. All neurons in a NN layer fire simultaneously, unlike brain neurons, which trigger each other async.
Afaik the properties you’re referring to on AlexNet (the network trained to perform on Image Net) have to do with the nature of repeated convolution operations which, while interesting, is not as deeply insightful as I believe the article is aiming for.
FTFY. Philosophy aside, humans understand complex things by building simpler abstractions for them. I think the key to NN understandability centers on building simpler NNs focused on specific tasks that build into bigger ones.
> The explanations have to be wrong, because if their explanations were always right, you could just replace the black box with the explanations.
I've never thought of it like that, but I think that really gets to the core of the issue...
To be able to reason about the behaviour of a neural network with 100% confidence you would need an explanation that incorporated every weight, otherwise you have to accept some degree of uncertainty.
But the idea you could ever explain something with billions of parameters in a way that a human could comprehend seems ridiculous. Imperfect generalisation therefore would inevitably be required – "this bit of the network seems to perform x function". But in doing this you have to accept that this generalisation comes with uncertainty because such a high-level approximation is unlikely to capture all of the nuances. And if it did capture all nuances then the black box isn't needed.
Neural networks aren’t like that. The whole thing works together simultaneously to derive a result. So I’m skeptical that there is a high level explanation other than “it’s doing it’s best to approximate the data it was trained on” which is not that useful.
Suppose you have the spinning color wheel on your MacBook. You give it to a computer scientist who says that the program is in an infinite loop, and the wheel will never stop spinning. Then you give the computer to a superhuman physicist who studies all of the components including the state of the transistors and internal components and concludes that the wheel will spin for 9 years before the screen gives out. [1]
The point is that both are correct, but we’d really prefer the first explanation, since it gives insight into what the program is doing.
The problem is that neural networks aren’t like programs. They don’t have reducible internal components unless you reduce down to the weights of the network itself. [2]
Or, at more length and in more depth, The Analytical Language of John Wilkins by Borges.
> Lewis Carroll, in Sylvie and Bruno Concluded (1893), made the point humorously with his description of a fictional map that had "the scale of a mile to the mile". A character notes some practical difficulties with such a map and states that "we now use the country itself, as its own map, and I assure you it does nearly as well."
But “this portion of the network appears to recognize faces” is still a useful abstraction. In the same way it is for neurology. Even if both NNs and brains sometimes see faces that aren’t there because the network isn’t “recognizing faces” but a system which signals on a complex set of heuristics that mostly detects faces.
How does the answer change if I only want 99% confidence and accept the existence of illusions? — can we describe the bulk of the behavior in a summary?
https://www.alignmentforum.org/posts/j84JhErNezMxyK4dH/llm-m...
However, to think we cannot comprehend them at all is too much. We can certainly ”query” a NN to understand the weights and activations that led to some decision in some specific scenario. How to consume and use that information is another matter, of course, but a debugging process is certainly feasible (as it is with any large scale computer system).
Plus, the architectures of neural networks are not as complicated as most people think. Sure it gets weird after training, but the basic architecture of components are not nearly as complicated as the hardware of a GPU for example, or even that of a complex OS kernel.
reminds me of: "The Tao that can be told is not the eternal Tao."
The real problem IMO is our ability to represent natural concepts as such graphs. Is love, for example, a DFA? Can we search for its isomorphism?
This is very insightful. I've long maintained that "understanding" requires some predictive ability in addition to an ability to explain what's happened in the past.
In the case of neural networks, we should be able to "simply" generate the network that has the desired behavior.
But we can't. We're still shackled to compiling some collection of training data that may or may not produce the desired behavior, tested by some other collection of testing data that may or may not test the desired phenomenon.
Also, remember that the explanation doesn't really have to fully contain the semantics of the model. It doesn't have to fully encode the weights, just activate sufficiently sympathetic structures in a human brain.
At a very low level, there’s no secret: all of the weights are there, all of the relationships between weights are known. The problem is that it doesn’t tell you anything about the emergent properties of the network, in the same way that quantum physics doesn’t give much insight into biology.
It may be possible that there is no English sentence you can utter about the network which is both explanatory and fully accurate. What is the network doing? It’s trying to approximate the function you’ve given it. That’s it.
You can try other things like ablation to find the effects of lobotomizing the network in certain ways, but this also can’t fully explain 2nd and higher order relationships.