That said, after skimming the piece and thinking about the amount of work that went into it, one question popped up in my mind: do you think it would be possible to train a DNN to learn to visualize the "most important" neuron activations / interactions of another DNN?
> do you think it would be possible to train a DNN to learn to visualize the "most important" neuron activations / interactions of another DNN?
That sounds like a really hard problem. I'm not entirely sure what it would mean even, but it would not surprise me at all if there was some refinement that could turn into an interesting research direction! :)
In terms of what it could mean, one idea I just had is to take a trained model, randomly remove (e.g., zero out) neurons, and then train a second model to predict how well the trained model continues to work without those removed neurons. The goal would not be to 'thin out' the first model to reduce computation, but to train the second one learn to identify important neurons/interactions. Perhaps the second model can learn to predict which neurons/interactions are most important to the first model, as a stepping stone for further analysis?
Can you elaborate on what you mean by "directions in activation space" ? If I understand it right:
You take a few neurons in a layer, and you follow some linear combination of their weights; you are then walking along a random direction. You take a single neuron and walk along its weights; you walk along this neuron's direction. Is this correct?
Also another comment: Szegedy et al [9] (https://arxiv.org/pdf/1312.6199.pdf) has the following abstract:
> First, we find that there is no distinction between individual high level units and random linear combinations of high level units, according to various methods of unit analysis. It suggests that it is the space, rather than the individual units, that contains the semantic information in the high layers of neural networks.
This is one of the predictions of the following paper, even though I think it came later: Opening the Black Box of Deep Neural Networks via Information (https://arxiv.org/abs/1703.00810). Here: https://youtu.be/FSfN2K3tnJU?t=1h19m23s, Tishby says: "One of the consequences of this theory is that single neurons in a layer don't tell us much."
Also, If you extend this idea to training examples, you get Mixup (https://arxiv.org/pdf/1710.09412.pdf)
distill.pub is really a really amazing place to learn.
(a reason why I'm moving into scientific visualization and GIS instead of backend work...)