Feature visualizations are described in this article: https://distill.pub/2017/feature-visualization/

The ones you see in this work are mostly a variant of the standard feature visualization, which tries to show different "facets" in neurons that respond to multiple things. The details are explained in the appendix of the paper (https://distill.pub/2021/multimodal-neurons/ ).

Worth noting that Chris Olah (who wrote this comment) has led much of the interesting work in making feature visualisations useful. If you look for "Visualizing Neural Networks" on his page https://colah.github.io/ you'll find lots of other interesting links in this area.

Thank you!

Has there been any study of variability in these activation images - like are there many disconnected local maxima depending on the initialization, or how they vary with retraining the network (or e.g. with dropout, etc), or varying the model parameters in some direction that keeps the loss in a local minimum.

I could picture that maybe they always look the same, but sometimes there would be cases where they have different modes that accomplish the same thing.

Quick unrelated question as you seem to be a subject expert. Are there currently any neural network models that deal with multiple separate networks that occasionally trade or share nodes? It might be useful for modeling a system where members of an organization may leave and join another organization such as a business or a church

Start with random input, then incrementally optimize the input to maximize the activation of one of the nodes in the graph, the neuron. The visualization is one of those inputs that hit a maximum.

During the initial research into multi-layer neural networks, it appeared that only the input and output layers had any human-comprehendible meaning; anything else would be an indecipherable vector of how much weight each item on the previous layer should have, e.g. 0.1*pixel[0] + 0.004*pixel[2] + … 0.4*pixel[1024]

In more recent research, people noticed that sensible labels can be given by seeing what sort of inputs activated it. For example, if that neuron lights up with any picture with a lot of right-angles, and nothing else, a researcher may call it “the right-angle neuron”.

In this case, the researchers did this to the CLIP network and found it was much easier than usual for humans to come up with sensible names for the type of thing any given neuron would recognise.

You can also demonstrate what a neuron does by running the maths in reverse to get an input which would activate that neuron more strongly than any other input; in this case an image.

> In more recent research, people noticed that sensible labels can be given by seeing what sort of inputs activated it. For example, if that neuron lights up with any picture with a lot of right-angles, and nothing else, a researcher may call it “the right-angle neuron”.

Isn’t this also how neuroscientists identified brain regions?

I guess it makes sense this would happen but it’s curious that we’ve been able to build a blackbox algorithm/technique that’s as difficult to reverse engineer as real brains.