Worth noting that Chris Olah (who wrote this comment) has led much of the interesting work in making feature visualisations useful. If you look for "Visualizing Neural Networks" on his page https://colah.github.io/ you'll find lots of other interesting links in this area.

Thank you!

Has there been any study of variability in these activation images - like are there many disconnected local maxima depending on the initialization, or how they vary with retraining the network (or e.g. with dropout, etc), or varying the model parameters in some direction that keeps the loss in a local minimum.

I could picture that maybe they always look the same, but sometimes there would be cases where they have different modes that accomplish the same thing.

Quick unrelated question as you seem to be a subject expert. Are there currently any neural network models that deal with multiple separate networks that occasionally trade or share nodes? It might be useful for modeling a system where members of an organization may leave and join another organization such as a business or a church