We seem to be lacking a theory of aerodynamics for neural networks. Most research seems to be judged by performance. The Aerodyanmic equivalent focusing on building a more powerful engine rather than wing design. The fact that we still visualize all the spurious w_ij weights in a network is a symptom of this problem. You wouldn’t show a fully connected circuit diagram for an 8 bit adder with spurious logical connections, but this is exactly what we do when we visualize NN topologies.

There’s some great work in Artificial Gene Networks that tries to tackle this problem. AGNs are mathematically the same as ANNs.

Here’s a paper I wrote in 2008 with about 170 citations that has a lot of relevant references: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2538912/