We seem to be lacking a theory of aerodynamics for neural networks. Most research seems to be judged by performance. The Aerodyanmic equivalent focusing on building a more powerful engine rather than wing design. The fact that we still visualize all the spurious w_ij weights in a network is a symptom of this problem. You wouldn’t show a fully connected circuit diagram for an 8 bit adder with spurious logical connections, but this is exactly what we do when we visualize NN topologies.

There’s some great work in Artificial Gene Networks that tries to tackle this problem. AGNs are mathematically the same as ANNs.

Here’s a paper I wrote in 2008 with about 170 citations that has a lot of relevant references: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2538912/

That's a fair point. And I don't mean to argue against continuing to try and understand how the brain works. But I don't think we should take "we don't know how the brain works" as a final argument against being able to build useful kind of AI.

Well, there's also observations from neuroscience. CNN's were directly inspired by observations of closely-grouped neurons firing when detecting certain images, and capsule networks are directly inspired by cortical columns.