You have to remember what came before 2012: SVMs, Random Forests etc, absolutely nothing like the brain (yes, NNs are old, but 2012 was the start of the deep learning revolution). With this frame of reference, the brain and neural networks are both a kind of Connectionism with similar properties, and I think it makes perfect sense to liken them with each other, draw inspiration from one and apply it to the other.

It is also odd to see such a weak argument as the brain-to-body mass ratio being used, as here: https://youtu.be/YD-9NG1Ke5Y?t=593

If this metric were truly indicative, what should we make of the remarkable ratios found in small birds (1:12), tree shrews (1:10), or even small ants (1:7)?

https://en.wikipedia.org/wiki/Brain%E2%80%93body_mass_ratio

> what should we make of the remarkable ratios found…

We also can’t implement those creatures’ control systems in silicon, so they too are doing things we can learn from?

Neural Networks are 200 years old (Legendre and Gauss defined Feed forward neural networks). Deep learning. The real difference between traditional ones and deep learning is a hierarchy of layers (hidden layers) which do different things to accomplish a goal. Even the concept of training is to provide weights on the neural network and there are many algorithms to do refinement, optimization and the network design.

Gauss did not define feed forward neural networks, it all stems from a tweet of a very confused person.

I mean, sure, you can model a simple linear regression fitted via Least Squares (pretty much what they did 200 years ago) with a one hidden layer feed-fwd Neural Network, but the theorical framework for NNs is quite different.

For Least Squares, you do not even use a hidden layer. Just a single dense layer from input directly to output. You also do not use an activation function (or use the identity activation function). That is, everything that makes neural networks special.

sorry, but i think neural-networks came way before 2012, notably the works of rumelhart, mccleland etc. see the 2 volume "parallel distributed processing" to read almost all about it.

the book(s): https://direct.mit.edu/books/monograph/4424/Parallel-Distrib...

a-talk: https://www.youtube.com/watch?v=yQbJNEhgYUw

I raise you Warren McCulloch in 1962: https://www.youtube.com/watch?v=wawMjJUCMVw

Jets and Sharks!

https://github.com/acmiceli/IACModel

I knew someone would bring it up, which is why I added "(yes, NNs are old, but 2012 was the start of the deep learning revolution)"

2012 was when the revolutionaries stormed the bastille and overthrew the old guard. But I say it was 2006 when the revolution started, when the manifesto was published: deep NNs can be trained end-to-end, learning their own features [1]. I think this is when "Deep Learning" became a term of art, and the paper has 24k citations. (Interestingly in a talk a Vector Hinton gave two weeks ago he said his paper on deep learning at NIPS 2006 was rejected because they already had one.)

[1] G. E. Hinton and R. R. Salakhutdinov, 2006, Science, Reducing the Dimensionality of Data with Neural Networks

What came before was regression. Which is to this day no 1 method if we want something interpretable, especially if we know which functions our variables follow. And self attention is very similar to correlation matrix. In a way neural networks are just bunch of regression models stacked on top of each other with some normalization and nonlinearity between them. It's cool however how closely it resembles biology.

Sorry, but they just called it "neuron" to sound nicer.

Is that wildly different from me calling a data structure where a parent node has child nodes a tree?

Depends — do you then start claiming that because your data structure is like a tree, it’s surely going to start bearing fruit and emitting oxygen?

Reading the replies to your comment, I think maybe the answer to your simple question is: "no". I also wonder if any "serious comparisons" have been made, and would be interested to read about it! A good question, I think.

You don't need to simulate every atom in a planet to predict its orbit. A mathematical neuron could have similar function to a real one even if it works completely differently.

what color are neurons? is that relevant? ml has proven that artificial networks can think. the other stuff may be necessary to do other things, or maybe simply evolved to support the requisite biological structures. ml is of course inspired by biology, but that does not mean we need to simulate everything.

> ml has proven that artificial networks can think

nope, at this point it's an ad-free version of google that summarizes results without linking to source websites.

Not excusing the lack of caveat in his talk but IMO, the old adage of: "All models are wrong, but some are useful." applies here.