I suspect the first ANN researchers called them neurons to get more funding.

In essence an ANN is just a complicated mathematical function that you can tweak to fit the training set. Originally the activation function of a neuron in an ANN resembled real neurons, but later it was found that other, very different, functions are much better, like RLE.

I agree- neural networks were called like that to be sexy. It's certainly what attracted me to them in the 1980s (because I was intersted in both computers and biology). I got in a lot of arguments in the early 2000s before I understood just how fundamentally different neural networks are from brains.

except that they were first called perceptrons.

the neuron analogy has been there all along, since ANNs are entirely analogous to "BNNs" - for instance the NIPS conference.

the firing-rate-vs-activation-level difference isn't that big a deal.

Not so much futile (well, perhaps it is considering there’s much we still don’t know about biological neurons) as slow - simulating all that in software won’t be fast.

I asked ChatGPT to estimate memory requirements of a brain simulation, with neurons from a paper linked in OP post ("Single cortical neurons as deep artificial neural networks"). This paper uses neuron models that are supposed to simulate real ones. The answer was 8-50 PB, with 10^7 weights shared between neurons, or 3.4 EB if each neuron has its own 10^7 weights. So it's not out of the realm of possibility.

I wonder where the 10^7 came from: standard numbers are 86B neurons and 10k synapses/neuron.

On the other hand it shows how utterly tiny and primitive state-of-the-art LLMs are compared to real-world brains. By the way our brain only runs on on a max clock speed of around 100-1000 Hz.

Yet LLMs can achive miraculous results with so little. It really begs the question, where is the limit? How smart would a artificial human brain be if it would clock in gigahertz and had the somehow orders of magnitudes more efficient reasoning and detail recall of LLMs?

That's why I think lower speed, high density 3D compute is a good candidate for future hardware.

By lowering the clock speed, power and heat drop exponentially.

I'm just hoping we figure out something logistically better than laminating chips together.

I completely agree. I'm betting on physical learning systems via reservoir computing personally. Will be going back to school to study nanotechnology soon to pursue this formally in addition to some experiments Ill be funding and running on my own dime as well. Have you ever seen steel ball bearings submerged in castor oil? If you apply an electrical current to that what do you think will happen? The most fascinating thing occurs, the ball bearings self organize into a network that can even display properties of hebbian learning if set up correctly! I suspect experiments such as these will eventually even raise the bigger question of, what is life actually? Another way to think about the potential of this area of technology is to think of a biological synapse, its about 1 micron in size right? Well, how many "neural atoms" can we fit in that space? 1000x1000x1000!! Its marvelous how much space there is when you shrink things down. Ill even go so far as to speculate that to create genuine superintelligence, youll need this technology Ill be working on to mature and form into some.new form of stochastic processor/computer

The greater the clock speed, the greater the energy consumption, but I guess compute would scale linearly with frequency.

However LLMs are not really tiny compared to real brains, brains FLOP/s have been estimated to be around 1e15 to 1e21, while a single H100 SXM can already do 1e15 BF16 FLOP/s.

The brain also has about 1e15 synapses, each holding the equivalent of about 4.7 bits of information, which would be about 590 TB worth of parameters, equivalent to 294T BF16 parameters (would need 7.4k H100s to hold those, at 80GB per H100).

So in my opinion, it's more of a memory bandwidth problem than a computing power problem, as H100s have comparatively very little memory bandwidth per FLOP than the brain, which holds information directly inside the computing units (synapses).

And I also think that detail recall is a double-edged sword. Memorizing details can certainly be useful, but it can also compete with pattern generalization. If you can perfectly remember the training set (overfit) you don't need to learn to generalize.

AI "neurons" (ie. combination of weight layer and following activation layer) are only a very crude model of the real thing, but they do model what appears to be the core function of integrating the synaptic inputs and firing in non-linear fashion when this integrated input crosses some threshold. So, it's more than just connectionism, but a lot less than attempting to closely mimic biology.

Nobody knows how much would be gained by using a more biologically accurate model of a neuron. Much of the low level detail may be irrelevant, but the timing aspect of it, or some equivalent, might be helpful if we want to make ANNs capable of learning realtime pattern recognition and control.

> they do model what appears to be the core function of integrating the synaptic inputs and firing in non-linear fashion when this integrated input crosses some threshold

> what appears to be

> Nobody knows how much would be gained by using a more biologically accurate model of a neuron.

Philosophical:

The neuron is an integral part of it's environment... so using models of much more primitive biological neurons ( less "processing" distance between env and self ) might yield something, I believe, but not evolved ones, where they are part of tens of thousands of systems in systems with only a few of them layered like onions. It's a completely different definition of distance and the neural nets are not part of anything.

> Much of the low level detail may be irrelevant, but the timing aspect of it

Tens of thousands of asynchronous contiguous and non-linear events firing at just the right timing in 10s of thousands different places at timings that create something even crazier than perfect synchrony ... That lower level detail is likely responsible for the timing ... it's a bit like why osmotic pressure inside and outside is important and implicit to all the ratios relevant in the different spots of the body/organ/tissue ....

I'm unimpressed by AI and genAI to this day. But I think it's cool. ... it can all be achieved without calling it "AI" at all. Just a bunch of nodes and well sorted and tagged input ... it's social proof laundered awe mostly ... no chance against that marketing machine, so LLM = AI. Every time you "think" it can be more than a dictionary and wikipedia I realize how stupid I am for thinking that briefly when I read the reply to my prompt. Same goes for code. It's holy shit, wow, ah wait, there's that open source code and that free frontend design bit and all the other things, snippets, assets I had already bookmarked somewhere or sometime in the past ...

It's so crude that it might not even be the way neurons work. Like just a programmed activation function? Who programs it? We inherit the weights some how? Are they compressed in DNA? Doesn't seem likely because we share 90% with fruit flies etc. So not much space is left to store weights. I don't think it's the weights doing the work in biological neurons, but the architecture. Every cell might be doing the same thing but it's how you use them that counts. Maybe a biologist can comment on this.