That would make each neuron a complicated computer, functionally equivalent to one pc in capability and data storage.
"The neuronal gene Arc is essential for long-lasting information storage in the mammalian brain, mediates various forms of synaptic plasticity, and has been implicated in neurodevelopmental disorders. However, little is known about Arc’s molecular function and evolutionary origins.
Here, we show that Arc self-assembles into virus-like capsids that encapsulate RNA. Endogenous Arc protein is released from neurons in extracellular vesicles that mediate the transfer of Arc mRNA into new target cells, where it can undergo activity-dependent translation. Purified Arc capsids are endocytosed and are able to transfer Arc mRNA into the cytoplasm of neurons.
These results show that Arc exhibits similar molecular properties to retroviral Gag proteins. Evolutionary analysis indicates that Arc is derived from a vertebrate lineage of Ty3/gypsy retrotransposons, which are also ancestors to retroviruses. These findings suggest that Gag retroelements have been repurposed during evolution to mediate intercellular communication in the nervous system."
What this mechanism does is take the transcription level of a gene in one cell, and induce a proportional signal in another cell. There are numerous mechanisms which do that in cells. Most of them are nowhere near as weird as Arc, but the net result is similar.
In a single cell I wouldn't call VDJ recombination memory, it's just an efficient way of encoding many different possibilities for types of receptors (and then selecting one). There exist many cells in your body that match a potential antigen you've never seen before - it's just that there are an extremely small number of them. The memory is really encoded by increasing the population of that cell in the body. Which has different storage properties than the individual cell's DNA does. Epigenetic change to a single neuron is unlikely to have a functional effect either, but it is a change that can occur to a mature cell as a reaction to the environment, in a way that VDJ recombination is not. AFAIK there is a lot left to be understood about how epigenetic modifications affect the brain.
I agree this is all interesting and can have implications for computational models, but there are models at many levels of abstraction. So I don't think it is necessary to get at the biological details in order to discuss higher level computational implications. Questions about the capabilities of the memory for each individual neuron would arise, but they could be theorized about without much knowledge of biological details. Of course it depends what your goals are whether you would consider that useful.
Anyway, I think we're both in agreement the OG comment was wrong, I just feel that "in DNA" is very ambiguous, and also not really the problem with his comment given the purpose of HN.
>That would make each neuron a complicated computer
Is this not an accepted fact? Each neuron is like a tiny processor in a larger distributed like system. This is why it's impossible to build bioaccurate NNs because each neuron has ~10K connections to various other neurons.
There are hard limits. No matter how you try you can't perfectly simulate three atoms using two atoms. If it turns out we have to, in software, represent fifty percent of neuronal activity to create consciousness we're in real trouble. A dragonfly can take inputs from thousands of ommatidia and use them to track targets in space using only sixteen neurons. How many transistors would it take us to do the same? Take that ratio and apply it to the 86 billion neurons in the human brain and you have a rough idea of what it will take to create strong AI. The numbers aren't promising.
I think that's largely known, depending of course on how you define "cognition".
Huge tracts (I don't have numbers) of the cortex are dedicated to things like vision, motor control, etc... Those aren't "cognition" as generally understood, and there are many stroke victims out there who can testify (like, actually "testify", in the sense of using their brain to explain it to you) to the fact that they can no longer see, or move their left side, etc... Their "cognition" is not impaired.
It gets fuzzier with things like speech and recognition, which also have dedicated real estate but are, kinda, "para-cognition" tasks.
Really, yes: you can have a "thinking" engine with a tiny fraction of the computation power of the human brain. I think most folks agree with that. The broader question is that with so limited an I/O structure: what is there for it to think about?
I thought this at least was fairly well understood: We do in fact use our whole brains, as anything less would be a fantastic waste of resources, which evolution would have taken care of long ago. We have numerous human-specific adaptations to deal with the relatively massive brains we're carrying around.
https://en.wikipedia.org/wiki/Brain-to-body_mass_ratio
"In animals, it is thought that the larger the brain, the more brain weight will be available for more complex cognitive tasks. However, large animals need more neurons to represent their own bodies and control specific muscles;[clarification needed][citation needed] thus, relative rather than absolute brain size makes for a ranking of animals that better coincides with the observed complexity of animal behaviour. The relationship between brain-to-body mass ratio and complexity of behaviour is not perfect as other factors also influence intelligence, like the evolution of the recent cerebral cortex and different degrees of brain folding,[5] which increase the surface of the cortex, which is positively correlated in humans to intelligence."
Amazing. Do you have a reference for that? I'd like to read about the findings as well as how they determined that!
The instantaneous efficiency is much lower, sure, but the lifetime efficiency is another question. The resource cost to create a plant is the nutrient/energy cost of producing and dispersing a seed. The energy cost of producing and installing a solar panel is enormous by comparison, and takes years if not decades to capture more resources than it took to produce.
Home solar panels capture 11-15% of incoming energy. Plant leaves capture 3-6%. Do you have solar panels now? If you could throw five dollars in seeds over your roof and get half that amount of electricity, would you?
So I tend to err on the side of biology being more complex than not.
But it's absolutely possible that the high-level behaviour is simple while the underlying implementation is complex and chaotic, as in the gas laws.
In principle one could imagine a processor design that works on these intermediate states that somehow vastly exceeds the computational power of a modern system despite using the same base transistors; in practice we have no idea how to build such a thing, and if we did, we wouldn't know how to build a second one of the same thing reliably either.
Biology lacks this restriction. That doesn't mean it's pure and utter chaos, either, there's bounds on that because it still needs systems to at least be metastable. But where humans engineer almost exclusively with stable systems, biology freely uses metastable systems all over the place. And then, even more remarkably, it deals with the question of how to replicate such a strange system in a way that no modern human engineer ever would by making every instance unique, and still somehow functional.
It's a tough act to follow.
Often the most efficient solution to a problem ends up being a simple one. It would be very surprising if the human brain's Rube Goldberg machine was anywhere near close to a mathematically optimal implementation of intelligence.
It's possible that much of the complexity in a biological neuron is simply working around other complexity introduced by biology, solving problems that we programmers do not even have to think about because we can simply directly use matrix multiplication.
Humans can attempt to multiply numbers and frequently get the wrong result. That doesn't scream "pushing the limits of intelligence".
Maybe there will be transistor-based human-level AGI soon, but I guess it would require several kilowatts of power compared to the 20 or so watts a human brain requires.
Given that entire datacenters don't come close to an AGI, let alone a human level one, I fear "several kilowatts of power" is lowballing it by a significant number of orders of magnitude.
More to the point, we don't even know how or what gives rise to a general intelligence, and even defining it is basically a philosophical question. To me, the optimism of some AI enthusiasts (and I don't mean specifically the parent) feels like cavemen contemplating an expedition to the stars shortly after they invented the sling.
I think the chances of us arriving to anything close to an AGI iteratively based on our current capabilities is a pipe dream.
There are already several computers more powerful than this that don’t appear close to a working full brain emulation, and we definitely don’t understand intelligence well enough yet to engineer something like ourselves, so it’s reasonable to be skeptical of estimates saying we’ll be at the kilowatts level “soon” even if it turns out we’re just missing a step which will be obvious in hindsight.
(Unless by “soon” you mean 15 years; I don’t want to bet on anything on that timescale).
Well, I'm old so by "soon" I mean within my lifetime, e.g. the next 3 to 4 decades.
edit: to clarify a little, when the term AI was first coined by McCarthy in 1956, researchers were confident in cracking AGI within a decade. Then AI-Winter came and people became more cautious. So when I say "soon", I mean it's probably not going to be another 65 years, but also not 5 months or 5 years (unexpected breakthroughs aside).
"You think thermodynamics means using humans as batteries is a dumb idea? Where do you think you learned thermodynamics?"
faint nyan cat music in the background
(part of the quadrillion atoms system of course)
Assuming AGI requires about as much hardware as it'd take to emulate a brain, at least.
That's a very very weak upper bound on how much hardware it takes. I think it's not all that different from emulating a Nintendo64 with a quantum simulation of the hardware.
For complex systems to work (not to mention evolve), they need to be robust to small perturbations -- there's no way the computation the brain is doing is sensitive to the details of particular atoms. There has to be redundancy, modularity, etc. These things aren't human inventions so much as they are the only way to meaningfully move in a 2^|giant-number| state-space.
You could argue despite the huge number of physical degrees of freedom, the operations on DNA can be reduced to copy, repair, express, suppress. On the other hand, there's still a ton of intrinsic complexity in storing a huge amount of data, and yeah some nucleotides are totally essential.
The other thing a wonder about: sure, maintaining a proteome is hugely complicated, but how much of this complexity goes into maintaining homeostasis (e.g. metabolism, cytoskeloton and membrane maintenance, replication,...) vs. enabling computation. Seems like silicon has the advantage here.
A grain of sand is around 6.02E19 Daltons, imagine how much computational power there is in a sand castle!
On the other hand, snails have been shown to use just two neurons to generate their complex (relatively, I mean they're snails after all) feeding behaviour with just two neurons: https://neurosciencenews.com/neurons-decision-making-4370/
(Veering slightly offtopic here but this is the first I've heard the term 'Dalton', I've always just heard them called AMUs.)
We can see this in practice by looking at e.g. a self-driving Tesla vs. a mouse.
Watch a frightened mouse run across an uneven landscape, climbing and clambering and avoiding obstacles as it controls four independent limbs and countless small muscles in real time.
A Tesla's auto-drive is nowhere near that good in spite of having only a few levers to control: accelerate, brake, left, and right. It also has far better sensors than the mouse including better eyesight, a wider field of vision, etc., and last I checked mice do not have access to a cellular network supplying them with a heads-up macro view of the local environment. They don't have "fleet-wide learning" either.
The Tesla's AI uses over a hundred watts of power. The mouse's brain uses milliwatts.
As I wrote this my brain was consuming between 30 and 60 watts. My laptop peaks out at 80.
We are not even close to what biological neurons accomplish in raw compute, and while we are getting pretty good at training giant regression models that we call AI I am not convinced we really understand things at the algorithmic level yet either.
Relevant read: "The thermodynamic efficiency of computations made in cells across the range of life" https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5686401/
Threshold neurons like Pitts-McCulloch neurons are rarely used in artificial neural network architectures. They can't learn.
ANN's uses a nonlinear activation function that is differentiable (at least in practice). That small change makes huge difference.
The strongest statement that might fit here is that they can't learn efficiently. Zero gradients just make learning slower (though I do agree that differentiability is something to strive for).
As a bit of an aside, in practice stochastic versions of algorithms assuming differentiability work on wide ranges of functions, and compositions of poorly behaved functions can be quite nicely.
For a couple [0] concrete examples:
(1) Throw the absolute value function into your favorite gradient/Newton's minimization routine. Blindly using differentiable techniques often works if a sub-gradient technique would work.
(2) Consider minimizing the magnitude of the smallest eigenvalue of the Jacobian matrix of your favorite function. Many of the intermediate components (e.g., trying to derive the eigenvalue with respect to matrix entries) are poorly defined, undefined, or have cusps and other nasty features. The composition is (under mild constraints) differentiable with non-zero gradients.
(3) Consider minimizing the absolute value of a step function. By using a wide difference quotient as an approximation of the derivative and feeding that into optimizers you'll still find the minimum near zero (See (1); it works similarly).
If the composite output is only constant on small regions in the input space (which holds if those neurons are modeling anything non-trivial), you can rig together something close enough to backprop to still learn efficiently.
Is there a good (i.e. theoretical) reason for this?
If I understand the PM neuron, it's outputs are boolean. This means there's no backprop signal, despite differentiability, since the outputs are constant (thus, gradient zero) in any neighborhood, which in turns zeros out any learning signal you would want to backprop through them. So you need a different learning strategy than backprop to use them.
(see also: the 'dead neuron' problem/phenomenon with ReLU activations.)
You may be implying there could be information stored in all those atoms, but I'm not sure that's possible. We live in a thermal bath, this means the behavior of atoms is usually stochastic. Their position cannot reliably hold information without dispersing it rapidly into the thermal environment. One way to get around this is to form chemical bonds, like in the DNA, where some kind of order or position is stable. But it necessitates this chemical structure, and importantly it necessitates as well a chemical reading mechanism and a chemical writing mechanism (or some kind of kinetic activation). Most parts of a cell are not prepared for any of that as far as I can tell. Information really should be carried by discrete elements such as neurotransmitters, as well as continuous but temporary (unstable) elements such as electric impulses (potentially caused by complex responses to electric potential and current inside the neuron). Even in the electric case, the fact about information stands; electric state is also encoded in atoms.
In other words, we almost certainly don't need the full fidelity to reproduce the behavior of a neuron.
I would need more rigorous examination of the neuron to give a confident estimate, but as a rule of thumb the concentration of relevant information everywhere is much less than DNA's (and mostly negligible everywhere) -- certainly a very interesting research program.
From a quick googling, E. Coli DNA has about 4.6 x 10^6 bp, so I would be reasonably confident in an upper bound to neuron information as (volume of neuron/volume of e. coli) x 5 x 10^6 bits (i.e. 1 megabit, 125 kb). The reality is probably much less. DNA is so dense because the reading and replication time is relatively slow. It's not made for rapid, random access at the speed of thought.
If I were to guess, I'd say long term information is probably retained within the concentration of compounds that can be read electro-chemically. The question of information then is how sensitive the neural system as a whole is to differences in concentration and differences in timing and amplitude of neural impulses. Again given thermal noise in the brain and limitations of amplitude, you can give strict upper bounds on neural communication (I'd be surprised at sensitivities more than a few ppm).
So essentially
1 neuron <= log2(distinguishable chemical state) (memory)
And also
1 spike <= log2(electrical degrees of freedom) (bandwidth)
Again, I'd require more information on spikes, but they carry maybe 20-40 bits at most -- so not more than a double f.p., although in ANNs again because of low sensitivities due to architecture most of LSB information doesn't contribute significantly to the computation (whereas the brain could multiplex information more effectively). So it's still likely in the order of 1 spike <= 10 flops.
I'm sure there's others.
A biological neuron is 6e17 Daltons[0], so on the order of quadrillions of atoms. A single synapse is a huge landscape studded with receptors of various kinds, and the whole thing is swimming in salt solution where chemicals diffuse stochastically. And then there's the glia.
This is why I dismiss any claims about the computational power of the human brain, most of which seem to begin with the assumption that "1 spike = 1 FLOP", and that'll all that goes on in the brain.
[0]: https://faculty.washington.edu/chudler/facts.html and https://www.wolframalpha.com/input/?i=1E-06+grams+in+dalton