It is not. There are many biological examples of random processes generating patterns and weights, such tiger stripes, fingerprints, etc. DNA only encodes the reaction, but the pattern itself is seeded by initial position of random molecules, not the DNA.
> that it influences the initial state of the brain.
You've gone from "DNA is probably the seed for the random initialization", as in the deterministic seed for a PRNG, to "influences". Those are 2 different claims.
Most scientists will agree that DNA encodes the large-scale connectivity in the brain, ie from region to region, thickness of corpus callosum etc.
Most scientists will disagree that it is "probable" that DNA encodes/seeds the weights of each and every synapse.
My own reading of the poster's intent aligns with this: "DNA encodes the large-scale connectivity in the brain, ie from region to region, thickness of corpus callosum etc."
Most biologists will fully disagree with the whole premise, and support the reverse claim that we are born with random weights, we learn to walk/speak English/etc after we're born. Not from weights in our DNA.
At least Penrose had the good manners to make a relevant contributions (meaning Chomsky effectively managed to steer his field in entirely the wrong direction and nobody uses his contributions today in study)
Something that more or less guarantees you’ll get about the same capabilities as the original model, but with a different final expression of certain characteristics
I wonder if this could be a good distribution model too, selling “dried” models (almost like raisins - or the beings from the 3 body problem), and then you can buy them to fully train (raise?) with your own data
A random seed that works would be the exact opposite of that.
Not sure exactly what this means
But there’s prior art on the concept of compressing models through lossy representations
In particular, procedural generation of models, check out HyperNEAT (2009), HyperNetworks (2017) and Developmental Symbolic Encoding (2011)
It means that neural networks are a mistake, that they don't really have to be as complex as they are and animal minds could just calculate directly from the seed as opposed to building and powering this very expensive organ. The brain uses more power than any muscle in essentially all mammals and even more in humans. Only olympic-level efforts come anywhere close to power usage like the brain on a single muscle. So your body is really giving up something serious to let you think, to keep track of 100 Trillion weights and constantly evaluate against them.
This would be something very strange for DNA/evolution to do. Especially where it comes to energy use, if it can be done with 10W of power, you can bet your firstborn DNA will NEVER under any circumstances get caught using 10.01W.
I like your idea of "base" models for sale.
Or that infants with a rooting response survive much more often.
Basically, weights encoded biologically from hereditary behavior.
Instincts.
Now, the real conundrum is that if our DNA can really only encode ~12gb then how are we passing this information down?
FractalDB proves that "synthetic data useful for training AI" can be produced by a very small program.
There is "no room" in DNA for a full dataset, or a full snapshot of model weights. But there's plenty of room for "this one little trick cuts the data required to learn a desirable thing X in half". Low Kolmogorov complexity, but high performance gains.
Also as aspects of nature... Its not just genes for a system that can feel pleasure or pain but also a mapping from that system to a set of textures, temperatures, and frequencies that take that system beyond certain thresholds. Plus a large set of environmental inputs that cause the system to grow in different ways in response to different stimuli.
A lot of people seem to think that training is limited to the 10 bits/s of processing mentioned here. But raw sensor input (mostly visual) is in the order of 21 quadrillion bits per year, according to the same paper. It seems likely to me that learning uses much more data than is ultimately expressed in behaviour.
[1] https://www.sciencedirect.com/science/article/pii/S089662732...
Even the shallow encoding patterns of AV1 can easily crush a raw video stream down by a factor of 10^3 without losing either spatial coherence or semantic loading.
Humans live in self-similar environments and perform self-similar activities. Any practical training dataset has the "natural human data" beat on diversity by a crushing amount. And a system that can learn deeper patterns makes the sensor streams more redundant, not less.
Incorporating non-text data into AI training runs has repeatedly demonstrated that for general reasoning capabilities, no data type beats text in value per byte.
LLMs used to "fail at spatial reasoning forever". But each new generation would fail a little less. Then we got the likes of Astra, and ARC-AGI-3 straight up died.
and then there's the abstraction of 'activation' used in neuralnets that is very useful computationally, but omits a lot of the complexity in actual neural signals. you don't get fine-detail in the timing of spikes or the fact that neurons use different types of neurotransmitters.
On the other hand, even such basics may well make a huge difference.
Pretty much every organism with a brain has a sophisticated survival reaction to physical contact with any part of its body, because its brain is trained on nerve signals. Everything with eyes or sonar learns spacial awareness of objects and their proximity, velocity, directoion of motion. I think the clue is that biological brains are trained on a physical world with very complex rules and physics as well as reactions that arise from chemistry.
Words and images, audio and video, even virtual worlds with soft-body physics, are just flat projected shadows of the complexity of the actual world.
While true, so too are our senses: https://en.wikipedia.org/wiki/Aristotelian_physics
I wondered wtf was going on so I looked around and realized that some sort of bird of prey was circling overhead that triggered that reaction and behaviour. It got me thinking. Somewhere inside of her brain and therefore her genome was the equivalent of a pixelated black and white image of the outline of a bird of prey, and the algorithm that can generate that zig-zag pattern in muscle firings and a whole overall heuristic that can apply these patterns and others into a coherent strategy that manifested as the fight or flight behaviour that I saw.
No one taught her this as she never really spent any time with her parents or her litter as she was adopted quite young, and spent most of her time doing house cat things so it wasn't something that she picked up from her environment, so it seemed pretty innate to me.
It was very influential to me as a teen to ponder that if that bird of prey's genome encodes the full shape and structure and behaviour of it then in some fractal sense there's a low fidelity copy of the shape of that bird and also implicity the bird's behaviour through a counter prey behaviour encoded in the cat's genome.
I'm also reminded of that car AI using a neural net with a single shared weight, where the topology of the neural graph was what made the AI work.
How stuff is connected is what our DNA is about, so seems obvious the topology must encode behavior and such and it's not all learned.
Take something we have zero pretraining for, like chess. It takes an AI billions of games to get to where a human is after 300.
I don’t think that’s a fair comparison since (as far as I know) humans aren’t born with randomly initialised neural weights that are setup in a very generic structure, but have at the very least extremely strong inductive biases. I think it’s fair to count the hundreds of millions of years of evolution between the first animals with a primitive neural system and humans today, and the amount of “tokens” that were “trained on” is likely enormous. I think it’s better to consider the 200M tokens as a sort of “fine tuning” to a sort of genetic “pre-training”.