Following that trail of thought, intelligence is an achievement and not a physicality.
Humans are simple in this model (just like Carmack asserts) because they aren't properly intelligent, sapient, or conscious 100% of the time.
Just because our DNA can be efficiently encoded doesn't mean that our brain is a tiny proportion of that encoding. Your DNA doesn't change much from when you're born to when you die (random degredation aside) and yet your cognative abilities change beyond all recognition. Why is that? Well maybe there's more to what's in the brain than just what's encoded in your DNA.
Secondly, how does he get to the 40Mb number? I don't think we know anywhere near enough to know how much information it would take to encode a brain, but 40Mb seems just made up. For starters, consider the amount of random stuff you can remember from your entire life. Are you saying that all can be encoded in just 40Mb? Seems very unlikely.
One of the things that everyone is sort of skipping over is the "sufficient training" part. There is no bootstrap reinforcement learning possible for AGI. You can't alphago this sucker and have it play simulations against itself because the whole idea of generality is that there isn't a simple rule framework in which you could run such a simulation. So training any kind of AGI is a really hard problem.
I would speculate that there are more ways to train on logical consistency of the output, and improve the models further.
He admits that the equivalent of years of "training" would still be needed to take an toddler-level consciousness to something approaching an adult human.
For me his statement is one of those things that is sort of not even wrong, like when people say humans only have two eyes and can drive a car therefore an automated driving system must be possible with only cameras. On the face of it, this seems like it could be true but of course it handwaves away the hardest and most mysterious part of intelligence and focusses instead on the easy bit that we already know reasonably well.
It's exactly like if someone said they had a secret formula to be an NBA superstar which is:
1. Be really tall
2. Be really agile
3. Be really good at basketball
Like yeah of course but all the hard parts are left out.
Is that not what humanity has developed from centuries/millennia of experience with all the approaches to child raising and education?
Engineering something trainable is clearly the difficult part given the entire conversation is about whether or not it's even possible.
We know a LOT about how neurons organize. Not even close to everything we 'should' know, but we do know a lot.
Most of this is in the development of the brain. How you get from one cell to the trillions that make up a person.
The real quick and dirty explanation is that cells follow multiple chemical gradients to find their 'home'. This is a HUGE topic though, and I'm being terse as I have a meeting to get to.
How adult cells organize also has a LOT of science behind it. Again, though, it's mostly about chemical gradients, with a bit of electrical stuff thrown in. Again, HUGE topic.
Kinda like asking if gravity and fluid dynamics are encoded in the blueprint for an aircraft.
The design relies on them, and exists in the form it does because of them.
To me, using a blueprint analogy you'd have to say the blueprint describes an airplane that once you construct enough of them interact in such a way as to build their own airports, plan their own routes, fly themselves and produce their own online booking software and that's still nowhere near as complex as what's happening inside a nematode let alone a human.
But though it's not literally about locations in the sense of physical coordinates, the way cell signalling and the molecular feedback loops that drive development are still reliant on basic physical laws.
It would be completely redundant and unnecessary to encode those laws themselves since they're invariant across time and space. Physics and chemistry are fixed.
It would never make sense for DNA to literally encode information about physical laws in the same way it wouldn't make sense to do so on an airplane blueprint, because the design of the blueprint was itself constrained by those laws, as would any alternative design.
Medium answer: Kinda. The chemical gradients cause a signaling cascade that modifies transcription of DNA (it's really complicated). This transcription change then causes the cell to become a XXX_neuron. However, there are many many waves of this process occurring with a lot of cell death along the way. When those cells are not the 'final' cell of the nervous system, these transcriptions can cause further and more complicated chemical gradients to exist in the fetus. These complicating recursive loops can also self-affect cells and cause them to change yet again.
We're still discovering a lot here.
Also, this is largely how ALL cells in a body work, not just neurons. Be careful though, this is very very complicated stuff and everything I've written has a caveat.
To use ML terms -- Humans have "Foundation Models" which are composed of: - Their Biological makeup - The culture into which they are raised
My interpretation is that Carmack is essentially confusing a config file for the computer program itself, then saying "look how small it is, this shouldn't be that hard to reverse engineer".
He's wrong. There is currently no practical way to produce a software system that possesses the ability for human thought, reasoning, and motivation without that system possessing uniquely human (let alone organic) properties: the biological and chemical makeup, plus the physical characteristics, of a human, and the ability to process using human senses. (Hint: a neural net processing video images is a mere shadow of how a human processes things with their senses of sight, sound, and touch.)
Carmack thinks humans can be reliably reduced to spherical cows in a vacuum, but that only holds true on paper. A real human being is not merely a meat machine: we are driven largely by emotions and physical desires, none of which exist in a computer except through elaborate simulation thereof.
Now, I'm sure over the next couple of decades we will make huge strides in mimicking a human being's ability to learn, i.e. creating ever more complex LLMs and AI models that act increasingly more humanlike, but they will be nothing but more and more elaborate parlor tricks which, when prodded just the right way, will fail completely, revealing that they were never human at all. They will be like Avatars (from the movie): sophisticated simulacra that rely on elaborate behind-the-scenes support systems, without which they are useless.
I think that’s actually pretty doable. Take for example flying. We don’t build airplanes that flap their wings because we have a deeper understanding of flight that allows us to build flying machines far beyond the capabilities of any animal.
Likewise, once we understand the mechanics of intelligence we should be able to build something that can learn that is completely computer based.
"Intelligence" is not something easily abstracted away from the physical world, especially if you want something to learn on its own. How will an AGI to learn that stoves are hot?
The main challenge I see to creating such a system that can truly learn is that you will have to constrain it to have motivation to learn and follow your directions, and nothing else. And even if you could add such constraints, what would "learning" mean to such a device? What would stop it from going off on useless tangents, like attempting to count every grain of sand you show it? Anything with as much autonomy as it takes to have AGI will likely start coming to conclusions we don't want it to.
My guess is that in the near future, either we'll create something that is beyond our ability to control effectively, or it will be yet another clever simulation of AGI that is not really AGI.
Nice thought