I am not so sure. The current success of deep learning seems to be a direct counterexample to this. These techniques are yielding great success by imitating some patterns we observe in the brain, but the resulting inferences more or less defy rigorous explanation. We might fully understand the structure of a neural network but explaining the "why" behind its decisions seems to elude us.
> subjected to all kinds of experiences, which, in turn, will result in this brain becoming conscious - gradually.
Again, there is no evidence that "all kinds of experiences" will result in gradual emergence of consciousness from anything we have observed.
To a computer, experience is just data. So we're not talking about computers.
Phrases like "software brain" and "conscious machines" are ways of adding something that is not software to computers. But trust me, if we're talking about software, the programmer will know every line of their code.
Now, if you want to get meta, we can consider programs that write programs. But again, the first program will be written by a human, and we will know exactly what we are doing.
> resulting inferences more or less defy rigorous explanation
We have written programs that take input and produce data models. There is no consciousness here.
Humans take input and produce models too. But I fear you may be confusing learning with consciousness and agency. These not to be found in the code of learning machines, nor in the models they've learned. I am sorry, but it's just not there.
For every abstraction there is an implementation. That is physics. With no implementation, we have metaphysics. Here, "consciousness" is the abstraction. The physics of it is the physical implementation, and is what Roger Penrose and other scientists are working on identifying. The simulation will have a coded implementation. Meaning, whether we generate it or we build something that generates it, when we have "consciousness" we will have the code that backs it. To be able to do this without any understanding of what consciousness is would be analogous to monkeys typing Shakespeare.
Of course there is. The human "wetware" brain does just that.
>know every line of code
But code is there to simply provide some basic mechanisms, not the brain's structure. It is better compared to laws of chemistry or something on that level. It does not need to rewrite itself for the data structures that represent the simulated brain to evolve.
That said, I think you should be a little more critical of deep learning. So-called neural networks are more meaningfully described as high-dimensional curve fits. When people "train" neural networks, they're fitting the parameters of a model (just like y = a*x + b is a model with a and b as parameters) to some data. The results are impressive, and sometimes downright spooky, but neural networks are closer in spirit to a curve fit in MS Excel than they are to a real brain. There is no deep why. These are just nested compositions of affine transformations and ramp functions.
A reductionist would say that we can explain any phenomenon in the Universe because we know how atoms interact and everything is built out of atoms. When you ask why a tree grows, the answer that it grows because of the interactions between atoms is not useful and approaches the problem on too low level. Similarly a programmer can create artificial neurons and perfectly understand how these neurons operate on a low level, but it doesn't mean the programmer will understand the high level emergent phenomenons that can result from these low level interactions.
This is not at all how modern AI systems work. Modern AI systems work by feeding lots of data into a system that uses randomized heuristics to come up with an answer that has high confidence. You change the training data a little bit and the output can be entirely different, for reasons that are far from clear.
I definitely believe things happen in large programs that no one really intended.
Any evidence? It's fun to think about, but most people who think this are guilty of one of the mistakes I've listed. Or don't consider it a mistake. Or are not programmers.
AI, as long as it is software, is programmed with a very specific understanding of what is being programmed. Every line of code is intentional. Nothing exists in between the lines.
> guiding this evolution
AI is perfectly guided. Every program is completely understood (insert incompetent programmer joke here). Every symbol is typed with purpose on purpose.
The point is, Roger Penrose is not talking about programs. He's talking about physics.
When a programmer successfully emulates consciousness, every line will be deliberate, and they will know exactly how they did it. Not only that, if their program behaves like real consciousness, the programmer will have had to have knowledge of real consciousness. Otherwise what they've built would be something else. Something exciting I'm sure, but something else.
All of our models (and programs) are based on our understanding of the real world. Hence, they are simulations. The deeper our understanding of the real world, the more accurate our simulations.