I for one fully agree. I am pretty sure that whatever true generic AI emerges in the future will definitely not share brain schemata with us.
It seems to me that you attack the idea of the AI singularity in a very narrow framework: human-like intelligence, human-like hardware, copyable consciousness / intelligence, etc.
What if none of those restrictions apply? (And I am pretty sure they don't apply indeed.)
I agree that singularity shouldn't become a religion. But you are too quick to discount the possibilities of an artificial mind with more processing power and more efficient algorithms (compared to our brains').
Tell that to Homo erectus. Yeah, whales don't have advanced technology, but they didn't evolve facility with opposable thumbs to manipulate their environment.
> You can't grow your brain as large as you want. There are physical limits on power consumption, power dissipation, and organisational complexity.
Whales show that biological brains could be 6x larger. As for electronic brains, I don't know how large the initial Einstein-brain would be. But if delays from Ethernet interconnect are permissible for intra-brain activity, then your brain could occupy a large datacenter.
> It's even debatable if you can grow an Einstein. What if GI is NP-hard?
I did state that "getting to a computer Einstein" was where skepticism might apply. Nevertheless, nature managed to do it by accident. Are humans NP-hard? They exist. Obviously they can be implemented somehow. You don't need to simulate a particular existing human perfectly; it just needs to be close enough to the way a human brain could work.
If you need an initial state for the brain, maybe you could create an artificial environment (i.e. with only machines, for easy simulation) that a hundred actual humans grew up in to the point where they developed language, verify that those humans later became functional adults; and then simulate an embryo growing up in that environment as well, and start evaluating the results and iterating. Kind of fun to think about.
> "What if we made a golem? We could do anything! What if we put ourselves into the golem? We could live forever!"
The data that computers contain is easy to copy—by design. I think this will be a property of all computers in the future, except possibly quantum computers that always keep lots of data in a superposition (or computers that, for security reasons, are designed to be uncopyable)—and none of the existing neural nets are quantum computers. So copying plus backups or RAID is approximately guaranteed to cover the "live forever" part. Most of what I've said is based solely on the assumption that we develop a computer program that simulates an Einstein, plus common facts about computer programs.
Even if there would be true generalized Artificial intelligence, it is not unreasonable to think that the formation of the right thought framework for those systems to function effectively will require a substantial amount of time. It is also reasonable to think that such change would probably be in symbiosis with human intelligence. This change would not be sudden, but happen over many generations. It is my opinion that this change has already started at least a generation ago.
Who is disputing that? I am 99.9% certain that this is true.
The point of the singularity isn't that AI is coming next summer -- it's that from one developmental point and on the intelligence and abilities explode exponentially. To get to that stage though, thousands of years might pass.
> It is also reasonable to think that such change would probably be in symbiosis with human intelligence.
Now that is not likely. At all. They have to have the same motivation, interests, emotions and needs like us. The chances of an artificial race being developed to cover all these at once are practically zero.
A race not motivated by the same things we are is practically impossible to sympathise with our own motives.
Well, the general horror story that is being told is that intelligence and the ability to use it go hand in hand. As soon as machines surpass our intelligence, all hell breaks loose and there is no turning back. The focus is very much on the individual (both human and machine) and not on the structure.
> They have to have the same motivation, interests, emotions and needs like us.
Here is where I need some more convincing. In my view, nature is full of systems and groups of species living in symbiosis. Due to the slow development of the thought framework, we will co-evolve with the machines, or become part machine ourselves. In a certain way, we already are.
Perhaps I don't really have the word for it and I am sorry for that. Our understanding of intelligence, whether mechanistic or biological, is mostly as points on a linear scale (IQ). I think this misses the big picture: we are not biological systems, but have lived in symbiosis with our child teknos since we made our first tools. In this regard, there is no subject nor object: they are one. I am fundamentally with Heidegger on this one.
In literature and popular culture people conflate concepts all the time. :) The way I always understood that horror story was that generic AIs get schooled pretty brutally and quickly and they learn to use anything they have access to -- thus materialising that popular horror story. I think that's not far fetched, having in mind that the organisations that are very serious about obtaining a true AI are mostly corporations and military.
> Here is where I need some more convincing.
Maybe here I got influenced by popular culture. The books and movies I've read and watched mostly setup the generic AIs in situations where they have to be either parasites (Hyperiond / Endymion books) or commandeer a lot of tech and personnel (entrusted to them, like Skynet in the Terminator franchise) for their own ends.
But to me, the mere fact that a future generic AI doesn't have body and emotions like our own already means it's not going to be at all motivated by the same things as us. And we might even serve some pretty grim purposes to it, like the robots recharging themselves with organic matter in the PS4 game "Horizon: Zero Dawn" (which of course leads to consumption and thus extinction of all life).
Meaning, an AI might just view us as a natural resource to be exploited, and nothing more.
Assuming GI is possible and you give it a robot body and then the body breaks after 20 years, how long will it take to get used to the new body?
It's even debatable if you can grow an Einstein. What if GI is NP-hard? What if the belief that GI - never mind consciousness - can be modelled with our version of computing hardware is just plain wrong? What if a GI System is uncopyable for practical or philosophical reasons?
The singularity looks more like plain old religious thinking than anything that could be recognised as real science.
"What if we made a golem? We could do anything! What if we put ourselves into the golem? We could live forever!"
And so on.