If you could grow your own brain as large as you wanted, and live as long as you wanted with the physical robustness and energy of a 25-year-old, that alone seems like a road to godhood, though perhaps a slow one. If you could download knowledge directly, as another commenter suggests, that becomes a much faster road.
> Apotheosis in the context of AI safety as well as for humankind is really about the accumulation of power, rather than the accumulation of knowledge. The extent of our apotheosis is limited only by what we can control, not by what we know.
If one country, even a smallish one, has 50-100 additional years of technological advancement, I think they might be able to use this knowledge to gain a lot more power. I'm pretty sure history has borne this out. Do you think we're at the limit of what new technology can do for military strength? Materials science (armor, weapons), biotechnology, energy and battery technology, cybersecurity, psychology and propaganda, lasers and anti-missile defense systems...
Suppose a country like North Korea had 100 Einsteins working on such things for a few years under the radar. I suppose it might be difficult for them to hide the construction of their army of thousands of robots invincible to all but the most powerful bombs... but then again, maybe they could do it underground, directly underneath some large innocuous manufacturing operation. And, of course, the cyber- and bio-weapons efforts would be very easy to conceal.
It's even debatable if you can grow an Einstein. What if GI is NP-hard? What if the belief that GI - never mind consciousness - can be modelled with our version of computing hardware is just plain wrong? What if a GI System is uncopyable for practical or philosophical reasons?
The singularity looks more like plain old religious thinking than anything that could be recognised as real science.
"What if we made a golem? We could do anything! What if we put ourselves into the golem? We could live forever!"
And so on.
I for one fully agree. I am pretty sure that whatever true generic AI emerges in the future will definitely not share brain schemata with us.
It seems to me that you attack the idea of the AI singularity in a very narrow framework: human-like intelligence, human-like hardware, copyable consciousness / intelligence, etc.
What if none of those restrictions apply? (And I am pretty sure they don't apply indeed.)
I agree that singularity shouldn't become a religion. But you are too quick to discount the possibilities of an artificial mind with more processing power and more efficient algorithms (compared to our brains').
Tell that to Homo erectus. Yeah, whales don't have advanced technology, but they didn't evolve facility with opposable thumbs to manipulate their environment.
> You can't grow your brain as large as you want. There are physical limits on power consumption, power dissipation, and organisational complexity.
Whales show that biological brains could be 6x larger. As for electronic brains, I don't know how large the initial Einstein-brain would be. But if delays from Ethernet interconnect are permissible for intra-brain activity, then your brain could occupy a large datacenter.
> It's even debatable if you can grow an Einstein. What if GI is NP-hard?
I did state that "getting to a computer Einstein" was where skepticism might apply. Nevertheless, nature managed to do it by accident. Are humans NP-hard? They exist. Obviously they can be implemented somehow. You don't need to simulate a particular existing human perfectly; it just needs to be close enough to the way a human brain could work.
If you need an initial state for the brain, maybe you could create an artificial environment (i.e. with only machines, for easy simulation) that a hundred actual humans grew up in to the point where they developed language, verify that those humans later became functional adults; and then simulate an embryo growing up in that environment as well, and start evaluating the results and iterating. Kind of fun to think about.
> "What if we made a golem? We could do anything! What if we put ourselves into the golem? We could live forever!"
The data that computers contain is easy to copy—by design. I think this will be a property of all computers in the future, except possibly quantum computers that always keep lots of data in a superposition (or computers that, for security reasons, are designed to be uncopyable)—and none of the existing neural nets are quantum computers. So copying plus backups or RAID is approximately guaranteed to cover the "live forever" part. Most of what I've said is based solely on the assumption that we develop a computer program that simulates an Einstein, plus common facts about computer programs.
Even if there would be true generalized Artificial intelligence, it is not unreasonable to think that the formation of the right thought framework for those systems to function effectively will require a substantial amount of time. It is also reasonable to think that such change would probably be in symbiosis with human intelligence. This change would not be sudden, but happen over many generations. It is my opinion that this change has already started at least a generation ago.
Who is disputing that? I am 99.9% certain that this is true.
The point of the singularity isn't that AI is coming next summer -- it's that from one developmental point and on the intelligence and abilities explode exponentially. To get to that stage though, thousands of years might pass.
> It is also reasonable to think that such change would probably be in symbiosis with human intelligence.
Now that is not likely. At all. They have to have the same motivation, interests, emotions and needs like us. The chances of an artificial race being developed to cover all these at once are practically zero.
A race not motivated by the same things we are is practically impossible to sympathise with our own motives.
Well, the general horror story that is being told is that intelligence and the ability to use it go hand in hand. As soon as machines surpass our intelligence, all hell breaks loose and there is no turning back. The focus is very much on the individual (both human and machine) and not on the structure.
> They have to have the same motivation, interests, emotions and needs like us.
Here is where I need some more convincing. In my view, nature is full of systems and groups of species living in symbiosis. Due to the slow development of the thought framework, we will co-evolve with the machines, or become part machine ourselves. In a certain way, we already are.
Perhaps I don't really have the word for it and I am sorry for that. Our understanding of intelligence, whether mechanistic or biological, is mostly as points on a linear scale (IQ). I think this misses the big picture: we are not biological systems, but have lived in symbiosis with our child teknos since we made our first tools. In this regard, there is no subject nor object: they are one. I am fundamentally with Heidegger on this one.
In literature and popular culture people conflate concepts all the time. :) The way I always understood that horror story was that generic AIs get schooled pretty brutally and quickly and they learn to use anything they have access to -- thus materialising that popular horror story. I think that's not far fetched, having in mind that the organisations that are very serious about obtaining a true AI are mostly corporations and military.
> Here is where I need some more convincing.
Maybe here I got influenced by popular culture. The books and movies I've read and watched mostly setup the generic AIs in situations where they have to be either parasites (Hyperiond / Endymion books) or commandeer a lot of tech and personnel (entrusted to them, like Skynet in the Terminator franchise) for their own ends.
But to me, the mere fact that a future generic AI doesn't have body and emotions like our own already means it's not going to be at all motivated by the same things as us. And we might even serve some pretty grim purposes to it, like the robots recharging themselves with organic matter in the PS4 game "Horizon: Zero Dawn" (which of course leads to consumption and thus extinction of all life).
Meaning, an AI might just view us as a natural resource to be exploited, and nothing more.
Assuming GI is possible and you give it a robot body and then the body breaks after 20 years, how long will it take to get used to the new body?
A larger brain don't necessarily provide a better survival advantage. It's well known that average Homo Neanderthalensis had larger brains than our modern average follow humans. And if you don't survive, you are nowhere toward the road to thaumaturgic existence.
What is the trade off is the most important question, as anywhere.
Does it really not though? If you watched the movie "Transcendence" you'd presume that a sufficiently [initially] sophisticated AI can decipher our financial markets quite snappily and make a very quick money off of that -- and use those to finance its endeavours. Which is exactly what the AI in this movie did, very successfully too.
My point is, given a much larger and better brain, the intelligence within it can optimise a much larger problem space at once, and that can very easily include resource acquisition as well.
If intelligence as we know it could ramp up to generally solve all problems, then surely evolution would keep increasing it and we'd all be gods or at least Einsteins by now?
Depends. If a future AI wants to make a civilisation that spans our own entire Solar System then it would have to tackle much more -- in number and complexity -- logistical, religious, political and what-have-you problems. So it might be too much for it as certain problems here on Earth are for us the humans.
However, if we postulate a self-aware AI who is Earth-bound and is having more processing power than a single human brain and has much more efficient algorithms than our own then it really easy follows that this AI would be much much better than us in solving our Earth-bound problems. Like in economy: more income and same expenses equal bigger profit.
> If intelligence as we know it could ramp up to generally solve all problems, then surely evolution would keep increasing it and we'd all be gods or at least Einsteins by now?
Well, we are limited by the physical size of our brains being able to squeeze out of our mother's hips during birth. Same way a future AI would, at least initially, be limited by the available hardware on which it can run, how much power it can consume and if its heat dissipation won't become too much.
However, an AI wouldn't grow in a natural environment, namely where natural limits would hamper an infinite growth. IMO if a private well-funded lab -- or the military -- invest heavily in such an AI they would just keep throwing hardware at it until it becomes efficient enough to be compared to a human and above. From that point on it is very likely it would become smart and proficient enough to optimise itself and start solving age-old problems we never were able to solve so far.
This, to me at least, is the so-called singularity point, in less extreme terms compared to its official meaning. Namely an entity that can optimise itself and achieve engineering perfection much better than what we can do.
I don't think that's something that's clear at all. Men and whales have larger brains than women, on average. Are they proportionally smarter? Without knowing how brains work, it's impossible to say what constraints exist due to size.
The peoples that worked bronze prevailed over the peoples that just used stones and sticks. Advances in AI are no different than that.
If you can automate production and defense systems, the natural consequence is that a few people will conspire against the rest and dominate them.
It is also a huge assumption that 'hostile' to humans AIs would be cooperative between themselves. In resource contention situation especially..
Those aren't actually limitations preventing us from becoming godlike. Those are limitations on us becoming more populous and collectively more experienced, which I argue isn't the same thing at all.
Apotheosis in the context of AI safety as well as for humankind is really about the accumulation of power, rather than the accumulation of knowledge. The extent of our apotheosis is limited only by what we can control, not by what we know.
Knowledge and intelligence only allow you to reason more effectively about paths to power given enough information about the environment, but it doesn't give you a strategic guarantee that you will acquire power - we know there are games that cannot be solved, and games that take too long to solve efficiently, and games that, even with sufficient computing power, rely on factors that cannot be controlled (such as luck), and games for which there are no Nash equilibria or stable winning strategies.
The real world is full of such games. Computers have only managed to beat humans at a small fraction of them, and even then only ones where brute-force and smart pruning (optimal game space search) are winning strategies - bots for games that require probabilistic reasoning, such as poker, typically don't fare any better against expert humans, largely because efficiently solving probabilistic games in general is (per a hazy seminar of AI many years ago, so I could be misremembering) an NP-hard problem.
There are plenty of both natural and artificial limitations around the accumulation of power besides game solvability and probabilistic reasoning's difficulty:
1) System designs that require a minimum set of resources that intelligences can't acquire. For example, individual humans usually can't build their own thermonuclear devices because they don't have access to weapons-grade uranium. Much of our information security relies on making it computationally expensive and infeasible for an attacker without state-level resources to break in, and getting state-level resources isn't a straightforward task at all. We humans haven't even gotten around to building our Dyson spheres yet, because the resources needed to do it far outstrip our ability to acquire them. Intelligence does not improve resource acquisition by itself.
2) Impossibility theorems and optimization difficulty. These place fundamental limits on power - as an example, we know Moore's law has to stop at some point because increasing chips increases power generation, and the amount of effort needed to generate such chips increases with every nanometer shaved off. Most of humanity's most difficult and important problems are optimization problems, and these optimization problems can be tricky to solve in polynomial time. In many cases, we have to rely on fast crude heuristics that give us less than optimal results.
The above limitations partly become easier if you try to parallelize your efforts by duplicating AIs (you can collude to acquire resources and solve problems), but this means you get all the fun parts of distributed systems theory for free: coordination, replication consistency, etc. If only all problems were embarassingly parallel! So it's not necessarily easier to add nodes to a knowledge graph just by adding more processors.