"First: super-intelligent AI is unlikely because, if you pursue Vernor's program, you get there incrementally by way of human-equivalent AI, and human-equivalent AI is unlikely."
The arguments that follow don't say anything about why it's impossible or even prohibitively difficult to do, they only provide reasons why people wouldn't want to try. There are, however, motivations beyond those he takes into consideration.
Uploading: Again, he doesn't make any arguments as to why uploading is not achievable, he just talks about the very hard ethical questions that arise when dealing with uploaded intelligences. That didn't stop us from inventing nuclear weapons and a host of other ethicaly challenging technologies, so why would it stop us from inventing uploading?
One particular statement he makes in this section I do want to address specifically: "Uploading implicitly refutes the doctrine of the existence of an immortal soul, and therefore presents a raw rebuttal to those religious doctrines that believe in a life after death."
This is so obviously untrue that I don't know how any intelligent person could use it in a serious argument. Religions that believe in immortal souls will simply maintain that the uploaded copy is just that: a copy, a soulless simulation of a real person. How can you possibly prove this one way or another? "Soul" is a purely religious concept, beyond any temporal means of observation or measurement, and therefore not subject to empirical study. I still think you would see plenty of religious people oppose uploading, but for different reasons.
He finishes by discussing the possibility that our entire universe is already a simulation being run in the "real" world, and admits that nobody can prove this one way or another, at least not anytime soon.
This might be true of some religious institutions but not of all. The concept of the 'Soul' is actually derived from Aristotelianism (there is no mention of it as such in the Bible) and many sects, (particularly Catholics) believe that in order for something to be intelligent it needs to have a soul, that is, an immaterial aspect. If an uploaded person, or an AI, or space alien or whatever is demonstrated to be intelligent, it would by definition have a 'soul.'
You would still get lots of folks who refused to accept the person-hood of uploaded persons and those folks might actually dominate, but the actual theological consequences would be different (and far more complex) than you imagine.
Wrong, and wrong. The notion of the "soul" in Greek philosophy predates Aristotle; it's found in Plato's Phaedo, for example-- and there are several words in the Hebrew bible that are usually translated as "soul" (nefesh and ruach, which correspond roughly to psyche and pneuma, respectively.)
I stand corrected on this.
"The notion of the "soul" in Greek philosophy predates Aristotle."
True, but my point is that certain sects of Christianity borrow heavily from Greek philosophy when trying to intellectualize what a 'soul' actually is. Some tend to follow the Aristotelian conception rather than the Platonic (they are distinct.) I'm not trying to say that Aristotle invented the concept of the soul if that's what you're thinking.
That being said, I'll accept your fallacious assumption for a moment and turn your question around: what if an emotional system is necessary for self-aware, volitional intelligence? Our emotions are inseperably intertwined with the other aspects of our minds, and it might very well be that they are a fundamental aspect of consciousness.
Many philosophers and theologians have attempted to make completely rational arguments in favor of the existence of God. My liberal arts theme in undergrad was philosophy so I've read a fair amount, but I have yet to see a compelling argument yet. Could you please make some suggestions?
EDIT: Proving or disproving the existence of God is, in general, a fool's game. One could probably prove that certain types of deities are impossible, but it's impossible to prove that no form of deity is possible. On the other hand, proving the existence of God would almost certainly require a well-documented divine intervention, and even then many people would (rightfully) remain skeptical.
Also, my previous post didn't say anything about proving the existence of God: I said that they made "arguments in favor of religion," which is not the same thing. First, not all religions involve deities. Even if you restrict the discussion to religions involving deities, it is possible to argue that it is desireable or advantageous to believe in God without first proving that God exists. Furthermore, even some atheists argue that religion in general or some religions in particular have had a net positive impact on humanity.
p.s. "...but it's impossible to prove that no form of deity is possible" - could you point to a proof of that?
As for your latter point: if you disprove the possibility of deities with a particular quality or set of qualities, I can then describe a deity which lacks those qualities and is therefore outside the set of deities you have proven impossible. For example, if you can somehow prove that there has never been divine intervention of any form (which I highly doubt is possible to prove), then I could respond by positing a deity that does not intervene in the temporal universe at all, but instead rules only in the afterlife. Since the afterlife is beyond temporal measurement, it is by definition beyond empirical observation, and there is therefore no way you can disprove the existence of an afterlife. By extension, there is no way you can disprove the existence of a deity which rules the afterlife.
b) You are right. AFAIK, religious experiences involves the parietal and temporal lobes, so its not just the emotional system. Agree that it's an important (irreplaceable?) part of our self-awareness. However the fact that non-religious people can act ethically in a harmonious society indicates that it might be reduntant. Still, religious beliefs arise in a brain, so there will probably be a way to turn them off (probably not without sideeffects).
Also interesting is that most of his arguments seem to be economic ones. In short, "it doesn't seem useful so the required investments to develop it will never be made". Not so much that it is impossible physically.
He might be right, he might be wrong, only time will tell. Personally I don't think it is very useful to make predictions about this (which is kind of the same stance that he takes "it's unwise to live on the assumption that they're coming down the pipeline within my lifetime...").
While that'd be a nice thing to have, it doesn't seem anything like an AI in the usual sense... i.e. the sense in which AIs have personhood and their own thoughts and feelings and goals and so forth.
It seems that conceptualizing AIs as human-like is the #1 top mistake that leads people to think AIs are impossible (or, alternatively, safe once they're built). It's more enlightening to think of an AI as a cold mathematical machine that uses computational power to achieve whatever goal the original programmer specified, possibly increasing its own power in the process. It will happily eat your babies if it needs more atoms. Its lack of personhood won't save you.
If a superhuman AI exists, exactly what evolutionary niche are humans going to fill? Because there's a word for species which don't have an evolutionary niche: extinct.
But really, even humans are just cold mathematical machines when you get below a certain layer.
Very surprising coming from a SciFi person - most people who write SciFi (Scientific themed) tend to perform exhaustive research into existing science. Disappointing...
- CS says the path to super-intelligent AI lies through human-equivalent AI (actually, he says Vinge's program says that, and he explicitly criticizes it)
- but I can find an uncomputable perfectly ideal math function
- and I can hypothesize that maybe it'll be possible to approximate it cheaply
- and I can hypothesize that easy instances of it are good enough
- and I will now view human progress as a way of approximating that function, and submit that only easy instances were used.
This isn't merely red herring, it's red herring full of contradictions. If easy instances are enough, what's the relevance of the provably ideal uncomputable problem, besides rhetoric? If humans used easy instances, and they're the only example we know of that achieved I, how is that not weak evidence that AI through easy instances will pass through human-like I?
It shows that there's a path to AI that's gated on solving formal math problems instead of imitating wishy-washy emotional humans. This point may be obvious to you, but apparently it wasn't obvious to Stross.
> If humans used easy instances, and they're the only example we know of that achieved I, how is that not weak evidence that AI through easy instances will pass through human-like I?
Humans are also the only animals that can play chess, but chessplaying software didn't pass through a human-like stage where they got nervous, forgetful, etc. My intuition says math can't be that radically different from chess (both involve search on trees with a complicated evaluation function), and we're only a handful of new insights from having computers decisively beat humans at math. And at that moment, by implication from my previous point, they might just also beat us at everything else :-) Remember that, on an absolute scale, humans are about as bad at math as you can be and still build a civilization (otherwise civilization would've happened at a lower level of development) and also that we probably suck at math because math is a conscious task (see Moravec's paradox).
Related link that should make you scared if you buy my reasoning: http://www.vetta.org/2009/09/monte-carlo-aixi/ .
Enhancements to primate evolutionary fitness are not much use to a machine, - Said who?
I strongly suspect that the hardest part of mind uploading won't be the mind part, but the body and its interactions with its surroundings. That's actually the easy part, the motor and sensory systems even in humans are largely deciphered.
Uploading implicitly refutes the doctrine of the existence of an immortal soul I thought this was refuted centuries ago, and anyway, it's a red herring.
our hard-wired biophilia will keep dragging us back to the real world, or to simulations indistinguishable from it Is the "real world" something stable? The same human brains performed great with both stone age cave living as well as modern megacity-life.
And so on and so on. The basic argument is "It is definately possible, but we may have reservations about it, ergo it's impossible"
b) Stephen Hawking
c) False, soul is not a corporal thing [for those who believe in it], whatever the corpus.
d) We are limited by what we cannot perceive through our brains. We are "dragged back" by our inability to perceive higher intelligence, not because we 're lazy to adapt.
EDIT: as a demonstration, I will deal with the first point in the essay. It says: "super-intelligent AI is unlikely because, if you pursue Vernor's program, you get there incrementally by way of human-equivalent AI, and human-equivalent AI is unlikely". Actually, getting to strong AI through math (rather than through mimicking humans) sounds more probable to me. We already have formalisms that can compute the most accurate possible prediction and the most efficient possible way to optimize a a utility function (inferring the right physical laws in the process) if given tons of computing power, for example look up Solomonoff induction or Marcus Hutter's AIXI. These count as superintelligences, or at least superweapons that can destroy the world. Stross's argument does not demonstrate the unlikelihood of someone implementing a fast approximation to AIXI tomorrow.