To quote Wittgenstein: “Whereof one cannot speak, thereof one must be silent.” Writing about the the unwritable makes books hard to write. The tools of philosophy, in the 20th century at least, rely heavily on obscurantism and impenetrable jargon, to which Bostrom is no stranger.
Re:What’s Next for Artificial Intelligence
by Zigurd on Wednesday June 22, 2016 @12:46AM
(#11951610)
I found Superintelligence very readable. Very little jargon. I suspect he is wrong and that superintelligence is the Y2K of the 21st century. I suspect that super-machine-brains will dominate us as much as power plants sneer at our puny energy output. The most alarming scenarios assume anything, like practical nanotech, could be created by bootstrapping from a powerful brain in a box. And once you make that assumption, kaboom.
Just don't enhance the chimps. That would be bad.
Bostrom is also using the tools of human philosophy assuming they are general enough to apply to superintelligence. So he comes off as inherently anthropomorphizing even as he warns against doing just that.
He said Superintelligence was a very difficult book to write and that's probably part of what he meant by "difficult."
There is plenty to doubt. One big doubt about the danger of AI is that AI is not an animal. It is not alive like an animal. It has no death like an animal has. It doesn't need a "self." It doesn't propagate genes. It did not evolve through natural selection. So, except for Bostrom's use of whole brain emulation as a yardstick, there isn't much of the commonplace things that makes humans dangerous that needs to be in an AI.
But if the ideas of "strategic advantage" are in general correct, in the way Bostrom uses them, then Bostrom is right to say we are like a child playing with a bomb.