>Bostrom is basically dealing in the outcomes of very long joint probabilities, which, multiplied by the probability of each individual variable, is very low.

That's not how probability works. You can't just assume Bostrom is wrong by default. That might work for futurists like Hanson or Kurzweil, which do that. But Bostrom isn't making a huge number of assumptions. His book presents a large number of possible futures, and the dangers of each.

Bostrom is also using the tools of human philosophy assuming they are general enough to apply to superintelligence. So he comes off as inherently anthropomorphizing even as he warns against doing just that.

He said Superintelligence was a very difficult book to write and that's probably part of what he meant by "difficult."

There is plenty to doubt. One big doubt about the danger of AI is that AI is not an animal. It is not alive like an animal. It has no death like an animal has. It doesn't need a "self." It doesn't propagate genes. It did not evolve through natural selection. So, except for Bostrom's use of whole brain emulation as a yardstick, there isn't much of the commonplace things that makes humans dangerous that needs to be in an AI.

But if the ideas of "strategic advantage" are in general correct, in the way Bostrom uses them, then Bostrom is right to say we are like a child playing with a bomb.

Just don't enhance the chimps. That would be bad.