So: Yes, we might all get along fine; ideally the more intelligent entity would be even more enlightened that we are; etc - those are all valid possibilities, and no one is saying they are implausible.
All people are saying is that some bad possibilities are also plausible (Eliezer Yudkowsky writes some great stuff about all the accidental ways we could screw up trying to program an AGI to be benevolent), and that, as the downside of the bad possibilities is potentially so great, then, in the absence of any argument why they definitely won't happen, we need to think carefully about them.
Some would say that, because the potential consequences of a bad outcome are so great, we need to be able to prove the bad outcomes won't happen before we build this.
In my opinion, Eliezer Yudkowsky and those who follow his argument are not intellectually credible and should dismissed with zero consideration.
In looking at this issue, I would note that Yudkowsky's (Bostrone's) ultimate argument is exactly homologous to Pascal's Wager. [1]
IE, since there's a hypothetical small probability that some entity [A Future AGI/Christian God/Flying Spaghetti Monster] could exist and the stakes for ignoring the possibility are "infinite" then we must "rationally" labor now to deal with possibility regardless of not understanding the mechanism involved in that entity.
And the thing is that modern science and modern statistics pretty hinge on ignoring, uh, "guff", demanding that extraordinary claims submit their extraordinary evidence and so-forth.
Further, the other fallacious argument pushed by Yudkowsky et al involves treating an AGI as a wish-granting Genie - human make overt logical requests and the AGIs "twist" these in a world-changing/destroying fashion. The problem is the over-request-format is identical to "logical specification" approach of gofai, the generally carded view that AI can be achieved with only the symbolic specification of how the world works (notably Yudkowsky and friend simultaneously acknowledge AI won't be based on symbolic specification/gofai and ground their argument on AGIs operating on explicit orders only, which boil down to symbolic specification. The Bostrone example is you tell the Genie "make me a thousand paper clips" and it turns the planet into paper clips "just to be sure").
[1] https://en.wikipedia.org/wiki/Pascal's_Wager
[2]https://en.wikipedia.org/wiki/Wish
[3]
I myself am on fence, on the one hand it is a bit like worrying about over population on mars, and there really is nothing we can productively do right now to research AI safety, the best thing we have are convolutional long/short term memory nets, and there's not much you can say about it.
But at the same time worrying is a great move if the alternative is to just wait until we can do something.
We haven't unintentionally produced a program that had a whole range of unexpected behaviors. An intelligence having human-like "survival instincts" and etc would be a big variation on a tool.
You're not getting it.
The effective argument of the danger seer is akin to "humans will learn to fly but then our cities will be covered with feathers" -
Humans have incentive to produce flexible, learning, language-understanding machines. Humans don't an incentive to produce a thing that will suddenly go rogue and decide to kill us.
It is entirely reasonable to argue that the tendency of humans themselves to go rogue and decide kill other humans comes from the evolutionary processes that produced humans rather than from the quality of human intelligence. We can see other relatively intelligent animals that live with in more (and in less) harmony than humans. No reason intelligence implies entity X won't fully cooperate with the people who intentionally engineered it.
I would argue that we human imagine that AGIs would have the same "downsides" as ourselves simply because we don't have any conception of intelligent things besides ourselves.
Maybe a carefully human-engineered intelligence would for reasons unknown turn into the things we see on movies. Maybe the next generation of supercolider will create an subatomic chain reaction that destroys the earth. But it's premature to worry about these particular hypothetical, especially given humans with ordinary AIs do pose problems.