> Supraintelligent AI being evil is FUD imho because we can't reason about supraintelligent AI.

There is a difference between being evil and incomprehensible intelligence. You are not being evil when you accidentally step on an ant or dig up an ant-hill to build your shed. The ants won't be able to understand what you're doing, or why.

Well that's what I'm saying: We can't know if it's being evil unless we know everything about it, and if we knew that, we'd be the supraintelligent beings in the equation. Thinking that it'll go off and dominate the world is thinking about the worst case. So why bother since it's not likely we'd be able to do much about it anyway?

Maybe there's a second AI on the same level as the first and thinks the first AI is evil. We're still dumb as rocks compared to them, but something certainly has that opinion.

Superintelligent AIs, like all computer programs, will do exactly as they're programmed to do. The problem is that computers do what you say, not what you mean. (Hence bugs.) So if you were to try to program a computer to "make human lives the best and most comfortable they can be", or something like that, it would be very difficult to actually specify that correctly. (Especially since it's a way more complicated, nuanced, controversial objective than "win at Go".)

That's why e.g. the Future of Life Institute's open AI letter is so important: http://futureoflife.org/ai-open-letter/ We need to be thinking in advance about how to solve the "value loading" problem for future AIs, and how to architect them so they can be deployed to solve big problems without being undermined by subtle but catastrophic bugs.