That doesn't mean that the bad scenarios are as bad as people make out, but whether the AI is itself seeking to destroy humanity or just being used by militaristic people to plan simultaneous preemptive wars against everyone they see as a threat, the technology looks very dangerous to me. We will try to build it, that is just the human way, but like with the creation of other dangerous tech, we should be thinking about the dangers and how we will cope with them.
A benevolent AI would immediately prioritize:
A) Eliminating hostile AI research
B) Advancing the good of humanity on the metrics it was given.
Make no mistake, I'm optimistic about technology in general, I just want to observe that this is a unique moment in history an we should treat it as anything but inevitable that we'll make it.
The world has seen doomsday predictions forever, and while there is always the possibility we'll kill ourselves off, one has to recognize that there is also the possibility that we won't. Good and bad things seem to happen in about equal measure when you look at it on a larger time scale.
Doomsday thinking is counter productive to getting stuff done anyway as it tends to sap people energy (why invest time in what you're doing if you're all going to die anyway? sorts of reasoning).
Why is that so much harder to consider than the bad scenarios? I suppose evolutionarily the optimistic people probably didn't reproduce as often as the pessimistic people did but still.