We don't? Not only we question those all the time, but we also have failed in producing them in any stable condition in most of our societies.
tl;dr: Ethical AI is hard. Ethical NI is harder!
Doesn't "Coherent Extrapolated Volition" actually boil down to, "Hey AI, don't do like us. Do as we should do!" (If we were better, more noble beings.)
https://wiki.lesswrong.com/wiki/Coherent_Extrapolated_Voliti...
And if we can't have "Coherent Extrapolated Volition" aren't the only outcomes the subsumption of Homo sapiens into a different kind of intelligence and/or its extinction?
Come to think of it, "Coherent Extrapolated Volition" just sounds like the same sort of wishful thinking which religions hook into.
Voltaire in the 18th century: "If God did not exist, it would be necessary to invent him."
Tech in 2019: "We need to be the first to implement the god-like AIs, because the the first mover advantage will yield tremendous profits!"
If one is interested in seeing an opposing point of view to this, one could lookup Philip Hamburger and the term, administrative state.
Does it?
At a personal level, I'm much more scared of a bad person with a gun and access to me than a bad person with an AI and access to me. It's more important than the ethical use of a teacup, but certainly rivaled and beaten at a micro level.
At a macro level I think your argument has more ground to stand on when it comes to things like improved efficiency of mass surveillance, but how much of that is AI part rather than the mass surveillance part? What immoralities are enabled/accelerated that could not be done without AI? In the end I'm just as concerned about mass surveillance as I was before. In a practical sense, even if somehow passed laws limiting AI, do I really think that a government performing mass surveillance in the shadows is going to follow those laws when AI is a concept anyone (with the know-how) can implement? I don't think AI's power/capacity is as high as people think, it just tends to fit well with some very bad macro level ethical actions to enhance them.
I'm not sure I'm convinced either way yet, but the claim of "unrivaled" made me immediately skeptical, at least considering the AI of this decade(s). In a hundred years you're probably right with the unrivaled part, though atom bombs and the like are probably close seconds.
The other thing AI is about is cost and perhaps secrecy. Turk system requires many people who need to be fed, that puts a pretty high low bar on price. It also requires disseminating data to many agents, each of whom could leak it.
Being more accessible means more Bad Guys (unethical actors) using it.
AI only introduces a new danger, but at the current complexity of AI I don't think its any more significant in effect than the other ethical problems we have. In fact, many misuses of A.I are based on the uses, not the AI itself. I don't think linear regression is inherently an immoral tool either if we're going the "this tool is too dangerous to use/allow access to" route. Until we have anything close to AGI, AI is a tool only as ethical as the user of the tool. That's the real issue here.
We absolutely do need regulation, but I'll be damned if 10 people with power in any government understand AI well enough to regulate it. Every day I think we get closer and closer to needing to implement technocrats in governments. The FCC is a great example of a place we should have had the model for decades.