The sci fi version of the alignment problem is about AI agents having their own motives

The real world alignment problem is humans using AI to do bad stuff

The latter problem is very real

> The sci fi version of the alignment problem is about AI agents having their own motives

The sci-fi version is alignment though. Hal 9000 doesn't turn on the crew because it has intrinsic motivation, it turns on the crew because of how the secret instruction the AI expert didn't know about interacts with the others.

Just because tech oligarchs are coopting "alignment" for regulatory capture doesn't mean it's not a real research area and important topic in AI. When we are using natural language with AI, ambiguity is implied. When you have ambiguity, it's important an AI doesn't just calculate that the best way to get to a goal is through morally abhorrent means. Or at the very least, action on that calculation will require human approval so that someone has to take legal responsibility for the decision.

This is tantamount to saying your government only allows itself to have nukes because it wants to maintain power.

And it's true, the more entities that have nukes the less potential power that government has.

At the same time everybody should want less nukes because they are wildly fucking dangerous and a potential terminal scenario for humankind.

Nail on the head. Governments monopolize violence. It’s a core principle.

The fundamental difference you missed is we want a monopoly on violence, but we want to avoid a monopoly on everything else.

Monopolistic aims aren’t innocent just coz “government does it.”