If we outlaw AI, because of "risk", then people might experiment with it in secret, which has its own unique risks. If we study AI in the open, that has unique risks. If we descend further into authoritarianism to punish AI, that has risks. And no matter what we do about AI, the risks of nuclear war or life ending asteroid impacts remain.

> In philosophy, Pascal's mugging is a thought experiment demonstrating a problem in expected utility maximization. A rational agent should choose actions whose outcomes, when weighted by their probability, have higher utility. But some very unlikely outcomes may have very great utilities, and these utilities can grow faster than the probability diminishes. Hence the agent should focus more on vastly improbable cases with implausibly high rewards; this leads first to counter-intuitive choices, and then to incoherence as the utility of every choice becomes unbounded.

Curiously enough, this idea can be traced to one of the most prominent AI Safety advocates.