There are numerous ways to look at safety here. All else being equal, non-nuclear-weapon states wishing to arm with nuclear weapons would hate a total ban on nuclear energy, due to dual-use mining acting as a cover for their arms development.

Is that a sufficient justification for actually imposing such a ban? I personally don’t think so, but reasonable people can disagree on that.