Unfortunately I think it's mostly the association with nuclear weapons. That has added to the not-totally-unreasonable fear of accidents like Chernobyl, but without actually evaluating whether other existing power stations are equally at risk, or if building new (and especially smaller / different designs) would be safe.