Maybe someday we'll be able to mine fissionable materials from asteroids and construct nuclear rockets in deep space with no risk to Earth. But that technology is firmly in the realm of science fiction today.
And, even if the fuel was a problem, reactors are going to be designed to be refueled. One could send up the reactor and fuel on different launches with the fuel sent in packaging the would survive any explosion (although the fuel itself already would)
RTGs are actually more of a radiological hazard, as the fuel for those are necessarily very radioactive.
In the end I do agree with your position that the safety of the people on Earth are likely to be a problem because safety is a feeling, not some well defined risk factor. Without some large generational educational project (or maybe change the name?), most people will think that nuclear power can never be safe, even when driving their gasoline powered car on a two lane road to work.
Maybe someone will start working on nuclear rockets again. I think a large number quality people would come out of the woodwork to be able to be part of such a project.