That nuclear powered space travel is more economical and efficient at large scales isn't disputed by experts. The barriers reaching that scale are funding and political, not the engineering ones you've highlighted.
If you can invent any kind of technology at will, sure, it can be great. Then let's have light and reliable space nuclear power plants. At the moment we are very far from that.
It might have looked like a good choice forty years ago, but solar cells have improved a lot, as well as batteries.
We do have some newish things on the nuclear side that could help, like gas cooled reactors or supercritical power cycles. You need to get the temperature up (to make the radiators small) and the power density too.
We are much closer to the tech for efficient nuclear rockets than we are to the tech for an (even partially) self-sustaining colony on Mars.
This is what I thought. To my (admittedly fairly uninformed) knowledge, nuclear power is really nuclear steam engines, at least the ones I know of. As awesome as a space steam engine sounds, it doesn't seem too feasible for the reasons you presented.
https://en.wikipedia.org/wiki/Radioisotope_thermoelectric_ge...
A steam turbine nuclear reactor is a Rankine cycle heat engine, their efficiency is above 40%.
The reason RTG are used is simplicity and no moving parts, but efficiency greatly suffers. If we could figure out more efficient thermocouple all sorts of opportunities open up, haven't seen much work in this area.
Though nowadays you could maybe improve them with advanced quantum technology, like long wavelength solar cells. The cell itself still has to be cooler than the source, so you still need radiators... https://en.wikipedia.org/wiki/Thermophotovoltaic
You don't have gravity so you don't have convection or separation of gas and liquid. Hard to test such systems on Earth.
You can't make repairs easily.
Solar cells start to look really good really fast.
Maybe on Mars surface they could make some sense, or on the Moon where the night lasts for two weeks. Though on the Moon one would preferably settle near the poles without such problems.
But even on the surface, just regular batteries look really good on the simplicity front, and fuel cells beyond that.