The fact the nuclear reactor hasn't been miniaturized to the size of a coin and included in every $15 childrens toy to power lights is in large part because of public perception and government regulation rather than real safety/health concerns - there exist plenty of typical household materials that are more dangerous than the safest nuclear materials.
RTGs can be made small, but they also completely sacrifice the power density of nuclear due to their abysmal efficiency. Their main advantage is maintenance-free longevity more so than compactness or anything.
Either way, I can totally imagine someone saying "punch cards and relays scale down horribly - you could never make a small computer".
A small steam turbine is an apparatus with about the size and complexity of a car engine. But if it has to be licensed and inspected by the NRC then you can forget about it.
And you could make a little RTG that could power a phone for a decade. Even with the terrible efficiency of thermoelectric generators it would still be much smaller than a lithium battery that could store the same number of kWh, because it's ~10% of the efficiency multiplied by much more than 10 times the power density. The main limit you hit on size is practicality -- why make a tiny, tiny, tiny one that lasts for a month when you can make an only marginally larger one that lasts for ten years?
Also the potential damage is a lot higher e.g. if you're in a car accident and the RTG gets punctured vs. just a Li-ion battery catching fire. In terms of cleanup and containment.
The point is that it's not a scaling problem. It's practical to make an RTG that small as a technical matter.
But it has, or at least close to:
RTGs[1] weighting a couple dozen kilograms are used in space exploration and in rare cases in cold climates. Their main problem is low power density.
Nuclear pacemakers[2] of just a few centimeters across used to be a thing - long lasting and reliable. Also used thermoelectric generators. Replaced by cheaper lithium-based batteries due to high costs of making the nuclear capsule resilient against various possible sources of damage - like body cremation, car accidents, etc.
Nonetheless we have reached certain natural limit of nuclear fission, and only use of nuclear fusion[3] (or major scientific and engineering break-throughs) could change the status quo. The three main unsolved problems for wide scale commercial deployment of fission power:
1) radiation shielding - right now we only have passive shielding via use of structural materials, weights a lot
2) critical mass for active chain reaction (heavy and requires active management) vs passive generation via spontaneous fission (low power density)
3) operational safety and non-proliferation issues: a fission device requires radioactive material, and radioactive material is by its nature dangerous
Notably all three issues are either minimized or entirely absent for nuclear fusion.
Lastly, in typical nuclear power plant, the power generation facilities are usually of comparable size to, or even larger than, the nuclear reactor building itself - so miniaturizing the reactor would yield only diminishing results. High-temperature superconductors ought to be of some help here.
--
[1] https://en.wikipedia.org/wiki/Radioisotope_thermoelectric_ge...
[2] https://www.medicaldesignandoutsourcing.com/medtech-memoirs-...
[3] "Typical fuel pellets are about the size of a pinhead and contain around 10 milligrams of fuel." - https://en.wikipedia.org/wiki/Inertial_confinement_fusion
Can be dramatically reduced with a neutron reflector [1]. Bare Sphere's [2] can be as small as 6.9cm diameter, and use of a neutron reflector could perhaps reduce that by half or more. 3.5cm is pretty small. Research is still ongoing into reflecting neutrons, and it's possible that some clever arrangement of neutron 'lenses' might allow criticality with almost no mass.[3]
[1]: https://en.wikipedia.org/wiki/Neutron_reflector
[2]: https://en.wikipedia.org/wiki/Critical_mass#Critical_mass_of...
[3]: https://www.isis.stfc.ac.uk/Pages/neutron-reflectivity---int...
Not to mention that those toys would end in a few days in the garbage, ready to be shipped by volume to be burnt ("recycled") in some other country.