Either way, I can totally imagine someone saying "punch cards and relays scale down horribly - you could never make a small computer".
A small steam turbine is an apparatus with about the size and complexity of a car engine. But if it has to be licensed and inspected by the NRC then you can forget about it.
And you could make a little RTG that could power a phone for a decade. Even with the terrible efficiency of thermoelectric generators it would still be much smaller than a lithium battery that could store the same number of kWh, because it's ~10% of the efficiency multiplied by much more than 10 times the power density. The main limit you hit on size is practicality -- why make a tiny, tiny, tiny one that lasts for a month when you can make an only marginally larger one that lasts for ten years?
Also the potential damage is a lot higher e.g. if you're in a car accident and the RTG gets punctured vs. just a Li-ion battery catching fire. In terms of cleanup and containment.
The point is that it's not a scaling problem. It's practical to make an RTG that small as a technical matter.
RTGs can be made small, but they also completely sacrifice the power density of nuclear due to their abysmal efficiency. Their main advantage is maintenance-free longevity more so than compactness or anything.