A hot object will radiate heat as infrared. The hotter it is, the more heat it will radiate. So, a sufficiently large heatsink designed to exploit that, can cool even in a vacuum. (Theoretically, this could even be used for reactionless propulsion.)

In the 70s and 80s, the USSR launched a whole series of radar satellites that were powered by several generations of nuclear reactors: https://en.m.wikipedia.org/wiki/RORSAT

Liquid metal coolant, as you suggested. Also, direct thermionic conversion to electricity -- no heat engine cycle. No control rods either -- control reflectors outside the core. Very different kind of nuclear engineering than is needed down here on Earth. No shielding!