I really ought to do a blog post on the math behind this but it's not really a simple matter of power density because drive currents have traditionally shrunk along with the area and hence capacitance of transistors. For a long time people didn't reduce voltage to achieve classical Dennard Scaling[1] because back in the day nobody really cared about power but you could always go back to scaling that if you wanted. I suppose that the reason it looms so large in our heads is that Intel was having problems with power on the Pentium IV right as things were starting to change around 90nm.

Some things that make increasing performance with feature size more difficult recently are that transistors spend lots of time velocity saturated, there's increased leakage current, and the capacitance of wires has come exceed transistor capacitance.