It depends on the mix of circuits. Memory don't follow same design rules as logic so can scale quite well. Logic density may well be limited more by power dissipation than by design rules. In other words you can pack some circuits a lot denser than others.

I do agree that it is frustrating for someone from the outside to understand exactly what kind of scaling one can expect.

Indeed. Performance per watt is all that ultimately matters IMO.

CPU purchase cost matters too. The electricity cost of one 100W CPU might be around 500 USD per year in a datacenter. Assuming 5 years useful lifetime that's 2500 USD per CPU. But cheaper CPUs can easily cost 1000 USD less, so even if they're 20% less efficient they may still come out ahead.

A 100 watt CPU in any modern data center would be at an incremental cost of well under $40.

Data centers get cheap bulk power, cooling is mostly passive now (evaporative) and backup power systems are also a lot cheaper now compared to the compute cost than in the past.

It’s especially important for CPUs at home and which are rarely stressed.