This is just a vague intuition, but I feel like space hardening typically is done to insure very very high levels of reliability in all conditions. Im not sure how much easier the job might be if you could accept a sizable amount of transient failures. For many of these infernce systems, it feels like some "bad" processing might not really be a problem.

As others have said though, and as your "extreme transistor density" points to, high heat dissipation & energy usage are absolutely very real factors here. Still, Coral, back in 2017, was a half watt 2tflops inference engine, on a non-cutting-edge (at the time) process.