Mainstream CPUs are heavily optimised for latency over energy consumption (using speculative execution to effectively waste a multiple of the energy required for the computation itself). It'd be an interesting exercise to try to design an architecture optimises energy consumption over latency (perhaps rather than deep cache hierarchies, using cores that can power gate while they are waiting for memory accesses...)

That's a very good point. Even when one generates optimal code, the CPU will run all sorts of computations that do not contribute to reach the right result, but only to get there more quickly.

A lot of the low-power ARM processors do that - just avoiding speculative execution like you pointed out is a huge win if power is your main concern.

Another interesting take are barrel processors that can avoid scheduling stalled contexts, but they are only effective when we can feed them enough threads the time it spends waiting is less than a "normal" processor would. Early Xeon Phi's (and Cell's PPUs) behaved like barrel processors - each core could dispatch 4 (2 on the PPY) threads, but it did so round-robin. Scheduling only one thread per core didn't make them run any faster.