If we were in the habit of recompiling from source with each CPU generation (this presumes source is available), that would allow much more innovation in machine language. And then the design of that machine language would more closely resemble the actual design of the CPU.
But there are such significant advantage to remaining binary compatible, that I'm not surprised with how things actually turned out.
I'm not at all sure that the best answer is "source code". For x86 CPUs the compatible layer has been the ISA and that has turned out pretty damn well, suggesting that source compatibility is not the only and may not be the most effective choice.
Particularly if the source is C, which is rather ambiguous compared to most machine languages.
I disagree, though I see your point.
If the x86 instruction set had been designed to be implemented in various ways depending on the available semiconductor technology, then that would be a much better situation. However, the basics of x86 were designed in the 1980s to be directly implemented in the hardware of the time.
If you were designing an ISA expressly to insulate software from the hardware, I'm sure there are dozens of design choices that would be made differently. Ditto for amd-64.
Or maybe said in a different way, even if we recompile from C to a new ISA, we still have to execute the code that (by virtue of its algorithm) has unpredictable branches and lots of cache misses.
I do think there's room for innovation at a higher semantic level, e.g. maybe a really fast thread message-passing/synchronization mechanism with a lot of hardware thread contexts. Or lots of other ideas, e.g. "helper threads" (user-provided speculative prefetch code) or "informing memory operations" (load-and-branch-on-cache-miss). (All of these come out of comparch research in the 90's and 00's.) But all of those would require programmer cooperation, since they introduce new semantics.
Luckily, not that much any more. In GPUs, for example, nobody cares, since nobody is carrying binaries around.
You say this as if it is a bad thing (or am I misinterpreting here?), but compatibility is enormously valuable. That's why the strategy of choosing compatibility over cleanliness of architecture is so widespread in successful complex systems - ISAs, OSs, the Web, programming languages, etc etc.
It's hard to love the resulting complexities, but remaining compatible really is almost always the right thing to do.