[1] http://en.wikipedia.org/wiki/Hyper-threading
[2] for example, CONFIG_SCHED_SMT in linux
However, this is part of what hyperthreading accomplishes. The OS gives the CPU two tasks ahead of time, then when one task stalls, the CPU can switch over to the other one and work on it for a while.
And you're right, it's missing a lot because I'm writing an article, not a book. It is fun to explore details, but ultimately you have to stop somewhere.
On the other hand, long periods of no cache misses followed by long periods of waiting after a cache miss are exactly what you expect from real code (especially optimized code). So I'd think that you'd have much bigger gains from that. The same goes for branch misprediction.
Of course it's hard to write about a complex topic, choose the right details, and make it all seem simple. Thumbs up for trying!
Interestingly, there is at least one academic proposal to do something like this [1], but I'm not aware of any real implementations.
And when that fails, it's exactly the use case for hyperthreading.