The TMS 9900 (and its predecessor, the TI 990 minicomputer) had a similar sort of idea, except back in the days of less layers of caching. https://en.wikipedia.org/wiki/Texas_Instruments_TMS9900#Arch...
(Of course, those days, there weren't orders of magnitude between the time to execute a single instruction and the time to access DRAM or ROM.)
http://en.wikipedia.org/wiki/AT%26T_Hobbit
It's an interesting idea. It probably makes thread switching very cheap (just change a pointer, a PC and maybe a condition code register).
[As the wiki article says, we were going to use it for the Newton, but the price kept going up and the chip was buggy.]
On game consoles, at least, successful developers know how the cache works at a detailed level; it's surprising how often it's cheaper to recompute something than to fetch it from memory again. It's a waste to fetch a whole cache line for a single byte, and you'll see good titles make very good utilization of their cache fetches (e.g., 70%).
DSP programming environments often provide ways to lock cache lines so replacement doesn't hose performance.
At the shared L2/L3 level, things get really complicated, since you suffer cache replacement from other parts of the system. (Why did our frame rate go down? / Oh, we sent an HTTPS request, which brought in the crypto stuff and ...). It's not something you can analyze or deal with statically, and good tools and test methodologies are essential.