Unwinding changes to the TLB on every mispredict would have a significant overhead and hurt overall performance. Removing valid data you just cached (speculatively or otherwise) is generally a bad idea.

User mode software requires a TLB (unless you want to do a page walk for every single instruction!)

Even if you could remove the TLB entirely from the CPU somehow, the attacker could just use the cache or some other microarchitectural structure.

Well that's disappointing. Thanks for the explanation!

I never have to worry about such low level details in my day-to-day work, so this is all new to me.