Which brings me to another something I don't understand about this. If mispredictions leave their calculations in cache why can't that cached data be cleared after the branch is determined to be bad?
It's all either ring 3 (user) or ring 0 (kernel) for the purposes of this discussion. The biggest architectural impediment is the single bit for ring selection in page table entries. And since long mode requires paging, there's currently not a way to select rings 1 and 2.
Although you might be able to play with some of the hypervisor extensions to give you guest ring 0 context inside a regular user process... See Akaros for a good example of what that looks like.
> Which brings me to another something I don't understand about this. If mispredictions leave their calculations in cache why can't that cached data be cleared after the branch is determined to be bad?
Because the cache is a fixed size, and that just kicks the can down the road. You'll just test for the eviction in your exploit instead.
"Clearing the cached data" is just as detectable as caching the data in the first place. You'd have to treat the line fill as a transaction that could be rolled back. Unfortunately, transactional memory and CPU caches lie at polar-opposite ends of the performance spectrum.
IMHO the only safe way to do a fetch from prohibited memory is to not do it at all.
You can also observe changes to the TLB state, though - so you'd have to do the same buffering-and-commit dance there as well.
Maybe you can also leak data through the BTB or RTB (double-Spectre!), or by tieing up particular execution units conditionally which is observable by a sibling hyperthread?
But (given the idea that we're talking about future chips) it could help describe protection boundaries in a finer grain way to protect against Spectre in the same way that AMD is protected against Meltdown.
The main issue with spectre (in a long term sense) is that the CPU really has no way of understanding that you're running lower privileged code in a higher context, whether that be BPF in the kernel, or JS in the browser. IMO, we're going to be fighting the same battle over and over again as the all of the mitigations so far are heavily microarchitecure specific.
But in the context of trends, we're moving towards User^Kernel memory mapping with features SMAP. That idea combined with kernel changes to holistically enforce separation boundaries more effectively along with cache partitioning AFAICT leads to potential end game win on this front.
Edit: And to be totally up front my major side project is adding a BPF/seccomp esque virtual machine into a formally verified kernel, so my head is in that space.
For instance brushing off rings 1 and 2 from the dust bin of history could give user and kernel code ways to describe multiple levels of memory protection in a way that the chip can understand. That way, the same way that AMD was protected from Meltdown ("I can tell from the TLB that address is outside of what this code is supposed to touch, so I'm not going to speculate past that") could be expanded to a more general form.