E.g. if syscall S reads kernel memory without properly checking pointers/offsets (maybe GCC optimized them out), then process A can possibly use that to steal process B's secrets. Or maybe the kernel does check the pointers/offsets, but isn't hardened against yet another speculative execution side channel attack your processor is vulnerable to, so leaks some information via timing information.
MAP_EXCLUSIVE breaks that exploit, and to unbreak it you need both syscall S and another buggy syscall (or maybe another row hammer attack) - to write kernel memory - and to successfully leverage that into tricking the kernel into remapping process B's pages.
No panacea, but I could see it rasing the bar in exploit difficulty as part of a defense in depth strategy.
But at that point we're at a microkernel design that needs context switches between kernel processes... Good, I'd welcome that. This feature just implements a level of security that the whole kernel needs to catch up to before it makes sense.
IMHO the question is what is the actual semantic here. What are you asking for ? Does it mean "at any cost", what does it guarantee (100% or statistically), what level of guarantee is acceptable, what level is -EOPNOTSUPP or similar ?
I think different semantics as Cox mentions would correspond to different threat models. With the right hardware support, it could mitigate against a kernel that falls under an attacker's complete control after the memory has been allocated. With less support it could mitigate against an attacker who can exploit read accesses at kernel privilege. But the details seem a bit open still...
It's supposed to be hardware based. If process A requests memory with MAP_EXCLUSIVE, and it verifies that it got it that way, the hardware is supposed to make sure the memory is only available to that particular process. No amount of context switching is allowing the kernel to read it, nor could should it be able to modify the running code in a process if it's properly hardened (e.g. put the code to run in the protected area). It pose some interesting challenges, but it seems quite possible
Hardware gets a flag that says "am I in privileged mode now?" and it does not (and fundamentally can not) know to whether the currently running code belongs to process A or something else, or to whom a particular memory region belongs - as far as hardware is concerned, there's no "whom" at all, that's an abstract construct arbitrarily introduced and maintained by the OS.
I'm pretty sure we've seen specialized CPUs in the past that did exactly that.
There's no reason hardware can't be given a notion of processes, and used to set boundaries between each other and the kernel. It's not an untested idea.
An exploit in your closed-source nvidia graphics driver should not get access to your private keys.
Easy fix: at the next context switch, the kernel overwrites the code of the process with its own code that copies the contents of the "secret" memory to somewhere the kernel can read it, and then executes its own code.
I guess I don't understand the threat model in which having data protected from the kernel is helpful.
If the idea is that the kernel is malicious, then it's not going to work in the first place (e.g. the kernel could just fail to honour the MAP_EXCLUSIVE flag in the first place). If the idea is that the kernel is initially trustworthy but gets compromised some time after the memory is made "exclusive", then it can execute its own arbitrary code in that process's context and get the contents that way.