I don't know but I can't imagine a highly predictable branch being slower than a TLB flush.

Well consider the fact that checking a table of “ok” programs is a branch and a lookup in of itself.

It would be a branch, but surely it would be a flag on the process struct set when the process started, rather than a lookup each time.

Yeah that should be really fast, still. Programs could also opt to just tell the OS "hey don't check this system call from me", on each system call, avoiding any lookup.

The impact of TLB flushing, not just the cost of the flush, is really significant - it's going to take a lot of work to be as expensive within the syscall path.

What would stop malware telling the os to not check it?

Nothing, but that only makes reading the malware's memory possible with these exploits. That malware won't be able to access memory of some other process, if that other process is using those flags itself.

Edit: For that to work that flag would have to work on the context switch level. So every time you switch away from a sensitive process, flush all buffers and whatever else, then switch. This also requires the kernel itself to enable mitigations as necessary when it touches encryption keys before switching back to user space.

That assumes that the malware can already have arbitrary control over system calls, at which point spectre isn't the issue.