This brings to mind articles and discussions I read in the early 90s about differences in performance between running in real (or "unreal") mode and protected mode --- yes, the extra permission checks, paging, segmentation, etc. definitely add some overhead:

https://groups.google.com/d/topic/comp.lang.asm.x86/YbQJ_m0d...

Thus they decided to do permissions checking in parallel with other operations, leading to Spectre. It's interesting to note that the original intent of protected mode was not as a real secure "security feature" immune from all attacks, but more to provide some isolation to guard against accidents and expand the available address space. The entire Win9x series of OSs embody this principle.

What would the TempleOS of CPUs look like, I wonder?

x86 running in unreal mode might be close. I wonder if anyone has done any benchmarking of recent x86 CPUs in that mode...

> Thus they decided to do permissions checking in parallel with other operations, leading to Spectre

No. Leading to Meltdown. Spectre is fundamental to the way we approached any speculative execution until now.

Spectre variant 2 is not fundamental to speculative execution. It can be fixed by tagging the indirect branch predictions with the full virtual address and the ASID, and flushing it together with the TLB.

Any idea what will be the solution? Keep more speculative data in the pipeline? Performance penalties on security faults? Complex rollback?

you could certainly extend the unused transactional support to do a chandy-lamport thing, with the difference that when you run out of space isolated cache versions, you just can't speculate any more.

it would be a lot of machinery

you could also do latency hiding with smt instead of trying to fight it head-on with speculation. ultimately probably more productive, but either the compiler or the programming model or the user has to expose that concurrency.

If all speculation used a separate cache that is only committed on retirement, that should keep the side channels at bay.

> This brings to mind articles and discussions I read in the early 90s about differences in performance between running in real (or "unreal") mode and protected mode

I remember a 10 years old Microsoft Research project that implemented an OS that would use the .NET managed runtime to implement security. IIRC, they had interesting differences with CPU memory isolation off.

I like the idea that you don't need hardware barriers to isolate programs when they are lobotomized.

You're speaking of Singularity and it's "software isolated processes", which amounted to static verification of IL before AOT compiling it to x86. From the perspective of Sing#, the only way to express IPC was in the form of protocols, which were essentially formalized function call dances between two processes.

Singularity would be just as vulnerable to the recent bugs as contemporary OSes are, possibly more so because there is even less timing uncertainty when crossing privilege domains, making the attacks even easier

Cosmos OS is another .NET-based OS toolkit that does this. Here's a video of one of the developers talking about this, and other things: https://channel9.msdn.com/Shows/On-NET/Chad-Z-Hower-aka-Kudz...

Plenty of embedded CPUs don't have MMUs. Having done multi-"process" programming on them, it is hard to appreciate how nice basic address space protection is from a debugging perspective.

does modern x86 even support real mode?

AFAIK all x86 processors start in real mode.

Intel tried making a 386 variant with only protected mode a long time ago, and it didn't sell very well: https://en.wikipedia.org/wiki/Intel_80376

32-bit protected mode only, but with all support for paging and virtual memory removed? I can see why that combination never really caught on.

They made it from 1989 to 2001 so it must have been okay for the embedded market that it targeted.

It might have a better chance these days, with UEFI becoming the default.

Rumor is Intel is trying to get rid of the BIOS and just start the processor in protected mode

Sure it does. All x86 CPUs boot up in real mode, for example. Bootloaders are written in real mode.

> Bootloaders are written in real mode.

Not when the bootloader is an UEFI application. UEFI puts the processor into 32 bit protected mode or typically long mode (64 bit). So when your bootloader that is implemented as an UEFI application is started, you are already out of real mode.

> We could get a >50% performance boost by ignoring security.

What makes you believe that? The (crude, emergency, not designed-in) workaround dropping performance by 50% doesn't mean that fixing the issue in the design requires reducing performance by 50%.

Indeed. Google's published data suggests negligible losses if you are using FDO, which you must be if you care about performance.

Are you referring to this?

  Performance Details
  Overhead
  Naturally, protecting an indirect branch means that no
  prediction can occur.  This is intentional, as we are
  “isolating” the prediction above to prevent its abuse. 
  Microbenchmarking on Intel x86 architectures shows that our
  converted sequences are within cycles of an native indirect
  branch (with branch prediction hardware explicitly
  disabled).

  For optimizing the performance of high-performance binaries,
  a common existing technique is providing manual direct
  branch hints.  I.e., Comparing an indirect target with its
  known likely target and instead using a direct branch when a
  match is found.

  Example of an indirect jump with manual prediction
    cmp %r11, known_indirect_target
    jne retpoline_r11_trampoline
    jmp known_indirect_target

  One example of an existing implementation for this type of
  transform is profile guided optimization, which uses run
  time information to emit equivalent direct branch hints.

  -- https://support.google.com/faqs/answer/7625886

Note the parenthetical in the first paragraph. The way I read the above, they're merely saying that the performance is the same as disabling branch prediction for the section of code. Which would be really slow. But, presumably, faster overall than _actually_ disabling branch prediction as there's probably no good way to explicitly do that in as localized a manner.[1]

And as I interpret this, the reference to FDO is unfortunately confusing and perhaps better placed under the heading of correctness. They're merely saying that the transformation of indirect branches to direct branches is something compilers already do to steer speculative execution down the correct path. Except in the case of a retpoline it's steering speculative execution into a trap.

[1] I'm no expert in this area (nor do I do much assembly programming) but AFAIU branch predictors these days have substantial caches of their own. Disabling branch prediction or flushing the predictor's caches is probably much more costly than simply steering the predictor into a trap. The single pipeline bubble you create is better than the many bubbles that would be created if the branch predictor had to warm up again for later code.

What is FDO? A quick google search doesn't result in anything meaningful.

Feedback directed optimization

More commonly called profile-guided optimization (PGO).

So they lost 5% of performance, then applied FDO, a generic optimization technique, and gained back about that 5%. In other words, they could be ~5% ahead without the bug (at least 2%, if you consider there are some helpful interactions between FDO and retpoline mitigations).

Not to my understanding. They were already applying FDO to all of their performance sensitive programs. They found that after FDO, the retpolines and the other work done to mitigate meltdown and spectre had negligible impact. This is because FDO replaces indirect branches with direct branch hints, so only in uncommon cases do you even have to execute the retpoline. That's my understanding, at least.

Why limit yourself? Just run everything in ring 0. No kernel calls overhead at all. Lightning-fast network stack. Use a unikernel, run your trusted code only, don't store secrets.

Beyond game servers, this could be the mode of operation of e.g. compute cluster nodes, relying on external firewall for security, and running zero untrusted code. Of course they won't care about meltdown or spectre either.

I’ve heard some rumblings that this is what some HFT shops were doing years ago to minimize latency and to reduce context switches. Because they already run bare metal and don’t share anything besides a physical facility with their competitors (if even that) the attack vectors are not the same as those running on shared machines either.

You can already get 1,000x speedup in JavaScript if you e.g. avoid CORS on images where you read all pixels one-by-one. What should take a few ns in fact easily takes 30s in such an image. Security and performance are adversarial; we can have almost-perfectly secure systems that would make 8th gen i7 slower than Z80 or superfast CPUs where anybody can do anything.

How can CORS possibly be related to image pixels in any way?

It's to prevent reading cross-origin images through JavaScript. A canvas is considered tainted if it contains data from a cross origin image that was loaded without CORS approval. Tainted canvases cannot be read from by JavaScript.

I don't think this requires a 1000x slowdown but I'm not super familiar with this area of the spec. It should be possible to cache the tainted state of the canvas in a single bit, only updating it when adding new images to the canvas, and checking the bit when converting the canvas to an array.

For some reason every single pixel access must be authorized through CORS. I am not sure if this is in the standard or just shoddy implementation in all browsers. Try it yourself, read pixels in one non-CORSed image, and then in CORSed and wonder what went wrong on the drawing board. Some people bypass it via WebGL.

> Some people bypass it via WebGL.

How? There's no concept of a tainted WebGL canvas because it just can't load textures without CORS.

Not sure if it still works but it used to work before (~2 years ago). Maybe it was fixed in the meantime.

That totally sounds like a bug to me, it shouldn't happen.

There's a trade-off, but I seriously doubt it's anywhere near that. For one, if you could get super fast by ignoring safety, that would surely be found in consoles. Then, a good old 486 or early Pentiums have none of these issues, having a multiple of z80 speed without some decades of progress we had in the interim.

I meant you could execute every instruction with cryptographic check, i.e. AES check for every single step if this instruction could be executed. That would bring speeds under Z80 and will be overwhelmingly secure if protocol is correct. I hope nobody would try to add it to normal CPUs in the future, but security guys are sometimes too idealistic.

Doesn't it also mean Intel kind of cheated their way to the performance crown by ignoring security while AMD does not seem to have these issues ? It certainly brings Ryzen closer again, however future improvements to the patch might make it negligible in the end.

Meltdown looks pretty much like cheating. Intel simply took security verification out of the critical path, and their public image ought to suffer for that.

Spectre is a much more complex issue. It is more of a "I would never thing of that" thing. Looks like there were people at the security community talking about it even 10 years ago, but it is certainly hard for knowledge to spread to chip designers - most of what security people study would be noise for them, and even the security people mostly ignore those.

AMD has the same issues. The difference in Meltdown, according to the paper, comes down to idiosyncratic peculiarities of timing differences, not anything fundamentally more sound in other processors.

WRONG. You fell for the Intel PR strategy. AMD is completely unaffected by Meltdown, since AMD checks permissions before fetching memory into the cache.

Section 6.4 of the Meltdown paper seems to indicate that the same flaw is present in AMD CPUs, but not easily exploitable.

Do you have a different source for that?

The toy example referenced in section 6.4 is actually closer to Spectre. It's missing the crucial distinction that the speculatively loaded address should belong to a page inaccessible from the current ring level.

To see why that toy example is insufficient, consider that you could simply execute the load directly without putting an exception in front of it and you would be able to read the value.

Some Amiga people still believe that "virtual memory" is a passing fad, and when the Amiga comes again in glory we'll all realize how futile and self-defeating it was to prevent people from accessing the raw hardware and OS.

Also, unikernels move in this direction. If you just run one program per VM, virtual memory doesn't gain you much security.

IncludeOS especially.

> One service pr. VM means no need for virtual address spaces, and no overhead due to address translation. Everything happens in a single, ring 0 address space.

> if you're running a game server you really don't have to care very much

Think also about scientific computing, which usually doesn't need to be done in a secure multi-user environment (strictly speaking). I.e. you could limit security to the entrance gate (log in).

It's more than that even - given that a high portion of the code being run now is using virtual machines, a lot of that protection is redundant. If all code is run inside VMs and zero 'native' code is allowed, then you could run without needing protection rings, system call overhead, memory mapping, etc - which in theory could more than make up for the virtual machine overhead.

This was being explored with Microsoft's Singularity and Midori projects but seems to be a dormant idea.

A fun talk on this idea with JavaScript: https://www.destroyallsoftware.com/talks/the-birth-and-death...

I don’t think it works like that. You still need protection between the rings inside the VM and the CPU is providing that protection. The VM is not only a user process for the host, it is executing its kernel code in the virtualized ring 0 on the CPU (where its still the CPU that provides protection).

As i understand it, with this approach you wouldn't have a separation between kernel code and the virtual machine - everything runs in a single address space and you rely on the virtual machine verifying the code as it JITs it.

And that, of course, is how you get Spectre.

It should be possible to apply retpoline in the JIT to mitigate that.

Emphasis on "mitigate".

Isn't that a unikernel?

No. Each copy of the VM is still it's own process. The kernel just trusts the VM to be safe.

"What would the TempleOS of CPUs look like, I wonder?"

In the old days, probably Burroughs B5000 for it being high-level with security and programming in large support built in before those were a thing. If about hackability, the LISP machines like Genera OS that you still can't quite imitate with modern stuff. If a correct CPU and you want it today, then I've been recommending people build on VAMP that was formally-verified for correctness in Verisoft project. Just make it multicore, 64-bit, and with microcode. Once that's done, use the AAMP7G techniques for verifying firmware and priveleged assembly on top of it. Additionally, microcode will make it easier to turn it into a higher-level CPU for cathedral-style CPU+OS combos such as Oberon, a LISP/Scheme machine, or Smalltalk machines. Heck, maybe even native support for Haskell runtime for House OS.

https://www.smecc.org/The%20Architecture%20%20of%20the%20Bur...

http://www.symbolics-dks.com/Genera-why-1.htm

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.217...

http://www.ccs.neu.edu/home/pete/acl206/slides/hardin.pdf

http://www.projectoberon.com

If you could design your own processors to run your only code and processing your only input, I’m pretty sure it would be 10x to 100x faster

ASICs and FPGAs do this.

They're used in bitcoin mining where you do the same operation over and over.

> We could get a >50% performance boost by ignoring security.

Perhaps by making it unnecessary, rather than ignoring it - SPIN is now over 20 years old:

http://www-spin.cs.washington.edu/

There are other single-address-space safe-language systems that are even older; Genera is from the early '80s.

I assume those save language would still fail to defend against spectre. The problem is that the „code path with data“ executed in a speculative branch is not considered by the compiler but „fantasized“ by the cpu.

Remember that Spectre can be exploited through JavaScript JIT. The JS sandbox is quite comparable to the models used for save languages.

Plus this is not a security issue on a single tenant server running only the game server code.

There are some plausible ways it could still be exploited, but, I would say those deserve higher in the stack mitigations.

As a contrived example, a game server that accepts maps over the network that can contain a scripting language like Lua/javascript could theoretically cause data leaks.

By higher in the stack mitigations, I mean things like neutering the javascript timer accuracy in chrome fix to Meltdown/Spectre.

You can get a performance of this magnitude because the baseline secure system is a hurried patch and wasn't designed for it. We're far from the optimal performance-security trade-off.

Nice thoughts but the vulnerabilities of Spectre and Meltdown do not directly threaten code which runs on the vulnerable CPU. Instead, the vulnerabilities allow malicious code running on the vulnerable CPUs to attack the system and other apps. A subtle difference, but makes the job to identify the applications which are not trustworthy and avoid running them on the performant but potentially vulnerable cores.

If every application were signed (like on the apple store), does that help reduce the risk here? If it were a game server then all apps run would presumably be provided by some official third party who could be relied upon to sign the binary.

If it were all signed, then is the risk of the box becoming compromised fairly small and worst case, replacing the entire box (rebuild) if there were an issue could address problems quickly?

[edit to make it clear that this is a question]

If the application can run sufficient complex arbitrary code it is not safe. Example: Safari with JavaScript JIT compilation.

It is my understanding that any sort of cloud setup would be completely out. Also, anything with a web-browser or other platform that runs third party code. But that still leaves an awful lot of servers that are under the control of a single company or individual and don't run any untrusted code (sandboxed or otherwise), doesn't it?

Even dedicated supercomputer processors (e.g. A2, SW26010, VIIIfx) use virtual memory. In fact, most GPUs nowadays include an MMU as well, something they historically never had.

(SC applications tend to be huge fans of using enormous page sizes, like 2 GB, which mitigates page table walk costs)

Yeah, you split your business into those who's customers need >1 physical machine, and those that need < 1 physical machine. Drop security for the former, increase for the latter. Both can phrase their application as a unikernel.

The performance boost is not about ignoring security, but about ignoring edge case of one particular security check on i386 which ended up being the only critical one for "modern" unix-derived (which for purposes of this discussion includes Windows NT) OSes.

In original x86 protected mode model (ie. real security flags on descriptor level and page protection bits only used for scheduling of paging) this would be non-issue.

No, it just means that it's better to design chips and software architectures with security in mind from day one. If you don't, then every single patch you add afterwards will bloat-up the whole system and the (software) performance will drop overtime.

It's just another way to say you're building technical debt into your chip/software if you don't consider security from day one. Because sooner or later you will need that security, if you have tens of millions or more people using that product. So it's not like you can just ignore it. You're just postponing it for later.

I might be wrong, but isn't this what trading companies do on their network-insulated trading machines?