It's not just OpenBSD. For example, here's an interesting comment from Matt Dillon's commit to DragonflyBSD to mitigate Meltdown [1]:

> I should note that we kernel programmers have spent decades trying to reduce system call overheads, so to be sure, we are all pretty pissed off at Intel right now. Intel's press releases have also been HIGHLY DECEPTIVE. In particular, they are starting to talk up 'microcode updates', but those are mitigations for the Spectre bug, not for the Meltdown bug. Spectre is another bug, far more difficult to exploit than Meltdown, which leaks information from other processes or the kernel based on those other processes or kernel doing speculative reads and executions which are partially managed by the originating user process. Spectre does NOT involve a protection domain violation like Meltdown, so the Meltdown mitigation cannot mitigate Spectre.

> These bugs (both Meltdown and Spectre) really have to be fixed in the CPUs themselves. Meltdown is the 1000 pound gorilla. I won't be buying any new Intel chips that require the mitigation. I'm really pissed off at Intel.

[1] http://lists.dragonflybsd.org/pipermail/users/2018-January/3...

The OpenBSD developers are mad that Intel didn't inform them, or any of the other BSDs. Only Microsoft and Linux developers where included in the information that was under embargo. I don't think they're upset that Intel made a mistake, not more so that most others anyway.

Don‘t they refuse to honor embargoes? That would be an explanation why they got no information privately.

They don't like embargoes but they don't go out of their way to break them. The only embargo I know of that OpenBSD broke was for OpenSSL, and that was an honest mistake, as explained here: https://www.tedunangst.com/flak/post/regarding-embargoes

There was another "incident" with the KRAK embargo, where OpenBSD got permission to silently patch it early and then the researcher who found it regretted giving them permission.

I think people put these two incidents together, combine it with the developers' attitudes towards embargoes and come out with: OpenBSD doesn't honour embargoes!

I guess if they made an explicit statement that they will comply with embargoes going forward, they'll be able to correct the record. But I don't see that that has happened. Lots of egos have to get out of the way, maybe?

Here's an LWN article that references Theo's position on embargoes: https://lwn.net/Articles/601958/

This article seems to suggest that of all the BSDs, only OpenBSD / Theo takes this position - at least in in the case of the OpenSSL problems, the others got advance notice and participated in the embargo.

It might be argued that the risk of the exploit making it into the wild would have been higher if the BSDs were notified, on account of the sort of accidental premature disclosure that actually happened. This would appear to be self-serving if stated by the embargo insiders, but it may still be valid, especially if my guesses that a) this problem's biggest potential impact is in cloud computing, and b) there is relatively little use of BSDs in cloud computing, are accurate.

To me, one of the interesting bits there is the accidental timing of a comment.

From https://www.krackattacks.com

> We notified OpenBSD of the vulnerability on 15 July 2017, before CERT/CC was involved in the coordination. Quite quickly, Theo de Raadt replied and critiqued the tentative disclosure deadline: “In the open source world, if a person writes a diff and has to sit on it for a month, that is very discouraging”. Note that I wrote and included a suggested diff for OpenBSD already, and that at the time the tentative disclosure deadline was around the end of August. As a compromise, I allowed them to silently patch the vulnerability. In hindsight this was a bad decision, since others might rediscover the vulnerability by inspecting their silent patch. To avoid this problem in the future, OpenBSD will now receive vulnerability notifications closer to the end of an embargo.

Note the date there that de Raadt was commenting on the discouragement of sitting on a fix for a month. What is the likelihood that he would be very discouraged to sit on it for six months? What if it was a three month embargo that changed to a six month embargo - when would the fix be released?

I would assume that those are questions that need to be asked prior to notifying a project.

They do, they break embargoes and that's why they are now paying the price for it.

This is why embargoes are stupid. Unless you're 'blessed' you get screwed. They only exist to give intangible benefits to large corporations at the expense of everyone else.

It's the technological version of insider trading

I think this is a valid opinion, but it its a bit circular.

'We don't respect embargoes' -> 'Company only releases to individuals who respect embargoes' -> repeat

Well, there is the ME for one, an omnipotent, mandatory backdoor. Then there are _many_ undocumented opcodes. What does 0f0d00 do? What 0f78c0? What dbe0? I can continue for a very long time. These are just some known unknowns. If you still need more reasons, they are uncooperative when it comes to certain other firmware blobs.

If you are open to arguments, there are many good reasons to take a negative stance towards Intel.

Fun fact: I sent an overview of unknown instructions in compressed text form to a gmail address, and Google rejected it citing potentially malicious content.

To be fair gmail flags basically any code, if it's zipped i've found it only needs to vaguely resemble code. basically gmail is only safe for sending image formats and document formats that it knows of and don't contain macros, then it's just a crapshoot that you don't get matched a false positive. Gmail is far from developer friendly anymore.

You can't even send encrypted 7zip archives through Gmail. Well, you can, but they'll block the download claiming it's not safe.

> Gmail is far from developer friendly anymore.

Was it ever?

Well, not positively, but there was a time when it didn't block you from zipping up a little snippet of code and sending it to someone or saving it a draft when your in a hurry.

> This exploit is very hard to imagine before now.

It's really not. It's the sort of thing you wonder after first learning about out-of-order and speculative execution in a computer architecture class, but your professor assures you that implementors have been very careful to ensure any partial execution is properly flushed and rolled back. Then it turns out that, nope, no one's actually been keeping an eye on this after all.

> but your professor assures you that implementors have been very careful to ensure any partial execution is properly flushed and rolled back.

And, at the time, your professor was correct, to the extent of what he/she defined as "any partial execution".

But his/her definition of "partial execution" only considered state changes to the programmer visible CPU architecture state (i.e., the user level register set and the flags register). Their definition ignored the cache, because at the time the cache was considered simply a transparent optimization system that did not effect the values of contents of the CPU architectural state.

And in a way, that definition is still valid, even after the knowledge of these exploits. The CPU architectural state (registers/flags), even after executing exploit code, is exactly what it would have been had sequential execution happened. The exploit takes advantage of the fact that you can arrange the right set of code to run just the right way to leave a different state in the _cache_ (i.e., that thing believed to have been merely a transparent optimization).

It turns out now that the belief in the cache being transparent and only providing optimizations was the flaw in the belief system at the time.

> But his/her definition of "partial execution" only considered state changes to the programmer visible CPU architecture state (i.e., the user level register set and the flags register). Their definition ignored the cache, because at the time the cache was considered simply a transparent optimization system that did not effect the values of contents of the CPU architectural state.

You're putting words in other people's mouths here.

...but if it is "so obvious" - why has it taken ~20 years to discover it (publically)?

It’s been discussed for years by people who didn’t need to see the exploit to know that it had to be there. Sane people have been avoiding timesharing for critical work on x86 since Core 2.

Life-safety work has no business in a public cloud.

This is a recurring pattern that just blows my mind: typically some small, often academic, group knows about some problem or potential problem that everybody else is ignoring until it becomes un-ignorable.

Anthopogenic climate change falls into this category.

I have to wonder, first, what sort of obvious stupid self-destructive things am I doing right now but ignoring. And second, how can we build systems that systematically take this into account? Is there some way to short-cut the process and find and fix problems in the early stages? Or better yet, design our systems so that they don't have the problems from the start?

It is not really comparable to climate change - people have been assuming that everything is OK, but there is no well-organized campaign to persuade people that these exploits don't exist now there is good evidence that they do.

There is one attitude that exacerbates those problems that are real - the tacit assumption that concerns do not count for anything until an exploit has been demonstrated.

The key thing to take into account here is that for every one of those early warnings that turn out to be true there's also a plethora of speculation that never amounts to anything.

Because like most inventions, its obvious after the fact. But getting there was time-intensive and wrought with many ways of failure.

Spectre is somewhat reasonable; at least some variants are forgivable. Meltdown isn't. The risks due specifically to speculation and cache timing attacks (and lots of related stuff) were well known and documented for over a decade. They shouldn't have been speculating across memory protection boundaries, and they should have known that - even if it was unclear exactly how one might exploit such information leakage.

If it is so well known why has it taken 20 years to be discovered?

I don't know for sure but i'm guessing it's because this is all proprietary stuff, security researchers working on closed source hardware have to to a lot of inference... turn an already difficult theory into practice on an obscured implementation that is substantially harder to reverse engineer than compiled code.

It didn't take 20 years to be discovered; PoC's for variations on this theme were found over 10 years ago.

The details really matter for making an exploit: so the amount of speculation and what gets speculated, how caches work, how good timing is (and how large the difference between cache and memory is) etc.

Merely knowing that the combination of speculation, caching, and timing have the potential to break through memory protection barriers is a far from enough to actually exploit that weakness.

For comparison: it was widely known that sha1 had weaknesses, yet it took many years for somebody to construct two pdfs that actually demonstrate a hash collision.

From my perspective: I've been singing Intel's praises for the last decade as they regularly (for my own tests/usage) beat out AMD in performance, clock for clock if not dollar for dollar.

With this recent news, I feel like part of the reason that Intel has been doing so well is that they have been _cheating_.

Clock speed has been an awful metric for 2 decades now (except for within a single CPU model series).

All processors (including AMD/ARM) since the early 90's have been using instruction pipelining. It's a universal performance improvement, not Intel cheating.

Intel get drastically more instructions per clock cycle than anyone else, which is a very useful metric. Intel processors still dominate in single-threaded workflows. Meltdown makes that performance advantage look slightly suspect - Intel might be cutting corners to squeeze out extra IPC at the expense of security, reliability or accuracy.

Intel is cheating by validating the protection ring of speculative executions asynchronously, aka Meltdown, which makes this timing attack work in kernel space and requires the PTI mitigation. ARM does this on some cores too. AMD insists that they do not.

I don't think this is equivalent to the VW emissions cheating ... I guess we could find out otherwise later but if so it would amaze me that such a large and variable organization kept a secret for 20 years.

Is it really cheating? What they are doing is well known and published, with one side effect that didn’t get rolled back (which seems more like a bug). At least that’s what I’ve understood from what I’ve read.

Implementing a cool optimization that happens to break your processor's security features is cheating, yes, even if you did it unintentionally, like accidentally taking a performance-enhancing drug.

They didn't unintentionally break security features; they knowingly did so - the potential for information disclosure was widely documented. In their defense - the solution isn't trivial in general, and large parts of at least spectre apply in ways a CPU would be hard pressed to prevent (how is a CPU to know that even though memory isn't marked as protected from currently running code in actual fact that's because the JIT and the JIT-tted code are running in the same memory space?). But the intel CPUs are leaking a lot more data than reasonable.

Your parent was talking about Meltdown; that's the one that breaks security features, specifically: memory safety of a higher privilege level from a lower one.

Intent matters, at least to me. This does not seem to be a case like VW's diesel-emissions shenanigans.

The problem still is: one of the units responsible for increasing the processor performance was not working correctly. So the increased performance of the processor was obtained by not working correctly. That qualifies as cheating - if not by intent but then by nature of the design. It remains to be investigated how much of the security implications of their design choices was or could have been known to Intel. The fundamental attack concept was known for many years.

No, it just doesn’t. You’re diluting the meaning of that term to irrelevance. There are plenty of other words that adequately describe what happened - sloppy, careless, short-sighted. Cheating isn’t one of them.

You can easily cheat, without bad faith, by being sloppy or careless. You can also cheat in bad faith and call it being sloppy or careless. Breaking rules often has nothing to do with intent.

It is working correctly. A side channel is unfortunate, but in no way comparable to a missed page fault.

In that case, probably no processors are “clean” as 100% elimination of all bugs is highly unlikely. Just like a bit of growth hormones slipped into your food supply without your knowledge.

All other chipmakers are 'immune' to Meltdown precisely because the attack is obvious. Cache timing is hard, but speculating execution before checking permissions is plain dumb.

If it was such a mythical attack why doesn't it work on anyone else's chips? Intel screwed up hard on this and I have no sympathy. Hopefully the incoming lawsuits will make up for the massive amount of money wasted for the performance losses

You may want to read the following from the Meltdown paper.

> We also tried to reproduce the Meltdown bug on several ARM and AMD CPUs. However, we did not manage to successfully leak kernel memory with the attack described in Section 5, neither on ARM nor on AMD. The reasons for this can be manifold. First of all, our implementation might simply be too slow and a more optimized version might succeed. For instance, a more shallow out-of-order execution pipeline could tip the race condition towards against the data leakage. Similarly, if the processor lacks certain features, e.g., no re-order buffer, our current implementation might not be able to leak data. However, for both ARM and AMD, the toy example as described in Section 3 works reliably, indicating that out-of-order execution generally occurs and instructions past illegal memory accesses are also performed.

The most recent ARM chips (cortex A75) are vulnerable.

Didn't know that. The chipmaking business is insular and pretty incestuous, I wonder if the same engineers built both speculative execution units.

I've got a friend in CPU design and he's only got about 50 companies he can work for in the world where he could do the same job he does now

There are only so many ways to accelerate a CPU by speculating.

I’ve written RTL that speculatively fetches data from memory in order to avoid bubbles in a pipeline. Not a CPU, but the concept is exactly the same.

If somebody had assigned me to a CPU project without guidance from a security architect and ask me to speculate reads, I’d probably have done the same as Intel.

The chance that the same guy did both CPUs is small. It’s just that it’s not an unreasonable way of doing thing if you’re not familiar with these kind of attack.

In the never ending quest for yet more performance out of memory systems that are ever slower than the CPU's, this is likely exactly how it happened. The speculative reads, because they were speculative, were likely viewed as harmless (because the instruction that generated the read would never be committed if the speculation missed anyway, so CPU state (registers/flags) would not be changed improperly).

And if anyone even considered the results of the reads, they likely saw them as nothing more than free cache pre-fetch instructions that would enhance performance should the speculative path turn out to be the correct path after-all.

And because the push was for yet more performance, free cache pre-fetch operations were likely viewed as a great bonus.

Actually, what I heard through the grapevine is that it became vulnerable after an engineer retired.

Sadly that doesn't fit OpenBSD's "Intel you suck" narrative.

> This exploit is very hard to imagine before now

theo de raadt of openbsd called this out, 10 years ago: https://marc.info/?l=openbsd-misc&m=118296441702631

Well they did release their Coffee Lake generation chip after they already new about the exploit.

Now i know it would take a lot for a major company to delay/cancel a release like that, but i think it could be argued that they were dishonest if they knew it would have a performance impact.

They would have had to delay it for a year or more.

So the decision was, go ahead and release it even though we know it's vulnerable because we'll make more money that way than we would if we took the time to fix it and make it secure?

If so, I hope the market punishes that decision mercilessly.

To be fair, Coffee Lake is presumably faster/better and not more vulnerable than other chips [0]. If you have to mitigate anyway, and you're desperate to get back to pre-mitigation performance parity, releasing a faster chip that still requires mitigation could be a reasonable option.

[0] Caveat: Skylake and later are extra-vulnerable to Spectre because they have even more aggressive speculative execution, but fortunately, the IBRS microcode update is roughly equally performant as retpoline on Skylake+ only.

What? No, that's not even remotely how it works. Not releasing Coffee Lake when it was ready would not have sped up the design of a new CPU which fixes Meltdown in any way. The only way that not shipping Coffee Lake would have helped anyone is by indirectly increasing sales of AMD processors.

Well, Intel sales are going to drop off a cliff, and might not entirely be made up once a "fixed" version comes out.

The meltdown paper explains the same side channel race condition vulnerability exists on ARM and AMD CPUs. Their toy project shows all three architectures have the same flaw.

However for unknown reasons it’s worse in Intel. They were not able to exploit the vulnerability to dump the kernel memory on AMD and ARM.

So I find the whole “Intel you suck” really out of line. I would not be surprised someone clever will come and figure out a Meltdown style attacks which works on AMD and ARM CPUs.

As Bruce Schneier says: “attacks always get better they never get worse”

I thought this exploit was imagined a decade ago? I could swear I have seen links posted here where concerns were raised as speculative execution entered the picture. But I agree that it’s hard to blame them when it took a decade for something to actually materialize.