Don’t be generous to Intel. I work in a vertical strategic to Intel, and get regularly called on by them and get NDA presentations and the ability to talk to engineers about strategic projects.

For this incident, I got an email a few hours after the embargo was lifted, that essentially said that it was no big deal and referenced public information. The purpose of the communication was to have people like me message up the chain that this was no big deal. That misdirection is inexcusable, particularly when they could have given meaningful guidance under NDA.

We had some follow up questions, which weren’t really answered. We were directed to hardware OEMs, as ETA for microcode updates are out of their control and according to Intel are the full responsibility of the OEM. In reality, Intel was struggling to deliver the code, and the OEMs we deal with issued patches in hours, and had to pull back updates due to Intel code revisions.

Personally, I do have alternatives for strategic parts of the business that drive high margin Intel sales. Many critical aspects of my business can run on Intel or Power platforms, and we can engineer solutions either way in similar cost footprints.

Less strategic aspects of the business, like end user compute now have niche competitors that can gobble up Intel business very quickly. Half of my desktop users run on VDI, mostly with AMD thin clients. 50% of my constituencies can run their core line of business functions on iOS. iPad with a keyboard could reduce my Intel desktop spend by 50-75% for 2-3 years.

I'm also under Intel NDA and it took them days to provide any meaningful guidance at all.

The only useful thing in these documents was a timeline/detailed list for the microcode patches, all of which should be public.

They also claim that Spectre/Meltdown are "not a bug or flaw in Intel products" and their slide deck has a whole slide dedicated to forward-looking statement disclaimers. Sigh.

Needless to say, we're not impressed.

They know what they're saying is bull hockey, but they have to say it, otherwise they'll lose their lawsuits. They have to be able to point to the public spec and say "See!? We're completely in spec! No bugs here." Nevermind that the bug is in the spec.

Thanks for sharing those details. Deferring to OEMs is such a copout given how they are most likely going to handle security issues:

https://twitter.com/avtargill/status/951195158229463046?s=17

At this point it’s up to users to scour motherboard manufacturers’ clunky forums to determine if their platforms will ever receive patches. Given the severity of the issue this really should be handled with more accountability and with a greater sense of urgency.

It’s also a cop out as they were leaning on the assessments of the vulnerability as a medium risk information disclosure by many security analysts. That is accurate in many respects, but not in the context of a multi tenant environment or in places where different compliance standards are in play.

It’s particularly obnoxious when you read about the heroic efforts that were put in place to put AWS, Azure and GCP right. If it was no big deal, why go through that?

> purpose of the communication was to have people like me message up the chain that this was no big deal. That misdirection is inexcusable

But perhaps this was no big deal. We've seen years of research suggesting that modern CPUs are full of issues like this. There's probably a good decade worth of papers on cache side channel attacks. See https://eprint.iacr.org/2013/448.pdf for example

Perhaps the big deal is that there are still people who think they can safely run multiple different things on a single machine?

The 'bombshell' of meltdown isn't about cache side-channel attacks. Instead, it is about the use of out-of-order execution (meltdown) and speculative execution (spectre) to get side effects. It doesn't matter much that these side effects happen to be cache state. For meltdown, it matters that page protection is essentially broken. For spectre it matters that speculation on indirect branches are essentially attacker controlled.

Both attacks, in their practical form, use cache timing as the side-channel to extract information. But the surprise is the control over (as the paper calls them) 'transient executions'.

Yeah, I know. My point was that it is very well established that CPUs are full of issues similar enough to meltdown and spectre. Therefore it is foolish for anyone to assume that they can safely run untrusted code on a machine with secrets on it.

I think you might be misunderstanding what the phrase "side channel attack" means. It's very difficult to protect against side-channel attacks, because of their very nature. There are a lot of ways someone can try to exploit a system and of course not all of them can be accounted for in advance. But not all side-channel attacks are practical, some are done for academic reasons and can only be done in controlled environments with contrived setup.

like check this out: https://www.tau.ac.il/~tromer/acoustic/

> Here, we describe a new acoustic cryptanalysis key extraction attack, applicable to GnuPG's current implementation of RSA. The attack can extract full 4096-bit RSA decryption keys from laptop computers (of various models), within an hour, using the sound generated by the computer during the decryption of some chosen ciphertexts.

So the seriousness of a side-channel attack is determined as a function of the impact of the exploit as well as how easy it is to carry out the exploit.

Meltdown in particular is nasty because it is relatively easy exploit and is undetectable and affects a ridiculously large range of hardware. So it is actually a pretty big deal. Yes there are side-channel attacks against Intel CPU's, but this isn't just any old side-channel attack.

I'm not misunderstanding anything. I'm simply of the opinion that secure multiuser systems aren't something we can do right now.

Here's just one of the more practical attacks http://palms.ee.princeton.edu/system/files/SP_vfinal.pdf

>but this isn't just any old side-channel attack.

It isn't, but you were already screwed. Now you're just slightly more screwed.

There’s a $250 billion cloud computing market that makes that assumption.

What about it? There's a huge market for cigarettes too.

And this is terrifying for me. I am giving away money so that people smarter than myself figure out how to secure my systems.

There is no such thing as practical form of these exploits. We have yet to see a real world demonstration even though everybody keeps saying how theoretically easy it is.

Poc in the url means proof of concept. Not very Real world at all.

Not some artificial poc that already knows an address to attack and needs to be helped by continually pulling the data into the L1 cache.

If it is so easy to do then why has nobody written anything that can read a password from a browser or sudo?

A proof of concept is a real world demonstration. That's all it is.

You seem to think that exploit devs should work for free, why is that?

I have seen the way Intel takes care of its important customers first hand. They are as pro as it gets. You won't find anyone better.

I’m sure that’s true, and it’s great if you’re the army or Amazon.

I’ve read Andy Grove’s account of the thinking behind the response to the Pentium math bug. I expect better from Intel. As a customer somewhere between an individual PC builder and Amazon web services, I don’t think they handled this incident well at all.

EPYC is already economically viable. If it's not seen as viable, that's for business rather than technical reasons. It has more cores, better IO, and after these patches it's no longer clear that Intel has simply better performance per core.

Just a random rant: Our data team onprem infra is based on EPYC and boy it's wonderful. We've just placed an order for another 200 EPYC-based servers for our onprem setup that's set to arrive in Q1 but folks already can't wait.

AMD people should be proud, as customers we're really happy. I hope that GCP would have EPYC-based platform at some point too.

Where do you source EPYC servers?

Right now your favorite Supermicro seller is the best bet. Right now.

In two weeks, it's Dell. R6415, R7415, R7425.

What chx said, but in our case we have custom racks with higher density of servers due to spacing constraints so it's not standard Supermicros.

> I'm skeptical that Intel really has that much of a problem, and the investors in the stock market seem to agree.

It's way, way, way too soon to judge the long-term implications of Meltdown and Spectre on Intel. If their clients want to switch, it'll take months and years to do that. That doesn't mean they won't do it, but it means we won't really know the full extent for a while.

The stock price is a really crude metric. For judging the long-term implications of this, we can't look at how the stock has performed in the last two weeks alone and extract any meaningful information.

My prediction: nobody big is going to switch away from Intel entirely, but they will start to prioritize investments in technology built on its competitors, as a way to hedge their future risk. That's definitely bad for Intel, because over time, it'll reduce their lock-in.

Intel stock is down ~10% and AMD up ~10% since end of December. I'm not even sure why people are saying the stock market thinks it's not a problem. The stock market represents a weighted average of possible futures and it has priced in some damage. Is the idea that Intel should instantly drop 90%?

As I write this Intel stock is around $44, up from $41 last October, and that $41 is at an almost 18-year high, it hasn't been this high since the Dot-com bubble. Look at the longer term graph[1]. Similar story for AMD[2] except it's at an all time high since 2007-ish.

The reason people are saying Intel stock isn't down is because of that, when what HN considers the worst thing ever is indistinguishable from fluctuations for the last 3 months the market doesn't think it's a big deal.

1. https://www.google.com/search?q=intel+stock

2. https://www.google.com/search?q=amd+stock

This didn’t become public last October.

Well yes, that's the point.

The point is that it's up from an arbitrary historical value? It's down since the story broke - that's the point.

It's down from 47 to 45, and it's almost impossible to attribute this to these bugs with any reliability. Yes, there was some initial panic in the first days after the bug. You can see this in the volume the day after it was announced. But things have largely recovered since the low midway through this month.

Of course, the stock market by no means knows everything. But the aggregate prediction of traders is that this doesn't matter very much to the bottom line, and I tend to agree.

To be fair AMD stock has a a fairly volatile history. I have traded it over the last fifteen years and my only thought is: Wall Street has a vendetta against AMD.

For example a while back amd announced in an earnings call: we are in the black and reduced out debt subtantially. The stock tanks by 15-20 percent

They just means they were expecting a bigger profit.

The analogy I would use is that investors may know something is going to happen, but there is a lag before it really happens (less market share), and the stock price decline lags that. In school they teach you that a stock tracks the future predictions but it doesn't quite work like that. For big, widely held things, the stock often lags at predicting future troubles until the last group of investors realize that the trouble exists. That also explains why when stocks fall they often fall quickly rather than gradually.

Disclaimer: I know relatively little about differences between processor architectures, so this might be totally wrong because Intel might just have them over a barrel on this.

I think the major difference between this and, say, the Equifax blowup, is that Intel's institutional clients are affected by this.

I'm not sure what they're thinking internally, but it stands to reason that they're probably a bit upset at least: Their CapEx just went up to maintain the same level of computing power. I'd be surprised if internally Google is buying the "AMD is just as affected" line that Intel's been throwing out.

So, I wouldn't be surprised if they're at least evaluating AMD.

Or, again, maybe Intel just totally has them over a barrel and transitioning isn't feasible at all. It certainly doesn't paint a great picture of Intel's future if AMD does catch up, though.

I think the big institutional clients are always evaluating AMD, and even more exotic options. But I don't think this bug has materially affected the equation here. Savvy buyers understand that there will occasionally be bugs, and this one has already been mitigated. How Intel verbally responds to the issue matters much less than their actions, because the words do not affect the costs of using their products, and the actions do.

I am deeply skeptical of the commentary on Intel's attitude and press releases. I really doubt that matters much to most buyers.

I think Intel has two problems:

First, ARM is doing to Intel what Intel did to the Unix workstation vendors in the 80’s.

Second, given that they’re being cornered into the server business, they need to have products that are rock solid there until they can regroup. This is one of a long parade of recent screwups with their big bets in this space:

(1) A while back, all their server atom chips (tons of crypto and I/O with piles of ECC DRAM and cores for < $1000 and < 20W) had a bug where they stopped booting af 18 months of uptime. These compete exactly in the space server-ARM has a chance, so many affected vendors were already dual sourcing.

(2) NVIDIA crushes them for AI, and Intel is a distant third for graphics in general

(3) Samsung SSDs generally trounce Intel ones.

(4) They’re rapidly losing client device share. Their big recent innovation there is AMT, which is increasingly considered an anti-feature.

That leaves conventional IT compute, (web services, DBMS, etc) for their core business, but even on-prem stuff is moving to private cloud, which needs multi tenancy, and they’re looking pretty risk for that use case too (vs AMD?)

They’ll certainly be around for a long time, but it’s not clear how long they’ll keep their “no one gets fired for buying IBM”-level of dominance.

> (3) Samsung SSDs generally trounce Intel ones.

This is true only for the consumer SSD market, where Intel outsources large portions of the product development. It's also a market that Intel may abandon completely in the next few years as Intel and Micron start to pursue separate flash memory development. If Intel doesn't score a solid win with a consumer SSD in the next two generations, it would be reasonable for them to pull out and focus solely on enterprise SSDs, where they have no trouble winning.

> this one has already been mitigated

Correct me if I'm wrong: It's been mitigated by applying a patch that has fairly severe performance implications, no? How does this not affect the institutional clients' bottom line in that case?

> fairly severe performance implications

1 - 30% impact range for best / worst case. On the kernel mitigations. So really workload dependant.

There's the 15-20% hit for going to database. Most servers out there do nothing but CRUD to some DB somewhere because most businesses don't do HPC. It's definitely an issue for them.

Then there are companies like Epic Games who reported horrific numbers. It seems if you do lots of simple communications (eg websockets or UDP), you can expect a huge slowdown.

https://www.epicgames.com/fortnite/forums/news/announcements...

>Or, again, maybe Intel just totally has them over a barrel and transitioning isn't feasible at all.

That was my hypothesis for why their stock didn't drop much. The problem is very bad but intel's quasi-monopoly and the very high switching costs involved will let them weather it.

Not only will they probably weather it (short term), it will probably result in a reasonable sales bump as affected datacenters replace capacity lost to the various patches and mitigations and also perhaps accelerate replacement schedules when hardware that has designed-in mitigations becomes available.

>I'm not sure what they're thinking internally, but it stands to reason that they're probably a bit upset at least: Their CapEx just went up to maintain the same level of computing power. I'd be surprised if internally Google is buying the "AMD is just as affected" line that Intel's been throwing out.

They don't have to buy it. Whether affected or not, AMD is a non starter at this moment for those things.

Semi-related question: Why is that? I've been looking for resources on why AMD is a non-starter, but most searches just turn up comparisons for low-level consumers.

AFAIK, it’s the ecosystem and availability. Let’s say you’re a Dell shop and you’re using mostly PowerEdge R730s (one of their middle-of-the-road 2-socket options). Your choice right now is Xeon E5-2600 v4 family, because that’s what that platform supports. These aren’t motherboards that you just pick up at Fry’s, they’re designed as an entire system, which takes a while to get going. Dell is not going to invest the R&D money unless they have demand, but there won’t be much demand without stories of success in the field, so we’re at a bit of a chicken and egg problem right now.

Speaking of Dell, they are launching some EPYC stuff: https://blog.dellemc.com/en-us/poweredge-servers-amd-epyc-pr...

But again, getting the ball rolling might take a couple of years. Look at what happened with Opteron as an example.

Pundits and editorial staff sensationalize. Investors collectively behave like a "drunken psycho". Neither are of themselves bellwethers of a company's future success.

We might instead point to facts. Major tech companies are getting into chip design. Apple's foray into fabless last year almost destroyed the value of Imagine and Dialog shares. If they and other companies are successful, we can see the same happen to Intel.

The article you’re mocking uses that very fact, and many more, to make its case:

The company has no competitor in server chips at the moment, but this episode could change that. Microsoft and Google have publicly praised Qualcomm Inc.’s first server chip, which went on sale in November, and Apple, Google, Microsoft, Amazon, and Facebook all have internal divisions working on chip designs.

While I don't underestimate the scope and possible ramifications, I still don't understand the immediate threat to the average person. The malicious hacker that is after secret personal information does not need to go to the trouble of writing a trojan that exploits Spectre or Meltdown when a simple keylogger and file scanner would be much simpler and probably more effective.

The article itself states: "So far, Meltdown and Spectre probably pose less risk to the average person than, say, a simple phishing attack in which a hacker tries to send you to a malicious website. They won’t lead to the kind of widespread panic that resulted from the 2017 hack of Equifax’s customer database.

But that could change. Hackers who hadn’t tried to break into Intel’s hardware, believing there was no way it would leave a side door open, are now seeking ways in."

"But that could change" is a vague term that doesn't mean much to me. Again - not trivialising this nor saying Intel shouldn't do some soul searching, but I'd like to better understand the justification for the apparent hysteria - unless, of course some people more experienced in security would care to explain what it is I'm missing here.

P.S. One thing I wanted to check was whether Spectre/Meltdown breaches could somehow be caused by manipulating a web browser. Some searching revealed that this is indeed a possibility so at this point, everyone feel free to panic :-)

Agree.

Intel has as much of a problem as VW had after diesel-gate: none. Same will be valid for Apple's throttling scandal.

Big corps like this may experience some little storms here and there, but there is no iceberg big enough for them.

Articles like this exist just for the sake of writing something and making some money.

It's not even close to being over -- https://skyfallattack.com/

There seems to be consensus that it's a hoax.

Is there an embargo? Is it a silly play on the James Bond reference that Spectre represented? Either way pretty funny since you can't know for sure after this fiasco.

Botched micro code updates, Intel engineers arguing with each other on the linux kernel mailing list, etc. Intel's best and brightest have had 6+ months to work on proper mitigations in secret and this is the result.

https://marc.info/?l=linux-kernel&m=151559244214217&w=2 https://marc.info/?l=linux-kernel&m=151559367514704&w=2

Silly play is what I heard as most likely.

Yeah, handling of mitigations for Spectre has been awful during the embargo period, but I am very happy with the result we're getting now. It's taking less than three weeks to get everything sorted out.

Does the embargo have a hard deadline?

> the investors in the stock market seem to agree

It's down several percent since the announcement in a rising market.

More generally they have underperformed the SP500 over the past 2 years and AMD in particular is blowing them away.

Yes they have a problem. The current CEO seems more interested in politics than technology. http://www.breitbart.com/big-government/2015/09/10/intel-cut... (inb4 I don't like Breitbart)

Using the current stock market reaction as a bellwether for anything but short term (1-6 months) outlook is a bit silly.

In the longer term (1-3 years) Intel has a very large problem. Especially as they are being eaten alive in non-desktop class cpus right now.

These issues may not be a knockout punch, or even have them on the ropes, but it made them stumble and they look vulnerable.

Regarding the stock price and business revenue, it might be naive and certainly depends on if lawsuits go forward, but these bugs could prompt customers to upgrade their chips when the next chip revisions are out, increasing sales over pre-disclosure expectations.

I remember the selling on insider information claims, and was skeptical that it would matter that much.

How do you factor into your thesis the possibility of a multinational class-action lawsuit?