When you think about your own workstation, it's not a big deal to build an Intel or AMD system. But when you buy 100k motherboards and spend the time adjusting your tooling to those, from packaging to power, to cooling, to support, to OS code, etc. and then you on a whim decide to get another 100k motherboards of a different architecture, you spend a non-trivial amount of time and money to support those as well. Again, if AMD provides better hardware, it's absolutely worth it. But I personally wouldn't do it based on this bug.
I don't own shares of either AMD or Intel.
But that's the problem. Nothing works 100% of the time, which is why monoculture is bad. When there is a bug that affects 20% of your systems, you can continue operating at 80% capacity, which at a reasonable level of reserve/redundancy means you're still entirely up. With a monoculture the bug affects everything and you're entirely down.
> But when you buy 100k motherboards and spend the time adjusting your tooling to those, from packaging to power, to cooling, to support, to OS code, etc. and then you on a whim decide to get another 100k motherboards of a different architecture, you spend a non-trivial amount of time and money to support those as well.
This is why hardware abstraction is a thing.
It's almost always less expensive to support diverse hardware from the beginning than to wait until after the market shifts.
Eventually the day comes to switch from 68K to PowerPC, or PowerPC to Intel, or Intel to ARM, or ARM to whatever else. Because eventually you save/gain a zillion dollars by switching and it "only" costs three quarters of a zillion to switch.
But it would have cost a tenth that much to have supported diverse hardware from the start, and then the transition is only a matter of using more of the now-superior hardware rather than being stuck on the now-inferior hardware for potentially years while everything is rearchitected from scratch.
This is the mistake in your argument. Motherboards, CPUs, RAM chips, GPUs are analog and physical objects. For AWS to switch a DC from one mobo to another just to find out that this one draws 5% more power and their standard backup generator can't handle it, which now starts a chain reaction of upgrades is going to incur real world costs. Costs that can't be amortized by writing some code to make the motherboards look the same.
This is basically the A/B testing/one-armed bandit problem. How much do you spend time exploring alternatives vs how much time to you reap the benefits of the fact that all your hardware is exactly the same and best of breed, as based on your testing?
> When there is a bug that affects 20% of your systems, you can continue operating at 80% capacity, which at a reasonable level of reserve/redundancy means you're still entirely up. With a monoculture the bug affects everything and you're entirely down.
These situations simply don't lose enough money to make up for the gains of a monoculture.
Think about it this way. You probably run or at least know someone who runs SaaS products. Do you/they use five different cloud providers in equal measure to make sure you have diversity if one has an issue? Do you/they use five different software stacks in case there is a remote exploit for RoR and PHP holds things up? Do you/they buy groceries at five different grocery stores in case one of them has an e. coli outbreak that the other four don't? The answer to all that is now, because no matter how you try to abstract these things, there is a meaningful difference between PHP vs Django vs RoR vs Express vs .NET and between AWS and GC and Azure, that it would cost you a lot more, not just in billing but in engineering effort to support.
Another example: chances are you've at some point built a RAID array. Did you put different size and performance drives from different manufacturers into it or did you buy N of the same drive type to ensure even performance? If so, why?
Put another way, how much more are you willing to pay on your AWS bill to ensure they are running a mix of ARM, AMD, PowerPC, and Intel chips? Because my guess is that it won't be in the range of 1-2%.
Being physical isn't different. The data center is designed to allow systems that consume up to, for example, 500W. When one consumes 400W and another consumes 420W, they're still fungible. A system that consumes 525W can't be used, but you know that so you don't use those.
> This is basically the A/B testing/one-armed bandit problem. How much do you spend time exploring alternatives vs how much time to you reap the benefits of the fact that all your hardware is exactly the same and best of breed, as based on your testing?
That isn't the relevant problem. Even if you choose monoculture, you still have to pay the cost of weighing your alternatives to decide which single model to use.
The cost of diversity is that the second best model on some metric is 20% worse than the best. But that is also the advantage, because on some other metric it's 20% better. You can use each model for its strength. And since you can't perfectly predict the future, when something unexpected happens you're better able to handle it, because for any given thing that only some systems can do, you will have some systems that can do it.
> These situations simply don't lose enough money to make up for the gains of a monoculture.
Beware survivorship bias. It's easier to find an active monoculture company that has never had a major problem than one that has, because having a major problem in a monoculture often results in bankruptcy.
> Do you/they use five different cloud providers in equal measure to make sure you have diversity if one has an issue?
For services with high availability requirements, people absolutely do that.
> Do you/they use five different software stacks in case there is a remote exploit for RoR and PHP holds things up?
That wouldn't reduce attack surface. The relevant thing people do is to use two factor authentication.
> Do you/they buy groceries at five different grocery stores in case one of them has an e. coli outbreak that the other four don't?
Having multiple local grocery stores is a thing people want. And people do actually use them, because different stores have the best price or quality for different products.
> chances are you've at some point built a RAID array. Did you put different size and performance drives from different manufacturers into it or did you buy N of the same drive type to ensure even performance? If so, why?
These are spec differences, not supplier differences. There is no issue with using drives of the same size and speed from different manufacturers.
Also compare ZFS, which allows you to efficiently use unmatched drives for the same filesystem.
When you're scaling up/maintaining your DC, you're much more likely to be looking for single sku, like for like products that allow similar tooling, knowledge base, experience etc... Like you said, monoculture has its benefits in some situations.
Personally even with this bug I'd be very hesitant to switch. Our previous tests between them for very specific workloads showed our best cost/performance was with Intel over successive generations, and the scaling/tip over points were different. We have a combination of experience and knowledge around the existing arch and how our applications and workloads interact with it that involved a number of pain points that I'm not sure its worth it to re-experience with another arch.
On the other hand, those running on non-bare metal, cloud based, auto-scaling/automated solutions that have a wider tolerance for individual app performance, are probably in a situation where they care less about this, but at the same time have little to no say in the arch they run on, that decision is left to the cloud providers they use.
just my 2 cents anyways.
The reason every server isn't POWER is: ecosystem. For any random company, switching archs for anything less than a multiple factor gain is a daunting multi-generation proposition. For a hyperscaler like Google the bar is a lot lower but you need a compliant vendor that will do a lot of the long haul platform work. IBM's been trying to establish that for many years and is just about to pull it off. Supply chain is also important, hyperscalers have come to expect buying and building systems a certain way and IBM will now just sell chips or even the IP for you to fab yourself. And of course the total cost calculus: capex, and opex in the form of TDP, support burden.
Google _will_ be using P9 for GPU servers internally. The inflection point for them was I/O and memory bandwidth. So, paradigm shift was what was needed to turn a juggernaut.. and that is what adding a bunch of accelerators to your platform is. Intel has no good solution there.
I believe POWER9 has the ability to be either big or little endian as well, so that helps for compatibility issues, and it's just a matter of whether your application can compile.
Google is pushing both PowerCPU development as well as ARM. They seem to be able to sort for this just fine. You can write tools to sort the differences. You cannot write tools to fix major HW issue.
Anyway my 2 cents based on experience and history for whatever the comments of a random person on the intertubes is worth.
At the same time AMD also has a golden opportunity to for some PR and marketing.
I'm not a believer in stock price as good indicator of anything, sorry to skip that part.
Intel's value is 99% engineering + manufacturing ability + customer relations. It would be a poor CEO indeed who'd direct their IT to start buying AMD because of this alone.
Let's be clear: It's both.
Major system vendors are now offering to apply bootleg removal situations at the factory on customer request[1]. That request is not free. People are willing to /pay extra/ for no-IME laptops.
Either Intels marketing and public relations department are asleep at the wheel, or they've gone to the top to request a friendly switch to disable this and been told by the legal department that they can't have one.
[1]https://liliputing.com/2017/12/dell-also-sells-laptops-intel...
I don't like the fact that you can't disable ME, that it's not open source, and that it's vulnerable any more than anyone else. But this does seem like hyperbole much more than fact.
95% speculation. The last 5% comes from exercising basic pattern recognition.
I remind you that we're probably talking about interference from the organization that arranged this:
https://arstechnica.com/tech-policy/2014/05/photos-of-an-nsa...
The existence of that program was pure speculation, until it turned out to be totally real.
>(b) fails Hanlon's razor.
This is completely irrelevant to any argument made between two informed participants. It's worse than speculation, it's a plea to glib colloquialisms. Any chance you've got evidence or even reasoned speculation supporting the theory that the worlds most successful CPU manufacturer has an incompetent marketing department?
> 95% speculation. The last 5% comes from exercising basic pattern recognition.
No, it's all speculation because pattern recognition is not evidence, as applied here. Like, is it possible that I am an NSA agent trying to persuade you that you are safe and shouldn't worry about ME? Of course it's possible. But do you have any evidence of that? No.
"Well, in the past the NSA has asked big companies for backdoors into their products" is a true statement with evidence. "That implies that in this case there is a 5% chance that is exactly what's happening" is 100% speculation because again there is no evidence. If you can find any, I am all ears because honestly I am not a fan of Intel, Intel ME, the NSA, government spying, big corporations taking advantage of consumers, or a number of other things I imagine you and I agree on. But I think I am being rational when I say that chances are this is a stupid bug or number of bugs, plus bad old school thinking on the part of the management team, and not a deliberate NSA feature.
Here is my bit of speculation: if the NSA asked Intel to include a backdoor, wouldn't they both have done a better job of creating it? Why introduce a bug when you can include whatever code you want in a closed source firmware? You can literally add any kind of C&C mechanism you want because nobody can see what you are doing and nobody would ever know. Is the NSA that stupid to to ask for a bug that can be found and exploited? Is Intel not able to offer a better technical solution? Wouldn't it be to both of their benefits to do this right from the start? Also, why only approach Intel and not AMD? AMD is not as popular but surely has enough market share to warrant spying on.
Why aren't you viewing the possibility of intelligence agencies ordering the Intel ME as one of these future historical facts? If the proof for that became known today, both the agency and Intel would scramble to introduce a better backdoor in the next generation CPUs / MBs and devise a marketing campaign to make it sound good -- and to bash their former selves for "making a mistake" while simply thinking "OK, we're gonna cover it up much better this time and we're gonna twist it in such a way that people would flock to buy it". It's what marketing and spies do; they twist facts. Why is that so non-legit for you?
Furthermore, you're asking why didn't they do a better job if it was a conspiracy. People in closed circles aren't exposed to public criticism and their thinking is affected in the process. They usually think "meh, good enough, nobody will ever find it anyway". They are humans like you and I and are susceptible to bad days or negligence due to being tired. Furthermore, it's very likely they were under pressure to make it work quickly so they took shortcuts. What makes you think the programmers of the intelligence agencies have godlike powers over their (very likely) military superiors? Answer is, they don't. Programmers have no executive powers and their counsel is usually met with skepticism if it doesn't fit the management's agenda.
When talking about intelligence, our best bet is to do educated guesses. If we had hard facts we would be targets. As mentioned in another reply of mine directed at you -- it's their job to hide the facts. So you requesting proof of these matters is basically refuting all possibility of intelligence agency commission of the Intel ME on the grounds of "hey, you are not the next Edward Snowden so your arguments are invalid".
Meh. You come across as a guy who basically says "my speculation is better than yours". Not constructive.
Your theory in the above comment is that the NSA or equivalent ordered Intel to build a C&C mechanism into their processors. Intel then did a perfect job covering up this request, but did a piss poor job of implementing it due to incompetence and has not managed to correct it for 10 years. There is no indication that this might be the case but because of other unsavory activities by the NSA or equivalent it can be assumed that at some point evidence will be uncovered that you are right and therefore we should accept it as fact. Do I have that right?
Judging by other activities of the intelligence agencies and working with pure speculation -- not hiding from these words, you are correct by calling it that -- I still think it's much more likely they commissioned the Intel ME.
You mention critical thinking in another comment. Critical thinking, the way I apply it, also requires a historical context to be applied to the situation one is analyzing. Agencies have been doing pretty shady stuff and some of it has been uncovered for the entire world to see.
Critical thinking, the way I apply it, says that the odds are there is a foul play. I merely wish you to recognize that this is the more likely scenario than a bunch of coincidences and/or people supposedly making the ME to serve data center sysadmins -- btw many of those sysadmins, including on several threads here in HN, said they never used the ME and named a plethora of other tools.
Obviously I am not trying to change the way you think in general. I believe we can both agree that none of us knows for sure. The human brain's strength is to work with many variables and be able to impose some order in the chaos by pattern recognition and using historical info. I am not gonna deny this can lead to people drawing awfully misguided conclusions sometimes -- and I've been guilty of that as well! -- but it's the best we have, especially having in mind what tiny imperfect brains we have to work with.
Everything I can name are circumstantial evidence. I accept that. It's the nature of the area. Intelligence data isn't easy to come by.
And I am saying that the confidence interval on that calculation is just orders of magnitude not tight enough. I am not denying that you could be right. It's just that I am giving that possibility something like a 1% chance of being true, while something like 85% chance of this being pure incompetence by Intel management and engineers (the rest being some other explanation that's neither malice nor direct incompetence). I don't think you and I can find a common ground on this estimation.
Again though, ME is a bad thing because it's not open source, it can't be turned of, and it's buggy. Regardless of who ordered its creation, it sucks.
You're also saying, implicitly, that therefore we must default to assuming it is incompetence.
That link isn't a given. Stating that it is incompetence is also speculation, not some kind of universal backup truth.
However, when it comes to that last 5%, I assert that the historical data does not back a claim that Intel's marketing department is incompetent.
Until it's financially worth their while, why would they spend money on it?
You request a proof that's impossible to procure. Are you now gonna claim the lack of this proof supports your thesis?
1. Intelligence agencies have been known to force companies to give them access to their products.
2. Companies have been known to comply, if reluctantly, at least until a whistleblower exposes the program.
3. Intel ME was developed as an on-chip version of an external card that is actually useful.
4. Intel has made poorly engineered products before.
5. Intel isn't in a habit of open sourcing firmware.
6. From a technical standpoint, Intel is fully capable of creating a system that doesn't allow C&C through a bug and an exploit.
7. AMD, the second largest computer chip maker does not have a matching system that can't be disabled and that has similar bugs.
Based on this, I'd say it's possible that the NSA (or equivalent) asked Intel to develop ME and add a bug to allow C&C, but very unlikely.
It's also possible that the NSA (or equivalent) asked Intel to develop ME and add C&C and Intel did it through a deliberate bug, but very unlikely.
It's also possible that Intel tried to develop a feature the market might want, and screwed up the implementation. This seems to me to be very likely. It's the simplest explanation (Occam's razor) and it requires only incompetence, not malice (Hanlan's razor), so it's sort of by default most likely.
If someone can produce an iota of evidence to the contrary I will change my allocation of probabilities appropriately, but so far the evidence is "it could have been done" and "they've been known to spy on people in the past". In my book that's not a strong enough argument.
I would say that too if I'd be waiting for everyone to sell so then I could buy INTC :-)