News for hackers. Stuff that matters.
Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by rbanffy on Wednesday January 03, 2018 @01:10PM (#16063384)
It's a good thing CPU is fairly compressible. Unless you meter it very carefully, you'll see the performance hit and it'll not impact you that much. Very few of my physical boxes are over 70% CPU utilization on a daily average.

It's, however, really bad if you sell CPU cycles for a living. You just lost between 5 and 30% of your capacity. If you have a large building, you just lost part of your parking lot to the Intel Kernel Page Problem building.

Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by devcpp on Wednesday January 03, 2018 @01:37PM (#16063676)
Problem is, most companies that need a lot of power only care about one thing - peak performance. And they tune it carefully in order to not overspend while guaranteeing minimal downtime. This means that they'll have to pretty much scale their infrastructure up by exactly 30%. That's a LOT for these big clients.

Honestly, I'd just make sure the server firewalls are super tight and not take in the future patches. At least for now.

Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by ajross on Wednesday January 03, 2018 @01:47PM (#16063791)
Very very few highly tuned "peak performance" workloads are dominated by syscall overhead like the test that produced that 30% number was. It's best to hold off on the hyperbole.
Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by dijit on Wednesday January 03, 2018 @04:18PM (#16065255)
I have a power dependent workload that scales horizontally and is currently already dominated by the cost of system calls. This will effectively, directly cause me to buy 30% more compute on a huge infrastructure. (2,000~physical machines. Quite beefy dual socket machines with a lot of memory)

I know I’m not alone.

Then again. Think of microservices, Kubernetes for instance; Network requests are system calls.

Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by rbanffy on Wednesday January 03, 2018 @04:41PM (#16065473)
If your workload has no code that's untrusted, you can safely skip this patch or disable it on boot. If not, at 2000+ physical machines, it may be worth to move some of that into kernel modules that would collapse a couple syscalls into a single higher level one.
Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by ufo on Wednesday January 03, 2018 @09:20PM (#16067460)
The VM host will still have the patch applied, won't it?
Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by rbanffy on Thursday January 04, 2018 @05:26AM (#16069229)
Yes, but if it's your metal, you don't need to.
Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by dijit on Thursday January 04, 2018 @04:11AM (#16068946)
Good idea. But it’s a windows executable and depends quite heavily on windows specifics.

(In my case anyway)

30% overhead might be inscentive to revisit the assumption we can’t rewrite it for Linux.

Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by rbanffy on Thursday January 04, 2018 @05:29AM (#16069240)
You still can move some functionality to a device driver or something else that runs in the NT kernel space.
Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by rogerthis on Wednesday January 03, 2018 @08:06PM (#16067095)
But then you have to release the module as GPL, no?
Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by lmm on Thursday January 04, 2018 @07:01AM (#16069685)
Only if you want to do something that would otherwise be a violation of copyright - e.g. distribute the module to other people (assuming it's sufficiently entangled with linux to be a derivative work therof). The GPL only licenses you to do things that you otherwise couldn't do, it doesn't restrict you from doing things that were never a violation of copyright (e.g. privately modifying your own things).
Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by KingMachiavelli on Wednesday January 03, 2018 @09:44PM (#16067575)
No kernel modules can be closed source; GPU drivers are a common example of a closed source kernel module.
Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by maxander on Wednesday January 03, 2018 @05:43PM (#16066082)
Will you be buying Intel-based machines? Or will you be running a hybrid-architecture cluster now?

I don’t know very much about computing on that scale, but I wonder if all the people selling off Intel stock are thinking this story through.

Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by lev99 on Wednesday January 03, 2018 @06:39PM (#16066585)
AMD server CPUs currently outperform Intel on some multi-threaded benchmarks. This usually isn't a problem for people buying for peak-performance because you can always buy more CPUs to increase parallel programming speeds, but it's harder to make single threads faster.

It's possible that the patches applied to fix this bug will cause some single-threaded benchmarks to change from Intel being the fastest to AMD being the fastest.

Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by myrandomcomment on Wednesday January 03, 2018 @09:54PM (#16067642)
So for what it is worth my company has all Intel kit. We run servers that run docker. In each docker container we do build / test for our product. That is all we use them for. 1RU with 2 blades, each blade is dual socket, 72 total cores, 512GB RAM. We will not apply this patch as none of this is public facing and we do not want the hit to build / test throughput. The one big thing that this has done is we were looking at AMD for new servers and that has now become a higher priority on the to do list. Given our environment we care about the number of containers we can run, period.
Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by dijit on Thursday January 04, 2018 @04:13AM (#16068954)
It is overwhelmingly likely that we’ll buy more intel. Power/Watt has always been superior and AMD has to prove itself over time before we’d buy it.

Not trying to kill expectations. This decision isn’t mine alone. You know the old saying “nobody got fired for buying Cisco” that applies to Intel too.

Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by bpchaps on Wednesday January 03, 2018 @03:02PM (#16064515)
No, but lots of workloads are built with latency in mind. For APIs that talk to each other in long serial chains, don't be surprised if request responses take significantly longer in many, many workflows.
Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by brazzledazzle on Wednesday January 03, 2018 @03:01PM (#16064500)
I wouldn’t isolate your concern to firewalls and bad actors that break in over SSH. If they manage to find a vulnerability in your app that allows remote code execution this could help them make that problem much worse. Also VM/container escapes are a big problem if you use a cloud provider.
Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by redcalx on Wednesday January 03, 2018 @03:20PM (#16064680)
> I'd just make sure the server firewalls are super tight and not take in the future patches. At least for now.

Good security is about layers. No one layer can be assumed to be watertight, but with enough layers you hopefully get to a good place.

Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by viraptor on Wednesday January 03, 2018 @02:19PM (#16064123)
If they really care about peak performance, I don't believe the PTI patch will affect them. If you can change your system in a way that the power-hungry part does not work on untrusted data, you can not with "nopti" and ignore it. Systems which both need lots of maxed-out CPUs and traffic directly from wild internet are pretty rare. They're unlikely to run on a virtualised systems either.
Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by btilly on Wednesday January 03, 2018 @02:39PM (#16064313)
Systems which both need lots of maxed-out CPUs and traffic directly from wild internet are pretty rare.

That's a good description of basically every cloud environment out there, from AWS on down.

In other words they are extremely common.

Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by rbanffy on Wednesday January 03, 2018 @03:02PM (#16064514)
There are many ways to tune such workloads and I suspect our software will get better as a result.

We'll start to get conscious about the number of syscalls we use on each operation, start using large buffers, start buffering stuff user-side...

Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by viraptor on Wednesday January 03, 2018 @03:40PM (#16064875)
The CPUs in cloud environments are not maxed out in general. There will be some area like batch processing and compute-specific VMs. For other cases, there's quite a bit of overcommitting of resources. And that's before you start doing scheduling that mixed workloads on a physical host for better utilisation. Source: worked on a public cloud environment.
Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by btilly on Wednesday January 03, 2018 @05:17PM (#16065816)
I agree with you on most VMs. But once you schedule mixed workloads, you want each host to be balanced so that all of its capacity is utilized evenly. Which means that if CPU use increases across the fleet, you will want new hardware with more CPU.
Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by rbanffy on Thursday January 04, 2018 @05:31AM (#16069246)
Either that, or you'll have to put up with some processes taking longer.
Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by steve_gh on Wednesday January 03, 2018 @11:12PM (#16067983)
About 10 years ago I was mentored by a guy who was an utter wizard at queuing theory, and who bugfixed a whole bunch of nasty issues in cellular telecoms hardware through his understanding of how queuing theory impacted code execution.

TL:DR - queue behaviour gets nonlinear as you approach the theoretical max load. If you are running your processors at a high load, even a small change in code throughput makes a huge difference to real world behaviour.

Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by azeirah on Thursday January 04, 2018 @06:37PM (#16075131)
I believe I saw a strangeloop talk about this specific issue in Clojure. The talk giver was talking about channels, not queues though.
Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by gleenn on Sunday January 07, 2018 @03:59AM (#16089970)
Do you have a link?
Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by topspin on Wednesday January 03, 2018 @01:59PM (#16063932)
Hmm. Since everyone that sells (Intel) CPU cycles for a living suffers the same loss of supply this boils down to pricing; the same demand chasing fewer cycles will drive up prices and the market will adapt.

30% is a big hit. I'm wondering if that isn't a bit exaggerated, or perhaps the consequence of a poorly optimized workarounds that will rapidly improve. I recall seeing figures on the order of 3% only a few days ago.

Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by btilly on Wednesday January 03, 2018 @02:41PM (#16064330)
30% is a worst case for a workload optimized to hit the performance bug as hard as possible.

How big it will be for your workload is a function of what your workload is. Benchmark if it is important to you.

Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by earenndil on Thursday January 04, 2018 @12:48AM (#16068333)
50%, rather, is the worst case scenario. 30% is a bad case scenario, and 5% best-case scenario. Which is still a lot for large cloud providers like amazon, google, microsoft.
Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by dx034 on Thursday January 04, 2018 @07:13AM (#16069751)
Any public information on how someone like Google or Facebook handle this? Do they have enough spare capacity to patch or will they need to build further capacity first? I could imagine 10% of Google's capacity (internal services, not Google Cloud) is at least a large datacentre.
Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by earenndil on Friday January 05, 2018 @08:29PM (#16083272)
I know someone who works for amazon and he said that they didn't need to do anything or buy any more servers.
Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by ktta on Wednesday January 03, 2018 @01:59PM (#16063925)
>It's, however, really bad if you sell CPU cycles for a living.

Who really sells CPU cycles? Cloud providers sell instances priced per core. So the real hit is by the customers since they have to shell out for more instances for the same amount of computing power.

The hit I see is by providers of 'serverless' computing, since they charge per request and have their margins reduced.

Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by zedpm on Wednesday January 03, 2018 @02:43PM (#16064359)
> The hit I see is by providers of 'serverless' computing, since they charge per request and have their margins reduced.

AWS, Azure, and GCP all bill serverless with a combination of per-request fees and compute (GB-seconds), so I'd expect the entire hit to be passed on to the user since this will cause increased compute time for each request. N requests that used to average 300ms each will now be N requests that average, say, 400ms, so the per-request billing remains the same and the compute billing will increase by approximately 30%.

Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by ktta on Wednesday January 03, 2018 @05:39PM (#16066049)
I don't understand what exactly you're saying. All of those services have serverless services, but they also have server based instances which abstract compute to amount of cores and RAM rather than CPU cycles. And most use is out of the services which aren't serverless.
Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by TotallyHuman on Thursday January 04, 2018 @03:13PM (#16073654)
Then it's because you don't know how modern cloud works.
Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by ktta on Thursday January 04, 2018 @05:23PM (#16074659)
I now see that my misunderstanding was about who exactly the users were and who the provider was. My opinion of provider was only GCE, AWS, etc. while the commenter I believe when talking about providers included users of those services (who again were providers of serverless services).
Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by user5994461 on Thursday January 04, 2018 @12:20AM (#16068240)
A lot of companies do, including many ycombinator startups. Think of anything that's analytics, data science, data warehouse or advertising related. The costs to run their service just took a hit.
Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by Houshalter on Wednesday January 03, 2018 @02:11PM (#16064028)
Their competitors are also affected.

Also a 30% decrease is also equivalent to setting Moore's law back 7 months. A 5% loss is only setting it back 1 month. I know that's a bit of a naive calculation. But the point is computing power has long operated in an exponential domain. So big differences in absolute numbers aren't necessarily a big deal.

Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by seccess on Wednesday January 03, 2018 @02:17PM (#16064097)
According to this patch comment, AMD x86 chips are not affected: https://lkml.org/lkml/2017/12/27/2
Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by mulmen on Wednesday January 03, 2018 @02:56PM (#16064469)
Sure but who is using AMD chips in place of Intel server chips? If company A competes in the widget market against company B and they both built their server infrastructure on Intel then neither company gained an advantage due to a performance degradation in Intel hardware.
Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by geezerjay on Friday January 05, 2018 @02:09PM (#16080781)
> Sure but who is using AMD chips in place of Intel server chips?

Well... Everyone who bought AMD. Some people managed to see beyond the hype and go for the optoon that made sense.

Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by mulmen on Friday January 05, 2018 @04:01PM (#16081597)
The overwhelming majority of the cloud runs on Intel. Saying AMD is slightly better off doesn't really help if my systems are built on Intel. This is the case for most people.

What hype are you referring to? Are you suggesting the people who bought AMD knew this was a problem for Intel?

Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by pi-rat on Wednesday January 03, 2018 @03:23PM (#16064702)
Azure got some AMD EPYC.
Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by JohnJamesRambo on Wednesday January 03, 2018 @03:24PM (#16064717)
>Sure but who is using AMD chips in place of Intel server chips?

Maybe a lot more now?

Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by jjwiseman on Wednesday January 03, 2018 @03:35PM (#16064823)
CPU speed hasn't followed Moore's law since 2003ish. (Number of transistors is still following Moore's law, but that doesn't necessarily directly help you when your program is suddenly 3-30% slower.)
Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by Houshalter on Wednesday January 03, 2018 @03:55PM (#16065033)
A CPU from 2017 is going to run your programs a hell of a lot faster than one from 2003. Even if they technically have the same clock speed. Look at benchmarks for instance: https://www.cpubenchmark.net/high_end_cpus.html
Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by RobAtticus on Wednesday January 03, 2018 @04:20PM (#16065285)
The claim wasn't "CPUs in 2017 are not faster than CPUs in 2003" or even "CPUs in 2017 are not much faster than CPUs in 2003"; the claim was that they haven't followed Moore's law since 2003, so applying it to CPU speed nowadays is inaccurate. Of course CPUs are faster now than they were 14 years ago, just not as fast as the case where Moore's law still applied to CPU speed.
Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by lttlrck on Wednesday January 03, 2018 @05:33PM (#16065976)
Moore’s Law doesn’t describe CPU clock speed increases.
Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by slededit on Thursday January 04, 2018 @04:35AM (#16069042)
Dennard scaling however did deal with clock speed (indirectly via power). It has failed since about 2005.
Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by bunderbunder on Wednesday January 03, 2018 @04:09PM (#16065172)
I don't think you can reliably phrase this in terms of Moore's law. Moore's law mostly concerns raw FLOPs. It's less useful for predicting hardware performance for operations that are governed by limitations like I/O and memory latency. And this slowdown, if I understand it correctly, is largely driven by memory latency.
Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by adrianratnapala on Wednesday January 03, 2018 @09:59PM (#16067667)
One of the rationales for cloud computing is it saves money by cracking up utilisation. Providers observe how much users "really" use and then provisioning that much.

True, sometimes you will leave boxes at low utilisation for various reasons, e.g. to deal with traffic spikes. But those reasons have not gone away. So now instead of heaving a predictable increase in CPU cost, you have an unpredictable increase in performance snafus.

The only good news is that the real performance hit will be less than 30% on many workloads. Especially once the providers start juggling and optimising.

Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by bogomipz on Wednesday January 03, 2018 @01:40PM (#16063705)
>"It's a good thing CPU is fairly compressible."

What do you mean by "compressible"?

Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by theoh on Wednesday January 03, 2018 @01:47PM (#16063798)
Presumably, for a certain important class of application, CPU is not used "densely", i.e. continually. Instead it's used intermittently, like a gas rather than a solid... Hence compressibility. Such applications are far from being CPU-bound, in other words.
Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by bogomipz on Wednesday January 03, 2018 @01:52PM (#16063848)
So a cloud provider would be an example. Compressible similar to a sparse file I guess as well. Thanks this makes sense.
Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by laythea on Wednesday January 03, 2018 @02:19PM (#16064115)
I think it was meant that a normal application does not utilise the CPU all the time, which can be seen by looking at the task manager CPU usage % = X. Any extra processing needed to fix this bug will have to come out of the remaining 100-X%. This is OK as long as you have enough spare %, and can afford the extra power usage for that processing.
Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by bogomipz on Wednesday January 03, 2018 @02:48PM (#16064400)
That makes sense, thanks. This is a big deal.
Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by rbanffy on Wednesday January 03, 2018 @03:27PM (#16064748)
Virtualization is one popular way to drive up CPU utilization. The more diverse workloads run on a given server, the more even the CPU usage tends to get. This way, if you have 100 workloads that peak at 100% but average at 1%, your CPU usage will tend to be smooth at 100%, any overallocation will smooth out over time (a job that would take 1 second may take up to 10).
Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by asplake on Wednesday January 03, 2018 @01:33PM (#16063629)
No vendor can afford to do it at a loss for long. One way or another the customer will end up paying
Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by exabrial on Wednesday January 03, 2018 @01:43PM (#16063746)
There's also latency though :/ it seems that programs that make a lot of syscalls will be affected more than programs that are doing in-process calculations
Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by rbanffy on Wednesday January 03, 2018 @03:30PM (#16064773)
We'll start being more syscall conscious when we write our programs. We'll batch more at the user mode side and try to use less syscalls to do the job.

Kernel ABIs will eventually reflect that and crop up higher level expensive calls that replace groups of currently cheap syscalls (that will become expensive after the fix).

And Intel will profit handsomely from next generation CPUs that'll get an instant up-to-30% performance boost for fixing this bug.

Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by pkaye on Wednesday January 03, 2018 @05:00PM (#16065666)
What about all the kernel interrupts due to network and storage traffic?
Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by exabrial on Wednesday January 03, 2018 @05:27PM (#16065913)
Maybe the scheduler could dedicate a core to interrupts and software that has small quanta and page tables? Can't really think of a code solution that doesn't sound stupid when I type it.
Re:Intel Confronts Potential ‘PR Nightmare’ With Reported Ch... by 0xTJ on Wednesday January 03, 2018 @08:39PM (#16067263)
Another consideration is power usage in data centers. Server power usage is annoyingly complex, and once you get above 70% utilization power usage may go up considerably.