Intel might find that these benchmarks achieve the opposite of what they intended. I had no idea that viable ARMv8 alternatives to Intel's architecture were on the market. There are likely others who were similarly ignorant. Intel has just informed all of us that these options are out there.

Although given the relative performance, I can't imagine who'd pick the arm option currently, unless the pricing difference is truly extreme. There's just too many downsides to arm servers at this time: it's not just total overall performance, but particuarly single threaded performance (which affects e.g. latency that your readers experience), power usage (affects costs), and of course availability of software.

It's getting close, that's for sure - and there may well be some niche where it already makes sense, but I wouldn't know which one.

Probably the best value proposition is that the gap is likely to shrink, and investing now may help you prepare for a future switch when it makes more sense.

> It's getting close, that's for sure - and there may well be some niche where it already makes sense, but I wouldn't know which one.

Proprietary blobs such as Intel's ME that cannot be removed from Intel Xeon systems pose a security risk. If the ARM systems are found to lack them and they achieve the status of being good enough, security minded individuals and organizations will buy them because of that.

As long as that happens, Cavium will be able to use the revenues to develop the next generation of chips to better compete on traditional metrics like Intel was able to do against the RISC vendors of the past. As a security minded individual myself, I am well aware of that and I look forward to supporting such developments by purchasing such hardware for applications where it can be used.

There are others thinking this too. Some of them are watching the Talos workstation motherboard come to market with that in mind. The people behind it are even talking about IBM's executives being willing to make the POWER9 processor's microcode open source if the POWER8 model succeeds.

https://www.raptorengineering.com/TALOS/prerelease.php

"Proprietary blobs such as Intel's ME that cannot be removed from Intel Xeon systems pose a security risk. If the ARM systems are found to lack them and they achieve the status of being good enough, security minded individuals and organizations will buy them because of that."

Do you expect the above to be anything more than a rounding error of the total server market? I doubt Cavium is going to finance its development from that.

If you want it to be so, then make it so. There's plenty of ammunition to use in all of the leaks.

> ThunderX2 is a 14nm FinFET design, and the company claims it will deliver 2 to 3 times the performance of ThunderX on select workloads with a higher performance to power ratio. Their plan for the X2 is to include 54 ARMv8.2 cores with a target frequency of 2.4 to 2.8 GHz and up to 3 GHz in turbo mode (versus their current 28nm, 48 core, 2.5GHz design), 32MB of what Cavium calls shared Last Level Cache (LLC), 6 DDR4 memory interfaces running at 3200MHz (1 DPC) or 2966MHz (2 DPC), multiple 10/25/40/100 Gb/s Ethernet (KR) interfaces, enhanced virtualization support (including IO), ARM Holdings Trustzone security and a variety of hardware accelerators. [...] [1]

Is ARM's TrustZone not the same as Intel's ME? At least I remember that AMD uses a ARM Core with TrustZone in it's x86 CPU's to basically achieve the same (or something similar) as Intel's Management Engine.

[1] quoted from: http://www.forbes.com/sites/moorinsights/2016/05/30/cavium-s...

TrustZone is basically the name for the ring -1/hypervisor access level for ARM CPUs, not a separate "engine" or product. It can be used to run and isolate code at a higher privilege level than an operating system eg. for DRM.

> Although given the relative performance,

AFAICT the benchmarks seem to focus on traditional use cases like CPU-bound/memory-bound computation. But the ever-growing use case is the physical hardware that underlies the commercial "cloud computing" focus. IMO much of that market is running I/O-bound code and if I were sourcing hardware for one of those providers I would want to offer a tier which could serve enormous numbers of clients on a single node.

Power consumption is another huge element of data center costs and scaling up ARM-based servers is likely to reap a big savings.

In my experience, many I/O bound applications tend to benefit less from parallelism, but still usually benefit from a fast (single threaded) core - there's often still quite a bit of cpu processing in supposedly I/O bound tasks. As in: you're still better off getting a fast CPU, you just don't need a bazillion cores. Since perf/watt is still poorer in general on ARM cores you'd need to be very very sure that you really don't need any CPU for that to be worth it.

Also; most of these CPU's aren't designed in a vacuum: faster CPUs tend to have better I/O subsystems. E.g. memory latency (which is I/O from a CPU's perspective) is much, much worse on ARM, probably because it hasn't historically mattered as much.

So I'd expect arm to make sense on special purpose I/O related stuff, where you can be sure the limitations aren't an issue. Do some fast e.g. SSDs include ARM chips?

Perhaps

I've been doing it a while with Cavium via Octeon's:

https://www.cavium.com/OCTEON-III_CN7XXX.html

Looked awesome. The ThunderX is the Octeon ported to ARM ISA far as I can tell. The Octeon II's with 16 cores plus Gbps I/O and crypto accelerators in PCI card form can be had on eBay for $200-300. Now, I have performance sheets from Intel confirming it performed as well as I hoped.

Great news for Cavium. ;)

Just for the processor? I'm asking because that's enough to buy an AMD CPU + 8GB RAM + a mainboard + powersupply to get 70% of the multicore performance of a intel 6700k (again the cpu costs more than this setup).

Intel and AMD are market leaders in raw price/performance. They drop mad money on custom designs at cutting-edge nodes to do it, too. The Cavium chips might be custom but probably standard cell. Cost way less. Further, you get 16 RISC cores with no x86 baggage, they run Linux, PCI form factor, and low power. Onboard are usually accelerators for TCP/IP, compression, encryption, and so on that run in Gbps that your processor cores don't have to handle. The CPU cores themselves run pretty fast.

The hybrid seemed like one of the best designs. Probably why AMD started copying it with their semi-custom business followed by Intel. But those are still full-on x86 processors with all kinds of circuits you probably don't need plus maybe some backdoors for AMT, etc. Cavium's design is much more interesting and maybe safer. I've mainly encouraged people to clone it with RISC-V cores rather than straight-up buy them. ;)

The newer ones are pretty impressive in performance and accelerators, though:

https://www.cavium.com/OCTEON-III_CN7XXX.html

I looked at that, but: * AMD no ECC, nor a similar chip with it. Intel has a Xeon E3 which is basically core as the 6700 with ECC enabled. Doesn't hurt that it's cheaper as well. * AMD had 70% of the throughput, but much less on anything single threaded. So many normal UI apps like say web browsing are slower. * the AMD solution runs hotter, so you need to either tolerate more noise or pay more for case/fans/power supplies * My use case includes running 24/7 so burning extra watts (and running the AC to cool them) adds up.

After considering it all I went with the E3-1230v5. Near 6700k multicore performance, near 6700k single thread performance, max 80 watt TDP, and idles much lower. I also get a M.2 slot which allows pretty amazing performance (well above SATA) and a tiny footprint (gum stick or so).

I don't buy systems often though, the machine it's replacing is a quad core 3.0 GHz machines (Q6600) with 8GB ram from 2007. So I don't mind spending a bit more if it's going to last. Hopefully with 32GB ram this new machine will last quite a while.

My machine is about as old given it's a Core Duo 2 with 4GB of RAM. One of the Dell laptops designed for Ubuntu. What do you use yours for where a quad-core 3GHz and 8GB RAM isn't enough now but was a while back? Just curious.

AFAIK, all modern AMD CPUs support ECC. You just need motherboard support.

That was true on the AM3+ Processors with the 890FX and 990FX chipsets. I do not believe it is the case on AMD's modern CPU sockets, but I would be happy to be shown to be wrong.

Here's one more, which I think should be the most competitive one out there:

https://www.apm.com/news/appliedmicro-announces-x-gene-3-the...

http://www.nextplatform.com/2016/04/28/arm-server-chips-xeon...

https://semiaccurate.com/2016/04/25/appliedmicros-x-gene-3-a...

>I had no idea that viable ARMv8 alternatives to Intel's architecture were on the market.

For some definitions of "on the market". The ARM server chips can be hard to get still.

Chips are easy to get. It's all that pesky supporting hardware everyone in the ARM world disagrees on that causes availability issues to end users.

There seems to be at least one incorrect statement in the article.

"The point Intel is making in the chart above is that it has mastered NUMA scaling, which is no surprise since the company has been at it for decades."

SGI was one the pioneers of ccNUMA (Origin 2000?) in the late 90s. That would constitute decades.

Around the same time, Intel delivered the ASCI Red supercomputer using P6 processors with a snooping coherency scheme.

I don't think Intel had significant NUMA experience (at least in any commercial products) prior to AMD K8 (2003), and their first NUMA product would have been IA64 (E8870 chipset?) or the Nehalem microarchitecture chips. That is not quite a decade, let alone decades.

I was under the impression that Itanium had it a while longer, but you're right. Itanium got it even later (2010) thanks to tape-out delays.

The SGI Altix 3000 had ccNUMA in 2003, but that was using SGI's ccNUMA technology to connect Intel's microprocessors rather than anything Intel developed. I believe that other UNIX vendors that adopted Intel's chips used their own ccNUMA technology to link them together too.

Anyway, the article's author likely made the mistake of thinking that multisocket implies NUMA. It is an easy mistake to make for those who do not know the details of how systems are implemented.

SGI did it with MIPS processors in the 90s before they used Intel chips. For example: http://dl.acm.org/citation.cfm?doid=264107.264206

This was the system in which the directory logic was formally verified: http://www.sgidepot.co.uk/origin/compcon97_dv.pdf

I agree the author probably made a simple mistake, but it is misleading to people at the "enthusiast" level who then parrot things like this as flamewar fodder. A lot of respect is owed to computer engineers at corporations that are defunct or a fraction of their former greatness (e.g., SGI, DEC, HP).

As a further example, Bob Colwell's book, "The Pentium Chronicles", is a great read and teaches some powerful lessons about how to successfully do engineering in the kind of large organizations that can scale production of a design. But if you take it at face value, the book makes no meaningful attempt to inform the reader that other companies had done Out-of-Order well before P6. It leaves a naive reader with the idea Intel pioneered OOO as well, which is no way true.

I guess I'm just (maybe overly) sensitive to sloppiness and omission leaving an inaccurate impression of the history, especially when it's so easy these days to fact-check.

My recollection is that "The Pentium Chronicles" said explicitly that OoO had been done, but not for x86 CPUs, and that there was a significant question at the time as to whether it was actually viable for x86.

Possibly the article is worded imprecisely. NUMA by itself is not the desirable property. What you want is multisocket scaling, NUMA is just a way to get there.

Certainly for 2 and 4 sockets Intel uncore (i.e. the connecting fabric) is state of the art and has been for a while (but not 2 decades).

I agree it's a nit, and your interpretation is probably on the right track, but it's an inaccurate statement by the original author as-written.

Also agree that the crux of the point is multisocket scaling. Though with enough sockets (if memory serves, in mid-2000s, the rough consensus was 4), I don't think computer architects think there is any way to scale well without a NUMA approach of some kind, so it's not just an academic point. With today's memory bandwidth and "memory transaction volume", perhaps even 2 sockets wouldn't scale well in a uniform memory access configuration.

With Moore's law slowing down, semiconductors can get commoditized. It happened with DRAM business in 1980s: https://en.wikipedia.org/wiki/Intel#From_DRAM_to_microproces...

Maybe CPU margins are going to drop too? ARM is likely the successor. ARM already won mobile market, maybe it start eating some of the cloud market? Do we really care on which CPU architecture runs S3 servers and etc.? That gives cloud providers huge leverage in negotiations. E.g. AWS can demand more discounts for Intel CPUs and if they won't comply slowing, but surely move some portion of it's infrastructure to ARM.

The problem with comoditization is that margins become razor thin, which in turn means less money to invest in R&D, which means that it is hard for a company that isn't already there to come up with high performance (or even entry level, really) server solutions.

Intel ate the server market in the late '90/eary '00 thanks to the fat margins it had on the desktop.

That's not currently the case in the ARM world were only Apple and maybe Qualcomm have the needed margins, and Apple currently doesn't seem interested in the server market.

If ARM is providing the expensive R&D-based IP, then any licensee can be a competitor.

But AFAIK ARM hasn't provided any server specific design. Most ARM based server attempts so far have been with custom designs (e.g. AppliedMicro X-Gene).

Give it time, why wouldn't ARM start offering server building blocks? There is obviously a market for it. ARM went to 64 bit, and it has just gone to advanced GPU designs. It supports extreme number of cores now too. Server-centric designs can not be far off.

While ARM Holdings reference processors are reasonably good, the company is specialized in low power designs. Apple for example had to do a custom design to get the high performance ARM it needed. It is not obvious that ARM has the knowledge, the money and even the will to pursue a server design.

> Do we really care on which CPU architecture runs S3 servers and etc.?

Not at all, specially when the language runtimes are rich enough to abstract OS and hardware architectures.

Even GPUs can be treated as an "acceleration device" by rich GPGPU runtimes.

> Do we really care on which CPU architecture runs S3 servers and etc.?

Somebody above the "we" in that statement has to care.