> I can't pull out my visa card and go buy a (atx, microatx, mini-itx) format motherboard for an ARM CPU, the CPU itself, RAM, etc, and build a system to run debian, centos, RHEL, ubuntu whatever on.

The difference between ARM and x86 is that there's no standardized ARM interface between the motherboard and the CPU—no ARM equivalent to ACPI. Every integration of an ARM CPU into a board is bespoke. So you have to buy "a board with an ARM CPU on it", not just an ARM CPU and board separately.

But, if you relax that restriction, it's not like it's hard to acquire "a board with an ARM CPU on it." 80% of single-board computers (e.g. the Raspberry Pi) are "a board with an ARM CPU on it." You can wipe pretty much any Android device (not just phones, but also HDMI "streaming boxes", which are a convenient form-factor for basing a workstation on) and install Linux on them. There are also some higher-end development/SDK boards for ARM embedded systems, like the Nvidia Jetson. What more do you need?

> The difference between ARM and x86 is that there's no standardized ARM interface between the motherboard and the CPU—no ARM equivalent to ACPI. Every integration of an ARM CPU into a board is bespoke. So you have to buy "a board with an ARM CPU on it", not just an ARM CPU and board separately.

Not exactly true. ARM server chips do have ACPI. The trick is buying a CPU that is SBSA[1] and SBBR[2] compliant. Then most things work as expected.

[1] https://static.docs.arm.com/den0029/a/Server_Base_System_Arc...

[2] http://infocenter.arm.com/help/topic/com.arm.doc.den0044d/Se...

You still can't have a choice of cpus bought seprately for those boards though.

My first question would be, why has no industry trade group attempted to define a standard socket, or use an existing physical socket pin-out.

It's not hard to acquire something with an ARM CPU on it, but at prices an ordinary person can afford, they're all in the category of toy computers. Try to find an affordable ARM system with a M.2 2280 NVME SSD slot on it like you can find on a $100 desktop x86-64 motherboard. Or multiple PCI-Express 3.0 x8/x16 slots.

I've previously spent many years working for a hardware manufacturer. Personal theory is that this is a real example of a chicken or egg problem related to economies of scale. Nobody wants to spend dozens of millions of dollars tooling up to produce ARM socketed CPUs and motherboards and stuff, which may or may not be price/performance competitive with current-gen Intel and AMD stuff by the time it's ready for release. And there's a huge risk in manufacturing something like that and then discovering that the sales volumes are really low.

Look at the sheer massive quantities that the top-ten Taiwanese motherboard manufacturers churn out every year.

> You can wipe pretty much any Android device (not just phones, but also HDMI "streaming boxes", which are a convenient form-factor for basing a workstation on) and install Linux on them

No you really can't, and phones aren't servers. Take any modern $600 smartphone and try installing something very close to a stock debian or centos on it. Being able to maybe boot a Linux kernel on something doesn't mean that there's anything like the market demand for that particular hardware platform target for a whole distribution.

> My first question would be, why has no industry trade group attempted to define a standard socket, or use an existing physical socket pin-out.

Because ARM is a design licensed to companies that want to make their own chips for their own purposes. Those purposes involve optimizing cost by choosing, per product, whether to integrate various cores onto the SoC, or to leave them as external devices on the board, or to maybe put them into a secondary chipset microcontroller and route some IO pins to connect the two.

In ARM designs, the licensee determines the required pin-out, because the licensee determines what the CPU does vs. what the board does. You can't really have a generic "ARM board", because no two "ARM CPUs" would have the same expectations for what are on that board.

A particular ARM licensee could standardize the socket between their own designs—but in doing so, they'd lose a lot of the advantage of licensing vs. buying an off-the-shelf chip in the first place.

a whole lot of arm chips are too small to reasonably be socketed

These have an M.2 slot. https://www.seeedstudio.com/ROCK-Pi-4-Model-B-4GB-p-4137.htm...

https://www.youtube.com/explainingcomputers is a good YouTube channel for reviewing these types of single board computers.

Has a M.2 slot doesn't mean you can put any M.2 ssd in it, especially M.2 2280 is the most common size and easily can buy on store/online, smaller than that is mostly for OEM devices and you don't have many choices available.

Not about the M.2 slot, but a shout-out to Explainingcomputers. I graduated as an embedded systems programmer, but changed careers along the way. This channel reignited my passion for embedded systems and now i'm an enthusiastic hobbyist again.

An economic viewpoint on your first question is that there is not enough margin (or energy, leverage, etc) left in the system level ecosystem to press for the need for a standard. In the old days Compaq etc very strongly pushed for standards to help them PCs move away from IBM. This began to come undone around the time of EISA, which was ratified but failed, and PCI, which was Intel sponsored. It only got worse after AMD (and other x86 semis) lost ground to Intel.

> My first question would be, why has no industry trade group attempted to define a standard socket, or use an existing physical socket pin-out.

There's Qseven [1] and SMARC [2].

1 https://en.m.wikipedia.org/wiki/Qseven

2 https://en.m.wikipedia.org/wiki/Smart_Mobility_Architecture

Isn't the answer quite simple? There is no demand.

We have reached the point where shipping 1.2B Smartphone per year is called massive, comparing to 200M PC market where 150M of those are laptop ( Soldered and non Socketed ).

You cant buy a self made ARM PC. But there will be Qualcomm ARM Laptop. And the AWS ARM CPU are based on ARM N1 Design, it is only a matter of time somebody make the same CPU, or System integrator selling CPU made from other ARM CPU vendor such as Fujisu, Ampere, Huawei, Marvell, or possibly Nvidia and others.

I dont see much of a problem in getting ARM access as consumers.

You're basically waiting for Apple to go ARM on its desktops/notebooks.

Folks already anticipating this in 2020.

https://www.macrumors.com/guide/arm-macs/

Microsoft, which is collaborating with Qualcomm on netbooks.

Google, which has previously collaborated (OP1) with Rockchip on Chrome OS.

Microsoft also as the Azure Sphere ARM SBC.

Sphere is a special purpose device, not for every day general computing, and I am yet to see when it will be finally made secure as adverstised (given the use of C without any kind of special security tooling).

> no ARM equivalent to ACPI

The ARM equivalent to ACPI is ACPI.

ACPI is not "between the board and CPU", it's "between the OS and the computer". Which is actually much more important. Writing drivers for bespoke PCIe, USB and SATA controllers is not fun. ACPI standardizes configuration of these things.

> ACPI is not "between the board and CPU", it's "between the OS and the computer".

Eh, kinda-sorta. ACPI (through the availability of a DSDT in BIOS flash) provides whatever's running on application processors (like CPUs, but also independent coprocessors like mobile baseband processors and server baseboard management controllers) the ability to 1. query "the board" for what's effectively a listing of the available busses, and the devices on those busses; and 2. to initialize those devices with wired address-space regions on those busses, such that nothing conflicts.

Now, certainly, ARM boards that have standardized "network-like" peripheral busses like PCIe, USB, or (maybe) SATA, have an ACPI controller on the SoC, to schedule those devices' address-space regions onto those busses.

But not all ARM devices have those busses. Some are entirely embedded, with the ARM core serving more of the function of a microcontroller than a true application processor. (There are ARM cores in keyboards and mice!) And some—especially cheap—consumer ARM "platforms" only have busses like SPI and I²C. (You know, like an Arduino!) Even older fully-featured ARM "computers", like smartphones and portable game consoles, didn't support ACPI until very recently, either (which is much of why devices like the 3DS and PSVita didn't support Bluetooth or USB OTG HNP.)

In either case, these devices more-often-than-not don't have ACPI controllers, instead "hard-wiring" each peripheral device on the board to a certain address-space range on a certain bus available to the CPU, just like PCs used to do before there were even IRQ jumpers. (One clear example: every Nintendo portable since the GBA was an ARM SoC, but none of them until the Switch have had ACPI. They just had a static IO memory map, that you could program against.)

And even those ARM devices that do have an ACPI controller on-board, almost universally don't use it the way that x86 does, where even chipset-specific devices like Platform Controller Hubs get probed and configured by ACPI rather than just sitting statically on a special bus.

We're not talking about embedded devices in a server thread :)

I'm not sure what you're calling "an ACPI controller". (You seem to be thinking of MMUs and PICs??)

ACPI is a software interface. The tables are the ACPI. It's possible to implement ACPI on a system that originally shipped with U-Boot and Flattened Device Trees. Say, for the Raspberry Pi: https://github.com/tianocore/edk2-platforms/tree/8b72f720d53...

> The ARM equivalent to ACPI is ACPI.

But most ARM devices aren't using it - their closest equivalent is the device tree.

That's certainly true for ARM devices taken as a whole, but I believe most ARM servers are SBBR and SBSA compliant, which means they do support ACPI.

That is one of the things that Nuvia is promising to change.

> You can wipe pretty much any Android device (not just phones, but also HDMI "streaming boxes", which are a convenient form-factor for basing a workstation on) and install Linux on them.

Nope, the PostmarketOS project is actively trying to make this feasible. It's quite non-trivial and heavily device-specific.

Enlighten me as to which HDMI "streaming box" I can install (mainline) Linux on. Bonus points for a model where the h264 encoding hardware block is supported.

Amlogic and Allwinner SoCs have varying degrees up media decode support in mainline kernels (h264 is supported, other codecs vary). Armbian supports a bunch of such boards with Linux 5.3. What exactly are you trying to do?

We're building libre streaming hardware for https://fosdem.org .

Our current approach is using the Allwinner A20 processor. Investing quite a bit of work in a free driver for the h264 encoding block on there (cedrus vpu).

It was an honest question if we had overlooked something obvious...

Yeah, the Cedrus VPU is already supported upstream: https://linux-sunxi.org/Sunxi-Cedrus

That page is a bit out of date, I think the support has improved since then.

I've not used them. YMMV. But they seem to match your description.

How closely coupled is ACPI to x86?

It used to be quite coupled to x86 hardware, but nowadays they have a profile that replaces a bunch of fixed hardware (such as the ACPI embedded controller) with an ACPI description of GPIO pins, I2C buses etc.

ACPI was a required feature of Itanium systems, claims slide 5 of this Intel presentation from 2000:

https://uefi.org/sites/default/files/resources/ACPI_2.0_Supp...

It isn’t.

If your credit card is massive: https://store.avantek.co.uk/arm-servers.html — both Ampere and Marvell (Cavium) are available.

As a normal person, you're mostly stuck with SolidRun's offerings if you want something that's affordable but serious (i.e. can run PCIe devices).

I run a MACCHIATObin as a desktop with FreeBSD, it's fine, great open source UEFI-ACPI firmware (upstream EDK2), generic ECAM PCIe works (with a quirk but still), but it's not a powerful system. Four A72 cores, single DDR4 channel. It's basically a rather-old-ultrabook level of performance.

Then there's their new thing with the NXP LX2160A 16-core dual-channel chip — there was a loooot of doubt about whether they could achieve good ACPI support on that SoC. But recently NXP have pushed a commit that introduces a generic description of the PCIe controller to the ACPI tables..

Interesting. I wonder if there are any actual niches where these boxes offer a better cost/performance compared to a similar x86 server.

It's perfectly possible to buy an ARM server. We recently got a ThunderX2-based server from Phoenics: https://www.phoenicselectronics.com/catalogsearch/result/?q=...

Would you please tell what the specifications are and how much it cost you?

Not sure if I'm allowed to disclose the price (it's pretty easy to ask them for a quote though), but the config was:

Gigabyte R181-T90 1U ThunderX2

16 x 32GB RDIMM-TX2

1 x 480GB SSD Micron

Did you get them in volume or just the one? Quick searching looks to be between $8k-$10k I'd expect a volume discount could bring it between $6k-$8k

Just one, and that's roughly the right ballpark.

Did some poking around, supermicro found nothing. CDW and NewEgg show no results. Gigabyte has a handful of systems.

https://www.gigabyte.com/ARM-Server

https://www.officedepot.com/a/products/6889935/Gigabyte-R120...

Rockpi, raspberry pi, orange pi all have 4 or 6 core arm64 sbc.

And the linked systems are mostly multiples of 48 cores and hundreds of GB of RAM. Totally different class.

A raspberry pi might be good for very small scale experimentation of little apps, but my 4GB model took months to arrive and doesn't come close to matching the performance of the smallest instance I would use at $DAYJOB.

This is not server-class hardware, which is what's been requested in this thread.

I'ts going that way though. ARM NAS server board with ECC RAM: https://shop.kobol.io/products/helios4-full-kit-2gb-ecc-3rd-...

We mentioned them in the article; keep your eye on Nuvia.

https://www.forbes.com/sites/moorinsights/2019/11/18/nuvia-d...

$20 gets you quad-core headless

https://www.friendlyarm.com/index.php?route=product/product&...

lscpu from that thing:

Architecture: aarch64

Byte Order: Little Endian

CPU(s): 4

On-line CPU(s) list: 0-3

Thread(s) per core: 1

Core(s) per socket: 4

Socket(s): 1

NUMA node(s): 1

Vendor ID: ARM

Model: 4

Model name: Cortex-A53

Stepping: r0p4

CPU max MHz: 1296.0000

CPU min MHz: 120.0000

BogoMIPS: 48.00

NUMA node0 CPU(s): 0-3

For comparison a raspberry pi4 is nominally 270 bogomips

As another data point, an rpi 3B+ 1.3 is only 38.40 BogoMIPS as reported by lscpu:

  pi@raspberrypi:~ $ lscpu
  Architecture:        armv7l
  Byte Order:          Little Endian
  CPU(s):              4
  On-line CPU(s) list: 0-3
  Thread(s) per core:  1
  Core(s) per socket:  4
  Socket(s):           1
  Vendor ID:           ARM
  Model:               4
  Model name:          Cortex-A53
  Stepping:            r0p4
  CPU max MHz:         1400.0000
  CPU min MHz:         600.0000
  BogoMIPS:            38.40
  Flags:               half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32

How did you get the output formatted like this? I tried, but failed.

edit: and somehow omitted the

Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid

For AArch64, there is this [1].

[1] https://www.solid-run.com/nxp-lx2160a-family/honeycomb-works...

That thing is $750 and probably performs much worse than a two year old Ryzen 1500X ($159 at the time) on a $105 motherboard. Nevermind a new Ryzen 3500 which is $194.

And probably close to an order of magnitude worse.

I agree, as someone that's had some "off mainstream" hardware over the years, I'd kind of like to have a 16+ core ARM server to play with; just because that's the kind of dork that I am.

Really though, I think this is other things, if amazon can become independent from Intel and AMD? that gives them huge leverage in terms of negotiations with them. Really, it sets AWS up quite nicely for the time when they are broken up and split out.

> I can't pull out my visa card and go buy a (atx, microatx, mini-itx) format motherboard for an ARM CPU, the CPU itself, RAM, etc, and build a system to run debian, centos, RHEL, ubuntu whatever on.

You can buy a SBC though, so you don't necessarily need to. There's SBC's with more power than the Raspberry Pi (at least up to 3). They just cost more. I'm starting to get into the habit of using more Raspberry Pi's at home for network stuff, and I'm debating setting one up for a web development target platform, something that can be code-pen like but on my Pi. Beats running a bunch of VMs and cheaper than buying a whole rig just for VMs.

What about single board computers?

They lack some important things you need in a datacenter like hardware management platforms. You also can't upgrade components or add interfaces (which at their cost point makes some sense).

SBCs have generally been hobbyist oriented. It would be interesting to see some be datacenter oriented.

I think there is bias against them because they are so cheap. But they are arm64 and with 4 or 6 cores now. Still slow compared to an Intel/amd desktop but just always ssh/sshfs into them from your primary machine. I do this with vscode and it is great. Even better if you primary machine is Linux based.

The main problem that I find with current SBCs is their limited DRAM. Trying to compile stuff I can't keep all cores busy.

On black Friday, I bought a AMD 2600, 64 gb ram and 256 SSD + 6 TB 7200. And a 8 gb GPU.

I paid 1000€, 140€ shipping

Planning to do something similar for a 8 node rpi 4 cluster, depending the results ( 600€ because of PoE)

I'm pretty sure I will come out insanely more cheaper to run my microservice experiments with docker then hosting it in the cloud.

I don't care/talk about ARM currently. Where I work they will go for performance / € in the cloud.

Not many people care about ARM in the cloud. I don't know why the topic of coming from.

Or what I'm missing?

What made you skimp on the SSD? I bought a 512GB SSD for 66€ last year and now the newer equivalent model sells for 57€.

Hmmm, at those prices I would worry about the quality of the SSD. (I've learned to trust a few brands, learned to avoid a few others, the hard way. The 512GB SSDs I'd consider purchasing go for 2X-4X your mentioned cost).

If you're feeling adventurous, you could try running bcache to speed up HDD access with the SSD.

https://wiki.archlinux.org/index.php/Bcache