I constantly forget how good Graviton2 is.

E: A64FX seems like a really cool chip, but it sure is much easier to get time on AWS than Fugaku. :)

Does any of these ARM systems work with a mainline kernel, or do they need some old custom kernel with binary blobs?

All server-class Arm hardware works with UEFI + ACPI, as such using regular OS images.

(mandated by Arm ServerReady SR/SBSA, you are not allowed to ship Device Tree there)

Could it be possible we get this too in the workstation and laptop ecosystems?

Windows on Arm64 machines are all UEFI + ACPI too. :-)

(even for 32-bit Windows on Arm, back in the Windows RT times, this was the case. It's a shame that Microsoft decided to lock those down back then)

On more affordable Arm machines, Honeycomb LX2K has an official UEFI firmware (with ACPI) and NVIDIA's Jetson AGX Xavier has an official (but experimental) UEFI+ACPI option.

Those aren't complete yet in mainline when using ACPI. For the LX2K, onboard networking currently doesn't work when using mainline kernels. And for the AGX Xavier, you lose PCIe (until the quirk to support it there is merged, or the ECAM through PSCI patchset) and integrated GPU support.

When using a Windows on Arm laptop, most Qualcomm drivers only have device tree bindings, not ACPI at least at this point. As such, you might need to load a device tree at boot for a more complete experience. (you can look at repos on https://github.com/aarch64-laptops if you have one)

For SBCs, forget about almost anything not named "Raspberry Pi", for which... a third-party UEFI + ACPI firmware is available at https://rpi4-uefi.dev, which allowed the original RPi4 to be certified as SystemReady ES.

A lot of the things that actually drive making people implement UEFI + ACPI is that Windows, RHEL and VMWare ESXi require UEFI + ACPI to boot on 64-bit Arm.

Don't forget about our Marvell series of boards :) While HoneyComb is our flagship for SystemsReady ES certification, We will also have our Armada 8040 and CN913x based products that will also be fully supported UEFI + ACPI systems.

Which distro would you recommend for ARM servers?

For my servers, I use Ubuntu so far. For more workstation/desktop machines, I use Fedora instead.

Yes, the kernel has great support for ARM. Distros that need to package up the kernel, tweak the bootloader, deal with peripherals, etc.... less so on great support. ARM is a bit of the wild west with SoCs defining all kinds of esoteric and exotic hardware configs managed by the kernel's device tree infrastructure instead of a typical BIOS. If you're dealing with server workloads it matters much less (and they probably support UEFI) so stick with a distro that has good support (Ubuntu, Fedora, etc.). If you're dealing with ARM hardware directly like little SBC boards, start getting comfortable rolling your own device tree, bootloader config, etc. as you're probably going to need it. In all cases though the stock mainline kernel is fine to use.