> and our shiny new M1/M2 Macs run the same binaries

Does that mean you're all running asahi linux, or are you somehow running linux elf binaries on macOS?

Or are you using VMs, just like the x86 + docker desktop folks are doing to run the same binaries on their laptops and servers?

VMs through things like Docker for Mac, in our case.

And yes - it’s exactly the same as you can do if you use x86 servers and dev machines.

It’s just the fact that we can also do this for ARM on hardware that doesn’t suck that makes using ARM in production more practical for us.

Sorry I missed the details:

"Does that mean you're all running asahi linux or are you somehow running linux elf binaries on macOS?" So you're running whatever linux districtuion you want or some particular flavor through Docker on Mac? And you don't have ELF binaries because you're using precompiled packages for ARM and don't need weirdo prebuilt ELFs, or you have a workaround?

Docker for Mac runs a full-blown Linux kernel last time I checked (Docker for Windows can use Windows containers). So whatever OS they run on the cloud (with OCI) these containers are going to work on the (very fast) M1/M2. The OS the Docker images are using is irrelevant. The OS running in the cloud is irrelevant. Only relevant factor is Docker for Mac as it means Linux kernel overhead. But this is offset by M1/M2. Clever setup!

Exactly this.

Plus as the Apple virtualisation framework now implements pretty standard virtual hardware - it uses virtio - we are finding things work vastly better in Docker for Mac - particularly as we can use virtiofs etc to pass files through from the host.

> Docker for Windows can use Windows containers

I thought Windows containers never went anywhere. Interesting to hear that they are still a thing.

Yes, but pretty much in the same sad state as they were a few years ago. We're running a few in production and next time we touch it we're replacing them with Windows Server Core EC2 instances built from baked images.

Note on Linux containers:

Running Linux containers on Windows requires the use of LinuxKit or WSL. Docker Desktop for Windows requires you switch modes between Linux container mode and Windows container mode as you can't run both simultaneously. A workaround is to install an additional Docker daemon inside the WSL environment. Most people are going to install Docker Desktop and use WSL in Linux mode. It's fine. Hopefully my facts are up-to-date.

Windows containers have improved quite a lot in Windows 11.

The interesting thing is, how is Docker for Mac able to run x86 containers at speed when Apple's Developer Documentation for Rosetta says it can't be used for virtual machines?

Or have they just embedded the QEMU on-demand translator with binfmt_misc in the ARM virtual machine and the M1/M2 is just powerful enough to make users not notice what's going on?

[1] https://developer.apple.com/documentation/apple-silicon/abou...

Rosetta can be used to emulate individual processes inside a linux-based virtual machine - Apple provide tooling for this: https://developer.apple.com/documentation/virtualization/run...

We deploy containers to production, and use variants of the same containers for local development - mostly Debian.

Those containers contain a mix of custom built software and packages from the distro.

Just as an aside, the same process works painlessly with podman, too, if your employer is allergic to Docker. Maybe once or twice a year I get reminded that I’m not using x86 - and it sure is nice to work all day and finish with 60% battery.

Is this battery savings from using podman instead of docker for mac or using docker for max with M1/M2?

Running on ARM itself is a big power savings. Docker Desktop also had issues with CPU load which they might have improved since I stopped using it a couple of years ago. There were a bunch of GitHub issues starting around 2018 which were closed, moved, reopened, and it looks like it’s still a concern whereas with Podman I’ve been able to use it without thinking about it.

asahi linux is only needed if you are running on the bare metal. A VM (which abstracts the hardware) means you can run existing ARM64 binaries.

You still need VM to run Linux ELF on macOS, but you don't need to use full-blown emulation like QEMU if the Linux ELF is for AARCH64, unlike when you try to run x86 ELF. It's much faster this way.

I assume he just meant running binaries compiled for the same ISA / processor architecture.

> None of the other cloud providers are using ARM at this scale yet. They’re using Ampere chips, whereas Amazon invested in building their own - it tells you the level of commitment Amazon have here.

James Hamilton was adamant AWS build ARM expertise in-house [0] (one which they accelerated by acquiring Annapurna Labs), and he's been proven right [1] despite the obstacles he himself foresaw [2].

[0] On the Origins of AWS Custom Silicon, https://archive.is/4k5ul

[1] https://mvdirona.com/jrh/talksandpapers/JamesHamilton_CEMS.p...

[2] AWS Designed Processor: Graviton, https://archive.is/WLFBW

> and he's been proven right despite the obstacles he himself foresaw

I wouldn't trust anybody to be right about the benefits of something this big if they don't also foresee the obstacles you'll face to realise those benefits.

Smaller dent then selling trampolin patches that take a already sold good and multiply the performance with a factor of 0.75.

>My employer is moving workloads to ARM on AWS as it’s much better for price/perf

It's a fool's errand to try to reason about Amazon's pricing. We don't have figures about how much a Graviton costs to build or to run, but we do know AWS has staggeringly high margins across the board. They have huge headroom to make price cuts if they want their homegrown solution to win.

And perhaps it is better! I have no doubt it's a good CPU. But we can't infer that from pricing.

It could also be simultaneously true that it's cheaper to rent ARM servers from AWS than it is to rent Intel servers from them, and that AWS makes more money renting you the ARM servers. In fact, given they're developed in house, it might even be likely that the ARM margins are higher for them than their Intel margins.

Quite right - you can't infer performance from pricing.

We did some assessments of how well our applications ran on ARM and x86 based AWS instances, and quite simply it costs us less to serve the same quantities of end user traffic using Amazon's ARM instances compared to the x86 ones.

I think ARM and AMD both are starting to cut deep into Intel's market share. I do think Ampere is a bit higher priced than I'd be comfortable with if I were setting up servers. The AMD 128-core Zen 4c server chip is very impressive in terms of per-core power use and on par on power usage with ARM even.

Incredible amount of computing capacity per u in the racks at this point on both fronts. Intel is betting on custom accelerators, but I don't know how well that fits on a rental (cloud) model unless you get a full server/cpu.