It's not, it has to be heavily integrated into the processor core.

Particularly the main piece of virtualization is an added level of indirection to the page table walking hardware.

There's some IOMMU stuff that can be bought off the shelf (and ARM's SMMU for this is really good actually), but I imagine Apple would build their own (or acquire it) looking at the rest of their IP blocks.

Edit: Actually I'm not even sure if they could pull an SMMU in. Do we have confirmation that they're using AXI/CHI or do they have something else for their NoC protocol?

(Disclaimer: I know much more about x86 virtualization than ARM.)

Apple isn’t really in the server business, and the kinds of high performance VMs that want direct hardware access seem unlikely to run on Apple silicon in the near future. It seems to me that virtualization could work just fine without an IOMMU in this scenario. (Certain GPU workloads would be an exception.)

That being said, I would expect Apple to have an IOMMU at launch for a different reason: Thunderbolt or any other external PCIe connection. Doing this without an IOMMU is a security catastrophe, in contrast to doing it with an IOMMU, which is merely an enormous attack surface that no one secured properly.

Apple Silicon Macs should not only have IOMMU but apparently each device should have its own. They talk about this in the "Explore the new system architecture of Apple Silicon Macs" video [0] (starting at ~9:14).

[0] https://developer.apple.com/videos/play/wwdc2020/10686

I wouldn't count Apple out of the server business for long.

After they have a couple generations of laptop silicon under their belts, there's nothing stopping them from dogfooding a real server macOS for awhile and booting up an Apple Service Cloud.

No special insight into whether they actually will, but it's a natural play.

IIRC the old macOS server had atrocious performance. Building a product that can compete with Linux or FreeBSD for general server workloads is a lot of work. Apple could do it, but the investment might be hard to justify.

There’s also an issue of margins. Apple sells attractive hardware and provides a software ecosystem, and they charge high margins for it. Big server users use a large numbers of servers, and they want a lot of bang for their buck. This is not a game that Apple has historically played very well, nor do I see why they would want to.

Thinking of the same. I was skeptical of ARM Mac Pro, but now Apple is going All in I thought they might as well use those for their iCloud.

Darwin, macOS Server. This sounds fun.

They may have build something based on the A13 for their own cloud needs, and thus why the iPad CPU hasn’t changed and skipped a year.

Apple is really bad at building products which their leadership doesn't want to use personally. Ping and iAd come to mind, server would be the same.

Yet Apple has quite the powerful chip family. If they were to spin off a subsidiary without the consumer-focused mission...well, that's what I would do!

I can read the English, I can Google the lingo, but I'm just a software guy still, and this only parses to me as "there's a bunch of memory management hardware on a CPU that is really important to efficient virtualization, some of it standardized by ARM, some which might make it trivial to support virtualization, but no one knows which standards Apple has settled upon in their silicon".

Is there a more ELI5'ish accessible walk through the context and why's behind this part of the discussion? It sounds really fascinating to me, but I'm not yet equipped to understand it well.

The simple answer is that Apple builds their own CPU cores, so they have to build their own virtualization. Virtualization is not something you tack on around a CPU core, it's something that's part of a CPU core. Since Apple aren't using ARM CPUs, they can't use ARM virtualization. Anyone suggesting otherwise is confused :-)

(Apple might be able to leverage off the shelf ARM technologies that might help with virtualization, but not the core feature of virtualization itself)

Thanks so much, that watered it down for me just right.

So are there any tells/indications that Apple's silicon took this into consideration from the beginning, or is virtualization something they had to retrofit into the core? It would be pretty amazing if say, way back in A1 days, we could point to something in the core that indicated they already started laying the groundwork to make virtualization feasible later.

A1 is not a thing, and the first 64-bit Apple Silicon was the A7, so I don't expect anything to have appeared before then.

Virtualization isn't terribly hard to add to a core, so they could've started thinking about it at any time. It's possible that some of their cores already support it and we just don't know; that would be a very Apple thing to do. The way virtualization on ARM works (at least the way ARM themselves implemented it; Apple could've done something differently) is that there are three execution levels: EL2 (VM), EL1 (guest kernel), and EL0 (guest userspace). So a device that supports EL2 but drops immediately to EL1 on boot to run a normal kernel (without virtualization active) would not necessarily have obvious "tells" that it supports virtualization, unless you broke into the boot process early enough to catch it in EL2.

It would be interesting to break into an A11 device using the checkm8 exploit and see if there is any evidence of EL2/virtualization support on that core.

Here's a fun one though: Apple CPUs did at least at one point support EL3 (that's one level higher, TrustZone), which they used for KPP:

https://xerub.github.io/ios/kpp/2017/04/13/tick-tock.html

Which suggests they might support EL2 and virtualization too. Honestly, I can't find any trustworthy reference claiming that existing Apple Silicon supports virtualization, nor that it doesn't. For all we know it does..

No shipping Apple CPUs support it EL2. KPP/WatchTower was inherently racy/bypassable and has been dead for years, replaced with KTRR which is baked into the silicon itself.

They’re using ARM CPUs but not arm’s CPUs ;)

They are using the ARM ISA on their own custom silicon.

A14 is rumored to have this already, it's not like they don't know how to do this but more that they had no reason to do so until now.

Parallel Desktop demos running linux on Apple Silicon in WWDC, maybe they have internal developer kit based on A14, but it cannot be made public.

What's the reason to do it now?

They’re shipping Macs that need it by the end of the year.

Macs need virtualization to run guest OSes since Macs are big in the pro and developer market. There is no current use for VMs on a phone, so they probably just left that module out in past chips.

Developers are something absurd like 20% of their install base, so Docker gets a first class seat at the table from a product requirements perspective? Albeit in aarch64 mode.

Developers != Someone using macOS as pretty Linux to deploy on Linux.

Also I am yet to see any benefit with docker in Apple platforms ecosystem.

WSL