Interesting. My reading is that there are two components. The first is the Linux bootstrap, which they are entirely in compliance for. The second is vmkernel, which includes some GPL code whose source they're releasing, but vmkernel as a whole is not GPL; this is almost certainly out of compliance (but also easily fixable), unless they're doing something to isolate "vmkdrivers" from the rest of vmkernel.

(Can I legally port, say, the GPLv2 ext4 filesystem driver from Linux to a non-GPL OS like OS X? Can Apple pick up that port, and distribute ext4.kext and its source in OS X, without putting the entire xnu kernel under GPLv2?)

The harder and fuzzier part is that the ESX boot process works by asking Linux to "load" vmkernel, using the vmklinux helper module. If this is in the sense of loading a kernel module, it's (mostly) clearly wrong. If this is in the sense of loading a userspace app (as if vmklinux was a binfmt module, and vmkernel was a non-ELF executable), then this is (mostly) clearly okay. But it's doing neither of those. It's loading vmkernel above Linux, making the Linux process the first world, analogous to a Xen domU. So this isn't userspace, but it's also arguably a new address space. Neither of which are legal concepts per se, so it's hard to tell where this lies.

This fuzzy part is neither the first component (Linux / "vmnix" / the bootstrap) nor the second (vmkernel), but the way in which they interact and are designed around each other. This itself might be an infringement, even if the first is not infringing, and the second is fixed.

vmnix aka Service Console aka Console OS aka the Fuzzy part, was part of ESX, which was discontinued with the 5.0 series of ESX products.

They still use it for bootstrapping, right? I know they've gotten rid of the Red Hat-ish userspace, but according to the LWN article, there's still a Linux kernel involved and a kernel module used to pass control to vmkernel.

(Is the word "vmnix" wrong? Does it apply only to the entire Red Hat-ish distribution in the COS, and not to the Linux stepping stone alone?)

Nope, that stuff is all gone with the ESXi product. The vmkernel can bootstrap itself now.

Basically the old 1.0 - 4.0 process for _ESX_ was like this:

* Bootloader

* Linux Kernel boot

* Load VMnix module.

* Load VMkernel

* hardware control passed to VMkernel

* Linux continues running as a sort of 'super' VM with some limited hardware access.

For _ESXi_ from 4.0 - now :

* Bootloader

* VMkernel

What's curious is that VMkernel's "personality" for ESXi userworlds (which are analogous to sessions IIRC) is it implements the x64 Linux ABI, with some signal/syscalls masked.

I don't know if this was to make the transition for stuff like vpxa from ESX 4 on easier internally, or to make it easier for 3rd parties to develop against, or both.

Is it lazily wrapping some functionality that should be open sourced that fills in where the service console would normally be? Probably not, but I can see why people would be a little suspicious.

To answer the 2nd part of the question:

vmnix was the name of the Linux kernel module that was loaded during the Service Console's boot process.

The terms "Service Console", "Console OS (COS)" were at times used interchangeably and referred to the Redhat distro you logged into at the terminal.

VSphere 4.0 had 2 versions: ESX (with COS) and ESXi (without COS). In 5.0 the COS was removed entirely.