If you're running a service, then you can use a much trimmer hypervisor, e.g.

https://github.com/siemens/jailhouse

Since the guest unikernel isn't a full kernel, the hypervisor interface is much more minimal, and the few host features it needs can be delegated to the CPU via VT-X (e.g. page table mapping).

At least, that's the dream. (I've never actually used Jailhouse or tried any of the research projects attempting this.)