It simply is. Even in a heavyweight VM with lots of hardware support, those hardware drivers are a tiny fraction of the user/kernel interface, no matter how you choose to count it (lines of code, number of foreign calls, number of exposed modules).

If you don't want to derive this axiomatically, fair enough: count vulnerabilities. The tally you're looking for is every Linux LPE versus every Linux KVM escape.