It simply is. Even in a heavyweight VM with lots of hardware support, those hardware drivers are a tiny fraction of the user/kernel interface, no matter how you choose to count it (lines of code, number of foreign calls, number of exposed modules).
If you don't want to derive this axiomatically, fair enough: count vulnerabilities. The tally you're looking for is every Linux LPE versus every Linux KVM escape.
I do think VMs are likely more secure than containers in a cloud environment, because of course at that level you have both problems, but I don't believe that.the number of vulnerabilities found at the VM layer is at all reflective of their actual vulnerability.