I don't see why this is necessarily the case. You can boot lightweight VMs with a unikernel in under 10ms [1].

[1] http://cnp.neclab.eu/projects/lightvm/lightvm.pdf