I don't see why this is necessarily the case. You can boot lightweight VMs with a unikernel in under 10ms [1].
I know it's an edge case that doesn't fit Amazon Lambdas atm. I do it by hand ATM, so I know it can be done, but sure I run it on too many preowned servers to handle peaks. Controlling serverless resources in a more fine grained way is a way to use your servers more effectively.
But if you've got time critical applications maybe serverless is not for you.