VM's tend to lose the overlay layered filesystem which can dramatically reduce disk usage. Having the filesystem reset to a clean state for every new container is a huge feature of containers. And VM's tend to need predefined dedicated resources for things like memory. A process in a container would only allocate memory when it needs and can free up for other processes to use. It's not all about the startup speed.

That said, VM's have their place, and docker has the option to switch out backends. It's entirely possible to replace runc with some other tool that starts VM's instead of containers. (That's already happening today with Windows containers.)