What do you mean mid-optimize?
I'm just saying if you're running on macOS and want to limit resource usage, running it in docker in order to limit resource usage, feels like solving the problem at the wrong layer as you already are using a VM in that case, so just put the resource limitation directly on the VM instead.
Hence the mid-optimization: don't go "the VM's already slow, tweak that", just keep going and tweak Docker, it's almost trivially easy, and trivially replicated on any other machine should you need to either scale or hand off.
It's just as easy as tuning docker. Every VM gets resources allocated to it, that you, as a user, can customize however you want. Most major VM implementations have this for you to use today already. Just as replicable as tuning docker container parameters.