The difference is absolutely marginal. The main slowdown sources is mounting huge volumes from the host, that's definitely works better with Linux. And emulating x86_64, if your container does not have arm64 build. But if you don't need it, I'd argue that M1 performance will yield faster containers compared to average Intel laptop.