I mean the performance penalty is just inherent to how Docker on Mac works. Instead of being a container like on Linux, it’s a virtual machine, which will necessarily be slower.

The difference is absolutely marginal. The main slowdown sources is mounting huge volumes from the host, that's definitely works better with Linux. And emulating x86_64, if your container does not have arm64 build. But if you don't need it, I'd argue that M1 performance will yield faster containers compared to average Intel laptop.

Docker Desktop still sucks at least on my corp mac.

I'd recommend you try OrbStack.