My Docker setup isn't slower than when it was running Arch Linux. The downside is that I have to start Docker Desktop compared to having the docker daemon start at boot. The upside is that VSCode and the browser is 10x smoother on Windows than it was on the same hardware on GNOME, so the whole dev experience feels much better.

RAM usage due to the Linux VMs might be the issue in your case, I have loads of it.