Unless you need x86 compatibility, the M-series machines will be much faster. Check out the old M1 reviews for comparisons to Intel machines. They've only gotten faster since then.

But Xcode's issues are Xcode, not the processor. It will still hang the main thread whenever it wants, just for a bit less time, and your Swift builds will be faster, even if it still doesn't scale linearly with core count.