Compilation is largely memory bandwidth limited - some of the acceleration from cores might be that they have different memory channels rather than more CPUs.

M1 builds a lot faster than Intel CPUs even with fewer cores because of the memory bandwidth.