If I have to get the clock to reach 2 cores/units at the same time that is easier then say 4 cores/units. As you increase the clock speed the time to reach all the cores decreases. So if you are wider you would need to reach more cores/units in the same period of time, hence "harder".
EDIT (To make it a little more clear) To make it more clear the voltage change is typically represented in books as a vertical line, but that is not the case it's diagonal and fuzzy. By fuzzy I mean not a straight line but will have some tiny mini downs on the way up.
Different parts of the circuits are going to respond to the up or the down. They are also can vary based on the exact voltage. For example if I have a 5V up one circuit might consider 4.8V to be up and another could be 4.9 or 4.7.
Silicon has improved, but there still is limits of scale based on size, volts and timing.
The other thing is that the M1 processor seems to be a lot wider than contemporary processors. This means it can (potentially) do more operations per clock. Wider processors are harder to clock faster, but it works for Apple. Apple has no desire to put in the cooling you need to get chips running at 4+ GHz, so it's not a big deal if their chips are clock limited. On the other hand, Intel and AMD like their cores to approach 5 GHz at the top end.