It's not. The limit is how fast you can switch transistors. Electrons can go a pretty long distance inside a clock cycle, and waiting multiple clock cycles for signals to go from core to core is normal and easy.