Well, basic physics -- it already takes nontrivial time for signals to cross the die. If you make the die bigger, you're probably just going to be using that to add more cores, and the time for the furthest-apart cores to communicate is going to grow.

Transmission delays suck, but this is something that the brain has evolved to cope with (consider the speed of an action potential vs the speed of electrons in a CPU and the distances between neurons in a typical mammalian brain vs the distances between transistors in a CPU). So I think there there are strategies we could use to deal with larger silicon, but I do agree that there will always be an incentive to keep things as small as possible.