In the case of a fork in the chain the network decides the winner based on which fork ends up being longer. If blocks are being mined too quickly then forks become more likely and the state of the chain is more volatile as the current "winner" keeps flipping around.

Blocks being mined too slowly is a more obvious problem: less transaction throughput.