Communication overhead.
Mostly that, but I would add cultural and competence issues as well - more internal politics, and C-teamers sneaking in that drag the team down.
It's easy to make sure everyone in a small team is elite and has value alignment, and it's hard to maintain that as it gets bigger.
What you're describing is negative marginal benefit.