FWIW-W, Susan Blackmore in her 1999 book, the Meme Machine, notes there appears to be an equilibrium for brain size. Neanderthal had a bigger brain size than modern humans. The larger brain requires more calories, and increases mortality for the mother.

Blackmore, Susan. The Meme Machine, Oxford and New York, Oxford University Press 1999