Brain matter is metabolically very expensive to build, and I assume maintain, so it has to provide a significant evolutionary advantage per mm^3 to make it evolutionarily worth having larger brains.