That's not the point. It's the massive amount of neurons and (finely tuned) synapses that allows human intelligence, and the vast amount of possible mental "states".

Functional organization is of course of prime importance, but AFAIK, it's the sheer cortical surfaces that allow us to be smarter than apes (who are able to use symbolic (sign) language, teach it to their offspring, use tools, and recognize themselves in the mirror, all feats that were once thought to be proper to the human species).

We might have reached a local maximum , though(related to the neurotransmission speed?). Neanderthals had a bigger brain, but were seemingly dumber and less culturally advanced than contemporaneous Homo Sapiens.