There's another point: big heads (whether they contain a big brain or not) are selected against, as they make birth a much more dangerous business that it would be otherwise.