I believe that is actually a "stupid cavemen" popular myth. From wikipedia:

  Neanderthal cranial capacity is thought to have been as
  large as that of a Homo sapiens, perhaps larger,
  indicating their brain size may have been comparable, as
  well. In 2008, a group of scientists created a study using
  three-dimensional computer-assisted reconstructions of
  Neanderthal infants based on fossils found in Russia and
  Syria, showing that they had brains as large as modern
  humans' at birth and larger than modern humans' as
  adults.[8] On average, the height of Neanderthals was
  comparable to contemporaneous Homo sapiens. Neanderthal
  males stood about 165–168 cm (65–66 in), and were heavily
  built with robust bone structure. They were much stronger
  than Homo sapiens, having particularly strong arms and
  hands.[9] Females stood about 152–156 cm (60–61 in)
  tall.[10]

Of course whales have big brains too and they are pretty dumb, so our mileage may vary ;)

edit: formatting

Thanks burgerbrain. It seems I didn't remember correctly (or did - it's been a long time since I last got really interested in this, perhaps the early 90's). And I also have some reading to do ;-).

OTOH, in regard to the parent post, reason never stopped a racist before. It's doubtful it will now.

Yep, it's not purely the brain size that matters, it's the ratio of brain size to total body mass. And Neanderthals did indeed have larger brains than humans. They also started wearing pigmented jewelry before we did, in fact we might have learned it from them.

I would consider a function of surface and volume, as well as the kinds and "bandwidth" of the senses and complexity of society. You have to process all the sensory input that comes through the skin, eyes, sonar and keep your friends friendly.

Slightly larger: http://newswatch.nationalgeographic.com/2008/09/09/neanderth...