The Neanderthals and the Cro-Magnon's had about 1 tennis-ball's volume more brain than we do, on average. If we lose another tennis-ball, we would have brains the size of Homo Erectus. Nobody really knows why, but our brains have been steadily shrinking since the late Paleolithic.

But the thing is that, unlike Neanderthals--which there probably weren't more than 10,000 or so living at once--there are billions of us, and brain size follows the same Gaussian curve. Which means there are those among us who have really big brains.