Actually that's not the case. Popes and priests used to have orgies and sex parties; the reason to go into the clergy was for money and power. They had a racket going, collecting money from poor villagers; joining up was how you cashed in. They were FAR from celibate.
http://www.cracked.com/article_15648_five-biggest-badass-pop...
Similar history for Dalai Lamas; they used to go to war, have illegitimate offspring, etc.
In outlying areas where there wasn't much power to be had, you joined up with the local monks/priests if you didn't have other job skills and thus no way to support a wife/family, so it wasn't as if the most promising males were all taking themselves out of the gene pool or anything close to it.
And you don't need a 1:1 ratio of fathers to mothers; a single smart, powerful male can father hundreds or thousands of children, and pass on his "smart genes".
For centuries, what did you do if you were a bright kid in China? Get your balls loped off and become a eunuch. Who needs children when you have the Imperial household to run? Even if you didn't become a eunuch, you at least became a Buddhist monk or something if you were smart.
Europe was pretty much the same thing. Smart people became priests and monks, who, while not completely celibate in the real world, still had much lower rates of reproduction.
On the other hand, in Africa, you gotta be smart, or your ass is lion fodder. Are you a lazy daydreamer with no head for thinking? Too bad, because when the drought comes, only smart dudes will be able to provide for their families through alliances and phony witch doctor magic tricks.
----
That a story like this can be so easily constructed is why I never put too much stock in things like evolutionary psychology per se. It's easy to come up with a fable that "explains" why the world is the way it is. Real history and real facts are much harder to piece together.