I think women liking tall men is pretty much universal which would lead to them getting taller and taller though evolution. As they men aren't all extra tall there are presumably counter factors like tall people needing more food in famines or having back problems I'd imagine. I heard someone say the average Frenchman used to be taller but Napoleon put a lot of the tall ones in his army and then lost them in battle.