I think women liking tall men is pretty much universal which would lead to them getting taller and taller though evolution. As they men aren't all extra tall there are presumably counter factors like tall people needing more food in famines or having back problems I'd imagine. I heard someone say the average Frenchman used to be taller but Napoleon put a lot of the tall ones in his army and then lost them in battle.

Height has definitely transcended other physical traits like physical strength in terms of hunter-gatherer advantageous traits.

Women still prefer many of these physical traits, but probably not as much as wealth, which is linked to intelligence, which comes by default with age.

So you have to factor in what traits are conducive to wealth accumulation, and the easiest and biggest one is probably longevity. If you can be an older, wealthier male with his shit together, that's probably one of the bigger factors over height.

It's important to note that women's preferences is not the only thing affecting evolution - shorter men can be selected due to higher survival rates (for example in jungles, as another commenter pointed out).

Definitely shorter people have higher survival rates, and also a short women would counter the taller man since half of the genes come from the women.

This is just speculation but I think most likely in a Jungle environment where being light on your feet helps during a hunt.