It's important to note that women's preferences is not the only thing affecting evolution - shorter men can be selected due to higher survival rates (for example in jungles, as another commenter pointed out).